.. Are there other fruit trees growing in this area? I'd assumed this was the broken, eroded, or crumbled form of 'rock'; but if it's compacted, it would act as any other type of 'hardpan' or impenetrable layer or strata... And it's on a slope - is it a 'ledge,' or outcropping of this material? If so ... that might limit your fruit tree growing in that location.
If you can eventually break through it, and there's a backhoe
or small track-hoe
in your neighborhood, I'd be inclined to hire one to come into the orchard area and dig up some good sized holes. --- Deep enough to get through the saprolite, and wide enough for the trees to establish themselves - as they'd hopefully penetrate the surrounding material.
Most 'hardpan' conditions I've known of have required such heavy equipment to break it up. If you were to stake out the locations of your envisioned trees, this could be a one-time disruption and expense. What have your neighbors done? --- My clay's sounding better already