Yes, it is true that many volunteers didn’t own slaves, but that was because they were young men. They did, however, live with family members who owned slaves or worked with older men who owned them. This quote below is about volunteers for the Army of Northern Virginia:
volunteers in 1861 were 42 percent more likely to own slaves themselves or to live with family members who owned slaves than the general population.
The attachment to slavery, though, was even more powerful. One in every ten volunteers in 1861 did not own slaves themselves but lived in households headed by non family members who did. This figure, combined with the 36 percent who owned or whose family members owned slaves, indicated that almost one of every two 1861 recruits lived with slaveholders. Nor did the direct exposure stop there. Untold numbers of enlistees rented land from, sold crops to, or worked for slaveholders. In the final tabulation, the vast majority of the volunteers of 1861 had a direct connection to slavery. For slaveholder and nonslaveholder alike, slavery lay at the heart of the Confederate nation. The fact that their paper notes frequently depicted scenes of slaves demonstrated the institution’s central role and symbolic value to the Confederacy.[link]