I am only using the second and third rinse waters (I use the first to help clean my bath tub) so there isn't a lot of soap, and I don't use fabric softener during the warm months of the year so it isn't an issue.

The issue with laundry soap, so far as my (admittedly limited) knowledge goes, is that it contains phosphorus, which impacts the environment. But phosphorus is the P in N-P-K fertilizers, and as I rarely fertilize my garden I don't expect for there to be any resulting problems. Time, as they say, will tell.