I assume you're speaking of scalloping. That is most often caused by not rotating your tires, and tends to occur most often on your non-drive wheels, yes.
However it usually takes 10,000 or more miles to become obvious unless you have very aggressive tires (offroad "MT" or mud tires) in which case it can set in in less than 5,000 miles on some brands of tires.
In any case, I'd suggest rotating the tires for sure. The scalloping will wear away to some extent as you accrue miles with them on the rear.