If everything on it looks good (bushings, frame straightness, etc.), the wheels are straight, and swapping the tires side-side made no significant difference, then most likely your issue is insufficient caster angle on the passenger side (it's also possible you have too much caster angle on the driver side).
If you don't have the skill to sort it out on your own, then I would suggest try taking it to a 4x4 shop, off road race shop, or some place that regularly works hands-on with modified suspensions (who would be more likely to recognize if there's an issue, and where it is). The turnover rate at chain shops is usually much too high for anyone to learn beyond what their training materials might've told them (and for decades now it's also been apparent that these training materials have little (if anything at all) about Ford's Twin-Beam suspensions, so most of them are just guessing at it).