I found the problem. The vacuum ball, which is between the inner fender and the front fender, was broken in half. This caused a large vacuum leak which of course contributed to a rough idle and the fact that my AC didn't work.
I tried gluing it back together, but it still leaked. So, my solution? I removed it completely and hooked the vacuum lines together. No more vacuum ball, no more leak, and my A/C works great.
For future people who are trying to search this, there is one vacuum line near the A/C evaporator box that feeds both the 4wd hubs and the vacuum-operated A/C stuff. It goes into that "ball" which stores vacuum for when you mash the gas or other situations where there is no vacuum available.
The way this ball works, it has two bolts in it below the airbox and if they flex just right, the ball will crack and leak.
You will need to remove the air filter box (pull it really hard. It won't break, trust me!). and unbolt a bunch of rusty crap to get to this thing. You'll also want to remove the washer fluid/antifreeze tank.