When engine idle is at 700 or above alternator should show 13.5v or higher at battery.
Below that RPM it is basically shut off so you would read battery voltage 12.3v to 12.8v, and with a 1 volt difference lights would look dimmer.
Manual trans idle is 700, automatic 800, that's warmed up.
Cold idle is based on outside/engine temp at startup, 1,100 is about average.
If your RPMs drop below 700 then yes lights would dim a bit, if an alternator has a field that has failed lights may even dim at normal idle.
Alternator has 3 fields that make AC voltage, it is not unusual for one to fail over the years, but the other 2 can pick up the slack for the most part, just not fully at idle.
An alternator won't cause RPMs to drop, if alternator bearing was bad the belt would start to slip, so you would hear squealing, an alternator simply can't cause enough drag so slow down 100+ horse power engine.
So your issue is the erratic idle, could be a vacuum leak or dirty IAC(idle air control) Valve.
In normal start up computer opens IAC valve all the way while cranking engine, when engine starts RPMs should go up to above 1,500, then computer will start closing the IAC Valve to set idle for engine temperature, 1,100 cold and 700 warm(or 800).
If your engine doesn't start like this then clean the IAC valve first, it isn't hard to do.
Anther thing to do is to warm up engine and unplug the IAC Valves wires(Valve closes all the way), idle should drop to 500, engine may even stall, either is good, it means no vacuum leak.
If idle stays high, you have a leak.
PCV hoses on the 4.0l were common leak points, especially the elbows, underside would wear thru.