It could be your alternator...but it can also be that the battery is old enough and starting to die off. When the battery gets old, it takes more power from the alternator to keep it topped up (it also doesn't have the capacity it did when new). When this happens, you don't have as much available power for the accessories.
The brushes in the alternator may be on the way out, which could be causing your troubles. Your alternator will put out minimal current at idle. It is probably going to put out the most current between 1900-2700rpm. This will also explain your problem at idle.
Start with the easy things though...check all your power cables and be sure that they are in good shape. Be sure that the ends are not corrorded and that the battery terminals are clean and making good connections. I'm not sure if you have the ability, but putting in new cables in as many areas as you can is a good bet. I replaced the main power and ground connections with 2gauge welding cable. I know it's overkill but I'll never have to worry about current draw issues, and the starter definitely has more than enough power now to turn over in even -30* weather.
I'm not sure what you have for a stereo...but your lights are going to draw about 17A of current at 12V. This plus whatever your stereo is pulling, plus the added load of your heater fan, headlights, etc will tax most electrical systems.