The booster acts on a vacuum line from the engine, so if there is a leak you are loosing vacuum, which can affect idle and fuel suction (the suction is actually caused by the intake cycle of the piston so if there is a leak, less or no fuel gets taken into the cylinder for compression/combustion)....
Check for other disconnected or broken lines on the engine...such as those into the intake side of the engine...the pcv line is a prime suspect on later models, but carbed engines have many vac lines to check sometimes...\
And you can check the tube going from the vac tree to the booster...look for leaks or loose connections (i use hose clamps on both ends) and that the plug going into the booster is tight...also make sure that there is slack in the tube because if the engine rocks it could yank the tube out of the booster...
I've replaced boosters for the same noise, only to find out it was a vac line problem...