A little bit of noise in neutral is normal, according to the service manual "Gear rollover noise, inherent in manual transmissions, is caused by the constant mesh gears turning at engine idle speed, while the clutch is engaged and the transmission is in neutral..........Gear rollover noise will disappear when the clutch is disengaged or when the transmission is engaged in gear. Release bearing rub will disappear when the clutch is engaged. In the event that a bearing is damaged, the noise is more pronounced while engaged in gear under load or coast than in neutral."
Does not sound like a pilot bearing issue to me. Could be a heat shield on the exhaust, something loose in the engine bay, behind the dash, damaged or missing shifter bushings, etc... lots of possibilities.