I bet you could try carefully hand glazing on the lichen... or Windex? Not sure I never had this problem yet. Must be a way to get it off without scratching the paint.
I'd try various approaches, maybe have a hose handy running slow and rinse it off as it removes. Anyone who's done much of this will know that if you get a clump of grit of any kind you can put a scratch in it, that's why with finer grits you are always wet sanding and rinsing things that way it doesn't happen.
The glaze works really nice on the last coat of alkyd paint I use on the instruments, that's actually why I got it, but it's made for vehicles. After you use it you can feel the difference in smoothness as well as see it.
Edit: maybe bleach but I'd go carefully on a trial spot. I got a replacement seat belt for the '99, it was greasy and grody and full strength bleach made it like new. It won't attack the nylon but gets all the crap out. I guess I simple-greened it first but that worked only so-so and bleach really cleaned it nice, just about 15 minute soak was all it took.