I tried a bunch of different things, finally what I did is this and I know it sounds ridiculous.
I make Windex out of water and about 15% isopropyl alcohol, with a half drop of Dawn (and a few drops of blue food coloring). I spray this on then scrub around the diamonds with a toothbrush which loosens up the grime, then I sop up the liquid to get most of the debris, and polish it with paper towels, being careful to not leave debris at the base of the diamonds which is where it collects. Kind of a tedious process but working quickly it doesn't take too long, then some places I have to go over a second time, but eventually it all comes off. At the end I spray wax it which probably doesn't do much, but I see water beading up on it this morning so maybe it helps a little.
I tried scubbing with a large brush but it didn't do better and it tends to make stuff go places you don't want.
The WIndex won't hurt paint though, in my experience, it might want to take off wax but no biggie even if so.
I try to get most of the crud off before polishing/wiping because aluminum is soft and if you have a grit there you can scratch it pretty easily. Mine was mostly just real fine black dust that was stuck in there.
Seems to help in the overall cosmetics None of the other things I tried like Noxon or Brassite or hand glaze or anything seemed as good.
3M hand glaze does help a ton with paint though it's safe for clear coat it's just super fine, for that I use soft cloth to polish. It cleans up fine scratches and leaves a nice smooth surface for waxing. I'm gradually doing the whole body; I did the roof which as you might expect was the worst and it really shined it up nice.
Not familiar with alumabrite but will check it out.