Same problem as you, and I initially use a soft horse hair hand brush to remove ALL the dust right after I finish the ride. It's an old hand broom with a handle that I bought ages ago at Home Depot but I see that that style has gotten a lot more expensive now. You can get any shoe shine brush which is horse hair for about four bucks on Amazon but the long handle style makes it so much easier to use I would spend a bit more and get that. It's amazing how easy it is to remove all of that dust with that if you do it right after the ride. It's safe and works better than any other brush I have ever used in the past 40 years of cycling.
Then wipe dust off chain with bike flipped over resting on saddle/handlebars w/ dedicated dry terry cloth wash rag, apply fresh dry lube. Immediately wipe the excess off w/ clean side of the wash rag before it dries, or get too much buildup & more difficult to remove if you're too lazy to use chain cleaner like I am. Wash rag replaced every six months when it is loaded up with lube.
Dedicated T-shirt cloth is used to clean stanchions/dust wipers to avoid cross contamination of wax and oil from chain rag. However this dust is going to get down below the dust wipers and more frequent fork lower and shock servicing is best for these conditions. Plenty of videos on this online and it gets easy to do.
Finally spray pledge furniture polish on 2nd dedicated T-shirt rag & polish the entire bike. I stock up on Pledge when they are on sale. Take care to walk a few steps away downwind when spray the rag, which is easy to do instead of taking everything off the bike to avoid contamination of rotors. Have never contaminated a rotor. Bike turned upside down initially and then flipped upright for polishing the parts that are hard to reach.
This may sound like a lot but it goes very quickly when I get to it right when our group finishes the ride while others are chatting:
Grab bag of bike cleaning gear from truck
Flip bike over
Horse hair brush entire bike quickly
Dedicated rag in its own Ziploc to clean stanchions
Dedicated Terry cloth rag in its own Ziploc wipes dusty chain , apply fresh lube, wipe excess w/ clean side of rag.
Polish frame mostly cause too tired after drive home, can touch up missed areas before next ride
Flip the bike over and done.
10 minutes max.
Join group chat with buddies
Of course if it is raining when I finish riding this gets done at home. Hope this helps.