"Context matters." That's my credo.
Appearance matters too. But in context.
If you are a Wall Street CEO, a tailor, a socialite, a PR person, or what have you, looking the part is a part of your life. Maybe you just like to look good. Maybe you like to see others look good, don't understand why they wouldnt want to be at their best, and are disheartened or turned off when others appear dishevled and sloppy. No one should fault you for any of that. Your advice would probably be welcomed by others who want to get their looks right too, whether for professional or personal reasons. Maybe also by others who don't know they need the advice.
There is also substance to the idea that you should respect yourself and your immediate environment enough to dress and groom the minimum for the occasion. Politely calling others to task (sometimes hinting is more appropriate) when they do not really can be for their benefit, and sometimes (this may sound pretentious) for the greater good.
But when you start insisting that the garbage man off duty, the minimum wage worker heading to the grocery store, and every other average Joe in the throws of every day life are lesser than if they are not always decked in the latest, greatest, high end gear (or at least look like a Saks catalogue, maybe Macy's), you become something else. Insensitive and out of touch. You become the asshole no one wants to listen to.
There's a ton of circumstances that dictate what's right for different people in different situations. The most gentlemanly thing to do is to consider that with everyone you meet. You are talking to people, after all. Treat them as more than things.