1. When you are a hermit, no one comes to your house, so they don't see the cat hair on the couch or the drips of spaghetti sauce on the stove top or the crumbs on the counter and think "TSK. Doesn't she ever tidy this place?"
2. When you are a hermit, you don't see anyone, so they don't look at your clothes and your hair and think "TSK. There's a stain on the collar of his shirt. And his hair is much too long. Would it kill him to get a haircut?"
3. When you are a hermit, you don't interact with anyone, so you don't have to be afraid of accidentally offending someone or hurting someone's feelings, as if you were some sort of lumbering cave troll who backs into someone's favorite heirloom vase and sends it tumbling to its doom.
4. When you are a hermit, you don't tell anyone what you do, so you can spend your time as you really honestly like. If you feel gardening that day, you can spend the whole day covered in dirt. If you feel like sitting in the sun reading a book, you can do that without someone thinking, "What a lazy bum. Doesn't she know there is grass to be mown, weeds to be pulled, leaves to be raked and plants to be watered?"
When you are a hermit, you are insulated from judgment. Sure, the people in the town 40 miles away, where you go to get supplies think, "Strange old duck. Half the marbles have rolled away," but they only think that on the day you go into town, every 3 months or so. And the rest of the time, you're free to be yourself.