I think even if I really hated them, I'd still give it back. When you loose your wallet, you loose everything...money, credit, liscense, photos(maybe), social security card, and loads of other stuff that's a pain in the arse to replace. Even if you hate them, they probably don't live that much different from you and probably relies on a lot that's in their wallet to get by. I wouldn't want someone to stress out like that.
Karma is also another reason I'd give it back. If I lost mine, I'd really hope someone would return it if they found it. It's just the right thing to do.