Your mind may be used to hit
CTRL+D all the time, but you won't have a hard time switching over. That's because the "Unselect" will be right in front of you most of the time. You just hit it, and all your selection disappears.
You also have the option to unselect only pieces of stuff, by using the marquee remove tool.