Actually,I think the answer is a qualified 'yes".
Now, I imagine as with just about every other feature in a biological organism there's a degree of variation. That might account for some of why people prefer certain colors over others. But speaking as an art major, I can tell you that the vast majority view colors more or less the same.
The reason I say that is the good old "color wheel". See, certain colors go well with certain other colors. Others clash horribly with the same. And those examples are pretty well universal. If you look on a color wheel:
http://www.makart.com/resources/artclass/cwheel.html you'll find that the same colors are 'complimentary' regardless of who the viewer is.
If, for example, I saw purple when you saw yellow, those color combinations would be wildly different for me than they are for you.
Also, people who describe colors using metaphor tend to come up with the same terms over and over again. Blue is cool, red is hot and vibrant, yellow is cheery, green is calming etc...
I think, for the most part, we're all seeing the same thing.