Genesis 9:20-27. Though, it should be pointed out: it wasn't anything Ham did; it was actually Canaan but for some reason Ham was blamed. But by many Biblical scholars "Ham is the reason people have black skin". It's a "curse". Meaning: it's racism.
Basically, it's a justification for slavery, and only of black people.
I think even the Talmud (sp?) Mentions the significance of it all. Something about Biblegod (or Talmudgod) turning Ham's skin black because he had sex on the Ark, or something.
There are many interpretations. But they all seem to come down to: black people are evil.