In the 1970s, Cal Poly had the highest population of black students in recent decades. Since then, the number has dwindled.