He was suggesting with his comments that those folks are behind that desk because they are black. And you're right, it is not really racist to say such a thing. He just disregards their talent when he makes those remark.
And, why can't the panel be all black? Why comment? What's so weird about it? The Basketball court is pretty much all black, no one cries racism? Maybe they're there because of their talent. They just happen to be black. Schaub ignores all this and chooses to make divisive remarks that are in these times considered sensitive.