For years he’s been saying things things that white supremacists love to hear: White people are being replaced; legal and illegal immigrants are ruining the US; George Soros hates the west and wants society to collapse (anti-semites love that one); immigrants are dirty, uncivilized; a race war is imminent; anything done to acknowledge racism is racist; BLM protests were violent riots; mass-shooters with racist manifestos aren’t white supremacists; etc.
There are several instances of him associating with and boosting white supremacists online, and even his show’s long-time top writer was pressured to quit after explicitly racist online posts came to light.
This page does a fairly good job of cataloging the sorts of rhetoric he’s used that’re amicable to white supremacists.
My case chiefly rests on his years of white supremacist rhetoric — which you didn’t address. I mentioned those other two bits at the end of my comment. On their own, I agree, they are slightly concerning but not damning; but in the context of his show’s content and rhetoric, they are.