It's actually a great opinion.
In racing a red flag doesn't mean stop and call the whole thing off. It's a chance to reset and see if/when it's safe to continue.
Just because you see a red flag doesn't mean someone thinks of you as less, It doesn't mean they're harmful or dangerous. That's your assumption based on whatever raised that flag for you. A lot of people might not even see the same thing as a red flag.
Sure, you're probably right most of the time and they're not worth the energy. If you shut them down straight away though, without investigating, you might end up missing more green flags than red.
And just never race again?
Sure that solves the one issue you had that one time at one track but would that be worth giving up racing altogether? What about the track you've never had an incident on? What about your favourite track to race on? What about all the new tracks you'd miss out trying? Would you give up driving altogether because of it? There's still risk there. That's a lot of fear and sacrifice for nothing more than a potential reason things could go bad.
Let's move away from the metaphor though. There's always the risk anybody could go out of their way to harm you. That friendly neighbour, the nice guy at work or even close friends and family.
I'm not saying ignore the red flags, especially any serious ones. There's a guy at my work who is very chatty, needy and desperate for my attention. Those are red flags for me so he doesn't get to find out where I live or socialise or go to the gym or anything like that because he's likely to just turn up. I'm not going to stop being friends with my colleagues because of one person though. Some of my best friends are current and ex-colleagues, I'd have missed out on some great people if I let one person/incident put me off.