Baylan's too young. If we assume he's about the same age as Ray Stevenson was (59) he'd have been around 8 or 9 years old when Anakin was born.
I think Baylan was disappointed that Anakin never really "restored balance to the force" the way he had hoped. He's made it clear he liked what the Jedi Order once represented, not what it had become and was probably hoping Anakin would be the one to finally end the destructive light vs. dark cycle and break the Jedi order away from its dogmatic inflexibility back to being a force for peace. Instead, he just wrecked everything, joined forces with Palps and set up another Light vs Dark cycle. Baylan wants to break that cycle.