First, the idea of holding the elbow is that you and the guide are walking with a more natural posture and communication can be streamlined through natural body movement. Give it a try.
Second, I go from gently and politely asking them to let go and explaining I don’t need help, to assertively ordering. It’s great to assume it’s all well-meaning ignorance, but you may end up needing to protect yourself.