Sounds like a bug, but I'd start by making sure to change the default audio output, and not the app specific audio output. I don't know what desktop environment you're using, but you can start by installing pavucontrol
and set the fallback/default device in the output devices
tab. Then when you need to change the output, change the fallback again.
You can also make changes with key bindings using the commandline app.
Since you're using Nobara, I'll assume it's using pipewire and not pulseaudio. If the problem turns out to be a bug, you can post a bug report here: https://gitlab.freedesktop.org/pipewire/pipewire/-/issues