The OOM killer is usually triggered after it starts hitting the disk. Which means your system is unresponsive for a long time until it finally kills something.

Using something like oomd can help trigger before it hits swap but then why are you using swap in the first place?

The bigger issue is that the kernel sometimes ignores the swappiness and will evict code/data pages long before file cache even when set to 0 or 1. I'm still not sure if that was because of an Ubuntu patch or if it was an issue that's been resolved in the years since I last saw this