It's absolutely possible, though. MPV has it. It definitely takes longer than going forward, and sometimes I have to press the "back one frame" shortcut 2-5 times per frame. But, it does exist.

A simple explanation: Compressed video is typically stored in such a way that you have say, one full frame every 5 frames or so. The in-between frames are just what changed from the previous frame. (Actually, smart compression is adaptive, changing how many full frames it needs depending on what the content is.)

So going forward a frame is easy. The current view is stored in the frame buffer, and you just add the changes to the next frame.

Ah, but how to go back? There are (at least) 3 possibilities.

1. Reverse the process. "Subtract" the last frame data. But this may leave nothing behind, and so probably isn't viable.
2. Double buffer. You have to keep the previous frame around in a second buffer. This adds a new copy operation every frame, and another buffer in memory, but performance for that probably isn't too bad. But you eat that performance all the time just so you can rewind one frame. And to rewind more than just 1 frame, you actually need many past frame's buffered! So something more like a stack of old frames would need to be kept around (like a stack of photos) and you can riffle back through them anytime. We'd throw away old frames past the last checkpoint, but ultimately this is just an increasingly expensive way to do it. This would need to be optional I think.
3. Last option is the classic comp sci tradeoff: if option 2 used a lot of memory, option 3 uses a lot of calculation. Each time you go back a frame, we find the previous checkpoint frame, and then apply each change frame up to where you rewound to. This takes a moment to calculate each time, but unless you're QTE-ing on the frame back button it's probably fast enough!

The best solution probably mixes 2 and 3. Maybe a double or triple frame buffer, with the option to calculate new back-frames as further needed?

Anyway, that was fun to think about!

I do get annoyed, setting the hotkeys & seek distances, to something actually useful, every time I install VLC

In MPV you can litteraly use shaders and upscaler to improve significantly the image.

See : https://github.com/bloc97/Anime4K https://github.com/butterw/bShaders https://artoriuz.github.io/blog/mpv_upscaling.html

VLC doesn't support shaders.

Other great stuff with MPV is change everything, from the UI to keyboard bind. And of course MPV is totaly usable with the command line (is better than use a gui btw, you can see the errors and other stuff).

The number of times I've needed to do something, then realized VLC already did it... Wow

Even had use for the video wall options, a few times.