There are many libraries and frameworks that take the grunt of game dev, without locking you to their specific physics engine, etc.
For example
2D: SFML, Easily create a window and move sprites around, rest is up to you. (C++)
3D: JMonkeyEngine (Based on lwjgl) is basically the same, with 3D rendering instead. It contains tools for physics and collisions, but those are optional. You can do whatever with meshes in 3d space (Java)
I probably should have realized this before fighting with complete game engines (unity, godot) for multiple years even though I prefer doing stuff myself. I find it easier to handle when I've made it myself.
Anyone else know any minimal but feature-rich engines?