Xournalpp is what you are looking for.

It has a horrible outdated interface, but it makes layerwise edits and preserves the original data of the doc.

Unfortunately not open source, but I have good experience with Master pdf Editor. It "is a proprietary application to edit PDF documents on Linux, Windows and macOS. It can create, edit (insert text or images), annotate, view, encrypt, and sign PDF documents.": https://www.linuxuprising.com/2019/04/download-master-pdf-editor-4-for-linux.html

I haven't noticed that w/ LibreOffice Draw and I'm quite happy with it

To the best of my knowledge, there isn't a good FOSS PDF editor.

There's LibreOffice Draw, but as you said, it messes up the formatting. There's also Inkscape, but good luck if you have pages of text. You can also try Scribus, but I wouldn't say it's good (or intuitive).

If all you want to do is write over a PDF, put (not edit, nor remove) text, formulae, or images, then Xournal++ is very decent.

If I need to edit a PDF, I use (*gags*) Adobe Acrobat running in Wine (with Lutris is very easy). It's as proprietary and evil as it can be, but it's good at editing PDFs.