this post was submitted on 23 Jul 2024
32 points (86.4% liked)
Programming
17522 readers
489 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities [email protected]
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
OK so, you are very much right. You should definitely benchmark it using a simulation of what your data might look like. It should not be that hard. Just make script, that creates bunch of files similar to your data. About the trailing white space, when I am in terminal I just use sed to remove the latest '\n' and in rust I just use .trim(), in go I think there is strings.trim(). It is honestly not that hard. The data structure and parser is not formed the same way as the json, where you have to parse the whole thing. So you don't have to. You just open the files you need read their content. It is a bit more difficult at first since you can't just translate a whole struct directly, but it pays for itself when you want to migrate the data to a new format. So if your structure never changes, probably those formats are easier.
Right, it's just kind of a thing to think about. If your program is something that might conceivably be used of sshfs (as an example) ... this is probably not a great option for your program's configuration.
Well a very common thing is to create a "config" object that lives in the long running process (and in some cases can be reloaded without restarting the program).
That model also saves you from unnecessary repeated IO operations (without one off caching and reloading mechanisms) and allows you to centralize any validation (which also means you can give configuration errors on start up).
I do wish various formats were more "streaming" friendly, but configuration isn't really one of them.
In a lot of languages moving between formats is also fairly trivial because the XYZ markup parser parses things into an object map and the ZYK markup writer can write an object map into ZYK format.
Maybe I'm not understanding what you mean by migrating the data to a new format though.
OK so, for example if you have to change the structure of the configuration file, in a statically typed language. You have to have two representation of the data, the old one, and the new one.You have to first deserialize the data, in the old format, then convert it back to the new format, then replace the old files. The FAMF alternative, allows you just to easily use copy and paste and delete to achieve the same goal. Please keep in mind that you can just make configuration data structure that you can keep in-memory. It is just that the representation of the persisted information is spread between different files and not just one file.