Programming

19258 readers

42 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 2 years ago

MODERATORS

snowe

Ategon

[email protected]

YAML, SQL, or something else? Looking for recommendations for making a database of stories. (lemm.ee)

submitted 3 days ago by [email protected] to c/programming

39 comments fedilink hide all child comments

Short version of the situation is that I have an old site I frequent for user written stories. The site is ancient (think early 2000's), and has terrible tools for sorting and searching the stories. Half of the time, stories disappear from author profiles. Thousands of stories and you can only sort by top, new, and 30-day top.

I'm in the process of programming a scraper tool so I can archive the stories and give myself a library to better find forgotten stories on the site. I'll be storing tags, dates, authors, etc, as well as the full body of the text.

Concerning the data, there are a few thousand stories- ascii only, and various data points for each story with the body of many stores reaching several pages long.

Currently, I'm using Python to compile the data and would like to know what storage solution is ideal for my situation. I have a little familiarity with SQL, json, and yaml, but not enough to know what might be best. I am also open to any other solutions that work well with Python.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 13 points 3 days ago (11 children)

I would scrape them into individual json files with more info than you think you need, just for the sake of simplicity. Once you have them all, then you can work out an ideal storage solution, probably some kind of SQL DB. Once that is done, you could turn the json files into a .tar.zst and archive it, or just delete them if you are confident in processed representation.

Source: I completed a similar but much larger story site archive and found this to be the easiest way.

[–] [email protected] 3 points 3 days ago (10 children)

That's a good idea! Would yaml be alright for this too? I like the readability and Python styled syntax compared to json.

[–] towerful 3 points 3 days ago (1 children)

I see no reason you can't use yaml.
Yaml and json are essentially identical for basic purposes.

Once the scraper has been confirmed working, are you going to be doing a lot of reading/editing of the raw data? It might as well be a binary blob (which is a bad idea as it couples the raw data to your specific implementation)

[–] [email protected] 1 points 3 days ago

I'm not entirely sure yet, but probably yes to both. The story text will likely stay unchanged, but I'll likely experiment with various ways to analyze the stories.

The main idea I want to try is assigning stories "likely tags" based on the frequency of keywords. So castle and sword could indicate fantasy while robot and ship could indicate sci-fi. There are a lot of stories missing tags, so something like this would be helpful.

load more comments (8 replies)