this post was submitted on 19 Aug 2024
74 points (96.2% liked)

Programming

17528 readers
233 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 2 years ago
MODERATORS
snowe
Ategon
[email protected]
74
The Potential Pitfalls of Pagination (jade.ellis.link)
submitted 3 months ago by JadedBlueEyes to c/programming
18 comments fedilink hide all child comments
top 18 comments
sorted by: hot top controversial new old
[–] Deebster 16 points 3 months ago (1 children)

In time-based pagination, the suggested fix to lots of data in a selected timespan is:

simply adding a limit to the amount of records returned (potentially via a query parameter) transparently solves it.

This means clients can't see all the results, unless you add a way to view other pages of data, which is just pagination again. Or is the intended design that clients view either the first x results (the default) or view all results?

The problem with articles like OPs and others is that they don't allow custom sorting, which is often a requirement, e.g. interfaces that present the data in a table, where column headers can be clicked to sort.

[–] JadedBlueEyes 1 points 3 months ago

Regarding your first paragraph, this results limit is per page. To get the next page, you take your timestamp of the last item and use it in from_time, or whatever you've called it. It's still a pagination technique.

Regarding custom sorting, some of the techniques in the article can do this, some of them can't. Obviously timestamp based pagination can't, however the ID-based pagination that I mentioned can.

[–] [email protected] 12 points 3 months ago (2 children)

Fairly lame article but yeah this is something lemmy used to get wrong. I haven't paid attention to whether it still does.

[–] JadedBlueEyes 12 points 3 months ago (3 children)

They fixed this in version 0.19 pr #3872 (note that the cursor here is a way of hiding a post ID to continue from, as far as I can see).

Also, lame article? 😖

[–] [email protected] 14 points 3 months ago (1 children)

Not lame, I appreciated it and as a result will be implementing ID-based pagination, sometime.

[–] JadedBlueEyes 6 points 3 months ago

Thank you! It's lovely to hear it was helpful to someone 😊

[–] [email protected] 7 points 3 months ago (2 children)

Also, lame article? 😖

It's pretty short, it could be improved with benchmarks or at least some examples of how that would be implemented efficiently. Maybe a bit of pros and cons, cursor pagination for example doesn't handle "go directly to page 200" very well. I've also seen some interesting hacks to make offset pagination surprisingly fast with a clever subquery.

I wouldn't call it lame though.

[–] [email protected] 15 points 3 months ago

I mean, for what it's worth, you calling it "pretty short" is what made me go read the article, because quite frankly, my attention span isn't long enough to read through an average-length article for a topic like this.

[–] [email protected] 2 points 3 months ago

Now we're judging articles for quantity instead of quality?

[–] [email protected] 5 points 3 months ago

Ah, thanks for linking that PR. The point made in the article is a good and helpful one, as can be seen by the way so many sites get it wrong, including Lemmy in the past. It's just that it is pretty thin to make a whole article around. It would be a good thing to bring up in an optimization guide that covers more topics.

I would be interested to know if there is substantial cost to keeping cursors open for long periods, while the db is being updated. I think given those labels, it may be preferable to do a new select each time the user requests the next page. But I haven't benchmarked that.

If you look at how Wikipedia pages through article histories, the labels are timestamps with 1 second resolution.

Here is another article about the same topic, part of the author's book on SQL performance:

https://use-the-index-luke.com/sql/partial-results/fetch-next-page

[–] [email protected] -3 points 3 months ago

Who uses "lame" in 2024? It was so pervasive during the Digg times.

[–] [email protected] 5 points 3 months ago (1 children)

Pretty useful article! I'm pretty sure I've seen data shifting happen a few times on Jerboa.

I wonder if uuid7 based pagination solves these issues. Precise enough that you don't overlap/over fetch data with time based pagination, while still being essentially time based so you have a consistent offset. Definitely important to limit the size if requests, though.

[–] Deebster 2 points 3 months ago

Probably not; I'd expect the places where you need something like UUIDv7 (large, eventually-consistent systems) to not be entirely suitable because you can have records added out of sequence. You'd have to add a received-at field - but in that case you may as well just use a standard incrementing ID as your primary key.

[–] [email protected] 2 points 3 months ago (1 children)

lazily loading notifications

...What?

[–] JadedBlueEyes 5 points 3 months ago (1 children)

This whole article was sprung from a discussion of exactly that case, because users often simply don't delete notifications. It's very common for users to have years of undismissed notifications stacked up under the notification bell, and it's not a good experience to load them all at once.

[–] [email protected] 1 points 3 months ago (2 children)

Ah, like smartphone/Windows notifications, not webpage popups? Yeah, always wonder how my dad doesn't miss everything, despite always having 20+ notifications.

[–] JadedBlueEyes 2 points 3 months ago

Yeah, or even the inbox in lemmy. It's a surprisingly common thing.

[–] GetOffMyLan 1 points 3 months ago

Yes when they said notifications they meant notifications hehe