Since this issue is most prevalent while browsing new, you could temporarily store (a hash of) the titles and contents of each post during a given browsing session, and if a post matches another post on the same page (e.g. one set of posts from Everything/new), hide it and allow the user to see comments from each instance of the post when they view the canonical (earliest, or most active, or returned first) post.
If another duplicate post is later loaded (through scrolling) the post could be hidden and replaced with a smaller indicator telling the user a duplicate post has been detected, and they can click it to view the comments.
This is really exciting! It's been interesting watching updates from this project; would love to try it out someday.