I recently read the trilogy, and absolutely fell in LOVE with this type of story. To clarify, to me, the three body problem trilogy felt like a modern take on cosmic horror, while im not looking specifically for horror. I'm looking for the feeling that this trilogy gave me, a sort of feeling of awe and excitement over the unknown. I love the unknown, but especially i love the idea of the unknown in reality, so stories that ground science fiction worries in reality a bit, but also aren't entirely afraid to pull out some less realistic ideas to get across certain points. I am looking for any books that remotely feel like this trilogy in that sense.
I feel like such books would be very uncommon however.
EDIT: Do any of you listen to music while you read? I started recently making specific playlists for reading various kinds of books, so far i’ve made a playlist for reading warhammer 40k books and a playlist for reading cosmic horror and creepy stories in general, i came up with the idea while reading three body problem. after i started listening while reading, it made the experience even creepier, which i loved.
EDIT EDIT: For anyone else making suggestions, since theres a few suggestions i already have in my collection i plan to read here. i will provide a list of books i have in my collection that i plan to read. This isn't all of them, this is just my "cosmic stuff" folder. Some of it is a bit less sci fi and is more just personally interesting for other reasons.
Ballad of black tom- victor lavalle
Blindsight+echopraxia- peter watts (suggested by this thread, already had it, more inclined to read now)
The void trilogy- peter hamilton
Phyl Undhu, Abstract horror, exterminator- Nick Land
The killing star- charles R pellegrino
The revelation space collection- alastair reynolds (newly added thanks to this thread)
The tinfoil dossier trilogy- Caitlin R Kiernan
vacuum diagrams/xeelee sequence- Stephen Baxter (newly added from this thread)
The expanse- james S.A. Corey (haven't grabbed yet, but plan to)
You might enjoy Peter F Hamilton's books Pandora's Star and its sequel, Judas Unchained. It's somewhere between space opera and hard sci-fi but there are significant plots and sub-plots involving alien creatures ranging from the vaguely comprehensible (to humans) through to creatures that are almost beyond our ability to understand.