Golang

2217 readers

1 users here now

This is a community dedicated to the go programming language.

Useful Links:

Rules:

Posts must be relevant to Go
No NSFW content
No hate speech, bigotry, etc
Try to keep discussions on topic
No spam of tools/companies/advertisements
It’s OK to post your own stuff part of the time, but the primary use of the community should not be self-promotion.

founded 1 year ago

MODERATORS

Ategon

RandomDevOpsDude

austin

Comparing memory usage between node.js object and go map[string]interface{} (self.golang)

submitted 1 month ago by bia to c/golang

4 comments fedilink hide all child comments

I'm fairly new to go and I've recently migrated a in-memory cache from node to go for concurrency improvements, but the memory usage difference between the two are huge. I've tried to read up on the map memory model in go but haven't been able to find a reason for the difference. I can't see that I'm doing anything special, so I'm looking for guidance here.

The documents that are stored are around 8 KB in size as a JSON file. In node the memory usage for 50000 documents stored as objects is 1,5 GB, and for go maps it is 10 GB.

To me, this doesn't seem reasonable but I can't find the source of the difference. Could anyone here give their take on this?

you are viewing a single comment's thread
view the rest of the comments

[–] kamstrup 2 points 1 month ago

Interesting observation! The most simple explanation would be that it is memory claimed by the Go runtime during parsing of the incoming bson from Mongo. You can try calling runtime.GC() 3 times after ingest and see if it changes your memory. Go does not free memory to the OS immediately, but this should do it.

2 other options, a bit more speculative:

Go maps have been known to have a bit of overhead in particular for small maps. Even when calling make() with the correct capacity. That doesn't fit well with the memory profile you posted well, as I didn't see any map container memory in there...

More probable might be that map keys are duplicated. So if you have 100 maps with the key "hello" you have 100 copies of the string "hello" in memory. Ideally all 100 maps qould share the same string instance. This often happens when parsing data from an incoming stream. You can either try to manually dedup the stringa, see if the mongo driver has the option, or use the new 'unique' package in Go 1.23