LocalLLaMA

2640 readers

2 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago

MODERATORS

[email protected]

Models not loading into RAM (lemmy.ml)

submitted 3 weeks ago by [email protected] to c/[email protected]

10 comments fedilink hide all child comments

I didn't expect a 8B-F16 model with 16GB on disk could be run in my laptop with only 16GB of RAM and integrated GPU, It was painfuly slow, like 0.3 t/s, but it ran. Then I learnt that you can effectively run a model from your storage without loading into memory and checked that it was exactly the case, the memory usage kept constant at around 20% with and without running the model. The problem is that gpt4all-chat is running all the models greater than 1.5B in this way, and the difference is huge as the 1.5b model runs at 20 t/s. Even a distilled 6.7B_Q8 model with roughly 7GB on disk that has plenty of room (12GB RAM free) didn't move the memory usage and it was also very slow (3 tokens/sec). I'm pretty new to this field so I'm probably missing something basic, but I just followed the instrucctions for downloading it and compile it.

you are viewing a single comment's thread
view the rest of the comments

[–] gens 1 points 2 weeks ago* (last edited 2 weeks ago)

The programs usually mmap the file into memory. That means that parts of it are loaded as used and unloaded if there is no memory left. That's why it does not say it is using memory. Check disc i/o as it is generating the message. For linux that can be seen in htop or iotop, for win idk.

Note that I use lmstudio, that uses llama.cpp to run models. Gpt4all, I think, uses a modified version of same. Doesn't matter they should all be using mmap to load the file.

PS Depending on the model I also get a couple tokens per sec on the cpu.

Edit: Didn't see someone already said the same, I'l leave this here anyway.