It should be split between VRAM and regular RAM, at least if it's a GGUF model. Maybe it's not, and that's what's wrong?
xcjs
joined 1 year ago
Ok, so using my "older" 2070 Super, I was able to get a response from a 70B parameter model in 9-12 minutes. (Llama 3 in this case.)
I'm fairly certain that you're using your CPU or having another issue. Would you like to try and debug your configuration together?
Unfortunately, I don't expect it to remain free forever.
No offense intended, but are you sure it's using your GPU? Twenty minutes is about how long my CPU-locked instance takes to run some 70B parameter models.
On my RTX 3060, I generally get responses in seconds.
It's a W3C managed standard, but there are tons of behavior not spelled out in the specification that platforms can choose to impose.
The standard doesn't impose a 500 character limit, but there's nothing that says there can't be a limit.
Or maybe just let me focus on who I choose to follow? I'm not there for content discovery, though I know that's why most people are.
I was reflecting on this myself the other day. For all my criticisms of Zuckerberg/Meta (which are very valid), they really didn't have to release anything concerning LLaMA. They're practically the only reason we have viable open source weights/models and an engine.
That's the funny thing about UI/UX - sometimes changing non-functional colors can hurt things.
At some point, you lose productivity and reduced work weeks have shown increases in productivity can happen.
My go-to solution for this is the Android FolderSync app with an SFTP connection.
Good luck! I'm definitely willing to spend a few minutes offering advice/double checking some configuration settings if things go awry again. Let me know how things go. :-)