Large Language Models

187 readers

1 users here now

A place to discuss large language models.

Rules

Please tag [not libre software] and [never on-device] services as such (those not green in the License column here).
Be useful to others

Resources

github.com/ollama/ollama
github.com/open-webui/open-webui
github.com/Aider-AI/aider
wikipedia.org/wiki/List_of_large_language_models

founded 2 years ago

MODERATORS

[email protected]

How to use GPUs over multiple computers for local AI? (lemmy.world)

submitted 6 days ago by [email protected] to c/[email protected]

1 comments fedilink hide all child comments

cross-posted from: https://lemmy.dbzer0.com/post/41844010

The problem is simple: consumer motherboards don't have that many PCIe slots, and consumer CPUs don't have enough lanes to run 3+ GPUs at full PCIe gen 3 or gen 4 speeds.

My idea was to buy 3-4 computers for cheap, slot a GPU into each of them and use 4 of them in tandem. I imagine this will require some sort of agent running on each node which will be connected through a 10Gbe network. I can get a 10Gbe network running for this project.

Does Ollama or any other local AI project support this? Getting a server motherboard with CPU is going to get expensive very quickly, but this would be a great alternative.

Thanks

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 3 points 6 days ago

You’re going to need a whole lot more than 10Gb, you just need a way better motherboard and chip. Get the AMD Threadripper and an ASUS Pro motherboard.