Selfhosted

GPT4ALL sounds like your best bet. It's one of the easiest to set up solutions at the moment. As best as I can tell none of the open and local options are gpt4 level yet. that's it, there are lots of models to play with and they seem to be getting better very quickly. GPT4ALL makes it pretty simple I specifically linking the models that work and helping you download them.

the one downside is that I don't believe they have implemented GPU models yet. that means things are easy to set up, but it's going to be a slow experience, especially if you don't have a really beefy CPU and lots of RAM.

Unfortunately there isn't any free/open source LLM that's as good as ChatGPT. The ones that are here like Llama 2 and Vicuna 1.5 are decent, but are pretty dumb and love spouting random incorrect info and are really unreliable. The biggest problem isn't even that, it's that hosting one yourself requires a TON of resources. The higher-end models (still not as good as ChatGPT) can need over 20gb of VRAM. For reference an RTX 4090 has 24gb vram.

You might have better luck trying alternatives to ChatGPT. I personally use Claude, it's less censored and has knowledge up to 2023.

Originally posted this to beehaw on another account:

Oobabooga is the main GUI used to interact with models.

https://github.com/oobabooga/text-generation-webui

FYI, you need to find checkpoint models. In the available chat models space, naming can be ambiguous for a few reasons I'm not going to ramble about here. The main source of models is Hugging Face. Start with this model (or get the censored version):

https://huggingface.co/TheBloke/llama2_7b_chat_uncensored-GGML

First, let's break down the title.

• This is a model based in Meta's Llama2.
• This is not "FOSS" in the GPL/MIT type of context. This model has a license that is quite broad in scope with the key point stipulating it can not be used commercially for apps that have more than 700 million users.
• Next, it was quantized by a popular user going by "The Bloke." I have no idea who this is IRL but I imagine this is a pseudonym or corporate alias given how much content is uploaded by this account on HF.
• This model is based on a 7 Billion parameter dataset, and is fine tuned for chat applications.
• This is uncensored meaning it will respond to most inputs as best it can. It can get NSFW, or talk about almost anything. In practice there are still some minor biases that are likely just over arching morality inherent to the datasets used, or it might be coded somewhere obscure.
• Last part of the title is that this is a GGML model. This means it can run on CPU or GPU or a split between the two.

As for options on the landing page or "model card"

• you need to get one of the older style models that have "q(numb)" as the quantization type. Do not get the ones that say "qK" as these won't work with the llama.cpp file you will get with Oobabooga.
• look at the guide at the bottom of the model card where it tells you how much ram you need for each quantization type. If you have a Nvidia GPU with the CUDA API, enabling GPU layers makes the model run faster, and with quite a bit less system memory from what is stated on the model card.

The 7B models are about like having a conversation with your average teenager. Asking technical questions yielded around 50% accuracy in my experience. A 13B model got around 80% accuracy. The 30B WizardLM is around 90-95%. I'm still working on trying to get a 70B running on my computer. A lot of the larger models require compiling tools from source. They won't work directly with Oobabooga.

I found

pi.ai

more reliable than gpt. It seems to have data about recent events

I’ve come across a FOSAI (free open source AI) community a few times - give it a search, there are several getting started type posts.

Non lemmy link

https://lemmy.world/c/[email protected]