so glad to find this sub on lemmy. i was trying to find out more about this and didn't want to re-open my reddit! thank you for posting
Edit: unusably slow when i first got it set up but much faster once I got the GPTQ-for-llama set up. Some really good responses.