This is a threat to LLMs, not AI itself. AI models looking for novel cures for diseases (for just one of many examples) are not trained on random Internet text.
this post was submitted on 26 Aug 2024
49 points (96.2% liked)
Fuck AI
1281 readers
1 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
founded 8 months ago
MODERATORS
This is a threat to any neural network that is being constantly trained. Hell it's even a problem with our brain's NN. We just call it "believing your own bullshit" or "getting high on your own supply".
The issue with NNs looking for cures or diseases (or anything that isn't trained off of the internet) is that they are basically out of training data. They'll need orders of magnitude more to get better and we just don't have that. We haven't figured out a way to get better off of less data and there's no real movement on that front either.
What we have right now is essentially a culmination of research that has been going on since the 1960's that was finally able to be realized with us figuring out:
- We can map our NN variables to a matrix
- We can use linear algebra to optimize the loss on that matrix
- We can leverage video cards to crunch the linear algebra
- We have the largest data set ever created in order to get our loss lower than ever before
Completely irrelevant. The title and posted article are talking about unintentionally training LLM text generation models with prior output of other AI models. Not having enough training data for other types of models is a completely different problem and not what the article is about.
Nobody is going to "trawl the web for new data to train their next models” (to quote the article) for a model trying to cure diseases.
AI labeled datasets have this to some extent too if they're not manually checked.