This is a most excellent place for technology news and articles.
I work with people who work in this field. Everyone knows this, but there's also an increased effort in improvements all across the stack, not just the final LLM. I personally suspect the current generation of LLMs is at its peak, but with each breakthrough the technology will climb again.
Put differently, I still suspect LLMs will be at least twice as good in 10 years.
I hope it all burns.
Oh nice, another Gary Marcus "AI hitting a wall post."
Like his "Deep Learning Is Hitting a Wall" post on March 10th, 2022.
Indeed, not much has changed in the world of deep learning between spring 2022 and now.
No new model releases.
No leaps beyond what was expected.
\s
Gary Marcus is like a reverse Cassandra.
Consistently wrong, and yet regularly listened to, amplified, and believed.
I just want a portable self hosted LLM for specific tasks like programming or language learning.
You can install Ollama in a docker container and use that to install models to run locally. Some are really small and still pretty effective, like Llama 3.2 is only 3B and some are as little as 1B. It can be accessed through the terminal or you can use something like OpenWeb UI to have a more "ChatGPT" like interface.
Welcome to the top of the sigmoid curve.
If you were wondering what 1999 felt like WRT to the internet, well, here we are. The Matrix was still fresh in everyone's mind and a lot of online tech innovation kinda plateaued, followed by some "market adjustments."
I think it's more likely a compound sigmoid (don't Google that). LLMs are composed of distinct technologies working together. As we've reached the inflection point of the scaling for one, we've pivoted implementations to get back on track. Notably, context windows are no longer an issue. But the most recent pivot came just this week, allowing for a huge jump in performance. There are more promising stepping stones coming into view. Is the exponential curve just a series of sigmoids stacked too close together? In any case, the article's correct - just adding more compute to the same exact implementation hasn't enabled scaling exponentially.
Oh no!
Anyway...
I've been hearing about the imminent crash for the last two years. New money keeps getting injected into the system. The bubble can't deflate while both the public and private sector have an unlimited lung capacity to keep puffing into it. FFS, bitcoin is on a tear right now, just because Trump won the election.
This bullshit isn't going away. Its only going to get forced down our throats harder and harder, until we swallow or choke on it.
With the right level of Government support, bubbles can seemingly go on for literal decades. Case in point, Australian housing since the late 90s has been on an uninterrupted tear (yes, even in ‘08 and ‘20).
But eventually, bubbles either deflate or pop, because eventually governments and investors will get tired of propping it up. It might take decades, but I think it's inevitable.
Short on the AI stocks before it crash!
The market can remain irrational longer than you can remain solvent.
A. Gary Shilling
Thank fuck. Can we have cheaper graphics cards again please?
I'm sure a RTX 4090 is very impressive, but it's not £1800 impressive.
nope, if normal gamers are already willing to pay that price, no reason for nvidia to reduce them.
There's more 4090 on steam than any AMD dedicated GPU, there's no competition
AMD will go back to the same strategy they had with the RX 580. They don't plan to release high end cards next generation. It seems they just want to pump out a higher volume of mid-tier (which is vague and subjective) while fixing hardware bugs plaguing the previous generation.
Hopefully, this means we can game on a budget while AMD is focusing primarily on marketshare.
It's so funny how all this is only a problem within a capitalist frame of reference.
I think I've heard about enough of experts predicting the future lately.
The hype should go the other way. Instead of bigger and bigger models that do more and more - have smaller models that are just as effective. Get them onto personal computers; get them onto phones; get them onto Arduino minis that cost $20 - and then have those models be as good as the big LLMs and Image gen programs.
Other than with language models, this has already happened: Take a look at apps such as Merlin Bird ID (identifies birds fairly well by sound and somewhat okay visually), WhoBird (identifies birds by sound, ) Seek (visually identifies plants, fungi, insects, and animals). All of them work offline. IMO these are much better uses of ML than spammer-friendly text generation.
those are all classification problems, which is a fundamentally different kind of problem with less open-ended solutions, so it's not surprising that they are easier to train and deploy.
Marcus is right, incremental improvements in AIs like ChatGPT will not lead to AGI and were never on that course to begin with. What LLMs do is fundamentally not "intelligence", they just imitate human response based on existing human-generated content. This can produce usable results, but not because the LLM has any understanding of the question. Since the current AI surge is based almost entirely on LLMs, the delusion that the industry will soon achieve AGI is doomed to fall apart - but not until a lot of smart speculators have gotten in and out and made a pile of money.
