Ask Lemmy

Citation needed that LLMs are passing these tests like they’re nothing.

LLMs don’t have intelligence, they are sentence generators. Sometimes those sentences are correct, sometimes they’re gobbledygook.

For instance, they fabricate real-looking but nevertheless totally fake citations in research papers https://www.nature.com/articles/s41598-023-41032-5

To your point we already know standardized tests are biased and poor tools to measure intelligence. Partly that’s because they don’t actually measure intelligence- they often measure rote knowledge. We don’t need LLMs to make that determination, we already can.

Standard tests don’t measure intelligence. They measure things like knowledge and skill. And ChatGPT is very knowledgeable and highly skilled.

IQ tests have the goal of measuring intelligence.

All standardized test is how well you prepared for that particular standardized test, doesn't matter if it is the SAT, MCAT, or Leetcode. You aren't suppose to think on the spot for these tests, you are suppose regurgitate everything you have rehearsed for weeks and months during the test.

And unthinking regurgitation is what LLMs do better than anything else.

There has been plenty of proof that standardized testing doesn't work long before ChatGTP ever existed. Institutions will keep using them though because that's what they've always done and change is hard

When I was at uni lecturers would often state that exams were thr worst measure of grasping the subject material but its all we have at the moment.

I saw this my self with some of my class mates testing very well but when discussing or problem solving outside of the class there was nothing there.

I think llms fall into this category but with way better recall.

I think it highlights how a lot of these exams are just about the amount of information one can memorize.

Standardized tests were always a poor measure of comprehensive intelligence.

But this idea that "LLMs aren't intelligent" popular on Lemmy is based on what seems to be a misinformed understanding of LLMs.

At this point there's been multiple replications of the findings that transformers build world models abstracted from the training data and aren't just relying on surface statistics.

The free version of ChatGPT (what I'm guessing most people have direct experience with) is several years old tech that is (and always has been) pretty dumb. But something like Claude 3 Opus is very advanced at critical thinking compared to GPT-3.5.

A lot of word problem examples that models 'fail' are evaluating the wrong thing. When you give a LLM a variation of a classic word problem, the frequency of the normal form biases the answer back towards it unless you take measures to break the token similarities. If you do that though, most modern models actually do get the variation completely correct.

So for example, if you ask it to get a vegetarian wolf, a carnivorous goat, and a cabbage across a river, even asking with standard prompt techniques it will mess up. But if you ask it to get a vegetarian 🐺, a carnivorous 🐐 and a 🥬 across, it will get it correct.

GPT-3.5 will always fail it, but GPT-4 and more advanced will get it correct. And recently I've started seeing models get it correct even without the variation and trip up less with variations.

The field is moving rapidly and much of what was true about LLMs a few years ago with GPT-3 is no longer true with modern models.

LLMs have a good time with standardized tests like SAT precisely because they're standardized, i.e. there's enough information on the internet for them to parrot on them

Try something more complex and free-form and where a human might have to work a little more to break it down into actual little subtasks with their intelligence - and then solve it, LLMs in the best case scenario will just say they don't know how to do it, and in the worst case scenario they'll hallucinate some actual bullshit.

Ask an LLM to explain a joke. It often won't understand why a joke is funny, but that won't stop it from trying to give you an explanation.

Those tests are not for intelligence. They're testing whether you've done the pre-requisite work and acquired the skills necessary to continue advancing towards your desired career.

Wouldn't want a lawyer that didn't know anything about how the law works, after all, maybe they just cheated through their classes or something.

Eh, yes and no. It might help illustrate the limitations of testing for some people, but it's not really telling us anything new about them. It is meant to cheaply provide an indication of how a student is fairing and has never been considered by anyone serious as some kind of comprehensive measure of intelligence. Their flaws have been known for a long time.

We use standardized tests because they’re cheap pieces of paper we can print out by the thousands and give out to a schoolfull of children and get an approximation of their relative intelligence among a limited range of types of intelligence. If we wanted an actual reliable measure of each kid’s intelligence type they’d get one-on-one attention and go through a range of tests, but that would cost too much (in time & money), so we just approximate with the cheap paper thing instead. Probably we could develop better tests that accounted for more kinds of intelligence, but I’m guessing those other types of intelligence aren’t as useful to capitalism, so we ignore them.

Everyone knew this.

Obviously 1:1 mentoring, optional cohort/Custom grouping, experiential, self paced, custom versioned assignment learning is best but that's simply not practical for a massive system.

Intelligence cannot be measured. It's a reification fallacy. Inelegance is colloquial and subjective.

If I told you that I had an instrument that could objectively measure beauty, you'd see the problem right away.

such tests are not standardized tests of intelligence, they are standardized tests of specific-competencies.

Thomas Armstrong's got a book "7 Kinds of Smart, revised", on 9 intelligences ( he kept the same title, but added 2 more ).

Social/relational intelligence was not included in IQ because it is one that girls have, but us guys tend to not have, so the men who devised IQ .. just never considered it to have any validity/significance.

Just as it is much easier to make a ML that can operate a commuter-train fuel-efficiently, than it is to get a human, with general function, to compete at that super-specialized task, each specialized-competency-test is going to become owned by some AI.

Full-self-driving being the possible exception, simply because there are waaaaay too many variables, & general competence seems to be required for that ( people deliberately driving into AI-managed vehicles, people throwing footballs at AI-managed vehicles, etc, it's lunacy to think that AI's going to get that kind of nonsense perfect.

I'd settle for 25% better-than-us. )

Just because an AI can do aviation-navigation more-perfectly than I can, doesn't mean that the test should be taken off potential-pilots, though:

Full-electrical-system-failures do happen in aviation.

Carrington-event level of jamming is possible, in-flight.

• Intelligence is "climbing the ladder efficiently".

• Wisdom is knowing when you're climbing the wrong ladder, & figuring-out how to discover which ladder you're supposed to be climbing.

Would you remove competence-at-soccer tests for pro sports-teams?

"Oh, James Windermere's an excellent athlete to add to our soccer-club! Look at his triathelon ratings!"..

.. "but he doesn't even understand soccer??"

.. "he doesn't need to: we got rid of that requirement, because AI got better than humans, so we don't need it anymore".

idiotic, right?

It doesn't matter if an AI is better than a human at a particular competency:

if a kind-of-work requires that competency, then test the human for it.

Tests built for humans are not tests built for machines.

No. It may be proof that standardized tests are not useful measures of LLM intelligence, but human brains operate differently from LLMs, so these tests may still be very useful measures of human intelligence.

I don't think any of those tests ever claimed to be a general intelligence test, they're specific knowledge tests. Books also contain a ton of specific knowledge but books are not intelligent.

No. It’s the opposite in fact. It shows that ChatGPT is not very intelligent. Just very well-read.

Intelligence is not the same as knowledge.

We already knew that intelligence is a complex and multifaceted property, and that being very intelligent and being very good at taking tests are two distinct (albeit loosely correlated) skills. It's just a too convenient measurement despite it's many flaws.

This is only news if you're an ignorant techbro who doesn't pay attention to any other field except computer programming.