this post was submitted on 15 May 2024
656 points (99.8% liked)

TechTakes

1489 readers
56 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 125 points 7 months ago (64 children)

What I find delightful about this is that I already wasn't impressed! Because, as the paper goes on to say

Moreover, although the UBE is a closed-book exam for humans, GPT-4’s huge training corpus largely distilled in its parameters means that it can effectively take the UBE “open-book”

And here I was thinking it not getting a perfect score on multiple-choice questions was already damning. But apparently it doesn't even get a particularly good score!

[–] [email protected] 36 points 7 months ago (44 children)

[...W]hen examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to 48th percentile overall, and 15th percentile on essays.

officially Not The Worst™, so clearly AI is going to take over law and governments any day now

also. what the hell is going on in that other reply thread. just a parade of people incorrecting each other going "LLM's don't work like [bad analogy], they work like [even worse analogy]". did we hit too many buzzwords?

[–] [email protected] 18 points 7 months ago (1 children)

But LLM’s don’t work like Typewriters, they work like Microwaves!

load more comments (1 replies)
[–] [email protected] 18 points 7 months ago* (last edited 7 months ago)

"Nooo you don't get it, LLMs are supposed to be shit"

load more comments (42 replies)
[–] [email protected] 31 points 7 months ago (3 children)

That’s like saying a person reading a book before a quiz is doing it open book because they have the memory of reading that book.

[–] [email protected] 64 points 7 months ago (2 children)

It's more like taking a digital copy into the test room with you and Ctrl+F'ing every question/answer.

[–] [email protected] 25 points 7 months ago (7 children)

Except it’s not, because they can’t perfectly recall everything.

It’s more like reading every book in the world, and someone asking you what comes next after “And I…”.

[–] realbadat 37 points 7 months ago

"will alwaaays love you...."

Easy. No other answer.

[–] [email protected] 21 points 7 months ago* (last edited 7 months ago) (12 children)

But the AI isn't "recalling" in the same way you do, it doesn't "remember" what it "read", it "reads" on demand and has instant access to essentially all of the information ~~available online~~ it was trained on (E: though it's becoming more or less the same thing, and is definitely the same when it comes to law books for example), from which it collects the necessary details if and when it needs it.

So yes, it is literally "sat" there with all the books open in front of it, and the ability to pinpoint a bit of information in any one of all the books in milliseconds.

load more comments (12 replies)
[–] [email protected] 15 points 7 months ago

and in conclusion an AI is very like an elephant, particularly the back end

load more comments (4 replies)
[–] [email protected] 20 points 7 months ago* (last edited 7 months ago) (7 children)

I'm not a big AI guy but it's really not quite like that, models do NOT contain all the data they were trained on.

Edit: I have no idea what's going on down below this comment

[–] [email protected] 27 points 7 months ago (11 children)

I’m not a big AI guy

we can tell

load more comments (11 replies)
load more comments (6 replies)
[–] [email protected] 21 points 7 months ago* (last edited 7 months ago) (14 children)

I'm not even going to engage in this thread cause it's a tar pit, but I do think I have the appropriate analogy.

When taking certain exams in my CS programme you were allowed to have notes but with two restrictions:

  1. Have to be handwritten;
  2. have to fit on a single A4 page.

The idea was that you needed to actually put a lot of work into making it, since the entire material was obviously the size of a fucking book and not an A4 page, and you couldn't just print/copy it from somewhere. So you really needed to distill the information and make a thought map or an index for yourself.

Compare that to an ML model that is allowed to train on data however long it wants, as long as the result is a fixed-dimension matrix with parameters that helps it answer questions with high reliability.

It's not the same as an open book, but it's definitely not closed book either. And the LLMs have billions of parameters in the matrix, literal gigabytes of data on their notes. The entire text of War and Peace is ~3MB for comparison. An LLM is a library of trained notes.

load more comments (14 replies)
load more comments (1 replies)
load more comments (62 replies)