487
this post was submitted on 17 Aug 2023
487 points (96.0% liked)
Technology
58303 readers
11 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Just now, I tried to get Llama-2 (I'm not using OpenAI's stuff cause they're not open) to reproduce the first few paragraphs of Harry Potter and the philosophers' stone, and it didn't work at all. It created something vaguely resembling it, but with lots of made-up stuff that doesn't make much sense. I certainly can't use it to read the book or pirate it.
Openai:
That doesn't mean the copyrighted material isn't in there. It also doesn't mean that the unrestricted model can't.
Edit: I didn't get it to tell me that it does have the verbatim text in its data.
Here we go, I can get chat gpt to give me sentence by sentence:
Most publically available/hosted (self hosted models are an exception to this) have an absolute laundry list of extra parameters and checks that are done on every query to limit the model as much as possible to tailor the outputs.
This wasn't even hard... I got it spitting out random verbatim bits of Harry Potter. It won't do the whole thing, and some of it is garbage, but this is pretty clear copyright violations.
Maybe it’s trained not to repeat JK Rowling’s horseshit verbatim. I’d probably put that in my algorithm. “No matter how many times a celebrity is quoted in these articles, do not take them seriously. Especially JK Rowling. But especially especially Kanye West.”
It's not repeating its training data verbatim because it can't do that. It doesn't have the training data stored away inside itself. If it did the big news wouldn't be AI, it would be the insanely magical compression algorithm that's been discovered that allows many terabytes of data to be compressed down into just a few gigabytes.
Do you remember quotes in english ascii /s
Tokens are even denser than ascii. simmlar to word "chunking" My guess is it's like lossy video compression but for text, [Attacked] with [lazers] by [deatheaters] apon [margret];[has flowery language]; word [margret] [comes first] (Theoretical example has 7 "tokens")
It may have actually impressioned a really good copy of that book as it's lilely read it lots of times.
If it's lossy enough then it's just a high-level conceptual memory, and that's not copyrightable.
It varries based on how much time its been given with the media.