this post was submitted on 21 Jun 2023
6 points (100.0% liked)

AI

4151 readers
3 users here now

Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals, which involves consciousness and emotionality. The distinction between the former and the latter categories is often revealed by the acronym chosen.

founded 3 years ago
6
TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (arxiv.org)
submitted 1 year ago by sisyphean to c/[email protected]
0 comments fedilink hide all child comments
 

cross-posted from: https://programming.dev/post/133153

Quote:

In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.

Related:

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here