Actually Useful AI

cross-posted from: https://lemmy.intai.tech/post/133548

https://arxiv.org/pdf/1706.03762.pdf

Attention Is All You Need

By Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin

Word count: 4221

Estimated read time: 17 minutes

Links:

• Paper: https://arxiv.org/abs/1706.03762
• Code: https://github.com/tensorflow/tensor2tensor

Summary: This paper proposes a new neural network architecture called the Transformer that is based solely on attention mechanisms, without using sequence aligned RNNs or convolutions. The Transformer achieves state-of-the-art results in machine translation while being more parallelizable and requiring significantly less time to train. Key contributions:

Proposes multi-head self-attention as a replacement for recurrence and convolutions in encoder-decoder architectures. Self-attention connects all positions with a constant number of sequentially executed operations, whereas recurrent layers require O(n) sequential operations.

Introduces scaled dot-product attention, which performs better than additive attention for large values of attention dimension. Applies attention scaling to improve training.

Employs positional encodings instead of recurrence to enable the model to make use of sequence order. Shows that learned positional embeddings can replace sinusoids with negligible loss in quality.

Achieves state-of-the-art BLEU scores on WMT 2014 English-to-German and English-to-French translation at a fraction of the training cost of previous models. Outperforms all previously published models on English constituency parsing with limited training data.

The Transformer's reliance on attention and positional encodings rather than recurrence make it very promising for parallelization and scaling to longer sequences. The results demonstrate the potential of attention-based models to supplant RNNs and CNNs in sequence transduction tasks.

Evaluation: The Transformer architecture presents several advantages for using large language models and generative adversarial networks:

The Transformer is highly parallelizable since it does away with sequence-aligned RNNs. This makes it very suitable for scaling up with more parameters and data.

The multi-head self-attention provides a way to jointly attend to information from different representation subspaces at different positions, allowing modeling of dependencies regardless of distance. This is useful for long-range dependencies in large contexts.

Positional encodings allow the model to make use of sequence order without recurrence. This can enable generating coherent, ordered outputs in GANs and large LMs.

The Transformer achieves excellent results with limited training data, suggesting its representations transfer well. This is promising for few-shot learning and fine-tuning large LMs.

The paper provides useful analysis into the roles different attention heads learn, which can inform work on interpretable attention-based representations.

Overall, the Transformer architecture seems very promising as a foundation for large scale language modeling and GAN training. The representations it learns appear powerful yet transparent. The results on parsing suggest it can capture linguistic phenomena well. The parallelizability enables scaling. Much follow-on work has already adapted and refined the Transformer, making it very relevant today.

Intelligence explosion arguments don’t require Platonism. They just require intelligence to exist in the normal fuzzy way that all concepts exist.

cross-posted from: https://lemmy.intai.tech/post/124795

• Github
• Paper

Large Language Models as Tool Makers Authors: Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, Denny Zhou

Word count: 4579 words

Estimated read time: 12 minutes

Source code: https://github.com/ctlllll/LLM-ToolMaker ↗

Summary:

This paper proposes a framework called LLMs As Tool Makers (LATM) that enables large language models (LLMs) to create and utilize their own tools for solving complex reasoning tasks. The key idea is to separate the process into two stages - tool making and tool using. In the tool making stage, a powerful yet expensive LLM acts as the "tool maker" to generate reusable Python functions for solving demonstrations of a task. In the tool using stage, a lightweight and cost-effective LLM acts as the "tool user" to call these tools to solve new instances of the task.

Experiments on tasks like logical deduction, tracking shuffled objects, Dyck language parsing, etc show that with tools made by GPT-4, GPT-3.5 Turbo as the tool user can match or exceed the performance of GPT-4 at lower cost. The authors also introduce a "dispatcher" LLM to handle streaming tasks by identifying when to reuse existing tools or request new ones.

Overall, this work demonstrates a promising approach to enabling LLMs to create their own tools, reducing reliance on human-crafted tools. The division of labor also allows using smaller models for most of the inferences, improving cost-efficiency. This technique could significantly expand the capabilities of LLMs in a scalable manner.

The proposed LATM framework demonstrates an interesting and promising approach to improving the reasoning and problem-solving capabilities of large language models in a cost-effective manner. Here are some thoughts on its applicability:

The ability for LLMs to create their own tools could be very useful for building practical applications. For any recurring task, the model could generate a reusable tool instead of solving from scratch each time. This could make applications more efficient and scalable.

The staged approach allows combining different sized models optimally - a powerful model makes tools, while lightweight models use the tools. This cost-effectiveness is attractive for real-world applications with budget constraints.

The tools being in Python allows them to integrate into application codebases easily. The dispatcher model also provides flexibility to handle new tasks.

The method's applicability does seem more geared towards logical reasoning, procedural and algorithmic tasks right now. Further research may be needed to extend it to other domains.

There are still open challenges around rigorously testing and validating the quality and safety of automatically generated tools. Methods to provide human oversight would be important.

Overall, the LATM paradigm does appear promising for augmenting LLMs and enabling them to participate more actively in their own learning and tooling. With further research to broaden its scope, it could become a general framework for efficiently enhancing LLM capabilities.

So in summary, LATM seems quite promising as a technique for unlocking more of the potential of LLMs for practical applications requiring complex reasoning in a scalable and cost-efficient manner. More research is still needed, but the principles demonstrated align well with enabling wider usage of LLMs and GANs in applications.

At OpenAI, protecting user data is fundamental to our mission. We do not train our models on inputs and outputs through our API.

We’re rolling out custom instructions to give you more control over how ChatGPT responds. Set your preferences, and ChatGPT will keep them in mind for all future conversations.

@AutoTLDR

GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on four diverse tasks: 1) solving math problems, 2) answering sensitive/dangerous questions, 3) generating code and 4) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was very good at identifying prime numbers (accuracy 97.6%) but GPT-4 (June 2023) was very poor on these same questions (accuracy 2.4%). Interestingly GPT-3.5 (June 2023) was much better than GPT-3.5 (March 2023) in this task. GPT-4 was less willing to answer sensitive questions in June than in March, and both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. Overall, our findings shows that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLM quality.

Using generative AI (like ChatGPT) in business improves users’ performance by 66%, averaged across 3 case studies. More complex tasks have bigger gains, and less-skilled workers benefit the most from AI use.

Introducing Llama 2 - The next generation of our open source large language model. Llama 2 is available for free for research and commercial use.

This release includes model weights and starting code for pretrained and fine-tuned Llama language models — ranging from 7B to 70B parameters.

@AutoTLDR

16 Mar, 2023

Kagi Search is pleased to announce the introduction of three AI features into our product offering.

We’d like to discuss how we see AI’s role in search, what are the challenges and our AI integration philosophy. Finally, we will be going over the features we are launching today.

@AutoTLDR

Wanted to share a resource I stumbled on that I can't wait to try and integrate into my projects.

A GPT4All model is a 3GB - 8GB file that you can download and plug into the GPT4All open-source ecosystem software. Nomic AI supports and maintains this software ecosystem to enforce quality and security alongside spearheading the effort to allow any person or enterprise to easily train and deploy their own on-edge large language models.

This is a game that tests your ability to predict ("forecast") how well GPT-4 will perform at various types of questions. (In caase you've been living under a rock these last few months, GPT-4 is a state-of-the-art "AI" language model that can solve all kinds of tasks.)

Many people speak very confidently about what capabilities large language models do and do not have (and sometimes even could or could never have). I get the impression that most people who make such claims don't even know what current models can do. So: put yourself to the test.

I have an upcoming interview and I'm curious if anyone has experimented with utilizing Nvidia's eye contact feature during interviews. What was your experience like?

Increasingly powerful AI systems are being released at an increasingly rapid pace. This week saw the debut of Claude 2, likely the second most capable AI system available to the public. The week before, Open AI released Code Interpreter, the most sophisticated mode of AI yet available. The week before that, some AIs got the ability to see images.

And yet not a single AI lab seems to have provided any user documentation. Instead, the only user guides out there appear to be Twitter influencer threads. Documentation-by-rumor is a weird choice for organizations claiming to be concerned about proper use of their technologies, but here we are.

@AutoTLDR

TL;DR: (by GPT-4 🤖)

The article by Chandler Kilpatrick on Medium discusses the new Code Interpreter feature of ChatGPT, which has been released to Beta from its previous Alpha testing phase. The Code Interpreter enhances ChatGPT's ability to process, generate, manipulate, and run code, currently supporting only Python. Users can upload files (with a limit of 100 MB per file) for the AI to interact with, although it cannot edit files directly. The Code Interpreter can be used in various fields such as software development, data analytics, documentation, and education, helping with tasks like code generation, error detection, code refactoring, creating data visualizations, and providing real-time programming tutoring. The article also highlights some impressive feats accomplished by users, including recreating the game Flappy Bird in less than 10 minutes.

LLM is my command-line utility and Python library for working with large language models such as GPT-4. I just released version 0.5 with a huge new feature: you can now install plugins that add support for additional models to the tool, including models that can run on your own hardware.

@AutoTLDR

An AI-first notebook, grounded in your own documents, designed to help you gain insights faster.

@AutoTLDR

Greetings Citizens of Hopefully Useful AI.

It has come to my attention that there are plenty of videos, as well as workflows that would get so much better if there was the possibility of textifying their audio content.

That being said, I hear Whisper, at least in the past 9 months or so was the cream of the crop when it came to audio recognition. And was also open source to boot (shocker).

Therefore, I'd be quite pleased to know if anyone created a method to more easily make use of the model. Because dedicating mental space to remembering specific adhoc commands does not make for a good long term tool.

For reference, I can throw a 24GB of VRAM at the problem if need be, and am running a Windows machine. Anything like Oobabooga or A1111? (Or a standard program would work just as nicely.) That would be very much appreciated.

Type in your answer, and ENRICH the future of Lemmy with your knowledge. (As well as answer one's question, pretty please.)

Thank you very much for reading and have a most fine of days!

We are pleased to announce Claude 2, our new model. Claude 2 has improved performance, longer responses, and can be accessed via API as well as a new public-facing beta website, claude.ai. We have heard from our users that Claude is easy to converse with, clearly explains its thinking, is less likely to produce harmful outputs, and has a longer memory. We have made improvements from our previous models on coding, math, and reasoning. For example, our latest model scored 76.5% on the multiple choice section of the Bar exam, up from 73.0% with Claude 1.3. When compared to college students applying to graduate school, Claude 2 scores above the 90th percentile on the GRE reading and writing exams, and similarly to the median applicant on quantitative reasoning.

@AutoTLDR

TL;DR: (by GPT-4 🤖)

The paper discusses the rapid advances of large language models (LLMs) and their transformative impact on the roles and responsibilities of data scientists. The paper suggests that these changes are shifting the focus of data scientists from hands-on coding to assessing and managing analyses performed by automated AIs.

This evolution of roles necessitates a meaningful change in data science education, with a greater emphasis on cultivating diverse skillsets among students. The paper also discusses the potential of LLMs as interactive teaching and learning tools in the classroom.

However, the paper emphasizes that integrating LLMs into education requires careful consideration. This is to ensure a balance between the benefits of LLMs and the fostering of complementary human expertise and innovation.

I'm looking for more cost-effective alternatives to Perplexity.ai that offer GPT-4 integration along with search capabilities for factual assistance, ideally around $5/month instead of the $20/month subscription fee for Perplexity.ai. I've come across Nuggt (https://github.com/Nuggt-dev/Nuggt), but it seems to rely solely on a local model without search functionality. I've also found Phind.com, a developer-focused search engine that uses GPT-4 to answer technical questions with code examples and detailed explanations. While it may not be as good as Perplexity.ai, it offers more free uses. Are there any other options that combine GPT-4 and search features at a lower price point?