this post was submitted on 16 Aug 2024
8 points (61.1% liked)

Programming

17325 readers
75 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 1 year ago
MODERATORS
 

Today I had a little aha moment. If anyone asked me yesterday about AI tools integrated into their editor, I would say its a bad idea. Ask me today, I would still say its bad idea. :D Because I don't want to rely on AI tools and get too comfortable with it. Especially if they are from big companies and communicate through internet. This is a nogo to me.

But since weeks I am playing around with offline AI tools and models I can download and execute locally on my low end gaming PC. Mostly for playing with silly questions and such. It's not integrated in any other software, other than the dedicated application: GPT4All (no it has nothing to do with ChatGPT)

I'm working on a small GUI application in Rust and still figure out stuff. I'm not good at it and there was a point where I had to convert a function into an async variant. After researching and trying stuff, reading documentation I could not solve it. Then I asked the AI. While the output was not functioning out of the box, it helped me finding the right puzzle peaces. To be honest I don't understand everything yet and I know this is bad. It would be really bad if this was a work for a company, but its a learning project.

Anyone else not liking AI, but taking help from it? I am still absolutely against integrated AI tools that also require an online connection to the servers of companies. Edit: Here the before and after (BTW the code block in beehaw is broken, as certain characters are automatically translated into < and & for lower than and ampersand characters respectively.)

From:

    pub fn collect(&self, max_depth: u8, ext: Option<&str>) -> Files {
        let mut files = Files::new(&self.dir);

        for entry in WalkDir::new(&self.dir).max_depth(max_depth.into()) {
            let Ok(entry) = entry else { continue };
            let path = PathBuf::from(entry.path().display().to_string());
            if ext.is_none() || path.extension().unwrap_or_default() == ext.unwrap() {
                files.paths.push(path);
            }
        }
        files.paths.sort_by_key(|a| a.name_as_string());

        files
    }

To:

    pub async fn collect(&self, max_depth: u8, ext: Option<&str>) -> Result {
        let mut files = Files::new(&self.dir);

        let walkdir = WalkDir::new(&self.dir);
        let mut walker =
            match tokio::task::spawn_blocking(move || -> Result {
                Ok(walkdir)
            })
            .await
            {
                Ok(walker) => walker?,
                Err(_) => return Err(anyhow::anyhow!("Failed to spawn blocking task")),
            };

        while let Some(entry) = walker.next().await {
            match entry {
                Ok(entry) if entry.path().is_file() => {
                    let path = PathBuf::from(entry.path().display().to_string());
                    if ext.is_none() || path.extension().unwrap_or_default() == ext.unwrap() {
                        files.paths.push(path);
                    }
                }
                _ => continue,
            }
        }

        files.paths.sort_by_key(|a| a.name_as_string());

        Ok(files)
    }
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 2 points 2 months ago (1 children)
[–] [email protected] 2 points 2 months ago (1 children)

It runs smoother and with no memory bottlenecks. Besides, you can load any gguf you want. You are not limited by the LLMs offered by GPT4ALL

[–] [email protected] 1 points 2 months ago (1 children)

I have no comparison, so cannot say much. But what do you mean by memory bottlenecks? Isn't this a limitation of the hardware itself? I mean if I have a memory bottleneck on GPT4All, then I would have it using with other software as well.

[–] [email protected] 2 points 2 months ago (1 children)

The app will freeze the computer if you use models that are too big. It also produces stuttering in the smaller models.

[–] [email protected] 2 points 2 months ago (1 children)

BTW both applications download from HuggingFace and I could just place any .gguf file downloaded manually too. And it runs through lamacpp too, so it shouldn't be too different. GPT4All got a few big updates recently.

But I lack experience and am open for any alternative. Unfortunately oobabooga is not available through Flatpak (I'm on Linux). If it becomes available as Flatpak, I'll give it a try. At the moment there is no hurry for me, as it works fine now. But the new Llama 3.1 128k is slow and I wonder if its a problem with AMD cards, with GPT4All or if this is a problem with this model in general. So that's why I'm open to try other software.

[–] [email protected] 2 points 2 months ago

ok. you run the start_linux.sh on oobabooga to run it on Linux. I've never run it on Linux, though.