Technology
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
view the rest of the comments
Something I found is that LLM struggle with weirder cases, when it comes to code.
I once tried getting ChatGPT (though admittedly only 3.5) to ~~generate code in~~ understand SaHuTOrEPoL, which is one of the more esoteric languages I created, and it really struggled with it.
Why would you expect ChatGPT to know how to write code in a language that you created yourself? According to that repository you created it last year, and ChatGPT was only trained on data up to 2021 so there's no way it could have been in its training set. Even though AIs have surprised us with their insights in a lot of cases they aren't magical.
Admittedly, I worded my comment poorly. What I meant is that ChatGPT struggled with understanding the semantics and structure of the language.
As an example, try from this this code block
You can, hopefully guess that
S__is a variable which has a methodm__w, accessed by using a hyphen, rather than a dot and statements end using adokeyword. ChatGPT missed on all marks.Have you tried including the docs in the prompt as well? I wouldn't have guessed that
S__is a variable in your example, but I would've known that after reading the docs. I'd expect ChatGPT to at least do OK if it had access to the docs.Huh, fair enough, its hard to know whats obvious and whats not.
As for the second part, to my slight surprise, it did manage to figure it out. I guess I just suck at using LLMs lol
edit: To clarify, previously I tried only to correct its mistakes over and over again untill I got frustated
edit2: I still feel like chatGPT is struggling with basics of the language. maybe its just me being shit at using LLMs but smh
I don't think the LLM is gonna do that great of a job with it for this reason, but still worth giving a shot. ChatGPT is a well trained coding chimp. You realistically could get a well trained chimp to start off a lot of projects and have people finish it. The fact that it can correct itself after you explain how it's wrong is very powerful as well.
LLM isn't gonna be useful for converting a single program from COBOL to Java, it is gonna be useful for converting many programs from COBOL to Java. I bet IBM is trying this on their own shit first before they try to sell it to customers, because language conversion software would be a HUGE but very boom based money.
As I stated on a different comment in this thread, I worded my comment poorly. Why I think this is relevant however that, at least in this case, if an LLM get code which is significantly different from what its trained with, it can make wildly incorrect guesses. While here its because of a language with a... unique syntax, I think this could also be the case for code with a lot of technical debt or weird design decisions.
Yeah, I don't think this will be as good as we want. The amount of technical debt I'd expect in these big programs would be off the charts. It would take a full team of people just to feed in corrections, and they would need to both know COBOL and be willing to kill all the COBOL jobs to do it. It's a tough ask
I'm still optimistic though because it's IBM and converting between languages is a billion dollar question. I don't think IBM would do a truly irresponsible deployment of such big corporate changes. I'd expect IBM to do a LOT of testing before sending anything out that would actually change the world, such as banking software. But in could be wrong.
IBM hawks new conversion tools all the time. None of them are amazing sliver bullets, all of them require humans to comb over the resulting output. And every single one Iβve ever used chokes on any weird case.
From the RPG fixed form to free form, DDS to DDL conversion, and so on all of them are usually more trouble to use than to not use.
IBM does this kind of stuff all the time. And for some folks itβll work some of the times. But at this point, I just skip any WS tool they put out and have a snippet on RDi and RDz that does all the required plugging away to call web services from the COBOL module.
Probably because it wasn't trained on it. That's not any different from asking the same of a human.
Sorry, I worded my comment poorly, see my reply to FaceDeer in this thread