I need to disagree with you on AI. We did not fail at it. Not because LLMs are good. But because any program processing arbitrary data, even a stupid simple calculator is AI – a machine performing work that human brain can do, ideally with the added benefit of maximized determinism and greater speed. If you reduce this generalistic term I believe is so overly broad we should cease to use it to LLMs, then these criteria seem to have been thrown out of the window since they are usually heuristic balls of python mud.
So having established that it is all just software that processes arbitrary data, let's go back to the basics of software design. Huge amounts of money and working hours have been thrown into the erratic attempts to create a software that can do everything at once. GPT extensions are fucking dystopian and here is why – we had a tool for that for decades that does it much more better, without imposing digital handcuffs on the user and burning the planet – IT'S CALLED AN OPERATING SYSTEM AND PROGRAMS.

General-purpose AI is a lie sold to you by monopolistic surveillance capitalists for whom it is a dream come true since making a decently reliable LLM requires prohibitively large resources but the endless stream of data much larger and contextualized than was the case for search engines thrown at it compensates that quite well, a pipe dream in terms of achieving what it is aimed to achieve with it's current design and a nightmare to build and test.

So if we discard this term as a meaningless overly broad buzzword it is since computation on non hardcoded data is what we've designed computers that are not just state machines for, let's talk about what makes Lisp is so good at data-driven programming:

Functional programming is generally more deterministic since you have immutable persistent data structures everywhere. This also makes it quite good at implementing safe, reliable concurrency.
This determinism is furthered by the homoiconicity – the fact that the boundary between code and data is the outcome of using S-expressions and has powerful implications for eliminating so many data conversion bugs and complexities, all while usually not using static typing (!) and also for the language's extensivity and building DSLs
Very simple syntax, again thanks to S-expressions - just (function arguments...) basically.

I think Eich understood that when he initially wanted to port Scheme to the web browser, after all html does have lispy semantics, but office politics in the heyday of Java forced him to give up on this idea and we've ended up with this goofy counterintuitive mess that bred hacky workarounds instead of the extensivity we could've had if he did so - take a look at Hiccup templating DSL and decide for yourself if this or jsx are simpler ways of writing out stuff to the DOM.