Seems no, to me: a human lawyer wouldn't, for instance, make up case law that doesn't exist. AI has done that one already. If it had even the most basic understanding of what the law is and does, it would have known not to do that.

LLMs don't have an understanding of anything, but that doesn't mean all AI in perpetuity is incapable of having an understanding of eg. what the law is. Edit: oh and also, it's not like human lawyers are incapable of mistakenly "inventing" case law just by remembering something wrong.

As to whether human intelligence is statistics, well… our brains are neural networks, and ultimately neural networks – whether made from meat or otherwise – are "just statistics." So in a way I guess our intelligence is "just statistics", but honestly the whole question is sort of missing the point; the problem with AI (which right now really means LLMs) isn't the fact that they're "just statistics", and whether you think human intelligence is or isn't "just statistics" doesn't really tell you anything about why our brains perform better than LLMs

People confabulate all the time.

False memories can also be deliberately created. Here's a classic: https://www.washington.edu/news/2001/06/11/i-tawt-i-taw-a-bunny-wabbit-at-disneyland-new-evidence-shows-false-memories-can-be-created/

Seems no, to me: a human lawyer wouldn't, for instance, make up case law that doesn't exist

You've never seen someone misremember something? The reason human lawyers don't usually get too far with made-up case law is because they have reference material and they know to go back to it. Not because their brains don't make stuff up.