Technology

I think there's a lot of Dunning–Kruger here.

The simple fact is that they aren’t able to reproduce their exact training data so no, they aren’t storing it in a highly compressed form.

See: jpeg analogy. You've described here lossy compression not something that is categorically different than compression. Perhaps the AI models are VERY lossy. But that doesn't mean it is original or creative.

But the reality is, we largely do not know how these chatbots work. They are black boxes even to the researchers themselves. That's just how neural networks are. But the thing I know is they are not themselves creative. All they can do is follow weights to reproduce the things human classifiers evaluated as subjectively "good" over the things they subjectively evaluated as "bad". All the creativity happened in the training process -- the inputs and the testing. All of the apparent creativity outputted is a product of the humans involved in training and testing the model, not the model itself. The actual creative force is somewhere far away.