This gets the question…how do we think? Are we not just language (and other inputs as well) processors? I’m not sure the answer is “no.”

I also listened to an interesting podcast, I believe it was this American life or some other npr one, about whether ai has intelligence. To avoid the just “compressed knowledge” they came up with questions that the ai almost certainly would not have found in the web. Early ai models were clearly just predicting the next word, and the example was asking it to stack a list of objects. And it just said to stack them one on top of another, in a way that would no way be stable.

However when they asked a new model to do the same, with the stipulation that it explain it’s reasoning, it stacked the objects in a way that would likely be stable. Even noting that the nail on top should be placed on the head so it doesn’t roll around, and laying eggs down in a grid between a book and a plank of wood so they wouldn’t roll out.

Another experiment they did was take a language model and asked it to use some obscure programming language to draw a picture of a unicorn. Now this is a language model, not trained on any images.

And you know what it did? It produced a picture of a unicorn. Just in rough shapes, but even when they moved the horn and flipped it around, it was able to put it back. Without even ever seeing a unicorn, or anything even, it was able to draw a picture of one.

I don’t think the answer is as simple and clear as you want it to be. And the fact that it “fucked up” on a vague prompt doesn’t really prove anything. Even humans do stupid shit like this if they learn something incorrectly.