Does or could ChatGPT understand text? [closed]

Question

The following argument concludes that the common understanding of ChatGPT (trained on text, receives online users' text questions, etc.) is not supported by the science. What criticisms are there of the argument which show it is unsound (one or more false premises) or invalid (breaks the rules of formal logical deduction)? The argument is:

In November 2022 AI research firm OpenAI released its online "text" chatbot ChatGPT. The following great interest was based on the popular and expert-led conviction that ChatGPT did, or with suitable (multi-billion-dollar) development might, understand the text questions online users type.

ChatGPT exists inside the electronics of computers. The research I conducted indicates that the science of the physics and chemistry of electronics doesn't support the idea that ChatGPT could understand online users' text questions. The reason is as follows. The only place text exists is on user keyboards and display screens. Everything else is electromagnetic radiation (WiFi, Bluetooth, fiber optics) and electrons (in wires and semiconductors).

ChatGPT never gets the text questions online users type. All it gets is electrons. The internet is made of electronic storage. All that's stored on the internet is electrons. There's no text on the internet. The ubiquitous claim that ChatGPT is trained on text stored on the internet is simply false.

Relevant quoted and referenced statements of OpenAI and respected media allegedly falsely describing ChatGPT can be found in Does ChatGPT understand text (PDF).

Answer 1

The argument is irrelevant. You can apply the same logic to say that humans don't understand text, since text doesn't exist in your brain. There are lots of software applications that read text- if you integrated one with ChatGPT, would that mean the integrated system suddenly 'understood' text in a way ChatGPT did not? You can tell Rodsmith from me that the argument is nonsense, and he should present himself at my study if he requires further clarification.

Answer 2

ChatGPT never gets the text questions online users type. All it gets is electrons. […] The ubiquitous claim that ChatGPT is trained on text stored on the internet is simply bonkers.

This argument is based entirely on a misinterpretation of what the word "text" means in this context.

In computing, "text" means a sequence of codes which are designed to represent written words, essentially by using one code for each character in the writing which is to be represented. It is obvious that ChatGPT is trained not on text in the sense of matter physically arranged into written words, but on text in the computing sense.

You're a software developer, so presumably you're familiar with the use of the word "text" to mean a sequence of character codes. It's not clear why you chose to interpret the word "text" in a way that produces a "bonkers" claim instead of a sensible one.

The argument says nothing about ChatGPT's ability or inability to understand these sequences of codes, so there is no need for me to say anything about that either.

---

By the way, I notice that the idea that symbols are shapes rather than other types of patterns seems to be very important to you. In this question from 2019, you asked why Alan Turing described computers as manipulating "symbols," given that the things that they manipulate are actually voltages and not shapes. In this question and this question, both from 2022, you point out that when people do what they call a Turing test in real life, the machine almost always receives electronic pulses instead of shapes, and you assert that "[w]ithout exposure to the shapes, the machine could not possibly understand the questions."

Why do you think that the distinction between shapes and other kinds of codes is important? When people listen to each other speaking, they don't hear any shapes, they only hear information about frequencies and amplitudes; does that mean that it is impossible for any human to ever understand speech? If a person has never seen any shapes representing words in their life, does that mean that that person is unable to understand language at all?

It seems like you keep repeating the argument "computers don't receive shapes as input, therefore computers can't understand anything" over and over, presumably hoping that someone will eventually agree with your argument (or at least understand it), but you've never given any reasoning for why you think the argument is valid. The argument seems like a non sequitur to me.

Answer 3

People are quick to point out that AI is merely electricity passing through some wires and transistors and such and to perform basic mathematical operations, and therefore AI is not intelligent like humans are.

But these arguments generally seem to take the form of "AI is like this, therefore it's not like humans in terms of understanding", without actually addressing what humans are like. That's comparing 2 things by only analysing 1 of those things. Considering the mechanics of human intelligence (as best we understand it) doesn't seem to end up in favour of those saying AI doesn't "understand" or aren't "conscious". Human brains are, after all, a bunch of connected neurons passing electrical signals between one another, in some chemical soup. Some might say there's more to human consciousness than our brains, but we have no evidence of this, and every indication that consciousness is strongly and directly tied to the physical parts of our brain.

It's probably fair to say the physical parts of the brain is more than just computation (which arguably set it apart from modern AI), although we're not currently able to say which mechanisms in the brain makes a human conscious, and how it does so. Even the people who says the mind exists beyond the brain still can't describe the mechanism for understanding or consciousness within such a mind (nor describe how such a mind would interact with the brain on a mechanical or logical level).

Knowing a thing or two about AI doesn't make someone an expert in neuroscience, yet some people act as if it does, as if they have a perfect understanding of the mechanisms of "consciousness" and "understanding" in humans, for them to be justified in saying AI isn't conscious.

---

This "it's all electrons" seems to be taking a few steps up on the scale of smart-sounding nonsense.

One could say "ChatGPT never gets the text questions, because it just gets electrons", but what does it even mean to "get" something? You could similarly say that your brain doesn't "get" the things you see or hear because all it receives is electric signals from your ears and eyes via your nerves, and the brain itself works largely by electric signals being passed between neurons.

* I briefly scanned through the paper. It does mention the need to "chose[sic] a place to start trying to understand the mind", but it's hard to make sense of the gibberish that follows as anything that could be said to differentiate human minds from AI.

Answer 4

First of all - LLM has no conscience so one can argue they do not understand anything at all. Does a flashlight understand that you switch it on when you push a button on it? No, it's just turned on cause there is electricity that starts flowing. If we continue this line of reasoning why does it matter at all in which form LLM process data? We shouldn't anthropomorphize LLMs as consciously understanding. They have no idea they even exist not to mention text understanding.

On a more technical note - the paper you provide leans quite heavily on Searle's CRA, but does not address counterarguments that have been made against it. The thing is reference, and semantics are not intrinsic qualities of arbitrary symbols. They emerge from the relationship between internal states of an agent and its external environment, through the intercession of the senses. Symbols must be "grounded" in sensorimotor interaction with the world to take on meaning.

The Chinese room, for example, cannot claim genuine semantics because it lacks grounding - its symbols derive from rules given by external designers, not internal perception and action. The manipulated symbols are not meaningful.

By relying on intrinsic semantics, the CRA ignores the actual relationship between representation and reference that gives rise to true semantics. That's a weakness.

Answer 5

First of all, I'm involved in NLP, so thanks for the paper. Some questions that come through here are malformed grammatically, so it's always a pleasure to find a semantically apropos, well cited question.

Obviously, it all hinges on what it means "to understand". The questions about the transformer model and their implications about semantics is a serious philosophical matter if you are willing to move past the claim computers only twiddle bits. The gulf between AGI and AI is best outlined in the recent book The Myth of Artificial Intelligence by Erik J. Larson (GB). Essentially, LLMs and other ML strategies have human-like abilities, but they don't rise to the level of general intelligence.

The point Larson makes is that big data methods like LLMs (which are statistical language models that rely on massive corpora) are inductive techniques and that whatever general intelligence is, it at a minimum is something that integrates deductive, inductive, and abductive methods. As he points out, there isn't even a theory to put to empirical test. In theory then, it's open philosophical as well as scientific question if general intelligence can be exhibited by non-biological systems of computation. (See the SEP's Physical Computation for a thorough treatment of what computation is best described as.)

Thus, it's strong to claim that computers do not understand with general intelligence. Note, there is no firm argument that computers cannot understand. John Searle's argument is considered by many thinkers to be severely deficient. For a good introduction to the criticisms thereof, see Preston & Bishop's Views into the Chinese Room. If you believe that the systems response carries weight for instance, then there is no serious argument that general intelligence cannot emerge in a significantly complex, non-biological system as far as I have found.

Does ChatGPT only get and process electrons? No, this claim is based on a category mistake. Electrons are ontological primitives in the discourse of electrical theory. ChatGPT is a statistical engine that processes vectorized, arbitrary token sequences based on frequency. Anyone conflating those two domains of discourse doesn't understand the basics of how reductionism in scientific theory (SEP) works. In fact, human understanding is built on processing morphemes (mostly) deterministically and contextually in the bigger picture of discourse (See dynamic semantics (SEP) for some insights there), and LLMs do not have access to that sort facility, even if you plunk an LLM into langchain and add some lightweight prompt engineering, use embedding strategies, etc.

Be cautious of anyone who philosophically abuses metaphysical reductionism and who couldn't explain in plain English what a transformer is. If you really are intrigued by the topic, I would encourage you to also read natural language ontology (SEP), categories (SEP), and concepts (SEP). At a minimum, the mimicry of language with formal semantics and human-level intelligence and natural language use are easy to confuse, but also easy with the right technical background to expose as distinct categories.

See also:

• Why is it impossible for a program or AI to have semantic understanding?
• Does a large language model show signs of an emergent awareness of semantics?
• AI-related PhilSE questions

Answer 6

I see from OP's profile that they wrote the paper!
It defines what they understand by "text".

4.1 What is text?

Text is written or printed language such as we find in textbooks. It comprises instances of shapes. But not every instance of a shape is text. A shape is text if it has been assigned a meaning, or interpretation. This is often done by a community.

Such shapes might be atomic such as A, B, C, D and 1, 2, 3, 4, that is, letters of an alphabet and digits. Or they might be such shapes sequenced into words, multi-digit numerals, and so on.

Types of text characters include the just-mentioned words and numerals, and also punctuation marks, special characters such as %, &, @, mathematical symbols, +, -, ^, = and others. In short, the shapes one finds imprinted on the top surfaces of keys of a computer or typewriter keyboard.

To understand text is to know the meanings of its shapes. Understanding text presupposes learning the meanings of the shapes. This is what many enthusiasts think ChatGPT does. They say it's rained on text scraped from the internet and that this training results in ChatGPT learning the meanings of the text shapes.

The last sentence undermines the whole premise of the paper:

The text shapes are not what is transmitted to ChatGPT. It is the electronic standard representations of the text, and the shapes of the letters are defined in the webpage style section (or overridden by the reader). ChatGPT won't care what font the webpage's author preferred the text to be read by humans. It is irrelevant. It is still text.

It is a symbolic representation of text, as defined more than 60 years ago with ASCII, and updated more recently with variants of Unicode. But it is still a symbology of the text, and not the actual shapes.

So ChatGPT is not sent shapes, but a representation of the text which allows it to process the semantics. Human readers are not sent shapes either, but instructions as to what shapes their browser should use, to make the text readable by humans. But ChatGPT does not convert the text representation to shapes, and then apply an OCR (optical character reader) to convert the shapes back to a simpler representation of text which it can process, which it was sent in the first place.

The author might just as well have written a paper explaining that Morse code wasn't about text, because it wasn't transmitting actual "letters" and "words" but pulses.

But they overlook the semantics of what the pulses, electrons, whatever carry, which is contextual (pun intended).

Answer 7

1. The question whether ChatGPT understands text has the pragmatic answer "Yes": Every user can confirm this by entering text into the text-interface of ChatGPT.

2. The deeper question is: Does ChatGPT understand the meaning of text?

   Here is the answer from ChatGPT itself:

   As an AI language model, I don't "understand" text in the same way humans do. Instead, I process and generate text based on patterns and associations learned from vast amounts of data during training. When you ask a question or input text, I analyze it, identify patterns, and generate responses based on those patterns. While I can provide coherent and contextually relevant responses, I don't have consciousness or subjective understanding like humans do. My responses are generated algorithmically based on statistical patterns in the data I've been trained on.

   The answer shows what ChatGPT is able to achieve and where it (i.e. it's designer team) locates its limit.

Answer 8

The old "It is nothing but a stochastic parrot" argument. Now it is true the main building blocks of ChatGPT and other LLMs do just that: User their neural net to predict on what word comes next. Not much intelligence there.

But then again: This is also a big building block on how our brain works: We lookup things in our own neural net: You are presented some new problem and immediately (without "thinking") some ideas come to your mind. People call it "intuition" or "experience". You hear a word and you find one that rhymes with it and you call it "creativity". Is there conscience? No. We do not know how we arrived at our "intuition" and neither does the LLM.

Now it is easy to see how these building blocks can be arranged to "mimic" conscience: Feed it some input and let it reflect on it. Let it checks the consistency of its results. Let it create new associations from that. Feed it back and let it reflect on it again, etc, etc..

Is this conscience? Well it would certainly look a bit like it. You could ask such a system what it was "thinking" and it could truthfully tell you that it was currently reflecting on some logical errors it found in the writings of Thomas Aquinas and then used its time to search the internet to find places where someone else spotted that but was not successful in finding much.

I think even a system that is based purely on text can get pretty far. Can you understand our 3D reality? At least such a system can reason about all its properties as it knows mathematics. Just as the human mathematician can reason about abstract 5D spaces. But can it understand what it means to enjoy a sunset on the beach when it has never "seen" a sunset? Leaving aside the fact that newer LLMs are "multi modal" and have images, video, sound as input and output, even the text can get you pretty far. All the works of literature that write about what humans feel. The understanding of biochemistry that governs the human body will also help. Sure this kind of intelligence is a true "alien" intelligence. It is the first alien intelligence mankind contacts.

And then the last part: The argument that these systems do not have a "world" they live in where they can collect their own experiences. Now this ignores that there is reinforcement learning.

A chess computer learns by playing endless games of chess. It has a 8 by 8 checkerboard world. Soon the coding assistance software will not only "autocomplete" your code but will write programs, then compile then, run and test them, profile their performance and learn from that "experience". It has an abstract world of "software and programming" where it navigates and learns.

The "universe" of these systems is quite different from ours. So there will always be a bridge where we do not understand them and they do not understand us. But I find it quite arrogant to assume that THEY are limited where actually WE are. THEY can create new universes where they "live" that we can't.

As Wittgenstein famously said:

if a lion could speak, we could not understand him’ because a lion’s form of life is so alien to ours that we cannot seriously claim to know what the lion means by what the lion says.

So the main fallacy of the critique on AI is that they confuse "intelligence" and "human intelligence".

And all of this is not science fiction: All the above either exists or is actively worked on and literally trillions of dollars of budget are directed that way.

So the answer to the question is:

Can it ChatGPT understand text:

• currently not but within the next few month most likely yes.
• yes. but it will never be able to understand it the same way as a human. as much as we can not understand the lion.

Answer 9

The answer is pretty simple and short:

The argument is false. Here's why:

The presence or absence of text in the guts of computing machines is completely irrelevant. What matters is information. Shannon information. That's what human language is about and what LLM models, and deep learning models in general, deal with.

You better coherently establish the meaning of understanding if you're gonna argue computers don't have it. Since I don't suppose the author successfully and satisfactorily defined it, his argument is doomed.