That’s what I thought but there’s slightly more than that.
The writer tried to trick ChatGPT 5, saying Vermont has no R in it. ChatGPT did say “wait, it does”. But then when pushed it said, “oh right there is no R in Vermont”.
I mean… the inability to know what it knows or not is a real problem for most use cases…
Yeah, the fact you can “gaslight” a chat is just as much of a symptom of a difficulty as the usual mistakes. It shows that it doesn’t deal with facts, but structurally sound content, which is correlated with facts, especially when the prompt has context/rag stuffing the prompt using more traditional approaches that actually will tend to get more factual stuff crammed in.
To all the people white knighting for the LLM, for the thousandth time, we know that it is useful, but it’s usefulness is only tenuously connected to the marketing reality. Making the mistake in counting letters is less important than the fact that it “acts” like it can when it can’t.
That’s what I thought but there’s slightly more than that.
The writer tried to trick ChatGPT 5, saying Vermont has no R in it. ChatGPT did say “wait, it does”. But then when pushed it said, “oh right there is no R in Vermont”.
I mean… the inability to know what it knows or not is a real problem for most use cases…
Yeah, the fact you can “gaslight” a chat is just as much of a symptom of a difficulty as the usual mistakes. It shows that it doesn’t deal with facts, but structurally sound content, which is correlated with facts, especially when the prompt has context/rag stuffing the prompt using more traditional approaches that actually will tend to get more factual stuff crammed in.
To all the people white knighting for the LLM, for the thousandth time, we know that it is useful, but it’s usefulness is only tenuously connected to the marketing reality. Making the mistake in counting letters is less important than the fact that it “acts” like it can when it can’t.