• NutWrench@lemmy.world
    link
    fedilink
    English
    arrow-up
    37
    arrow-down
    2
    ·
    9 months ago

    Each conversation lasted a total of five minutes. According to the paper, which was published in May, the participants judged GPT-4 to be human a shocking 54 percent of the time. Because of this, the researchers claim that the large language model has indeed passed the Turing test.

    That’s no better than flipping a coin and we have no idea what the questions were. This is clickbait.

    • Hackworth@lemmy.world
      link
      fedilink
      English
      arrow-up
      14
      ·
      9 months ago

      On the other hand, the human participant scored 67 percent, while GPT-3.5 scored 50 percent, and ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time.

      54% - 67% is the current gap, not 54 to 100.

    • NutWrench@lemmy.world
      link
      fedilink
      English
      arrow-up
      14
      arrow-down
      3
      ·
      9 months ago

      The whole point of the Turing test, is that you should be unable to tell if you’re interacting with a human or a machine. Not 54% of the time. Not 60% of the time. 100% of the time. Consistently.

      They’re changing the conditions of the Turing test to promote an AI model that would get an “F” on any school test.

    • BrianTheeBiscuiteer@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      arrow-down
      1
      ·
      9 months ago

      It was either questioned by morons or they used a modified version of the tool. Ask it how it feels today and it will tell you it’s just a program!

  • dustyData@lemmy.world
    link
    fedilink
    English
    arrow-up
    26
    arrow-down
    1
    ·
    edit-2
    9 months ago

    Turing test isn’t actually meant to be a scientific or accurate test. It was proposed as a mental exercise to demonstrate a philosophical argument. Mainly the support for machine input-output paradigm and the blackbox construct. It wasn’t meant to say anything about humans either. To make this kind of experiments without any sort of self-awareness is just proof that epistemology is a weak topic in computer science academy.

    Specially when, from psychology, we know that there’s so much more complexity riding on such tests. Just to name one example, we know expectations alter perception. A Turing test suffers from a loaded question problem. If you prompt a person telling them they’ll talk with a human, with a computer program or announce before hand they’ll have to decide whether they’re talking with a human or not, and all possible combinations, you’ll get different results each time.

    Also, this is not the first chatbot to pass the Turing test. Technically speaking, if only one human is fooled by a chatbot to think they’re talking with a person, then they passed the Turing test. That is the extend to which the argument was originally elaborated. Anything beyond is alterations added to the central argument by the author’s self interests. But this is OpenAI, they’re all about marketing aeh fuck all about the science.

    EDIT: Just finished reading the paper, Holy shit! They wrote this “Turing originally envisioned the imitation game as a measure of intelligence” (p. 6, Jones & Bergen), and that is factually wrong. That is a lie. “A variety of objections have been raised to this idea”, yeah no shit Sherlock, maybe because he never said such a thing and there’s absolutely no one and nothing you can quote to support such outrageous affirmation. This shit shouldn’t ever see publication, it should not pass peer review. Turing never, said such a thing.

    • Kogasa@programming.dev
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      1
      ·
      9 months ago

      Your first two paragraphs seem to rail against a philosophical conclusion made by the authors by virtue of carrying out the Turing test. Something like “this is evidence of machine consciousness” for example. I don’t really get the impression that any such claim was made, or that more education in epistemology would have changed anything.

      In a world where GPT4 exists, the question of whether one person can be fooled by one chatbot in one conversation is long since uninteresting. The question of whether specific models can achieve statistically significant success is maybe a bit more compelling, not because it’s some kind of breakthrough but because it makes a generalized claim.

      Re: your edit, Turing explicitly puts forth the imitation game scenario as a practicable proxy for the question of machine intelligence, “can machines think?”. He directly argues that this scenario is indeed a reasonable proxy for that question. His argument, as he admits, is not a strongly held conviction or rigorous argument, but “recitations tending to produce belief,” insofar as they are hard to rebut, or their rebuttals tend to be flawed. The whole paper was to poke at the apparent differences between (a futuristic) machine intelligence and human intelligence. In this way, the Turing test is indeed a measure of intelligence. It’s not to say that a machine passing the test is somehow in possession of a human-like mind or has reached a significant milestone of intelligence.

      https://academic.oup.com/mind/article/LIX/236/433/986238

      • dustyData@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        9 months ago

        Turing never said anything of the sort, “this is a test for intelligence”. Intelligence and thinking are not the same. Humans have plenty of unintelligent behaviors, that has no bearing on their ability to think. And plenty of animals display intelligent behavior but that is not evidence of their ability to think. Really, if you know nothing about epistemology, just shut up, nobody likes your stupid LLMs and the marketing is tiring already, and the copyright infringement and rampant privacy violations and property theft and insatiable power hunger are not worthy.

  • phoneymouse@lemmy.world
    link
    fedilink
    English
    arrow-up
    21
    arrow-down
    1
    ·
    9 months ago

    Easy, just ask it something a human wouldn’t be able to do, like “Write an essay on The Cultural Significance of Ogham Stones in Early Medieval Ireland“ and watch it spit out an essay faster than any human reasonably could.

  • tourist@lemmy.world
    link
    fedilink
    English
    arrow-up
    17
    ·
    9 months ago

    The participants judged GPT-4 to be human a shocking 54 percent of the time.

    ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time

    Okay, 22% is ridiculously high for ELIZA. I feel like any half sober adult could clock it as a bot by the third response, if not immediately.

    Try talking to the thing: https://web.njit.edu/~ronkowit/eliza.html

    I refuse to believe that 22% didn’t misunderstand the task or something.

  • NeoNachtwaechter@lemmy.world
    link
    fedilink
    English
    arrow-up
    16
    arrow-down
    5
    ·
    9 months ago

    Turing test? LMAO.

    I asked it simply to recommend me a supermarket in our next bigger city here.

    It came up with a name and it told a few of it’s qualities. Easy, I thought. Then I found out that the name does not exist. It was all made up.

    You could argue that humans lie, too. But only when they have a reason to lie.

    • Lmaydev@programming.dev
      link
      fedilink
      English
      arrow-up
      5
      arrow-down
      2
      ·
      9 months ago

      That’s not what LLMs are for. That’s like hammering a screw and being irritated it didn’t twist in nicely.

      The turing test is designed to see if an AI can pass for human in a conversation.

      • NeoNachtwaechter@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        ·
        edit-2
        9 months ago

        turing test is designed to see if an AI can pass for human in a conversation.

        I’m pretty sure that I could ask a human that question in a normal conversation.

        The idea of the Turing test was to have a way of telling humans and computers apart. It is NOT meant for putting some kind of ‘certified’ badge on that computer, and …

        That’s not what LLMs are for.

        …and you can’t cry ‘foul’ if I decide to use a question for which your computer was not programmed :-)

  • bandwidthcrisis@lemmy.world
    link
    fedilink
    English
    arrow-up
    7
    ·
    edit-2
    9 months ago

    Did they try asking how to stop cheese falling off pizza?

    Edit: Although since that idea came from a human, maybe I’ve failed.

  • foggy@lemmy.world
    link
    fedilink
    English
    arrow-up
    6
    ·
    9 months ago

    Meanwhile, me:

    (Begin)

    [Prints error statement showing how I navigated to a dir, checked to see a files permissions, ran whoami, triggered the error]

    Chatgpt4: First, make sure you’ve navigated to the correct directory.

    cd /path/to/file

    Next, check the permissions of the file

    ls -la

    Finally, run the command

    [exact command I ran to trigger the error]>

    Me: stop telling me to do stuff that I have evidently done. My prompt included evidence of me having do e all of that already. How do I handle this error?

    (return (begin))

  • dhork@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    9 months ago

    In order for an AI to pass the Turing test, it must be able to talk to someone and fool them into thinking that they are talking to a human.

    So, passing the Turing Test either means the AI are getting smarter, or that humans are getting dumber.

    • pewter@lemmy.world
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      9 months ago

      Humans are as smart as they ever were. Tech is getting better. I know someone who was tricked by those deepfake Kelly Clarkson weight loss gummy ads. It looks super fake to me, but it’s good enough to trick some people.

  • werefreeatlast@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    arrow-down
    1
    ·
    9 months ago

    It does great at Python programming… everything it tries is wrong until I try and I tell tell it to do it again.

    • A_A@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      9 months ago

      Edit :
      oops : you were saying it is like a human since it does errors ? maybe i “wooshed”.


      Hi @werefreeatlast,
      i had successes asking LLaMA 3 70B with simple specific questions …
      Context : i am bad at programming and it help me at least to see how i could use a few function calls in C from Python … or simply drop Python and do it directly in C.
      Like you said, i have to re-write & test … but i have a possible path forward. Clearly you know what you do on a computer but i’m not really there yet.

      • werefreeatlast@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        9 months ago

        But people don’t just know code when you ask them. The llms fo because they got trained on that code. It’s robotic in nature, not a natural reaction yet.