Chatbots fail to convince judges that they're human

Paul Marks, senior technology correspondent

Hugh Loebner with the bronze award, picturing Alan Turing (Image: Paul Marks)

A chatbot called Rosette won the $4000 annual Loebner Prize in Artificial Intelligence at the University of Exeter yesterday - but once again none of the four chatbots that were competing managed to convince any of the judges that they were human.

After computer pioneer Alan Turing in 1950 posited the notion that machines might one day be thought of as "thinking", the competition attempts to find a computer program whose chat responses are indistinguishable from a human's. They are nowhere near it.

Every year since 1991, the prize's founder, Hugh Loebner, has asked four judges to sit at computer terminals where they can talk to a both a human (who's hiding in another room) and a chatbot - but they are not told which is which. It's up to the judges to decide which is the person and which is the software and then rate the chatbots on how good they are at human mimicry. A chatbot has only seemed more human than a human once in the competition's history - but that, says Loebner, only occured when one human volunteer decided to behave like an early chatbot, skewing the results.

This year I was one of the judges, alongside Exeter computer scientist Anthony Galton, University of Sheffield AI expert Noel Sharkey and How It Works technology writer Jonny O'Callaghan. I found all the entrants to be extremely disappointing: the chatbot's identity was evident after only three or four lines of chat (sometimes less) and they often came up with irrelevant, off-the-wall responses. First place went to Rosette by Bruce Wilcox, second went to Adeena Mignona's Zoe, third to ChipVivant by Mohan Embar and finally Tutor, by Ron Lee, came last. All the programmers are based in the US.?

Conspicuous by its absence in the final was the increasingly impressive CleverBot, a chatbot which is learning how to converse like a human from crowdsourced online conversations.

Developer Rollo Carpenter, of Dawlish in the UK, rues entering a less-capable, cut-down version of Cleverbot (which was easier to download) in the Loebner prize's selection round, which saw it finish outside the top four headed for the final. The cutdown version, he says, could not cope with a certain unexpected style of question that the full version could easily have handled.

It may be time to move on from the traditional text-based, linguistics-centred Turing test in any case, says Galton. He's proposing a computer intelligence test that takes into account the contribution the human vision system makes to intelligence. You can try it out here.

Source: http://feeds.newscientist.com/c/749/f/10897/s/196d41ab/l/0L0Snewscientist0N0Cblogs0Conepercent0C20A110C10A0Cturing0Etest0Echatbots0Ekneel0Ebef0Bhtml0DDCMP0FOTC0Erss0Gnsref0Fonline0Enews/story01.htm

direct tv lion king photon plane crash plane crash lake powell reno nevada