OT: Jeopardy - Humans vs. Machine

Submitted by Bodogblog on

Starts tonight at 7:30.  Fascinating that this computer ("Watson", after IBM founder) will be able to understand human idioms and natural language.  It's quite complex if you've ever studied linguistics.  Jennings seems like a bit of a tool, but I watched almost every one of his episodes during that long run.  Brad Rutter is the other contestant, the record holder for highest cumulative winnings. 

Watson will likely be the forerunner of technologies that affect our lives for several decades.  Bonus opportunity for all those folks posting "Go Big Blue" to actually be correct in their rooting interest, and be the first to bow down to the machines.  I side with John Connor.  

rdlwolverine

February 14th, 2011 at 11:02 PM ^

I didn't get a chance to see the show, but from the description and as someone who has been on the show multiple times, it sounds like Watson had an advantage on the buzzer.  Typically, top players know the answer from reading the clue before Alex has finished reading it out loud.  But, they are not able to buzz in until he is finished reading (changed rule because of Chuck Forest a UM law student who would buzz in as soon as the clue was revealed)..  After Alex is done reading the question out loud a light goes on and the players are then free to buzz in.  If you wait to see the light, you will  be too late though.  So, you have to try and time your buzz to match Alex finishing the reading and hope you have timed it right with the human that activates the switch that allows you to buzz in.  It sounds to me that Watson who is electronically fast at responding to the light coming on can wait til Alex is done and then only buzz in when it gets the electronic signal that it is now free to answer.  I hope to watch tomorrow myself.

WolverineBlue

February 15th, 2011 at 12:39 AM ^

I agree that Watson potentially has a huge advantage in terms of being able to buzz in first. Since Watson has no vision, it has not been made clear how it knows when the lock-out lights are illuminated. They are triggered by a production assistant who flicks a switch behind a screen when Alex is finished reading the question. If Watson is directly wired to that button, that is a gigantic advantage, in that a computer can respond in microseconds and human reflexes are more in the range of one or two tenths of a second.

As a former Jeopardy! champion myself, I have more than a passing interest in this experiment and must say that I have not been blown away by Watson's performance on the first day. It is particularly telling how incredibly stupid its second and third choice answers usually are. They often are not even remotely responsive to the question and suggests that Watson has not very accurately parsed the meaning of the question or even the category. I have also worked in a Natural Language research group, and feel that the algorithms they are using are less than impressive in terms of really understanding what question is being asked. They are mostly just using huge amounts of computer horsepower to do pattern matching.

BlockM

February 15th, 2011 at 8:32 AM ^

First of all, that second part about not being impressed with the natural language processing  is just dumb. 

Also, do you really think that Watson wouldn't be just as fast if they had a camera hooked up to it that was focused in on the light? It's not a "huge advantage," it's just something computers are better at than us. That's the whole point of the exercise.

rdlwolverine

February 15th, 2011 at 12:31 PM ^

As someone that has been on ten shows, believe me it is a huge advantage.  For most clues, at least two of the players know the answer before Alex is finished reading it and it is a contest to see who can coordinate his or her buzz-in with the guy who decides that it is OK to buzz in.  The ability to buzz in successfully in those situations is the single most important factor in succeeding in single jeopardy and doube jeopardy.  I would say the second most important factor is knowing when not to guess.

Yes, it is "just something that computers are better at than us,"  but it is still a huge advantage.

BlockM

February 15th, 2011 at 12:59 PM ^

What's the point of the exercise then? If I were trying to prove I was better than Michael Jordan in basketball, would I be able to say, "Well, you can't use your huge muscles and superior skills, because that would be an advantage."

rdlwolverine

February 15th, 2011 at 2:06 PM ^

by allowing the players to buzz in at any time after the clue is revealed and Alex starts reading (which is how it used to be during the Art Fleming era and into the mid-1980s of the Trebeck era).

At least then it would come down to who can take the information, process it and come up with the correct response the quickest, rather than who can respond to the freeing up of the buzzer system the fastest. 

WolverineBlue

February 15th, 2011 at 5:38 PM ^

If a camera was hooked up, and had a frame rate of, say,  60 frames a second, it could always answer within a 60th of a second, which is faster than any human is capable of (absent occasional luck in anticipating).Rdlwolverine is exactly right in that speed is the deciding factor. I guarantee you that both human contestants know at least 75-85% of the answers and are just getting beaten to the buzzer. If it is hooked up directly, it can answer even faster than 1/60th of a second.

As far as not being impressed with the natural language processing, that is not dumb. When the category is "Name the Decade" and Watson's third best answer is something like "Athens", that is a very clear indication that it has not parsed the category or question very well.

BlockM

February 15th, 2011 at 7:04 PM ^

Oh please. This computer is using a fraction of a second to estimate as closely as possible the correct answer to often cryptically worded problems. Any algorithm has certain types of situations it functions best in, and Watson's algorithm is performing incredibly well in a massive range of situations. If this isn't bleeding edge technology, I don't know what is.

WolverineBlue

February 16th, 2011 at 2:20 AM ^

The computer has 3-5 seconds to come up with an answer. Not knowing that "Athens" is an invalid answer in the "Name the Decade" category is sad. I have worked with some of the leading researchers in natural language processing (former IBMers now at Microsoft). Watson is bleeding edge hardware, but the software is something of a hack that is very specifically targetted at playing Jeopardy! and not at finding answers to generic questions.

Blazefire

February 15th, 2011 at 5:46 PM ^

I would be interested to read more on the programming for Watson's decision making capabilities. Obviously, it must work on some sort of probability, and reaching a conclusion that a given answer is "likely" to be correct. I wonder how certain it has to be, and how it can find that percentage for the various types of questions.

Example: Who was the 8th recognized Pope of the Roman Catholic Church? That's a given. The answer is 100%. Either you've got it in time, or you don't.

Then you've got something like:

An off key musical note and a flavor. Sour, right? Yes, but there are a LOT of language databases to search, including words that have fallen out of use in modern english, slang, words borrowed from other languages, etcetera. Maybe that has a 90% chance of being correct. Does Watson buzz in on that? Does it keep searching to get certianty to 95%?

Blazefire

February 15th, 2011 at 8:01 PM ^

A few things:

First, I'm shocked, but it appears that programers either forgot or neglected to consider that category names would be important to Watson's understanding of a question. Clearly, it did not realize that the category was "U.S. Cities".

Secondly, I would imagine that one of Watson's troubles is that it does not know how to define a "hero". So unless the full text of articles from local newspapers describing Edward O'Hare as recieving a "Hero's Welcome" was scanned into its memory, and unless it could access that much full text that fast, it'd have no hope.

I have no doubt that it probably DID have the full text of the Chicago Tribune in its databanks, but with no premise given as to the year, OR the chicago tibune, Watson would basically have to search its entire database line by line to put that one together.

MGoDubs

February 15th, 2011 at 8:03 PM ^

You could tell that the humans were obviously frustrated today that they could never click in. On numerous occasions I saw the humans clicking frantically. This is a huge disadvantage and makes it much less interesting to watch IMO. But I have to agree with past post that if the humans want to have any chance they are just going to have to click immediately and hope they can get the right answer afterwards, simply trusting their knowledge. 

rdlwolverine

February 15th, 2011 at 11:17 PM ^

I was able to watch tonight and confirmed my feeling that the buzzer set up gave the humans no chance.  Not sure why they were able to buzz in the few times they did; whether it was Watson not buzzing, Watson needing more time to process on those few questions, or just totally lucky timing. 

Mgodubs is right that only chance for humans would be to buzz in early, but that cannot be done.  Thus the humans only chance was that Watson would miss enough to give them a shot.