The Jeopardy! Challenge and Beyond Eric W. Brown (IBM Research)
To win The Jeopardy! Challenge, a computer system needs to have broad domain knowledge, understand complex language, and provide high precision answers with high confidence at high speed. The answer itself could be a type, a form, an “it” or a “this”. After manual analysis of Jeopardy clues identified 2,500 types, the team reached the conclusion that a manually built taxonomy approach would not work. They needed to leverage the knowledge in text and (semi)-structured data sources, collect evidence by using entity extraction, relationship extraction, keyword search, temporal reasoning, statistical paraphrasing, and geo-reasoning, then analyze the evidence and use machine learning approaches to combine the evidence and reach the final answer. Three elements to build success: 1) right metrics to measure and improve the system; 2) disciplined engineering procedure and evaluation to make the tests reproducible; and 3) extreme collaboration. A team of 12-20+ researchers over four years reached the relevance goal, and then four engineers spent 18 months to scale the system to an acceptable speed. Crawled 20,000 clues for experiments, half of which (11,000) were kept blind all the time. The rest were used for training and testing.
