Good Questions, part 4

Sad to say, I just found out I’ll miss the next Basement Bowl, due to vacation. Drat! On the plus side, I’m scheming to attend the Trivia Championships of North America, a weekend-long trivia explosion scheduled for Las Vegas in July. In any case, it’s time for one more installment of this series. Previous posts have focused more on the philosophical aspects of question construction, but in this one, I’ll get a little more technical — more about the craft than the art, as it were. I think I’m about out of gas after this, so let’s call it the season finale and get rolling.


Good Questions, part 3

I had another great trivia day last Saturday, this time a “Clubhouse Bowl” — just like a Basement Bowl, except held in a guy’s apartment clubhouse rather than a basement. There were trivia bowl-style games along with a bunch of Jeopardy! games run on a magic Jeopardy! simulator created by one of the gang. There was even “Trivia Battleship” — a wild cross of quiz-bowl questions with the classic strategy game. Correctly answered toss-ups would earn one shot against the other team, while bonuses could earn up to four more shots. Very fun.

Predictably, the whole thing primed me to whip up another episode in this series. Since I am apparently an endless font of opinions about good practices for trivia question-writing, let’s get started:


If you’re a Ghostbuster, crossing the streams is a bad thing. If you’re a trivia question writer, crossing the streams, by which I mean mixing the broad categories to find interesting hybrids, can be a very good thing indeed. There are plenty of sports questions, and plenty of movie questions, but how about sports movie questions? How about athletes who played bit parts in movies? How about movie-related nicknames given by Chris Berman to various athletes? The intersections between trivia categories can be fertile ground for some appealing questions, and can allow people who are normally weak in a category to kick ass in unexpected ways.

A great example of this came up at the recent Clubhouse Bowl. Dave Gatch handed out a sheet of movie stills, and asked us to tell him what song was playing during that point in the movie. Sound tough? Take a look at these examples (not the ones he used) and see if you can’t do exactly that:





Combining two categories (in this case movies and music) opens up new avenues of fun, activates players’ brains in new ways, and gives your game a feeling of greater unity.


I am no good at sports questions. Whenever I hear a sports toss-up begin, my hand relaxes on the buzzer, and I know there is very little chance that I will have anything to contribute. I start looking over at our team’s sports guy (yeah, it’s nearly always a guy) with hope and gratitude. I know the basics, but I just do not follow sports enough to know much beyond that.

Nevertheless, I include sports questions in all my regular (i.e. non-specialized) trivia games. Why? Writing sports questions helps improve me, both as a writer and as a player. Writing questions that are outside my comfort zone forces me to research things I don’t already know, some of which I may even remember later on down the line. This research also turns up unexpected gems of information which are quirky enough both to make a great question and to make the piece of information it concerns memorable enough to stick with me. Like, for example, did you know that Guy LaFleur, all-time leading scorer for the Montreal Canadiens (hockey team), recorded a disco album?

Incidentally, I do the same thing on Sporcle, a great site for trivia quizzes. I like to take Sporcle quizzes in areas where I’m strong, like music, movies, and literature. But I also like to take them in my weaker areas, like geography, history, and sports. Generally, I like to take a new random quiz, and then retake an old quiz. I pick which one to retake by sorting the list of quizzes I’ve taken, and identifying the one with the lowest percentage of right answers. Consequently, I’ve taken the NHL all-time team leaders quiz about 10 times so far, and my best score is 26 out of 120. That’s a huge improvement on my first score, though, which was 8 out of 120. And now I can tell you about a bunch of hockey players I’d never heard of before I started in on that quiz. (Which is how I learned the weird fact above.)

Another way to stretch is to try broadening your knowledge of areas in which you’re already strong. For instance, I love movies, and I know some Oscar trivia, but there is so much more for me to learn, and the Academy Awards are a very common trivia topic. So I write Oscar questions in areas I don’t know well, both to challenge players and to make me a better player and writer. (Just as an aside, if you want to improve as a trivia player, be on the lookout for creative ways to strengthen your knowledge. For instance, Windows 7 has a feature which lets you rotate through a set of images for desktop wallpaper, changing automatically at an interval you select. So I went out and snagged an image of every Best Picture winner, dropped them all into a folder, and have the wallpaper machine circulating among them. Now when I use my computer, I also get a little help remembering which movies have won the Best Picture Oscar.)


Something that makes trivia games great fun is their ability to point you to wonderful corners of culture that you never knew existed. I’ve been introduced to lots of great movies, music, TV, and other stuff via trivia, and I try to do the same for others. It’s great fun to write questions on topics you feel passionate about. At the same time, at least for me, that’s too narrow a field. I’m a white guy who grew up in the 80’s, and I have an endless well of questions I could write about cultural artifacts I got attached to in my time. But while I want my games to be fun for me, I also want them to be fun for people other than me. Consequently, I try to include questions about areas of culture that don’t mean as much to me, sometimes even things I actively dislike. As I sometimes point out, inclusion does not imply endorsement. I suppose this might seem like a restatement of the “stretch” point, but there’s a slightly different intention behind it. I try to spread the love among all different kinds of knowledge not just to make myself a better player, but to remember to include a diverse variety of topics so that my games are fun for a wide variety of people.

Okay, that’s all for now. How about those movie songs?

1. “Bohemian Rhapsody” by Queen plays during that scene from Wayne’s World.

2. The Righteous Brothers’ “Unchained Melody” is the music behind that scene from Ghost.

3. Any boy who grew up in the 80s (see, I told you that was my wheelhouse) is likely to remember The Cars’ “Moving In Stereo” as the soundtrack to Judge Reinhold’s fantasy sequence about Phoebe Cates in Fast Times At Ridgemont High.

4. Before Tom Cruise was famous for being crazy, he was famous for dancing around to Bob Seger’s “Old Time Rock & Roll” in Risky Business.

Good Questions, part 2

As promised/threatened, here’s another installment of “Paul’s Random Thoughts About Trivia Questions.” Carrying on with question-writing principles:


In my last post, I spent some time harping on the fact that trivia games are supposed to be fun. Your job as trivia question writer is to provide an enjoyable experience to your players, and humor is a crucial tool for that job. On its most basic level, it can liven up a somewhat bland question, like this bonus from the 2005 TRASH regionals:

Given clues, name the subjects of the following celebrity biographies, all of whom share a favorite hobby, for ten points each.
1. A Paper Life details her adventures with her allegedly abusive actor father, with ex-husband John McEnroe and with heroin.
2. Don’t Try This At Home, his chronicle of the year he decided to turn his house into a crack den, details his struggles with bandmates, his record label and heroin.
3. Scar Tissue recounts his life from toddlerhood and his drug dealer dad’s felonius additions to his mashed bananas, through forming a band with best friend Hillel Slovak, to his long standing affairs with Ione Skye, Sinead O’Connor and sweet, sweet heroin.

This question boils down to, “Name the celebrities based on the titles of their autobiographies and maybe some clues about their connections to other famous people,” which is just fine, but when you tie them together by their heroin addictions, dryly understate that as “all of whom share a favorite hobby”, and hurl a fastball zinger punchline at the end like “long-standing affairs with Ione Skye, Sinead O’Connor, and sweet, sweet heroin,” a run-of-the-mill question turns into one of the best ones in the game. (Though in my opinion the second part needs another clue or two.)

The other thing that the humor in that question does is to tone down a fairly dour topic. There’s a bit in the crosswords documentary Wordplay in which puzzle creator Merl Reagle explains that there are some words that you don’t see in crossword puzzles, even though they might be very useful cruciform words, just because their content is too distasteful. He calls this “the Sunday morning breakfast test”: “They’ve waited all week for this. They’re sitting there relaxing…and here comes RECTAL? I don’t think so.” Trivia games allow a wider latitude, especially when written for an informal event like a Basement Bowl, but still, you don’t want to offend, disgust, or annoy your audience. So when you find that question that you just have to write but whose subject matter is a little questionable, a little humor smooths the way.

Something about being in a Billy Wilder movie frequently makes people want to kill themselves. Fortunately, they rarely succeed. I’ll give you an actor and a suicide method, you tell me the Billy Wilder-directed movie, for five points each, 40 points for all 7:
1. Audrey Hepburn, carbon monoxide poisoning
2. Shirley MacLaine, an overdose of sleeping pills
3. Gloria Swanson, slashed wrists
4. Ray Milland, handgun
5. Marthe Keller, jumping in front of a train
6. Carol Burnette, jumping off a building
7. Jack Lemmon, hanging

That’s a question I wrote for the Basement Bowl a couple of years ago. I’d just come off a Wilder-watching jag and was amazed at the number of suicide attempts in his movies. It was great trivia fodder, except that quizzing about suicide after suicide is kind of heavy. Thus, I lead with a joke to lighten it up.

Perhaps the best reason to joke liberally is to relieve the tension that can sometimes build in trivia competitions. Relaxed players not only have more fun, I think they play better too.


Trivia games can and should be more than memory and speed tests. Yes, of course, those two skills (in varying proportions, depending on the format) constitute the backbone of a trivia game, but crucial to the art is creating opportunities to connect recall with thought. One of my favorite recent examples of this was invented (as far as I know) by Bill Schantz:

I’ll give you the year and the first letter of each word in its title, you name the 80’s pop song, for 10 points each:
1. (1983) E.B.Y.T.
2. (1983) O.T.L.T.A.
3. (1984) W.M.U.B.Y.G.G.
4. (1984) O.N.I.B.

You may have awesome recall of 80’s songs, but that by itself won’t get you very far with this question. Instead, you have to use part of your brain to generate plausible strings of words based on a breadcrumb trail of letters, while another part of your brain tries to connect those words to song titles that you know from that era. Even better is when you hear the letters, feel a moment of instantaneous synthesis, and just know what the answer is. Hitting this answer is even more satisfying than the average trivia pull, because of the additional solving effort necessary. Of course, in any kind of timed competition, it’s imperative to balance the time constraints (and attendant pressure) against puzzle elements in your questions. If they require too much thought, they’ll bog the game down.

Puzzle questions like this make regular appearances on Jeopardy!, and that show’s writers are particularly good at coming up with clever new ways to keep contestants on their toes. Generally the twist is in the nature of the category. Some recent examples, courtesy of the J! Archive:

Clue: Riesling, Syrah
[A fairly gentle example, in which the puzzle is to figure out how the category is working. The first clue in any Jeopardy! game is usually easier than its successors, but in a category like this, there’s generally a pointedly clear indicator in that first clue.]

Clue: In Romania they say “da”, in Japan, “hai”, & in Panama, this
[Here’s the flip side — a clue that can be answered on its own, but the category narrows down the set of possible answers. Jeopardy! does this all the time with its “quotation mark” categories, in which some piece of the category is in quotes, meaning that correct answers must start with or contain the quoted string of letters. A clueful category like this, though, is much more fun to discover.]

Category: GOD SPELL
Clue: Vulcan is the Roman equivalent of this Greek fire god (10 letters)
[There’s no puzzle element to this one — Alex makes clear at the beginning that contestants will have to spell their answers. It’s notable, though, because it exercises a different sort of recall than the average trivia question.]

Clue: Of presidential surnames
[This is an example of a question in which contestants must recall a set of data and then do some kind of processing on it. And because of the high-pressure nature of Jeopardy!, they must do it very quickly, leading to the sort of intuitive pulls that supply the pleasure of answering this style of question.]

Category: OF ORDER
Clue: U.S. cities, from west to east: Newport News, Milwaukee, New Orleans
[In this one, they provide the set of data, but the answer is still the result of some processing on the part of the contestant.]

This could go on and on, and does, but I trust the point is made. Finding new angles from which to challenge the player is a great way to increase the fun of the game.


Another surefire way to make your game more fun is to write questions that step outside the typical text format and engage some of your players’ senses. This is one of my favorite things to do, and consequently my Basement Bowl games have been littered with visual clues:

Logo design is an art, and sometimes it takes a few tries to get it right. Given the defunct logo, tell me the NFL team it represents for five points each.
4 different old-time logos for NFL teams

I find images via Google Image Search, copy and paste them into a Word document, resize as necessary, then print out a couple of copies on the color printer. I hand these out to players before I read the clue. A couple of things I’ve learned: make two copies (since in the Basement Bowl missed bonus answers can transfer over to the other team), and make the pictures large enough that they can be seen in a dimly lit basement.

Then there’s the audio. I’m a huge fan of the audio. For instance, I’ve written a couple of all-audio games of one-hit wonders, another one of mellow gold tunes, and another couple of all female artists. Thanks to the wonders of digital audio and the fabulous Audacity, it’s extremely easy to create mp3 song clips and burn them onto a CD. I bring my nifty mini-boombox to the basement, and play the game as all tossups, no teams. Each person has a buzzer, and they score points by buzzing in first to correctly identify a song, generally both the title and artist.

Sometimes I’ll even go a little more complicated. For instance, with my female artists game, I recognized that for a number of the clips, someone might be able to identify the artist even if they couldn’t name the song. So the rules were that if you buzzed in before the clip ended, you had to identify both the title and artist. Correct answers scored two points, with a bonus point for naming the singer in the case of groups (e.g. if I played The Pretenders you could get a bonus point for naming Chrissie Hynde.) If you buzzed in after the song was over, you could name the artist for one point.

One thing I learned for my second round of this type of game was to end the clip with a distinctive sound, like a ding. The first time, nobody was sure when the clip would be over, so there was some hesitation from people who didn’t want to get penalized for buzzing too early.

I’ve done audio toss-ups in regular games too, and not just music. There’s plenty of great fodder in movie clips, comedians, interviews, and miscellaneous distinctive sounds, such as the sound of Pac-Man dying. And there is further sensory fun to be had. I once wrote a bonus in which I handed out soda-pop flavored Jelly Bellies, and had players identify the soda from each one. I’ve had players get up and dance the Batusi. Another quizmaster had people do 4 different dances from A Charlie Brown Christmas, which I thought was brilliant.

The point is that you can add a lot of variety and excitement to your game by creatively extending your questions into nonverbal (or more-than-verbal) realms. Not only that, these kinds of questions can bring out hidden strengths in your players, allowing them to have more fun by kicking ass in new ways.

That’s enough for tonight. More installments to come. But never fear, I would never sign off without providing the long-awaited answers:

Sweet, sweet heroin:
1. A Paper Life: Tatum O’Neal
2. Don’t Try This At Home: Dave Navarro
3. Scar Tissue: Anthony Kiedis

Wilder suicides:
1. Audrey Hepburn, carbon monoxide poisoning: Sabrina
2. Shirley MacLaine, an overdose of sleeping pills: The Apartment
3. Gloria Swanson, slashed wrists: Sunset Boulevard
4. Ray Milland, handgun: The Lost Weekend
5. Marthe Keller, jumping in front of a train: Fedora
6. Carol Burnette, jumping off a building: The Front Page
7. Jack Lemmon, hanging: Buddy Buddy

80’s initial songs:
1. (1983) E.B.Y.T.: Every Breath You Take by The Police
2. (1983) O.T.L.T.A.: One Thing Leads To Another by The Fixx
3. (1984) W.M.U.B.Y.G.G. Wake Me Up Before You Go-Go by Wham!
4. (1984) O.N.I.B. One Night In Bangkok by Murray Head

Clue: Riesling, Syrah
Question: What are grapes?

Clue: In Romania they say “da”, in Japan, “hai”, & in Panama, this
Question: What is “si”? (Homophone with “C”)

Category: GOD SPELL
Clue: Vulcan is the Roman equivalent of this Greek fire god (10 letters)
Question: What is H-E-P-H-A-E-S-T-U-S?

Clue: Of presidential surnames
Question: What is Wilson

Category: OF ORDER
Clue: U.S. cities, from west to east: Newport News, Milwaukee, New Orleans
Question: What is New Orleans, Milwaukee, Newport News?

Olde-tyme NFL logos:
1. New York Giants
2. Buffalo Bills
3. Denver Broncos
4. Washington Redskins

Oh, and finally: Each heading in this edition is a cultural reference.
“Be more funny!” is a classic Simpsons gag.
“Cudgel thy brains” is from Hamlet by Wiliam Shakespeare
“Senses Working Overtime” is a song by XTC.

Good Questions, part 1

Now, it’s true that I’ve had some trivia experience. However, I wouldn’t exactly claim to be an expert on question-writing. Compared to many of the people I know from that world, I’m a raw newbie. Not only that, there are people out there in the world who actually make their living (or at least a side income) from writing questions, like for example Paul Paquet, who penned just about the best article I’ve seen on writing good quiz questions.

Nevertheless, I’ve been writing trivia questions for a number of years now, and along the way I’ve contracted some opinions on what makes a question good or not-so-good. And what is a blog for if not to toss your unsolicited, inexpert opinions out to a disinterested world? So without further preamble, here are some of the principles I’ve found important in question-writing.


This is one of those pieces of advice that really applies to any kind of writing. It’s even more crucial in writing for a game, though, because when you’re writing something static like a novel or (say) a blog post, your reader can walk away with impunity. Interaction increases audience investment in the experience, giving you a bit more of a captive audience, especially when the interaction is of a social nature, like trivia. Be a good captor. Write questions suited for your, um, prisoners. Okay, I’m walking away from this metaphor before it turns into an extended meditation on Stockholm syndrome.

I’ve primarily written trivia questions for two sets of people. One set is the Basement Bowl regulars, trivia enthusiasts (and often champions) whose collective knowledge is astonishing. The other set is my co-workers — for several years I published a weekly trivia quiz at my job to promote social mixing, have fun at work, and raise morale. (It worked great until I got too isolated, overworked, and demoralized.) The trivia came in different formats, but the differences were deeper than that. I pitched the questions differently, because the two groups have different ideas of fun. For instance, I might write a question like this for the Basement Bowl:

Some directors’ first films were huge, memorable hits. Others… not so much. I’ll give you a director’s first film along with its year of release, you name the director, for ten points each. The films I’m choosing are feature-length, theatrically released in the U.S., and solely directed by that person.
1. Who’s Afraid Of Virginia Woolf? (1966)
2. Hard Eight (1996)
3. Eight (1998)
4. Cars That Eat People (1974)

When I posed that same question for my co-workers, I gave the director’s first three films:

1. Who’s Afraid Of Virginia Woolf? (1966), The Graduate (1967), Catch-22 (1970)
2. Hard Eight (1996), Boogie Nights (1997), Magnolia (1999)
3. Eight (1998), Billy Elliot (2000), The Hours (2002)
4. Cars That Eat People (1974), Picnic At Hanging Rock (1975), Black Rain (1977)

[Answers at the bottom of this post]

On my co-workers’ quiz, I also provided 26 more questions, including people with a very recognizable first few films: Spike Lee, Quentin Tarantino, Michael Moore, David Lynch, Woody Allen, etc. Why? Because your basic group of office workers, who are doing trivia as a fun break, are not likely to be the sort of film fiends who would recognize a lot of obscure early work from now-famous directors. If they don’t have a clue on any of the questions, they’re likely to roll their eyes and never take one of these stupid quizzes again. Nobody likes to be completely stumped over and over again. Which leads me to my next point…


Trivia is supposed to be fun, and there’s nothing fun about feeling like an idiot. A good trivia game should leave you feeling smart, not stupid. That means that you should pitch your questions to a range with “fairly easy” on one end and “fairly difficult” on the other. Actually, “pitch” is the wrong metaphor — a pitcher is trying to prevent the batter from getting a hit, but a quizmaster should not be trying to prevent players from getting right answers. It’s not very hard to stump trivia players, even the greatest trivia players. In fact, I’d say that anybody can write a question that will stump a given person. All you have to do is ask for obscure enough information. Nobody knows it all, nor should they need to. The experience you should be trying to provide is one of success mixed with challenge.

I aim for about a 70/30 ratio between these two, but of course I rarely hit that. Gauging difficulty is one of the hardest things to do in writing questions, especially in areas where you either know a whole lot or very little. In the former case, all the information feels so familiar that it’s hard to get a sense of what a regular person might know. For instance, I’d have difficulty writing a full Simpsons quiz without a little feedback from somebody who hasn’t watched the show for ages. Is asking for the Simpsons’ address a hard question or an easy question? Seems easy to me, but I don’t trust my perceptions about it.

By the same token, when you know very little about a subject, all the questions seem hard. One of my running jokes in the Basement Bowl is that I always apologize for my sports questions in advance, because I very frequently have only a dim sense of how easy or hard they might be. I look at each one and think, “I sure would never have known that without looking it up!” Knowing I’m going to err, I try to err on the side of success.

So if kicking ass is more fun, why not make all the questions super easy? Well, because if you’re answering questions that practically any literate person could get without effort, you don’t feel like you’re kicking ass. Consider this question: “This network is home to Good Morning America, Cougar Town, Grey’s Anatomy, Nightline, and The Academy Awards. It also shares its name with the first three letters of the alphabet. What is it?” The first sentence is a fair question for general audiences, but the second sentence makes it into a terrible question for almost all audiences. What’s fun about being quizzed on things that a four-year-old would know? (Unless you are a four-year-old, of course — see “Know Your Audience.”) The one situation in which I could see this question working is a buzzer-beater game, in which the questions start out giving an advantage based on knowledge, but if nobody can capitalize on that advantage, it turns into a “fastest thumb” challenge.

On the “fairly difficult” side of the spectrum, as I said, the goal shouldn’t be to stump people. There are some questions in every game that a player just isn’t going to know, and that’s fine. What’s important is to avoid asking questions that nobody would know. Ideally, you want players to look back on the questions they missed and think, “I should have known that!”, not “Who on earth would know that?” The other good reason to have fairly difficult questions in the mix is that sometimes they give players the opportunity to reach out and clock an unexpected home run, which is the most kick-ass feeling of all. (Wow, the baseball metaphors sure are offering themselves to me tonight.)


Here’s a sample question from my “Rock And Roll Hall Of Fame Trivia Challenge” calendar:

In 1965, Leo Fender sold his Fender Guitar Company to CBS for what price?
a. $13 million
b. $19 million
c. $22 million

Remember when I said that you don’t want players to look at a question and think, “Who on earth would know that?” Well, an even worse reaction to generate is, “Who on earth would care about that?” Which is exactly what I think when I see this question. The sale amount for a guitar company is a very banal piece of information. About the only person who’d have an emotional attachment to the difference between these numbers is Leo Fender himself. Trivia calendar, Leo Fender is not your audience. In general, this calendar feels like the product of somebody combing through a book of “This Day In Rock History” facts. Which means that you get some pretty good and interesting questions (“Which album by Johnny Cash was the first country album to top the U.S. pop chart?”) but a lot of questions like the one above. If I were this person’s editor, I would emphasize the fact that we’re supposed to care about the answer.

What could make this question better? Well, if it were actually an interesting amount of money, that’d make it reasonable. For instance, if the multiple choice answers were “a) $1,300 b) $13 million c) $130 million”, that’d be a step in the right direction. Each of those answers tells a pretty different story about what that sale might have meant to Fender, and whether the company was valued properly, which is likely to be more compelling information. Of course, it’d be even better if the answer were one of the outliers, but it isn’t.

Another way to improve it might be to switch around what the question is actually asking about: “In 1965, who sold his guitar company to CBS for $13 million?” That doesn’t quite get us there, though, because (in my opinion) it falls outside the realm of something the average music fan could reasonably be expected to know. So we’d want to inject a hint or two in there: “In 1965, what designer of the Telecaster and the Stratocaster sold his guitar company to CBS for $13 million?” Now we’re asking for an association that is fair game for music fans, and we’ve got ourselves a reasonable trivia question.

I’ve got more, but it’ll have to wait, because this post has gone on long enough. Meanwhile, how about some answers?
First films:
1. Who’s Afraid Of Virginia Woolf? (1966) — Mike Nichols
2. Hard Eight (1996) — Paul Thomas Anderson
3. Eight (1998) — Stephen Daldry
4. Cars That Eat People (1974) — Peter Weir

The Simpsons live at 742 Evergreen Terrace in Springfield, which is famously vague about what state it’s in.

According to the calendar, Ring Of Fire was the first country album to top the pop chart. Wikipedia doesn’t seem to want to corroborate that today — it says the album only reached #26, though it was “the first #1 album when Billboard debuted their Country Album Chart on Jan. 11, 1964.” Not that Wikipedia is an authoritative source or anything, but neither is the RRHOF calendar.

What is “the future”?

That post about the art of the trivia question is still brewing, but I got sidetracked this week by another event in the trivia world. You may have heard about it. Watson, an IBM supercomputer, played two games of Jeopardy! against that show’s most famous champions, and thoroughly trounced the both of them.

A number of friends who watched the match complained that it was boring. If what you were looking for was a tense, movie-like contest with the drama of close scores or a come-from-behind victory, I can certainly see why you’d be disappointed. It had all the drama of the 49ers annihilating the Broncos 55-10 in Super Bowl XXIV. On the other hand, if what you were looking for was a glimpse of the world to come, in the form of a breathtaking technical achievement, this match absolutely delivered the goods.

See, some people tend to think computers are smart, and that of course a computer could beat a human at Jeopardy!, given a sufficiently broad knowledge base for its answers. But really, that’s a case of misplaced signifiers. Many human brains find rapid mental arithmetic of large or complex numbers difficult, and therefore associate it with intelligence. Computers happen to be fantastic at this kind of thing. The chess club is full of smart kids, and therefore chess must be a smart person’s game. Knowing that a computer could defeat the chess world champion must mean that computers are smart, right?

Here’s the thing, though. Computers are great at one thing: computing. Arithmetic is computation. Chess, at a sufficient level of abstraction, is also computation. The further away from numbers you move, the dumber computers become, meaning that for the vast majority of tasks our brains do each day, computers are extremely stupid. “Natural language”, aka the way we humans talk to each other, is an enormous challenge for a computer to deal with, as anyone playing interactive fiction for the first time could tell you. (Though the idea that better parsing of natural language will automatically make for better IF is another case of misplaced signifiers — better understanding of language is great and everything, but the more important part of IF is its model world. Advancing the parser just means the model world’s seams show more quickly.) Because computers lack human experience, they are stunningly bad at dealing with linguistic context, and are therefore capable of spectacular misunderstandings when faced with any language outside the very limited domains for which they’re programmed.

Watson is no exception to this, but it has a few advantages that other machines lack. For one thing, there’s an enormous amount of processing power behind it: some 90 servers, over 21 terabytes of data, 15 terabytes of RAM, and 80 teraflops of throughput. More important, though, are a couple of its conceptual approaches to knowledge.

First, through a paradigm called machine learning, Watson learns by example, getting better and better at the game as he sees more and more Jeopardy [leaving the exclamation point off from here on out] clues and their correct answers. It would be ridiculously impractical to try to construct a set of rules that would allow a computer to recognize every possible Jeopardy question, so instead Watson’s creators gave it a framework for recognizing associations between question words, answer words, and source texts, then fed it tens of thousands of Jeopardy clues as examples. This technique enabled Watson to make a huge leap in its Jeopardy prowess.

The other key aspect of Watson is its embrace of uncertainty. Watson doesn’t deal in right answers and wrong answers. It deals in answers that are more likely to be right vs. less likely to be right. Thus, when faced with the clue, “The parents of this 52nd governor of New York immigrated to the United States from Salerno, Italy,” we see its top three answers thus:

Three answers, with "Mario Cuomo" listed first and a certainty of 98% indicated. "motorcycle club" and "Marine Corps" are below, listed at 8% certainty.

Watson was quite certain that “Mario Cuomo” was the correct answer, but hadn’t entirely ruled out the far crazier answers “motorcycle club” and “Marine Corps.” Indeed, if what you’re seeking is comedy, look no further than Watson’s runner-up answers.

Laughs aside, though, it’s this uncertainty which makes Watson so formidable. In a frequently-cited example, Watson can look at the name “Alice Cooper” and weigh the evidence that Alice is a woman’s name against the evidence that Alice Cooper is a man, give each pile of evidence a score, and come to its own conclusion. A strictly rule-bound computer would have to be given a specific exception to handle this case. Watson can generate its own exception, thereby improving its knowledge base. As a co-worker of mine pointed out, isn’t this a hallmark of intelligence? The capacity to allow for the possibility that we may not know everything or fully understand the world is an incredibly powerful tool in the search for truth.

So as a computer, Watson rocks. But Jeopardy is an entertainment program, not a science program. Is it fun to watch Watson play Jeopardy? George Doro, my teammate in the Anti-Social Network, called it “more fascinating than exciting,” and that’s right on target. IBM branded the hell out of this show, and it would have been a black eye for them had Watson lost. Consequently, a few gameplay decisions were made which helped Watson win, but made the show a little less fun.

First off, Watson was allowed to be lightning-fast on the buzzer. People think of Jeopardy as a purely mental game, but unlike chess, there’s a physical component of Jeopardy. People (and computers) with faster reflexes do far better on the show — it doesn’t matter if you know 100% of the answers when you’re getting outbuzzed 80% of the time. Trying to play buzzer-beaters against a computer is like running a 500-yard dash against a car. Watson didn’t have to be this quick — just subtract a little of that processing power until the computer’s average buzz-in time equals the average human’s buzz-in time (or even Ken Jennings’ average) and you’ve got a fairer battle, but instead, when Watson was certain enough of its answer, no human thumb could possibly outrace its mechanical plunger. (There were a few exceptions, but overall it was clear that Watson’s buzzing speed was what allowed it to dominate the match.)

Secondly, there’s the fact that each human had not only Watson to contend with, but also another top-notch Jeopardy player! Consequently, anytime Watson doesn’t pick up a clue in time, the two humans tended to split the points between them. I know Jeopardy is traditionally played by three contestants, but there was plenty about this match that was non-traditional. I would be very interested to see how Jennings would do against Watson by himself, especially if the buzzer advantage were corrected. As he put it in an NPR interview: “It’s the worst of both worlds, you know? The ideal scenario would be to have a human versus a computer, or maybe a computer versus a very good human and a lousy ‘Jeopardy!’ player. I don’t know if you saw Wolf Blitzer on the show, but I’d like to have Wolf back.”

That’s not to say that Watson was flawless. One of its major weaknesses was its inability to see or hear. Instead of listening to Alex Trebek read the clue, Watson was fed the clue via (essentially) a text message, so it saw and started processing the clue at the same time as Ken and Brad saw it. The show neutralized the most obvious disadvantage of this blindness and deafness by eliminating the audio or visual clues it often features. Jeopardy has made this sort of accommodation before, to serve disabled human players, and while it’s certainly true that Ken and Brad could have whomped the computer on those clues, that’s really not what Watson was built to do, so it would rather miss the point. A more pertinent disadvantage was that it could not hear what the other contestants were answering. It was told whether its own answer was correct, and told the correct answers provided by humans, but was not told of wrong answers, leading to this exchange:

Ken: “‘Name That Decade’ for a thousand.”
Alex: “The first modern crossword puzzle is published & Oreo cookies are introduced.” [Ken buzzes in] “Ken?”
Ken: “What are the ’20s?”
Alex: “No.” [Watson buzzes in] “Watson?”
Watson: “What is 1920s?”
Alex: “No. Ken said that.”

[The correct answer was “The 1910s.”] Trebek’s schoolmarmish correction of a machine that had just that moment proven it can’t hear him was amusing, and perhaps reflexive. Watson’s error was the kind of mistake that humans rarely make, though it’s not unheard of. When a human does it, though, it’s a sign of frazzled nerves. With Watson, it’s an Achilles heel. Well, maybe an Achilles toenail.

Another major weakness Watson displayed was its difficulty leveraging the category title to come up with the answer. Humans completely dominated that “Name The Decade” category — Watson was having trouble processing quickly enough to outbuzz them, and at one point its top guess for one of the clues was “2002,” even though it did come up with decades for the others. Most famously, in the Final Jeopardy round of the first game, it encountered the category “U.S. Cities,” and the clue, “Its largest airport is named for a World War II hero; its second largest, for a World War II battle,” which it answered thus:

Watson answering "What is Toronto?????"

(This inspired the funniest Watson joke I’ve yet seen: “Me: Hey Doc, I’ve got this pain in my left arm and an awful headache. Doc: What is Toronto?????”) The answer was in fact “Chicago,” but even if a human didn’t know the answer, he very likely would have guessed an actual U.S. city based on the category, rather than a Canadian city.

As some of the IBM guys pointed out, Daily Doubles and Final Jeopardy are a tough area for Watson, because it has to guess something, and therefore risk looking stupid. When it’s not sure about its answers on a regular clue, it can just refrain from buzzing in. Watching the show, I thought perhaps that Watson’s creators forced it to simply focus on the question, more or less ignoring the category. Turns out this isn’t quite true. In fact, it considers the category in its approach, but it’s learned from its thousands of Jeopardy clues that category is often only weakly tied to the answer. For instance, that Chicago question could have been reworded, “Chicago’s O’Hare airport is named after a World War II hero; this airport, its second largest, was named after a World War II battle.” The question still would have fit the category, but the answer would have been an airport, not a city. Watson has seen that scenario play out many times, and is thus wary of assuming that the answer in a “U.S. Cities” category will always be a U.S. city.

In the end, Watson defeated the humans soundly, with a score of $77,147 to Jennings’ $24,000 and Rutter’s $21,600. A lot of the press coverage has focused on the “man vs. machine” angle, and of course the match was set up to emphasize that. In fact, it was rather poignant to see Watson beat one of its human practice match opponents on the clue, “This African-American folklore laborer: ‘Before I let that steam drill beat me down I’ll die with my hammer in my hand.'” I guess there’s this sort of pastoral vs. industrial thing that gets set up when machines attempt a traditionally human activity, even though people holding buzzers and answering trivia questions doesn’t exactly fit neatly into the pastoral mold.

I don’t feel much solidarity with the OMG SKYNET IS HERE!!!!! response. As somebody who works in IT, I’m fascinated by the achievement. I think about how satisfying it must have been to have worked on the team that created this. Those people just finished a massive four-year project, and the result was an incredible leap forward in information processing, with a world-famous, historic, televised, wildly successful debut. I just finished my time as a team member on a three-year project, and the result is a shakily implemented student system whose portal is currently driving everyone crazy with how incomplete and slow it is. I’m sure there is mental, emotional, and physical damage associated with both project teams, but wouldn’t it have been wonderful to have been on the one whose final product worked so well?

In his Final Jeopardy answer, Ken Jennings wrote, “(I, for one, welcome our new computer overlords.)” It’s a reference to a hilarious moment on The Simpsons. And interestingly, it may not have been one Jennings thought of himself. Here’s an excerpt from his NPR interview with Neal Conan:

Mr. JENNINGS: Maybe it’s just my own ego, but yeah, I feel like I’ve somehow, through some weird coincidence, been elected as the champion of carbon-based life on Earth against, you know, our new future oppressor.
CONAN: Silicon, yeah.
Mr. JENNINGS: And I would like to strike a blow while I have the chance.
CONAN: I, for one, welcome our robot overlords.
Mr. JENNINGS: You may have no choice, Neal.

Then again, it’s quite possible that this interview was taped after the Jeopardy round was taped, so who knows? But whether Jennings was lifting a joke or simply making a reference, isn’t this the skill for which we celebrate him? He gathers knowledge from various sources, and retrieves it quickly, using it when it can make the most impact. His graciousness and humor in that final moment certainly set him apart from his predecessor in IBM challenge history, Garry Kasparov, who famously stalked away in an enormous huff after being beaten by Deep Blue. But in that graciousness and humor, he also subtly made the point that for all Watson’s skill and speed at information retrieval, humans can still wield that information with a precision and effect that Watson could never hope to achieve.

Trivial Matters With a Vengeance

Okay, so some of this is covered in my review of Wordplay, but here it is from a slightly different angle. But before we go there, answers to the last entry’s questions:

Following in the footsteps of Anthony Michael Hall and Jason Lively, he played Rusty Griswold in National Lampoon’s Christmas Vacation. More famously, he appeared as David Healey, Darlene’s boyfriend and eventual husband, in 92 episodes of Roseanne. Name this actor who currently stars as Dr. Leonard Hofstader on The Big Bang Theory.
Answer: Johnny Galecki

In the 1960s, Marvel Comics loved to liven up its titles by throwing in an extra adjective. I’ll give you a comic book title, you fill in the missing adjective, for five points each.
1. The Incredible Hulk
2. The Amazing Spider-Man
3. The Invincible Iron Man
4. The Uncanny X-Men
5. The Mighty Thor
6. The Astonishing Ant-Man

During CU‘s 2001 revival of the Trivia Bowl, I heard rumors of this thing called a “Basement Bowl.” I gathered it was some kind of trivia-oriented deal held in somebody’s basement.

Trivial Matters 2: Electric Boogaloo

In my last entry, I explained the structure of the trivia bowl, and talked about its history at CU. Before I continue, let me provide some answers to the questions I posed there:

Q: James Rado, Gerome Ragni, and Galt MacDermot were the writers behind what song medley, which won Record Of The Year, topped the charts for six weeks in 1969, and was the biggest hit in the 5th Dimension’s career?
Answer: “Aquarius/Let the Sunshine In”

I’ll name a fictional computer from a movie, you name the movie, for ten points each.
1. MU-TH-R 182 model 2, the ship-board computer on the space ship Nostromo, known by the crew as ‘mother.’

Answer: Alien
2. Deep Thought, a computer created by a pan-dimensional, hyper-intelligent race of beings who look to us exactly like white mice.
Answer: The Hitchhiker’s Guide To The Galaxy
3. EMERAC, a room-sized computer recently acquired by the Federal Broadcasting Network, whose worth is advocated by inventor Richard Sumner and doubted by reference librarian Bunny Watson.
Answer: Desk Set
4. WOPR, or War Operations Plan Response, a military simulator housed at NORAD.
Answer: Wargames

Now for my trivia autobiography (a redundant phrase, perhaps.)

The Beatles — Love

I have to confess, the whole Love thing completely passed me by. I vaguely heard, four years ago or so, that Cirque du Soleil was doing some Beatles tribute show. Because I’ve never seen a Cirque show, and because my imagination has been stunted by television and video games, all I can picture is a bunch of guys in tights, swinging from trapezes, forming human pyramids, and so forth. To Beatles music. I yawn, and move on.

Then my friend Trrish made a special trip to Las Vegas to see the show, and told me a bit more about the way the music worked, and I was intrigued enough to pick up the album. And now that I’ve listened to it ten times in a row and had my mind completely BLOWN, I have to write about it, and I have to evangelize it to everybody I know who loves the Beatles. So I will gladly acknowledge the gooniness of gushing over an album 4 years after it comes out, just because it happened to take me that long to get around to it. It’s like writing a bunch of breathless blog posts about Buffy The Vampire Slayer in 2006, 3 years after it was cancelled. Which, y’know, I also did. So I guess this sort of thing is really my stock in trade nowadays.

If you’re way out of the loop like me, let me tell you a little about this album. It’s the soundtrack to the Cirque show, yes. But more importantly, it’s an exquisite, loving collage of Beatles sounds, made by people who have the music in their DNA: George Martin and his son Giles. They take the songs and pull apart their elements, mixing them together ingeniously, so that a solo from one song might float through the intro of another, fading smoothly into a third. Every sound on the record (with one exception, which I’ll discuss later) is from a Beatles recording. Sometimes the songs stand on their own for a while, sometimes the mixtures are subtle, and sometimes they jump out and grab you by the throat.

The sound is uniformly fantastic. I’ve never heard the songs sound better, even (or, perhaps, especially) when they’re not mixed up. Listening to this CD is an enormously thrilling experience if you have the music deeply engraved in your brain like I do. Seriously, if you love the Beatles, and you haven’t heard this album, stop reading right now, buy it, listen to it, and then come back. I’ll be here. You probably want to experience it fresh. I recommend headphones.

And just so you know, if you’re sensitive to this kind of thing in an album review: here be spoilers from this point forward.