Pictionary-Playing AI Sketches the Future of Human-Machine Collaborations

What do the games of chess, Jeopardy!, Go, Texas Hold’em, and StarCraft have in common? In each of these competitive arenas, an AI has resoundingly beat the best human players in the world. These victories are astounding feats of artificial intelligence—yet they’ve become almost humdrum. Another day, another triumph over humans. 

At the Allen Institute for Artificial Intelligence (AI2), in Seattle, researchers set out to do something different. Their AllenAI collaborates with a human player in a Pictionary-style drawing and guessing game, which is won through human-AI cooperation. 

Want to see for yourself? Go play it. AI2 has just launched a public version of the game, a simplified version of Pictionary that it calls Iconary. The current version of AllenAI has limited abilities—but as it engages with a diverse set of players, with different aptitudes and game strategies, it will get better. 

However, the goal isn’t to build “the world’s best Pictionary player,” says Ali Farhadi, senior research manager of the computer vision group at AI2. Rather, Farhadi sees Pictionary as a way to cultivate skills that can be transferred out of the game realm. 

“To play Pictionary, the AI has to do some common sense reasoning, it has to know about abstraction, and it needs a little bit of theory of mind,” Farhadi tells IEEE Spectrum. (When you hypothesize about another person’s emotions or thoughts, that’s what psychologists call theory of mind.) “By learning to play Pictionary, the AI can acquire skills and knowledge that transfer to real world applications.”

The lessons learned may be applicable to any system that requires human-AI communication, says Farhadi. He imagines more productive interactions with voice assistants like Alexa, and with helper robots that adapt based on feedback.

The Allen Institute isn’t the only institution that’s investigating collaborative AI via gameplay. OpenAI, a San Francisco-based research organization, has a team of 5 AI agents that work together to play the video game Dota 2, matching their skills against a team of humans (the AIs took on human champions last summer, but didn’t win). And Spectrum recently covered a competition that requires AI agents to cooperate in the game of Minecraft. 

Those efforts require AIs to collaborate with each other, as a step toward human-AI collaboration. But the Iconary project jumps straight to that end goal.

In the classic game of Pictionary, one member of a team has to draw a sketch representing a certain word or phrase, and the other team members have to guess it. In Iconary, AllenAI can play the role of either drawer or guesser. When it’s the drawer, it pops up a series of icons, and the human tries to guess the phrase. If the human is stumped, the AI will elaborate on its image.

When AllenAI is guessing, the human player draws on a sketch pad, then selects the icon that best represents whatever she was trying to draw. By repeating this process and arranging the resulting icons on a board, she creates a little narrative to help AllenAI to guess the phrase. 

Currently, the game includes 75,000 phrases, which must be depicted via 12,000 icons. The researchers say they restricted the number of icons so both the AI and human players would have to be creative about combining them, using simple elements to build up to larger concepts. 

Throughout the course of a game, AllenAI adapts to the individual player’s needs, says Aniruddha Kembhavi, a research scientist at AI2 who co-led the Iconary project. This is where it displays a rudimentary theory of mind, he says. “It needs to put itself in the mind of its partner, and decide, ‘What do I need to draw to get this human to guess right?’”

For AllenAI’s training, it observed some 100,000 Iconary games played by workers on the crowd-labor platform Mechanical Turk, gradually learning successful game strategies. To accelerate the learning curve, it also played games of Iconary against itself, racing through games at a speed no human could match.

That type of self-play has been key to other AI gaming victories, such as DeepMind’s AlphaGo system that taught itself to play chess, Go, and Shogi with no instructions. But Kembhavi says his team couldn’t rely exclusively on self-play to train AllenAI: “It might do a great job of playing Pictionary with itself—but its drawings might not be understandable to humans,” he says. 

The AI2 team has yet to publish any papers on the project or its methods. Several AI researchers told IEEE Spectrum that they can’t comment on the work’s scientific significance without a better understanding of “what’s going on beneath the hood,” as Mark Riedl, an associate professor at Georgia Tech, puts it. Still, he says the Iconary project seems to be “a positive step.” Riedl, who directs the Entertainment Intelligence Lab, works on AI collaboration and creativity; he’s currently working on an AI that can play the role-playing game Dungeons & Dragons. 

Riedl argues that AI research needs to move away from games with a known set of rules and possible actions, which includes complex games like Go and StarCraft. He sees real potential for the advancement of artificial intelligence in games that are unconstrained and open, that require ingenuity and imagination. While Iconary is a greatly simplified version of Pictionary, “I think it has worthy goals,” he says. “A full version of Pictionary could be a very interesting research problem.” 

Even in the current version, Riedl says, he appreciates the emphasis on common sense reasoning. An AI system that can transfer its skills from a game to real-world applications will need a grounding of basic knowledge, he says: “How does the world work? How do social interactions work? What are the scripts we tend to follow?”

AI2’s Farhadi says that even as Iconary helps the AI learn about society, the human players will also learn about AI. The hoopla about AIs defeating human grandmasters in certain games has led to a misperception that AI will soon surpass humans in general intelligence—when in fact, he says, “it’s not as intelligent as a dog.” 

While Elon Musk and some other tech luminaries have raised the specter of a superintelligent AI that will wipe out humanity, or maybe just take all our jobs, Farhadi puts those ideas in the category of science fiction. “It’s far from reality, and it’s far from our vision,” he says. “We envision a world in which humans and AI are working together.”