How AI Machines Are Learning to Navigate in Minecraft Mazes
Behavioral psychologists have long used mazes to study memory and learning; their subjects, mostly rats and mice.
Now researchers are beginning to use the same approach to test an entirely new kind of subject—the latest breed of artificial intelligence machine. They have started by putting these machines through their paces in mazes created in the online world of Minecraft.
Mazes have a long history in behavioral psychology. At the beginning of the 20th century, scientists became interested in the ability of rats and mice to learn and remember. In particular, they began to study learning mechanisms such as reinforcement learning.
The maze became the standard workhorse for this kind of work. Researchers would devise a complex labyrinth, place some kind of reward at the center, and then set a rat loose inside and see how quickly it solved the puzzle.
Psychologists quickly discovered that rats learned rapidly and could find their way even with various sensory impairments such as being blinded, deafened, or having their whiskers plucked.
But the complexity of early mazes meant that experiments were hard to compare. So eventually psychologists settled on simple mazes in the shape of Ts or Ys, for example, that could easily be reproduced in any lab.
That helped show how rats learn, that genes can determine how quickly rats solve puzzles, and so on. In recent years, computer scientists have even developed virtual reality mazes in which the rats are held stationary and forced to look at a screen while standing on top of a kind of trackball that moves as they walk or run. In this way, the rat advances through the virtual maze.
Now Junhyuk Oh and pals at the University of Michigan have begun experimenting with an entirely new kind of maze to test the cognitive skills of an entirely new kind of being. The new mazes are constructed in Minecraft, a 3-D world in which players use textured cubes to build almost anything. Creating a simple maze is trivial here.
But the beings Oh and co are testing are even more exotic—they are artificial intelligence machines. However, while these machines learn easily in ideal environments, they have difficulty in real world situations where objects can be partially obscured, where vision and movement have to be carefully coordinated to succeed and the resulting reward often delayed.
An important question is what kinds of AI systems are best at this. But studying how AI systems cope is hard because tricky environments are difficult to reproduce. That’s where Minecraft comes in.
Oh and co have created a set of mazes in which they set their AI algorithms increasingly complex tasks. For example, one task might be to find the red cube in a maze, the next task to find a red block if the first block it sees is yellow but otherwise to look for a blue block, and so on (see video).
The maze ensures that there is not always a clean line of site to the blocks and that the algorithm must coördinate its movement and vision to explore. The team can also give different kinds of rewards for successfully completing the task. Crucially, the same task with the same level of difficulty can be set over and over again.
The results are the first systematic exploration of this kind of AI cognitive ability. The team says the best performing AI system uses deep reinforcement learning enhanced with additional memory. These machines retrieve relevant memories based on the context in which they were stored and in which the device finds itself. That’s different from many existing memory systems that do not rely on context for memory retrieval.
“Our main empirical result is that context dependent memory retrieval can more effectively solve our set of tasks,” they say.
That’s interesting work that paves the way for much future work. Minecraft mazes are easy to reproduce allowing anybody with an AI system to compare their performance to these and other results.
It also allows a straightforward expansion of the research. “In future work, we intend to take advantage of the flexibility of the Minecraft domain to construct even more challenging cognitive tasks to further evaluate our architectures,” say Oh and co.
And that raises an interesting prospect. Perhaps the first place we’ll see AIs performing advanced tasks will be in virtual reality settings like Minecraft. This also provides a benign environment in which to explore some of the ethical issues that AI research raises.
So next time you, or your kids, are immersed in a Minecraft world, take a careful look at the other players. It’s just possible that they may not be all that they seem.
Ref: arxiv.org/abs/1605.09128: Control of Memory, Active Perception, and Action in Minecraft