Machine Learning Taps Power of Curiosity

(Image: Sarah Holmlund/Shutterstock)

(Image: Sarah Holmlund/Shutterstock)

As data analytics become increasingly driven by artificial intelligence (AI), researchers search for a way to drive machine learning. The key ingredient its future development may be a dash of curiosity.

There are all kinds of AI systems currently used by various businesses with different names like Alexa and Albert to personalize then. Perhaps it's time for an AI system named George after the monkey whose curiosity propels him into various adventures.

That would be an apt choice for the Intrinsic Curiosity Module (ICM) developed by a group of four researchers at University of California, Berkeley. The attempt to inject curiosity to achieve self-motivated advances in machine learning was the subject of their paper, Curiosity-driven Exploration by Self-supervised Prediction, that was just submitted to the 34th International Conference on Machine Learning (ICML 2017).

[Learn more about AI and Machine Learning. Sign up now for the next edition of the A2 Academy.]

Their premise is that external rewards for learning are of necessity limited and actually rather rare in real life. That doesn't mean that people stop exploring or seeking out answers even when there are no prizes for doing. They are motivated by their own human curiosity. Infusing that kind of motivation in a virtual agent gets it to test things out for itself even when not directed to do so. The test of the effect was done in monitoring how far it would proceed in two video games, VizDoom and Super Mario Bros. as you see in the demo video here:

As the paper explains, the utility of curiosity is twofold: one is that it motivates exploration of one's "environment in the quest for new knowledge (a desirable characteristic of exploratory behavior is that it should improve as the agent gains more knowledge)." Another is that it serves as "a mechanism for an agent to learn skills that might be helpful in future scenarios."

Capitalizing on the latter allowed the virtual agent to build on "knowledge gained from earlier experience" to successfully navigate through "new places much faster than" it would have done if "starting from scratch." The researchers demonstrated that effective transfer of knowledge in the agents' extrapolation of what they have learned from one level of the video game to another one even within "a completely new map with new textures. "

Though the agent trained in Super Mario World did not receive any incentive "for killing or dodging enemies or avoiding fatal events," it still came to learn to do these things and arrive at "over 30% of Level-1." The researchers surmise that the motivation for the agent is to do whatever it takes to stay alive in the game in order to be exposed to more of the game environment.

"In order to remain curious, it is in the agent's interest to learn how to kill and dodge enemies so that it can reach new parts of the game space," they say. In that way, "curiosity provides indirect supervision for learning interesting behaviors." There could be far-reaching ramification for such built-in motivation.

The researchers suggest that the course-plotting behaviors learned by the agent like that of learning to avoid "bumping into walls" in the VizDoom game could be adapted to "a navigation system." That would be a design that responds to the stimulus in the environment rather than "reward signals." Given that those occur very infrequently in the "real world," they believe that their "approach excels in this setting and converts unexpected interactions that affect the agent into intrinsic rewards."

In the interest of collaboration, which itself is based on appreciation for shared curiosity, the researchers said they will share their algorithm's code and what was involved in the tests online at no charge. So you can exercise your own human curiosity in learning more about the development of artificial curiosity. Perhaps in future, AI will also entail AC. The advantage of calling it George is that it prevents anyone from confusing it with an air conditioner.

Ariella Brown,

Ariella Brown is a social media consultant, editor, and freelance writer who frequently writes about the application of technology to business. She holds a PhD in English from the City University of New York. Her Twitter handle is @AriellaBrown.

Machine Learning Taps Power of Curiosity

Learning can be motivated by rewards, but mostly it is motivated by curiosity. New research examines whether machine learning can embed curiosity, too.

Why Machine Learning Can Improve Customer Service

Machine learning-powered chatbots offer companies the possibility of saving money while improving customer service.

Re: Curiosity
  • 6/22/2017 2:27:04 PM

@kq4ym You beiev human curiosity is really a pragmatic thing, something that seeks out ways of perpetutating the species? I don't think I'd agree. Human curiosity actually can place us in great peril at times, and it won't necessarily translate into anything practical. It is what drives pure science -- the kind of study that is not directed toward anything of direct use -- and exploration of places where we really cannot live like deep under the sea, the arctic poles, and apace. That doesn't mean that pure science and exploration don't lead to practical outcomes in the end, only that the motivation for them is something other than the survival instinct.

What they found is that the AI derived a preference for survival within the game in order to learn more about what it had to offer. 

Re: Curiosity
  • 6/16/2017 10:20:45 AM

I'm not so sure I'd agree with the premise noted. "Their premise is that external rewards for learning are of necessity limited and actually rather rare in real life." It would seem rather that humans if not all sentient creatures have an instictive necessity to be curious which results in learning what is useful or not to continue the species in the long run or to find food and shelter in the short run. Now, getting machines to learn likewise will be an interesting problem to consider especially since those machines are still millions of years behind us in neural networks.

Re: Curiosity
  • 6/8/2017 5:12:32 PM

@PC Quite true. Curiosity can get people into trouble. Packing into a mobile robot could yield some unexpected results, though, for now it seems to be just a way to keep machine learning motivated without external rewards.

  • 6/8/2017 4:04:44 PM

Curious George would serve as an excellent namesake for the artificial curiosity module.

Monkeys and children, not to equivocate the two, both impress me with their relentless curiosity. This is the reason that small children, at least, can never be left alone for long (maybe 30 seconds if you're within earshot).