Machine Learning Taps Power of Curiosity


(Image: Sarah Holmlund/Shutterstock)

(Image: Sarah Holmlund/Shutterstock)

As data analytics become increasingly driven by artificial intelligence (AI), researchers search for a way to drive machine learning. The key ingredient its future development may be a dash of curiosity.

There are all kinds of AI systems currently used by various businesses with different names like Alexa and Albert to personalize then. Perhaps it's time for an AI system named George after the monkey whose curiosity propels him into various adventures.

That would be an apt choice for the Intrinsic Curiosity Module (ICM) developed by a group of four researchers at University of California, Berkeley. The attempt to inject curiosity to achieve self-motivated advances in machine learning was the subject of their paper, Curiosity-driven Exploration by Self-supervised Prediction, that was just submitted to the 34th International Conference on Machine Learning (ICML 2017).

[Learn more about AI and Machine Learning. Sign up now for the next edition of the A2 Academy.]

Their premise is that external rewards for learning are of necessity limited and actually rather rare in real life. That doesn't mean that people stop exploring or seeking out answers even when there are no prizes for doing. They are motivated by their own human curiosity. Infusing that kind of motivation in a virtual agent gets it to test things out for itself even when not directed to do so. The test of the effect was done in monitoring how far it would proceed in two video games, VizDoom and Super Mario Bros. as you see in the demo video here:

As the paper explains, the utility of curiosity is twofold: one is that it motivates exploration of one's "environment in the quest for new knowledge (a desirable characteristic of exploratory behavior is that it should improve as the agent gains more knowledge)." Another is that it serves as "a mechanism for an agent to learn skills that might be helpful in future scenarios."

Capitalizing on the latter allowed the virtual agent to build on "knowledge gained from earlier experience" to successfully navigate through "new places much faster than" it would have done if "starting from scratch." The researchers demonstrated that effective transfer of knowledge in the agents' extrapolation of what they have learned from one level of the video game to another one even within "a completely new map with new textures. "

Though the agent trained in Super Mario World did not receive any incentive "for killing or dodging enemies or avoiding fatal events," it still came to learn to do these things and arrive at "over 30% of Level-1." The researchers surmise that the motivation for the agent is to do whatever it takes to stay alive in the game in order to be exposed to more of the game environment.

"In order to remain curious, it is in the agent's interest to learn how to kill and dodge enemies so that it can reach new parts of the game space," they say. In that way, "curiosity provides indirect supervision for learning interesting behaviors." There could be far-reaching ramification for such built-in motivation.

The researchers suggest that the course-plotting behaviors learned by the agent like that of learning to avoid "bumping into walls" in the VizDoom game could be adapted to "a navigation system." That would be a design that responds to the stimulus in the environment rather than "reward signals." Given that those occur very infrequently in the "real world," they believe that their "approach excels in this setting and converts unexpected interactions that affect the agent into intrinsic rewards."

In the interest of collaboration, which itself is based on appreciation for shared curiosity, the researchers said they will share their algorithm's code and what was involved in the tests online at no charge. So you can exercise your own human curiosity in learning more about the development of artificial curiosity. Perhaps in future, AI will also entail AC. The advantage of calling it George is that it prevents anyone from confusing it with an air conditioner.

Ariella Brown,

Ariella Brown is a social media consultant, editor, and freelance writer who frequently writes about the application of technology to business. She holds a PhD in English from the City University of New York. Her Twitter handle is @AriellaBrown.

Analytics and School Attendance: A Laundry Story

Appliance maker Whirlpool suspected there was a correlation between student attendance and access to clean clothes. Here's the story of how the company placed washers and dryers in schools and tracked the difference it made in student performance.

How AI Can Moderate Comments, Eliminate Trolls

The New York Times has only allowed comments to be posted on about 10% of its stories because moderating such comments is a slow process. Now the media company is using artificial intelligence to speed up the process.


Re: Curiosity
  • 7/3/2017 6:18:44 PM
NO RATINGS

@Lyndon_Henry My pleasure. I enjoy getting the perspective from the people on the board here. Your comments are always insightful. 

Re: Curiosity
  • 7/3/2017 5:26:31 PM
NO RATINGS

..

Ariella writes

 ... agreed that the programming does appear to be set for trail-and-error. The researchers attribute the agent's motivation to survive the games to curiosity because they claim there is no other motivation for it to keep going beyond discovering new parts of the game. I suppose they equate that with human curiosity to explore a new terrain, though it is possible to argue that it is qualitatively not the same.

Unanswered questions ... Sounds to me like it would make for a very interesting A2 radio discussion to invite one of these researchers as a guest ...

In any case, thanks to Ariella for finding this fascinating research experiment and writing it up to intrigue us ...

..

Re: Curiosity
  • 7/3/2017 11:14:05 AM
NO RATINGS

@Lyndon_Henry agreed that the programming does appear to be set for trail-and-error. The researchers attribute the agent's motivation to survive the games to curiosity because they claim there is no other motivation for it to keep going beyond discovering new parts of the game. I suppose they equate that with human curiosity to explore a new terrain, though it is possible to argue that it is qualitatively not the same.

Re: Curiosity
  • 7/3/2017 11:11:39 AM
NO RATINGS

@rbaz There are some nice observations on the child's questioning nature in James E Ryan's book Wait, What? What's also interesting about his observation about his persistence in questioning even into adulthood, making it a part of his career. Interestingly, it turns out that he was adopted, which may be a factor in some of the differences between himself and the parents he grew up with.

Re: Curiosity
  • 7/1/2017 10:00:36 AM
NO RATINGS

..

Ariella writes "What they found is that the AI derived a preference for survival within the game in order to learn more about what it had to offer."

I'm somewhat puzzled as to how the performance of this model (operational "agent") is considered an example of curiosity.

In their paper the authors write

This work belongs to the broad category of methods that generate an intrinsic reward signal based on how hard it is for the agent to predict the consequences of its own actions, i.e. predict the next state given the current state and the executed action. However, we manage to escape most pitfalls of previous prediction approaches with the following key insight: we only predict those changes in the environment that could possibly be due to the actions of our agent or affect the agent, and ignore the rest.

The "agent" seemed to exhibit a persistent objective of achieving the next "state" or level of what it was set to achieve (e.g., game). Its algorithms enabled it to compare what it was predicting in order to proceed with the consequences of what it was doing. This sounds like a kind of trial-and-error process, which is probably what enables a Roomba vacuum cleaner to bump itself around the furniture on the floor of a room.

The "agent" didn't seem to exhibit "curiosity" for solving anything of its own selection, but rather was set in virtual environments (games) by the researchers, where it set about trying to figure its way to each subsequently higher "state" or "level" in the game.

It apparently would get feedback when the "state" it achieved conformed with the state it had predicted. It didn't need some extrinsic (external) signal to confirm achievement of a goal, so this "signal" was intrinsically generated. (Although how it was able to detect achievement of the next subsequent state doesn't seem to be explained ...) 

However, I still fail to see this as an example of curiosity in action ...

..

Re: Curiosity
  • 6/30/2017 5:52:27 PM
NO RATINGS

'Curiosity in humans varies greatly and has a great impact on how they interact with the world. ' Seth, agreed that the level of curiosity varies in humans, but not initially. Our experiences leads us to become more curious or discourages us. Early childhood interaction like the kid that ask alot of questions, only to be yelled at or disregarded, will not be as curious as the one that finds positive engagement. Most of our difference is experience based versus genetics.

Re: Curiosity
  • 6/28/2017 3:21:26 PM
NO RATINGS

I think if you can program for curiosity you're getting very close to programing for personality and consciousness.   Curiosity in humans varies greatly and has a great impact on how they interact with the world.  

Re: Curiosity
  • 6/22/2017 2:27:04 PM
NO RATINGS

@kq4ym You beiev human curiosity is really a pragmatic thing, something that seeks out ways of perpetutating the species? I don't think I'd agree. Human curiosity actually can place us in great peril at times, and it won't necessarily translate into anything practical. It is what drives pure science -- the kind of study that is not directed toward anything of direct use -- and exploration of places where we really cannot live like deep under the sea, the arctic poles, and apace. That doesn't mean that pure science and exploration don't lead to practical outcomes in the end, only that the motivation for them is something other than the survival instinct.

What they found is that the AI derived a preference for survival within the game in order to learn more about what it had to offer. 

Re: Curiosity
  • 6/16/2017 10:20:45 AM
NO RATINGS

I'm not so sure I'd agree with the premise noted. "Their premise is that external rewards for learning are of necessity limited and actually rather rare in real life." It would seem rather that humans if not all sentient creatures have an instictive necessity to be curious which results in learning what is useful or not to continue the species in the long run or to find food and shelter in the short run. Now, getting machines to learn likewise will be an interesting problem to consider especially since those machines are still millions of years behind us in neural networks.

Re: Curiosity
  • 6/8/2017 5:12:32 PM
NO RATINGS

@PC Quite true. Curiosity can get people into trouble. Packing into a mobile robot could yield some unexpected results, though, for now it seems to be just a way to keep machine learning motivated without external rewards.

Page 1 / 2   >   >>
INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +