- by Lyndon_Henry, Blogger
- 7/3/2017 5:26:31 PM
... agreed that the programming does appear to be set for trail-and-error. The researchers attribute the agent's motivation to survive the games to curiosity because they claim there is no other motivation for it to keep going beyond discovering new parts of the game. I suppose they equate that with human curiosity to explore a new terrain, though it is possible to argue that it is qualitatively not the same.
Unanswered questions ... Sounds to me like it would make for a very interesting A2 radio discussion to invite one of these researchers as a guest ...
In any case, thanks to Ariella for finding this fascinating research experiment and writing it up to intrigue us ...
- 7/3/2017 11:14:05 AM
@Lyndon_Henry agreed that the programming does appear to be set for trail-and-error. The researchers attribute the agent's motivation to survive the games to curiosity because they claim there is no other motivation for it to keep going beyond discovering new parts of the game. I suppose they equate that with human curiosity to explore a new terrain, though it is possible to argue that it is qualitatively not the same.
- 7/3/2017 11:11:39 AM
@rbaz There are some nice observations on the child's questioning nature in James E Ryan's book Wait, What? What's also interesting about his observation about his persistence in questioning even into adulthood, making it a part of his career. Interestingly, it turns out that he was adopted, which may be a factor in some of the differences between himself and the parents he grew up with.
- by Lyndon_Henry, Blogger
- 7/1/2017 10:00:36 AM
Ariella writes "What they found is that the AI derived a preference for survival within the game in order to learn more about what it had to offer."
I'm somewhat puzzled as to how the performance of this model (operational "agent") is considered an example of curiosity.
In their paper the authors write
This work belongs to the broad category of methods that generate an intrinsic reward signal based on how hard it is for the agent to predict the consequences of its own actions, i.e. predict the next state given the current state and the executed action. However, we manage to escape most pitfalls of previous prediction approaches with the following key insight: we only predict those changes in the environment that could possibly be due to the actions of our agent or affect the agent, and ignore the rest.
The "agent" seemed to exhibit a persistent objective of achieving the next "state" or level of what it was set to achieve (e.g., game). Its algorithms enabled it to compare what it was predicting in order to proceed with the consequences of what it was doing. This sounds like a kind of trial-and-error process, which is probably what enables a Roomba vacuum cleaner to bump itself around the furniture on the floor of a room.
The "agent" didn't seem to exhibit "curiosity" for solving anything of its own selection, but rather was set in virtual environments (games) by the researchers, where it set about trying to figure its way to each subsequently higher "state" or "level" in the game.
It apparently would get feedback when the "state" it achieved conformed with the state it had predicted. It didn't need some extrinsic (external) signal to confirm achievement of a goal, so this "signal" was intrinsically generated. (Although how it was able to detect achievement of the next subsequent state doesn't seem to be explained ...)
However, I still fail to see this as an example of curiosity in action ...
- by rbaz, Data Doctor
- 6/30/2017 5:52:27 PM
'Curiosity in humans varies greatly and has a great impact on how they interact with the world. ' Seth, agreed that the level of curiosity varies in humans, but not initially. Our experiences leads us to become more curious or discourages us. Early childhood interaction like the kid that ask alot of questions, only to be yelled at or disregarded, will not be as curious as the one that finds positive engagement. Most of our difference is experience based versus genetics.
- by SethBreedlove, Data Doctor
- 6/28/2017 3:21:26 PM
I think if you can program for curiosity you're getting very close to programing for personality and consciousness. Curiosity in humans varies greatly and has a great impact on how they interact with the world.
- 6/22/2017 2:27:04 PM
@kq4ym You beiev human curiosity is really a pragmatic thing, something that seeks out ways of perpetutating the species? I don't think I'd agree. Human curiosity actually can place us in great peril at times, and it won't necessarily translate into anything practical. It is what drives pure science -- the kind of study that is not directed toward anything of direct use -- and exploration of places where we really cannot live like deep under the sea, the arctic poles, and apace. That doesn't mean that pure science and exploration don't lead to practical outcomes in the end, only that the motivation for them is something other than the survival instinct.
What they found is that the AI derived a preference for survival within the game in order to learn more about what it had to offer.
- by kq4ym, Data Doctor
- 6/16/2017 10:20:45 AM
I'm not so sure I'd agree with the premise noted. "Their premise is that external rewards for learning are of necessity limited and actually rather rare in real life." It would seem rather that humans if not all sentient creatures have an instictive necessity to be curious which results in learning what is useful or not to continue the species in the long run or to find food and shelter in the short run. Now, getting machines to learn likewise will be an interesting problem to consider especially since those machines are still millions of years behind us in neural networks.
- 6/8/2017 5:12:32 PM
@PC Quite true. Curiosity can get people into trouble. Packing into a mobile robot could yield some unexpected results, though, for now it seems to be just a way to keep machine learning motivated without external rewards.