Deep Learning: Make Sense of Humongous Datasets

The human mind, on its own, cannot possibly fathom the gargantuan datasets such as those holding geophysical data for oil exploration. Machines augment human intelligence by doing the grunt work of data crunching and finding the patterns, which humans can use to tell a story from the data.

New machine learning techniques going by the moniker of deep learning, commonly referred to as neural network algorithms, find the clusters in the data and the links between them to make sense of the data. It is a lot like solving a crossword puzzle -- the answers are found by looking at the relationships in the clues.

There are two types of deep learning -- supervised learning and unsupervised learning. Supervised learning begins with an understanding of the categories that define the data while unsupervised learning does not presume any knowledge of the categories.

Supervised deep learning is analogous to searching for an undersea destination like an oil well with the knowledge of the coastline alone. It reads the relationships in the geophysical data in the layers underneath the seashore to reach, progressively, the oil well. Unsupervised learning first establishes whether a relationship exists between the contours of the coastline and the subterranean topography.

I spoke with Dr. Charles H. Martin, a long-time expert in machine learning and the founder of Calculation Consulting, about the prospects for enterprise applications of supervised and unsupervised deep learning. “Many in the business world recognize the vast potential of applications of deep learning and the technology has matured for widespread adoption,” he said. “The most hospitable culture for machine learning is scientific and open to recurring experimentation with ideas and evolving business models. The legacy enterprise fixation on engineering and static processes is a barrier to its progress,” he underscored.

Unstructured data abounds, and the familiar methods of analyzing it with categories and correlations do not necessarily exist. The size and variety of such databases can elude modeling. These unstructured databases have valuable information like social media conversations about brands, video from traffic cameras, sensor data of factory equipment, and trading data from exchanges that are akin to finding a needle in a haystack. Deep learning algorithms find the brand value from positive and negative remarks on social media, elusive fugitives in the video from traffic cameras, the failing equipment from the factory data, or the investment opportunity in the trading data.

“Unsupervised deep learning helps in detecting patterns and hypothesis formulation while supervised deep learning is for hypothesis testing and deeper exploration,” Martin said. “Unsupervised deep learning has proved to be useful for fraud detection and oil exploration -- anamolies in the data point to cybercrime and oil respectively,” he explained. “The prediction of corporate performance using granular data such as satellite imagery of traffic in the parking lots of retail companies is an example of second generation of supervised deep learning.”

Early detection of illnesses from medical imaging is one category of problems that deep learning is well suited to address. Citing the example of Chronic Obstructive Pulmonary Disease (COPD), Dave Sullivan, the CEO of Ersatz Labs, a cloud-based deep learning company based in San Francisco, told me, “The imaging data shows nodules and not all of them indicate COPD. It is hard for even a trained eye to tell one from another. Deep learning techniques evolve as they are calibrated and recalibrated (trained) on vast volumes of data gathered in the past, and they learn to distinguish with a high degree of accuracy for individual cases.”

Clarifai has democratized access to its deep learning with its API, which allows holders of data to analyze and benefit from the insights. I spoke with Matthew Zeiler, the CEO and Founder of Clarifai, to understand how its partners use the technology. One of them is France-based I-nside, a healthcare company, which uses smartphones to conduct routine examinations of the mouth, ear, and throat to generate data for diagnosis. “In developing countries where doctors are scarce, the analysis of the data points to therapies that are reliable,” Zeiler said. “In developed countries, the analysis of the data supports the judgment of doctors, and they have reported satisfactory results.”

Enterprise is not the only place where deep learning has found a home. Consumer applications like Google Now, Microsoft’s Cortana, and Assistant are available in the market. Folks are often anxious and distracted, at work or play, when they are unable to keep track of critical events that could affect them or their family. Home surveillance watches pets, the return of young children from school, elderly relatives falling, the arrival of critical packages and more. What matters is an alert on an unusual event. Camio uses the camera of a handheld phone or any other home device like a computer to capture video of happenings at home. If-This-Then-That (IFTTT) alerts are sent when something irregular happens. Deep Learning algorithms separate significant events from the routine.

Unstructured data presents unique problems with its size, variety, and apparent formlessness. Crucial insights lurk in that data. Deep learning helps to find the meaningful information in the clutter of the data.

Kishore Jethanandani, Market Researcher

Kishore Jethanandani's current professional focus is content development and communications of complex subject matter for clients in the technology and financial services industry in the San Francisco Bay and Boston area. His career has evolved from industrial economic research with an accent on policy reform to a business journalist while he was in India to marketing writing and research in the USA. A technology buff and a futurist, he has an instinct for spotting emerging technology and market trends and their likely impact on business strategy and competition.  His personal slogan is "A hedgehog who senses the future". My website can be found at

Mesh Networks Promise to Tap a Wealth of IoT Data

The Internet of Thing's last mile holds rich data seams that can be opened for data mining with mesh networks.

What You Need to Know About Prescriptive Analytics

Even though companies have just scratched the surface with adoption of prescriptive analytics, some are getting real-world benefits, based on better decision making.

Deep learning technique
  • 1/18/2016 12:20:07 PM

Kishore, can you tell me which deep learning technique you have been talking about: dropout, no-drop, dropconnect or gradient descent? Which technique would you consider the most effective in reducing health care costs?

Re: Better medicine
  • 11/29/2015 4:41:15 PM

I'm not confident that deep learning tech in healthcare will reduce costs. We have imaging technologies that make diagnosis more accurate, targeted nanoscale treatments with new drugs, doctors handing off routine care to nurse practitioners and physicians assistants, robotic surgery, and limits on hospital stays. Yet, costs continue to soar when each of those steps should have cut costs. What doctor or hospital is going to reduce their fees?

Re: Better medicine
  • 11/29/2015 3:50:42 PM

Predictions about human health will be accurate because these are chemical and biological processes.

I'm less confident about predicting the financial markets since these depend on human behavior. Am I the only one that finds it difficult to predict how people will act and react?

Re: Better medicine
  • 11/29/2015 12:46:42 PM

Deep learning should eventually prove to be real lifesavers for the health industry. And if the costs of doctors reading charts and numbers can be reduced, theoretically healthcare cost should be able to be reducted significantly. I wonder if deep learning of financial data may lead to some important understandings, or just demonstrate that perhaps financial markets are more random that we think?

Re: A Rethorical Look at the Potential of the Human Brain
  • 11/23/2015 2:46:21 PM

"...So I see humans and machines playing complementary roles."


@Kishore    No disagreement here.   I have just been noticing more and more that no one wants to think or learn anymore.   Just Google it, get an answer and not have enough intellectual background to even question the results.   It is as if we have all given up on our potential because of the almighty computer.


Re: A Rethorical Look at the Potential of the Human Brain
  • 11/18/2015 12:37:12 AM

@Louis Watson: There are pros and cons of the human brain and that of deep learning by machines. Humans are certainly more creative, analytical and intuitive while machines process much larger volumes of data at a higher speed. So I see humans and machines playing complementary roles.

Re: Let's automate this
  • 11/18/2015 12:33:48 AM

@Terry Sweeney: It is more like rushing towards automated analytics. Humans play a role later when you try to make sense of the patterns identified. For real-time analytics, the entire process is automated. The technology is quite advanced and is especially useful in medical diagnostics, speech recognition, etc

A Rethorical Look at the Potential of the Human Brain
  • 11/17/2015 11:13:44 PM

Interesting arguments for the wonders of computing.  This argument that the human brain ( even though we do not even understand it completely) is not capable of coming to the same conclusions as say, Watson is something I am not prepared to throw the towel in on. 

Sure only a small percentage ( Einstein and the rest) have even come close to harnessing the power of one's own intellect or brain for the sake of argument.

But do we know for a fact that the brain cannot process as effectively Watson ?   Because few could ever operate it (their brain) to it's fullest potential ?

There is no doubt I would lose to Watson rather quickly in chess for instance but this result does not inherently demenish the ability of the brain to function almost as efficiently as Watson itself.

Just because we have never seen this result (often) does not mean this possibity cannot exist. 

Let's automate this
  • 11/17/2015 10:42:40 PM

Are we tiptoeing up to automated analytics with deep un/supervised learning? This makes a huge amount of sense for big data volumes where human power can't touch its subtleties, nuances or important insights.

Predicitng ear infections
  • 11/17/2015 4:02:46 AM

The idea of predicting ear infection with a mobile device connected to a supercomputer such as Watson would be a great idea.  It would help prevent hearing problems by predicting early signs of ear infections when a child is very young.

Page 1 / 2   >   >>