"Algorithms and AI are not objective," she said. "They're opinions embedded in code." O'Neil's talk, which took the title from her book, Weapons of Math Destruction, was about how even with good intentions, data scientists can create toxic algorithms that end up doing harm instead of good. That can mean failure in trying to solve a problem, like keeping good teachers or hiring fairly.
When it comes to how we create algorithms, said O'Neil, "we haven't established standards on what is good enough."
One way to correct this issue is by examining the laws and regulations that govern each industry, O'Neil says.
"It's one thing to build an algorithm that you find useful, like how much to exercise, but if you are building an algorithm that's used widely for important decisions that are already regulated, then you have to ask the question: What is the industry set standard for this and how do I make sure I'm meeting that standard?"
If this issue is ignored, O'Neil says that we can expect to see the inequality gap widen, and our democracy dissolve.