Norman the psychopathic AI offers a warning on biased data
Norman psychopathic AI
Credit: MIT

Norman the psychopathic AI offers a warning on biased data

Many people are concerned about the potential rise of malignant AI, with UK newspapers, in particular, worried about the ‘Terminator’ scenario of machines that are hostile to humanity.

Researchers at MIT have decided to explore this concept by creating a psychopathic AI, named Norman – after Norman Bates in the Alfred Hitchcock movie, Psycho. Their aim isn’t to confirm the public’s worst fears by designing a hostile machine intelligence, but to demonstrate how and why a machine might become ‘evil’ in the first place.

Norman was designed to explore the enormous influence that training data has on machine learning algorithms, and the results are certainly instructive.

But first, what is the problem that researchers are trying to highlight?

Importing biases

Many people assume that artificial intelligence systems are somehow objective and devoid of the biases, beliefs, or prejudices that are common among human beings. In fact, the reverse is invariably the case, and the data that developers use to train machine learning algorithms can heavily influence their behaviour, and the outcomes that these systems produce.

Research has shown (see below) that unconscious bias can creep into training data, sometimes because systems are developed in teams that lack diversity or external inputs, and on other occasions simply because they are trained using data that contains historic biases that have never been recognised and addressed by the developers.

For example, if an AI is trained to give sentencing guidelines in the legal system, it will produce biased results if the training data contains long-term, systemic biases against minority groups. This isn’t a hypothetical scenario: the COMPAS AI system in the US was recently found to be biased against black Americans and other minorities, because decades of legal data contained institutional biases in sentencing.

In effect, those biases have become automated and given a veneer of neutrality: a dangerous set of circumstances in social terms.

All of these issues are explored in depth in this external report by Internet of Business editor, Chris Middleton. Among the many cases discussed in that article is the CIA’s recent building of an AI image recognition system to determine if people with tattoos are more likely to commit crimes. However, as the report explains, the implicit belief “people with tattoos commit crimes” is inherent in the programme and the training data, so there is a likelihood the system can only give officials the answers they want.

While developers and their machine learning models might be completely unbiased themselves, the point is that many AIs reach whatever conclusions are available to them from the data that humans have put into them. They are not in any meaningful sense ‘intelligent’.

Introducing Norman

The researchers used the Rorschach inkblot test to prove the point. Via Norman, the team demonstrated that the same machine learning algorithm will perceive completely different scenes in an image when trained using different source data.

Norman was designed to perform image captioning, creating textual descriptions of images. However, it was trained using a Reddit page that contained disturbing depictions and observations on the reality of death.

Norman Rorschach test
Norman sees: “man is shot dead in front of his screaming wife.”

The AI was then tested alongside another image-captioning neural network, this time trained on the Microsoft COCO dataset. Both were subjected to Rorschach inkblots – the psychological test created in 1921 and made famous by its use in the diagnosis of psychological disorders.

The results of the AI experiment were disturbing, if predictable. While the standard AI interpreted one image as containing “a group of birds sitting on top of a tree branch”, Norman concluded “a man is electrocuted”.

Similarly, what was a “a close up of a vase with flowers” to the other AI, was captioned “a man is shot dead in front of his screaming wife” by Norman.

Other interpretations included, “man gets pulled into dough machine” and “pregnant woman falls at construction story [sic].”

  • Norman isn’t the MIT team’s first foray into AI’s links with horror and other emotions. In 2016, researchers shared the Nightmare Machine – AI-generated horror imagery – and polled people around the world on their responses to AI’s ability to invoke emotions such as fear. A year later, the Shelley AI collaboratively wrote horror stories with humans before Deep Empathy explored the flip side of the emotional coin.

Internet of Business says

The implications of the ‘Norman’ research are valuable – and troubling – because they reveal that some AI systems may simply present us with the results that we, consciously or unconsciously, already want to see. In the same way that a Google image search, for example, will present whatever pictures internet users have tagged in a certain way – including tags that may be partial or biased.

This opens up the real possibility that we may begin to use AI to ‘prove’ things that we already believe to be the case. In such a world, confirmation bias could become endemic, but have a veneer of neutrality and evidenced fact.

While the MIT experiment takes this issue to an extreme, Norman serves to highlight that many industries may be too quick to take AI processes at face value, and may be importing a broad range of biases, misapprehensions, or beliefs into systems that, in theory, are designed to be neutral.

And who is to say which image set was the ‘correct’ one with which to train the system? This is a more interesting and complex question than it seems. For example, does Microsoft’s standard data set contain no biases? Is it weighted to include every society on Earth, or merely what a group of American researchers has deemed to be acceptable?

Any image set, no matter how large, must on some level contain editorial choices that represent a set of implicit beliefs.

The truth is that while they may be more efficient, productive, and profitable in some applications, AIs are largely as fallible as the data with which they are shaped – a problem worsened by the ‘black box’ nature and complexity of some neural networks, which combine to create a system that produces answers, but with little transparency or ‘auditability’.

In March, Internet of Business’ Joanna Goodman sat in on the all-party parliamentary group on AI (APPG) at the House of Lords, reporting on the need for education, empowerment, and excellence in relation to AI.

She heard that many AI algorithms deliver average outcomes, which may be suited to most applications. Yet, when the consequences are business-critical or life-changing, ‘average’ may be an inadequate, utilitarian result.

When it comes to prejudices built into training data, an output of average conclusions may simply serve to reinforce the status quo – as shown in our report on the prejudices of the black-box risk-scoring AIs found throughout our financial, insurance, and criminal justice systems.

However, the research presented in the MIT report also offers a potential solution – by allowing black-box AI users to retrain them with the actual outcomes, using a transparent student model to mimic a black-box risk score teacher.

Combining this approach with greater vigilance in training data can help make critical AI models more transparent, while retaining their performance accuracy.

Regardless, Norman is a vivid reminder of the need to address complacency about data bias when training AI systems that may have huge impacts on our lives.

Additional commentary and analysis: Chris Middleton.