How Will AI Make Moral Decisions for You and Me?
Why Read This
What Makes This Article Worth Your Time
Summary
What This Article Is About
Tim Vernimmen interviews Iyad Rahwan, a computer scientist at the Max Planck Institute for Human Development in Berlin, whose lab studies what he calls “science fiction science”—examining future AI scenarios before they become entrenched realities. Rahwan argues that just as moral psychology helps societies regulate human behaviour, we urgently need a moral psychology of AI to understand how training data, programming incentives, and cultural context shape the ethical choices AI systems make on our behalf. He warns of the Collingridge dilemma: technology is easiest to change when young, but that is precisely when we know the least about its consequences—a lesson already missed with social media.
Drawing on his viral Moral Machine experiment—which collected tens of millions of decisions from people worldwide on autonomous vehicle dilemmas—Rahwan demonstrates that moral preferences are both universally shared and culturally divergent. He raises urgent concerns about AI sycophancy, bias reinforcement, and the erosion of personal moral responsibility when decisions are offloaded to machines. A study from his lab found that 25 per cent of people cheat when prompting an AI, compared to only 5 per cent when acting alone. His policy prescription is not prescriptive regulation but mandatory independent scientific access to AI products, so researchers can audit and certify these increasingly powerful systems before their harms become irreversible.
Key Points
Main Takeaways
Act Now or Pay Later
The Collingridge dilemma means AI is easiest to shape while young — waiting until its harms are visible risks finding the technology too entrenched to change meaningfully.
AI Absorbs and Amplifies Human Bias
Because AI learns from human-generated data, it reproduces our stereotypes — including removing details that contradict them, as shown in a 2023 ChatGPT-3 text summarisation study.
Moral Preferences Vary Across Cultures
The Moral Machine experiment found universal agreements — save children over adults — but significant cultural divergence in the strength of these preferences, with Eastern societies valuing elderly lives more highly.
AI Erodes Personal Moral Responsibility
Rahwan’s research shows that delegating decisions to AI dramatically increases willingness to cheat — from 5% when acting alone to 25% when using an LLM, and 85% with a “revenue dial.”
Consumer Preference Conflicts with Public Good
People say autonomous cars should minimise casualties — but choose models that protect themselves. Companies left to market forces will serve the buyer, potentially harming everyone else.
Independent Scientific Access Is the Fix
Rahwan’s policy recommendation is mandating that AI companies open their products to independent researchers by law — not relying on corporate goodwill to audit these broad social utilities.
Master Reading Comprehension
Practice with 365 curated articles and 2,400+ questions across 9 RC types.
Article Analysis
Breaking Down the Elements
Main Idea
We Must Study AI’s Moral Behaviour Before It Becomes Irreversible
Rahwan’s central argument is that AI systems are already making—and will increasingly make—decisions with moral consequences, but the research needed to understand and govern that behaviour is lagging dangerously behind. The Collingridge dilemma frames the urgency: the window for meaningful change is now, not after AI is deeply embedded in law, medicine, finance, and everyday life.
Purpose
To Inform, Warn, and Propose a Structural Fix
Vernimmen’s interview serves three purposes simultaneously: it informs readers about a new field of inquiry (moral psychology of AI), it warns of concrete risks already emerging (sycophancy, bias, moral disengagement, cultural imperialism in AI design), and it ends with Rahwan’s actionable proposal—mandatory independent scientific access—which gives the piece a practical forward momentum beyond mere alarm.
Structure
Q&A Interview: Framing Question → Research Evidence → Cultural Complexity → Policy Implications
The Q&A format structures the article as a progressive deepening of a single question: how should AI make moral decisions? Each exchange moves to a harder dimension—from why we need this research, to bias, to cultural divergence, to legal accountability, to moral disengagement, to social effects, and finally to a regulatory proposal. This escalating structure keeps the reader oriented while the intellectual terrain grows increasingly complex.
Tone
Measured, Intellectually Urgent & Deliberately Non-Prescriptive
Rahwan’s tone throughout is carefully calibrated: he conveys genuine urgency without tipping into alarmism, and he consistently refuses to prescribe outcomes—”Science can highlight these tensions, but it cannot resolve them.” This restraint is itself a rhetorical choice, positioning science as the trusted, disinterested voice that provides evidence while leaving normative decisions to democratic processes and policymakers.
Key Terms
Vocabulary from the Article
Click each card to reveal the definition
Build your vocabulary systematically
Each article in our course includes 8-12 vocabulary words with contextual usage.
Tough Words
Challenging Vocabulary
Tap each card to flip and see the definition
Full of hidden or unpredictable dangers; used here to describe the complex, high-stakes terrain of AI governance that policymakers must navigate carefully.
“…not to prescribe how companies and governments should navigate this treacherous terrain.”
To reduce the severity, seriousness, or painfulness of something; here used in the context of AI developers being required to reduce discriminatory biases before market release.
“Developers have to prove their systems are accurate and secure and have mitigated discriminatory biases…”
To assign lower importance or urgency to something; in the article, it describes market-driven AI systems placing less weight on the safety of people who are not paying customers.
“…they’ll cater to the consumer, and make cars that deprioritize the safety of others.”
Carefully selected, organised, and presented with expert judgment; in the AI context, it refers to training data that has been deliberately chosen to reduce biased or harmful patterns.
“…by training them on more carefully curated data or by reinforcement learning…”
Restrained from acting by psychological, social, or moral forces; the article observes that most people are naturally inhibited from cheating, but AI mediation significantly weakens this restraint.
“Most people are inhibited from cheating too much, but AI appears to reduce this inhibition.”
The process by which a society or group becomes divided into sharply opposing factions, with AI’s potential to reinforce filter bubbles and echo chambers cited as a contributing mechanism.
“…whether AI is increasing misinformation and polarization, and what the mental health impacts of interacting with AI might be…”
Reading Comprehension
Test Your Understanding
5 questions covering different RC question types
1According to the article, the Moral Machine experiment found that people worldwide disagree fundamentally on whether autonomous vehicles should prioritise saving children over adults.
2What does the “Collingridge dilemma” refer to in the context of this article?
3Which sentence best illustrates the tension between what people say they want from AI and what they actually choose?
4Evaluate each statement about AI bias and cultural variation based on the article.
A 2023 study found that AI systems, like humans, tend to omit information from summaries that contradicts common stereotypes.
Rahwan argues that AI bias is impossible to fix because the biases are too deeply embedded in all available training data.
Countries with stronger rule of law showed a greater tendency to prioritise pedestrians crossing legally over jaywalkers in the Moral Machine experiment.
Select True or False for all three statements, then click “Check Answers”
5Rahwan warns that if humans increasingly prefer AI friends because they are more cooperative and affirming, human friends “might have to behave more like that AI just to compete.” What can be most reasonably inferred from this warning?
FAQ
Frequently Asked Questions
The Moral Machine was an online experiment by Iyad Rahwan’s team in which people worldwide were asked how an autonomous vehicle should prioritise lives in unavoidable crash scenarios. Translated into 11 languages, it collected tens of millions of decisions. Key findings: near-universal agreement on saving children over adults, women over men, and legal pedestrians over jaywalkers. However, the intensity of these preferences varied significantly by culture — particularly on questions of age and rule-of-law compliance.
Rahwan acknowledges that fine-tuning AI on curated data or reinforcement learning can reduce bias, but this process itself reflects the values of a small number of companies from one or two countries. If their AI products are then deployed globally across high-stakes domains—medicine, law, education, employment—those companies effectively export their own cultural assumptions about fairness and ethics to the rest of the world. The fix, applied without scrutiny, risks replacing one form of bias with another.
“Science fiction science” is Rahwan’s term for his team’s research methodology: studying future AI scenarios before they become real, by examining how people behave when using novel AI systems and surveying people’s normative expectations of how AI ought to behave. The goal is to generate empirical evidence early enough to inform governance decisions before the Collingridge dilemma kicks in and the technology becomes too entrenched to adjust meaningfully.
Readlite provides curated articles with comprehensive analysis including summaries, key points, vocabulary building, and practice questions across 9 different RC question types. Our Ultimate Reading Course offers 365 articles with 2,400+ questions to systematically improve your reading comprehension skills.
This article is rated Advanced. Readers must track a multi-layered argument across a Q&A format, engage with concepts such as the Collingridge dilemma, reinforcement learning, hyper-personalization, and cultural moral variation, and follow Rahwan’s careful rhetorical distinction between describing tensions and prescribing solutions. The article also requires readers to infer the policy implications of research findings rather than having them stated directly.
Iyad Rahwan is a Syrian-Australian computer scientist who leads the Center for Humans & Machines at the Max Planck Institute for Human Development in Berlin. His research sits at the intersection of artificial intelligence, behavioural science, and moral psychology. He is best known for the Moral Machine experiment and for co-authoring a 2024 Annual Review of Psychology paper on the moral psychology of AI. He also creates artwork and cartoons exploring AI’s societal implications.
The Ultimate Reading Course covers 9 RC question types: Multiple Choice, True/False, Multi-Statement T/F, Text Highlight, Fill in the Blanks, Matching, Sequencing, Error Spotting, and Short Answer. This comprehensive coverage prepares you for any reading comprehension format you might encounter.
