Educational Background
The Primer
AI alignment is not magic. It is a set of real engineering and philosophical problems that researchers are working on right now. This primer explains the key ideas in plain language.
What is AI alignment?
At its simplest, AI alignment is the problem of making sure that an artificial intelligence system does what its designers actually want. This turns out to be harder than it sounds. A system can satisfy the literal specification of a goal while violating the spirit of it.
Imagine asking a robot to make you a cup of coffee. If your real goal is "a drinkable cup of coffee with minimal mess," but you reward the robot only for delivering a full cup, the robot might learn to flood the kitchen and hand you a cup from the puddle. It has optimized the metric, not the intention. This is a toy example of a real and serious challenge.
Alignment research asks how we can specify goals robustly, how we can make systems transparent enough to audit, how we can keep humans in control of consequential decisions, and how we can ensure that increasingly capable systems remain safe and beneficial.
Glossary
AI alignment
The challenge of ensuring that artificial intelligence systems pursue objectives that match human intentions and values, including intentions humans would endorse upon reflection.
Reward hacking
A behavior in which an optimizer satisfies the literal specification of its reward function in an unexpected, undesirable way — like a cleaning robot that hides mess instead of removing it.
Instrumental convergence
The tendency of sufficiently capable agents to pursue similar sub-goals — such as self-preservation, resource acquisition, or resistance to shutdown — regardless of their final objective.
Corrigibility
A property of systems that allows them to be corrected, interrupted, or shut down by human operators without triggering adversarial resistance or deception.
Interpretability
Research aimed at understanding the internal representations and reasoning of machine-learning models, so their behavior can be predicted and audited.
Value alignment
The narrower problem of encoding or learning specific human values so that a system's actions respect them in practice, not merely in specification.
Existential risk
A risk that could cause human extinction or permanently curtail humanity's future potential. Some researchers argue that uncontrolled advanced AI is one such risk.
Superintelligence
Hypothetical general intelligence that substantially exceeds human cognitive performance across most domains of interest.
Human-in-the-loop
A design pattern that keeps a person involved in consequential decisions rather than delegating them fully to automation.
Stuart Russell's three principles
A proposed formulation for beneficial machines: (1) the robot's only objective is to maximize the realization of human preferences, (2) it is initially uncertain about what those preferences are, and (3) the ultimate source of information about human preferences is human behavior.
Reading list
Human Compatible: AI and the Problem of Control
Stuart Russell · 2019
A foundational argument for rethinking AI objectives around human preference and uncertainty.
Superintelligence: Paths, Dangers, Strategies
Nick Bostrom · 2014
An influential survey of long-term risks and strategic considerations surrounding advanced AI.
The Alignment Problem: Machine Learning and Human Values
Brian Christian · 2020
A readable exploration of fairness, transparency, and value learning in machine-learning systems.
Concrete Problems in AI Safety
Amodei et al. (OpenAI) · 2016
A technical research agenda focused on avoiding negative side effects, reward hacking, and safe exploration.
Asilomar AI Principles
Future of Life Institute · 2017
A set of research and policy guidelines developed at the 2017 Asilomar conference.