On the Measure of Intelligence

François Chollet · 5 Nov 2019 · arxiv.org ↗

5 korrents from this paper

In plain words

Being smart is not scoring high on a fixed task after lots of built-in knowledge or training; it is how efficiently you acquire skills on new problems. This settles why current AI tests mislead us and opens fairer ways to compare systems to humans using better benchmarks. It argues a position against skill-only measures, defines intelligence as skill-acquisition efficiency, and presents the Abstraction and Reasoning Corpus as a test built on human-like starting knowledge.

Our summary of the paper, not the authors' words — written to be readable without the field's vocabulary, from the stored copy of the paper and nothing else. Drafted with xai:grok-4.5 and checked by a person. The authors' own sentences are the quotes below.

Near this, by wording

Papers whose claims are worded most like this one's, found by the same hourly pass that draws the map. It is a measure of LANGUAGE, not of agreement or of citation: two papers can be near each other here and flatly contradict one another.

François Chollet did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Progress toward more intelligent artificial systems requires defining and evaluating intelligence so systems can be compared with each other and with humans.

Their wordsTo make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans.
Our reading
Solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience.

Their wordsWe argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power.
Our reading
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.

Their wordsWe argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.
Our reading
Both the special-purpose-programs view and the blank-slate view of human intelligence are likely incorrect.

Their wordsToday, it is increasingly apparent that both of these views of the nature of human intelligence - either a collection of special-purpose programs or a general-purpose Tabula Rasa - are likely incorrect
Our reading
If intelligence is the process of acquiring skills, no single task demonstrates intelligence unless it is a meta-task of skill-acquisition across many tasks.

Their wordsIf intelligence lies in the process of acquiring skills, then there is no task X such that skill at X demonstrates intelligence, unless X is actually a meta-task involving skill-acquisition across a broad range of tasks.