measuring intelligence
What it would mean to measure intelligence in a machine, as opposed to measuring how well it does one task it was trained for.
FilterEveryone, all time
- FC
François Chollet quoted
Our reading · no longer heldThe LLM line of research will reach a capability plateau.
Recalls holdingIn the era of base LLM scaling (2022-2024), I believed the LLM line of research would reach a capability plateau (as later seen with base LLMs). In late 2024, after the o3 test-time compute demo, I changed my views: the new models were showing genuine fluid intelligence, and with this new line of work, the LLM line of research could achieve unbounded capability scaling. "There will be no wall."
SaidIn the era of base LLM scaling (2022-2024), I believed the LLM line of research would reach a capability plateau (as later seen with base LLMs). In late 2024, after the o3 test-time compute demo, I changed my views: the new models were showing genuine fluid intelligence, and with this new line of work, the LLM line of research could achieve unbounded capability scaling. "There will be no wall."
- 12 months earlier
- AG
Alexey Guzey quoted
Our reading · no longer heldWork on large language models is not progress toward artificial general intelligence, because the models do not understand what they read.
No longer holdsIt’s become very difficult for me to maintain the belief in the stupidity of ChatGPT when every time I laugh at it, it ends up ridiculing me 6 months later.
SaidFirst, I’m now convinced that ChatGPT understands what it reads.
- 6 months earlier
- NL
Nathan Lambert quoted
Their wordsI think my personal definition of AGI is much simpler. I think language models are a form of AGI and all of this super powerful stuff is a next step that's great if we get these tools. But a language model has so much value in so many domains that it's a general intelligence to me.
↗DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459youtube.com 8th of 44 in this recording
- 6 weeks earlier
- FC
François Chollet quoted
Their wordsThis "memorize, fetch, apply" paradigm can achieve arbitrary levels of skills at arbitrary tasks given appropriate training data, but it cannot adapt to novelty or pick up new skills on the fly (which is to say that there is no fluid intelligence at play here.)
↗OpenAI o3 Breakthrough High Score on ARC-AGI-Pubarcprize.org
- 4 months earlier
- AG
Alexey Guzey quoted
Their wordsThis means that working on neural networks is NOT getting us closer to AGI, except indirectly.
- 5 years earlier
- FC
François Chollet quoted
Their wordsWe argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power.
↗On the Measure of Intelligencearxiv.org 2nd of 5 in this piece
- FC
François Chollet quoted
Their wordsWe argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.
↗On the Measure of Intelligencearxiv.org 3rd of 5 in this piece
- FC
François Chollet quoted
Their wordsIf intelligence lies in the process of acquiring skills, then there is no task X such that skill at X demonstrates intelligence, unless X is actually a meta-task involving skill-acquisition across a broad range of tasks.
↗On the Measure of Intelligencearxiv.org 5th of 5 in this piece