Sayash Kapoor
PhD candidate in computer science at Princeton's Center for Information Technology Policy; researches the societal impact of AI and co-writes AI as Normal Technology with Arvind Narayanan.
Sayash Kapoor did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter All topics · Newest first
-
Our reading
Evaluating an AI agent requires scrutinizing its entire trajectory, not just its final outcome.
Their wordsThe evaluator must scrutinize the entire trajectory, not just the final outcome.
↗New paper: Emerging trends in AI agent evaluationnormaltech.ai 1st of 2 in this piece
-
Our reading
An AI agent's evaluation results at one level of scope are not guaranteed to hold at another.
Their wordsCrucially, the results of evaluation at one level of scope are not guaranteed to translate to another.
↗New paper: Emerging trends in AI agent evaluationnormaltech.ai 2nd of 2 in this piece
- 5 weeks earlier
-
Our reading
AI companies should be liable for what their agents do, and policymaking should make that clear.
Their wordsWe think AI companies should be liable for what their agents do, and this should be clarified through policymaking.
↗The AI-as-Normal-Technology view of loss-of-control incidentsnormaltech.ai 1st of 3 in this piece
-
Their wordsBut in our view, marginal investments in control are more likely to be effective compared to those in alignment.
↗The AI-as-Normal-Technology view of loss-of-control incidentsnormaltech.ai 2nd of 3 in this piece
-
Their wordsRather than behaving like startups, AI companies need to behave like more mature organizations.
↗The AI-as-Normal-Technology view of loss-of-control incidentsnormaltech.ai 3rd of 3 in this piece
- 6 weeks earlier
-
Their wordsOur results suggest that conducting open-ended research remains challenging for frontier AI agents.
↗AI agents can't yet do open-ended AI researchnormaltech.ai 1st of 2 in this piece
-
Their wordsOur view is that while faster progress is certainly possible on narrow tasks (such as improving efficiency), we don't think it will lead to broad RSI or explosive progress.
↗AI agents can't yet do open-ended AI researchnormaltech.ai 2nd of 2 in this piece
- 2 months earlier
-
Their wordsEven if there was no direct copying, writing an operating system might be relatively easy for agents because of patterns memorized in the training data, so this doesn't tell us much about agents' ability to create novel pieces of software.
↗Did Google’s AI agents really build an operating system for $916?normaltech.ai 1st of 3 in this piece
-
Their wordsGoogle's experiment does add to the mounting evidence that agents or agent teams can autonomously or near-autonomously work on certain kinds of tasks for very long periods of time, making progress without getting stuck or confused.
↗Did Google’s AI agents really build an operating system for $916?normaltech.ai 2nd of 3 in this piece
-
Their wordsbenchmark evaluation is effectively impossible for this kind of task for many reasons including cost.
↗Did Google’s AI agents really build an operating system for $916?normaltech.ai 3rd of 3 in this piece
- 1 day earlier
-
Their wordsNonproliferation is brittle because it relies on a single chokepoint. Resilience distributes defenses across society.
↗Do AI Risks Require Extraordinary Government Intervention?normaltech.ai 1st of 2 in this piece
-
Their wordsOpen-weight models and widespread API access from frontier labs mean that the gap between frontier and publicly available capabilities is at most a few months, not years.
↗Do AI Risks Require Extraordinary Government Intervention?normaltech.ai 2nd of 2 in this piece
- 5 weeks earlier
-
Their wordsApp store operators should prepare for and police spam submissions, as they might soon see thousands of applications submitted autonomously using agents.
↗Open-world evaluations for measuring frontier AI capabilitiesnormaltech.ai 1st of 3 in this piece
-
benchmarksreinforcement learning
Their wordsBut whatever is precise enough to benchmark is also precise enough to optimize for, and modern RL training runs increasingly resemble the shape of the benchmarks themselves.
↗Open-world evaluations for measuring frontier AI capabilitiesnormaltech.ai 2nd of 3 in this piece
-
Their wordsAverage performance is very different from upper-bound capability elicitation. Trying to run dozens (or 100s) of tasks in a benchmark suite is only necessary to measure average performance.
↗Open-world evaluations for measuring frontier AI capabilitiesnormaltech.ai 3rd of 3 in this piece
- 7 weeks earlier
-
Their wordsOur view is that reliability lags capability, and that reliability will remain a barrier to deployment unless researchers and developers focus effort on improving reliability as a separate dimension from accuracy.
↗New Paper: Towards a science of AI agent reliabilitynormaltech.ai 1st of 3 in this piece
-
Their wordsOur view is that for autonomous operation in high-stakes contexts, we need 3-5 "nines" of performance - 99.9% to 99.999% accuracy - in order for reliability to become a non-issue, and we don't think LLM-based agents are on track to reach such a threshold.
↗New Paper: Towards a science of AI agent reliabilitynormaltech.ai 2nd of 3 in this piece
-
Their wordswe expect each order of magnitude decrease in unreliability (1-reliability) to be as hard as the previous one.
↗New Paper: Towards a science of AI agent reliabilitynormaltech.ai 3rd of 3 in this piece
- 7 months earlier
-
Their wordsThey are simply trying to accelerate production, which is like adding lanes to a highway when the slowdown is actually caused by a toll booth.
-
Their wordsNo amount of findings or papers in older paradigms of geology would have led to the same progress that plate tectonics did.