korrents
Nathan Lambert

What Nathan Lambert thinks about reinforcement learning

@nathan-lambert · 55 positions · 1 change of mind · On ppll ↗

Machine-learning researcher on open language models; writes the Interconnects newsletter and the RLHF Book, after leading post-training at Ai2.

Nathan Lambert did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter reinforcement learning · Newest first
Topic
All 55LLMs 30open source 11AI alignment 6America 6China 5reinforcement learning 5OpenAI 4AGI 3Anthropic 3
23 more topics
innovation 3AI and science 2infrastructure 2recursive self-improvement 2scaling laws 2semiconductors 2AI agents 1AI and jobs 1AI writing 1censorship 1creativity 1cybersecurity 1education 1energy 1measuring intelligence 1motivation 1neural networks 1politics 1quantum mechanics 1Silicon Valley 1startups 1taste 1writing 1
Sort
NewestOldest

Only Nathan Lambert on reinforcement learning. See everyone on reinforcement learning

4 dated positions, 2024 to 2025, in their own words. Our reading of what Nathan Lambert has said — not written or endorsed by them.

  1. Our readingRLHF is not a tax on capability: preference tuning also raises maths and code scores, which is why the labs keep reaching for it.

    Their wordsAnd the important thing to say is that no matter how you want the model to behave, these RLHF and preference-tuning techniques also improve performance. So, on things like math evals and code evals, there is something innate to these, what is called contrastive loss functions.

    ↗DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459youtube.com 22nd of 44 in this recording

  3. Our readingReasoning is not taught to a model by people: it emerges from reinforcement learning on questions with checkable answers, with no human preference data at all.

    Their wordsAnd these reasoning behaviors emerge naturally. So these things like, "Wait, let me see. Wait, let me check this. Oh, that might be a mistake." And they emerge from only having questions and answers.

    ↗DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459youtube.com 23rd of 44 in this recording

  5. Our readingHumans keep their place in AI as judges rather than authors, because telling which of two answers is better is far easier than writing a good one.

    Their wordsAnd humans are actually very good at reading or judging between two things versus... This goes back to the core of what RLHF and preference tuning is that it's hard to generate a good answer for a lot of problems, but it's easy to see which one is better.

    ↗DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459youtube.com 41st of 44 in this recording

  6. 2 months earlier

  7. Our reading · no longer heldOpenAI's o1 does an online search over reasoning steps at test time, unlike a traditional autoregressive language model.

    LLMsOpenAI

    No longer holdsThis is the argument that all of the “search” in OpenAI’s o1 (and some of the replications) is all within the reinforcement learning training behavior.

    SaidIn my first post on o1, I tried to take OpenAI’s corporate communications and the media of their employees at face value to figure out what is going on: What would it mean for OpenAI’s o1 to be using search?

    ↗OpenAI's o1 using "search" was a PSYOPinterconnects.ai