AI and machine learning

What you can read here

1. 1 The perceptron: a probabilistic model for information storage and organization in the brain

   Frank Rosenblatt · 1 Jan 1958 · 1 korrent

   To understand organisms' recognition, generalization, recall and thinking we need how they sense, store and use information; the perceptron is a probabilistic model for brain storage and organization. Those three questions have to be answered before the higher abilities make sense. It proposes a probabilistic model for information storage and organization in the brain, set against the open problems of sensing, remembering, and how memory shapes recognition and behavior.

   neural networks

2. 2 Long Short-Term Memory

   Jürgen Schmidhuber, Sepp Hochreiter · 1 Nov 1997 · 2 korrents

   Networks that process sequences lose the teaching signal over long delays; LSTM keeps that signal constant inside special memory units with gates, learning across more than 1000 steps. It gets many more successful runs and learns much faster than older methods, and solves complex long-delay tasks previous algorithms never managed. It proposes a new learning method built on an earlier analysis of why that signal fades, tested against several older sequence-network algorithms on artificial data.

   neural networks

3. 3 Gradient-based learning applied to document recognition (with 3 co-authors)

   Yoshua Bengio · 1 Jan 1998 · 4 korrents

   Neural nets built for how 2D shapes vary beat every other method at reading handwritten digits, and whole multi-part document readers can be trained end to end the same way. Global training helps real handwriting systems, and a cheque reader built this way reaches record accuracy in commercial use on several million cheques a day. It compares methods on a standard digit task and proposes training multi-part systems as one, building on multilayer networks adjusted by sending errors backward through the layers.

   neural networks

4. 4 ImageNet: A large-scale hierarchical image database (with 5 co-authors)

   Fei-Fei Li · 1 Jun 2009 · 4 korrents

   A new image database organizes 3.2 million clean pictures under thousands of word categories from a big dictionary of meanings, far larger and more accurate than earlier image sets. That scale and structure give researchers a shared resource for object recognition, image sorting, and automatic grouping. It is a dataset release that describes gathering pictures with paid online workers and shows three simple uses.

   computer vision

5. 5 Generative Adversarial Networks (with 7 co-authors)

   Yoshua Bengio · 10 Jun 2014 · 2 korrents

   Two networks play a game where one creates fake samples to fool the other into thinking they are real data, until the creator matches the true data distribution. It lets generative models train with simple backpropagation without Markov chains or hard probability steps, and experiments show competitive samples on image sets. It proposes this adversarial training method for generative models, sidestepping the inference and sampling difficulties of earlier deep generative approaches.

   GANs

6. 6 Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)

   Diederik P. Kingma, Jimmy Ba · 22 Dec 2014 · 4 korrents

   A new algorithm called Adam finds good solutions to huge noisy problems by tracking running averages of gradients and their sizes so steps adjust themselves with almost no tuning. It suits large data or many parameters, noisy or sparse signals, uses little memory and compute, works well in tests against other methods, and comes with a matching best-known convergence guarantee. It proposes the method with links to earlier algorithms it builds on, theoretical analysis, empirical comparisons, and a simple variant.

   optimizers

7. 7 He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (arXiv)

   Jian Sun, Kaiming He · 10 Dec 2015 · 4 korrents

   Making neural nets much deeper for image recognition works better if layers learn only the difference from their inputs instead of full new mappings, letting accuracy rise with depth. These let them build networks eight times deeper than earlier ones yet simpler, cutting error to 3.57 percent on a major image test and improving object spotting by 28 percent. It proposes a training method for very deep nets and reports the contest-winning image results that follow when depth is no longer so hard to optimize.

   neural networkscomputer vision

8. 8 Attention is all you need

   Aidan N. Gomez, Ashish Vaswani, Illia Polosukhin, Jakob Uszkoreit, Llion Jones, Niki Parmar, Noam Shazeer, Łukasz Kaiser · 12 Jun 2017 · 3 korrents

   A new network for translating languages uses only attention to link words, dropping the usual step-by-step loops and filters entirely, and it works better while training faster. It scores 28.4 on English-to-German, beating prior bests including groups of models by over 2, and sets a single-model record of 41.8 on English-to-French after 3.5 days on eight GPUs. It proposes this architecture built solely on attention, against the dominant recurrent and convolutional sequence models.

   transformers

9. 9 Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)

   Jonathan Frankle, Michael Carbin · 9 Mar 2018 · 5 korrents

   Big networks started at random hide smaller ones inside that, kept at those starting weights, train alone to the same accuracy just as fast. Finding them means networks under 10-20% the size can train from scratch to match or beat the original. It reports experiments plus a prune-and-reset method supporting this against the usual trouble training sparse nets from the start.

   neural networkspruning

10. 10 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Kenton Lee, Kristina Toutanova, Ming-Wei Chang · 11 Oct 2018 · 4 korrents

    Train a language model to guess randomly hidden words using the full surrounding text so it sees both directions, then lightly adjust it for many different tasks. This lets one model handle question answering and other jobs with almost no custom design and raises scores a lot on eleven language tests, including 7.7 points to 80.5 on the main suite. It proposes a training method that builds on prior language models but pushes against their one-way limits.

    LLMsscaling laws

11. 11 On the Measure of Intelligence

    François Chollet · 5 Nov 2019 · 5 korrents

    Being smart is not scoring high on a fixed task after lots of built-in knowledge or training; it is how efficiently you acquire skills on new problems. This settles why current AI tests mislead us and opens fairer ways to compare systems to humans using better benchmarks. It argues a position against skill-only measures, defines intelligence as skill-acquisition efficiency, and presents the Abstraction and Reasoning Corpus as a test built on human-like starting knowledge.

    ARC-AGImeasuring intelligence

12. 12 Language models are few-shot learners (with 30 co-authors)

    Dario Amodei · 28 May 2020 · 3 korrents

    Larger language models can do new tasks after seeing just a few examples in plain text, with no extra training, sometimes matching systems that got thousands of examples. This reduces the need to collect huge labeled datasets for every new task. It reports results on a 175 billion parameter model tested this way, building on prior pre-training work that still required task-specific extra training.

    LLMs

13. 13 Highly accurate protein structure prediction with AlphaFold (with 33 co-authors)

    Demis Hassabis · 15 Jul 2021 · 4 korrents

    A computer program regularly predicts a protein's three-dimensional shape from its amino acid sequence alone at accuracy matching lab experiments, even with no similar structure known. It opens large-scale study of how proteins work since experiments have solved only around 100,000 while billions of sequences are known and each takes months to years. It reports results from a redesigned neural network incorporating physical and biological knowledge that greatly outperformed other methods in a blind test.

    protein structure prediction

14. 14 Training language models to follow instructions with human feedback (with 19 co-authors)

    Jan Leike · 4 Mar 2022 · 4 korrents

    Teaching language models via human good-answer examples and output rankings makes a small one preferred to a 100x larger plain model, with less lying and toxicity. This shows just making them bigger fails to make them do what users want and gives a workable fix that keeps most other skills intact. It reports experimental results on fine-tuning with human feedback for broad instructions, extending earlier techniques used mainly for summarization.

    LLMsAI alignment

15. 15 Managing extreme AI risks amid rapid progress (with 24 co-authors)

    Yoshua Bengio · 26 Oct 2023 · 4 korrents

    AI races toward goal-chasing systems that act alone, risking massive harms and irreversible loss of human control, while safety work and governance lag far behind. It demands one-third of AI budgets for safety research plus fast government rules that tighten automatically as capabilities grow. It is a consensus position paper outlining risks and a tech-plus-oversight plan, drawing lessons from other safety-critical technologies against today's weak response.

    AI alignment

16. 16 Assessing Adaptive World Models in Machines with Novel Games (with 13 co-authors)

    François Chollet · 17 Jul 2025 · 5 korrents

    People adapt fast by quickly building mental models of how new places work, yet AI checks only look at fixed knowledge from big training sets. This opens concrete tests of whether machines can figure out brand-new rules through limited play the way people do. It is a perspective proposing suites of games with genuine hidden novelty as benchmarks, drawing on cognitive science and pushing past static model evaluations.

    world models

17. 17 From Solvers to Research: Large Language Model-Driven Formal Mathematics at the Research Frontier (with 18 co-authors)

    Terence Tao · 8 Jul 2026 · 4 korrents

    AI systems that check math proofs with computers are great at fixed problems but still cannot discover new theorems or settle open questions the way research mathematicians do. Closing that gap would let them expand mathematical knowledge instead of just rediscovering known results or winning contests. It is a position paper that surveys the field and maps five concrete barriers blocking the shift from solvers to research agents.

    LLMsmathematics