The Expressive Power of Large Language Models
Large language models process vast sequences of input tokens by alternating between classical multi-layer perceptron layers and self-attention mechanisms. While the approximation capabilities of perceptrons are relatively well understood, those of attention mechanisms remain less explored. In this talk, I will compare the proof techniques and approximation results associated with these two types of layers, emphasizing key open questions that connect large language models with approximation theory in infinite-dimensional spaces representing input token distributions.
Contatto:
marco.verani@polimi.it
Gabriel Peyré is a CNRS senior researcher and professor at the École Normale Supérieure, Paris. He works at the interface between applied mathematics and machine learning, developing theoretical and numerical methods in Optimal Transport to analyze the training of deep neural networks and to address applications in single-cell genomics. He is also engaged in promoting reproducible research and coding education, particularly through the platform numerical-tours.com. He obtained 3 ERC grants (starting in 2010, consolidator in 2017 and advanced in 2024), the Blaise Pascal prize from the French Academy of Sciences in 2017, the Magenes Prize from the Italian Mathematical Union in 2019, and the silver medal from CNRS in 2021. He is an invited speaker at the European Congress for Mathematics in 2020 and the International Congress of Mathematicians in 2026. He is the deputy director of the Prairie Institute for Artificial Intelligence, the director of the ENS IA Lab, and of the ELLIS Paris Unit.