Jean Kaddour

Publications

1. No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

   Kaddour, Jean, Key, Oscar, Nawrot, Piotr, Minervini, Pasquale, and Kusner, Matt J.

   In Advances in Neural Information Processing Systems 2023

   Abs PDF Code

   The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

2. Evaluating Self-Supervised Learning for Molecular Graph Embeddings

   Wang, Hanchen, Kaddour, Jean, Liu, Shengchao, Tang, Jian, Lasenby, Joan, and Liu, Qi

   In Advances in Neural Information Processing Systems 2023

   Abs PDF Code

   Graph Self-Supervised Learning (GSSL) provides a robust pathway for acquiring embeddings without expert labelling, a capability that carries profound implications for molecular graphs due to the staggering number of potential molecules and the high cost of obtaining labels. However, GSSL methods are designed not for optimisation within a specific domain but rather for transferability across a variety of downstream tasks. This broad applicability complicates their evaluation. Addressing this challenge, we present ’Molecular Graph Representation Evaluation’ (MOLGRAPHEVAL), generating detailed profiles of molecular graph embeddings with interpretable and diversified attributes. MOLGRAPHEVAL offers a suite of probing tasks grouped into three categories: (i) generic graph, (ii) molecular substructure, and (iii) embedding space properties. By leveraging MOLGRAPHEVAL to benchmark existing GSSL methods against both current downstream datasets and our suite of tasks, we uncover significant inconsistencies between inferences drawn solely from existing datasets and those derived from more nuanced probing. These findings suggest that current evaluation methodologies fail to capture the entirety of the landscape.

3. Challenges and Applications of Large Language Models

   Kaddour, Jean, Harris, Joshua, Mozes, Maximilian, Bradley, Herbie, Raileanu, Roberta, and McHardy, Robert

   2023

   Abs PDF

   Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identify the remaining challenges and already fruitful application areas. In this paper, we aim to establish a systematic set of open problems and application successes so that ML researchers can comprehend the field’s current state more quickly and become productive.

4. Spawrious: A benchmark for fine control of spurious correlation biases

   Lynch, Aengus, Dovonon, Gbètondji JS, Kaddour, Jean, and Silva, Ricardo

   arXiv preprint arXiv:2303.05470 2023

   PDF Website
5. The MiniPile Challenge for Data-Efficient Language Models

   Kaddour, Jean

   2023

   Abs PDF Website

   The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse datasets are often too large for academic budgets; hence, most research on Transformer architectures, training procedures, optimizers, etc. gets conducted on smaller, homogeneous datasets. To this end, we present The MiniPile Challenge, where one pre-trains a language model on a diverse text corpus containing at most 1M documents. MiniPile is a 6GB subset of the deduplicated 825GB The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using \k\-means, and (3) filter out low-quality clusters. To verify MiniPile’s suitability for language model pre-training, we use it to pre-train a BERT and T5 model, yielding a performance drop of only \1.9}%\/\2.5}% on the GLUE and SNI benchmarks compared to the original pre-trained checkpoints trained on \2.6\x/\745\x the amount of data. MiniPile is available at https://huggingface.co/datasets/JeanKaddour/minipile.

6. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging

   Kaddour, Jean

   In Has it Trained Yet? NeurIPS 2022 Workshop 2022

   PDF Code
7. Causal Machine Learning: A Survey and Open Problems

   Kaddour, Jean, Lynch, Aengus, Liu, Qi, Kusner, Matt J., and Silva, Ricardo

   2022

   PDF Website
8. When Do Flat Minima Optimizers Work?

   Kaddour, Jean, Liu, Linqing, Silva, Ricardo, and Kusner, Matt

   In Advances in Neural Information Processing Systems 2022

   PDF
9. DAG Learning on the Permutahedron

   Zantedeschi, Valentina, Franceschi, Luca, Kaddour, Jean, Kusner, Matt, and Niculae, Vlad

   In The Eleventh International Conference on Learning Representations 2023

   PDF
10. Causal Effect Inference for Structured Treatments

    Kaddour, Jean, Zhu, Yuchen, Liu, Qi, Kusner, Matt J., and Silva, Ricardo

    In NeurIPS 2021

    PDF Code Video Poster
11. Probabilistic Active Meta-Learning

    Kaddour, Jean, Sæmundsson, Steindór, and Deisenroth, Marc

    In NeurIPS 2020

    PDF Code Video Poster