A taxonomy and review of generalisation research in NLP

Find paper here

The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

@article{hupkes2023taxonomy,
    author = {
        Dieuwke Hupkes and
        Mario Giulianelli and
        Verna Dankers and
        Mikel Artetxe and
        Yanai Elazar and
        Tiago Pimentel and
        Christos Christodoulopoulos and
        Karim Lasri and
        Naomi Saphra and
        Arabella Sinclair and
        Dennis Ulmer and
        Florian Schottmann and
        Khuyagbaatar Batsuren and
        Kaiser Sun and
        Koustuv Sinha and
        Leila Khalatbari and
        Maria Ryskina and
        Rita Frieske and
        Ryan Cotterell and
        Zhijing Jin
    },
    article = {Nature Machine Intelligence},
    title = {A taxonomy and review of generalisation research in NLP},
    year = {2023},
    number = {10},
    doi = {10.1038/s42256-023-00729-y},
    url = {https://doi.org/10.1038/s42256-023-00729-y},
    pages = {1161--1174},
}