Résumé IA · arXiv: 2005.00052v1 · cs.CL
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer
Posez des questions sur cet article à notre assistant IA
Comment discuter avec cet article ? Guide illustré
Vous pouvez aussi discutez avec plusieurs papiers à la fois ici.
Gardez cette conversation — un compte gratuit conserve vos questions et réponses, vos collections d’articles et vos alertes.
Continuer avec Google Créer un compte gratuitOutils pour cet article
Outils avancés Premium
Pour suivre un domaine de près : alertes de citations, diapositives de journal club, comparaison des versions et envoi vers Zotero. Découvrir Premium
Parmi les 31 articles autour de celui-ci, le nom qui revient le plus : Alan Ansell (2 articles)
7 autres personnes en signent plusieurs. La carte complète ajoute leurs institutions et leurs approches, d’après les articles des dernières années.
Les articles autour de celui-ci
Les voir sur une carteClassés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual TransferAlan Ansell, E. Ponti, Jonas Pfeiffer et al. · 2021 · Conference on Empirical Methods in Natural Language Processing · cité 103 fois · Pas encore résumé
Résumé scientifique
Adapter modules have emerged as a general parameter-efficient means to specialize a pre-trained encoder to new domains. Massively multilingual transformers (MMTs) have particularly benefited from additional training of language-specific adapters. However, this approach is not viable for the vast majority of languages, due to limitations in their corpus size or compute budgets. In this work, we propose MAD-G ( M ultilingual AD apter G eneration), which contextually generates language adapters from language representations based on typological features. In contrast to prior work, our time-and space-efficient MAD-G approach enables (1) sharing of linguistic knowledge across languages and (2) zero-s…
-
Cross-Lingual Transfer with Target Language-Ready Task AdaptersMarinela Parovi'c, Alan Ansell, Ivan Vulic et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cité 19 fois · Pas encore résumé
Résumé scientifique
Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer. The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language. Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity by creating"bilingual"adapters specific to the source-target language pair. In this work, we aim to take the best of both worlds by (i) fine-tuning task adapters adapted to the target language(s) (so-called"target language-ready"(TLR) adapters) to maintain high transfer performance, but (ii) without sac…
-
X-METRA-ADA: Cross-lingual Meta-Transfer learning Adaptation to Natural Language Understanding and Question AnsweringMeryem M'hamdi, Doo Soon Kim, Franck Dernoncourt et al. · 2021 · North American Chapter of the Association for Computational Linguistics · cité 21 fois · Pas encore résumé
Résumé scientifique
Multilingual models, such as M-BERT and XLM-R, have gained increasing popularity, due to their zero-shot cross-lingual transfer learning capabilities. However, their generalization ability is still inconsistent for typologically diverse languages and across different benchmarks. Recently, meta-learning has garnered attention as a promising technique for enhancing transfer learning under low-resource scenarios: particularly for cross-lingual transfer in Natural Language Understanding (NLU). In this work, we propose X-METRA-ADA, a cross-lingual MEta-TRAnsfer learning ADAptation approach for NLU. Our approach adapts MAML, an optimization-based meta-learning approach, to learn to adapt to new la…
-
Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual TransferA. Ustun, Arianna Bisazza, G. Bouma et al. · 2022 · Conference on Empirical Methods in Natural Language Processing · cité 40 fois · Pas encore résumé
Résumé scientifique
Massively multilingual models are promising for transfer learning across tasks and languages. However, existing methods are unable to fully leverage training data when it is available in different task-language combinations. To exploit such heterogeneous supervision, we propose Hyper-X, a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation. It generates weights for adapter modules conditioned on both tasks and language embeddings. By learning to combine task and language-specific knowledge, our model enables zero-shot transfer for unseen languages and task-language combinations. Our experiments on a diverse set of languages demonstrate that Hyper-X…
-
WAD-X: Improving Zero-shot Cross-lingual Transfer via Adapter-based Word AlignmentAhtamjan Ahmat, Ya-Ting Yang, Bo Ma et al. · 2023 · ACM Trans. Asian Low Resour. Lang. Inf. Process. · cité 7 fois · Pas encore résumé
Résumé scientifique
Multilingual pre-trained language models (mPLMs) have achieved remarkable performance on zero-shot cross-lingual transfer learning. However, most mPLMs implicitly encourage cross-lingual alignment in pre-training stage, making it hard to capture accurate word alignment across languages. In this paper, we propose Word-align ADapters for Cross-lingual transfer (WAD-X) to explicitly align word representations of mPLMs via language-specific subspace. Taking a mPLM as the backbone model, WAD-X constructs subspace for each source-target language pair via adapters. The adapters use statistical alignment as the prior knowledge to guide word-level aligning in the corresponding bilingual semantic subs…
-
Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal et al. · 2019 · Computation and Language · cité 9 795 fois · Résumé IA disponible
Résumé scientifique
This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We also present a de…
-
FAD-X: Fusing Adapters for Cross-lingual Transfer to Low-Resource LanguagesJaeseong Lee, Seung-won Hwang, Taesup Kim · 2022 · AACL · cité 11 fois · Pas encore résumé
Résumé scientifique
Adapter-based tuning, by adding light-weight adapters to multilingual pretrained language models (mPLMs), selectively updates language-specific parameters to adapt to a new language, instead of finetuning all shared weights. This paper explores an effective way to leverage a public pool of pretrained language adapters, to overcome resource imbalances for low-resource languages (LRLs). Specifically, our research questions are, whether pretrained adapters can be composed, to complement or replace LRL adapters. While composing adapters for multi-task learning setting has been studied, the same question for LRLs has remained largely unanswered. To answer this question, we study how to fuse adapt…
-
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter MergingYi-Ran Zhao, Wen-Xuan Zhang, Huiming Wang et al. · 2024 · North American Chapter of the Association for Computational Linguistics · cité 47 fois · Pas encore résumé
Résumé scientifique
As an effective alternative to the direct fine-tuning on target tasks in specific languages, cross-lingual transfer addresses the challenges of limited training data by decoupling ''task ability'' and ''language ability'' by fine-tuning on the target task in the source language and another selected task in the target language, respectively. However, they fail to fully separate the task ability from the source language or the language ability from the chosen task. In this paper, we acknowledge the mutual reliance between task ability and language ability and direct our attention toward the gap between the target language and the source language on tasks. As the gap removes the impact of tasks…
Les travaux antérieurs sur lesquels il s’appuie
-
On the Cross-lingual Transferability of Monolingual RepresentationsMikel Artetxe, Sebastian Ruder, Dani Yogatama · 2019 · Annual Meeting of the Association for Computational Linguistics · cité 1 019 fois · Pas encore résumé
Résumé scientifique
State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of al…
-
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralizationJun-Jie Hu, Sebastian Ruder, Aditya Siddhant et al. · 2020 · International Conference on Machine Learning · cité 1 235 fois · Pas encore résumé
Résumé scientifique
Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that whi…
-
Parameter-Efficient Transfer Learning for NLPN. Houlsby, A. Giurgiu, Stanislaw Jastrzebski et al. · 2019 · International Conference on Machine Learning · cité 7 837 fois · Pas encore résumé
Résumé scientifique
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE b…
-
AdapterHub: A Framework for Adapting TransformersJonas Pfeiffer, Andreas Rücklé, Clifton A. Poth et al. · 2020 · Conference on Empirical Methods in Natural Language Processing · cité 793 fois · Pas encore résumé
Résumé scientifique
The current modus operandi in NLP involves downloading and fine-tuning pre-trained models consisting of millions or billions of parameters. Storing and sharing such large trained models is expensive, slow, and time-consuming, which impedes progress towards more general and versatile NLP methods that learn from and for many tasks. Adapters—small learnt bottleneck layers inserted within each layer of a pre-trained model— ameliorate this issue by avoiding full fine-tuning of the entire model. However, sharing and integrating adapter layers is not straightforward. We propose AdapterHub, a framework that allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. The f…
-
Simple, Scalable Adaptation for Neural Machine TranslationAnkur Bapna, N. Arivazhagan, O. Firat · 2019 · Conference on Empirical Methods in Natural Language Processing · cité 481 fois · Pas encore résumé
Résumé scientifique
Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multilingual NMT. Experiments on domain adaptation demonstrate that our proposed…
Les articles qui le citent
-
Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal RetrievalRui Cai, Zhi-Yu Dong, Jian-Feng Dong et al. · 2024 · AAAI Conference on Artificial Intelligence · cité 5 fois · Pas encore résumé
Résumé scientifique
Existing cross-modal retrieval methods typically rely on large-scale vision-language pair data. This makes it challenging to efficiently develop a cross-modal retrieval model for under-resourced languages of interest. Therefore, Cross-lingual Cross-modal Retrieval (CCR), which aims to align vision and the low-resource language (the target language) without using any human-labeled target-language data, has gained increasing attention. As a general parameter-efficient way, a common solution is to utilize adapter modules to transfer the vision-language alignment ability of Vision-Language Pretraining (VLP) models from a source language to a target language. However, these adapters are usually s…
-
No Train but Gain: Language Arithmetic for training-free Language Adapters enhancementMateusz Klimaszewski, P. Andruszkiewicz, Alexandra Birch · 2024 · International Conference on Computational Linguistics · cité 6 fois · Pas encore résumé
Résumé scientifique
Modular deep learning is the state-of-the-art solution for lifting the curse of multilinguality, preventing the impact of negative interference and enabling cross-lingual performance in Multilingual Pre-trained Language Models. However, a trade-off of this approach is the reduction in positive transfer learning from closely related languages. In response, we introduce a novel method called language arithmetic, which enables training-free post-processing to address this limitation. Extending the task arithmetic framework, we apply learning via addition to the language adapters, transitioning the framework from a multi-task to a multilingual setup. The effectiveness of the proposed solution is…
-
Sequential Adapter Stacking for Cross-Lingual Low-Resource ASRThai-Thinh Dang, Meng-Jie Qian, Kate Knill · 2026 · cité 1 fois · Pas encore résumé
Résumé scientifique
Extending large-scale multilingual automatic speech recognition (ASR) models to low-resource languages remains challenging. Model performance is skewed toward high-resource languages and degrades sharply for languages with limited labeled data and pre-training exposure. To address this, we investigate parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper. Alongside warm initialization and attention-based fusion, we propose Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter. Under controlled experiments, these approaches are evaluated on thr…
-
Typologically Informed Parameter AggregationStef Accou, Wessel Poelman · 2026 · Conference of the European Chapter of the Association for Computational Linguistics · cité 1 fois · Pas encore résumé
Résumé scientifique
Massively multilingual language models enable cross-lingual generalization but underperform on low-resource and unseen languages. While adapter-based fine-tuning offers a parameter-efficient solution, training language-specific adapters at scale remains costly. We introduce Typologically Informed Parameter Aggregation (TIPA), a training-free method that constructs proxy language adapters by aggregating existing ones, weighted by typological similarity. Integrated into the MAD-X framework, these proxies enable zero-shot cross-lingual transfer without additional training. We evaluate TIPA on five NLP tasks and over 230 languages. TIPA consistently outperforms or matches baselines such as Engli…
-
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual TransformersAnne Lauscher, Vinit Ravishankar, Ivan Vulić et al. · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP) · cité 250 fois · Pas encore résumé
Résumé scientifique
Massively multilingual transformers (MMTs) pretrained via language modeling (e.g., mBERT, XLM-R) have become a default paradigm for zero-shot language transfer in NLP, offering unmatched transfer performance.Current evaluations, however, verify their efficacy in transfers (a) to languages with sufficiently large pretraining corpora, and (b) between close languages.In this work, we analyze the limitations of downstream language transfer with MMTs, showing that, much like cross-lingual word embeddings, they are substantially less effective in resource-lean scenarios and for distant languages.Our experiments, encompassing three lower-level tasks (POS tagging, dependency parsing, NER) and two hi…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 4 novembre 2026.
Le résumé précédent a été créé il y a plus d'un an et peut être réexécuté (si nécessaire) en cliquant sur le bouton Exécuter ci-dessous.
Avertissement : Notre outil de synthèse basé sur l'IA et l'assistant virtuel fournis sur ce site Web peuvent ne pas toujours fournir des résumés complets ou des réponses exactes. Nous vous encourageons à examiner attentivement et à évaluer le contenu généré pour vous assurer de sa qualité et de sa pertinence par rapport à vos besoins.