Publications

2024

1. EMNLP

   Corpus

   MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages

   Marco Gaido^*, Sara Papi^*, Luisa Bentivogli, and 6 more authors

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024

   HF🤗 arXiv PDF Video Code Poster Slides
2. EMNLP

   What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study

   Beatrice Savoldi, Sara Papi, Matteo Negri, and 2 more authors

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Social Impact Paper Award) , Nov 2024

   Awarded

   Social Impact Paper Award

3. IWSLT Foundation Model

   SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation

   Sara Papi, Marco Gaido, Matteo Negri, and 1 more author

   In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), Aug 2024

   Abs

   This paper describes the FBK’s participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024. For this year’s submission in the speech-to-text translation (ST) sub-track, we propose SimulSeamless, which is realized by combining AlignAtt and SeamlessM4T in its medium configuration. The SeamlessM4T model is used ‘off-the-shelf’ and its simultaneous inference is enabled through the adoption of AlignAtt, a SimulST policy based on cross-attention that can be applied without any retraining or adaptation of the underlying model for the simultaneous task. We participated in all the Shared Task languages (English-\textgreaterGerman, Japanese, Chinese, and Czech-\textgreaterEnglish), achieving acceptable or even better results compared to last year’s submissions. SimulSeamless, covering more than 143 source languages and 200 target languages, is released at: https://github.com/hlt-mt/FBK-fairseq/.

4. Automatic Subtitling and Subtitle Compression: FBK at the IWSLT 2024 Subtitling track

   Marco Gaido, Sara Papi, Mauro Cettolo, and 4 more authors

   In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), Aug 2024

   Abs

   The paper describes the FBK submissions to the Subtitling track of the 2024 IWSLT Evaluation Campaign, which covers both the Automatic Subtitling and the Subtitle Compression task for two language pairs: English to German (en-de) and English to Spanish (en-es). For the Automatic Subtitling task, we submitted two systems: i) a direct model, trained in constrained conditions, that produces the SRT files from the audio without intermediate outputs (e.g., transcripts), and ii) a cascade solution that integrates only free-to-use components, either taken off-the-shelf or developed in-house. Results show that, on both language pairs, our direct model outperforms both cascade and direct systems trained in constrained conditions in last year’s edition of the campaign, while our cascade solution is competitive with the best 2023 runs. For the Subtitle Compression task, our primary submission involved prompting a Large Language Model (LLM) in zero-shot mode to shorten subtitles that exceed the reading speed limit of 21 characters per second. Our results highlight the challenges inherent in shrinking out-of-context sentence fragments that are automatically generated and potentially error-prone, underscoring the need for future studies to develop targeted solutions.

5. StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection

   Sara Papi, Marco Gaido, Matteo Negri, and 1 more author

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

   Abs

   Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.

6. Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?

   Marco Gaido, Sara Papi, Matteo Negri, and 1 more author

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

   Abs

   The field of natural language processing (NLP) has recently witnessed a transformative shift with the emergence of foundation models, particularly Large Language Models (LLMs) that have revolutionized text-based NLP. This paradigm has extended to other modalities, including speech, where researchers are actively exploring the combination of Speech Foundation Models (SFMs) and LLMs into single, unified models capable of addressing multimodal tasks. Among such tasks, this paper focuses on speech-to-text translation (ST). By examining the published papers on the topic, we propose a unified view of the architectural solutions and training strategies presented so far, highlighting similarities and differences among them. Based on this examination, we not only organize the lessons learned but also show how diverse settings and evaluation approaches hinder the identification of the best-performing solution for each architectural building block and training choice. Lastly, we outline recommendations for future works on the topic aimed at better understanding the strengths and weaknesses of the SFM+LLM solutions for ST.

7. SBAAM! Eliminating Transcript Dependency in Automatic Subtitling

   Marco Gaido, Sara Papi, Matteo Negri, and 2 more authors

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

   Abs

   Subtitling plays a crucial role in enhancing the accessibility of audiovisual content and encompasses three primary subtasks: translating spoken dialogue, segmenting translations into concise textual units, and estimating timestamps that govern their on-screen duration. Past attempts to automate this process rely, to varying degrees, on automatic transcripts, employed diversely for the three subtasks. In response to the acknowledged limitations associated with this reliance on transcripts, recent research has shifted towards transcription-free solutions for translation and segmentation, leaving the direct generation of timestamps as uncharted territory. To fill this gap, we introduce the first direct model capable of producing automatic subtitles, entirely eliminating any dependence on intermediate transcripts also for timestamp prediction. Experimental results, backed by manual evaluation, showcase our solution’s new state-of-the-art performance across multiple language pairs and diverse conditions.

8. When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP

   Sara Papi, Marco Gaido, Andrea Pilzer, and 1 more author

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

   Abs

   Despite its crucial role in research experiments, code correctness is often presumed solely based on the perceived quality of results. This assumption, however, comes with the risk of erroneous outcomes and, in turn, potentially misleading findings. To mitigate this risk, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We support our arguments with a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As countermeasures, we release pangoliNN, a library dedicated to testing neural models, and propose a Code-quality Checklist, with the goal of promoting coding best practices and improving software quality within the NLP community.

9. Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation

   Sara Papi, Peidong Wang, Junkun Chen, and 4 more authors

   In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Aug 2024

   DOI
10. How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyena

    Marco Gaido, Sara Papi, Matteo Negri, and 1 more author

    Aug 2024

2023

1. Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments

   Sara Papi, Peidong Wang, Junkun Chen, and 3 more authors

   In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Aug 2023

   DOI
2. Direct Speech Translation for Automatic Subtitling

   Sara Papi, Marco Gaido, Alina Karakanta, and 3 more authors

   Transactions of the Association for Computational Linguistics, Nov 2023

   Abs DOI

   Automatic subtitling is the task of automatically translating the speech of audiovisual content into short pieces of timed text, i.e., subtitles and their corresponding timestamps. The generated subtitles need to conform to space and time requirements, while being synchronized with the speech and segmented in a way that facilitates comprehension. Given its considerable complexity, the task has so far been addressed through a pipeline of components that separately deal with transcribing, translating, and segmenting text into subtitles, as well as predicting timestamps. In this paper, we propose the first direct speech translation model for automatic subtitling that generates subtitles in the target language along with their timestamps with a single model. Our experiments on 7 language pairs show that our approach outperforms a cascade system in the same data condition, also being competitive with production tools on both in-domain and newly released out-domain benchmarks covering new scenarios.

3. Joint Speech Translation and Named Entity Recognition

   Marco Gaido, Sara Papi, Matteo Negri, and 1 more author

   In INTERSPEECH 2023, Aug 2023

   DOI
4. Attention as a Guide for Simultaneous Speech Translation

   Sara Papi, Matteo Negri, and Marco Turchi

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2023

   DOI
5. When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP

   Sara Papi, Marco Gaido, Andrea Pilzer, and 1 more author

   Aug 2023

   arXiv:2303.16166 [cs]

   Abs DOI

   Despite its crucial role in research experiments, code correctness is often presumed only on the basis of the perceived quality of results. This assumption comes with the risk of erroneous outcomes and potentially misleading findings. To address this issue, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We present a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As a countermeasure, we propose a Code-quality Checklist and release pangoliNN, a library dedicated to testing neural models, with the goal of promoting coding best practices and improving research software quality within the NLP community.

6. Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

   Sara Papi, Marco Gaido, and Matteo Negri

   In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), Aug 2023

   DOI
7. AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation

   Sara Papi, Marco Turchi, and Matteo Negri

   In INTERSPEECH 2023, Aug 2023

   DOI
8. Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection

   Dennis Fucci, Marco Gaido, Sara Papi, and 3 more authors

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023

   Abs

   When translating words referring to the speaker, speech translation (ST) systems should not resort to default masculine generics nor rely on potentially misleading vocal traits. Rather, they should assign gender according to the speakers’ preference. The existing solutions to do so, though effective, are hardly feasible in practice as they involve dedicated model re-training on gender-labeled ST data. To overcome these limitations, we propose the first inference-time solution to control speaker-related gender inflections in ST. Our approach partially replaces the (biased) internal language model (LM) implicitly learned by the ST decoder with gender-specific external LMs. Experiments on en}rightarrowes/fr/it show that our solution outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms. The gains are even larger (up to 32.0 and 3.4) in the challenging condition where speakers’ vocal traits conflict with their gender.

2022

1. Does Simultaneous Speech Translation need Simultaneous Models?

   Sara Papi, Marco Gaido, Matteo Negri, and 1 more author

   In Findings of the Association for Computational Linguistics: EMNLP 2022, Dec 2022

   DOI
2. Efficient yet Competitive Speech Translation: FBK@IWSLT2022

   Marco Gaido, Sara Papi, Dennis Fucci, and 3 more authors

   In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), Dec 2022

   DOI
3. Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation

   Sara Papi, Marco Gaido, Matteo Negri, and 1 more author

   In Proceedings of the Third Workshop on Automatic Simultaneous Translation, Dec 2022

   DOI
4. Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora

   Sara Papi, Alina Karakanta, Matteo Negri, and 1 more author

   In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Nov 2022

   Abs

   Speech translation for subtitling (SubST) is the task of automatically translating speech data into well-formed subtitles by inserting subtitle breaks compliant to specific displaying guidelines. Similar to speech translation (ST), model training requires parallel data comprising audio inputs paired with their textual translations. In SubST, however, the text has to be also annotated with subtitle breaks. So far, this requirement has represented a bottleneck for system development, as confirmed by the dearth of publicly available SubST corpora. To fill this gap, we propose a method to convert existing ST corpora into SubST resources without human intervention. We build a segmenter model that automatically segments texts into proper subtitles by exploiting audio and text in a multimodal fashion, achieving high segmentation quality in zero-shot conditions. Comparative experiments with SubST systems respectively trained on manual and automatic segmentations result in similar performance, showing the effectiveness of our approach.

2021

1. Speechformer: Reducing Information Loss in Direct Speech Translation

   Sara Papi, Marco Gaido, Matteo Negri, and 1 more author

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021

   Abs DOI

   Transformer-based models have gained increasing popularity achieving state-of-the-art performance in many research fields including speech translation. However, Transformer’s quadratic complexity with respect to the input sequence length prevents its adoption as is with audio signals, which are typically represented by long sequences. Current solutions resort to an initial sub-optimal compression based on a fixed sampling of raw audio features. Therefore, potentially useful linguistic information is not accessible to higher-level layers in the architecture. To solve this issue, we propose Speechformer, an architecture that, thanks to reduced memory usage in the attention layers, avoids the initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria. Experiments on three language pairs (en→de/es/nl) show the efficacy of our solution, with gains of up to 0.8 BLEU on the standard MuST-C corpus and of up to 4.0 BLEU in a low resource scenario.

2. Simultaneous Speech Translation for Live Subtitling: from Delay to Display

   Alina Karakanta, Sara Papi, Matteo Negri, and 1 more author

   In Proceedings of the 1st Workshop on Automatic Spoken Language Translation in Real-World Settings (ASLTRW), Aug 2021

   Abs

   With the increased audiovisualisation of communication, the need for live subtitles in multilingual events is more relevant than ever. In an attempt to automatise the process, we aim at exploring the feasibility of simultaneous speech translation (SimulST) for live subtitling. However, the word-for-word rate of generation of SimulST systems is not optimal for displaying the subtitles in a comprehensible and readable way. In this work, we adapt SimulST systems to predict subtitle breaks along with the translation. We then propose a display mode that exploits the predicted break structure by presenting the subtitles in scrolling lines. We compare our proposed mode with a display 1) word-for-word and 2) in blocks, in terms of reading speed and delay. Experiments on three language pairs (en→it, de, fr) show that scrolling lines is the only mode achieving an acceptable reading speed while keeping delay close to a 4-second threshold. We argue that simultaneous translation for readable live subtitles still faces challenges, the main one being poor translation quality, and propose directions for steering future research.

3. Dealing with training and test segmentation mismatch: FBK@IWSLT2021

   Sara Papi, Marco Gaido, Matteo Negri, and 1 more author

   In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), Aug 2021

   Abs DOI

   This paper describes FBK’s system submission to the IWSLT 2021 Offline Speech Translation task. We participated with a direct model, which is a Transformer-based architecture trained to translate English speech audio data into German texts. The training pipeline is characterized by knowledge distillation and a two-step fine-tuning procedure. Both knowledge distillation and the first fine-tuning step are carried out on manually segmented real and synthetic data, the latter being generated with an MT system trained on the available corpora. Differently, the second fine-tuning step is carried out on a random segmentation of the MuST-C v2 En-De dataset. Its main goal is to reduce the performance drops occurring when a speech translation model trained on manually segmented data (i.e. an ideal, sentence-like segmentation) is evaluated on automatically segmented audio (i.e. actual, more realistic testing conditions). For the same purpose, a custom hybrid segmentation procedure that accounts for both audio content (pauses) and for the length of the produced segments is applied to the test data before passing them to the system. At inference time, we compared this procedure with a baseline segmentation method based on Voice Activity Detection (VAD). Our results indicate the effectiveness of the proposed hybrid approach, shown by a reduction of the gap with manual segmentation from 8.3 to 1.4 BLEU points.

2020

1. Mixtures of Deep Neural Experts for Automated Speech Scoring

   Sara Papi, Edmondo Trentin, Roberto Gretter, and 2 more authors

   In Proc. Interspeech 2020, Aug 2020

   DOI