World Library  
Flag as Inappropriate
Email this Article

Europarl corpus

Article Id: WHEBN0036200511
Reproduction Date:

Title: Europarl corpus  
Author: World Heritage Encyclopedia
Language: English
Subject: European Parliament, Thesaurus Linguae Graecae
Collection: Corpora, European Parliament
Publisher: World Heritage Encyclopedia
Publication
Date:
 

Europarl corpus

The Europarl Corpus is a corpus (set of documents) that consists of the proceedings of the European Parliament from 1996 to the present. In its first release in 2001, it covered eleven official languages of the European Union (Danish, Dutch, English, Finnish, French, German, Greek, Italian, Portuguese, Spanish, and Swedish).[1] With the political expansion of the EU the official languages of the ten new member states have been added to the corpus data.[1] The latest release (2012)[2] comprised up to 50 million words per language with the newly added languages being slightly underrepresented as data for them is only available from 2007 onwards.[1]

The data that makes up the corpus was extracted from the website of the European Parliament and then prepared for linguistic research.[1] After sentence splitting and tokenization the sentences were aligned across languages with the help of an algorithm developed by Gale & Church (1993).[1]

The corpus has been compiled and expanded by a group of researchers led by Philipp Koehn at Edinburgh University. Initially it was designed for research purposes in statistical machine translation (SMT). However, since its first release it has been used for multiple other research purposes, including for example word sense disambiguation.

Contents

  • Europarl Corpus and Statistical Machine Translation 1
    • Quality Assessment 1.1
    • Back Translation 1.2
  • References 2
  • External links 3

Europarl Corpus and Statistical Machine Translation

In his paper “Europarl: A Parallel Corpus for Statistical Machine Translation” (2005) Koehn sums up in how far the Europarl corpus is useful for research in SMT. He uses the corpus to develop SMT systems translating each language into each of the other ten languages of the corpus making it 110 systems. This enables Koehn to establish SMT systems for uncommon language pairs that have not been considered by SMT developers beforehand, such as Finnish-Italian for example.

Quality Assessment

The Europarl corpus may not only be used for developing SMT systems but also for their assessment. By measuring the output of the systems against the original corpus data for the target language the adequacy of the translation can be assessed. Koehn uses the BLEU metric by Papineni et al. (2002) for this, which counts the coincidences of the two compared versions—SMT output and corpus data—and calculates a score on this basis.[3] The more similar the two versions are, the higher the score, and therefore the quality of the translation.[1] Results reflect that some SMT systems perform better than others, e.g., Spanish–French (40.2) in comparison to Dutch–Finnish (10.3).[1] Koehn states that the reason for this is that related languages are easier to translate into each other than those that are not.[1]

Back Translation

Furthermore, Koehn uses the SMT systems and the Europarl corpus data to investigate whether back translation is an adequate method for the evaluation of machine translation systems. For each language except English he compares the BLEU scores for translating that language from and into English (e.g. English > Spanish, Spanish > English) with those that can be achieved by measuring the original English data against the output obtained by translation from English into each language and back translation into English (e.g. English > Spanish > English).[1] The results indicate that the scores for back translation are far higher than those for monodirectional translation and what is more important they do not correlate at all with the monodirectional scores. For example the monodirectional scores for English<>Greek (27.2 and 23.2) are lower than those for English<>Portuguese (30.1 and 27.2). Yet the back translation score of 56.5 for Greek is higher than the one for Portuguese, which gets 53.6.[1] Koehn explains this with the fact that errors committed in the translation process might simply be reversed by back translation resulting in high coincidences of in- and output.[1] This, however, does not allow any conclusions about the quality of the text in the actual target language.[1] Therefore Koehn does not consider back translation an adequate method for the assessment of machine translation systems.

References

  1. ^ a b c d e f g h i j k l Koehn, Philipp (2005): "Europarl: A Parallel Corpus for Statistical Machine Translation", in: MT Summit, pp. 79–86. (http://homepages.inf.ed.ac.uk/pkoehn/publications/europarl-mtsummit05.pdf)
  2. ^ European Parliament Proceedings Parallel Corpus 1996-2011 (http://www.statmt.org/europarl)
  3. ^ Papineni, Kishore et al (2002): BLEU. A method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association of Computational Linguistics (ACL), pp. 311–318. (http://acl.ldc.upenn.edu/P/P02/P02-1040.pdf)

External links

  • Europarl homepage
  • Europarl (v3 + v7) can be downloaded from the Opus corpora site in TMX/Moses format
Help improve this article
Sourced from World Heritage Encyclopedia™ licensed under CC BY-SA 3.0
Help to improve this article, make contributions at the Citational Source
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.
 


Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.