Language Identification as part of the Text Corpus Creation Pipeline at the Language Bank of Finland
Abstrakt
The Language Bank of Finland hosts text corpora originating from Finland. Two of the most used ones are the Newspaper and Periodical Corpus of the National Library of Finland and the Suomi24 Corpus.
The Language Bank has received considerable additions to both corpora and is currently creating new versions of the corpora. We are debuting language identification as part of the corpus creation pipeline.
As a language identifier, we are using our recently published HeLI-OTS software. This paper investigates the results and the quality of language identification.
We created a new dataset for evaluating the efficacy of language identification by extracting random samples from both corpora and manually annotating their language. We were especially interested in seeing how the relatively low OCR quality of the oldest part of the KLK-fi collection will affect language identification when using an off-the-shelf program like HeLI-OTS.
The oldest part of the KLK-fi collection and many parts of the Suomi24 corpus contain Finnish written dialectally or otherwise differing from the standard written Finnish. HeLI-OTS software includes several separate language models for dialectal Finnish, and in this article, we evaluate their usefulness using the new data set.
For both corpora, the overall micro F1 scores increase when using the additional dialectal models. Additionally, we take a detailed look at some language identification errors and discuss possible solutions.