Dealing with Newly Emerging OOVs in Broadcast Programs by Daily Updates of the Lexicon and Language Model
Abstract
This paper deals with out-of-vocabulary (OOV) word recognition in the task of 24/7 broadcast stream transcription. Here, the majority of OOVs newly emerging over time are constituted of names of politicians, athletes, major world events, disasters, etc.
The absence of these content OOVs, e.g. COVID-19, is detrimental to human understanding of the recognized text and harmful to further NLP processing, such as machine translation, named entity recognition or any type of semantic or dialogue analysis.
In production environments, content OOVs are of extreme importance and it is essential that their correct transcription is provided as soon as possible. For this purpose, an approach based on daily updates of the lexicon and language model is proposed.
It consists of three consecutive steps: a) the identification of new content OOVs from already existing text sources, b) their controlled addition into the lexicon of the transcription system and c) proper tuning of the language model. Experimental evaluation is performed on an extensive data-set compiled from various Czech broadcast programs.
This data was produced by a real transcription platform over the course of 300 days in 2019. Detailed statistics and analysis of new content OOVs emerging within this period are also provided.