Abstract
Speech recognition and machine translation have made major progress over the past decades, providing practical systems to map one language sequence to another. Although multiple modalities such as sound and video are becoming increasingly available, the state-of-the-art systems are inherently unimodal, in the sense that they take a single modality-either speech or text-as input.
Evidence from human learning suggests that additional modalities can provide disambiguating signals crucial for many language tasks. Here, we describe the How2 dataset, a large, open-domain collection of videos with transcriptions and their translations.
We then show how this single dataset can be used to develop systems for a variety of language tasks and present a number of models meant as starting points. Across tasks, we find that building multi-modal architectures that perform better than their unimodal counterpart remains a challenge.
This leaves plenty of room for the exploration of more advanced solutions that fully