COST Action Multi3Generation: Multi-task, Multilingual, Multi-modal Language Generation

Abstract

Multi3Generation aims to foster an interdisciplinary network of research groups working on different aspects of language generation (LG). We frame LG broadly as the set of tasks where the ultimate goal involves generating language. In contrast to the more classical definition of NLG, this also includes tasks not concerned with LG in an immediate sense, but that could obviously inform or improve LG models. The Action will focus on the four core challenges: 1) Data and information representations: in modern applications, inputs can be different sources such as images, videos, KBs and graphs. 2) Machine Learning (ML): modern ML approaches face additional challenges when applied to LG; inputs should be mapped to different correct outputs, i.e. challenges involve e.g. structured prediction and representation learning. 3) Interaction: Applications of LG, e.g. Dialogue Systems, Conversational Search Interfaces and Human-Robot Interaction pose additional challenges to LG due to uncertainty derived from the changing environment and the non-deterministic fashion of interaction. 4) KB exploitation: structured knowledge is key to many NLP tasks, including NLG, supporting ML methods that require expansion, filtering, disambiguation or user adaptation of generated content. This Action aims to address these challenges by answering the following questions: How can we efficiently exploit common-sense, world knowledge and multi-modal information from various inputs such as knowledge bases, images and videos to address LG tasks such as multi-modal machine translation (MT), video description and summarisation? How can modern ML methods such as multi-task learning (MTL), representation learning and structured prediction be leveraged for LG? How can the models from (1) and (2) be exploited to develop dialogue-based, conversational Human-Computer and Human-Robot interaction methods?