Back to previous page
Conference Publication

A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation

ZHAW, Switzerland
Kurt Stockinger
Jan Deriu, Katsiaryna Mlynchyk, Philippe Schläpfer, Alvaro Rodrigo, Dirk von Grünigen, Nicolas Kaiser, Eneko Agirre, Mark Cieliebak
accepted in:
ACL 2020
Yet to be published
April 7, 2020
Read full article

In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database, called Operation Trees (OT). This representation allows us to invert the annotation process without loosing flexibility in the types of queries that we generate. Furthermore, it allows for fine-grained alignment of the tokens to the operations.

In our method, we randomly generate OTs from a context free grammar, and annotators just have to write the appropriate question and assign the tokens. We apply the method to create a new corpus  OTTA (Operation Trees and Token Assignment), a large semantic parsing corpus for evaluating natural language interfaces to databases. We compare OTTA to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art semantic parsing model on our data and show that our corpus is a challenging dataset and that the token alignment can be leveraged to significantly increase the performance.

Will be available soon to download

Get in touch

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form, try again please.