PhrasIS: Phrase Inference and Similarity benchmark
No hay miniatura disponible
Fecha
2024-12
Título de la revista
ISSN de la revista
Título del volumen
Editor
Oxford University Press
Resumen
We present PhrasIS, a benchmark dataset composed of natural occurring Phrase pairs with Inference and Similarity annotations for the evaluation of semantic representations. The described dataset fills the gap between word and sentence-level datasets, allowing to evaluate compositional models at a finer granularity than sentences. Contrary to other datasets, the phrase pairs are extracted from naturally occurring text in image captions and news headlines. All the text fragments have been annotated by experts following a rigorous process also described in the manuscript achieving high inter annotator agreement. In this work we analyse the dataset, showing the relation between inference labels and similarity scores. With 10K phrase pairs split in development and test, the dataset is an excellent benchmark for testing meaning representation systems.
Palabras clave
Descripción
Materias
Cita
Lopez-Gazpio, Gaviria, García, Sanjurjo-González, Sanz, Zarranz, Maritxalar, & Agirre. (2024). PhrasIS: Phrase Inference and Similarity benchmark. Logic Journal of the IGPL, 32(6), 1088-1101. https://doi.org/10.1093/JIGPAL/JZAE037