arXiv subjects

Javier Conde

Publications and source records attributed to Javier Conde.

43 records · Page 3Linked to original sources

arXiv2024

Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal

This study investigates the potential of large language models (LLMs) to provide accurate estimates of concreteness, valence and arousal for multi-word expressions. Unlike previous artificial intelligence (AI) methods, LLMs can capture the nuanced meanings of multi-word expressions. We systematically evaluated ChatGPT-4o's ability to predict concreteness, valence and arousal. In Study 1, ChatGPT-4o showed strong correlations with human concreteness ratings (r = .8) for multi-word expressions. In Study 2, these findings were repeated for valence and arousal ratings of individual words, matching or outperforming previous AI models. Study 3 extended the prevalence and arousal analysis to multi-word expressions and showed promising results despite the lack of large-scale human benchmarks. These findings highlight the potential of LLMs for generating valuable psycholinguistic data related to multiword expressions. To help researchers with stimulus selection, we provide datasets with AI norms of concreteness, valence and arousal for 126,397 English single words and 63,680 multi-word expressions

cs.CL↗

arXiv2024

Spanish and LLM Benchmarks: is MMLU Lost in Translation?

The evaluation of Large Language Models (LLMs) is a key element in their continuous improvement process and many benchmarks have been developed to assess the performance of LLMs in different tasks and topics. As LLMs become adopted worldwide, evaluating them in languages other than English is increasingly important. However, most LLM benchmarks are simply translated using an automated tool and then run in the target language. This means that the results depend not only on the LLM performance in that language but also on the quality of the translation. In this paper, we consider the case of the well-known Massive Multitask Language Understanding (MMLU) benchmark. Selected categories of the benchmark are translated into Spanish using Azure Translator and ChatGPT4 and run on ChatGPT4. Next, the results are processed to identify the test items that produce different answers in Spanish and English. Those are then analyzed manually to understand if the automatic translation caused the change. The results show that a significant fraction of the failing items can be attributed to mistakes in the translation of the benchmark. These results make a strong case for improving benchmarks in languages other than English by at least revising the translations of the items and preferably by adapting the tests to the target language by experts.

cs.CL↗

arXiv2024

Fostering the integration of European Open Data into Data Spaces through High-Quality Metadata

The term Data Space, understood as the secure exchange of data in distributed systems, ensuring openness, transparency, decentralization, sovereignty, and interoperability of information, has gained importance during the last years. However, Data Spaces are in an initial phase of definition, and new research is necessary to address their requirements. The Open Data ecosystem can be understood as one of the precursors of Data Spaces as it provides mechanisms to ensure the interoperability of information through resource discovery, information exchange, and aggregation via metadata. However, Data Spaces require more advanced capabilities including the automatic and scalable generation and publication of high-quality metadata. In this work, we present a set of software tools that facilitate the automatic generation and publication of metadata, the modeling of datasets through standards, and the assessment of the quality of the generated metadata. We validate all these tools through the YODA Open Data Portal showing how they can be connected to integrate Open Data into Data Spaces.

cs.DB↗

arXiv2024

Collaboration of Digital Twins through Linked Open Data: Architecture with FIWARE as Enabling Technology

The collaboration of the real world and the virtual world, known as Digital Twin, has become a trend with numerous successful use cases. However, there are challenges mentioned in the literature that must be addressed. One of the most important issues is the difficulty of collaboration of Digital Twins due to the lack of standardization in their implementation. This article continues a previous work that proposed a generic architecture based on the FIWARE components to build Digital Twins in any field. Our work proposes the use of Linked Open Data as a mechanism to facilitate the communication of Digital Twins. We validate our proposal with a use case of an urban Digital Twin that collaborates with a parking Digital Twin. We conclude that Linked Open Data in combination with the FIWARE ecosystem is a real reference option to deploy Digital Twins and to enable the collaboration between Digital Twins.

cs.NI↗

arXiv2024

Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models

Vocabulary tests, once a cornerstone of language modeling evaluation, have been largely overlooked in the current landscape of Large Language Models (LLMs) like Llama, Mistral, and GPT. While most LLM evaluation benchmarks focus on specific tasks or domain-specific knowledge, they often neglect the fundamental linguistic aspects of language understanding and production. In this paper, we advocate for the revival of vocabulary tests as a valuable tool for assessing LLM performance. We evaluate seven LLMs using two vocabulary test formats across two languages and uncover surprising gaps in their lexical knowledge. These findings shed light on the intricacies of LLM word representations, their learning mechanisms, and performance variations across models and languages. Moreover, the ability to automatically generate and perform vocabulary tests offers new opportunities to expand the approach and provide a more complete picture of LLMs' language skills.

cs.CL↗

arXiv2023

How many words does ChatGPT know? The answer is ChatWords

The introduction of ChatGPT has put Artificial Intelligence (AI) Natural Language Processing (NLP) in the spotlight. ChatGPT adoption has been exponential with millions of users experimenting with it in a myriad of tasks and application domains with impressive results. However, ChatGPT has limitations and suffers hallucinations, for example producing answers that look plausible but they are completely wrong. Evaluating the performance of ChatGPT and similar AI tools is a complex issue that is being explored from different perspectives. In this work, we contribute to those efforts with ChatWords, an automated test system, to evaluate ChatGPT knowledge of an arbitrary set of words. ChatWords is designed to be extensible, easy to use, and adaptable to evaluate also other NLP AI tools. ChatWords is publicly available and its main goal is to facilitate research on the lexical knowledge of AI tools. The benefits of ChatWords are illustrated with two case studies: evaluating the knowledge that ChatGPT has of the Spanish lexicon (taken from the official dictionary of the "Real Academia Española") and of the words that appear in the Quixote, the well-known novel written by Miguel de Cervantes. The results show that ChatGPT is only able to recognize approximately 80% of the words in the dictionary and 90% of the words in the Quixote, in some cases with an incorrect meaning. The implications of the lexical knowledge of NLP AI tools and potential applications of ChatWords are also discussed providing directions for further work on the study of the lexical knowledge of AI tools.

cs.CL↗

arXiv2023

Modeling Digital Twin Data and Architecture: A Building Guide with FIWARE as Enabling Technology

The use of Digital Twins in the industry has become a growing trend in recent years, allowing to improve the lifecycle of any process by taking advantage of the relationship between the physical and the virtual world. Existing literature formulates several challenges for building Digital Twins, as well as some proposals for overcoming them. However, in the vast majority of the cases, the architectures and technologies presented are strongly bounded to the domain where the Digital Twins are applied. This article proposes the FIWARE Ecosystem, combining its catalog of components and its Smart Data Models, as a solution for the development of any Digital Twin. We also provide a use case to showcase how to use FIWARE for building Digital Twins through a complete example of a Parking Digital Twin. We conclude that the FIWARE Ecosystem constitutes a real reference option for developing DTs in any domain.

cs.CY↗