arXiv subjects

Audun Josang

Publications and source records attributed to Audun Josang.

6 recordsLinked to original sources

arXiv2024

Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence

As machine intelligence evolves, the need to test and compare the problem-solving abilities of different AI models grows. However, current benchmarks are often simplistic, allowing models to perform uniformly well and making it difficult to distinguish their capabilities. Additionally, benchmarks typically rely on static question-answer pairs that the models might memorize or guess. To address these limitations, we introduce Dynamic Intelligence Assessment (DIA), a novel methodology for testing AI models using dynamic question templates and improved metrics across multiple disciplines such as mathematics, cryptography, cybersecurity, and computer science. The accompanying dataset, DIA-Bench, contains a diverse collection of challenge templates with mutable parameters presented in various formats, including text, PDFs, compiled binaries, visual puzzles, and CTF-style cybersecurity challenges. Our framework introduces four new metrics to assess a model's reliability and confidence across multiple attempts. These metrics revealed that even simple questions are frequently answered incorrectly when posed in varying forms, highlighting significant gaps in models' reliability. Notably, API models like GPT-4o often overestimated their mathematical capabilities, while ChatGPT-4o demonstrated better performance due to effective tool usage. In self-assessment, OpenAI's o1-mini proved to have the best judgement on what tasks it should attempt to solve. We evaluated 25 state-of-the-art LLMs using DIA-Bench, showing that current models struggle with complex tasks and often display unexpectedly low confidence, even with simpler questions. The DIA framework sets a new standard for assessing not only problem-solving but also a model's adaptive intelligence and ability to assess its limitations. The dataset is publicly available on the project's page: https://github.com/DIA-Bench.

cs.AI↗

arXiv2024

TIPS: Threat Sharing Information Platform for Enhanced Security

There is an increasing need to share threat information for the prevention of widespread cyber-attacks. While threat-related information sharing can be conducted through traditional information exchange methods, such as email communications etc., these methods are often weak in terms of their trustworthiness and privacy. Additionally, the absence of a trust infrastructure between different information-sharing domains also poses significant challenges. These challenges include redactment of information, the Right-to-be-forgotten, and access control to the information-sharing elements. These access issues could be related to time bounds, the trusted deletion of data, and the location of accesses. This paper presents an abstraction of a trusted information-sharing process which integrates Attribute-Based Encryption (ABE), Homomorphic Encryption (HE) and Zero Knowledge Proof (ZKP) integrated into a permissioned ledger, specifically Hyperledger Fabric (HLF). It then provides a protocol exchange between two threat-sharing agents that share encrypted messages through a trusted channel. This trusted channel can only be accessed by those trusted in the sharing and could be enabled for each data-sharing element or set up for long-term sharing.

cs.CR↗

arXiv2023

Active Learning on Neural Networks through Interactive Generation of Digit Patterns and Visual Representation

Artificial neural networks (ANNs) have been broadly utilized to analyze various data and solve different domain problems. However, neural networks (NNs) have been considered a black box operation for years because their underlying computation and meaning are hidden. Due to this nature, users often face difficulties in interpreting the underlying mechanism of the NNs and the benefits of using them. In this paper, to improve users' learning and understanding of NNs, an interactive learning system is designed to create digit patterns and recognize them in real time. To help users clearly understand the visual differences of digit patterns (i.e., 0 ~ 9) and their results with an NN, integrating visualization is considered to present all digit patterns in a two-dimensional display space with supporting multiple user interactions. An evaluation with multiple datasets is conducted to determine its usability for active learning. In addition, informal user testing is managed during a summer workshop by asking the workshop participants to use the system.

cs.HC↗

arXiv2007

Cumulative and Averaging Fission of Beliefs

Belief fusion is the principle of combining separate beliefs or bodies of evidence originating from different sources. Depending on the situation to be modelled, different belief fusion methods can be applied. Cumulative and averaging belief fusion is defined for fusing opinions in subjective logic, and for fusing belief functions in general. The principle of fission is the opposite of fusion, namely to eliminate the contribution of a specific belief from an already fused belief, with the purpose of deriving the remaining belief. This paper describes fission of cumulative belief as well as fission of averaging belief in subjective logic. These operators can for example be applied to belief revision in Bayesian belief networks, where the belief contribution of a given evidence source can be determined as a function of a given fused belief and its other contributing beliefs.

cs.AI↗

arXiv2006

The Cumulative Rule for Belief Fusion

The problem of combining beliefs in the Dempster-Shafer belief theory has attracted considerable attention over the last two decades. The classical Dempster's Rule has often been criticised, and many alternative rules for belief combination have been proposed in the literature. The consensus operator for combining beliefs has nice properties and produces more intuitive results than Dempster's rule, but has the limitation that it can only be applied to belief distribution functions on binary state spaces. In this paper we present a generalisation of the consensus operator that can be applied to Dirichlet belief functions on state spaces of arbitrary size. This rule, called the cumulative rule of belief combination, can be derived from classical statistical theory, and corresponds well with human intuition.

cs.AI↗

arXiv2006

Belief Calculus

In Dempster-Shafer belief theory, general beliefs are expressed as belief mass distribution functions over frames of discernment. In Subjective Logic beliefs are expressed as belief mass distribution functions over binary frames of discernment. Belief representations in Subjective Logic, which are called opinions, also contain a base rate parameter which express the a priori belief in the absence of evidence. Philosophically, beliefs are quantitative representations of evidence as perceived by humans or by other intelligent agents. The basic operators of classical probability calculus, such as addition and multiplication, can be applied to opinions, thereby making belief calculus practical. Through the equivalence between opinions and Beta probability density functions, this also provides a calculus for Beta probability density functions. This article explains the basic elements of belief calculus.

cs.AI↗