arXiv subjects

Johann Higl

Publications and source records attributed to Johann Higl.

3 recordsLinked to original sources

arXiv2026

GLM-RAG: Graph Language Models for Graph-Based Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) over knowledge graphs requires retrievers that can effectively capture both graph structure and semantic information. Recent approaches have explored graph neural network (GNN)-based retrievers to model graph topology in multi-hop reasoning tasks. In parallel, graph language models (GLMs) have emerged as a promising paradigm that integrates graph reasoning and the semantic capabilities of language models. In this work, we introduce a GLM-based retriever and investigate the comparative strengths of GLM-based, GNN-based, and traditional vector-search-based retrievers in single- and multi-hop RAG settings, and with a particular focus on transferability to unseen domains. Our findings suggest that finetuned GLM retrievers generalize better out of domain, achieving SOTA on two multi-hop benchmarks. On in-domain multi-hop QA datasets they remain comparable to prior work, with promising scaling as parameters and subgraph coverage increase. GNN-based retrievers achieve higher graph coverage with an efficient training setup, whereas the vector-search baseline excels at single-hop datasets.

cs.AI↗

arXiv2026

A Family of LLMs Liberated from Static Vocabularies

Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face.

cs.CL↗

arXiv2020

Calibrating Core Overshooting Parameters With Two-dimensional Hydrodynamical Simulations

The extent of mixed regions around convective zones is one of the biggest uncertainties in stellar evolution. 1D overshooting descriptions introduce a free parameter ($f_{ov}$) that is in general not well constrained from observations. Especially in small central convective regions the value is highly uncertain due to its tight connection to the pressure scale height. Long-term multi-dimensional hydrodynamic simulations can be used to study the size of the overshooting region and the involved mixing processes. Here we show how one can calibrate an overshooting parameter by performing 2D Maestro simulations of Zero-Age-Main-Sequence stars ranging from $1.3$ to $3.5 M_\odot$. The simulations cover the convective cores of the stars and a large fraction of the surrounding radiative envelope. We follow the convective flow for at least 20 convective turnover times, while the longest simulation covers 430 turnover time scales. This allows us to study how the mixing as well as the convective boundary evolve with time, and how the resulting entrainment can be interpreted in terms of overshooting parameters. We find that increasing the overshooting parameter $f_{ov}$ beyond a certain value in the initial model of our simulations, changes the mixing behaviour completely. This result can be used to put limits on the overshooting parameter. We find $0.010 < f_{ov} < 0.017$ to be in good agreement with our simulations of a $3.5 M_\odot$ mass star. We also identify a diffusive mixing component due to internal gravity waves (IGW) that is active throughout the convectively stable layer, but likely overestimated in our simulations. Furthermore, applying our calibration method to simulations of less massive stars suggests a need for a mass-dependent overshooting description where the mixing in terms of the pressure scale height is reduced for small convective cores.

astro-ph.SR↗