arXiv subjects

Ahmetcan Yavuz

Publications and source records attributed to Ahmetcan Yavuz.

3 recordsLinked to original sources

arXiv2026

Objective vs. Search: Decomposing What Makes a Good Tokeniser

Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. We train language models with tokenisers produced by each algorithm, varying: model size, vocabulary sizes, and domain (English-only vs. multilingual). Evaluating models on bits-per-byte, we find that the search procedure -- not the objective -- is the dominant factor: bottom-up tokenisers consistently achieve lower bits-per-byte in most settings. Evaluating models on the BLiMP task, however, shows no consistent relationship between design choice and performance. Overall, our results disentangle the effect of tokeniser design choices on language modelling performance, offering concrete guidance for their more principled construction.

cs.CL↗

arXiv2026

What Language is This? Ask Your Tokenizer

Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm. In short, to predict a string's language label, we simply ask: under which language's unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fasttext, GlotLID-M, and CLD3, show that UniLID achieves competitive performance on standard benchmarks, reaches 69% accuracy with five labeled samples per language and 89% with 25, and delivers large gains on fine-grained dialect identification.

cs.CL↗

arXiv2026

Pixel-wise Planarity for High-Precision Monocular Plane Segmentation

Plane segmentation from a single RGB image remains challenging due to imprecise region grouping and geometrically inconsistent supervision, often leading to over-segmentation and false planar detections. We propose instead a pixel-wise planarity prediction framework for robust monocular plane segmentation. Building on a pretrained monocular geometric backbone predicting depth and surface normals, we introduce a dedicated planarity head that estimates per-pixel planarity confidence. During inference, predicted depth, normals, and planarity are combined in a lightweight region-growing procedure that enforces geometric consistency when forming plane segments. We further analyze existing plane ground-truth annotations and demonstrate substantial geometric inconsistencies under strict distance thresholds. Across multiple datasets, our method achieves improved geometric precision and segmentation quality compared to prior state-of-the-art approaches, while improving computational efficiency. Our code and models are available at https://github.com/alpayozkan/PixelwisePlanarity.

cs.CV↗