arXiv subjects

Darko Stosic

Publications and source records attributed to Darko Stosic.

10 recordsLinked to original sources

arXiv2025

NVIDIA Nemotron 3: Efficient and Open Intelligence

We introduce the Nemotron 3 family of models - Nano, Super, and Ultra. These models deliver strong agentic, reasoning, and conversational capabilities. The Nemotron 3 family uses a Mixture-of-Experts hybrid Mamba-Transformer architecture to provide best-in-class throughput and context lengths of up to 1M tokens. Super and Ultra models are trained with NVFP4 and incorporate LatentMoE, a novel approach that improves model quality. The two larger models also include MTP layers for faster text generation. All Nemotron 3 models are post-trained using multi-environment reinforcement learning enabling reasoning, multi-step tool use, and support granular reasoning budget control. Nano, the smallest model, outperforms comparable models in accuracy while remaining extremely cost-efficient for inference. Super is optimized for collaborative agents and high-volume workloads such as IT ticket automation. Ultra, the largest model, provides state-of-the-art accuracy and reasoning performance. Nano is released together with its technical report and this white paper, while Super and Ultra will follow in the coming months. We will openly release the model weights, pre- and post-training software, recipes, and all data for which we hold redistribution rights.

cs.CL↗

arXiv2025

Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

We present Nemotron 3 Nano 30B-A3B, a Mixture-of-Experts hybrid Mamba-Transformer language model. Nemotron 3 Nano was pretrained on 25 trillion text tokens, including more than 3 trillion new unique tokens over Nemotron 2, followed by supervised fine tuning and large-scale RL on diverse environments. Nemotron 3 Nano achieves better accuracy than our previous generation Nemotron 2 Nano while activating less than half of the parameters per forward pass. It achieves up to 3.3x higher inference throughput than similarly-sized open models like GPT-OSS-20B and Qwen3-30B-A3B-Thinking-2507, while also being more accurate on popular benchmarks. Nemotron 3 Nano demonstrates enhanced agentic, reasoning, and chat abilities and supports context lengths up to 1M tokens. We release both our pretrained Nemotron 3 Nano 30B-A3B Base and post-trained Nemotron 3 Nano 30B-A3B checkpoints on Hugging Face.

cs.CL↗

arXiv2025

Pretraining Large Language Models with NVFP4

Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improvements in computational speed and resource utilization. However, quantization at this level poses challenges to training stability, convergence, and implementation, notably for large-scale models trained on long token horizons. In this study, we introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format. Our method integrates Random Hadamard transforms (RHT) to bound block-level outliers, employs a two-dimensional quantization scheme for consistent representations across both the forward and backward passes, utilizes stochastic rounding for unbiased gradient estimation, and incorporates selective high-precision layers. We validate our approach by training a 12-billion-parameter model on 10 trillion tokens -- the longest publicly documented training run in 4-bit precision to date. Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline. These findings highlight that NVFP4, when combined with our training approach, represents a major step forward in narrow-precision LLM training algorithms.

cs.CL↗

arXiv2022

Ising models of deep neural networks

This work maps deep neural networks to classical Ising spin models, allowing them to be described using statistical thermodynamics. The density of states shows that structures emerge in the weights after they have been trained -- well-trained networks span a much wider range of realizable energies compared to poorly trained ones. These structures propagate throughout the entire network and are not observed in individual layers. The energy values correlate to performance on tasks, making it possible to distinguish networks based on quality without access to data. Thermodynamic properties such as specific heat are also studied, revealing a higher critical temperature in trained networks.

cond-mat.stat-mech↗

arXiv2022

Generalized Weighted Permutation Entropy

A novel heuristic approach is proposed here for time series data analysis, dubbed Generalized weighted permutation entropy, which amalgamates and generalizes beyond their original scope two well established data analysis methods: Permutation entropy, and Weighted permutation entropy. The method introduces a scaling parameter to discern the disorder and complexity of ordinal patterns with small and large fluctuations. Using this scaling parameter, the complexity-entropy causality plane is generalized to the complexity-entropy-scale causality box. Simulations conducted on synthetic series generated by stochastic, chaotic, and random processes, as well as real world data, are shown to produce unique signatures in this three dimensional representation.

cond-mat.stat-mech↗

arXiv2021

A new look at calendar anomalies: Multifractality and day of the week effect

Stock markets can become inefficient due to calendar anomalies known as day-of-the-week effect. Calendar anomalies are well-known in financial literature, but the phenomena remain to be explored in econophysics. In this paper we use multifractal analysis to evaluate if the temporal dynamics of market returns also exhibits calendar anomalies such as day-of-the-week effects. We apply the multifractal detrended fluctuation analysis (MF-DFA) to daily returns of market indices around the world for each day of the week. Our results indicate that individual days of the week are characterized by distinct multifractal properties. Monday returns tend to exhibit more persistent behavior and richer multifractal structures than other day-resolved returns. Shuffling the series reveals that multifractality arises both from a broad probability density function and from long-term correlations. From the time-dependent multifractal analysis we find that multifractal spectra for Monday returns are much wider than for other days during periods of financial crises. The presence of day-of-the-week effects in multifractal dynamics of market returns motivates further research on calendar anomalies from an econophysics perspective.

q-fin.ST↗

arXiv2021

Distance Metric Learning through Minimization of the Free Energy

Distance metric learning has attracted a lot of interest for solving machine learning and pattern recognition problems over the last decades. In this work we present a simple approach based on concepts from statistical physics to learn optimal distance metric for a given problem. We formulate the task as a typical statistical physics problem: distances between patterns represent constituents of a physical system and the objective function corresponds to energy. Then we express the problem as a minimization of the free energy of a complex system, which is equivalent to distance metric learning. Much like for many problems in physics, we propose an approach based on Metropolis Monte Carlo to find the best distance metric. This provides a natural way to learn the distance metric, where the learning process can be intuitively seen as stretching and rotating the metric space until some heuristic is satisfied. Our proposed method can handle a wide variety of constraints including those with spurious local minima. The approach works surprisingly well with stochastic nearest neighbors from neighborhood component analysis (NCA). Experimental results on artificial and real-world data sets reveal a clear superiority over a number of state-of-the-art distance metric learning methods for nearest neighbors classification.

cs.LG↗

arXiv2021

Search Spaces for Neural Model Training

While larger neural models are pushing the boundaries of what deep learning can do, often more weights are needed to train models rather than to run inference for tasks. This paper seeks to understand this behavior using search spaces -- adding weights creates extra degrees of freedom that form new paths for optimization (or wider search spaces) rendering neural model training more effective. We then show how we can augment search spaces to train sparse models attaining competitive scores across dozens of deep learning workloads. They are also are tolerant of structures targeting current hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.

cs.LG↗

arXiv2021

Accelerating Sparse Deep Neural Networks

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.

cs.LG↗

arXiv2019

Nonextensive triplets in stock market indices

Stock market indices are one of the most investigated complex systems in econophysics. Here we extend the existing literature on stock markets in connection with nonextensive statistical mechanics. We explore the nonextensivity of price volatilities for 34 major stock market indices between 2010 and 2019. We discover that stock markets follow nonextensive statistics regarding equilibrium, relaxation and sensitivity. We find nonextensive behavior in stock markets for developed countries, but not for developing countries. Distances between nonextensive triplets suggest that some stock markets might share similar nonextensive dynamics, while others are widely different. The current findings strongly indicate that the stock market represents a system whose physics is properly described by nonextensive statistical mechanics. Our results shed light on the complex nature of stock market indices, and establish another formal link with the nonextensive theory.

q-fin.ST↗