arXiv subjects

Tsuguchika Tabaru

Publications and source records attributed to Tsuguchika Tabaru.

9 recordsLinked to original sources

arXiv2025

Direct Quantized Training of Language Models with Stochastic Rounding

Although recent quantized Large Language Models (LLMs), such as BitNet, have paved the way for significant reduction in memory usage during deployment with binary or ternary weights, training these models still demands substantial memory footprints. This is partly because high-precision (i.e., unquantized) weights required for straight-through estimation must be maintained throughout the whole training process. To address this, we explore directly updating the quantized low-precision weights without relying on straight-through estimation during backpropagation, aiming to save memory usage during training. Specifically, we employ a stochastic rounding technique to minimize the information loss caused by the use of low-bit weights throughout training. Experimental results on our LLaMA-structured models of various sizes indicate that (1) training with only low-precision weights is feasible even when they are constrained to ternary values; (2) extending the bit width to 8 bits achieves performance on par with BitNet b1.58; (3) our models remain robust to precision scaling and memory reduction, showing minimal performance degradation when moving from FP32 to lower-memory environments (BF16/FP8); and (4) our models also support inference using ternary weights, showcasing their flexibility in deployment.

cs.LG↗

arXiv2025

Velocity dependence of the mass modifications of $ρ$ and $ω$ mesons in 12 GeV $p+A$ reactions

This study measured the invariant mass spectra of $ρ$ and $ω$ mesons in the $e^+e^-$ decay channel for 12 GeV (12.9 GeV/$c$) $p+\mathrm{C}$ and $p+\mathrm{Cu}$ reactions ($\sqrt{s}_{NN}=5.1$ GeV) at the KEK 12-GeV Proton Synchrotron. The measured spectra were divided into three $βγ$ regions to examine their velocity dependence. Across all regions, significant excesses were observed on the low-mass side of the $ω$ meson peak, beyond the contributions of known hadronic sources, in the data of the C and Cu targets. Model calculations were subsequently performed to evaluate the magnitudes of the mass modifications of $ρ$ and $ω$ mesons.

nucl-ex↗

arXiv2025

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models

Training large language models (LLMs) typically involves pre-training on massive corpora, only to restart the process entirely when new data becomes available. A more efficient and resource-conserving approach would be continual pre-training, where models are updated with new data rather than retraining from scratch. However, the introduction of new data often causes distribution shifts, leading to performance degradation on previously learned tasks. In this paper, we take a deeper look at two popular proposals for addressing this distribution shift within the continual learning literature: experience replay and gradient alignment. We consider continual pre-training of models within the Llama family of architectures at a large scale across languages with 100 billion tokens of training data in each language, finding that both replay and gradient alignment lead to more stable learning without forgetting. This conclusion holds both as we vary the model scale and as we vary the number and diversity of tasks. Moreover, we are the first to demonstrate the effectiveness of gradient alignment techniques in the context of LLM pre-training and propose an efficient implementation of meta-experience replay (MER) that imbues experience replay with the benefits of gradient alignment despite negligible compute and memory overhead. Our scaling analysis across model sizes and replay rates indicates that small rates of replaying old examples are definitely a more valuable use of compute than investing in model size, but that it is more compute efficient to scale the size of the model than invest in high rates of replaying old examples.

cs.LG↗

arXiv2025

Analysis of spectral modification of $ϕ$ mesons at finite density using a transport approach in the 12 GeV pA reactions

The hadron spectrum at finite density is an important observable for exploring the origin of hadron masses. In the KEK-PS E325 experiment, the di-electron decays of phi mesons inside and outside nuclei were measured using 12 GeV pA reactions. In the previous analysis, a significant excess was observed on the low-mass side of the phi meson peak in the data for slow-moving phi mesons ($βγ=p_ϕ/m_ϕ<1.25$) with the Cu target, and in-medium vector meson spectral modification was verified. We newly employed the PHSD transport approach to take into account the time evolution of spatial density distribution of the target nuclei. Consistent with the previous analysis, a significant excess was observed in the present analysis as well. It was found that incorporating momentum dependence into the spectral modification leads to better agreement with the experimental results. For the slow-moving $ϕ$ mesons with the Cu target, the newly obtained modification parameters are consistent with those from the previous analysis within the uncertainties.

nucl-ex↗

arXiv2023

Accuracy Improvement Technique of DNN for Accelerating CFD Simulator

There is a Computational fluid dynamics (CFD) method of incorporating the DNN inference to reduce the computational cost. The reduction is realized by replacing some calculations by DNN inference. The cost reduction depends on the implementation method of the DNN and the accuracy of the DNN inference. Thus, we propose two techniques suitable to infer flow field on the CFD grid. The first technique is to infer the flow field of the steady state from the airfoil shape. We use the position on the coordinates of the grid point and the distance from the surface of the airfoil as input information for the DNN. The second method uses the customized mean square error as a loss function. The size of the associated area for each grid point was multiplied by the square error. This method compensates for the effect caused by the size of the associated area of nonuniform allocation of grid points. The evaluation results show that the CFD incorporated first technique achieves 1.7x speedup against the CFD without DNN, while maintaining equivalent result quality. By implementing the second technique, the CFD realized further 2.3x speed up against the CFD with first technique only.

physics.flu-dyn↗

arXiv2021

MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems

Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights. High performance computing systems are pushing the frontiers of performance with a rich diversity of hardware resources and massive scale-out capabilities. There is a critical need to understand fair and effective benchmarking of machine learning applications that are representative of real-world scientific use cases. MLPerf is a community-driven standard to benchmark machine learning workloads, focusing on end-to-end performance metrics. In this paper, we introduce MLPerf HPC, a benchmark suite of large-scale scientific machine learning training applications driven by the MLCommons Association. We present the results from the first submission round, including a diverse set of some of the world's largest HPC systems. We develop a systematic framework for their joint analysis and compare them in terms of data staging, algorithmic convergence, and compute performance. As a result, we gain a quantitative understanding of optimizations on different subsystems such as staging and on-node loading of data, compute-unit utilization, and communication scheduling, enabling overall $>10 \times$ (end-to-end) performance improvements through system scaling. Notably, our analysis shows a scale-dependent interplay between the dataset size, a system's memory hierarchy, and training convergence that underlines the importance of near-compute storage. To overcome the data-parallel scalability challenge at large batch sizes, we discuss specific learning techniques and hybrid data-and-model parallelism that are effective on large systems. We conclude by characterizing each benchmark with respect to low-level memory, I/O, and network behavior to parameterize extended roofline performance models in future rounds.

cs.LG↗

arXiv2020

MLPerf Training Benchmark

Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.

cs.LG↗

arXiv2019

Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds

There has been a strong demand for algorithms that can execute machine learning as faster as possible and the speed of deep learning has accelerated by 30 times only in the past two years. Distributed deep learning using the large mini-batch is a key technology to address the demand and is a great challenge as it is difficult to achieve high scalability on large clusters without compromising accuracy. In this paper, we introduce optimization methods which we applied to this challenge. We achieved the training time of 74.7 seconds using 2,048 GPUs on ABCI cluster applying these methods. The training throughput is over 1.73 million images/sec and the top-1 validation accuracy is 75.08%.

cs.LG↗

arXiv2006

Heavy quark and J/psi production at RHIC/PHENIX

Single electrons from open heavy quarks and di-leptons from J/psi mesons have been studied systematically at RHIC/PHENIX using data from p + p, d + Au and Au + Au collisions at $\sqrt{s_{NN}}$ = 62.4 GeV, 130 GeV and 200 GeV. From the single electron study, the charm quark yield is found to scale with the number of binary collisions. This scaling has recently been confirmed using Au+Au collisions at 62.4 GeV. A new analysis shows that a high p_T suppression of single electrons is observed in Au + Au collisions at 200 GeV. This suppression suggests that heavy quarks lose significant energy in the medium. A weak rapidity dependence is seen in J/psi yields from d + Au data, which can be interpreted as a cold matter effect. We report also the results of a measurement of the spin alignment of the J/psi in the p_T range around 1.5 GeV/c.

nucl-ex↗