arXiv subjects

Shinichi Hemmi

Publications and source records attributed to Shinichi Hemmi.

3 recordsLinked to original sources

arXiv2025

PLaMo 2 Technical Report

In this report, we introduce PLaMo 2, a series of Japanese-focused large language models featuring a hybrid Samba-based architecture that transitions to full attention via continual pre-training to support 32K token contexts. Training leverages extensive synthetic corpora to overcome data scarcity, while computational efficiency is achieved through weight reuse and structured pruning. This efficient pruning methodology produces an 8B model that achieves performance comparable to our previous 100B model. Post-training further refines the models using a pipeline of supervised fine-tuning (SFT) and direct preference optimization (DPO), enhanced by synthetic Japanese instruction data and model merging techniques. Optimized for inference using vLLM and quantization with minimal accuracy loss, the PLaMo 2 models achieve state-of-the-art results on Japanese benchmarks, outperforming similarly-sized open models in instruction-following, language fluency, and Japanese-specific knowledge.

cs.CL↗

arXiv2024

Fast and Numerically Stable Implementation of Rate Constant Matrix Contraction Method

The rate constant matrix contraction (RCMC) method, proposed by Sumiya et al. (2015, 2017), enables fast and numerically stable simulations of chemical kinetics on large-scale reaction path networks. Later, Iwata et al. (2023) mathematically reformulated the RCMC method as a numerical algorithm to solve master equations whose coefficient matrices, known as rate constant matrices, possess the detailed balance property. This paper aims to accelerate the RCMC method. The bottleneck in the RCMC method lies in the greedy selection of steady states, which is actually equivalent to the greedy algorithm for the MAP inference in DPPs under cardinality constraints. Hemmi et al. (2022) introduced a fast implementation of the greedy DPP MAP inference, called LazyFastGreedy, by combining the greedy algorithm of Chen et al. (2018) with the lazy greedy algorithm by Minoux (1978), a practically efficient greedy algorithm that exploits the submodularity of the objective function. However, for instances arising from chemical kinetics, the straightforward application of LazyFastGreedy suffers from catastrophic cancellations due to the wide range of reaction time scales. To address this numerical instability, we propose a modification to LazyFastGreedy that avoids the subtraction of like-sign numbers by leveraging the properties of rate constant matrices and the connection of the DPP MAP inference to Cholesky decomposition. For faster implementation, we utilize a segment tree, a data structure that manages one-dimensional arrays of elements in a semigroup. We also analyze the increase in relative errors caused by like-sign subtractions and permit such subtractions when they do not lead to catastrophic cancellations, aiming to further accelerate the process. Using real instances, we confirm that the proposed algorithm is both numerically stable and significantly faster than the original RCMC method.

math.NA↗

arXiv2022

Lazy and Fast Greedy MAP Inference for Determinantal Point Process

The maximum a posteriori (MAP) inference for determinantal point processes (DPPs) is crucial for selecting diverse items in many machine learning applications. Although DPP MAP inference is NP-hard, the greedy algorithm often finds high-quality solutions, and many researchers have studied its efficient implementation. One classical and practical method is the lazy greedy algorithm, which is applicable to general submodular function maximization, while a recent fast greedy algorithm based on the Cholesky factorization is more efficient for DPP MAP inference. This paper presents how to combine the ideas of "lazy" and "fast", which have been considered incompatible in the literature. Our lazy and fast greedy algorithm achieves almost the same time complexity as the current best one and runs faster in practice. The idea of "lazy + fast" is extendable to other greedy-type algorithms. We also give a fast version of the double greedy algorithm for unconstrained DPP MAP inference. Experiments validate the effectiveness of our acceleration ideas.

cs.DS↗