arXiv subjects

Alexey Kutalev

Publications and source records attributed to Alexey Kutalev.

6 recordsLinked to original sources

arXiv2026

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.

cs.AI↗

arXiv2024

Investigating on RLHF methodology

In this article, we investigate the alignment of Large Language Models according to human preferences. We discuss the features of training a Preference Model, which simulates human preferences, and the methods and details we found essential for achieving the best results. We also discuss using Reinforcement Learning to fine-tune Large Language Models and describe the challenges we faced and the ways to overcome them. Additionally, we present our experience with the Direct Preference Optimization method, which enables us to align a Large Language Model with human preferences without creating a separate Preference Model. As our contribution, we introduce the approach for collecting a preference dataset through perplexity filtering, which makes the process of creating such a dataset for a specific Language Model much easier and more cost-effective.

cs.LG↗

arXiv2022

Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting

Following the invention in 2017 of the EWC method, several methods have been proposed to calculate the importance of neural network weights for use in the EWC method. Despite the significant difference in calculating the importance of weights, they all proved to be effective. Accordingly, a reasonable question arises as to how similar the importances of the weights calculated by different methods. To answer this question, we calculated layer-by-layer correlations of the importance of weights calculated by all those methods. As a result, it turned out that the importances of several of the methods correlated with each other quite strongly and we were able to present an explanation for such a correlation. At the same time, for other methods, the correlation can vary from strong on some layers of the network to negative on other layers. Which raises a reasonable question: why, despite the very different calculation methods, all those importances allow EWC method to overcome the catastrophic forgetting of neural networks perfectly?

cs.LG↗

arXiv2022

Empirical investigations on WVA structural issues

In this paper we want to present the results of empirical verification of some issues concerning the methods for overcoming catastrophic forgetting in neural networks. First, in the introduction, we will try to describe in detail the problem of catastrophic forgetting and methods for overcoming it for those who are not yet familiar with this topic. Then we will discuss the essence and limitations of the WVA method which we presented in previous papers. Further, we will touch upon the issues of applying the WVA method to gradients or optimization steps of weights, choosing the optimal attenuation function in this method, as well as choosing the optimal hyper-parameters of the method depending on the number of tasks in sequential training of neural networks.

cs.LG↗

arXiv2021

Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning

This paper is devoted to the features of the practical application of the Elastic Weight Consolidation (EWC) method for continual learning of neural networks on several training sets. We will more rigorously compare the well-known methodologies for calculating the importance of weights used in the EWC method. These are the Memory Aware Synapses (MAS), Synaptic Intelligence (SI) methodologies and the calculation of the importance of weights based on the Fisher information matrix from the original paper on EWC. We will consider these methodologies as applied to deep neural networks with fully connected and convolutional layers, find the optimal hyperparameters for each of the methodologies, and compare the results of continual neural network learning using these hyperparameters. Next, we will point out the problems that arise when applying the EWC method to deep neural networks with convolutional layers and self-attention layers, such as the "gradient explosion" and the loss of meaningful information in the gradient when using the constraint of its norm (gradient clipping). Then, we will propose a stabilization approach for the EWC method that helps to solve these problems, evaluate it in comparison with the original methodology and show that the proposed stabilization approach performs on the task of maintaining skills during continual learning no worse than the original EWC, but does not have its disadvantages. In conclusion, we present an interesting fact about the use of various types of weight importance in the problem of neural network pruning.

cs.LG↗

arXiv2021

Natural Way to Overcome the Catastrophic Forgetting in Neural Networks

Not so long ago, a method was discovered that successfully overcomes the catastrophic forgetting in neural networks. Although we know about the cases of using this method to preserve skills when adapting pre-trained networks to particular tasks, it has not obtained widespread distribution yet. In this paper, we would like to propose an alternative method of overcoming catastrophic forgetting based on the total absolute signal passing through each connection in the network. This method has a simple implementation and seems to us essentially close to the processes occurring in the brain of animals to preserve previously learned skills during subsequent learning. We hope that the ease of implementation of this method will serve its wide application.

cs.LG↗