arXiv subjects

Eghbal Hosseini

Publications and source records attributed to Eghbal Hosseini.

5 recordsLinked to original sources

arXiv2026

Linear representations in language models can change dramatically over a conversation

Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evolve along these dimensions within the context of (simulated) conversations. We find that linear representations can change dramatically over a conversation; for example, information that is represented as factual at the beginning of a conversation can be represented as non-factual at the end and vice versa. These changes are content-dependent; while representations of conversation-relevant information may change, generic information is generally preserved. These changes are robust even for dimensions that disentangle factuality from more superficial response patterns, and occur across different model families and layers of the model. These representation changes do not require on-policy conversations; even replaying a conversation script written by an entirely different model can produce similar changes. However, adaptation is much weaker from simply having a sci-fi story in context that is framed more explicitly as such. We also show that steering along a representational direction can have dramatically different effects at different points in a conversation. These results are consistent with the idea that representations may evolve in response to the model playing a particular role that is cued by a conversation. Our findings may pose challenges for interpretability and steering -- in particular, they imply that it may be misleading to use static interpretations of features or directions, or probes that assume a particular range of features consistently corresponds to a particular ground-truth value. However, these types of representational dynamics also point to exciting new research directions for understanding how models adapt to context.

cs.CL↗

arXiv2023

WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words

Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of these systems as well. We focus on text--audio and introduce Whisbert, which is inspired by the text--image approach of FLAVA (Singh et al., 2022). In accordance with Babylm guidelines (Warstadt et al., 2023), we pretrain Whisbert on a dataset comprising only 100 million words plus their corresponding speech from the word-aligned version of the People's Speech dataset (Galvez et al., 2021). To assess the impact of multimodality, we compare versions of the model that are trained on text only and on both audio and text simultaneously. We find that while Whisbert is able to perform well on multimodal masked modeling and surpasses the Babylm baselines in most benchmark tasks, it struggles to optimize its complex objective and outperform its text-only Whisbert baseline.

cs.CL↗

arXiv2023

Tri-Level Model for Hybrid Renewable Energy Systems

In practical scenarios, addressing real-world challenges often entails the incorporation of diverse renewable energy sources, such as solar, energy storage systems, and greenhouse gas emissions. The core purpose of these interconnected systems is to optimize a multitude of factors and objectives concurrently. Hence, it is imperative to formulate models that comprehensively cover all these objectives. This paper introduces tri-level mathematical models for Hybrid Renewable Energy Systems (HRESs), offering a framework to concurrently tackle diverse objectives and decision-making levels within the realm of renewable energy integration. The proposed model seeks to maximize the efficiency of solar PV, enhance the performance of energy storage systems, and minimize greenhouse gas emissions.

eess.SY↗

arXiv2023

Tabu Search and Simulated Annealing metaheuristic algorithms applied to the RoRo vessel stowage problem

The search heuristics Tabu search and Simulated annealing are commonly used meta-heuristics. The two heuristics have different ways of ensuring diversification. The heuristics can be implemented for solving the stowage planning problem. The stowage planning problem occurs every time a vessel is loaded. The idea is to stow the cargo in an optimal manner satisfying a set of constraints and specifically the stability constraints. An optimal plan can be to assign the cargo to spots on the vessel so that the vessel trim and draft are optimized ensuring a low fuel consumption. Here the problem considered is the stowage planning of trailers on a Roll-on/Roll-off vessel. Roll-on/Roll-off vessels carries vehicles or trailers on the different levels of the vessel. When making a stowage plan of a vessel ballast tanks can be adjusted to improve stability and to change draft. This leads to stability constraints which can be very complex to model and solve using a mixed integer model and solver. Complicating constraints occur in many different applications and different forms and are the cause of the popularity of search algorithms, such as tabu search and simulated annealing, for solving real-life applications. Results of the two meta-heuristics are shown for real-life stowage planning cases for Roll-on/Roll-off vessels.

math.OC↗

arXiv2020

Cost-Flow Summation Algorithm Based on Table Form to Solve Minimum Cost-Flow Problem

The minimum cost-flow problems have been attracted recently in optimization because of their applications in several areas of applied science and real life. Therefore, finding optima solution of these problems would be significant. Although some heuristic approaches have been proposed for solving the problem, but there is no any method to summarize information and converting from a graph form to a table. In this paper, at first all information of the problem are summarized in a table and then an efficient algorithm based on considering costs and flows is proposed. The algorithm is strongly efficient for problems with large size and it has sufficiently suitable results by solving some our generated problems.

math.OC↗