Searcharxiv⌕ Search

arXiv subjects

Nika Zahedi

Publications and source records attributed to Nika Zahedi.

4 recordsLinked to original sources

MicroLens: Read, Trace, and Rewire the Residual Stream with Two Lenses

Lenses that decode hidden states into words are a popular way to look inside language models, but a heatmap of decoded words only shows what a vector reads as and cannot distinguish whether, or how much, the model uses the information. We present an interactive viewer that puts reading, tracing and editing on the same layers-by-tokens grid of the residual stream. In the viewer, every cell can be decoded with two lenses, the logit lens and the Jacobian lens (J-lens), traced back to the tokens it drew from, and edited, with the rest of the model rerun on the spot. We demonstrate three usage scenarios on Gemma-3-4B-it, backed by sweeps over a few hundred prompts, to show what this tool can discover. First, the two lenses give different directions to write with, each suiting a different edit: the logit lens's word direction sets what a token outputs, while the J-lens's direction changes what later tokens read. Second, in two-hop trivia questions (the capital of the state or country containing a given city), the state or country is readable at the city token from as early as layer 7, and swapping it there can redirect the answer 92-98% of the time; however, at the answer position, the state or country information is not used. Third, when a chat model answers questions with a step-by-step reasoning prompt, the correct answer is often already the top word in the residual stream before any reasoning is written: for 89-92% of retrieval questions and 39-63% of multi-hop and serial ones, far above their respective distractors (at most 5%). However, ordering puzzles are an exception, as the model works them out only once it writes its final reasoning steps. Furthermore, for answers that are visible in the residual stream, the J-lens reads them a layer or two earlier than the logit lens across the different types of question.

cs.LG↗

Closed-Form Statistics and Design of Mode-Division-Multiplexing Systems Employing Group-Delay Compensation and Mode Permutation

Excessive accumulation of group-delay (GD) spread increases computational complexity and affects tracking of receiver-based multi-input multi-output signal processing, posing challenges to long-haul mode-division multiplexing in multimode fiber (MMF). GD compensation, which involves periodically exchanging propagating signals between modes with lower and higher GDs, can potentially reduce GD spread. In this work, we investigate two GD compensation schemes: conventional compensation, which alternates fiber types with opposite GD orderings, and self-compensation, which employs a single fiber type with periodically inserted mode permuters. We provide analytical expressions for GD statistics in compensated systems with arbitrary MMF types and mode permuters, accounting for random inter-group coupling, mode scrambling, and refractive index errors. To enhance the effectiveness of GD compensation over the C-band, we propose optimized graded-index depressed-cladding MMFs with index profiles tailored to control mode-dependent chromatic dispersion. We also analyze the impact of fiber fabrication errors and explore mitigation strategies based on GD characterization and sorting. Design examples and numerical simulations demonstrate improved system performance, highlighting the trade-offs between compensation effectiveness, system complexity, and transmission losses.

physics.optics↗

Low-Loss All-Fiber Mode Permuter Design Exploiting Propagation Constant Engineering and Cascaded Bragg Gratings

Managing group-delay (GD) spread is vital for reducing the complexity of digital signal processing (DSP) in long-haul systems using multi-mode fibers. GD compensation through mode permutation, which involves periodically exchanging power between modes with lower and higher GDs, can reduce GD spread. GD spread is maximally reduced when the modal GDs of the transmission fiber satisfy a specific relation and the mode permuter exchanges power between specific modes. Mode permuters with fiber Bragg gratings have been developed for links accommodating $D=6$ guided spatial and polarization modes; however, to the best of our knowledge, there have been no designs for links supporting a greater number of modes. We present two mode permuter designs based on fiber Bragg gratings for links employing graded-index transmission fibers with $D=12$ guided spatial and polarization modes. One design utilizes a step-index (SI) transverse profile, and the other features a free-form-optimized transverse profile achieving improved performance. For the SI design, we achieve mode-dependent loss standard deviation (MDL STD) and mode-averaged loss (MAL) of less than $0.14$ dB and $0.27$ dB, respectively, over the C-band. For the design with a free-form-optimized transverse profile, we achieve MDL STD and MAL of less than $0.11$ dB and $0.07$ dB, respectively, over the C-band. We numerically evaluate the designs through link simulations and quantify the reduction in GD spread for different levels of random inter-group coupling in the fiber. Our results show that in a link with periodic mode permutation and mode scrambling, the GD STD is reduced by a factor over $3.13$ compared to a link relying solely on periodic mode scrambling.

physics.optics↗

Bridging Associative Memory and Probabilistic Modeling

Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. The first studies recurrent neural networks designed to denoise, complete and retrieve data, whereas the second studies learning and sampling from probability distributions. Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions. We showcase four examples: First, we propose new energy-based models that flexibly adapt their energy functions to new in-context datasets, an approach we term \textit{in-context learning of energy functions}. Second, we propose two new associative memory models: one that dynamically creates new memories as necessitated by the training data using Bayesian nonparametrics, and another that explicitly computes proportional memory assignments using the evidence lower bound. Third, using tools from associative memory, we analytically and numerically characterize the memory capacity of Gaussian kernel density estimators, a widespread tool in probababilistic modeling. Fourth, we study a widespread implementation choice in transformers -- normalization followed by self attention -- to show it performs clustering on the hypersphere. Altogether, this work urges further exchange of useful ideas between these two continents of artificial intelligence.

cs.LG↗