Searcharxiv⌕ Search

arXiv subjects

Abhishek Kumar

Publications and source records attributed to Abhishek Kumar.

At least 109 records · Page 6Linked to original sources
arXiv2023

Phonon-Induced Collective Modes in Spin-Orbit Coupled Polar Metals

We study the interplay between collective electronic and lattice modes in polar metals in an applied magnetic field aligned with the polar axis. Static spin-orbit coupling leads to the appearance of a particle-hole spin-flip continuum that is gapped at low energies in a finite field. We find that a weak spin-orbit assisted coupling between electrons and polar phonons induces the emergence of electronic collective modes. The strength of the applied magnetic field tunes the number of modes and their energies, which can lie both above and below the particle-hole continuum. For a range of field values, we identify Fano-like interference between the electronic continuum and phonons. We show that signatures of these collective modes can be observed in electron spin resonance experiments, and we provide the corresponding theoretical predictions.

cond-mat.mes-hall↗
arXiv2023

Early Weight Averaging meets High Learning Rates for LLM Pre-training

Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a simple idea checkpoint averaging along the trajectory of a training run to improve both convergence and generalization quite early on during training. Here we show that models trained with high learning rates observe higher gains due to checkpoint averaging. Furthermore, these gains are amplified when checkpoints are sampled with considerable spacing in training steps. Our training recipe outperforms conventional training and popular checkpoint averaging baselines such as exponential moving average (EMA) and stochastic moving average (SWA). We evaluate our training recipe by pre-training LLMs, where high learning rates are inherently preferred due to extremely large batch sizes. Specifically, we pre-trained nanoGPT-2 models of varying sizes, small (125M), medium (335M), and large (770M)on the OpenWebText dataset, comprised of 9B tokens. Additionally, we present results for publicly available Pythia LLMs, ranging from 1B to 12B, which were trained on the PILE-deduped dataset containing 207B tokens.

cs.LG↗
arXiv2023

Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion

Sampling from the posterior distribution poses a major computational challenge in solving inverse problems using latent diffusion models. Common methods rely on Tweedie's first-order moments, which are known to induce a quality-limiting bias. Existing second-order approximations are impractical due to prohibitive computational costs, making standard reverse diffusion processes intractable for posterior sampling. This paper introduces Second-order Tweedie sampler from Surrogate Loss (STSL), a novel sampler that offers efficiency comparable to first-order Tweedie with a tractable reverse process using second-order approximation. Our theoretical results reveal that the second-order approximation is lower bounded by our surrogate loss that only requires $O(1)$ compute using the trace of the Hessian, and by the lower bound we derive a new drift term to make the reverse process tractable. Our method surpasses SoTA solvers PSLD and P2L, achieving 4X and 8X reduction in neural function evaluations, respectively, while notably enhancing sampling quality on FFHQ, ImageNet, and COCO benchmarks. In addition, we show STSL extends to text-guided image editing and addresses residual distortions present from corrupted images in leading text-guided image editing methods. To our best knowledge, this is the first work to offer an efficient second-order approximation in solving inverse problems using latent diffusion and editing real-world images with corruptions.

cs.LG↗
arXiv2023

Towards Last-layer Retraining for Group Robustness with Fewer Annotations

Empirical risk minimization (ERM) of neural networks is prone to over-reliance on spurious correlations and poor generalization on minority groups. The recent deep feature reweighting (DFR) technique achieves state-of-the-art group robustness via simple last-layer retraining, but it requires held-out group and class annotations to construct a group-balanced reweighting dataset. In this work, we examine this impractical requirement and find that last-layer retraining can be surprisingly effective with no group annotations (other than for model selection) and only a handful of class annotations. We first show that last-layer retraining can greatly improve worst-group accuracy even when the reweighting dataset has only a small proportion of worst-group data. This implies a "free lunch" where holding out a subset of training data to retrain the last layer can substantially outperform ERM on the entire dataset with no additional data or annotations. To further improve group robustness, we introduce a lightweight method called selective last-layer finetuning (SELF), which constructs the reweighting dataset using misclassifications or disagreements. Our empirical and theoretical results present the first evidence that model disagreement upsamples worst-group data, enabling SELF to nearly match DFR on four well-established benchmarks across vision and language tasks with no group annotations and less than 3% of the held-out class annotations. Our code is available at https://github.com/tmlabonte/last-layer-retraining.

cs.LG↗
arXiv2023

Anomalous pumping in the non-Hermitian Rice-Mele model

We study topological charge pumping (TCP) in the Rice-Mele (RM) model with irreciprocal hopping. The non-Hermiticity gives rise to interesting pumping physics, owing to the presence of skin effect and exceptional points. In the static 1D RM model, we observe two independent tuning knobs that drive the topological transition, viz., non-Hermitian parameter $γ$ and system size $N$. To elucidate the system-size dependency, we made use of the finite-size generalized Brillouin zone (GBZ) scheme. This scheme captures the state pumping of topological edge modes in the static 1D RM model and provides further insight into engineering novel gapless exceptional edge modes with the help of adiabatic drive. Finally, we apply three types of adiabatic protocols to study TCP in the 1+1D RM model. We further explain the number of pumped charges (in each period) using a non-Bloch topological invariant. This exactly explains the presence of different pumping phases in the non-Hermitian RM model as we tune the non-Hermitian parameter $γ$. We observe that in a non-Hermitian system, even a trivial adiabatic protocol can lead to pumping that has no Hermitian counterpart.

cond-mat.mes-hall↗
arXiv2023

On-chip Amorphous THz Topological Photonic Interconnects

Valley Hall photonic crystals (VPCs) offer the potential to create topological waveguides capable of guiding light through sharp bends on a chip. They can seamlessly integrate with functional components while occupying minimal space, making them a promising technology for terahertz (THz) topological photonic integrated circuits. However, a significant limitation for THz-scale integrated VPC-based devices has been the absence of arbitrary bend interconnects. Due to the crystalline symmetry, the traditional VPC designs restrict waveguides to the principal lattice axes (i.e., only 0-, 60- or 120- degree orientations). Here, we present an on-chip, all silicon implementation of deformed VPCs enabling topological waveguides with a variety of shapes and bends. Although the lattice is amorphous and lacks long-range periodicity, the topological protection of the waveguides is sustained by short-range order. We experimentally demonstrate the robust on-chip transmission of THz waves through waveguides of complicated shapes and arbitrary bends. We implement an amorphous lattice that serves as a frequency-dependent router capable of splitting the input signal into two perpendicular output ports, which cannot be achieved with an undeformed VPC. In addition, we showcase on-chip THz communication through 90 degree and P-shaped VPC waveguides, achieving data rates of 72 Gbit/s and 32 Gbit/s, respectively. Our findings demonstrate that the amorphous topological photonic crystals significantly enhance the adaptability of on-chip interconnections while preserving the performance of the topological waveguides.

physics.optics↗
arXiv2023

Enhancing Clustering Representations with Positive Proximity and Cluster Dispersion Learning

Contemporary deep clustering approaches often rely on either contrastive or non-contrastive techniques to acquire effective representations for clustering tasks. Contrastive methods leverage negative pairs to achieve homogenous representations but can introduce class collision issues, potentially compromising clustering performance. On the contrary, non-contrastive techniques prevent class collisions but may produce non-uniform representations that lead to clustering collapse. In this work, we propose a novel end-to-end deep clustering approach named PIPCDR, designed to harness the strengths of both approaches while mitigating their limitations. PIPCDR incorporates a positive instance proximity loss and a cluster dispersion regularizer. The positive instance proximity loss ensures alignment between augmented views of instances and their sampled neighbors, enhancing within-cluster compactness by selecting genuinely positive pairs within the embedding space. Meanwhile, the cluster dispersion regularizer maximizes inter-cluster distances while minimizing within-cluster compactness, promoting uniformity in the learned representations. PIPCDR excels in producing well-separated clusters, generating uniform representations, avoiding class collision issues, and enhancing within-cluster compactness. We extensively validate the effectiveness of PIPCDR within an end-to-end Majorize-Minimization framework, demonstrating its competitive performance on moderate-scale clustering benchmark datasets and establishing new state-of-the-art results on large-scale datasets.

cs.LG↗
arXiv2023

Player Re-Identification Using Body Part Appearences

We propose a neural network architecture that learns body part appearances for soccer player re-identification. Our model consists of a two-stream network (one stream for appearance map extraction and the other for body part map extraction) and a bilinear-pooling layer that generates and spatially pools the body part map. Each local feature of the body part map is obtained by a bilinear mapping of the corresponding local appearance and body part descriptors. Our novel representation yields a robust image-matching feature map, which results from combining the local similarities of the relevant body parts with the weighted appearance similarity. Our model does not require any part annotation on the SoccerNet-V3 re-identification dataset to train the network. Instead, we use a sub-network of an existing pose estimation network (OpenPose) to initialize the part substream and then train the entire network to minimize the triplet loss. The appearance stream is pre-trained on the ImageNet dataset, and the part stream is trained from scratch for the SoccerNet-V3 dataset. We demonstrate the validity of our model by showing that it outperforms state-of-the-art models such as OsNet and InceptionNet.

cs.CV↗
arXiv2023

Small-scale proxies for large-scale Transformer training instabilities

Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smaller scales. Although the causes of such instabilities are of scientific interest, the amount of resources required to reproduce them has made investigation difficult. In this work, we seek ways to reproduce and study training stability and instability at smaller scales. First, we focus on two sources of training instability described in previous work: the growth of logits in attention layers (Dehghani et al., 2023) and divergence of the output logits from the log probabilities (Chowdhery et al., 2022). By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates, and that mitigations previously employed at large scales are equally effective in this regime. This prompts us to investigate the extent to which other known optimizer and model interventions influence the sensitivity of the final loss to changes in the learning rate. To this end, we study methods such as warm-up, weight decay, and the $μ$Param (Yang et al., 2022), and combine techniques to train small models that achieve similar losses across orders of magnitude of learning rate variation. Finally, to conclude our exploration we study two cases where instabilities can be predicted before they emerge by examining the scaling behavior of model activation and gradient norms.

cs.LG↗
arXiv2023

Robust Unsupervised Domain Adaptation by Retaining Confident Entropy via Edge Concatenation

The generalization capability of unsupervised domain adaptation can mitigate the need for extensive pixel-level annotations to train semantic segmentation networks by training models on synthetic data as a source with computer-generated annotations. Entropy-based adversarial networks are proposed to improve source domain prediction; however, they disregard significant external information, such as edges, which have the potential to identify and distinguish various objects within an image accurately. To address this issue, we introduce a novel approach to domain adaptation, leveraging the synergy of internal and external information within entropy-based adversarial networks. In this approach, we enrich the discriminator network with edge-predicted probability values within this innovative framework to enhance the clarity of class boundaries. Furthermore, we devised a probability-sharing network that integrates diverse information for more effective segmentation. Incorporating object edges addresses a pivotal aspect of unsupervised domain adaptation that has frequently been neglected in the past -- the precise delineation of object boundaries. Conventional unsupervised domain adaptation methods usually center around aligning feature distributions and may not explicitly model object boundaries. Our approach effectively bridges this gap by offering clear guidance on object boundaries, thereby elevating the quality of domain adaptation. Our approach undergoes rigorous evaluation on the established unsupervised domain adaptation benchmarks, specifically in adapting SYNTHIA $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Mapillary. Experimental results show that the proposed model attains better performance than state-of-the-art methods. The superior performance across different unsupervised domain adaptation scenarios highlights the versatility and robustness of the proposed method.

cs.CV↗
arXiv2023

Distributionally Robust Post-hoc Classifiers under Prior Shifts

The generalization ability of machine learning models degrades significantly when the test distribution shifts away from the training distribution. We investigate the problem of training models that are robust to shifts caused by changes in the distribution of class-priors or group-priors. The presence of skewed training priors can often lead to the models overfitting to spurious features. Unlike existing methods, which optimize for either the worst or the average performance over classes or groups, our work is motivated by the need for finer control over the robustness properties of the model. We present an extremely lightweight post-hoc approach that performs scaling adjustments to predictions from a pre-trained model, with the goal of minimizing a distributionally robust loss around a chosen target distribution. These adjustments are computed by solving a constrained optimization problem on a validation set and applied to the model during test time. Our constrained optimization objective is inspired by a natural notion of robustness to controlled distribution shifts. Our method comes with provable guarantees and empirically makes a strong case for distributional robust post-hoc classifiers. An empirical implementation is available at https://github.com/weijiaheng/Drops.

cs.LG↗
arXiv2023

SoK: Distributed Computing in ICN

Information-Centric Networking (ICN), with its data-oriented operation and generally more powerful forwarding layer, provides an attractive platform for distributed computing. This paper provides a systematic overview and categorization of different distributed computing approaches in ICN encompassing fundamental design principles, frameworks and orchestration, protocols, enablers, and applications. We discuss current pain points in legacy distributed computing, attractive ICN features, and how different systems use them. This paper also provides a discussion of potential future work for distributed computing in ICN.

cs.NI↗
arXiv2023

How Can AI be Distributed in the Computing Continuum? Introducing the Neural Pub/Sub Paradigm

This paper proposes the neural publish/subscribe paradigm, a novel approach to orchestrating AI workflows in large-scale distributed AI systems in the computing continuum. Traditional centralized broker methodologies are increasingly struggling with managing the data surge resulting from the proliferation of 5G systems, connected devices, and ultra-reliable applications. Moreover, the advent of AI-powered applications, particularly those leveraging advanced neural network architectures, necessitates a new approach to orchestrate and schedule AI processes within the computing continuum. In response, the neural pub/sub paradigm aims to overcome these limitations by efficiently managing training, fine-tuning and inference workflows, improving distributed computation, facilitating dynamic resource allocation, and enhancing system resilience across the computing continuum. We explore this new paradigm through various design patterns, use cases, and discuss open research questions for further exploration.

cs.NI↗
arXiv2023

RemoteNet: Remote Sensing Image Segmentation Network based on Global-Local Information

Remotely captured images possess an immense scale and object appearance variability due to the complex scene. It becomes challenging to capture the underlying attributes in the global and local context for their segmentation. Existing networks struggle to capture the inherent features due to the cluttered background. To address these issues, we propose a remote sensing image segmentation network, RemoteNet, for semantic segmentation of remote sensing images. We capture the global and local features by leveraging the benefits of the transformer and convolution mechanisms. RemoteNet is an encoder-decoder design that uses multi-scale features. We construct an attention map module to generate channel-wise attention scores for fusing these features. We construct a global-local transformer block (GLTB) in the decoder network to support learning robust representations during a decoding phase. Further, we designed a feature refinement module to refine the fused output of the shallow stage encoder feature and the deepest GLTB feature of the decoder. Experimental findings on the two public datasets show the effectiveness of the proposed RemoteNet.

cs.CV↗
arXiv2023

Learnable Digital Twin for Efficient Wireless Network Evaluation

Network digital twins (NDTs) facilitate the estimation of key performance indicators (KPIs) before physically implementing a network, thereby enabling efficient optimization of the network configuration. In this paper, we propose a learning-based NDT for network simulators. The proposed method offers a holistic representation of information flow in a wireless network by integrating node, edge, and path embeddings. Through this approach, the model is trained to map the network configuration to KPIs in a single forward pass. Hence, it offers a more efficient alternative to traditional simulation-based methods, thus allowing for rapid experimentation and optimization. Our proposed method has been extensively tested through comprehensive experimentation in various scenarios, including wired and wireless networks. Results show that it outperforms baseline learning models in terms of accuracy and robustness. Moreover, our approach achieves comparable performance to simulators but with significantly higher computational efficiency.

cs.NI↗
arXiv2023

Score-based Causal Representation Learning with Interventions

This paper studies the causal representation learning problem when the latent causal variables are observed indirectly through an unknown linear transformation. The objectives are: (i) recovering the unknown linear transformation (up to scaling) and (ii) determining the directed acyclic graph (DAG) underlying the latent variables. Sufficient conditions for DAG recovery are established, and it is shown that a large class of non-linear models in the latent space (e.g., causal mechanisms parameterized by two-layer neural networks) satisfy these conditions. These sufficient conditions ensure that the effect of an intervention can be detected correctly from changes in the score. Capitalizing on this property, recovering a valid transformation is facilitated by the following key property: any valid transformation renders latent variables' score function to necessarily have the minimal variations across different interventional environments. This property is leveraged for perfect recovery of the latent DAG structure using only \emph{soft} interventions. For the special case of stochastic \emph{hard} interventions, with an additional hypothesis testing step, one can also uniquely recover the linear transformation up to scaling and a valid causal ordering.

stat.ML↗
arXiv2023

Odd symmetry planar Hall effect: A method of detecting current-induced in-plane magnetization switching

Type-x device attracts considerable interest in the field of spintronics due to its robust spin-orbit torque (SOT) induced magnetization switching, and easy deposition technique. However, universally applicable and straightforward detection of type-X magnetization reversal is still elusive, unlike type-Z switching, which employs DC-based anomalous Hall effect measurement. Here, we, demonstrated that the odd planar Hall signal (O-PHV) exhibits an odd symmetry with the application of an external magnetic field which motivates us to develop a reading mechanism for detecting magnetization switching of in-plane magnetized type-X devices. We verified our DC-based reading mechanism in the Pt/Co/NiFe/Pt stack where a thin Co layer is inserted to create dissimilar interfaces about the NiFe layer. Remarkably, the current-induced in-plane fields are found to be significantly large in Pt/Co/NiFe/Pt stack. Further, we successfully employed the O-PHV method to detect the current-induced magnetization switching. The pure DC nature of the writing and reading mechanism of our proposed type-X detection technique through O-PHV makes it the easiest in-plane magnetization detection technique. Moreover, the high repeatability and easy detection of our proposed method will open new avenues toward in-plane SOT switching based memory devices and sensors.

cond-mat.mes-hall↗
arXiv2023

Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestration

Future AI applications require performance, reliability and privacy that the existing, cloud-dependant system architectures cannot provide. In this article, we study orchestration in the device-edge-cloud continuum, and focus on edge AI for resource orchestration. We claim that to support the constantly growing requirements of intelligent applications in the device-edge-cloud computing continuum, resource orchestration needs to embrace edge AI and emphasize local autonomy and intelligence. To justify the claim, we provide a general definition for continuum orchestration, and look at how current and emerging orchestration paradigms are suitable for the computing continuum. We describe certain major emerging research themes that may affect future orchestration, and provide an early vision of an orchestration paradigm that embraces those research themes. Finally, we survey current key edge AI methods and look at how they may contribute into fulfilling the vision of future continuum orchestration.

cs.MA↗