arXiv subjects

Jie Luo

Publications and source records attributed to Jie Luo.

At least 73 records · Page 4Linked to original sources

arXiv2024

TinyLLaVA: A Framework of Small-scale Large Multimodal Models

We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public.

cs.LG↗

arXiv2024

Ambient-Pix2PixGAN for Translating Medical Images from Noisy Data

Image-to-image translation is a common task in computer vision and has been rapidly increasing the impact on the field of medical imaging. Deep learning-based methods that employ conditional generative adversarial networks (cGANs), such as Pix2PixGAN, have been extensively explored to perform image-to-image translation tasks. However, when noisy medical image data are considered, such methods cannot be directly applied to produce clean images. Recently, an augmented GAN architecture named AmbientGAN has been proposed that can be trained on noisy measurement data to synthesize high-quality clean medical images. Inspired by AmbientGAN, in this work, we propose a new cGAN architecture, Ambient-Pix2PixGAN, for performing medical image-to-image translation tasks by use of noisy measurement data. Numerical studies that consider MRI-to-PET translation are conducted. Both traditional image quality metrics and task-based image quality metrics are employed to assess the proposed Ambient-Pix2PixGAN. It is demonstrated that our proposed Ambient-Pix2PixGAN can be successfully trained on noisy measurement data to produce high-quality translated images in target imaging modality.

eess.IV↗

arXiv2024

Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization

[$^{18}$F]fluorodeoxyglucose (FDG) positron emission tomography (PET) has emerged as a crucial tool in identifying the epileptic focus, especially in cases where magnetic resonance imaging (MRI) diagnosis yields indeterminate results. FDG PET can provide the metabolic information of glucose and help identify abnormal areas that are not easily found through MRI. However, the effectiveness of FDG PET-based assessment and diagnosis depends on the selection of a healthy control group. The healthy control group typically consists of healthy individuals similar to epilepsy patients in terms of age, gender, and other aspects for providing normal FDG PET data, which will be used as a reference for enhancing the accuracy and reliability of the epilepsy diagnosis. However, significant challenges arise when a healthy PET control group is unattainable. Yaakub \emph{et al.} have previously introduced a Pix2PixGAN-based method for MRI to PET translation. This method used paired MRI and FDG PET scans from healthy individuals for training, and produced pseudo normal FDG PET images from patient MRIs that are subsequently used for lesion detection. However, this approach requires a large amount of high-quality, paired MRI and PET images from healthy control subjects, which may not always be available. In this study, we investigated unsupervised learning methods for unpaired MRI to PET translation for generating pseudo normal FDG PET for epileptic focus localization. Two deep learning methods, CycleGAN and SynDiff, were employed, and we found that diffusion-based method achieved improved performance in accurately localizing the epileptic focus.

cs.CV↗

arXiv2024

Modulate Your Spectrum in Self-Supervised Learning

Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL.

cs.LG↗

arXiv2023

Broadband CPW-based impedance-transformed Josephson parametric amplifier

Quantum-limited Josephson parametric amplifiers play a pivotal role in advancing the field of circuit quantum electrodynamics by enabling the fast and high-fidelity measurement of weak microwave signals. Therefore, it is necessary to develop robust parametric amplifiers with low noise, broad bandwidth, and reduced design complexity for microwave detection. However, current broadband parametric amplifiers either have degraded noise performance or rely on complex designs. Here, we present a device based on the broadband impedance-transformed Josephson parametric amplifier (IMPA) that integrates a horn-like coplanar waveguide (CPW) transmission line, which significantly decreases the design and fabrication complexity, while keeping comparable performance. The device shows an instantaneous bandwidth of 700(200) MHz for 15(20) dB gain with an average saturation power of -110 dBm and near quantum-limited added noise. The operating frequency can be tuned over 1.4 GHz using an external flux bias. We further demonstrate the negligible back-action from our device on a transmon qubit. The amplification performance and simplicity of our device promise its wide adaptation in quantum metrology, quantum communication, and quantum information processing.

quant-ph↗

arXiv2023

Testing Lorentz symmetry with space-based gravitational-wave detectors

Lorentz symmetry (LS), one of the most fundamental physical symmetries, has been extensively studied in the context of quantum gravity and unification theories. Many of these theories predict a LS violation, which could arise from the discreteness of spacetime, or extra dimensions. Standard-model extension (SME) is an effective field theory to describe Lorentz violation whose effects can be explored using precision instruments such as atomic clocks and gravitational-wave (GW) detectors. Considering the pure-gravity sector and matter-gravity coupling sector in the SME, we studied the leading Lorentz-violating modifications to the time delay of light and the relativistic frequency shift of the clock in the space-based GW detectors. We found that the six data streams from the GW mission can construct various combinations of measurement signals, such as single-arm round-trip path, interference path, triangular round-trip path, etc. These measurements are sensitive to the different combinations of SME coefficients and provide novel linear combinations of SME coefficients different from previous studies. Based on the orbits of TianQin, LISA, and Taiji missions, we calculated the response of Lorentz-violating effects on the combinations of the measurement signal data streams. Our results allow us to estimate the sensitivities for SME coefficients: $10^{-6}$ for the gravity sector coefficient $\bar{s}^{TT}$, $10^{-6}$ for matter-gravity coupling coefficients $(\bar{a}^{(e+p)}_{\text{eff}})_{T}$ and $\bar{c}^{(e+p)}_{TT}$, and $10^{-5}$ for $(\bar{a}^{n}_{\text{eff}})_{T}$ and $\bar{c}^{n}_{TT}$.

gr-qc↗

arXiv2023

A comparison of controller architectures and learning mechanisms for arbitrary robot morphologies

The main question this paper addresses is: What combination of a robot controller and a learning method should be used, if the morphology of the learning robot is not known in advance? Our interest is rooted in the context of morphologically evolving modular robots, but the question is also relevant in general, for system designers interested in widely applicable solutions. We perform an experimental comparison of three controller-and-learner combinations: one approach where controllers are based on modelling animal locomotion (Central Pattern Generators, CPG) and the learner is an evolutionary algorithm, a completely different method using Reinforcement Learning (RL) with a neural network controller architecture, and a combination `in-between' where controllers are neural networks and the learner is an evolutionary algorithm. We apply these three combinations to a test suite of modular robots and compare their efficacy, efficiency, and robustness. Surprisingly, the usual CPG-based and RL-based options are outperformed by the in-between combination that is more robust and efficient than the other two setups.

cs.RO↗

arXiv2023

Exploring Robot Morphology Spaces through Breadth-First Search and Random Query

Evolutionary robotics offers a powerful framework for designing and evolving robot morphologies, particularly in the context of modular robots. However, the role of query mechanisms during the genotype-to-phenotype mapping process has been largely overlooked. This research addresses this gap by conducting a comparative analysis of query mechanisms in the brain-body co-evolution of modular robots. Using two different query mechanisms, Breadth-First Search (BFS) and Random Query, within the context of evolving robot morphologies using CPPNs and robot controllers using tensors, and testing them in two evolutionary frameworks, Lamarckian and Darwinian systems, this study investigates their influence on evolutionary outcomes and performance. The findings demonstrate the impact of the two query mechanisms on the evolution and performance of modular robot bodies, including morphological intelligence, diversity, and morphological traits. This study suggests that BFS is both more effective and efficient in producing highly performing robots. It also reveals that initially, robot diversity was higher with BFS compared to Random Query, but in the Lamarckian system, it declines faster, converging to superior designs, while in the Darwinian system, BFS led to higher end-process diversity.

cs.RO↗

arXiv2023

Lamarck's Revenge: Inheritance of Learned Traits Can Make Robot Evolution Better

Evolutionary robot systems offer two principal advantages: an advanced way of developing robots through evolutionary optimization and a special research platform to conduct what-if experiments regarding questions about evolution. Our study sits at the intersection of these. We investigate the question ``What if the 18th-century biologist Lamarck was not completely wrong and individual traits learned during a lifetime could be passed on to offspring through inheritance?'' We research this issue through simulations with an evolutionary robot framework where morphologies (bodies) and controllers (brains) of robots are evolvable and robots also can improve their controllers through learning during their lifetime. Within this framework, we compare a Lamarckian system, where learned bits of the brain are inheritable, with a Darwinian system, where they are not. Analyzing simulations based on these systems, we obtain new insights about Lamarckian evolution dynamics and the interaction between evolution and learning. Specifically, we show that Lamarckism amplifies the emergence of `morphological intelligence', the ability of a given robot body to acquire a good brain by learning, and identify the source of this success: `newborn' robots have a higher fitness because their inherited brains match their bodies better than those in a Darwinian system.

cs.RO↗

arXiv2023

Radiology-Llama2: Best-in-Class Large Language Model for Radiology

This paper introduces Radiology-Llama2, a large language model specialized for radiology through a process known as instruction tuning. Radiology-Llama2 is based on the Llama2 architecture and further trained on a large dataset of radiology reports to generate coherent and clinically useful impressions from radiological findings. Quantitative evaluations using ROUGE metrics on the MIMIC-CXR and OpenI datasets demonstrate that Radiology-Llama2 achieves state-of-the-art performance compared to other generative language models, with a Rouge-1 score of 0.4834 on MIMIC-CXR and 0.4185 on OpenI. Additional assessments by radiology experts highlight the model's strengths in understandability, coherence, relevance, conciseness, and clinical utility. The work illustrates the potential of localized language models designed and tuned for specialized domains like radiology. When properly evaluated and deployed, such models can transform fields like radiology by automating rote tasks and enhancing human expertise.

cs.CL↗

arXiv2023

ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has achieved promising results in recent years. However, most existing reinforcement learning methods require a large amount of data for model training. In addition, data-efficient reinforcement learning requires the construction of strong inductive biases, which are ignored in the current MARL approaches. Inspired by the symmetry phenomenon in multi-agent systems, this paper proposes a framework for exploiting prior knowledge by integrating data augmentation and a well-designed consistency loss into the existing MARL methods. In addition, the proposed framework is model-agnostic and can be applied to most of the current MARL algorithms. Experimental tests on multiple challenging tasks demonstrate the effectiveness of the proposed framework. Moreover, the proposed framework is applied to a physical multi-robot testbed to show its superiority.

cs.MA↗

arXiv2023

A Comparative Study of Brain Reproduction Methods for Morphologically Evolving Robots

In the most extensive robot evolution systems, both the bodies and the brains of the robots undergo evolution and the brains of 'infant' robots are also optimized by a learning process immediately after 'birth'. This paper is concerned with the brain evolution mechanism in such a system. In particular, we compare four options obtained by combining asexual or sexual brain reproduction with Darwinian or Lamarckian evolution mechanisms. We conduct experiments in simulation with a system of evolvable modular robots on two different tasks. The results show that sexual reproduction of the robots' brains is preferable in the Darwinian framework, but the effect is the opposite in the Lamarckian system (both using the same infant learning method). Our experiments suggest that the overall best option is asexual reproduction combined with the Lamarckian framework, as it obtains better robots in terms of fitness than the other three. Considering the evolved morphologies, the different brain reproduction methods do not lead to differences. This result indicates that the morphology of the robot is mainly determined by the task and the environment, not by the brain reproduction methods.

cs.RO↗

arXiv2023

Dynamically Reconfigurable Photon Exchange in a Superconducting Quantum Processor

Realizing the advantages of quantum computation requires access to the full Hilbert space of states of many quantum bits (qubits). Thus, large-scale quantum computation faces the challenge of efficiently generating entanglement between many qubits. In systems with a limited number of direct connections between qubits, entanglement between non-nearest neighbor qubits is generated by a series of nearest neighbor gates, which exponentially suppresses the resulting fidelity. Here we propose and demonstrate a novel, on-chip photon exchange network. This photonic network is embedded in a superconducting quantum processor (QPU) to implement an arbitrarily reconfigurable qubit connectivity graph. We show long-range qubit-qubit interactions between qubits with a maximum spatial separation of $9.2~\text{cm}$ along a meandered bus resonator and achieve photon exchange rates up to $g_{\text{qq}} = 2π\times 0.9~\text{MHz}$. These experimental demonstrations provide a foundation to realize highly connected, reconfigurable quantum photonic networks and opens a new path towards modular quantum computing.

quant-ph↗

arXiv2023

Electrostatic effect due to patch potentials between closely spaced surfaces

The spatial variation and temporal variation in surface potential are important error sources in various precision experiments and deserved to be considered carefully. In the former case, the theoretical analysis shows that this effect depends on the surface potentials through their spatial autocorrelation functions. By making some modification to the quasi-local correlation model, we obtain a rigorous formula for the patch force, where the magnitude is proportional to ${\frac{1}{{{a}^{2}}}{{(\frac{a}{w})}^{β(a/w)+2}}}$ with ${a}$ the distance between two parallel plates, ${w}$ the mean patch size, and $β$ the scaling coefficient from ${-2}$ to ${-4}$. A torsion balance experiment is then conducted, and obtain a 0.4 mm effective patch size and 20 mV potential variance. In the latter case, we apply an adatom diffusion model to describe this mechanism and predicts a ${f^{-3/4}}$ frequency dependence above 0.01 ${\rm mHz}$. This prediction meets well with a typical experimental results. Finally, we apply these models to analyze the patch effect for two typical experiments. Our analysis will help to investigate the properties of surface potentials.

cond-mat.soft↗

arXiv2023

BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance

Deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications while suffering expensive computation and storage. Therefore, network compression technologies like binarization are studied to deploy KWS models on edge. In this paper, we present a strong yet efficient binary neural network for KWS, namely BiFSMNv2, pushing it to the real-network accuracy performance. First, we present a Dual-scale Thinnable 1-bit-Architecture to recover the representation capability of the binarized computation units by dual-scale activation binarization and liberate the speedup potential from an overall architecture perspective. Second, we also construct a Frequency Independent Distillation scheme for KWS binarization-aware training, which distills the high and low-frequency components independently to mitigate the information mismatch between full-precision and binarized representations. Moreover, we propose the Learning Propagation Binarizer, a general and efficient binarizer that enables the forward and backward propagation of binary KWS networks to be continuously improved through learning. We implement and deploy the BiFSMNv2 on ARMv8 real-world hardware with a novel Fast Bitwise Computation Kernel, which is proposed to fully utilize registers and increase instruction throughput. Comprehensive experiments show our BiFSMNv2 outperforms existing binary networks for KWS by convincing margins across different datasets and achieves comparable accuracy with the full-precision networks (only a tiny 1.51% drop on Speech Commands V1-12). We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1x speedup and 20.2x storage-saving on edge hardware.

cs.CL↗

arXiv2022

BiFSMN: Binary Neural Network for Keyword Spotting

The deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. However, computational resources for these networks are significantly constrained since they usually run on-call on edge devices. In this paper, we present BiFSMN, an accurate and extreme-efficient binary neural network for KWS. We first construct a High-frequency Enhancement Distillation scheme for the binarization-aware training, which emphasizes the high-frequency information from the full-precision network's representation that is more crucial for the optimization of the binarized network. Then, to allow the instant and adaptive accuracy-efficiency trade-offs at runtime, we also propose a Thinnable Binarization Architecture to further liberate the acceleration potential of the binarized network from the topology perspective. Moreover, we implement a Fast Bitwise Computation Kernel for BiFSMN on ARMv8 devices which fully utilizes registers and increases instruction throughput to push the limit of deployment efficiency. Extensive experiments show that BiFSMN outperforms existing binarization methods by convincing margins on various datasets and is even comparable with the full-precision counterpart (e.g., less than 3% drop on Speech Commands V1-12). We highlight that benefiting from the thinnable architecture and the optimized 1-bit implementation, BiFSMN can achieve an impressive 22.3x speedup and 15.5x storage-saving on real-world edge hardware. Our code is released at https://github.com/htqin/BiFSMN.

cs.CL↗

arXiv2022

Environment induced emergence of collective behaviour in evolving swarms with limited sensing

Designing controllers for robot swarms is challenging, because human developers have typically no good understanding of the link between the details of a controller that governs individual robots and the swarm behavior that is an indirect result of the interactions between swarm members and the environment. In this paper we investigate whether an evolutionary approach can mitigate this problem. We consider a very challenging task where robots with limited sensing and communication abilities must follow the gradient of an environmental feature and use Differential Evolution to evolve a neural network controller for simulated robots. We conduct a systematic study to measure the flexibility and scalability of the method by varying the size of the arena and number of robots in the swarm. The experiments confirm the feasibility of our approach, the evolved robot controllers induced swarm behavior that solved the task. We found that solutions evolved under the harshest conditions (where the environmental clues were the weakest) were the most flexible and that there is a sweet spot regarding the swarm size. Furthermore, we observed collective motion of the swarm, showcasing truly emergent behavior that was not represented in- and selected for during evolution.

cs.RO↗

arXiv2022

Delving into the Estimation Shift of Batch Normalization in a Network

Batch normalization (BN) is a milestone technique in deep learning. It normalizes the activation using mini-batch statistics during training but the estimated population statistics during inference. This paper focuses on investigating the estimation of population statistics. We define the estimation shift magnitude of BN to quantitatively measure the difference between its estimated population statistics and expected ones. Our primary observation is that the estimation shift can be accumulated due to the stack of BN in a network, which has detriment effects for the test performance. We further find a batch-free normalization (BFN) can block such an accumulation of estimation shift. These observations motivate our design of XBNBlock that replace one BN with BFN in the bottleneck block of residual-style networks. Experiments on the ImageNet and COCO benchmarks show that XBNBlock consistently improves the performance of different architectures, including ResNet and ResNeXt, by a significant margin and seems to be more robust to distribution shift.

cs.CV↗