arXiv subjects

Lin Luo

Publications and source records attributed to Lin Luo.

34 records · Page 2Linked to original sources

arXiv2024

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

The advancement of large Vision-Language-Action (VLA) models has significantly improved robotic manipulation in terms of language-guided task execution and generalization to unseen scenarios. While existing VLAs adapted from pretrained large Vision-Language-Models (VLM) have demonstrated promising generalizability, their task performance is still unsatisfactory as indicated by the low tasks success rates in different environments. In this paper, we present a new advanced VLA architecture derived from VLM. Unlike previous works that directly repurpose VLM for action prediction by simple action quantization, we propose a omponentized VLA architecture that has a specialized action module conditioned on VLM output. We systematically study the design of the action module and demonstrates the strong performance enhancement with diffusion action transformers for action sequence modeling, as well as their favorable scaling behaviors. We also conduct comprehensive experiments and ablation studies to evaluate the efficacy of our models with varied designs. The evaluation on 5 robot embodiments in simulation and real work shows that our model not only significantly surpasses existing VLAs in task performance and but also exhibits remarkable adaptation to new robots and generalization to unseen objects and backgrounds. It exceeds the average success rates of OpenVLA which has similar model size (7B) with ours by over 35% in simulated evaluation and 55% in real robot experiments. It also outperforms the large RT-2-X model (55B) by 18% absolute success rates in simulation. Code and models can be found on our project page (https://cogact.github.io/).

cs.RO↗

arXiv2024

Human-in-the-loop Fairness: Integrating Stakeholder Feedback to Incorporate Fairness Perspectives in Responsible AI

Fairness is a growing concern for high-risk decision-making using Artificial Intelligence (AI) but ensuring it through purely technical means is challenging: there is no universally accepted fairness measure, fairness is context-dependent, and there might be conflicting perspectives on what is considered fair. Thus, involving stakeholders, often without a background in AI or fairness, is a promising avenue. Research to directly involve stakeholders is in its infancy, and many questions remain on how to support stakeholders to feedback on fairness, and how this feedback can be integrated into AI models. Our work follows an approach where stakeholders can give feedback on specific decision instances and their outcomes with respect to their fairness, and then to retrain an AI model. In order to investigate this approach, we conducted two studies of a complex AI model for credit rating used in loan applications. In study 1, we collected feedback from 58 lay users on loan application decisions, and conducted offline experiments to investigate the effects on accuracy and fairness metrics. In study 2, we deepened this investigation by showing 66 participants the results of their feedback with respect to fairness, and then conducted further offline analyses. Our work contributes two datasets and associated code frameworks to bootstrap further research, highlights the opportunities and challenges of employing lay user feedback for improving AI fairness, and discusses practical implications for developing AI applications that more closely reflect stakeholder views about fairness.

cs.AI↗

arXiv2024

Generalized Skyrmions

Skyrmions are important topologically non-trivial fields characteristic of models spanning scales from the microscopic to the cosmological. However, the Skyrmion number can only be defined for fields with specific boundary conditions, limiting its use in broader contexts. Here, we address this issue through a generalized notion of the Skyrmion derived from the De Rham cohomology of compactly supported forms. This allows for the definition of an entirely new $\coprod_{i=1}^\infty \mathbb{Z}^i$-valued topological number that assigns a tuple of integers $(a_1, \ldots, a_k)\in \mathbb{Z}^k$ to a field instead of a single number, with no restrictions to its boundary. The notion of the generalized Skyrmion presented in this paper is completely abstract and can be applied to vector fields in any discipline, not unlike index theory within dynamical systems. To demonstrate the power of our new formalism, we focus on the propagation of optical polarization fields and show that our newly defined generalized Skyrmion number significantly increases the dimension of data that can be stored within the field while also demonstrating strong robustness. Our work represents a fundamental paradigm shift away from the study of fields with natural topological character to engineered fields that can be artificially embedded with topological structures.

physics.optics↗

arXiv2024

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

Pathological diagnosis remains the definitive standard for identifying tumors. The rise of multimodal large models has simplified the process of integrating image analysis with textual descriptions. Despite this advancement, the substantial costs associated with training and deploying these complex multimodal models, together with a scarcity of high-quality training datasets, create a significant divide between cutting-edge technology and its application in the clinical setting. We had meticulously compiled a dataset of approximately 45,000 cases, covering over 6 different tasks, including the classification of organ tissues, generating pathology report descriptions, and addressing pathology-related questions and answers. We have fine-tuned multimodal large models, specifically LLaVA, Qwen-VL, InternLM, with this dataset to enhance instruction-based performance. We conducted a qualitative assessment of the capabilities of the base model and the fine-tuned model in performing image captioning and classification tasks on the specific dataset. The evaluation results demonstrate that the fine-tuned model exhibits proficiency in addressing typical pathological questions. We hope that by making both our models and datasets publicly available, they can be valuable to the medical and research communities.

cs.CV↗

arXiv2024

Topological Protection of Optical Skyrmions through Complex Media

Optical Skyrmions have many important properties that make them ideal units for high-density data applications, including the ability to carry digital information through a discrete topological number and the independence of spatially varying polarization to other dimensions. More importantly, the topological nature of the optical Skyrmion heuristically suggests a strong degree of robustness to perturbations, which is crucial for reliably carrying information in noisy environments. However, the study of the topological robustness of optical Skyrmions is still in its infancy. Here, we quantify this robustness precisely by proving that the topological nature of the Skyrmion arises from its structure on the boundary and, by duality, is therefore resilient to complex perturbations provided they respect the relevant boundary conditions of the unperturbed Skyrmion. We then present experimental evidence validating this robustness in the context of paraxial Skyrmion beams against different polarization aberrations. Our work provides a framework for handling various perturbations of Skyrmion fields and offers guarantees of robustness in a general sense. This, in turn, has implications for applications of the optical Skyrmion where their topological nature is exploited explicitly, and, in particular, provides an underpinning for the use of Skyrmions in optical communications and photonic computing.

physics.optics↗

arXiv2024

Rethinking Generative Large Language Model Evaluation for Semantic Comprehension

Despite their sophisticated capabilities, large language models (LLMs) encounter a major hurdle in effective assessment. This paper first revisits the prevalent evaluation method-multiple choice question answering (MCQA), which allows for straightforward accuracy measurement. Through a comprehensive evaluation of 24 models across 11 benchmarks, we highlight several potential drawbacks of MCQA, for instance, the inconsistency between the MCQA evaluation and the generation of open-ended responses in practical scenarios. In response, we introduce an RWQ-Elo rating system, engaging 24 LLMs such as GPT-4, GPT-3.5, Google-Gemini-Pro and LLaMA-1/-2, in a two-player competitive format, with GPT-4 serving as the judge. Each LLM receives an Elo rating thereafter. This system is designed to mirror real-world usage, and for this purpose, we have compiled a new benchmark called ``Real-world questions'' (RWQ), comprising 20,772 authentic user inquiries. Additionally, we thoroughly analyze the characteristics of our system and compare it with prior leaderboards like AlpacaEval and MT-Bench. Our analysis reveals the stability of our RWQ-Elo system, the feasibility of registering new models, and its potential to reshape LLM leaderboards.

cs.CL↗

arXiv2023

Giant magnetocaloric effect and hysteresis loss in Mn$_x$Fe$_{2-x}$P$_{0.5}$Si$_{0.5}$ ($x$ = 0.7-1.2) microwires at ambient temperatures

Magnetocaloric microwires are very promising for energy-efficient magnetic refrigeration in micro electromechanical systems (MEMS) and nano electromechanical systems (NEMS). Creating microwires that exhibit large magnetocaloric effects around room temperature represents an important but challenging task. Here, we report a tunable giant magnetocaloric effect around room temperature in Mn$_x$Fe$_{2-x}$P$_{0.5}$Si$_{0.5}$ ($x$ = 0.7-1.2) microwires by utilizing a melt-extraction technique paired with thermal treatment and chemical engineering. The isothermal magnetic entropy change DeltaSiso and Curie temperature (TC) can be tuned by adjusting the Mn/Fe ratio. The TC varies from 351 to 190 K as x increases from 0.8 to 1.2. Among the compositions investigated, the x = 0.9 sample shows the largest value of DeltaSiso = 18.3 J kg$^{-1}$ K$^{-1}$ for a field change of 5 T around 300 K. After subtracting magnetic hysteresis loss, a large refrigerant capacity of ~284.6 J kg$^{-1}$ is achieved. Our study paves a new pathway for the design of novel magnetocaloric microwires for active magnetic refrigeration at ambient temperatures.

physics.app-ph↗

arXiv2023

Advanced magnetocaloric microwires: What does the future hold?

Magnetic refrigeration (MR) based on the magnetocaloric effect (MCE) is a promising alternative to conventional vapor compression refrigeration techniques. The cooling efficiency of a magnetic refrigerator depends on its refrigeration capacity and operation frequency. Existing refrigerators possess limited cooling efficiency due to the low operating frequency (around tens of Hz). Theory predicts that reducing geometrical effects can increase the operation frequency by reducing the relaxation time of a magnetic material. As compared to other shapes, magnetocaloric wires transfer heat most effectively to a surrounding environment, due to their enhanced surface area. The wire shape also yields a good mechanical response, reducing the relaxation time and consequently increasing the operation frequency of the cooling device. Experiments have validated the theoretical predictions. By assembling microwires with different magnetocaloric properties and Curie temperatures into a laminate structure, a table-like magnetocaloric bed can be created and used as an active cooling device for micro-electro-mechanical system (MEMS) and nano-electro-mechanical system (NEMS). This paper assesses recent progress in the development of magnetocaloric microwires and sheds light on the important factors affecting the magnetocaloric behavior and cooling efficiency in microwire systems. Challenges, opportunities, and strategies regarding the development of advanced magnetocaloric microwires are also discussed.

physics.app-ph↗

arXiv2023

SCAAT: Improving Neural Network Interpretability via Saliency Constrained Adaptive Adversarial Training

Deep Neural Networks (DNNs) are expected to provide explanation for users to understand their black-box predictions. Saliency map is a common form of explanation illustrating the heatmap of feature attributions, but it suffers from noise in distinguishing important features. In this paper, we propose a model-agnostic learning method called Saliency Constrained Adaptive Adversarial Training (SCAAT) to improve the quality of such DNN interpretability. By constructing adversarial samples under the guidance of saliency map, SCAAT effectively eliminates most noise and makes saliency maps sparser and more faithful without any modification to the model architecture. We apply SCAAT to multiple DNNs and evaluate the quality of the generated saliency maps on various natural and pathological image datasets. Evaluations on different domains and metrics show that SCAAT significantly improves the interpretability of DNNs by providing more faithful saliency maps without sacrificing their predictive power.

cs.CV↗

arXiv2023

Assessing and Enhancing Robustness of Deep Learning Models with Corruption Emulation in Digital Pathology

Deep learning in digital pathology brings intelligence and automation as substantial enhancements to pathological analysis, the gold standard of clinical diagnosis. However, multiple steps from tissue preparation to slide imaging introduce various image corruptions, making it difficult for deep neural network (DNN) models to achieve stable diagnostic results for clinical use. In order to assess and further enhance the robustness of the models, we analyze the physical causes of the full-stack corruptions throughout the pathological life-cycle and propose an Omni-Corruption Emulation (OmniCE) method to reproduce 21 types of corruptions quantified with 5-level severity. We then construct three OmniCE-corrupted benchmark datasets at both patch level and slide level and assess the robustness of popular DNNs in classification and segmentation tasks. Further, we explore to use the OmniCE-corrupted datasets as augmentation data for training and experiments to verify that the generalization ability of the models has been significantly enhanced.

eess.IV↗

arXiv2023

What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning

Pathological captioning of Whole Slide Images (WSIs), though is essential in computer-aided pathological diagnosis, has rarely been studied due to the limitations in datasets and model training efficacy. In this paper, we propose a new paradigm Subtype-guided Masked Transformer (SGMT) for pathological captioning based on Transformers, which treats a WSI as a sequence of sparse patches and generates an overall caption sentence from the sequence. An accompanying subtype prediction is introduced into SGMT to guide the training process and enhance the captioning accuracy. We also present an Asymmetric Masked Mechansim approach to tackle the large size constraint of pathological image captioning, where the numbers of sequencing patches in SGMT are sampled differently in the training and inferring phases, respectively. Experiments on the PatchGastricADC22 dataset demonstrate that our approach effectively adapts to the task with a transformer-based model and achieves superior performance than traditional RNN-based methods. Our codes are to be made available for further research and development.

cs.CV↗

arXiv2023

AGMDT: Virtual Staining of Renal Histology Images with Adjacency-Guided Multi-Domain Transfer

Renal pathology, as the gold standard of kidney disease diagnosis, requires doctors to analyze a series of tissue slices stained by H&E staining and special staining like Masson, PASM, and PAS, respectively. These special staining methods are costly, time-consuming, and hard to standardize for wide use especially in primary hospitals. Advances of supervised learning methods have enabled the virtually conversion of H&E images into special staining images, but achieving pixel-to-pixel alignment for training remains challenging. In contrast, unsupervised learning methods regarding different stains as different style transfer domains can utilize unpaired data, but they ignore the spatial inter-domain correlations and thus decrease the trustworthiness of structural details for diagnosis. In this paper, we propose a novel virtual staining framework AGMDT to translate images into other domains by avoiding pixel-level alignment and meanwhile utilizing the correlations among adjacent tissue slices. We first build a high-quality multi-domain renal histological dataset where each specimen case comprises a series of slices stained in various ways. Based on it, the proposed framework AGMDT discovers patch-level aligned pairs across the serial slices of multi-domains through glomerulus detection and bipartite graph matching, and utilizes such correlations to supervise the end-to-end model for multi-domain staining transformation. Experimental results show that the proposed AGMDT achieves a good balance between the precise pixel-level alignment and unpaired domain transfer by exploiting correlations across multi-domain serial pathological slices, and outperforms the state-of-the-art methods in both quantitative measure and morphological details.

eess.IV↗

arXiv2023

Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation

Continual Test-Time Adaptation (CTTA) aims to adapt the source model to continually changing unlabeled target domains without access to the source data. Existing methods mainly focus on model-based adaptation in a self-training manner, such as predicting pseudo labels for new domain datasets. Since pseudo labels are noisy and unreliable, these methods suffer from catastrophic forgetting and error accumulation when dealing with dynamic data distributions. Motivated by the prompt learning in NLP, in this paper, we propose to learn an image-level visual domain prompt for target domains while having the source model parameters frozen. During testing, the changing target datasets can be adapted to the source model by reformulating the input data with the learned visual prompts. Specifically, we devise two types of prompts, i.e., domains-specific prompts and domains-agnostic prompts, to extract current domain knowledge and maintain the domain-shared knowledge in the continual adaptation. Furthermore, we design a homeostasis-based prompt adaptation strategy to suppress domain-sensitive parameters in domain-invariant prompts to learn domain-shared knowledge more effectively. This transition from the model-dependent paradigm to the model-free one enables us to bypass the catastrophic forgetting and error accumulation problems. Experiments show that our proposed method achieves significant performance gains over state-of-the-art methods on four widely-used benchmarks, including CIFAR-10C, CIFAR-100C, ImageNet-C, and VLCS datasets.

cs.CV↗

arXiv2022

AnoDFDNet: A Deep Feature Difference Network for Anomaly Detection

This paper proposed a novel anomaly detection (AD) approach of High-speed Train images based on convolutional neural networks and the Vision Transformer. Different from previous AD works, in which anomalies are identified with a single image using classification, segmentation, or object detection methods, the proposed method detects abnormal difference between two images taken at different times of the same region. In other words, we cast anomaly detection problem with a single image into a difference detection problem with two images. The core idea of the proposed method is that the 'anomaly' usually represents an abnormal state instead of a specific object, and this state should be identified by a pair of images. In addition, we introduced a deep feature difference AD network (AnoDFDNet) which sufficiently explored the potential of the Vision Transformer and convolutional neural networks. To verify the effectiveness of the proposed AnoDFDNet, we collected three datasets, a difference dataset (Diff Dataset), a foreign body dataset (FB Dataset), and an oil leakage dataset (OL Dataset). Experimental results on above datasets demonstrate the superiority of proposed method. Source code are available at https://github.com/wangle53/AnoDFDNet.

cs.CV↗

arXiv2021

Magnetic and magnetocaloric properties of melt-extracted Mn1.26Fe0.60P0.48Si0.52 microwires

The polycrystalline Mn1.26Fe0.60P0.48Si0.52 microwires were successfully fabricated for the first time by the melt-extraction technique, and their magnetic and magnetocaloric properties were investigated systematically. The structural analysis shows that the microwires possess a hexagonal phase with Fe2P type, with a homogeneous composition distribution. Magnetometry measurements show that the microwires undergo a weak first-order magnetic phase transition at a temperature of 142 K. The maximum magnetic entropy change of the microwires reaches 4.64 Jkg-1K-1 for a field change of 5 T. These low-cost Mn1.26Fe0.60P0.48Si0.52 microwires are promising for active magnetic refrigeration in the liquid nitrogen temperature range.

physics.app-ph↗

arXiv2021

A perspective magnetic bed comprising Gd alloy multi-microwires for energy-efficient magnetic refrigeration

We have designed a new magnetic bed structure with desirable table-like magnetocaloric effect (MCE) by using three kinds of soft ferromagnetic Gd-Al-Co microwire arrays with different Curie temperatures ($T_C$). The $T_C$ interval of these three wires is ~10 K and the designed new structure named Sample A. This sample shows a smooth table-like magnetic entropy change ($ΔS_M$) at high applied field change ($μ_0 ΔH=5 T$) ranging from ~92 K to ~107 K. The maximum entropy change ($-ΔS_M^{\rm max}$) and refrigerant capacity (RC) for Sample A at $μ_0 ΔH=5 T$ are calculated to be ~9.42 Jkg$^{-1}$K$^{-1}$ and ~676 Jkg$^{-1}$. The calculated curves of $-ΔS_M(T)$ and the corresponding experimental data match well with each other, suggesting that the desirable magnetocaloric properties of the microwire arrays can be designed. Simulation shows that the RC values of the designed systems increase when increasing the interval of $T_C$. The table-like MCE and the enhanced heat-transfer efficiency due to the enhanced surface areas of the microwires make this newly designed magnetic bed very promising for use in energy-efficient magnetic refrigerators.

physics.app-ph↗