arXiv subjects

Yi Zhang

Publications and source records attributed to Yi Zhang.

At least 613 records · Page 34Linked to original sources

arXiv2024

Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

Graphical User Interfaces (GUIs) are central to our interaction with digital devices and growing efforts have been made to build models for various GUI understanding tasks. However, these efforts largely overlook an important GUI-referring task: screen reading based on user-indicated points, which we name the Screen Point-and-Read (ScreenPR) task. Currently, this task is predominantly handled by rigid accessible screen reading tools, in great need of new models driven by advancements in Multimodal Large Language Models (MLLMs). In this paper, we propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism, to address the ScreenPR task. Based on the input point coordinate and the corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout Tree. Based on the tree, our ToL agent not only comprehends the content of the indicated area but also articulates the layout and spatial relationships between elements. Such layout information is crucial for accurately interpreting information on the screen, distinguishing our ToL agent from other screen reading tools. We also thoroughly evaluate the ToL agent against other baselines on a newly proposed ScreenPR benchmark, which includes GUIs from mobile, web, and operating systems. Last but not least, we test the ToL agent on mobile GUI navigation tasks, demonstrating its utility in identifying incorrect actions along the path of agent execution trajectories. Code and data: https://screen-point-and-read.github.io

cs.CL↗

arXiv2024

Improving Instance Optimization in Deformable Image Registration with Gradient Projection

Deformable image registration is inherently a multi-objective optimization (MOO) problem, requiring a delicate balance between image similarity and deformation regularity. These conflicting objectives often lead to poor optimization outcomes, such as being trapped in unsatisfactory local minima or experiencing slow convergence. Deep learning methods have recently gained popularity in this domain due to their efficiency in processing large datasets and achieving high accuracy. However, they often underperform during test time compared to traditional optimization techniques, which further explore iterative, instance-specific gradient-based optimization. This performance gap is more pronounced when a distribution shift between training and test data exists. To address this issue, we focus on the instance optimization (IO) paradigm, which involves additional optimization for test-time instances based on a pre-trained model. IO effectively combines the generalization capabilities of deep learning with the fine-tuning advantages of instance-specific optimization. Within this framework, we emphasize the use of gradient projection to mitigate conflicting updates in MOO. This technique projects conflicting gradients into a common space, better aligning the dual objectives and enhancing optimization stability. We validate our method using a state-of-the-art foundation model on the 3D Brain inter-subject registration task (LUMIR) from the Learn2Reg 2024 Challenge. Our results show significant improvements over standard gradient descent, leading to more accurate and reliable registration results.

cs.CV↗

arXiv2024

Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention for its ability to utilize MM data and reduce reliance on labeled images. However, current methods face several challenges: (1) Complex network designs hinder scalability to scenarios with more than two modalities. (2) Focusing solely on modality-invariant representation while neglecting modality-specific features, leads to incomplete MM learning. (3) Leveraging unlabeled data with generative methods can be unreliable for SSL. To address these problems, we propose Double Bank Dual Consistency (DBDC), a novel MM-SSL approach for medical image segmentation. To address challenge (1), we propose a modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count. To address challenge (2), we design two learnable plug-in banks, Modality-Level Modulation bank (MLMB) and Modality-Level Prototype (MLPB) bank, to capture both modality-invariant and modality-specific knowledge. These banks are updated using our proposed Modality Prototype Contrastive Learning (MPCL). Additionally, we design Modality Adaptive Weighting (MAW) to dynamically adjust learning weights for each modality, ensuring balanced MM learning as different modalities learn at different rates. Finally, to address challenge (3), we introduce a Dual Consistency (DC) strategy that enforces consistency at both the image and feature levels without relying on generative methods. We evaluate our method on a 2-to-4 modality segmentation task using three open-source datasets, and extensive experiments show that our method outperforms state-of-the-art approaches.

cs.CV↗

arXiv2024

Inverse scattering transform for the defocusing-defocusing coupled Hirota equations with non-parallel boundary conditions at infinity

The inverse scattering transform for the defocusing-defocusing coupled Hirota equations is strictly discussed with non-zero boundary conditions at infinity including non-parallel boundary conditions, specifically referring to the asymptotic polarization vectors. To address the non-analyticity encountered in some of the Jost eigenfunctions, the "adjoint" Lax pair is employed. The inverse problem is formulated as an appropriate matrix Riemann-Hilbert problem. A key difference between non-parallel and parallel boundary conditions lies in the asymptotic behavior of the scattering coefficients, which significantly impacts the normalization of the eigenfunctions and the properties of sectionally meromorphic matrices within the Riemann-Hilbert problem framework. When the asymptotic polarization vectors are non-orthogonal, two distinct methodologies are introduced to convert the Riemann-Hilbert problem into a series of linear algebraic-integral equations. In contrast, when the asymptotic polarization vectors are orthogonal, only one method is feasible. Ultimately, it is demonstrated that pure soliton solutions do not exist in both orthogonal and non orthogonal polarization vector cases. This study provides a comprehensive framework for analyzing the defocusing-defocusing coupled Hirota equations using the inverse scattering transform, offering new insights into the characteristics and solutions of the equations.

nlin.SI↗

arXiv2024

Lost in Tracking: Uncertainty-guided Cardiac Cine MRI Segmentation at Right Ventricle Base

Accurate biventricular segmentation of cardiac magnetic resonance (CMR) cine images is essential for the clinical evaluation of heart function. However, compared to left ventricle (LV), right ventricle (RV) segmentation is still more challenging and less reproducible. Degenerate performance frequently occurs at the RV base, where the in-plane anatomical structures are complex (with atria, valve, and aorta) and vary due to the strong interplanar motion. In this work, we propose to address the currently unsolved issues in CMR segmentation, specifically at the RV base, with two strategies: first, we complemented the public resource by reannotating the RV base in the ACDC dataset, with refined delineation of the right ventricle outflow tract (RVOT), under the guidance of an expert cardiologist. Second, we proposed a novel dual encoder U-Net architecture that leverages temporal incoherence to inform the segmentation when interplanar motions occur. The inter-planar motion is characterized by loss-of-tracking, via Bayesian uncertainty of a motion-tracking model. Our experiments showed that our method significantly improved RV base segmentation taking into account temporal incoherence. Furthermore, we investigated the reproducibility of deep learning-based segmentation and showed that the combination of consistent annotation and loss of tracking could enhance the reproducibility of RV segmentation, potentially facilitating a large number of clinical studies focusing on RV.

eess.IV↗

arXiv2024

Domain-Conditioned Transformer for Fully Test-time Adaptation

Fully test-time adaptation aims to adapt a network model online based on sequential analysis of input samples during the inference stage. We observe that, when applying a transformer network model into a new domain, the self-attention profiles of image samples in the target domain deviate significantly from those in the source domain, which results in large performance degradation during domain changes. To address this important issue, we propose a new structure for the self-attention modules in the transformer. Specifically, we incorporate three domain-conditioning vectors, called domain conditioners, into the query, key, and value components of the self-attention module. We learn a network to generate these three domain conditioners from the class token at each transformer network layer. We find that, during fully online test-time adaptation, these domain conditioners at each transform network layer are able to gradually remove the impact of domain shift and largely recover the original self-attention profile. Our extensive experimental results demonstrate that the proposed domain-conditioned transformer significantly improves the online fully test-time domain adaptation performance and outperforms existing state-of-the-art methods by large margins.

cs.CV↗

arXiv2024

Cornwall-Jackiw-Tomboulis effective field theory to nonuniversal equation of state of an ultracold Bose gas

The equation of state (EOS) serves as a cornerstone in elucidating the properties of quantum many-body systems. A recent highlight along this research line consists of the derivation of the nonuniversal Lee-Huang-Yang (LHY) EOS for an ultracold quantum bosonic gas with finite-range interatomic interactions using one-loop effective path-integral field theory. The purpose of this work is to extend Salasnich's pioneering work to uncover beyond-LHY corrections to the EOS by employing the Cornwall-Jackiw-Tomboulis (CJT) effective field theory, leveraging its two-loop approximation. In this end, we expand Salasnich's remarkable findings of EOS to the next leading order characterized by $\left(ρa_{\text{s}}^{3}\right)^{2}$, with $ρ$ and $a_{\text{s}}$ being the density and the $s$-wave scattering length. Notably, we derive analytical expressions for quantum depletion and chemical potential, representing the next-to-LHY corrections to nonuniversal EOS induced by finite-range effects. Moreover, we propose an experimental protocol of observing the nonuniversal next-to-LHY corrections to the EOS by calculating fractional frequency shifts in the breathing modes. The nonuniversal beyond-LHY EOS in this work paves the way of using LHY effects in quantum simulation experiments and for investigations beyond the LHY regime.

cond-mat.quant-gas↗

arXiv2024

Strong lensing as a probe of braneworld

For the first time, we use the Event Horizon Telescope (EHT) data to constrain the parameters of braneworld black holes which constrain $ε=0.0285^{+0.0888+0.1456}_{-0.0895-0.1475}$ for the anisotropic black hole and $q=-0.0305^{+0.1034+0.1953}_{-0.0895-0.1470}$ for the tidal Reissner-Nordstr$\ddot{o}$m (RN) black hole. Based on the fitted data and physical requirement, we calculate the photon deflection, the angular separation and time delay between different relativistic images of the anisotropic black hole and the tidal RN black hole in the ranges $-0.1190<ε<0$ and $-0.1775 <q<0$. And furthermore, we study the quasinormal modes (QNMs) for the braneworld black holes. The results shed light on existence of extra dimension.

gr-qc↗

arXiv2024

Structured List-Grounded Question Answering

Document-grounded dialogue systems aim to answer user queries by leveraging external information. Previous studies have mainly focused on handling free-form documents, often overlooking structured data such as lists, which can represent a range of nuanced semantic relations. Motivated by the observation that even advanced language models like GPT-3.5 often miss semantic cues from lists, this paper aims to enhance question answering (QA) systems for better interpretation and use of structured lists. To this end, we introduce the LIST2QA dataset, a novel benchmark to evaluate the ability of QA systems to respond effectively using list information. This dataset is created from unlabeled customer service documents using language models and model-based filtering processes to enhance data quality, and can be used to fine-tune and evaluate QA models. Apart from directly generating responses through fine-tuned models, we further explore the explicit use of Intermediate Steps for Lists (ISL), aligning list items with user backgrounds to better reflect how humans interpret list items before generating responses. Our experimental results demonstrate that models trained on LIST2QA with our ISL approach outperform baselines across various metrics. Specifically, our fine-tuned Flan-T5-XL model shows increases of 3.1% in ROUGE-L, 4.6% in correctness, 4.5% in faithfulness, and 20.6% in completeness compared to models without applying filtering and the proposed ISL method.

cs.CL↗

arXiv2024

Stability of a class of supercritical volume-filling chemotaxis-fluid model near Couette flow

Consider a class of chemotaxis-fluid model incorporating a volume-filling effect in the sense of Painter and Hillen (Can. Appl. Math. Q. 2002; 10(4): 501-543), which is a supercritical parabolic-elliptic Keller-Segel system. As shown by Winkler et al., for any given mass, there exists a corresponding solution of the same mass that blows up in either finite or infinite time. In this paper, we investigate the stability properties of the two dimensional Patlak-Keller-Segel-type chemotaxis-fluid model near the Couette flow $ (Ay, 0) $ in $ \mathbb{T}\times\mathbb{R}, $ and show that the solutions are global in time as long as the initial cell mass $M<\frac{2π}{\sqrt{3}} $ and the shear flow is sufficiently strong ($A$ is large enough).

math.AP↗

arXiv2024

CVVLSNet: Vehicle Location and Speed Estimation Using Partial Connected Vehicle Trajectory Data

Real-time estimation of vehicle locations and speeds is crucial for developing many beneficial transportation applications in traffic management and control, e.g., adaptive signal control. Recent advances in communication technologies facilitate the emergence of connected vehicles (CVs), which can share traffic information with nearby CVs or infrastructures. At the early stage of connectivity, only a portion of vehicles are CVs. The locations and speeds for those non-CVs (NCs) are not accessible and must be estimated to obtain the full traffic information. To address the above problem, this paper proposes a novel CV-based Vehicle Location and Speed estimation network, CVVLSNet, to simultaneously estimate the vehicle locations and speeds exclusively using partial CV trajectory data. A road cell occupancy (RCO) method is first proposed to represent the variable vehicle state information. Spatiotemporal interactions can be integrated by simply fusing the RCO representations. Then, CVVLSNet, taking the Coding-RAte TransformEr (CRATE) network as a backbone, is introduced to estimate the vehicle locations and speeds. Moreover, physical vehicle size constraints are also considered in loss functions. Extensive experiments indicate that the proposed method significantly outperformed the existing method under various CV penetration rates, signal timings, and volume-to-capacity ratios.

cs.CV↗

arXiv2024

Creation of independently controllable and long lifetime polar skyrmion textures in ferroelectric-metallic heterostructures

Topological textures like vortices, labyrinths and skyrmions formed in ferroic materials have attracted extensive interests during the past decade for their fundamental physics, intriguing topology, and technological prospects. So far, polar skyrmions remain scarce in ferroelectrics as they require a delicate balance between various dipolar interactions. Here, we report that PbTiO3 thin films in a metallic contact undergo a topological phase transition and stabilize a broad family of skyrmion-like textures (e.g., skyrmion bubbles, multiple π-twist target skyrmions, and skyrmion bags) with independent controllability, analogous to those reported in magnetic systems. Weakly-interacted skyrmion arrays with a density over 300 Gb/inch2 are successfully written, erased and read-out by local electrical and mechanical stimuli of a scanning probe. Interestingly, in contrast to the relatively short lifetime <20 hours of the skyrmion bubbles, the multiple π-twist target skyrmions and skyrmion bags show topology-enhanced stability with lifetime over two weeks. Experimental and theoretical analysis implies the heterostructures carry electric Dzyaloshinskii-Moriya interaction mediated by oxygen octahedral tiltings. Our results demonstrate ferroelectric-metallic heterostructures as fertile playground for topological states and emergent phenomena.

cond-mat.mtrl-sci↗

arXiv2024

Few-Shot Testing of Autonomous Vehicles with Scenario Similarity Learning

Testing and evaluation are critical to the development and deployment of autonomous vehicles (AVs). Given the rarity of safety-critical events such as crashes, millions of tests are typically needed to accurately assess AV safety performance. Although techniques like importance sampling can accelerate this process, it usually still requires too many numbers of tests for field testing. This severely hinders the testing and evaluation process, especially for third-party testers and governmental bodies with very limited testing budgets. The rapid development cycles of AV technology further exacerbate this challenge. To fill this research gap, this paper introduces the few-shot testing (FST) problem and proposes a methodological framework to tackle it. As the testing budget is very limited, usually smaller than 100, the FST method transforms the testing scenario generation problem from probabilistic sampling to deterministic optimization, reducing the uncertainty of testing results. To optimize the selection of testing scenarios, a cross-attention similarity mechanism is proposed to learn to extract the information of AV's testing scenario space. This allows iterative searches for scenarios with the smallest evaluation error, ensuring precise testing within budget constraints. Experimental results in cut-in scenarios demonstrate the effectiveness of the FST method, significantly enhancing accuracy and enabling efficient, precise AV testing.

eess.SY↗

arXiv2024

Trustworthy Hate Speech Detection Through Visual Augmentation

The surge of hate speech on social media platforms poses a significant challenge, with hate speech detection~(HSD) becoming increasingly critical. Current HSD methods focus on enriching contextual information to enhance detection performance, but they overlook the inherent uncertainty of hate speech. We propose a novel HSD method, named trustworthy hate speech detection method through visual augmentation (TrusV-HSD), which enhances semantic information through integration with diffused visual images and mitigates uncertainty with trustworthy loss. TrusV-HSD learns semantic representations by effectively extracting trustworthy information through multi-modal connections without paired data. Our experiments on public HSD datasets demonstrate the effectiveness of TrusV-HSD, showing remarkable improvements over conventional methods.

cs.CV↗

arXiv2024

Robust Coulomb Gap and Varied-temperature Study of Epitaxial 1T'-WSe$_2$ Monolayers

The transition metal dichalcogenides (TMDCs) with a 1T' structural phase are predicted to be two-dimensional topological insulators at zero temperature. Although the quantized edge conductance of 1T'-WTe$_2$ has been confirmed to survive up to 100 K, this temperature is still relatively low for industrial applications. Addressing the limited studies on temperature effects in 1T'-TMDCs, our research focuses on the electronic and crystal properties of the epitaxial 1T'-WSe$_2$ monolayers grown on bilayer graphene (BLG) and SrTiO$_3$(100) substrates at various temperatures. For the 1T'-WSe$_2$ grown on BLG, we observed a significant thermal expansion effect on its band structures with a thermal expansion coefficient of $\sim$60$\times$10$^{-6}$ K$^{-1}$. In contrast, the 1T'-WSe$_2$ grown on SrTiO$_3$(100) exhibits minimal changes with varied temperatures due to the enhanced strain exerted by the substrate. Besides, A significant Coulomb gap (CG) was observed pinned at the Fermi level in the angle-resolved photoemission spectroscopy (ARPES) and scanning tunneling spectroscopy (STS). The CG was founded to decrease with increasing temperatures, and can persist up to 200 K for 1T'-WSe$_2$/BLG, consistent with our Monte Carlo simulations. The robustness of the CG and the positive fundamental gap endow the epitaxial 1T'-WSe$_2$ monolayers with huge potential for realizing the quantum spin Hall devices.

cond-mat.mes-hall↗

arXiv2024

Further remarks on absorbing Markov decision processes

In this note, based on the recent remarkable results of Dufour and Prieto-Rumeau, we deduce that for an absorbing MDP with a given initial state, under a standard compactness-continuity condition, the space of occupation measures has the same convergent sequences, when it is endowed with the weak topology and with the weak-strong topology. We provided two examples demonstrating that imposed condition cannot be replaced with its popular alternative, and the above assertion does not hold for the space of marginals of occupation measures on the state space. Moreover, the examples also clarify some results in the previous literature.

math.OC↗

arXiv2024

Braiding Topology of Non-Hermitian Open-Boundary Bands

There has been much recent interest and progress on topological structures of the non-Hermitian Bloch bands. Here, we study the topological structures of non-Bloch bands of non-Hermitian multiband quantum systems under open boundary conditions, which has received limited attention in prior studies. Using a continuity criterion and an efficient sub-generalized Brillouin zone (sub-GBZ) algorithm, we establish a homotopic characterization -- braiding topology, e.g., characterized by the band's total vorticity -- for open-boundary bands and sub-GBZs. Such topological identification is robust without topological transition and emergent degenerate points, such as exceptional points. We further analyze the transition's impact on bands and spectral flows, including interesting properties unique to open boundaries, and numerically demonstrate our conclusions with tight-binding model examples. We unveil a crucial insight that open-boundary bands interchange their portions after encountering certain exceptional points. Our results enrich the foundational understanding of topological characterizations for generic non-Hermitian quantum systems.

cond-mat.mes-hall↗

arXiv2024

Safe Policy Learning under Regression Discontinuity Designs with Multiple Cutoffs

The regression discontinuity (RD) design is widely used for program evaluation with observational data. The primary focus of the existing literature has been the estimation of the local average treatment effect at the existing treatment cutoff. In contrast, we consider policy learning under the RD design. Because the treatment assignment mechanism is deterministic, learning better treatment cutoffs requires extrapolation. We develop a robust optimization approach to finding optimal treatment cutoffs that improve upon the existing ones. We first decompose the expected utility into point-identifiable and unidentifiable components. We then propose an efficient doubly-robust estimator for the identifiable parts. To account for the unidentifiable components, we leverage the existence of multiple cutoffs that are common under the RD design. Specifically, we assume that the heterogeneity in the conditional expectations of potential outcomes across different groups vary smoothly along the running variable. Under this assumption, we minimize the worst case utility loss relative to the status quo policy. The resulting new treatment cutoffs have a safety guarantee that they will not yield a worse overall outcome than the existing cutoffs. Finally, we establish the asymptotic regret bounds for the learned policy using semi-parametric efficiency theory. We apply the proposed methodology to empirical and simulated data sets.

stat.ME↗