arXiv subjects

Ziyang Meng

Publications and source records attributed to Ziyang Meng.

At least 37 records · Page 2Linked to original sources

arXiv2025

ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

Recent advancements in Multi-modal Large Language Models (MLLMs) have led to significant progress in developing GUI agents for general tasks such as web browsing and mobile phone use. However, their application in professional domains remains under-explored. These specialized workflows introduce unique challenges for GUI perception models, including high-resolution displays, smaller target sizes, and complex environments. In this paper, we introduce ScreenSpot-Pro, a new benchmark designed to rigorously evaluate the grounding capabilities of MLLMs in high-resolution professional settings. The benchmark comprises authentic high-resolution images from a variety of professional domains with expert annotations. It spans 23 applications across five industries and three operating systems. Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%. Our experiments reveal that strategically reducing the search area enhances accuracy. Based on this insight, we propose ScreenSeekeR, a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-of-the-art performance with 48.1% without any additional training. We hope that our benchmark and findings will advance the development of GUI agents for professional applications. Code, data and leaderboard can be found at https://gui-agent.github.io/grounding-leaderboard.

cs.CV↗

arXiv2024

VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning

Recent advances in Large Vision-Language Models (LVLMs) have significantly improve performance in image comprehension tasks, such as formatted charts and rich-content images. Yet, Graphical User Interface (GUI) pose a greater challenge due to their structured format and detailed textual information. Existing LVLMs often overly depend on internal knowledge and neglect image content, resulting in hallucinations and incorrect responses in GUI comprehension. To address these issues, we introduce VGA, a fine-tuned model designed for comprehensive GUI understanding. Our model aims to enhance the interpretation of visual data of GUI and reduce hallucinations. We first construct a Vision Question Answering (VQA) dataset of 63.8k high-quality examples with our propose Referent Method, which ensures the model's responses are highly depend on visual content within the image. We then design a two-stage fine-tuning method called Foundation and Advanced Comprehension (FAC) to enhance both the model's ability to extract information from image content and alignment with human intent. Experiments show that our approach enhances the model's ability to extract information from images and achieves state-of-the-art results in GUI understanding tasks. Our dataset and fine-tuning script will be released soon.

cs.CV↗

arXiv2024

Evolution from quantum anomalous Hall insulator to heavy-fermion semimetal in magic-angle twisted bilayer graphene

The ground states of twisted bilayer graphene (TBG) at chiral and flat-band limit with integer fillings are known from exact solutions, while their dynamical and thermodynamical properties are revealed by unbiased quantum Monte Carlo (QMC) simulations. However, to elucidate experimental observations of correlated metallic, insulating and superconducting states and their transitions, investigations on realistic, or non-chiral cases are vital. Here we employ momentum-space QMC method to investigate the evolution of correlated states in magic-angle TBG away from chiral limit at charge neutrality with polarized spin/valley, which approximates to an experimental case with filling factor $ν=-3$. We find that the ground state evolves from quantum anomalous Hall insulator into an intriguing correlated semimetallic state possessing heavy-fermion features as AA hopping strength reaches experimental values. Such a state resembles the recently proposed heavy-fermion representations with localized electrons residing at AA stacking regions and delocalized electrons itinerating via AB/BA stacking regions. The spectral signatures of the localized and itinerant electrons in the heavy-fermion semimetal phase are revealed, with the connection to experimental results being discussed.

cond-mat.str-el↗

arXiv2023

Robust Kalman filters with unknown covariance of multiplicative noise

In this paper, state and noise covariance estimation problems for linear system with unknown multiplicative noise are considered. The measurement likelihood is modelled as a mixture of two Gaussian distributions and a Student's t distribution, respectively. The unknown covariance of multiplicative noise is modelled as an inverse Gamma/Wishart distribution and the initial condition is formulated as the nominal covariance. By using robust design and choosing hierarchical priors, two variational Bayesian based robust Kalman filters are proposed. Stability and covergence of the proposed filters, the covariance parameters, the VB inference, and the estimation error dynamics are analyzed. The lower and upper bounds are also provided to guarantee the performance of the proposed filters. A target tracking simulation is provided to validate the effectiveness of the proposed filters.

eess.SP↗

arXiv2023

A Unified Distributed Method for Constrained Networked Optimization via Saddle-Point Dynamics

This paper develops a unified distributed method for solving two classes of constrained networked optimization problems, i.e., optimal consensus problem and resource allocation problem with non-identical set constraints. We first transform these two constrained networked optimization problems into a unified saddle-point problem framework with set constraints. Subsequently, two projection-based primal-dual algorithms via Optimistic Gradient Descent Ascent (OGDA) method and Extra-gradient (EG) method are developed for solving constrained saddle-point problems. It is shown that the developed algorithms achieve exact convergence to a saddle point with an ergodic convergence rate $O(1/k)$ for general convex-concave functions. Based on the proposed primal-dual algorithms via saddle-point dynamics, we develop unified distributed algorithm design and convergence analysis for these two networked optimization problems. Finally, two numerical examples are presented to demonstrate the theoretical results.

math.OC↗

arXiv2022

Distributed Event-Triggered Algorithm for Convex Optimization with Coupled Constraints

This paper develops a distributed primal-dual algorithm via event-triggered mechanism to solve a class of convex optimization problems subject to local set constraints, coupled equality and inequality constraints. Different from some existing distributed algorithms with the diminishing step-sizes, our algorithm uses the constant step-sizes, and is shown to achieve an exact convergence to an optimal solution with an $O(1/k)$ convergence rate for general convex objective functions, where $k>0$ is the iteration number. Moreover, by applying event-triggered communication mechanism, the proposed algorithm can effectively reduce the communication cost without sacrificing the convergence rate. Finally, a numerical example is presented to verify the effectiveness of the proposed algorithm.

math.OC↗

arXiv2021

Distributed algorithms for the least square solution of linear equations

This paper proposes distributed algorithms for solving linear equations to seek a least square solution via multi-agent networks. We consider that each agent has only access to a small and imcomplete block of linear equations rather than the complete row or column in the existing literatures. Firstly, we focus on the case of a homogeneous partition of linear equations. A distributed algorithm is proposed via a single-layered grid network, in which each agent only needs to control three scalar states. Secondly, we consider the case of heterogeneous partitions of linear equations. Two distributed algorithms with doubled-layered network are developed, which allows each agent's states to have different dimensions and can be applied to heterogeneous agents with different storage and computation capability. Rigorous proofs show that the proposed distributed algorithms collaboratively obtain a least square solution with exponential convergence, and also own a solvability verification property, i.e., a criterion to verify whether the obtained solution is an exact solution. Finally, some simulation examples are provided to demonstrate the effectiveness of the proposed algorithms.

math.NA↗

arXiv2019

Distributed stochastic subgradient-free algorithm for Nash equilibrium seeking in two-network zero-sum games

This paper investigates the distributed Nash equilibrium seeking problem for two-network zero-sum games with set constraints, where the two networks have the opposite nonsmooth cost functions. The interaction of the agents in each network is characterized by an unbalanced directed graph. We are particularly interested in the case that the local cost function of each agent in the two networks is unknown in this paper. We first construct the stochastic subgradient-free two-variable oracles based on the measurements of the local cost functions. Instead of using subgradients of the local cost functions, the subgradient-free two-variable oracles are employed to design the distributed algorithm for the agents to search the Nash equilibrium. Under the strong connectivity assumption, it is shown that the proposed algorithm guarantees that the states of all the agents converge almost surely to the neighborhood of the Nash equilibrium, where the scale of the neighborhood can be arbitrarily small by selecting suitable smooth parameter in the oracles. Numerical simulations are finally given to validate the theoretical results.

math.OC↗

arXiv2018

Extended Coulomb liquid of paired hardcore boson model on a pyrochlore lattice

There is a growing interest in the $U(1)$ Coulomb liquid in both quantum materials in pyrochlore ice and cluster Mott insulators and cold atom systems. We explore a paired hardcore boson model on a pyrochlore lattice. This model is equivalent to the XYZ spin model that was proposed for rare-earth pyrochlores with "dipole-octupole" doublets. Since this model has no sign problem for quantum Monte Carlo (QMC) simulations in a large parameter regime, we carry out both analytical and QMC calculations. We find that the $U(1)$ Coulomb liquid is quite stable and spans a rather large portion of the phase diagram with boson pairing. Moreover, we numerically find thermodynamic evidence that the boson pairing could induce a possible $\mathbb{Z}_2$ liquid in the vicinity of the phase boundary between Coulomb liquid and $\mathbb{Z}_2$ symmetry-broken phase. Besides the materials' relevance with quantum spin ice, we point to quantum simulation with cold atoms on optical lattices.

cond-mat.str-el↗

arXiv2017

Coordinated trajectory tracking of multiple vertical take-off and landing UAVs

This paper investigates the coordinated trajectory tracking problem of multiple vertical takeooff and landing (VTOL) unmanned aerial vehicles (UAVs). The case of unidirectional information flow is considered and the objective is to drive all the follower VTOL UAVs to accurately track the trajectory of the leader. Firstly, a novel distributed estimator is developed for each VTOL UAV to obtain the leader's desired information asymptotically. With the outputs of the estimators, the solution to the coordinated trajectory tracking problem of multiple VTOL UAVs is transformed to individually solving the tracking problem of each VTOL UAV. Due to the under-actuated nature of the VTOL UAV, a hierarchical framework is introduced for each VTOL UAV such that a command force and an applied torque are exploited in sequence, then the position tracking to the estimated desired position and the attitude tracking to the command attitude are achieved. Moreover, an auxiliary system with proper parameters is implemented to guarantee the singularity-free command attitude extraction and to obviate the use of the unavailable desired information. The stability analysis and simulations effectively validate the achievement of the coordinated trajectory tracking of multiple VTOL UAVs with the proposed control approach.

eess.SY↗

arXiv2017

Persistent Flows and Non-Reciprocal Interactions in Deterministic Networks

This paper studies deterministic consensus networks with discrete-time dynamics under persistent flows and non-reciprocal agent interactions. An arc describing the interaction strength between two agents is said to be persistent if its weight function has an infinite $l_1$ norm. We discuss two balance conditions on the interactions between agents which generalize the arc-balance and cut-balance conditions in the literature respectively. The proposed conditions require that such a balance should be satisfied over each time window of a fixed length instead of at each time instant. We prove that in both cases global consensus is reached if and only if the persistent graph, which consists of all the persistent arcs, contains a directed spanning tree. The convergence rates of the system to consensus are also provided in terms of the interactions between agents having taken place. The results are obtained under a weak condition without assuming the existence of a positive lower bound of all the nonzero weights of arcs and are compared with the existing results. Illustrative examples are provided to show the critical importance of the nontrivial lower boundedness of the self-confidence of the agents.

eess.SY↗

arXiv2017

A State Transition Matrix-Based Approach to Separation of Cooperations and Antagonisms in Opinion Dynamics

This paper is concerned with the dynamics evolution of opinions in the presence of both cooperations and antagonisms. The class of Laplacian flows is addressed through signed digraphs subject to switching topologies. Further, a state transition matrix-based approach is developed for the analysis of opinion dynamics, regardless of any assumptions on connectivity, structural balance or digon sign-symmetry of signed digraphs. It is shown that based on the separation of cooperations and antagonisms, a relationship can be bridged between opinion dynamics under signed digraphs and under conventional digraphs. This helps to solve convergence problems for opinion dynamics. In particular, bipartite consensus (or stability) emerges if and only if the associated switching signed digraph is simultaneously structurally balanced (or unbalanced), which generalizes the use of structural balance theory in opinion dynamics to the case study of changing network topologies.

eess.SY↗

arXiv2016

Modulus Consensus over Networks with Antagonistic Interactions and Switching Topologies

In this paper, we study the discrete-time consensus problem over networks with antagonistic and cooperative interactions. Following the work by Altafini [IEEE Trans. Automatic Control, 58 (2013), pp. 935--946], by an antagonistic interaction between a pair of nodes updating their scalar states we mean one node receives the opposite of the state of the other and naturally by an cooperative interaction we mean the former receives the true state of the latter. Here the pairwise communication can be either unidirectional or bidirectional and the overall network topology graph may change with time. The concept of modulus consensus is introduced to characterize the scenario that the moduli of the node states reach a consensus. It is proved that modulus consensus is achieved if the switching interaction graph is uniformly jointly strongly connected for unidirectional communications, or infinitely jointly connected for bidirectional communications. We construct a counterexample to underscore the rather surprising fact that quasi-strong connectivity of the interaction graph, i.e., the graph contains a directed spanning tree, is not sufficient to guarantee modulus consensus even under fixed topologies. Finally, simulation results using a discrete-time Kuramoto model are given to illustrate the convergence results showing that the proposed framework is applicable to a class of networks with general nonlinear node dynamics.

math.OC↗

arXiv2015

Network Synchronization with Nonlinear Dynamics and Switching Interactions

This paper considers the synchronization problem for networks of coupled nonlinear dynamical systems under switching communication topologies. Two types of nonlinear agent dynamics are considered. The first one is non-expansive dynamics (stable dynamics with a convex Lyapunov function $φ(\cdot)$) and the second one is dynamics that satisfies a global Lipschitz condition. For the non-expansive case, we show that various forms of joint connectivity for communication graphs are sufficient for networks to achieve global asymptotic $φ$-synchronization. We also show that $φ$-synchronization leads to state synchronization provided that certain additional conditions are satisfied. For the globally Lipschitz case, unlike the non-expansive case, joint connectivity alone is not sufficient for achieving synchronization. A sufficient condition for reaching global exponential synchronization is established in terms of the relationship between the global Lipschitz constant and the network parameters. We also extend the results to leader-follower networks.

eess.SY↗

arXiv2015

Multi-agent Systems with Compasses

This paper investigates agreement protocols over cooperative and cooperative--antagonistic multi-agent networks with coupled continuous-time nonlinear dynamics. To guarantee convergence for such systems, it is common in the literature to assume that the vector field of each agent is pointing inside the convex hull formed by the states of the agent and its neighbors, given that the relative states between each agent and its neighbors are available. This convexity condition is relaxed in this paper, as we show that it is enough that the vector field belongs to a strict tangent cone based on a local supporting hyperrectangle. The new condition has the natural physical interpretation of requiring shared reference directions in addition to the available local relative states. Such shared reference directions can be further interpreted as if each agent holds a magnetic compass indicating the orientations of a global frame. It is proven that the cooperative multi-agent system achieves exponential state agreement if and only if the time-varying interaction graph is uniformly jointly quasi-strongly connected. Cooperative--antagonistic multi-agent systems are also considered. For these systems, the relation has a negative sign for arcs corresponding to antagonistic interactions. State agreement may not be achieved, but instead it is shown that all the agents' states asymptotically converge, and their limits agree componentwise in absolute values if and in general only if the time-varying interaction graph is uniformly jointly strongly connected.

cs.MA↗

arXiv2015

Sampled-Data Consensus over Random Networks

This paper considers the consensus problem for a network of nodes with random interactions and sampled-data control actions. We first show that consensus in expectation, in mean square, and almost surely are equivalent for a general random network model when the inter-sampling interval and network size satisfy a simple relation. The three types of consensus are shown to be simultaneously achieved over an independent or a Markovian random network defined on an underlying graph with a directed spanning tree. For both independent and Markovian random network models, necessary and sufficient conditions for mean-square consensus are derived in terms of the spectral radius of the corresponding state transition matrix. These conditions are then interpreted as the existence of critical value on the inter-sampling interval, below which global mean-square consensus is achieved and above which the system diverges in mean-square sense for some initial states. Finally, we establish an upper bound on the inter-sampling interval below which almost sure consensus is reached, and a lower bound on the inter-sampling interval above which almost sure divergence is reached. Some numerical simulations are given to validate the theoretical results and some discussions on the critical value of the inter-sampling intervals for the mean-square consensus are provided.

eess.SY↗

arXiv2014

Coordinated Output Regulation of Heterogeneous Linear Systems under Switching Topologies

This paper constructs a framework to describe and study the coordinated output regulation problem for multiple heterogeneous linear systems. Each agent is modeled as a general linear multiple-input multiple-output system with an autonomous exosystem which represents the individual offset from the group reference for the agent. The multi-agent system as a whole has a group exogenous state which represents the tracking reference for the whole group. Under the constraints that the group exogenous output is only locally available to each agent and that the agents have only access to their neighbors' information, we propose observer-based feedback controllers to solve the coordinated output regulation problem using output feedback information. A high-gain approach is used and the information interactions are allowed to be switched over a finite set of fixed networks containing both graphs that have a directed spanning tree and graphs that do not. The fundamental relationship between the information interactions, the dwell time, the non-identical dynamics of different agents, and the high-gain parameters is given. Simulations are shown to validate the theoretical results.

eess.SY↗

arXiv2014

Cooperative Set Aggregation for Multiple Lagrangian Systems

In this paper, we study the cooperative set tracking problem for a group of Lagrangian systems. Each system observes a convex set as its local target. The intersection of these local sets is the group aggregation target. We first propose a control law based on each system's own target sensing and information exchange with neighbors. With necessary connectivity for both cases of fixed and switching communication graphs, multiple Lagrangian systems are shown to achieve rendezvous on the intersection of all the local target sets while the vectors of generalized coordinate derivatives are driven to zero. Then, we introduce the collision avoidance control term into set aggregation control to ensure group dispersion. By defining an ultimate bound on the final generalized coordinate between each system and the intersection of all the local target sets, we show that multiple Lagrangian systems approach a bounded region near the intersection of all the local target sets while the collision avoidance is guaranteed during the movement. In addition, the vectors of generalized coordinate derivatives of all the mechanical systems are shown to be driven to zero. Simulation results are given to validate the theoretical results.

eess.SY↗