arXiv subjects

Jingyi Yang

Publications and source records attributed to Jingyi Yang.

36 records · Page 2Linked to original sources

arXiv2025

RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents

With the rapid development of multimodal large language models (MLLMs), they are increasingly deployed as autonomous computer-use agents capable of accomplishing complex computer tasks. However, a pressing issue arises: Can the safety risk principles designed and aligned for general MLLMs in dialogue scenarios be effectively transferred to real-world computer-use scenarios? Existing research on evaluating the safety risks of MLLM-based computer-use agents suffers from several limitations: it either lacks realistic interactive environments, or narrowly focuses on one or a few specific risk types. These limitations ignore the complexity, variability, and diversity of real-world environments, thereby restricting comprehensive risk evaluation for computer-use agents. To this end, we introduce \textbf{RiOSWorld}, a benchmark designed to evaluate the potential risks of MLLM-based agents during real-world computer manipulations. Our benchmark includes 492 risky tasks spanning various computer applications, involving web, social media, multimedia, os, email, and office software. We categorize these risks into two major classes based on their risk source: (i) User-originated risks and (ii) Environmental risks. For the evaluation, we evaluate safety risks from two perspectives: (i) Risk goal intention and (ii) Risk goal completion. Extensive experiments with multimodal agents on \textbf{RiOSWorld} demonstrate that current computer-use agents confront significant safety risks in real-world scenarios. Our findings highlight the necessity and urgency of safety alignment for computer-use agents in real-world computer manipulation, providing valuable insights for developing trustworthy computer-use agents. Our benchmark is publicly available at https://yjyddq.github.io/RiOSWorld.github.io/.

cs.AI↗

arXiv2025

DADM: Dual Alignment of Domain and Modality for Face Anti-spoofing

With the availability of diverse sensor modalities (i.e., RGB, Depth, Infrared) and the success of multi-modal learning, multi-modal face anti-spoofing (FAS) has emerged as a prominent research focus. The intuition behind it is that leveraging multiple modalities can uncover more intrinsic spoofing traces. However, this approach presents more risk of misalignment. We identify two main types of misalignment: (1) \textbf{Intra-domain modality misalignment}, where the importance of each modality varies across different attacks. For instance, certain modalities (e.g., Depth) may be non-defensive against specific attacks (e.g., 3D mask), indicating that each modality has unique strengths and weaknesses in countering particular attacks. Consequently, simple fusion strategies may fall short. (2) \textbf{Inter-domain modality misalignment}, where the introduction of additional modalities exacerbates domain shifts, potentially overshadowing the benefits of complementary fusion. To tackle (1), we propose a alignment module between modalities based on mutual information, which adaptively enhances favorable modalities while suppressing unfavorable ones. To address (2), we employ a dual alignment optimization method that aligns both sub-domain hyperplanes and modality angle margins, thereby mitigating domain gaps. Our method, dubbed \textbf{D}ual \textbf{A}lignment of \textbf{D}omain and \textbf{M}odality (DADM), achieves state-of-the-art performance in extensive experiments across four challenging protocols demonstrating its robustness in multi-modal domain generalization scenarios. The codes will be released soon.

cs.CV↗

arXiv2025

Context Parallelism for Scalable Million-Token Inference

We present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method achieves 1M context prefill with Llama3 405B model in 77s (93% parallelization efficiency, 63% FLOPS utilization) and 128K context prefill in 3.8s. We develop two lossless exact ring attention variants: pass-KV and pass-Q to cover a wide range of use cases with the state-of-the-art performance: full prefill, persistent KV prefill and decode. Benchmarks on H100 GPU hosts inter-connected with RDMA and TCP both show similar scalability for long-context prefill, demonstrating that our method scales well using common commercial data center with medium-to-low inter-host bandwidth.

cs.DC↗

arXiv2025

STGCN-LSTM for Olympic Medal Prediction: Dynamic Power Modeling and Causal Policy Optimization

This paper proposes a novel hybrid model, STGCN-LSTM, to forecast Olympic medal distributions by integrating the spatio-temporal relationships among countries and the long-term dependencies of national performance. The Spatial-Temporal Graph Convolution Network (STGCN) captures geographic and interactive factors-such as coaching exchange and socio-economic links-while the Long Short-Term Memory (LSTM) module models historical trends in medal counts, economic data, and demographics. To address zero-inflated outputs (i.e., the disparity between countries that consistently yield wins and those never having won medals), a Zero-Inflated Compound Poisson (ZICP) framework is incorporated to separate random zeros from structural zeros, providing a clearer view of potential breakthrough performances. Validation includes historical backtracking, policy shock simulations, and causal inference checks, confirming the robustness of the proposed method. Results shed light on the influence of coaching mobility, event specialization, and strategic investment on medal forecasts, offering a data-driven foundation for optimizing sports policies and resource allocation in diverse Olympic contexts.

cs.LG↗

arXiv2025

Shall Your Data Strategy Work? Perform a Swift Study

This work presents a swift method to assess the efficacy of particular types of instruction-tuning data, utilizing just a handful of probe examples and eliminating the need for model retraining. This method employs the idea of gradient-based data influence estimation, analyzing the gradient projections of probe examples from the chosen strategy onto evaluation examples to assess its advantages. Building upon this method, we conducted three swift studies to investigate the potential of Chain-of-thought (CoT) data, query clarification data, and response evaluation data in enhancing model generalization. Subsequently, we embarked on a validation study to corroborate the findings of these swift studies. In this validation study, we developed training datasets tailored to each studied strategy and compared model performance with and without the use of these datasets. The results of the validation study aligned with the findings of the swift studies, validating the efficacy of our proposed method.

cs.CL↗

arXiv2025

G$^2$V$^2$former: Graph Guided Video Vision Transformer for Face Anti-Spoofing

In videos containing spoofed faces, we may uncover the spoofing evidence based on either photometric or dynamic abnormality, even a combination of both. Prevailing face anti-spoofing (FAS) approaches generally concentrate on the single-frame scenario, however, purely photometric-driven methods overlook the dynamic spoofing clues that may be exposed over time. This may lead FAS systems to conclude incorrect judgments, especially in cases where it is easily distinguishable in terms of dynamics but challenging to discern in terms of photometrics. To this end, we propose the Graph Guided Video Vision Transformer (G$^2$V$^2$former), which combines faces with facial landmarks for photometric and dynamic feature fusion. We factorize the attention into space and time, and fuse them via a spatiotemporal block. Specifically, we design a novel temporal attention called Kronecker temporal attention, which has a wider receptive field, and is beneficial for capturing dynamic information. Moreover, we leverage the low-semantic motion of facial landmarks to guide the high-semantic change of facial expressions based on the motivation that regions containing landmarks may reveal more dynamic clues. Extensive experiments on nine benchmark datasets demonstrate that our method achieves superior performance under various scenarios. The codes will be released soon.

cs.CV↗

arXiv2025

Kronecker Mask and Interpretive Prompts are Language-Action Video Learners

Contrastive language-image pretraining (CLIP) has significantly advanced image-based vision learning. A pressing topic subsequently arises: how can we effectively adapt CLIP to the video domain? Recent studies have focused on adjusting either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we propose \textbf{CLAVER}: a \textbf{C}ontrastive \textbf{L}anguage-\textbf{A}ction \textbf{V}ideo Learn\textbf{er}, designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity inductive bias, mitigating the issue of spatiotemporal homogenization, and 3) it can be seamlessly plugged into transformer-based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and semantically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learning scenarios demonstrate the superiority and generality of our approach.

cs.CV↗

arXiv2024

FedPAE: Peer-Adaptive Ensemble Learning for Asynchronous and Model-Heterogeneous Federated Learning

Federated learning (FL) enables multiple clients with distributed data sources to collaboratively train a shared model without compromising data privacy. However, existing FL paradigms face challenges due to heterogeneity in client data distributions and system capabilities. Personalized federated learning (pFL) has been proposed to mitigate these problems, but often requires a shared model architecture and a central entity for parameter aggregation, resulting in scalability and communication issues. More recently, model-heterogeneous FL has gained attention due to its ability to support diverse client models, but existing methods are limited by their dependence on a centralized framework, synchronized training, and publicly available datasets. To address these limitations, we introduce Federated Peer-Adaptive Ensemble Learning (FedPAE), a fully decentralized pFL algorithm that supports model heterogeneity and asynchronous learning. Our approach utilizes a peer-to-peer model sharing mechanism and ensemble selection to achieve a more refined balance between local and global information. Experimental results show that FedPAE outperforms existing state-of-the-art pFL algorithms, effectively managing diverse client capabilities and demonstrating robustness against statistical heterogeneity.

cs.LG↗

arXiv2024

Generalized Face Anti-spoofing via Finer Domain Partition and Disentangling Liveness-irrelevant Factors

Face anti-spoofing techniques based on domain generalization have recently been studied widely. Adversarial learning and meta-learning techniques have been adopted to learn domain-invariant representations. However, prior approaches often consider the dataset gap as the primary factor behind domain shifts. This perspective is not fine-grained enough to reflect the intrinsic gap among the data accurately. In our work, we redefine domains based on identities rather than datasets, aiming to disentangle liveness and identity attributes. We emphasize ignoring the adverse effect of identity shift, focusing on learning identity-invariant liveness representations through orthogonalizing liveness and identity features. To cope with style shifts, we propose Style Cross module to expand the stylistic diversity and Channel-wise Style Attention module to weaken the sensitivity to style shifts, aiming to learn robust liveness representations. Furthermore, acknowledging the asymmetry between live and spoof samples, we introduce a novel contrastive loss, Asymmetric Augmented Instance Contrast. Extensive experiments on four public datasets demonstrate that our method achieves state-of-the-art performance under cross-dataset and limited source dataset scenarios. Additionally, our method has good scalability when expanding diversity of identities. The codes will be released soon.

cs.CV↗

arXiv2024

Understanding Human-AI Collaboration in Music Therapy Through Co-Design with Therapists

The rapid development of musical AI technologies has expanded the creative potential of various musical activities, ranging from music style transformation to music generation. However, little research has investigated how musical AIs can support music therapists, who urgently need new technology support. This study used a mixed method, including semi-structured interviews and a participatory design approach. By collaborating with music therapists, we explored design opportunities for musical AIs in music therapy. We presented the co-design outcomes involving the integration of musical AIs into a music therapy process, which was developed from a theoretical framework rooted in emotion-focused therapy. After that, we concluded the benefits and concerns surrounding music AIs from the perspective of music therapists. Based on our findings, we discussed the opportunities and design implications for applying musical AIs to music therapy. Our work offers valuable insights for developing human-AI collaborative music systems in therapy involving complex procedures and specific requirements.

cs.HC↗

arXiv2023

Design and Verification of a Novel Triphibian Robot

Multi-modal robots expand their operations from one working medium to another, land to air for example. The majorities of multi-modal robots mainly refer to platforms that operate in two different media. However, for all-terrain tasks, there are seldom research to date in the literature. Generally, locomotions in different working media, i.e. land, water and air, require different propelling actuators, and thus the triphibian system becomes bulky. To overcome this challenge, we proposed a triphibian robot and provide the robot with driving forces to perform all-terrain operations in an efficient way. A morphable mechanism is designed to enable the transition between different motion modes, and specifically a cylindrical body is implemented as the rolling mechanism in land mode. Detailed design principles of different mechanisms and the transition between various locomotion modes are analyzed. Finally, a triphibian robot prototype is fabricated and tested in various working media with both mono-modal and multi-modal functionalities. Experiments have verified our platform, and the results show promising adaptions in future exploration tasks in various working scenarios.

cs.RO↗

arXiv2022

Ai4EComponentLib.jl: A Component-base Model Library in Julia

Ai4EComponentLib.jl(Ai4EComponentLib) is a component-base model library based on Julia language, which relies on the differential equation solver DifferentialEquations.jl and the symbolic modeling tool Modelingtoolkit.jl. To handle problems in different physical domains, Ai4EComponentLib tries to build them with component-base model. Supported by a new generation of symbolic modeling tools, models built with Ai4EComponentLib are more flexible and scalable than models built with traditional tools like Modelica. This paper will introduce the instance and general modeling methods of Ai4EComponentLib model library.

cs.SE↗

arXiv2022

OptControl.jl: An Interpreter for Optimal Control Problem

OptControl.jl(OptControl) implements that modeling optimal control problems with symbolic algebra system based on Julia language, and generates the corresponding numerical optimization codes to solve them with packages from Julia. OptControl does not define a data type, but generates a solution script by handling Julia strings and runs the script automatically. It also provides an interface to save script files. Meanwhile, OptControl supports component-based modeling, which makes it easy to build the optimal control problem of complex systems. All of OptControl's dependency packages come from ecosystem of Julia.

math.OC↗

arXiv2022

Shape-morphing structures based on perforated kirigami

Shape-morphing structures, which are able to change their shapes from one state to another, are important in a wide range of engineering applications. A popular scenario is morphing from an initial two-dimensional (2D) shape that is flat to a three-dimensional (3D) target shape. One of the exciting manufacturing paradigms is transforming flat 2D sheets with prescribed cuts (i.e. kirigami) into 3D structures. By employing the formalism of the 'tapered elastica' equation, we develop an inverse design framework to predict the shape of the 2D cut pattern that would generate a desired axisymmetric 3D shape. Our previous work has shown that tessellated 3D structures can be achieved by designing both the width and thickness of the cut 2D sheet to have particular tapered designs. However, the fabrication of a sample with variable thickness is quite challenging. Here we propose a new strategy -- perforating the cut sheet with tapered width but uniform thickness to introduce a distribution of porosity. We refer to this strategy as perforated kirigami and show how the porosity function can be calculated from our theoretical model. The porosity distribution can easily be realized by laser cutting and modifies the bending stiffness of the sheet to yield a desired elastic deformation upon buckling. To verify our theoretical approach, we conduct finite element simulations and physical experiments. We also examine the loading-bearing capacity of morphed structures via indentation tests in both FEM simulations and experiments. As an example, the relationship between the measured geometric rigidity of morphed half-ellipsoids and their aspect ratio is investigated in details.

cond-mat.soft↗

arXiv2021

Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation

Cardinality estimation (CardEst) plays a significant role in generating high-quality query plans for a query optimizer in DBMS. In the last decade, an increasing number of advanced CardEst methods (especially ML-based) have been proposed with outstanding estimation accuracy and inference latency. However, there exists no study that systematically evaluates the quality of these methods and answer the fundamental problem: to what extent can these methods improve the performance of query optimizer in real-world settings, which is the ultimate goal of a CardEst method. In this paper, we comprehensively and systematically compare the effectiveness of CardEst methods in a real DBMS. We establish a new benchmark for CardEst, which contains a new complex real-world dataset STATS and a diverse query workload STATS-CEB. We integrate multiple most representative CardEst methods into an open-source database system PostgreSQL, and comprehensively evaluate their true effectiveness in improving query plan quality, and other important aspects affecting their applicability, ranging from inference latency, model size, and training time, to update efficiency and accuracy. We obtain a number of key findings for the CardEst methods, under different data and query settings. Furthermore, we find that the widely used estimation accuracy metric(Q-Error) cannot distinguish the importance of different sub-plan queries during query optimization and thus cannot truly reflect the query plan quality generated by CardEst methods. Therefore, we propose a new metric P-Error to evaluate the performance of CardEst methods, which overcomes the limitation of Q-Error and is able to reflect the overall end-to-end performance of CardEst methods. We have made all of the benchmark data and evaluation code publicly available at https://github.com/Nathaniel-Han/End-to-End-CardEst-Benchmark.

cs.DB↗

arXiv2021

Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning

We propose Partially Interpretable Estimators (PIE) which attribute a prediction to individual features via an interpretable model, while a (possibly) small part of the PIE prediction is attributed to the interaction of features via a black-box model, with the goal to boost the predictive performance while maintaining interpretability. As such, the interpretable model captures the main contributions of features, and the black-box model attempts to complement the interpretable piece by capturing the "nuances" of feature interactions as a refinement. We design an iterative training algorithm to jointly train the two types of models. Experimental results show that PIE is highly competitive to black-box models while outperforming interpretable baselines. In addition, the understandability of PIE is comparable to simple linear models as validated via a human evaluation.

cs.LG↗

arXiv2019

Photonic Crystal Fiber Metalens

Optical fiber is a well-established efficient way to guide and manipulate light and allowing high bandwidth optical transmission for long distance communication with low attenuation. Although a dielectric optical waveguide is efficient for transmitting light, its functionality is limited by the dielectric properties of the materials of core and cladding.The light coming out from optical fiber is typically diverging, and the numerical aperture is determined by the refractive index of the fiber materials. Thus, the light intensity decreases significantly upon exiting the fiber. The use of metasurfaces provides the opportunity to tailor light properties for advanced light manipulation and to develop novel optical applications that are flat. By producing a specific phase profile using spatially-varied nano-antenna elements, metasurfaces can control the wavefront of the transmitted, reflected, and scattered light, and enable novel ultrathin optical components such as flat lenses. Here, we report an ultrathin optical metalens cascaded on the facet of optical fiber that enables light focusing in the telecommunication regime. In-fiber metalenses with focal lengths of 28 and 40 um and maximum efficiency of ~16.4% at a wavelength of 1550 nm are demonstrated. The integration of an ultrathin metalens and optical fiber will open the path to revolutionary in-fiber optical devices for practical applications in optical imaging, sensing, and fiber laser.

physics.optics↗

arXiv2017

Excitation of epsilon-near-zero resonance in ultra-thin indium tin oxide shell embedded nanostructured optical fiber

We report a novel optical waveguide design of a hollow step index fiber modified with a thin layer of indium tin oxide (ITO). We show an excitation of highly confined waveguide mode in the proposed fiber near the wavelength where permittivity of ITO approaches zero. Due to the high field confinement within thin ITO shell inside the fiber, the epsilon-near-zero (ENZ) mode can be characterized by a peak in modal loss of the hybrid waveguide. Our results show that such in-fiber excitation of ENZ mode is due to the coupling of the guided core mode to the thin-film ENZ mode. We also show that the phase matching wavelength, where the coupling takes place, varies depending on the refractive index of the constituents inside the central bore of the fiber. These ENZ nanostructured optical fibers have many potential applications, for example, in ENZ nonlinear and magneto-optics, as in-fiber wavelength-dependent filters, and as subwavelength fluid channel for optical and bio-photonic sensing.

physics.optics↗