Searcharxiv⌕ Search

arXiv subjects

Di Feng

Publications and source records attributed to Di Feng.

At least 19 recordsLinked to original sources
arXiv2026

MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox

cs.CV↗
arXiv2026

LNN-Fly: Continuous-Time UAV Navigation for Robust Obstacle Avoidance under Timing Mismatch

End-to-end unmanned aerial vehicle (UAV) navigation can achieve impressive agility in simulation, yet its obstacle-avoidance behavior often degrades after deployment because the policy must tolerate simulator mismatch, sensing irregularity, and variable-rate control. These effects are especially dangerous in cluttered environments, where stale observations or short control irregularities can directly lead to collisions. We present LNN-Fly, a deployment-oriented continuous-time navigation policy for LiDAR-based UAV obstacle avoidance. The policy combines a dynamic-programming-inspired structured recurrent update, explicit conditioning on the elapsed control interval {\Delta}t, and an input-driven adaptive forgetting gate that refreshes stale latent state near hazards while preserving consistency during sustained maneuvers. It is trained with differentiable rollouts that incorporate deployment-relevant sensing and timing perturbations. In simulation, LNN-Fly improves obstacle-avoidance performance in the tested settings and shows better tolerance to reduced control frequency, sparse observations, and control-period jitter. It also transfers zero-shot from a simplified differentiable simulator to a physical quadrotor. In indoor cross-frequency real-world tests, the system achieves 100% success over 20 flights, while policy inference has a median latency of 0.514 ms on a desktop graphics processing unit (GPU) and about 2.5 ms on the onboard central processing unit (CPU), with onboard P95 latency below 30 ms.

cs.RO↗
arXiv2026

Incentivizing User Data Contributions for LLM Improvement under Withdrawal Rights

The continued improvement of large language models (LLMs) increasingly depends on eliciting high-quality, user-generated data, yet such data are costly to provide and often withheld due to privacy and effort concerns. This creates a fundamental design challenge: how to incentivize data contribution when model improvements require coordinated, threshold-level inputs, while contributions remain privately costly and partially reversible. We develop and theoretically analyze incentive mechanisms for user data contribution that explicitly account for threshold effects and reversibility, focusing on how subsidies and withdrawal rights can be jointly designed to overcome coordination failure. As a natural benchmark, we first consider subsidy-based incentives, under which users respond to posted payments with privately optimal floor contributions. These decentralized responses may fall below the improvement threshold, resulting in subsidy expenditure without model improvements. We then analyze mechanisms with withdrawal rights, in which users report costs, the provider centrally assigns contribution burdens, and users may withdraw before training. We prove that combining cost reporting with personalized assignment can eliminate inefficient provision by ensuring that data are collected only when improvement is sustainable, converting infeasible instances into a null outcome rather than subsidy leakage. Finally, we compare two withdrawal protocols. The simultaneous protocol can achieve lower total cost, while the small-first sequential protocol better incentivizes participation, encouraging greater data provision and thereby increasing the probability of crossing the improvement threshold.

cs.GT↗
arXiv2026

The Price of Ignorance: Information-Free Quotation for Data Retention in Machine Unlearning

When users exercise data deletion rights under the General Data Protection Regulation (GDPR) and similar regulations, mobile network operators face a tradeoff: excessive machine unlearning degrades model accuracy and incurs retraining costs, yet existing pricing mechanisms for data retention require the server to know every user's private privacy and accuracy preferences, which is infeasible under the very regulations that motivate unlearning. We ask: what is the welfare cost of operating without this private information? We design an information-free ascending quotation mechanism where the server broadcasts progressively higher prices and users self-select their data supply, requiring no knowledge of users' parameters. Under complete information, the protocol admits a unique subgame-perfect Nash equilibrium characterized by single-period selling. We formalize the Price of Ignorance -- the welfare gap between optimal personalized pricing (which knows everything) and our information-free quotation (which knows nothing) -- and prove a three-regime efficiency ordering. Numerical evaluation across seven mechanisms and 5000 Monte Carlo runs shows that this price is near zero: the information-free mechanism achieves >=99% of the welfare of its information-intensive benchmarks, while providing noise-robust guarantees and comparable fairness.

cs.GT↗
arXiv2026

PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice

As large language models (LLMs) are increasingly applied to legal domain-specific tasks, evaluating their ability to perform legal work in real-world settings has become essential. However, existing legal benchmarks rely on simplified and highly standardized tasks, failing to capture the ambiguity, complexity, and reasoning demands of real legal practice. Moreover, prior evaluations often adopt coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. To address these limitations, we introduce PLawBench, a Practical Law Benchmark designed to evaluate LLMs in realistic legal practice scenarios. Grounded in real-world legal workflows, PLawBench models the core processes of legal practitioners through three task categories: public legal consultation, practical case analysis, and legal document generation. These tasks assess a model's ability to identify legal issues and key facts, perform structured legal reasoning, and generate legally coherent documents. PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics, resulting in approximately 12,500 rubric items for fine-grained assessment. Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs. Experimental results show that none achieves strong performance on PLawBench, revealing substantial limitations in the fine-grained legal reasoning capabilities of current LLMs and highlighting important directions for future evaluation and development of legal LLMs. Data is available at: https://github.com/skylenage/PLawbench.

cs.CL↗
arXiv2025

SO-Bench: A Structural Output Evaluation of Multimodal LLMs

Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We make the benchmark and evaluation publicly available at https://github.com/apple/ml-sobench

cs.CV↗
arXiv2025

Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper, we present Ferret-UI Lite, a compact, end-to-end GUI agent that operates across diverse platforms, including mobile, web, and desktop. Utilizing techniques optimized for developing small models, we build our 3B Ferret-UI Lite agent through curating a diverse GUI data mixture from real and synthetic sources, strengthening inference-time performance through chain-of-thought reasoning and visual tool-use, and reinforcement learning with designed rewards. Ferret-UI Lite achieves competitive performance with other small-scale GUI agents. In GUI grounding, Ferret-UI Lite attains scores of $91.6\%$, $53.3\%$, and $61.2\%$ on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI navigation, Ferret-UI Lite achieves success rates of $28.0\%$ on AndroidWorld and $19.8\%$ on OSWorld. We share our methods and lessons learned from developing compact, on-device GUI agents.

cs.CV↗
arXiv2025

On fairness of multi-center allocation problems

We investigate Ekici (2024b)'s multi-center allocation problems, focusing on fairness in this context. We introduce three fairness notions that respect centers' priorities: internal fairness, external fairness, and procedural fairness. The first notion eliminates envy among agents within the same center, the second prohibits envy across different centers, and the third rules out envy from an ex-ante perspective through agents' trading opportunities. We provide two characterizations of a natural extension of the top-trading-cycles mechanism (TTC) through our fairness notions. Precisely, we show that in the presence of strategy-proofness and pair efficiency, internal fairness and external fairness together characterize TTC (Theorem 1). Also, strategy-proofness combined solely with procedural fairness also characterizes TTC (Theorem 2). Furthermore, by adding internal fairness, we establish our third TTC characterization, by relaxing Ekici's queuewise rationality to another voluntary participation condition, the center lower bound (Theorem 3). Finally, we define a core solution within this model and characterize it through TTC (Theorem 4). Our findings offer practical insights for market designers, particularly in contexts such as international cooperation in medical programs and worker exchange programs.

econ.TH↗
arXiv2025

Quotation-Based Data Retention Mechanism for Data Privacy in LLM-Empowered Network Services

The deployment of large language models (LLMs) for next-generation network optimization introduces novel data governance challenges. mobile network operators (MNOs) increasingly leverage generative artificial intelligence (AI) for traffic prediction, anomaly detection, and service personalization, requiring access to users' sensitive network usage data-including mobility patterns, traffic types, and location histories. Under the General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and similar regulations, users retain the right to withdraw consent and demand data deletion. However, extensive machine unlearning degrades model accuracy and incurs substantial computational costs, ultimately harming network performance for all users. We propose an iterative price discovery mechanism enabling MNOs to compensate users for data retention through sequential price quotations. The server progressively raises the unit price for retaining data while users independently determine their supply at each quoted price. This approach requires no prior knowledge of users' privacy preferences and efficiently maximizes social welfare across the network ecosystem.

cs.LG↗
arXiv2024

Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.

cs.CV↗
arXiv2024

LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies

We tackle the challenge of predicting models' Out-of-Distribution (OOD) performance using in-distribution (ID) measurements without requiring OOD data. Existing evaluations with "Effective Robustness", which use ID accuracy as an indicator of OOD accuracy, encounter limitations when models are trained with diverse supervision and distributions, such as class labels (Vision Models, VMs, on ImageNet) and textual descriptions (Visual-Language Models, VLMs, on LAION). VLMs often generalize better to OOD data than VMs despite having similar or lower ID performance. To improve the prediction of models' OOD performance from ID measurements, we introduce the Lowest Common Ancestor (LCA)-on-the-Line framework. This approach revisits the established concept of LCA distance, which measures the hierarchical distance between labels and predictions within a predefined class hierarchy, such as WordNet. We assess 75 models using ImageNet as the ID dataset and five significantly shifted OOD variants, uncovering a strong linear correlation between ID LCA distance and OOD top-1 accuracy. Our method provides a compelling alternative for understanding why VLMs tend to generalize better. Additionally, we propose a technique to construct a taxonomic hierarchy on any dataset using K-means clustering, demonstrating that LCA distance is robust to the constructed taxonomic hierarchy. Moreover, we demonstrate that aligning model predictions with class taxonomies, through soft labels or prompt engineering, can enhance model generalization. Open source code in our Project Page: https://elvishelvis.github.io/papers/lca/.

cs.LG↗
arXiv2024

Informational Size in School Choice

This paper introduces a novel measurement of informational size to school choice problems, which inherits its ideas from Mount and Reiter (1974). This concept measures a matching mechanism's information size by counting the maximal relevant preference and priority rankings to secure a certain pairwise assignment of a student to a school across all possible matching problems. Our analysis uncovers two key insights. First, the three prominent strategy-proof matching mechanisms, the deferred acceptance (DA) mechanism, the top trading cycles (TTC) mechanism, and the serial dictatorship (SD) mechanism, is (strictly) less informative than the non-strategy-proof immediate acceptance (IA) mechanism. This result highlights a previously omitted advantage of IA in term of its information demand, which partially explain the its popularity in real-world matching problems especially when acquiring information is both pecuniarily and cognitively costly. Second, when the matching problem contains at least four students, the TTC demands less information compared to the DA to implement a desired allocation. The issue of comparison between TTC and DA has puzzled researchers both in theory (Gonczarowski and Thomas, 2023) and in experiment (Hakimov and Kubler, 2021). Our result responds to this issue from an informational perspective: in experiments with relatively fewer students, agents tend to prefer DA over TTC as DA requires fewer information to secure one's allocation in all problems (Guillen and Veszteg, 2021), while the opposite is true when the market size increases (Pais et al., 2011). Among others, our informational size concept offers a new perspective to understand the differences in auditability (Grigoryan and Moller, 2024), manipulation vulnerability (Pathak and Sonmez, 2013), and privacy protection (Haupt and Hitzig, 2022), among some commonly used matching mechanisms.

econ.TH↗
arXiv2024

Axioms for Top Trading Cycles in Multi-Object Reallocation

This paper studies multi-object reallocation without monetary transfers, where agents initially own multiple indivisible objects and have strict preferences over bundles (e.g., shift exchange among workers at a firm). Focusing on marginal rules that elicit only rankings over individual objects, we provide axiomatic characterizations of the generalized Top Trading Cycles rule (TTC) on the lexicographic and responsive domains. On the lexicographic domain, TTC is characterized by balancedness, individual-good efficiency, the worst-endowment lower bound, and either truncation-proofness or drop strategy-proofness. On the responsive domain, TTC is the unique marginal rule satisfying individual-good efficiency, truncation-proofness, and either the worst-endowment lower bound or individual rationality. In the Shapley--Scarf housing market, TTC is characterized by Pareto efficiency, individual rationality, and truncation-proofness. Finally, on the conditionally lexicographic domain, the augmented Top Trading Cycles rule is characterized by balancedness, Pareto efficiency, the worst-endowment lower bound, and drop strategy-proofness. The conditionally lexicographic domain is a maximal domain on which Pareto efficiency coincides with individual-good efficiency.

econ.TH↗
arXiv2024

FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View

In autonomous driving, 3D occupancy prediction outputs voxel-wise status and semantic labels for more comprehensive understandings of 3D scenes compared with traditional perception tasks, such as 3D object detection and bird's-eye view (BEV) semantic segmentation. Recent researchers have extensively explored various aspects of this task, including view transformation techniques, ground-truth label generation, and elaborate network design, aiming to achieve superior performance. However, the inference speed, crucial for running on an autonomous vehicle, is neglected. To this end, a new method, dubbed FastOcc, is proposed. By carefully analyzing the network effect and latency from four parts, including the input image resolution, image backbone, view transformation, and occupancy prediction head, it is found that the occupancy prediction head holds considerable potential for accelerating the model while keeping its accuracy. Targeted at improving this component, the time-consuming 3D convolution network is replaced with a novel residual-like architecture, where features are mainly digested by a lightweight 2D BEV convolution network and compensated by integrating the 3D voxel features interpolated from the original image features. Experiments on the Occ3D-nuScenes benchmark demonstrate that our FastOcc achieves state-of-the-art results with a fast inference speed.

cs.CV↗
arXiv2023

Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving

LiDAR and camera are two critical sensors for multi-modal 3D semantic segmentation and are supposed to be fused efficiently and robustly to promise safety in various real-world scenarios. However, existing multi-modal methods face two key challenges: 1) difficulty with efficient deployment and real-time execution; and 2) drastic performance degradation under weak calibration between LiDAR and cameras. To address these challenges, we propose CPGNet-LCF, a new multi-modal fusion framework extending the LiDAR-only CPGNet. CPGNet-LCF solves the first challenge by inheriting the easy deployment and real-time capabilities of CPGNet. For the second challenge, we introduce a novel weak calibration knowledge distillation strategy during training to improve the robustness against the weak calibration. CPGNet-LCF achieves state-of-the-art performance on the nuScenes and SemanticKITTI benchmarks. Remarkably, it can be easily deployed to run in 20ms per frame on a single Tesla V100 GPU using TensorRT TF16 mode. Furthermore, we benchmark performance over four weak calibration levels, demonstrating the robustness of our proposed approach.

cs.CV↗
arXiv2023

Efficiency in Multiple-Type Housing Markets

We consider multiple-type housing markets (Moulin, 1995), which extend Shapley-Scarf housing markets (Shapley and Scarf, 1974) from one dimension to higher dimensions. In this model, Pareto efficiency is incompatible with individual rationality and strategy-proofness (Konishi et al., 2001). Therefore, we consider two weaker efficiency properties: coordinatewise efficiency and pairwise efficiency. We show that these two properties both (i) are compatible with individual rationality and strategy-proofness, and (ii) help us to identify two specific mechanisms. To be more precise, on various domains of preference profiles, together with other well-studied properties (individual rationality, strategy-proofness, and non-bossiness), coordinatewise efficiency and pairwise efficiency respectively characterize two extensions of the top-trading-cycles mechanism (TTC): the coordinatewise top-trading-cycles mechanism (cTTC) and the bundle top-trading-cycles mechanism (bTTC). Moreover, we propose several variations of our efficiency properties, and we find that each of them is either satisfied by cTTC or bTTC, or leads to an impossibility result (together with individual rationality and strategy-proofness). Therefore, our characterizations can be primarily interpreted as a compatibility test: any reasonable efficiency property that is not satisfied by cTTC or bTTC could be considered incompatible with individual rationality and strategy-proofness. The external validity of our results in the context of general environments is also discussed. For multiple-type housing markets with strict preferences, our characterization of bTTC constitutes the first characterization of an extension of the prominent TTC mechanism

econ.TH↗
arXiv2023

Priors are Powerful: Improving a Transformer for Multi-camera 3D Detection with 2D Priors

Transfomer-based approaches advance the recent development of multi-camera 3D detection both in academia and industry. In a vanilla transformer architecture, queries are randomly initialised and optimised for the whole dataset, without considering the differences among input frames. In this work, we propose to leverage the predictions from an image backbone, which is often highly optimised for 2D tasks, as priors to the transformer part of a 3D detection network. The method works by (1). augmenting image feature maps with 2D priors, (2). sampling query locations via ray-casting along 2D box centroids, as well as (3). initialising query features with object-level image features. Experimental results shows that 2D priors not only help the model converge faster, but also largely improve the baseline approach by up to 12% in terms of average precision.

cs.CV↗