arXiv subjects

Zheming Liang

Publications and source records attributed to Zheming Liang.

10 recordsLinked to original sources

arXiv2026

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.

cs.CV↗

arXiv2026

AdaCodec: A Predictive Visual Code for Video MLLMs

Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a \emph{predictive visual code}, and instantiate it for video MLLMs as \textbf{AdaCodec}. AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens. Across all eleven benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-token budget. Even at $1/7$ the budget, AdaCodec with 32k tokens surpasses the 224k baseline on all long-video benchmarks; on five general-video benchmarks, it raises the average score while substantially cutting time-to-first-token from 9.26s to 1.62s.

cs.CV↗

arXiv2026

LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

Vision-Language Models (VLMs) have achieved substantial progress across a wide range of understanding and reasoning tasks, driven by large-scale image-text training aimed at multimodal fusion. Ideally, replacing a textual question with its rendered-image counterpart should leave model performance essentially unaffected. In practice, however, such modality substitution induces dramatic performance degradation. We attribute this "carrier sensitivity" issue to an inherent bias in current training corpora. Across prevalent datasets such as image captioning, VQA, OCR, and web-sourced interleaved data, text and images are typically organized into distinct and asymmetric roles, with text serving as linguistic queries and images as visual references. Such data bias leads VLMs to exhibit distinct preferences for information acquisition across different modalities. Consequently, VLMs fail to align representations of semantically equivalent content across textual and visual carriers, making model reasoning fragile under modality substitution. To address this, we propose Local Modality Substitution (LoMo), a lightweight, architecture-agnostic data curation paradigm designed to provide supervision for cross-modal representational invariance between semantically equivalent text and image carriers. LoMo achieves this by reformulating single-modality prompts into seamlessly interleaved multimodal sequences. It dynamically selects target text spans and recasts them as rendered images, thereby preserving the same semantics across "text, visual, text" carriers. Extensive experiments across 13 diverse multimodal benchmarks demonstrate that LoMo significantly improves overall multimodal reasoning and yields deeper cross-modal fusion. Specifically, it delivers consistent gains across foundational models, improving over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.

cs.CV↗

arXiv2026

Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

Recent paradigms in Random Projection Layer (RPL)-based continual representation learning have demonstrated superior performance when building upon a pre-trained model (PTM). These methods insert a randomly initialized RPL after a PTM to enhance feature representation in the initial stage. Subsequently, a linear classification head is used for analytic updates in the continual learning stage. However, under severe domain gaps between pre-trained representations and target domains, a randomly initialized RPL exhibits limited expressivity under large domain shifts. While largely scaling up the RPL dimension can improve expressivity, it also induces an ill-conditioned feature matrix, thereby destabilizing the recursive analytic updates of the linear head. To this end, we propose the Stochastic Continual Learner with MemoryGuard Supervisory Mechanism (SCL-MGSM). Unlike random initialization, MGSM constructs the projection layer via a principled, data-guided mechanism that progressively selects target-aligned random bases to adapt the PTM representation to downstream tasks. This facilitates the construction of a compact yet expressive RPL while improving the numerical stability of analytic updates. Extensive experiments on multiple exemplar-free Class Incremental Learning (CIL) benchmarks demonstrate that SCL-MGSM achieves superior performance compared to state-of-the-art methods.

cs.LG↗

arXiv2025

UniREditBench: A Unified Reasoning-based Image Editing Benchmark

Recent advances in multi-modal generative models have driven substantial improvements in image editing. However, current generative models still struggle with handling diverse and complex image editing tasks that require implicit reasoning, underscoring the need for a comprehensive benchmark to systematically assess their performance across various reasoning scenarios. Existing benchmarks primarily focus on single-object attribute transformation in realistic scenarios, which, while effective, encounter two key challenges: (1) they largely overlook multi-object interactions as well as game-world scenarios that involve human-defined rules, which are common in real-life applications; (2) they only rely on textual references to evaluate the generated images, potentially leading to systematic misjudgments, especially in complex reasoning scenarios. To this end, this work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation. It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions. To improve evaluation reliability, we introduce multimodal dual-reference evaluation, providing both textual and ground-truth image references for each sample assessment. Furthermore, we design an automated multi-scenario data synthesis pipeline and construct UniREdit-Data-100K, a large-scale synthetic dataset with high-quality chain-of-thought (CoT) reasoning annotations. We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings. Through thorough benchmarking of both open-source and closed-source image editing models, we reveal their strengths and weaknesses across various aspects.

cs.CV↗

arXiv2025

RLFR: Extending Reinforcement Learning for LLMs with Flow Environment

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a promising framework for improving reasoning abilities in Large Language Models (LLMs). However, policy optimized with binary verification prone to overlook potential valuable exploration in reasoning trajectory. In view of heavy annotation cost of golden Process Reward Models (PRMs), recent works attempt using auxiliary signals for reward shaping of process tokens, involving entropy and likelihood collected from logit space. In this work, we offer a novel perspective on shaping RLVR with flow rewards derived from latent space, and propose RLFR, where the flow fields of model latents are constructed from either off-policy high-quality data and on-policy rejection sampling data, and the velocity deviations of policy latents within it are quantified to serve as a reward signal. RLFR first demonstrates that a well-established flow field can be a sound environment for reward signal collection, highlighting the expressive latent space is much underexplored. Moreover, RLFR is able to compress any off-policy expert data as reference for constituting reward signals, and we show that the efficient context dependence compressed within the hidden states are utilized, rather than individual token-level denotation for context comprehending. Experiments on both language and multimodal reasoning benchmarks demonstrate the reliability of flow rewards, and suggesting a promising paradigm for reward shaping with auxiliary signals.

cs.LG↗

arXiv2019

Adaptive Robust Energy Management Strategy for Campus-Based Commercial Buildings Considering Comprehensive Comfort Levels

Neglecting consumers' comfort always leads to failure or slow-response to demand response request. In this paper, we propose several comprehensive comfort level models for various appliances in campus-based commercial buildings (CBs). The objective of the proposed system is to minimize O\&M costs of campus-based CBs and maximize various comfort levels simultaneously under the worst-case scenarios. Adaptive robust optimization (ARO) is leveraged to handle various uncertainties within the proposed system: (i) demand response signals sending from the distribution system operator (DSO); (ii) arrival state-of-charge (SoC) conditions of plug-in electric vehicles (PEVs); (iii) power outputs of renewable energy sources (RESs); and (iv) load demand of other appliances. Benders decomposition, such as column-and-constraint generation (C\&CG) algorithm, is used to solve the reformulated NP-hard min-max problem. Extensive simulation results demonstrate the effectiveness of the proposed optimal energy management strategy for campus-based CBs in both minimizing O\&M costs and maximizing comprehensive comfort levels.

math.OC↗

arXiv2018

Optimal Energy Management for Commercial Buildings Considering Comprehensive Comfort Levels in a Retail Electricity Market

Demand response has been implemented by distribution system operators to reduce peak demand and mitigate contingency issues on distribution lines and substations. Specifically, the campus based commercial buildings make the major contributions to peak demand in a distribution system. Note that prior works neglect the consumers comfort level in performing demand response, which limits their applications as the incentives are not worth as compared to the loss in comfort levels for most time. Thus, a framework to comprehensively consider both operating costs and comfort levels is necessary. Moreover, distributed energy resources are widely deployed in commercial buildings such as roof top solar panels, plug in electric vehicles, and energy storage units, which bring various uncertainties to the distribution systems, i.e., output of renewable; electricity prices; arrival and departure of plug-in electric vehicles; business hour demand response signals and flexible energy demand. In this paper, we propose an optimal demand response framework to enable local control of demand-side appliances that are usually too small to participate in a retail electricity market. Several typical small demand side appliances, i.e., heating, ventilation, and air conditioning systems, electric water heaters and plug-in electric vehicles, are considered in our proposed model. Their operations are coordinated by a central controller, whose objective is to minimize the total cost and maximize the customers comfort levels for multiple commercial buildings. A scenario-based stochastic programming is leveraged to handle the aforementioned uncertainties. Numerical results based on the practical data demonstrate the effectiveness of the proposed framework. In addition, the trade off between the operation costs of commercial buildings and customers comfort levels is illustrated.

math.OC↗

arXiv2018

Smart Charging and Parking of Plug-in Hybrid Electric Vehicles in Microgrids Considering Renewable Energy Sources

With the current trend of transforming a centralized power system into a decentralized one for efficiency, reliability, and environment reasons, the concept of microgrid that integrates a variety of distributed energy resources (DERs) on the distribution network is gaining popularity. In this paper, we investigate the smart charging and parking of plug-in hybrid electric vehicles (PHEVs) in microgrids with renewable energy sources (RES), such as solar panels, in grid-connected mode. To address the uncertainties associated with RES power output and PHEVs charging condition in the microgrid, we propose a two-stage scenario-based stochastic optimization approach with the objective of providing a proper scheduling for parking and charging of PHEVs that minimizes the average total operating cost while maintaining the reliability of the microgrid. A case study is conducted to show the effectiveness of the proposed approach. Extensive simulation results show that the microgrid can minimize the operating cost and ensure its reliability.

math.OC↗

arXiv2017

Intelligent Home Energy Management System for Distributed Renewable Generators, Dispatchable Residential Loads and Distributed Energy Storage Devices

This paper presents an intelligent home energy management system integrated with dispatchable loads (e.g., clothes washers and dryers), distributed renewable generators (e.g., roof-top solar panels), and distributed energy storage devices (e.g., plug-in electric vehicles). The overall goal is to reduce the total operating costs and the carbon emissions for a future residential house, while satisfying the end-users comfort levels. This paper models a wide variety of home appliances and formulates the economic operation problem using mixed integer linear programming. Case studies are performed to validate and demonstrate the effectiveness of the proposed solution algorithm. Simulation results also show the positive impact of dispatchable loads, distributed renewable generators, and distributed energy storage devices on a future residential house.

math.OC↗