arXiv subjects

Jiaotuan Wang

Publications and source records attributed to Jiaotuan Wang.

4 recordsLinked to original sources

arXiv2026

Who Leads Now? Token-Level Modality Arbitration for Chart-to-Code Generation

Chart-to-code generation requires a model to read the fine-grained visual details of a chart and write executable code that reproduces it. Existing chart-to-code methods either train visual and coding abilities separately, or fine-tune on chart-to-code data with the two abilities entangled. Neither strategy accounts for the distinct nature of the two abilities or the interference that arises when they are optimized together. We propose MoCA (Mixture of Cross-modal Arbitration), which separates the two abilities rather than blending them. MoCA is built on Cross-modal Arbitration Block (CAB), which maintains a visual branch and a code branch as two distinct pathways, and a lightweight arbiter that arbitrates their relative contributions at every layer and generated token. We train MoCA in two stages: a supervised warm-up on self-distilled reasoning trajectories that decomposes visual understanding into explicit steps, followed by reinforcement learning with rewards on both the reasoning process and the final code. Analysis shows that the arbiter learns structured rather than arbitrary allocations, with expert contributions varying systematically across tokens, layers, and instances. Across three benchmarks, MoCA delivers competitive performance against general-domain and chart-specialized models. Ablation results show that the gains cannot be attributed to a larger model size alone, but instead arise from the joint contributions of complementary visual and code branch initialization and input-conditioned arbitration through CAB.

cs.AI↗

arXiv2024

Enhancing DETRs Variants through Improved Content Query and Similar Query Aggregation

The design of the query is crucial for the performance of DETR and its variants. Each query consists of two components: a content part and a positional one. Traditionally, the content query is initialized with a zero or learnable embedding, lacking essential content information and resulting in sub-optimal performance. In this paper, we introduce a novel plug-and-play module, Self-Adaptive Content Query (SACQ), to address this limitation. The SACQ module utilizes features from the transformer encoder to generate content queries via self-attention pooling. This allows candidate queries to adapt to the input image, resulting in a more comprehensive content prior and better focus on target objects. However, this improved concentration poses a challenge for the training process that utilizes the Hungarian matching, which selects only a single candidate and suppresses other similar ones. To overcome this, we propose a query aggregation strategy to cooperate with SACQ. It merges similar predicted candidates from different queries, easing the optimization. Our extensive experiments on the COCO dataset demonstrate the effectiveness of our proposed approaches across six different DETR's variants with multiple configurations, achieving an average improvement of over 1.0 AP.

cs.CV↗

arXiv2024

CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM

Automatic Chinese classical poetry generation has attracted much research interest, but achieving effective control over format and content simultaneously remains challenging. Traditional systems usually accept keywords as user inputs, resulting in limited control over content. Large language models (LLMs) improve content control by allowing unrestricted user instructions, but the token-by-token generation process frequently makes format errors. Motivated by this, we propose CharPoet, a Chinese classical poetry generation system based on token-free LLM, which provides effective control over both format and content. Our token-free architecture generates in a character-by-character manner, enabling precise control over the number of characters. Pruned from existing token-based LLMs, CharPoet inherits their pretrained capabilities and can generate poetry following instructions like "Write me a poem for my mother's birthday." CharPoet achieves format accuracy above 0.96, outperforming Jiuge-GPT-2 (0.91) and GPT-4 (0.38). In terms of content quality, CharPoet surpasses traditional systems including Jiuge, and is comparable to other LLMs. Our system is open source and available at https://modelscope.cn/models/CharPoet/CharPoet. A video demonstration of CharPoet is available at https://youtu.be/voZ25qEp3Dc.

cs.CL↗

arXiv2024

Multimodal Urban Areas of Interest Generation via Remote Sensing Imagery and Geographical Prior

Urban area-of-interest (AOI) refers to an integrated urban functional zone with defined polygonal boundaries. The rapid development of urban commerce has led to increasing demands for highly accurate and timely AOI data. However, existing research primarily focuses on coarse-grained functional zones for urban planning or regional economic analysis, and often neglects the expiration of AOI in the real world. They fail to fulfill the precision demands of Mobile Internet Online-to-Offline (O2O) businesses. These businesses require accuracy down to a specific community, school, or hospital. In this paper, we propose a comprehensive end-to-end multimodal deep learning framework designed for simultaneously detecting accurate AOI boundaries and validating the reliability of AOI by leveraging remote sensing imagery coupled with geographical prior, titled AOITR. Unlike conventional AOI generation methods, such as the Road-cut method that segments road networks at various levels, our approach diverges from semantic segmentation algorithms that depend on pixel-level classification. Instead, our AOITR begins by selecting a point-of-interest (POI) of specific category, and uses it to retrieve corresponding remote sensing imagery and geographical prior such as entrance POIs and road nodes. This information helps to build a multimodal detection model based on transformer encoder-decoder architecture to regress the AOI polygon. Additionally, we utilize the dynamic features from human mobility, nearby POIs, and logistics addresses for AOI reliability evaluation via a cascaded network module. The experimental results reveal that our algorithm achieves a significant improvement on Intersection over Union (IoU) metric, surpassing previous methods by a large margin.

cs.CV↗