arXiv subjects

Sheng Jin

Publications and source records attributed to Sheng Jin.

At least 37 records · Page 2Linked to original sources

arXiv2025

EMNLP: Educator-role Moral and Normative Large Language Models Profiling

Simulating Professions (SP) enables Large Language Models (LLMs) to emulate professional roles. However, comprehensive psychological and ethical evaluation in these contexts remains lacking. This paper introduces EMNLP, an Educator-role Moral and Normative LLMs Profiling framework for personality profiling, moral development stage measurement, and ethical risk under soft prompt injection. EMNLP extends existing scales and constructs 88 teacher-specific moral dilemmas, enabling profession-oriented comparison with human teachers. A targeted soft prompt injection set evaluates compliance and vulnerability in teacher SP. Experiments on 14 LLMs show teacher-role LLMs exhibit more idealized and polarized personalities than human teachers, excel in abstract moral reasoning, but struggle with emotionally complex situations. Models with stronger reasoning are more vulnerable to harmful prompt injection, revealing a paradox between capability and safety. The model temperature and other hyperparameters have limited influence except in some risk behaviors. This paper presents the first benchmark to assess ethical and psychological alignment of teacher-role LLMs for educational AI. Resources are available at https://e-m-n-l-p.github.io/.

cs.CL↗

arXiv2025

COIG-Writer: A High-Quality Dataset for Chinese Creative Writing with Thought Processes

Large language models exhibit systematic deficiencies in creative writing, particularly in non-English contexts where training data is scarce and lacks process-level supervision. We present COIG-Writer, a novel Chinese creative writing dataset that captures both diverse outputs and their underlying thought processes through systematic reverse-engineering of high-quality texts. Unlike existing datasets that provide only input-output pairs, COIG-Writer comprises 1,665 meticulously curated triplets spanning 51 genres, each containing: (1) a reverse-engineered prompt, (2) detailed creative reasoning documenting decision-making processes, and (3) the final text. Through comprehensive experiments, we identify a two-component model of creative writing: narrative logic (provided by process supervision) and linguistic expression (maintained by general-purpose data). Our findings reveal three critical insights: (1) Process supervision is highly effective but requires stabilization with general data. A ratio of at least one creative sample to twelve general samples is needed to achieve optimal performance; below this threshold, the win rate progressively degrades (from 62.75% down to 35.78%)., (2) creative capabilities are culturally-bound with no cross-lingual transfer (89.26pp gap between Chinese and English performance), and (3) lexical diversity inversely correlates with creative quality (TTR paradox), suggesting high diversity signals compensatory behavior for logical deficiencies. These findings establish that creative excellence emerges from the interaction between logical scaffolding and linguistic grounding, analogous to how mathematical reasoning enhances but cannot replace linguistic competence in foundation models.

cs.CL↗

arXiv2025

Evolution in Simulation: AI-Agent School with Dual Memory for High-Fidelity Educational Dynamics

Large language models (LLMs) based Agents are increasingly pivotal in simulating and understanding complex human systems and interactions. We propose the AI-Agent School (AAS) system, built around a self-evolving mechanism that leverages agents for simulating complex educational dynamics. Addressing the fragmented issues in teaching process modeling and the limitations of agents performance in simulating diverse educational participants, AAS constructs the Zero-Exp strategy, employs a continuous "experience-reflection-optimization" cycle, grounded in a dual memory base comprising experience and knowledge bases and incorporating short-term and long-term memory components. Through this mechanism, agents autonomously evolve via situated interactions within diverse simulated school scenarios. This evolution enables agents to more accurately model the nuanced, multi-faceted teacher-student engagements and underlying learning processes found in physical schools. Experiment confirms that AAS can effectively simulate intricate educational dynamics and is effective in fostering advanced agent cognitive abilities, providing a foundational stepping stone from the "Era of Experience" to the "Era of Simulation" by generating high-fidelity behavioral and interaction data.

cs.AI↗

arXiv2025

KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model

The emergence of Multimodal Large Language Models (MLLMs) has revolutionized image understanding by bridging textual and visual modalities. However, these models often struggle with capturing fine-grained semantic information, such as the precise identification and analysis of object keypoints. Keypoints, as structure-aware, pixel-level, and compact representations of objects, particularly articulated ones, play a crucial role in applications such as fine-grained image analysis, object retrieval, and behavior recognition. In this paper, we propose KptLLM++, a novel multimodal large language model that specifically designed for generic keypoint comprehension through the integration of diverse input modalities guided by user-defined instructions. By unifying keypoint detection across varied contexts, KptLLM++ establishes itself as an advanced interface, fostering more effective human-AI collaboration. The model is built upon a novel identify-then-detect paradigm, which first interprets keypoint semantics and subsequently localizes their precise positions through a structured chain-of-thought reasoning mechanism. To push the boundaries of performance, we have scaled up the training dataset to over 500K samples, encompassing diverse objects, keypoint categories, image styles, and scenarios with complex occlusions. This extensive scaling enables KptLLM++ to unlock its potential, achieving remarkable accuracy and generalization. Comprehensive experiments on multiple keypoint detection benchmarks demonstrate its state-of-the-art performance, underscoring its potential as a unified solution for fine-grained image understanding and its transformative implications for human-AI interaction.

cs.CV↗

arXiv2025

OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation

In this report, we present OpenUni, a simple, lightweight, and fully open-source baseline for unifying multimodal understanding and generation. Inspired by prevailing practices in unified model learning, we adopt an efficient training strategy that minimizes the training complexity and overhead by bridging the off-the-shelf multimodal large language models (LLMs) and diffusion models through a set of learnable queries and a light-weight transformer-based connector. With a minimalist choice of architecture, we demonstrate that OpenUni can: 1) generate high-quality and instruction-aligned images, and 2) achieve exceptional performance on standard benchmarks such as GenEval, DPG- Bench, and WISE, with only 1.1B and 3.1B activated parameters. To support open research and community advancement, we release all model weights, training code, and our curated training datasets (including 23M image-text pairs) at https://github.com/wusize/OpenUni.

cs.CV↗

arXiv2025

First systematic experimental 2D mapping of linearly polarized $γ$-ray polarimetric distribution in relativistic Compton scattering

The interaction of photons with relativistic electrons constitutes a fundamental electromagnetic process whose polarization transfer mechanics remain incompletely characterized. We report the first systematic measurement of spatial polarization distribution for $γ$-rays generated via \SI{45}{\degree} slant inverse Compton scattering (ICS) between linearly polarized \SI{0.117}{\eV} photons and \SI{3.5}{\GeV} electrons, performing full 2D mapping of intensity, polarization angle (AOP), and degree of polarization (DOP). Measurements reveal an asymmetric beam profile along the laser's polarization direction that resembles \SI{180}{\degree} backward ICS observations. The central beam region exhibits DOP $\approx$ 1.0 with AOP rigidly aligned at \SI{45}{\degree}, while peripheral regions display complex non-uniform polarization distributions. These findings confirm quantum electrodynamics predictions of near-complete polarization transfer along the beam axis in slant geometries, thus establishing slant scattering as a viable alternative to head-on configurations for generating high DOP $γ$-rays.

nucl-ex↗

arXiv2025

The Narrow Formation Pathway of Hot Saturns: Constraints on Initial Planetary Properties

The observed exoplanet population exhibits a scarcity of short-period Saturn-mass planets, a phenomenon referred to as the ``hot Saturn desert". This observational scarcity can be utilized to validate the theories regarding the formation and evolution of gas planets. In this study, we conduct large-scale numerical simulations to explore how the initial conditions of gas planets orbiting solar-type and M-dwarf stars influence their evolutionary trajectories in the semi-major axis versus planetary radius ($a$-$R$) parameter space. We generate a synthetic population of 10,000 short-period gaseous planets by systematically varying their initial planetary masses ($M_{\rm p}$), initial planetary luminosities ($L_{\rm p}$), initial core mass fractions ($f_{\rm core}$), and semi-major axis ($a$). Furthermore, we assume these gaseous planets have ceased orbital migration and model their long-term thermal evolution, taking into account the impacts of atmospheric evaporation. Our results show that the initial mass, $L_{\rm p}$, and $f_{\rm core}$ are the dominant factors controlling radius evolution for short-period gas planets. The key to survival as a hot Saturn analogue appears to be having just the right combination of properties after gas disk dissipation: an $M_{\rm p}$ below 0.5 Jupiter Mass ($M_{\rm Jup}$), a substantial $f_{\rm core}$ of $\geq$ 30%, and relatively low $L_{\rm p}$ on the order of $10^{-6}$ solar luminosity ($L_{\odot}$) or less. The survival criteria for hot Saturn analogs align with theoretically unfavorable initial conditions of gas planets formed via core accretion scenario, naturally explaining the observed boundaries of the hot Saturn desert.

astro-ph.EP↗

arXiv2025

Harmonizing Visual Representations for Unified Multimodal Understanding and Generation

Unifying visual understanding and generation within a single multimodal framework remains a significant challenge, as the two inherently heterogeneous tasks require representations at different levels of granularity. Current approaches that utilize vector quantization (VQ) or variational autoencoders (VAE) for unified visual representation prioritize intrinsic imagery features over semantics, compromising understanding performance. In this work, we take inspiration from masked image modelling (MIM) that learns rich semantics via a mask-and-reconstruct pre-training and its successful extension to masked autoregressive (MAR) image generation. A preliminary study on the MAR encoder's representation reveals exceptional linear probing accuracy and precise feature response to visual concepts, which indicates MAR's potential for visual understanding tasks beyond its original generation role. Based on these insights, we present \emph{Harmon}, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder. Through a three-stage training procedure that progressively optimizes understanding and generation capabilities, Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks while matching the performance of methods with dedicated semantic encoders (e.g., Janus) on image understanding benchmarks. Our code and models will be available at https://github.com/wusize/Harmon.

cs.CV↗

arXiv2025

F-LMM: Grounding Frozen Large Multimodal Models

Endowing Large Multimodal Models (LMMs) with visual grounding capability can significantly enhance AIs' understanding of the visual world and their interaction with humans. However, existing methods typically fine-tune the parameters of LMMs to learn additional segmentation tokens and overfit grounding and segmentation datasets. Such a design would inevitably cause a catastrophic diminution in the indispensable conversational capability of general AI assistants. In this paper, we comprehensively evaluate state-of-the-art grounding LMMs across a suite of multimodal question-answering benchmarks, observing drastic performance drops that indicate vanishing general knowledge comprehension and weakened instruction following ability. To address this issue, we present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations -- a straightforward yet effective design based on the fact that word-pixel correspondences conducive to visual grounding inherently exist in the attention mechanism of well-trained LMMs. Using only a few trainable CNN layers, we can translate word-pixel attention weights to mask logits, which a SAM-based mask refiner can further optimise. Our F-LMM neither learns special segmentation tokens nor utilises high-quality grounded instruction-tuning data, but achieves competitive performance on referring expression segmentation and panoptic narrative grounding benchmarks while completely preserving LMMs' original conversational ability. Additionally, with instruction-following ability preserved and grounding ability obtained, F-LMM can be directly applied to complex tasks like reasoning segmentation, grounded conversation generation and visual chain-of-thought reasoning. Our code can be found at https://github.com/wusize/F-LMM.

cs.CV↗

arXiv2025

Frame-Voyager: Learning to Query Frames for Video Large Language Models

Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

cs.CV↗

arXiv2025

The heating efficiency of hot Jupiters from a data-driven perspective

The inflated radii of hot Jupiters have been explored by various theoretical mechanisms. By connecting planetary thermal evolution models with the observed properties of hot Jupiters using hierarchical Bayesian models, a theoretical parameter called the heating efficiency has been introduced to describe the heating of the interiors of these planets. Previous studies have shown that the marginal distribution of this heating efficiency parameter has a single-peak distribution along the planetary equilibrium temperature (Teq). Since the foundation of these Bayesian inference models are the observed properties of hot Jupiters, there must be a corresponding feature in the observed data that leads to the inferred single-peak distribution of the heating efficiency. This study aims to find the underlying cause of the single-peak heating efficiency distribution without relying on specific theoretical models. By analyzing the relationship between different observed physical properties, we obtained a similar single-peak distribution of the radius expansion efficiency of hot Jupiters along Teq, which can be explained by the correlation with the stellar effective temperature. However, a detailed investigation suggests that this single-peak distribution is actually the result of straightforward physical processes. Specifically, the increase in heating efficiency can be attributed to the increase in incident stellar flux, while the decrease in heating efficiency can be attributed to the rise in gravitational binding energy associated with the increase in planetary mass.

astro-ph.EP↗

arXiv2024

AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks

Automated machine learning (AutoML) is a collection of techniques designed to automate the machine learning development process. While traditional AutoML approaches have been successfully applied in several critical steps of model development (e.g. hyperparameter optimization), there lacks a AutoML system that automates the entire end-to-end model production workflow for computer vision. To fill this blank, we propose a novel request-to-model task, which involves understanding the user's natural language request and execute the entire workflow to output production-ready models. This empowers non-expert individuals to easily build task-specific models via a user-friendly language interface. To facilitate development and evaluation, we develop a new experimental platform called AutoMMLab and a new benchmark called LAMP for studying key components in the end-to-end request-to-model pipeline. Hyperparameter optimization (HPO) is one of the most important components for AutoML. Traditional approaches mostly rely on trial-and-error, leading to inefficient parameter search. To solve this problem, we propose a novel LLM-based HPO algorithm, called HPO-LLaMA. Equipped with extensive knowledge and experience in model hyperparameter tuning, HPO-LLaMA achieves significant improvement of HPO efficiency. Dataset and code are available at https://github.com/yang-ze-kang/AutoMMLab.

cs.LG↗

arXiv2024

NADER: Neural Architecture Design via Multi-Agent Collaboration

Designing effective neural architectures poses a significant challenge in deep learning. While Neural Architecture Search (NAS) automates the search for optimal architectures, existing methods are often constrained by predetermined search spaces and may miss critical neural architectures. In this paper, we introduce NADER (Neural Architecture Design via multi-agEnt collaboRation), a novel framework that formulates neural architecture design (NAD) as a LLM-based multi-agent collaboration problem. NADER employs a team of specialized agents to enhance a base architecture through iterative modification. Current LLM-based NAD methods typically operate independently, lacking the ability to learn from past experiences, which results in repeated mistakes and inefficient exploration. To address this issue, we propose the Reflector, which effectively learns from immediate feedback and long-term experiences. Additionally, unlike previous LLM-based methods that use code to represent neural architectures, we utilize a graph-based representation. This approach allows agents to focus on design aspects without being distracted by coding. We demonstrate the effectiveness of NADER in discovering high-performing architectures beyond predetermined search spaces through extensive experiments on benchmark tasks, showcasing its advantages over state-of-the-art methods. The codes will be released soon.

cs.CV↗

arXiv2024

Circular-shift-based Vector Linear Network Coding and Its Application to Array Codes

Circular-shift linear network coding (LNC) is a class of vector LNC with local encoding kernels selected from cyclic permutation matrices, so that it has low coding complexities. However, it is insufficient to exactly achieve the capacity of a multicast network, so the data units transmitted along the network need to contain redundant symbols, which affects the transmission efficiency. In this paper, as a variation of circular-shift LNC, we introduce a new class of vector LNC over arbitrary GF($p$), called circular-shift-based vector LNC, which is shown to be able to exactly achieve the capacity of a multicast network. The set of local encoding kernels in circular-shift-based vector LNC is nontrivially designed such that it is closed under multiplication by elements in itself. It turns out that the coding complexity of circular-shift-based vector LNC is comparable to and, in some cases, lower than that of circular-shift LNC. The new results in circular-shift-based vector LNC further facilitates us to characterize and design Vandermonde circulant maximum distance separable (MDS) array codes, which are built upon the structure of Vandermonde matrices and circular-shift operations. We prove that for $r \geq 2$, the largest possible $k$ for an $L$-dimensional $(k+r, k)$ Vandermonde circulant $p$-ary MDS array code is $p^{m_L}-1$, where $L$ is an integer co-prime with $p$, and $m_L$ represents the multiplicative order of $p$ modulo $L$. For $r = 2, 3$, we introduce two new types of $(k+r, k)$ $p$-ary array codes that achieves the largest $k = p^{m_L}-1$. For the special case that $p = 2$, we propose scheduling encoding algorithms for the 2 new codes, so that the encoding complexity not only asymptotically approaches the optimal $2$ XORs per original data bit, but also slightly outperforms the encoding complexity of other known Vandermonde circulant MDS array codes with $k = p^{m_L}-1$.

cs.IT↗

arXiv2024

Ultra-High Resolution Segmentation via Boundary-Enhanced Patch-Merging Transformer

Segmentation of ultra-high resolution (UHR) images is a critical task with numerous applications, yet it poses significant challenges due to high spatial resolution and rich fine details. Recent approaches adopt a dual-branch architecture, where a global branch learns long-range contextual information and a local branch captures fine details. However, they struggle to handle the conflict between global and local information while adding significant extra computational cost. Inspired by the human visual system's ability to rapidly orient attention to important areas with fine details and filter out irrelevant information, we propose a novel UHR segmentation method called Boundary-enhanced Patch-merging Transformer (BPT). BPT consists of two key components: (1) Patch-Merging Transformer (PMT) for dynamically allocating tokens to informative regions to acquire global and local representations, and (2) Boundary-Enhanced Module (BEM) that leverages boundary information to enrich fine details. Extensive experiments on multiple UHR image segmentation benchmarks demonstrate that our BPT outperforms previous state-of-the-art methods without introducing extra computational overhead. Codes will be released to facilitate research.

cs.CV↗

arXiv2024

ESOD: Efficient Small Object Detection on High-Resolution Images

Enlarging input images is a straightforward and effective approach to promote small object detection. However, simple image enlargement is significantly expensive on both computations and GPU memory. In fact, small objects are usually sparsely distributed and locally clustered. Therefore, massive feature extraction computations are wasted on the non-target background area of images. Recent works have tried to pick out target-containing regions using an extra network and perform conventional object detection, but the newly introduced computation limits their final performance. In this paper, we propose to reuse the detector's backbone to conduct feature-level object-seeking and patch-slicing, which can avoid redundant feature extraction and reduce the computation cost. Incorporating a sparse detection head, we are able to detect small objects on high-resolution inputs (e.g., 1080P or larger) for superior performance. The resulting Efficient Small Object Detection (ESOD) approach is a generic framework, which can be applied to both CNN- and ViT-based detectors to save the computation and GPU memory costs. Extensive experiments demonstrate the efficacy and efficiency of our method. In particular, our method consistently surpasses the SOTA detectors by a large margin (e.g., 8% gains on AP) on the representative VisDrone, UAVDT, and TinyPerson datasets. Code is available at https://github.com/alibaba/esod.

cs.CV↗

arXiv2024

Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions

The key to OOD detection has two aspects: generalized feature representation and precise category description. Recently, vision-language models such as CLIP provide significant advances in both two issues, but constructing precise category descriptions is still in its infancy due to the absence of unseen categories. This work introduces two hierarchical contexts, namely perceptual context and spurious context, to carefully describe the precise category boundary through automatic prompt tuning. Specifically, perceptual contexts perceive the inter-category difference (e.g., cats vs apples) for current classification tasks, while spurious contexts further identify spurious (similar but exactly not) OOD samples for every single category (e.g., cats vs panthers, apples vs peaches). The two contexts hierarchically construct the precise description for a certain category, which is, first roughly classifying a sample to the predicted category and then delicately identifying whether it is truly an ID sample or actually OOD. Moreover, the precise descriptions for those categories within the vision-language framework present a novel application: CATegory-EXtensible OOD detection (CATEX). One can efficiently extend the set of recognizable categories by simply merging the hierarchical contexts learned under different sub-task settings. And extensive experiments are conducted to demonstrate CATEX's effectiveness, robustness, and category-extensibility. For instance, CATEX consistently surpasses the rivals by a large margin with several protocols on the challenging ImageNet-1K dataset. In addition, we offer new insights on how to efficiently scale up the prompt engineering in vision-language models to recognize thousands of object categories, as well as how to incorporate large language models (like GPT-3) to boost zero-shot applications. Code is publicly available at https://github.com/alibaba/catex.

cs.CV↗

arXiv2024

Enhanced Stability in Planetary Systems with Similar Masses

This study employs numerical simulations to explore the relationship between the dynamical instability of planetary systems and the uniformity of planetary masses within the system, quantified by the Gini index. Our findings reveal a significant correlation between system stability and mass uniformity. Specifically, planetary systems with higher mass uniformity demonstrate increased stability, particularly when they are distant from first-order mean motion resonances (MMRs). In general, for non-resonant planetary systems with a constant total mass, non-equal mass systems are less stable than equal mass systems for a given spacing in units of mutual Hill radius. This instability may arise from the equipartition of the total random energy, which can lead to higher eccentricities in smaller planets, ultimately destabilizing the system. This work suggests that the observed mass uniformity within multi-planet systems detected by \textit{Kepler} may result from a combination of survival bias and ongoing dynamical evolution processes.

astro-ph.EP↗