arXiv subjects

Jia Bu

Publications and source records attributed to Jia Bu.

4 recordsLinked to original sources

arXiv2026

SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence

We introduce SciEvalKit, a unified benchmarking toolkit designed to evaluate AI models for science across a broad range of scientific disciplines and task capabilities. Unlike general-purpose evaluation platforms, SciEvalKit focuses on the core competencies of scientific intelligence, including Scientific Multimodal Perception, Scientific Multimodal Reasoning, Scientific Multimodal Understanding, Scientific Symbolic Reasoning, Scientific Code Generation, Science Hypothesis Generation and Scientific Knowledge Understanding. It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science. SciEvalKit builds a foundation of expert-grade scientific benchmarks, curated from real-world, domain-specific datasets, ensuring that tasks reflect authentic scientific challenges. The toolkit features a flexible, extensible evaluation pipeline that enables batch evaluation across models and datasets, supports custom model and dataset integration, and provides transparent, reproducible, and comparable results. By bridging capability-based evaluation and disciplinary diversity, SciEvalKit offers a standardized yet customizable infrastructure to benchmark the next generation of scientific foundation models and intelligent agents. The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.

cs.AI↗

arXiv2025

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.

cs.AI↗

arXiv2025

Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning

Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.

cs.AI↗

arXiv2019

Constraints on axionlike particles with different magnetic field models from the PKS 2155-304 energy spectrum

Axion Like Particles (ALPs) are one promising kind of dark matter candidate particles that are predicted to couple with photons in the presence of magnetic fields. The oscillations between photons and ALPs travelling in the magnetic fields have been used to constrain ALP properties. In this work, we obtain some new constraints on the ALP mass $m_{\rm a}$ and the photon-ALP coupling constant $g$ with two different magnetic field models through TeV photons from PKS 2155-304. One is the discrete-$φ$ model that the magnetic field has the orientation angle $φ$ changes discretely and randomly from one coherent domain to the next, another is the linearly-continuous-$φ$ model that the magnetic field orientation angle $φ$ varies continuously across neighboring coherent domains. For the discrete-$φ$ model, we can obtain the best constraints on the ALP mass $m_{1}=m_{\rm a}/(1\ {\rm neV})=0.1$ and on the photon-ALP coupling constant $g_{11}=g/(10^{-11}\ {\rm GeV^{-1}})=5$, the reasonable range of the ALP mass $m_{1}$ is 0.08 $\thicksim$ 0.2 when $g_{11}$=5, and the only reasonable value of the photon-ALP coupling constant is $g_{11}$=5 when $m_{1}$=0.1. For the linearly-continuous-$φ$ model, we can obtain the best constraints on the ALP mass $m_{1}=0.1$ and on the photon-ALP coupling constant $g_{11}=0.7$, the reasonable range of the ALP mass $m_{1}$ is 0.05 $\thicksim$ 0.4 when $g_{11}$=0.7, and the reasonable range of the photon-ALP coupling constant $g_{11}$ is 0.5 $\thicksim$ 1 when $m_{1}$=0.1. All the results are consistent with the upper bound ($g<6.6\times10^{-11}\ {\rm GeV^{-1}}$, i.e. $g_{11}<6.6$) set by the CAST experiment.

astro-ph.HE↗