arXiv2026
Whether large language models can assess the innovativeness of academic publications matters for literature reviews, research funding screening, and academic recommendations. We introduce the Scientific Research Taste Benchmark (SciTaste), a gold-standard benchmark covering six disciplines: physics, mathematics, chemistry, biology, medicine, and finance. SciTaste measures research taste, that is, the ability to judge a paper's scientific value and potential, of which innovativeness is one of seven dimensions. It integrates metadata from NASA ADS, OpenAlex, and arXiv, aligns papers to field-specific classification systems, and scores each paper with the seven-dimensional Academic Taste Index. Dimension weights come from a bi-objective machine-learning approach (five regressors, including random forests) and are shared across all six disciplines. The benchmark includes 300 selected papers (50 per discipline) and a backup pool of 5,700 papers (950 per field) covering 1996-2023. SciTaste supports four evaluation modes: pure_qa, native_search, our structured tool-calling agent DiM (9 bibliometric tools in ADS/OpenAlex), and dimagent_v3_native. Outputs are evaluated by two complementary schemes: correlation with dimension-level accuracy, and WAPE relative error. Across ten state-of-the-art LLMs, tool-based structured evaluation generally outperformed toolless and raw web search modes, though gains were model-dependent. Physics is easiest; chemistry, biology, and medicine remain more challenging. SciTaste supplies the value-judgment substrate auto-research systems need to screen, rank, and prioritize candidate ideas. Dataset, harness, and scoring code are available on Hugging Face at https://huggingface.co/datasets/skylenage/SciTaste-benchmark.