Searcharxiv⌕ Search

arXiv subjects

Zhiyuan Fan

Publications and source records attributed to Zhiyuan Fan.

54 records · Page 3Linked to original sources
arXiv2025

SedarEval: Automated Evaluation using Self-Adaptive Rubrics

The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach. We release our dataset at https://github.com/wwn1233/sedareval.

cs.CV↗
arXiv2024

Achieving Constant Regret in Linear Markov Decision Processes

We study the constant regret guarantees in reinforcement learning (RL). Our objective is to design an algorithm that incurs only finite regret over infinite episodes with high probability. We introduce an algorithm, Cert-LSVI-UCB, for misspecified linear Markov decision processes (MDPs) where both the transition kernel and the reward function can be approximated by some linear function up to misspecification level $ζ$. At the core of Cert-LSVI-UCB is an innovative \method, which facilitates a fine-grained concentration analysis for multi-phase value-targeted regression, enabling us to establish an instance-dependent regret bound that is constant w.r.t. the number of episodes. Specifically, we demonstrate that for a linear MDP characterized by a minimal suboptimality gap $Δ$, Cert-LSVI-UCB has a cumulative regret of $\tilde{\mathcal{O}}(d^3H^5/Δ)$ with high probability, provided that the misspecification level $ζ$ is below $\tilde{\mathcal{O}}(Δ/ (\sqrt{d}H^2))$. Here $d$ is the dimension of the feature space and $H$ is the horizon. Remarkably, this regret bound is independent of the number of episodes $K$. To the best of our knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.

cs.LG↗
arXiv2024

Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent

Contemporary Text-to-Image (T2I) models frequently depend on qualitative human evaluations to assess the consistency between synthesized images and the text prompts. There is a demand for quantitative and automatic evaluation tools, given that human evaluation lacks reproducibility. We believe that an effective T2I evaluation metric should accomplish the following: detect instances where the generated images do not align with the textual prompts, a discrepancy we define as the `hallucination problem' in T2I tasks; record the types and frequency of hallucination issues, aiding users in understanding the causes of errors; and provide a comprehensive and intuitive scoring that close to human standard. To achieve these objectives, we propose a method based on large language models (LLMs) for conducting question-answering with an extracted scene-graph and created a dataset with human-rated scores for generated images. From the methodology perspective, we combine knowledge-enhanced question-answering tasks with image evaluation tasks, making the evaluation metrics more controllable and easier to interpret. For the contribution on the dataset side, we generated 12,000 synthesized images based on 1,000 composited prompts using three advanced T2I models. Subsequently, we conduct human scoring on all synthesized images and prompt pairs to validate the accuracy and effectiveness of our method as an evaluation metric. All generated images and the human-labeled scores will be made publicly available in the future to facilitate ongoing research on this crucial issue. Extensive experiments show that our method aligns more closely with human scoring patterns than other evaluation metrics.

cs.CV↗
arXiv2024

CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare. However, the trustworthiness of Med-LVLMs remains unverified, posing significant risks for future model deployment. In this paper, we introduce CARES and aim to comprehensively evaluate the Trustworthiness of Med-LVLMs across the medical domain. We assess the trustworthiness of Med-LVLMs across five dimensions, including trustfulness, fairness, safety, privacy, and robustness. CARES comprises about 41K question-answer pairs in both closed and open-ended formats, covering 16 medical image modalities and 27 anatomical regions. Our analysis reveals that the models consistently exhibit concerns regarding trustworthiness, often displaying factual inaccuracies and failing to maintain fairness across different demographic groups. Furthermore, they are vulnerable to attacks and demonstrate a lack of privacy awareness. We publicly release our benchmark and code in https://cares-ai.github.io/.

cs.LG↗
arXiv2024

Calibrated Self-Rewarding Vision Language Models

Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR enhances performance and reduces hallucinations across ten benchmarks and tasks, achieving substantial improvements over existing methods by 7.62%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning. Our data and code are available at https://github.com/YiyangZhou/CSR.

cs.LG↗
arXiv2024

On the Optimality of Dilated Entropy and Lower Bounds for Online Learning in Extensive-Form Games

First-order methods (FOMs) are arguably the most scalable algorithms for equilibrium computation in large extensive-form games. To operationalize these methods, a distance-generating function, acting as a regularizer for the strategy space, must be chosen. The ratio between the strong convexity modulus and the diameter of the regularizer is a key parameter in the analysis of FOMs. A natural question is then: what is the optimal distance-generating function for extensive-form decision spaces? In this paper, we make a number of contributions, ultimately establishing that the weight-one dilated entropy (DilEnt) distance-generating function is optimal up to logarithmic factors. The DilEnt regularizer is notable due to its iterate-equivalence with Kernelized OMWU (KOMWU) -- the algorithm with state-of-the-art dependence on the game tree size in extensive-form games -- when used in conjunction with the online mirror descent (OMD) algorithm. However, the standard analysis for OMD is unable to establish such a result; the only current analysis is by appealing to the iterate equivalence to KOMWU. We close this gap by introducing a pair of primal-dual treeplex norms, which we contend form the natural analytic viewpoint for studying the strong convexity of DilEnt. Using these norm pairs, we recover the diameter-to-strong-convexity ratio that predicts the same performance as KOMWU. Along with a new regret lower bound for online learning in sequence-form strategy spaces, we show that this ratio is nearly optimal. Finally, we showcase our analytic techniques by refining the analysis of Clairvoyant OMD when paired with DilEnt, establishing an $\mathcal{O}(n \log |\mathcal{V}| \log T/T)$ approximation rate to coarse correlated equilibrium in $n$-player games, where $|\mathcal{V}|$ is the number of reduced normal-form strategies of the players, establishing the new state of the art.

cs.LG↗
arXiv2024

Long-Term Energy Management for Microgrid with Hybrid Hydrogen-Battery Energy Storage: A Prediction-Free Coordinated Optimization Framework

This paper studies the long-term energy management of a microgrid coordinating hybrid hydrogen-battery energy storage. We develop an approximate semi-empirical hydrogen storage model to accurately capture the power-dependent efficiency of hydrogen storage. We introduce a prediction-free two-stage coordinated optimization framework, which generates the annual state-of-charge (SoC) reference for hydrogen storage offline. During online operation, it updates the SoC reference online using kernel regression and makes operation decisions based on the proposed adaptive virtual-queue-based online convex optimization (OCO) algorithm. We innovatively incorporate penalty terms for long-term pattern tracking and expert-tracking for step size updates. We provide theoretical proof to show that the proposed OCO algorithm achieves a sublinear bound of dynamic regret without using prediction information. Numerical studies based on the Elia and North China datasets show that the proposed framework significantly outperforms the existing online optimization approaches by reducing the operational costs and loss of load by around 30% and 80%, respectively. These benefits can be further enhanced with optimized settings for the penalty coefficient and step size of OCO, as well as more historical references.

math.OC↗
arXiv2024

Efficient Data Learning for Open Information Extraction with Pre-trained Language Models

Open Information Extraction (OpenIE) is a fundamental yet challenging task in Natural Language Processing, which involves extracting all triples (subject, predicate, object) from a given sentence. While labeling-based methods have their merits, generation-based techniques offer unique advantages, such as the ability to generate tokens not present in the original sentence. However, these generation-based methods often require a significant amount of training data to learn the task form of OpenIE and substantial training time to overcome slow model convergence due to the order penalty. In this paper, we introduce a novel framework, OK-IE, that ingeniously transforms the task form of OpenIE into the pre-training task form of the T5 model, thereby reducing the need for extensive training data. Furthermore, we introduce an innovative concept of Anchor to control the sequence of model outputs, effectively eliminating the impact of order penalty on model convergence and significantly reducing training time. Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the training time (3 minutes) to achieve comparable results.

cs.CL↗
arXiv2023

Efficient Algorithms for Sparse Moment Problems without Separation

We consider the sparse moment problem of learning a $k$-spike mixture in high-dimensional space from its noisy moment information in any dimension. We measure the accuracy of the learned mixtures using transportation distance. Previous algorithms either assume certain separation assumptions, use more recovery moments, or run in (super) exponential time. Our algorithm for the one-dimensional problem (also called the sparse Hausdorff moment problem) is a robust version of the classic Prony's method, and our contribution mainly lies in the analysis. We adopt a global and much tighter analysis than previous work (which analyzes the perturbation of the intermediate results of Prony's method). A useful technical ingredient is a connection between the linear system defined by the Vandermonde matrix and the Schur polynomial, which allows us to provide tight perturbation bound independent of the separation and may be useful in other contexts. To tackle the high-dimensional problem, we first solve the two-dimensional problem by extending the one-dimensional algorithm and analysis to complex numbers. Our algorithm for the high-dimensional case determines the coordinates of each spike by aligning a 1d projection of the mixture to a random vector and a set of 2d projections of the mixture. Our results have applications to learning topic models and Gaussian mixtures, implying improved sample complexity results or running time over prior work.

cs.LG↗
arXiv2023

OpenFE: Automated Feature Generation with Expert-level Performance

The goal of automated feature generation is to liberate machine learning experts from the laborious task of manual feature generation, which is crucial for improving the learning performance of tabular data. The major challenge in automated feature generation is to efficiently and accurately identify effective features from a vast pool of candidate features. In this paper, we present OpenFE, an automated feature generation tool that provides competitive results against machine learning experts. OpenFE achieves high efficiency and accuracy with two components: 1) a novel feature boosting method for accurately evaluating the incremental performance of candidate features and 2) a two-stage pruning algorithm that performs feature pruning in a coarse-to-fine manner. Extensive experiments on ten benchmark datasets show that OpenFE outperforms existing baseline methods by a large margin. We further evaluate OpenFE in two Kaggle competitions with thousands of data science teams participating. In the two competitions, features generated by OpenFE with a simple baseline model can beat 99.3% and 99.6% data science teams respectively. In addition to the empirical results, we provide a theoretical perspective to show that feature generation can be beneficial in a simple yet representative setting. The code is available at https://github.com/ZhangTP1996/OpenFE.

cs.LG↗
arXiv2023

On the Interplay Between Misspecification and Sub-optimality Gap in Linear Contextual Bandits

We study linear contextual bandits in the misspecified setting, where the expected reward function can be approximated by a linear function class up to a bounded misspecification level $ζ>0$. We propose an algorithm based on a novel data selection scheme, which only selects the contextual vectors with large uncertainty for online regression. We show that, when the misspecification level $ζ$ is dominated by $\tilde O (Δ/ \sqrt{d})$ with $Δ$ being the minimal sub-optimality gap and $d$ being the dimension of the contextual vectors, our algorithm enjoys the same gap-dependent regret bound $\tilde O (d^2/Δ)$ as in the well-specified setting up to logarithmic factors. In addition, we show that an existing algorithm SupLinUCB (Chu et al., 2011) can also achieve a gap-dependent constant regret bound without the knowledge of sub-optimality gap $Δ$. Together with a lower bound adapted from Lattimore et al. (2020), our result suggests an interplay between misspecification level and the sub-optimality gap: (1) the linear contextual bandit model is efficiently learnable when $ζ\leq \tilde O(Δ/ \sqrt{d})$; and (2) it is not efficiently learnable when $ζ\geq \tilde Ω(Δ / {\sqrt{d}})$. Experiments on both synthetic and real-world datasets corroborate our theoretical results.

cs.LG↗
arXiv2020

Generation of even and odd high harmonics in resonant metasurfaces using single and multiple ultra-intense laser pulses

High harmonic generation (HHG) opens a window on the fundamental science of strong-field light-mater interaction and serves as a key building block for attosecond optics and metrology. Resonantly enhanced HHG from hot spots in nanostructures is an attractive route to overcoming the well-known limitations of gases and bulk solids. We demonstrate a nanoscale platform for highly efficient HHG driven by strong mid-infrared laser pulses: an ultra-thin resonant gallium phosphide (GaP) metasurface. The wide bandgap and the lack of inversion symmetry of the GaP crystal enable the generation of even and odd harmonics covering a wide range of photon energies between 1.3 and 3 eV with minimal reabsorption. The resonantly enhanced conversion efficiency facilitates single-shot measurements that avoid material damage and pave the way to controllable transition between perturbative and non-perturbative regimes of light-matter interactions at the nanoscale.

physics.optics↗
arXiv2019

Nonlinear manifestations of photon acceleration in time-dependent metasurfaces: tunable broadband harmonics generation

Time-dependent nonlinear media, such as rapidly generated plasmas produced via laser ionization of gases, can increase the energy of individual laser photons and generate tunable high-order harmonic pulses. This phenomenon, known as photon acceleration, has traditionally required extreme-intensity laser pulses and macroscopic propagation lengths. Here, we report on a novel nonlinear material$-$an ultrathin semiconductor metasurface$-$that exhibits efficient photon acceleration at low intensities. We observe a signature nonlinear manifestation of photon acceleration: third-harmonic generation of near-infrared photons with tunable frequencies reaching up to $\approx3.1ω$. A simple time-dependent coupled-mode theory, found to be in good agreement with experimental results, is utilized to predict a new path towards nonlinear radiation sources that combine resonant upconversion with broadband operation.

physics.optics↗
arXiv2018

Active Valley-topological Plasmonic Crystal in Metagate-tuned Graphene

A valley plasmonic crystal for graphene surface plasmons (GSPs) is proposed. We demonstrate that a designer metagate, placed within a few nanometers from graphene, can be used to impose a triangular periodic Fermi energy landscape on the latter. For specific metagate geometries and bias voltages, the combined metagate-graphene structure is shown to produce sufficiently strong Bragg scattering of GSPs to produce complete propagation bandgaps, and to impart the GSPs with nontrivial valley-linked topological properties. Valley-selective kink states supported by a domain wall between differently patterned metagates are shown to propagate without reflections along sharply curved interfaces owing to suppressed inter-valley scattering. Our approach paves the way for non-magnetic dynamically reconfigurable topological nanophotonic devices.

cond-mat.mes-hall↗
arXiv2018

Perfect Diffraction with Bianisotropic Metagratings

One highly desirable function of a diffraction grating is its ability to deflect incident light into a specific diffraction order with near-perfect efficiency. While such asymmetry can be achieved in a variety of ways, e.g., by using a sawtooth (blazed) geometry, a recently emerged approach is to use a planar metagrating comprised of designer multi-resonant periodic units (metamolecules). Here we demonstrate that a bianisotropic unit cell supporting four resonances interfering in the far field can be used as a building block for achieving the prefect deflection. A coupled mode analysis shows that these modes provide a small number of orthogonal electromagnetic radiation patterns that are needed to suppress transmission/reflection into all but one diffraction order. Bianisotropy caused by a mirror symmetry breaking enables a normally incident wave to excite, through near-field couplings, two otherwise "dark" resonant modes. We design and experimentally realize bianisotropic metamolecules which are sub-wavelength in all three dimensions, and whose optical properties are desensitized to fabrication imperfections by their geometric simplicity. We show that optical beams tightly focused onto the metagratings with just a few unit cells can also be asymmetrically deflected with high efficiency, paving the way for compact broadband optical devices.

physics.optics↗
arXiv2017

Ultrathin wave plates based on bi-resonant silicon Huygens' metasurfaces

An all-dielectric Huygens metasurface supporting electric and magnetic resonances is a promising platform for a variety of optical applications that require a combination of extremely high transmission and broad-range control of the phase of the transmitted light. A highly efficient wave plate is one example of such an application. In combination with high spectral selectivity and strong optical energy concentration, such phase plates are desirable for precision sensing and high efficiency nonlinear optics. We propose a novel approach to realizing such ultra-thin optical plates: an anisotropic Fano-resonant optical metasurface (AFROM) employing the combination of a spectrally sharp electric and a relatively broadband magnetic resonance. The phase shift coverage approaching 2π is achieved through judicious choice of the geometric parameters of a complex unit cell. A new methodology based on eigenvalue simulations of leaky magnetic/electric resonances enables rapid computational design of such metasurfaces.

physics.optics↗
arXiv2011

DNA-based Self-Assembly of Chiral Plasmonic Nanostructures with Tailored Optical Response

Surface plasmon resonances generated in metallic nanostructures can be utilized to tailor electromagnetic fields. The precise spatial arrangement of such structures can result in surprising optical properties that are not found in any naturally occurring material. Here, the designed activity emerges from collective effects of singular components equipped with limited individual functionality. Top-down fabrication of plasmonic materials with a predesigned optical response in the visible range by conventional lithographic methods has remained challenging due to their limited resolution, the complexity of scaling, and the difficulty to extend these techniques to three-dimensional architectures. Molecular self-assembly provides an alternative route to create such materials which is not bound by the above limitations. We demonstrate how the DNA origami method can be used to produce plasmonic materials with a tailored optical response at visible wavelengths. Harnessing the assembly power of 3D DNA origami, we arranged metal nanoparticles with a spatial accuracy of 2 nm into nanoscale helices. The helical structures assemble in solution in a massively parallel fashion and with near quantitative yields. As a designed optical response, we generated giant circular dichroism and optical rotary dispersion in the visible range that originates from the collective plasmon-plasmon interactions within the nanohelices. We also show that the optical response can be tuned through the visible spectrum by changing the composition of the metal nanoparticles. The observed effects are independent of the direction of the incident light and can be switched by design between left- and right-handed orientation. Our work demonstrates the production of complex bulk materials from precisely designed nanoscopic assemblies and highlights the potential of DNA self-assembly for the fabrication of plasmonic nanostructures.

cond-mat.mes-hall↗