arXiv subjects

Zhen Lin

Publications and source records attributed to Zhen Lin.

At least 37 records · Page 2Linked to original sources

arXiv2024

Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation

The advent of large language models (LLMs) has dramatically advanced the state-of-the-art in numerous natural language generation tasks. For LLMs to be applied reliably, it is essential to have an accurate measure of their confidence. Currently, the most commonly used confidence score function is the likelihood of the generated sequence, which, however, conflates semantic and syntactic components. For instance, in question-answering (QA) tasks, an awkward phrasing of the correct answer might result in a lower probability prediction. Additionally, different tokens should be weighted differently depending on the context. In this work, we propose enhancing the predicted sequence probability by assigning different weights to various tokens using attention values elicited from the base LLM. By employing a validation set, we can identify the relevant attention heads, thereby significantly improving the reliability of the vanilla sequence probability confidence measure. We refer to this new score as the Contextualized Sequence Likelihood (CSL). CSL is easy to implement, fast to compute, and offers considerable potential for further improvement with task-specific prompts. Across several QA datasets and a diverse array of LLMs, CSL has demonstrated significantly higher reliability than state-of-the-art baselines in predicting generation quality, as measured by the AUROC or AUARC.

cs.CL↗

arXiv2024

Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

Tabular data from different tables exhibit significant diversity due to varied definitions and types of features, as well as complex inter-feature and feature-target relationships. Cross-dataset pretraining, which learns reusable patterns from upstream data to support downstream tasks, have shown notable success in various fields. Yet, when applied to tabular data prediction, this paradigm faces challenges due to the limited reusable patterns among diverse tabular datasets (tables) and the general scarcity of tabular data available for fine-tuning. In this study, we fill this gap by introducing a cross-table pretrained Transformer, XTFormer, for versatile downstream tabular prediction tasks. Our methodology insight is pretraining XTFormer to establish a "meta-function" space that encompasses all potential feature-target mappings. In pre-training, a variety of potential mappings are extracted from pre-training tabular datasets and are embedded into the "meta-function" space, and suited mappings are extracted from the "meta-function" space for downstream tasks by a specified coordinate positioning approach. Experiments show that, in 190 downstream tabular prediction tasks, our cross-table pretrained XTFormer wins both XGBoost and Catboost on 137 (72%) tasks, and surpasses representative deep learning models FT-Transformer and the tabular pre-training approach XTab on 144 (76%) and 162 (85%) tasks.

cs.LG↗

arXiv2024

Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models

Large language models (LLMs) specializing in natural language generation (NLG) have recently started exhibiting promising capabilities across a variety of domains. However, gauging the trustworthiness of responses generated by LLMs remains an open challenge, with limited research on uncertainty quantification (UQ) for NLG. Furthermore, existing literature typically assumes white-box access to language models, which is becoming unrealistic either due to the closed-source nature of the latest LLMs or computational constraints. In this work, we investigate UQ in NLG for *black-box* LLMs. We first differentiate *uncertainty* vs *confidence*: the former refers to the ``dispersion'' of the potential predictions for a fixed input, and the latter refers to the confidence on a particular prediction/generation. We then propose and compare several confidence/uncertainty measures, applying them to *selective NLG* where unreliable results could either be ignored or yielded for further assessment. Experiments were carried out with several popular LLMs on question-answering datasets (for evaluation purposes). Results reveal that a simple measure for the semantic dispersion can be a reliable predictor of the quality of LLM responses, providing valuable insights for practitioners on uncertainty management when adopting LLMs. The code to replicate our experiments is available at https://github.com/zlin7/UQ-NLG.

cs.CL↗

arXiv2023

Conformal Drug Property Prediction with Density Estimation under Covariate Shift

In drug discovery, it is vital to confirm the predictions of pharmaceutical properties from computational models using costly wet-lab experiments. Hence, obtaining reliable uncertainty estimates is crucial for prioritizing drug molecules for subsequent experimental validation. Conformal Prediction (CP) is a promising tool for creating such prediction sets for molecular properties with a coverage guarantee. However, the exchangeability assumption of CP is often challenged with covariate shift in drug discovery tasks: Most datasets contain limited labeled data, which may not be representative of the vast chemical space from which molecules are drawn. To address this limitation, we propose a method called CoDrug that employs an energy-based model leveraging both training data and unlabelled data, and Kernel Density Estimation (KDE) to assess the densities of a molecule set. The estimated densities are then used to weigh the molecule samples while building prediction sets and rectifying for distribution shift. In extensive experiments involving realistic distribution drifts in various small-molecule drug discovery tasks, we demonstrate the ability of CoDrug to provide valid prediction sets and its utility in addressing the distribution shift arising from de novo drug design models. On average, using CoDrug can reduce the coverage gap by over 35% when compared to conformal prediction sets not adjusted for covariate shift.

cs.LG↗

arXiv2023

Fast Online Value-Maximizing Prediction Sets with Conformal Cost Control

Many real-world multi-label prediction problems involve set-valued predictions that must satisfy specific requirements dictated by downstream usage. We focus on a typical scenario where such requirements, separately encoding $\textit{value}$ and $\textit{cost}$, compete with each other. For instance, a hospital might expect a smart diagnosis system to capture as many severe, often co-morbid, diseases as possible (the value), while maintaining strict control over incorrect predictions (the cost). We present a general pipeline, dubbed as FavMac, to maximize the value while controlling the cost in such scenarios. FavMac can be combined with almost any multi-label classifier, affording distribution-free theoretical guarantees on cost control. Moreover, unlike prior works, it can handle real-world large-scale applications via a carefully designed online update mechanism, which is of independent interest. Our methodological and theoretical contributions are supported by experiments on several healthcare tasks and synthetic datasets - FavMac furnishes higher value compared with several variants and baselines while maintaining strict cost control. Our code is available at https://github.com/zlin7/FavMac

cs.LG↗

arXiv2023

The Laplacian spectral ratio of connected graphs

Let $G$ be a simple connected undirected graph. The Laplacian spectral ratio of $G$, denoted by $R_L(G)$, is defined as the quotient between the largest and second smallest Laplacian eigenvalues of $G$, which is closely related to the structural parameters of a graph (or network), such as diameter, $t$-tough, perfect matching, average density of cuts, and synchronizability, etc. In this paper, we obtain some bounds of the Laplacian spectral ratio, which improves the known results. In addition, we give counter-examples on the upper bound of the Laplacian spectral ratio conjecture of trees, and propose a new conjecture.

math.CO↗

arXiv2022

Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networks

Deep neural network (DNN) classifiers are often overconfident, producing miscalibrated class probabilities. In high-risk applications like healthcare, practitioners require $\textit{fully calibrated}$ probability predictions for decision-making. That is, conditioned on the prediction $\textit{vector}$, $\textit{every}$ class' probability should be close to the predicted value. Most existing calibration methods either lack theoretical guarantees for producing calibrated outputs, reduce classification accuracy in the process, or only calibrate the predicted class. This paper proposes a new Kernel-based calibration method called KCal. Unlike existing calibration procedures, KCal does not operate directly on the logits or softmax outputs of the DNN. Instead, KCal learns a metric space on the penultimate-layer latent embedding and generates predictions using kernel density estimates on a calibration set. We first analyze KCal theoretically, showing that it enjoys a provable $\textit{full}$ calibration guarantee. Then, through extensive experiments across a variety of datasets, we show that KCal consistently outperforms baselines as measured by the calibration error and by proper scoring rules like the Brier Score.

stat.ML↗

arXiv2022

Conformal Prediction Intervals with Temporal Dependence

Cross-sectional prediction is common in many domains such as healthcare, including forecasting tasks using electronic health records, where different patients form a cross-section. We focus on the task of constructing valid prediction intervals (PIs) in time series regression with a cross-section. A prediction interval is considered valid if it covers the true response with (a pre-specified) high probability. We first distinguish between two notions of validity in such a setting: cross-sectional and longitudinal. Cross-sectional validity is concerned with validity across the cross-section of the time series data, while longitudinal validity accounts for the temporal dimension. Coverage guarantees along both these dimensions are ideally desirable; however, we show that distribution-free longitudinal validity is theoretically impossible. Despite this limitation, we propose Conformal Prediction with Temporal Dependence (CPTD), a procedure that is able to maintain strict cross-sectional validity while improving longitudinal coverage. CPTD is post-hoc and light-weight, and can easily be used in conjunction with any prediction model as long as a calibration set is available. We focus on neural networks due to their ability to model complicated data such as diagnosis codes for time series regression, and perform extensive experimental validation to verify the efficacy of our approach. We find that CPTD outperforms baselines on a variety of datasets by improving longitudinal coverage and often providing more efficient (narrower) PIs.

stat.ML↗

arXiv2022

On the Ky Fan $k$-norm of the $LI$-matrix of graphs

Let $A(G)$ and $D(G)$ be the adjacency matrix and the degree diagonal matrix of a graph $G$, respectively. Then $L(G)=D(G)-A(G)$ is called Laplacian matrix of the graph $G$. Let $G$ be a graph with $n$ vertices and $m$ edges. Then the $LI$-matrix of $G$ are defined as $LI(G)=L(G)-\frac{2m}{n}I_n$, where $I_n$ is the identity matrix. In this paper, we are interested in extremal properties of the Ky Fan $k$-norm of the $LI$-matrix of graphs, which is closely related to the well known problems and results in spectral graph theory, such as the Laplacian spectral radius, the Laplacian spread, the sum of the $k$ largest Laplacian eigenvalues, the Laplacian energy, and other parameters. Some bounds on the Ky Fan $k$-norm of the $LI$-matrix of graphs are given, and the extremal graphs are partly characterized. In addition, upper and lower bounds on the Ky Fan $k$-norm of $LI$-matrix of trees, unicyclic graphs and bicyclic graphs are determined, and the corresponding extremal graphs are characterized.

math.CO↗

arXiv2022

Conformal Prediction with Temporal Quantile Adjustments

We develop Temporal Quantile Adjustment (TQA), a general method to construct efficient and valid prediction intervals (PIs) for regression on cross-sectional time series data. Such data is common in many domains, including econometrics and healthcare. A canonical example in healthcare is predicting patient outcomes using physiological time-series data, where a population of patients composes a cross-section. Reliable PI estimators in this setting must address two distinct notions of coverage: cross-sectional coverage across a cross-sectional slice, and longitudinal coverage along the temporal dimension for each time series. Recent works have explored adapting Conformal Prediction (CP) to obtain PIs in the time series context. However, none handles both notions of coverage simultaneously. CP methods typically query a pre-specified quantile from the distribution of nonconformity scores on a calibration set. TQA adjusts the quantile to query in CP at each time $t$, accounting for both cross-sectional and longitudinal coverage in a theoretically-grounded manner. The post-hoc nature of TQA facilitates its use as a general wrapper around any time series regression model. We validate TQA's performance through extensive experimentation: TQA generally obtains efficient PIs and improves longitudinal coverage while preserving cross-sectional coverage.

stat.ML↗

arXiv2021

Locally Valid and Discriminative Prediction Intervals for Deep Learning Models

Crucial for building trust in deep learning models for critical real-world applications is efficient and theoretically sound uncertainty quantification, a task that continues to be challenging. Useful uncertainty information is expected to have two key properties: It should be valid (guaranteeing coverage) and discriminative (more uncertain when the expected risk is high). Moreover, when combined with deep learning (DL) methods, it should be scalable and affect the DL model performance minimally. Most existing Bayesian methods lack frequentist coverage guarantees and usually affect model performance. The few available frequentist methods are rarely discriminative and/or violate coverage guarantees due to unrealistic assumptions. Moreover, many methods are expensive or require substantial modifications to the base neural network. Building upon recent advances in conformal prediction [13, 33] and leveraging the classical idea of kernel regression, we propose Locally Valid and Discriminative prediction intervals (LVD), a simple, efficient, and lightweight method to construct discriminative prediction intervals (PIs) for almost any DL model. With no assumptions on the data distribution, such PIs also offer finite-sample local coverage guarantees (contrasted to the simpler marginal coverage). We empirically verify, using diverse datasets, that besides being the only locally valid method for DL, LVD also exceeds or matches the performance (including coverage rate and prediction accuracy) of existing uncertainty quantification methods, while offering additional benefits in scalability and flexibility.

cs.LG↗

arXiv2021

The biharmonic index of connected graphs

Let $G$ be a simple connected graph with the vertex set $V(G)$ and $d_{B}^2(u,v)$ be the biharmonic distance between two vertices $u$ and $v$ in $G$. The biharmonic index $BH(G)$ of $G$ is defined as $$BH(G)=\frac{1}{2}\sum\limits_{u\in V(G)}\sum\limits_{v\in V(G)}d_{B}^2(u,v)=n\sum\limits_{i=2}^{n}\frac{1}{λ_i^2(G)},$$ where $λ_i(G)$ is the $i$-th smallest eigenvalue of the Laplacian matrix of $G$ with $n$ vertices. In this paper, we provide the mathematical relationships between the biharmonic index and some classic topological indices: the first Zagreb index, the forgotten topological index and the Kirchhoff index. In addition, the extremal value on the biharmonic index for trees and firefly graphs of fixed order are given. Finally, some graph operations on the biharmonic index are presented.

math.CO↗

arXiv2021

The general Albertson irregularity index of graphs

We introduce the general Albertson irregularity index of a connected graph $G$ and define it as $A_{p}(G) =(\sum_{uv\in E(G)}|d(u)-d(v)|^p)^{\frac{1}{p}}$, where $p$ is a positive real number and $d(v)$ is the degree of the vertex $v$ in $G$. The new index is not only generalization of the well-known Albertson irregularity index and $σ$-index, but also it is the Minkowski norm of the degree of vertex. We present lower and upper bounds on the general Albertson irregularity index. In addition, we study the extremal value on the general Albertson irregularity index for trees of given order. Finally, we give the calculation formula of the general Albertson index of generalized Bethe trees and Kragujevac trees.

math.CO↗

arXiv2021

On the first Banhatti-Sombor index

Let $d_v$ be the degree of the vertex $v$ in a connected graph $G$. The first Banhatti-Sombor index of $G$ is defined as $BSO(G) =\sum_{uv\in E(G)}\sqrt{\frac{1}{d^2_u}+\frac{1}{d^2_v}}$, which is a new vertex-degree-based topological index introduced by Kulli. In this paper, the mathematical relations between the first Banhatti-Sombor index and some other well-known vertex-degree-based topological indices are established. In addition, the trees extremal with respect to the first Banhatti-Sombor index on trees and chemical trees are characterized, respectively.

math.CO↗

arXiv2021

The Sombor index of trees and unicyclic graphs with given maximum degree

Let $d_G(v)$ be the degree of the vertex $v$ in a graph $G$. The Sombor index of $G$ is defined as $SO(G) =\sum_{uv\in E(G)}\sqrt{d^2_G(u)+d^2_G(v)}$, which is a new degree-based topological index introduced by Gutman. Let $\mathscr{T}_{n,Δ}$ and $\mathscr{U}_{n,Δ}$ be the set of trees and unicyclic graphs with $n$ vertices and maximum degree $Δ$, respectively. In this paper, the tree and the unicyclic graph with minimum Sombor index among $\mathscr{T}_{n,Δ}$ and $\mathscr{U}_{n,Δ}$ are characterized.

math.CO↗

arXiv2021

The Sombor index of trees and unicyclic graphs with given matching number

In 2021, the Sombor index was introduced by Gutman, which is a new degree-based topological molecular descriptors. The Sombor index of a graph $G$ is defined as $SO(G) =\sum_{uv\in E(G)}\sqrt{d^2_G(u)+d^2_G(v)}$, where $d_G(v)$ is the degree of the vertex $v$ in $G$. Let $\mathscr{T}_{n,m}$ and $\mathscr{U}_{n,m}$ be the set of trees and unicyclic graphs on $n$ vertices with fixed matching number $m$, respectively. In this paper, the tree and the unicyclic graph with the maximum Sombor index are determined among $\mathscr{T}_{n,m}$ and $\mathscr{U}_{n,m}$, respectively.

math.CO↗

arXiv2021

DeepSZ: Identification of Sunyaev-Zel'dovich Galaxy Clusters using Deep Learning

Galaxy clusters identified from the Sunyaev Zel'dovich (SZ) effect are a key ingredient in multi-wavelength cluster-based cosmology. We present a comparison between two methods of cluster identification: the standard Matched Filter (MF) method in SZ cluster finding and a method using Convolutional Neural Networks (CNN). We further implement and show results for a `combined' identifier. We apply the methods to simulated millimeter maps for several observing frequencies for an SPT-3G-like survey. There are some key differences between the methods. The MF method requires image pre-processing to remove point sources and a model for the noise, while the CNN method requires very little pre-processing of images. Additionally, the CNN requires tuning of hyperparameters in the model and takes as input, cutout images of the sky. Specifically, we use the CNN to classify whether or not an 8 arcmin $\times$ 8 arcmin cutout of the sky contains a cluster. We compare differences in purity and completeness. The MF signal-to-noise ratio depends on both mass and redshift. Our CNN, trained for a given mass threshold, captures a different set of clusters than the MF, some of which have SNR below the MF detection threshold. However, the CNN tends to mis-classify cutouts whose clusters are located near the edge of the cutout, which can be mitigated with staggered cutouts. We leverage the complementarity of the two methods, combining the scores from each method for identification. The purity and completeness of the MF alone are both 0.61, assuming a standard detection threshold. The purity and completeness of the CNN alone are 0.59 and 0.61. The combined classification method yields 0.60 and 0.77, a significant increase for completeness with a modest decrease in purity. We advocate for combined methods that increase the confidence of many lower signal-to-noise clusters.

astro-ph.CO↗

arXiv2021

SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Models

Despite deep learning (DL) success in classification problems, DL classifiers do not provide a sound mechanism to decide when to refrain from predicting. Recent works tried to control the overall prediction risk with classification with rejection options. However, existing works overlook the different significance of different classes. We introduce Set-classifier with Class-specific RIsk Bounds (SCRIB) to tackle this problem, assigning multiple labels to each example. Given the output of a black-box model on the validation set, SCRIB constructs a set-classifier that controls the class-specific prediction risks with a theoretical guarantee. The key idea is to reject when the set classifier returns more than one label. We validated SCRIB on several medical applications, including sleep staging on electroencephalogram (EEG) data, X-ray COVID image classification, and atrial fibrillation detection based on electrocardiogram (ECG) data. SCRIB obtained desirable class-specific risks, which are 35\%-88\% closer to the target risks than baseline methods.

cs.LG↗