arXiv subjects

Yair Goldberg

Publications and source records attributed to Yair Goldberg.

18 recordsLinked to original sources

arXiv2025

Silent Abandonment in Text-Based Contact Centers: Identifying, Quantifying, and Mitigating its Operational Impacts

In the quest to improve services, companies offer customers the option to interact with agents via texting. Such contact centers face unique challenges compared to traditional call centers, as measuring customer experience proxies like abandonment and patience involves uncertainty. A key source of this uncertainty is silent abandonment, where customers leave without notifying the system, wasting agent time and leaving their status unclear. Silent abandonment also obscures whether a customer was served or left. Our goals are to measure the magnitude of silent abandonment and mitigate its effects. Classification models show that 3%-70% of customers across 17 companies abandon silently. In one study, 71.3% of abandoning customers did so silently, reducing agent efficiency by 3.2% and system capacity by 15.3%, incurring $5,457 in annual costs per agent. We develop an expectation-maximization (EM) algorithm to estimate customer patience under uncertainty and identify influencing covariates. We find that companies should use classification models to estimate abandonment scope and our EM algorithm to assess patience. We suggest strategies to operationally mitigate the impact of silent abandonment by predicting suspected silent-abandonment behavior or changing service design. Specifically, we show that while allowing customers to write while waiting in the queue creates a missing data challenge, it also significantly increases patience and reduces service time, leading to reduced abandonment and lower staffing requirements.

cs.SI↗

arXiv2024

Silent Abandonment in Contact Centers: Estimating Customer Patience from Uncertain Data

In the quest to improve services, companies offer customers the opportunity to interact with agents through contact centers, where the communication is mainly text-based. This has become one of the favorite channels of communication with companies in recent years. However, contact centers face operational challenges, since the measurement of common proxies for customer experience, such as knowledge of whether customers have abandoned the queue and their willingness to wait for service (patience), are subject to information uncertainty. We focus this research on the impact of a main source of such uncertainty: silent abandonment by customers. These customers leave the system while waiting for a reply to their inquiry, but give no indication of doing so, such as closing the mobile app of the interaction. As a result, the system is unaware that they have left and waste agent time and capacity until this fact is realized. In this paper, we show that 30%-67% of the abandoning customers abandon the system silently, and that such customer behavior reduces system efficiency by 5%-15%. To do so, we develop methodologies to identify silent-abandonment customers in two types of contact centers: chat and messaging systems. We first use text analysis and an SVM model to estimate the actual abandonment level. We then use a parametric estimator and develop an expectation-maximization algorithm to estimate customer patience accurately, as customer patience is an important parameter for fitting queueing models to the data. We show how accounting for silent abandonment in a queueing model improves dramatically the estimation accuracy of key measures of performance. Finally, we suggest strategies to operationally cope with the phenomenon of silent abandonment.

cs.SI↗

arXiv2023

Estimating Mean Viral Load Trajectory from Intermittent Longitudinal Data and Unknown Time Origins

Viral load (VL) in the respiratory tract is the leading proxy for assessing infectiousness potential. Understanding the dynamics of disease-related VL within the host is very important and help to determine different policy and health recommendations. However, often only partial followup data are available with unknown infection date. In this paper we introduce a discrete time likelihood-based approach to modeling and estimating partial observed longitudinal samples. We model the VL trajectory by a multivariate normal distribution that accounts for possible correlation between measurements within individuals. We derive an expectation-maximization (EM) algorithm which treats the unknown time origins and the missing measurements as latent variables. Our main motivation is the reconstruction of the daily mean SARS-Cov-2 VL, given measurements performed on random patients, whose VL was measured multiple times on different days. The method is applied to SARS-Cov-2 cycle-threshold-value data collected in Israel.

stat.AP↗

arXiv2022

A zero-estimator approach for estimating the signal level in a high-dimensional model-free setting

We study a high-dimensional regression setting under the assumption of known covariate distribution. We aim at estimating the amount of explained variation in the response by the best linear function of the covariates (the signal level). In our setting, neither sparsity of the coefficient vector, nor normality of the covariates or linearity of the conditional expectation are assumed. We present an unbiased and consistent estimator and then improve it by using a zero-estimator approach, where a zero-estimator is a statistic whose expected value is zero. More generally, we present an algorithm based on the zero estimator approach that in principle can improve any given estimator. We study some asymptotic properties of the proposed estimators and demonstrate their finite sample performance in a simulation study.

math.ST↗

arXiv2021

Improved Estimators for Semi-supervised High-dimensional Regression Model

We study a linear high-dimensional regression model in a semi-supervised setting, where for many observations only the vector of covariates $X$ is given with no response $Y$. We do not make any sparsity assumptions on the vector of coefficients, and aim at estimating $\mathrm{Var}(Y|X)$. We propose an estimator, which is unbiased, consistent, and asymptotically normal. This estimator can be improved by adding zero-estimators arising from the unlabelled data. Adding zero-estimators does not affect the bias and potentially can reduce variance. In order to achieve optimal improvement, many zero-estimators should be used, but this raises the problem of estimating many parameters. Therefore, we introduce covariate selection algorithms that identify which zero-estimators should be used in order to improve the above estimator. We further illustrate our approach for other estimators, and present an algorithm that improves estimation for any given variance estimator. Our theoretical results are demonstrated in a simulation study.

math.ST↗

arXiv2021

Improving Efficiency of Tests for Composite Null Hypotheses

The goal of mediation analysis is to study the effect of exposure on an outcome interceded by a mediator. Two simple hypotheses are tested: the effect of the exposure on the mediator, and the effect of the mediator on the outcome. When either of these hypotheses is true, a predetermined significance level can be assured. When both nulls are true, the same test becomes conservative. Adaptively finding the correct scenario enables customizing the tests and consequently enlarges their efficiency, which is most important in a multiple testing framework. In this work, we link between adaptive two-stage procedures and shrinkage estimators. We first study the properties of shrinkage estimators, and characterize their behavior at different parameter points using local asymptotics. We formulate theoretical results regarding shrinkage estimators, compared to regular estimators. We then discuss the multiple-testing framework and state results about using shrinkage estimator in two-stage procedures for controlling the FWER. Taking advantage of these theoretical results, we suggest a number of estimators and test statistics for the two-stage mediation procedures. We then investigate their empirical FWER and power, compared to regular estimators and tests, through simulations.

math.ST↗

arXiv2019

Kernel Machines for Current Status Data

In survival analysis, estimating the failure time distribution is an important and difficult task, since usually the data is subject to censoring. Specifically, in this paper we consider current status data, a type of data where all of the observations are censored. The format of the data is such that the failure time is restricted to knowledge of whether or not the failure time exceeds a random monitoring time. We propose a flexible kernel machine approach for estimation of the failure time expectation as a function of the covariates, with current status data. In order to obtain the kernel machine decision function, we minimize a regularized version of the empirical risk with respect to a new loss function. Using finite sample bounds and novel oracle inequalities, we prove that the obtained estimator converges to the true conditional expectation for a large family of probability measures. Finally, we present a simulation study and an analysis of real-world data that compares the performance of the proposed approach to existing methods. We show empirically that our approach is comparable to current state of the art, and in some cases is even better.

math.ST↗

arXiv2019

Semiparametric Analysis of the Proportional Likelihood Ratio Model and Omnibus Estimation Procedure

We provide a semi-parametric analysis for the proportional likelihood ratio model, proposed by Luo & Tsai (2012). We study the tangent spaces for both the parameter of interest and the nuisance parameter, and obtain an explicit expression for the efficient score function. We propose a family of Z-estimators based on the score functions, including an approximated efficient estimator. Using inverse probability weighting, the proposed estimators can also be applied to different missing-data mechanisms, such as right censored data and non-random sampling. A simulation study that illustrates the finite-sample performance of the estimators is presented.

math.ST↗

arXiv2019

Hoeffding-Type and Bernstein-Type Inequalities for Right Censored Data

We present Hoeffding-type and Bernstein-type inequalities for right-censored data. The inequalities bound the difference between an inverse of the probability of censoring weighting (IPCW) estimator and its expectation. We first discuss the asymptotic properties of the estimator and provide conditions for its efficiency. We present standard, data dependent, and uniform Hoeffding-type inequalities. We then present a Bernstein-type inequality. Finally, we show how to apply these inequalities in an empirical risk minimization setting.

math.ST↗

arXiv2018

Kernel Machines With Missing Responses

Missing responses is a missing data format in which outcomes are not always observed. In this work we develop kernel machines that can handle missing responses. First, we propose a kernel machine family that uses mainly the complete cases. For the quadratic loss, we then propose a family of doubly-robust kernel machines. The proposed kernel-machine estimators can be applied to both regression and classification problems. We prove oracle inequalities for the finite-sample differences between the kernel machine risk and Bayes risk. We use these oracle inequalities to prove consistency and to calculate convergence rates. We demonstrate the performance of the two proposed kernel machine families using both a simulation study and a real-world data analysis.

stat.ML↗

arXiv2018

Self-reporting and screening: Data with current-status and censored observations

We consider survival data that combine three types of observations: uncensored, right-censored, and left-censored. Such data arises from screening a medical condition, in situations where self-detection arises naturally. Our goal is to estimate the failure-time distribution, based on these three observation types. We propose a novel methodology for distribution estimation using both parametric and nonparametric techniques. We then evaluate the performance of these estimators via simulated data. Finally, as a case study, we estimate the patience of patients who arrive at an emergency department and wait for treatment. Three categories of patients are observed: those who leave the system and announce it, and thus their patience time is observed; those who get service and thus their patience time is right-censored by the waiting time; and those who leave the system without announcing it. For the third category, the patients' absence is revealed only when they are called to service, which is after they have already left; formally, their patience time is left-censored. Other applications of our proposed methodology are discussed.

math.ST↗

arXiv2015

Feature Elimination in Kernel Machines in moderately high dimensions

We develop an approach for feature elimination in statistical learning with kernel machines, based on recursive elimination of features.We present theoretical properties of this method and show that it is uniformly consistent in finding the correct feature space under certain generalized assumptions.We present four case studies to show that the assumptions are met in most practical situations and present simulation results to demonstrate performance of the proposed approach.

stat.ML↗

arXiv2014

A Quantile Regression Model for Failure-Time Data with Time-Dependent Covariates

Since survival data occur over time, often important covariates that we wish to consider also change over time. Such covariates are referred as time-dependent covariates. Quantile regression offers flexible modeling of survival data by allowing the covariates to vary with quantiles. This paper provides a novel quantile regression model accommodating time-dependent covariates, for analyzing survival data subject to right censoring. Our simple estimation technique assumes the existence of instrumental variables. In addition, we present a doubly-robust estimator in the sense of Robins and Rotnitzky (1992). The asymptotic properties of the estimators are rigorously studied. Finite-sample properties are demonstrated by a simulation study. The utility of the proposed methodology is demonstrated using the Stanford heart transplant dataset.

stat.ME↗

arXiv2014

Modeling and Optimization of Genetic Screens via RNA Interference and FACS

We study mathematically a method for discovering which gene is related to a cell phenotype of interest. The method is based on RNA interference -- a molecular process for gene deactivation -- and on coupling the phenotype with fluorescence in a FACS machine. A small number of candidate genes are thus isolated, and then tested individually. We model probabilistically this process, prove a limit theorem for its outcome, and derive operational guidelines for maximizing the probability of successful gene discovery.

stat.AP↗

arXiv2013

Support Vector Regression for Right Censored Data

We develop a unified approach for classification and regression support vector machines for data subject to right censoring. We provide finite sample bounds on the generalization error of the algorithm, prove risk consistency for a wide class of probability measures, and study the associated learning rates. We apply the general methodology to estimation of the (truncated) mean, median, quantiles, and for classification problems. We present a simulation study that demonstrates the performance of the proposed approach.

stat.ML↗

arXiv2012

Q-learning with censored data

We develop methodology for a multistage decision problem with flexible number of stages in which the rewards are survival times that are subject to censoring. We present a novel Q-learning algorithm that is adjusted for censored data and allows a flexible number of stages. We provide finite sample bounds on the generalization error of the policy learned by the algorithm, and show that when the optimal Q-function belongs to the approximation space, the expected survival time for policies obtained by the algorithm converges to that of the optimal policy. We simulate a multistage clinical trial with flexible number of stages and apply the proposed censored-Q-learning algorithm to find individualized treatment regimens. The methodology presented in this paper has implications in the design of personalized medicine trials in cancer and in other life-threatening diseases.

math.ST↗

arXiv2010

The Best Linear Unbiased Estimator for Continuation of a Function

We show how to construct the best linear unbiased predictor (BLUP) for the continuation of a curve in a spline-function model. We assume that the entire curve is drawn from some smooth random process and that the curve is given up to some cut point. We demonstrate how to compute the BLUP efficiently. Confidence bands for the BLUP are discussed. Finally, we apply the proposed BLUP to real-world call center data. Specifically, we forecast the continuation of both the call arrival counts and the workload process at the call center of a commercial bank.

stat.AP↗

arXiv2008

LLE with low-dimensional neighborhood representation

The local linear embedding algorithm (LLE) is a non-linear dimension-reducing technique, widely used due to its computational simplicity and intuitive approach. LLE first linearly reconstructs each input point from its nearest neighbors and then preserves these neighborhood relations in the low-dimensional embedding. We show that the reconstruction weights computed by LLE capture the high-dimensional structure of the neighborhoods, and not the low-dimensional manifold structure. Consequently, the weight vectors are highly sensitive to noise. Moreover, this causes LLE to converge to a linear projection of the input, as opposed to its non-linear embedding goal. To overcome both of these problems, we propose to compute the weight vectors using a low-dimensional neighborhood representation. We prove theoretically that this straightforward and computationally simple modification of LLE reduces LLE's sensitivity to noise. This modification also removes the need for regularization when the number of neighbors is larger than the dimension of the input. We present numerical examples demonstrating both the perturbation and linear projection problems, and the improved outputs using the low-dimensional neighborhood representation.

stat.ML↗