arXiv subjects

Teng Andrea Xu

Publications and source records attributed to Teng Andrea Xu.

6 recordsLinked to original sources

arXiv2026

Scaling Point-in-Time Language Models

Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models--trained exclusively on text available up to each calendar date--eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline--including dataset construction, training infrastructure, and evaluation code--to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

cs.CL↗

arXiv2025

Training NTK to Generalize with KARE

The performance of the data-dependent neural tangent kernel (NTK; Jacot et al. (2018)) associated with a trained deep neural network (DNN) often matches or exceeds that of the full network. This implies that DNN training via gradient descent implicitly performs kernel learning by optimizing the NTK. In this paper, we propose instead to optimize the NTK explicitly. Rather than minimizing empirical risk, we train the NTK to minimize its generalization error using the recently developed Kernel Alignment Risk Estimator (KARE; Jacot et al. (2020)). Our simulations and real data experiments show that NTKs trained with KARE consistently match or significantly outperform the original DNN and the DNN- induced NTK (the after-kernel). These results suggest that explicitly trained kernels can outperform traditional end-to-end DNN optimization in certain settings, challenging the conventional dominance of DNNs. We argue that explicit training of NTK is a form of over-parametrized feature learning.

cs.LG↗

arXiv2025

DeFi versus TradFi: Valuation Using Multiples and Discounted Cash Flows

As of August 2022, blockchain-based assets boast a combined market capitalisation exceeding one trillion USD, among which the most prominent are the decentralised autonomous organisation (DAO) tokens associated with decentralised finance (DeFi) protocols. In this work, we seek to value DeFi tokens using the canonical multiples and discount cash flow (DCF) approaches. We examine a subset of DeFi services including decentralised exchanges (DEXs), protocol for loanable funds (PLFs), and yield aggregators. We apply the same analysis to some publicly traded firms and compare them with DeFi tokens of the analogous category. Interestingly, despite the crypto bear market lasting for more than one year as of August 2022, both approaches evidence overvaluation in DeFi.

econ.GN↗

arXiv2023

Benign Autoencoders

Recent progress in Generative Artificial Intelligence (AI) relies on efficient data representations, often featuring encoder-decoder architectures. We formalize the mathematical problem of finding the optimal encoder-decoder pair and characterize its solution, which we name the "benign autoencoder" (BAE). We prove that BAE projects data onto a manifold whose dimension is the optimal compressibility dimension of the generative problem. We highlight surprising connections between BAE and several recent developments in AI, such as conditional GANs, context encoders, stable diffusion, stacked autoencoders, and the learning capabilities of generative models. As an illustration, we show how BAE can find optimal, low-dimensional latent representations that improve the performance of a discriminator under a distribution shift. By compressing "malignant" data dimensions, BAE leads to smoother and more stable gradients.

cs.LG↗

arXiv2023

A Short Survey on Business Models of Decentralized Finance (DeFi) Protocols

Decentralized Finance (DeFi) services are moving traditional financial operations to the Internet of Value (IOV) by exploiting smart contracts, distributed ledgers, and clever heterogeneous transactions among different protocols. The exponential increase of the Total Value Locked (TVL) in DeFi foreshadows a bright future for automated money transfers in a plethora of services. In this short survey paper, we describe the business model for different DeFi domains - namely, Protocols for Loanable Funds (PLFs), Decentralized Exchanges (DEXs), and Yield Aggregators. We claim that the current state of the literature is still unclear how to value thousands of different competitors (tokens) in DeFi. With this work, we abstract the general business model for different DeFi domains and compare them. Finally, we provide open research challenges that will involve heterogeneous domains such as economics, finance, and computer science.

econ.GN↗

arXiv2023

A Simple Algorithm For Scaling Up Kernel Methods

The recent discovery of the equivalence between infinitely wide neural networks (NNs) in the lazy training regime and Neural Tangent Kernels (NTKs) (Jacot et al., 2018) has revived interest in kernel methods. However, conventional wisdom suggests kernel methods are unsuitable for large samples due to their computational complexity and memory requirements. We introduce a novel random feature regression algorithm that allows us (when necessary) to scale to virtually infinite numbers of random features. We illustrate the performance of our method on the CIFAR-10 dataset.

cs.LG↗