Birth, Death and Diff: Insertion-Based Sequence Generation with Corrections

Durga Prasad Maram; Dhruvesh Patel; Tahira Naseem; Benjamin Rozonoyer; Gaurav Pandey; Ramón Fernandez Astudillo; Andrew McCallum

arXiv · 2610.34310

Birth, Death and Diff: Insertion-Based Sequence Generation with Corrections

Durga Prasad Maram·Dhruvesh Patel·Tahira Naseem·Benjamin Rozonoyer·Gaurav Pandey·Ramón Fernandez Astudillo·Andrew McCallum

View original source ↗Read source document ↗Export metadata ↓

Abstract

Insertion-based masked diffusion models enable flexible-length, parallel generation by interleaving mask insertion with token unmasking. However, because their generation decisions are irreversible, aggressive parallel decoding can introduce errors that subsequent steps cannot correct, necessitating conservative decoding with more forward passes (NFEs). Moreover, they exhibit higher entropy than standard MDMs because they marginalize over multiple alignments during training and therefore require even more conservative decoding in the absence of a correction mechanism. To address this limitation, we introduce $δ$MDM, which makes intermediate generations revisable by equipping an insertion-based model with substitution and deletion operations. During training, we expose the model to errors caused by aggressive unmasking and insertion, teaching it to correct its own mistakes. At inference time, we interleave insertion and unmasking with correction steps that can revise previously unmasked tokens and remove unnecessary masks. This mechanism makes aggressive decoding practical, improving both generation speed and quality. $δ$MDM Pareto-dominates FlexMDM, a vanilla insertion-based MDM, on the speed--quality trade-off. On high-diversity unconditional text generation on OpenWebText, it halves the generative perplexity relative to FlexMDM at the highest inference budget. It also achieves lower generative perplexity at all inference budgets compared to MDM trained from scratch with variable-length padded sequences. On low-diversity conditional text generation on TinyGSM, it matches FlexMDM accuracy with half as many NFEs.

Explore related subjects

cs.LG cs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Durga Prasad Maram, Dhruvesh Patel, Tahira Naseem, Benjamin Rozonoyer, Gaurav Pandey, Ramón Fernandez Astudillo, Andrew McCallum. 2026-10-25. Birth, Death and Diff: Insertion-Based Sequence Generation with Corrections. https://arxiv.org/abs/2610.34310

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Generative Semi-supervised Graph Anomaly Detection

This work considers a practical semi-supervised graph anomaly detection (GAD) scenario, where part of the nodes in a graph are known to be normal, contrasting to the extensively explored unsupervised setting with a fully unlabeled graph. We reveal that having access to the normal nodes, even just a small percentage of normal nodes, helps enhance the detection performance of existing unsupervised GAD methods when they are adapted to the semi-supervised setting. However, their utilization of these normal nodes is limited. In this paper, we propose a novel Generative GAD approach (namely GGAD) for the semi-supervised scenario to better exploit the normal nodes. The key idea is to generate pseudo anomaly nodes, referred to as 'outlier nodes', for providing effective negative node samples in training a discriminative one-class classifier. The main challenge here lies in the lack of ground truth information about real anomaly nodes. To address this challenge, GGAD is designed to leverage two important priors about the anomaly nodes -- asymmetric local affinity and egocentric closeness -- to generate reliable outlier nodes that assimilate anomaly nodes in both graph structure and feature representations. Comprehensive experiments on six real-world GAD datasets are performed to establish a benchmark for semi-supervised GAD and show that GGAD substantially outperforms state-of-the-art unsupervised and semi-supervised GAD methods with varying numbers of training normal nodes. Code will be made available at https://github.com/mala-lab/GGAD.

cs.LG↗

arXiv2026

Adaptive Momentum Benefits Score-based Generative Models

Score-based Generative Models (SGMs) have achieved remarkable success in generation tasks by establishing a diffusion process that gradually perturbs real data into Gaussian noise and then learning a reverse process to reconstruct the data from noise. However, conventional denoising methods, such as Langevin dynamics and numerical stochastic differential equation solvers, rely on randomness but suffer from slow generation speeds due to the high number of score function evaluations. To address this, we propose a plug-and-play Markov adaptive momentum sampling method, inspired by the strong connection between the SGMs' sampling process and stochastic gradient descent optimization. Our approach accelerates the transformation process without introducing additional hyperparameters. Theoretically, we establish a rigorous foundation by analyzing the relationship between the distribution of samples generated by our method and the original data distribution. This provides a novel perspective on the analysis of sampling distributions. Empirically, our method demonstrates superior efficiency, generating high-quality images/graphs in significantly fewer sampling steps, achieving a speedup of 2x to 5x while maintaining or surpassing baseline performance on various image and graph generation benchmarks. These results underscore the effectiveness of adaptive momentum in enhancing the efficiency of SGMs.

cs.LG↗

arXiv2026

A Comparative Study of Convolutional and Recurrent Neural Networks for Storm Surge Prediction in Tampa Bay

In this paper, we compare the performance of three popular deep learning architectures: Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN), and CNN-LSTM in water level prediction over different time window sizes. We used wind, pressure, and tide data from National Oceanic and Atmospheric Administration (NOAA) stations. Our results show that the LSTM model generally outperformed the other models in forecasting water level trends. By applying LSTM, we achieved a mean squared error (MSE) of 0.006, making it the most reliable option for real-world use. We predicted water levels using wind, pressure, and tides. Wind, pressure, and tide data are time-based features that lack spatial patterns. Consequently, LSTM performed better than spatial methods, such as CNN. We also investigated Hurricane Ian, which occurred in Tampa Bay in September 2022 and caused a significant storm surge. Our findings show that CNN and CNN-LSTM performed slightly better than LSTMs during powerful storms, such as hurricanes. Hurricanes are seldom observed in training data, and CNN-based models are more reliable when predicting data that were not included in the training. To explain how environmental factors affect the predicted water level, we used the perturbation method. The perturbation shows that the tide is more effective at predicting water levels than wind and pressure. This study demonstrates the potential of deep learning to improve storm surge prediction accuracy.

cs.LG↗