arXiv · 2611.18678
Not All Steps Matter: Temporally Adaptive Spectral Control for Concept Removal in Diffusion Models
Abstract
As text-to-image diffusion models see widespread deployment, removing unsafe or undesirable concepts without retraining has become a pressing need. Spectral methods are attractive because they are training-free and reversible, yet current variants apply the same output-value (OV) circuit intervention at every denoising step, assuming concept information is temporally uniform. This wastes intervention where concept energy is low while degrading image quality, an inefficiency that compounds across concept types. Through timestep-resolved spectral analysis, concept-specific energy is found to follow a consistent temporal hierarchy: content concepts decay fastest, object concepts are intermediate, and style concepts persist longest. This paper proposes Timestep-Adaptive Spectral Nullification (TASN), which adapts suppression strength and spectral components to the concept energy profile. Cross-attention at each timestep is decomposed into concept-aligned and concept-orthogonal components, and it is proved that when the orthogonal component is approximately stable, the collateral-minimizing schedule sets suppression proportional to concept energy. TASN implements this with a timestep-adaptive spectral subset tracking only active concept directions, strictly generalizing uniform Spectral Nullification (SN). On five red-teaming benchmarks, TASN lowers attack success rate below uniform SN while recovering the majority of its quality loss; ablations confirm gains arise from temporal targeting rather than reduced suppression, generalizing across concept categories and architectures.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Chanaka Thushitha Bandara, Lakmali Nadeesha Kumari, Sen-Ching Samson Cheung. 2026-10-30. Not All Steps Matter: Temporally Adaptive Spectral Control for Concept Removal in Diffusion Models. https://searcharxiv.com/abs/2611.18678
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.