Searcharxiv⌕ Search

arXiv subjects

Sang-Chul Kim

Publications and source records attributed to Sang-Chul Kim.

2 recordsLinked to original sources

Operating-Point Failure of Frozen Guardrail Probes Under Backbone Updates

We freeze a guardrail probe and threshold, then measure its operating point across 18 cells covering 16 public language-model updates. At a 1% target, the threshold fails closed in 8 of 9 cells in the first phase and either fails closed or open in 3 of 9 cells in a registered second phase (descriptive). The two phases also differ in update type and input format. The largest increase is from 1.01% to 15.12%, while the area under the receiver operating characteristic curve changes by no more than 0.0058 in any of these cells. A benchmark's staleness criteria classify none of them as stale. A rule based on these criteria flags 0.019 of layer cells where the rate doubles or halves. A label-free rate estimate flags 0.918, clearing 0.898 of the remaining cells. A two-moment model matches the change within a factor of 1.5 in 0.905 of 63 registered cells and in 0.773 of the 22 cells that moved. Resetting the threshold using stored benign prompts restores the false-positive rate without requiring new labels. However, poisoning 1% of these prompts leaves at most 0.017 of harmful prompts flagged. An attacker controlling the update cut detection of one of three targeted categories in three of five seeds. Both checks missed this attack, while a post hoc labelled check detected it.

cs.CR↗

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

Prompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under distribution shift, particularly on the benign side of the decision boundary, where false positives impose direct operational cost yet are seldom measured. We present PIDS-Bench, a frozen multi-axis benchmark that jointly evaluates attack detection and benign false-positive behavior at fixed thresholds, spanning in-distribution inputs, hard-benign prompts that mimic injection structure without malicious intent, obfuscated attacks, and domain and structural distribution shifts. We evaluate seven detectors (learned baselines, external prompt-injection classifiers, and broad-safety comparators) alongside a rule-based lower-bound reference. Multi-axis evaluation exposes a failure mode that aggregate F1 conceals. A detector exceeding F1 = 0.98 on the held-out split still misclassifies roughly one-third of an externally-sourced benign subset drawn from public corpora and restricted to security-adjacent content. Across a full threshold sweep and five training seeds, no internal detector reaches an operating point satisfying F1 >= 0.95 and hard-benign FPR <= 0.10 together on this stress distribution. Decomposing by provenance, we find that hard-negative augmentation nearly eliminates over-defense on curated stress inputs but leaves it substantially intact on externally-sourced prompts, a pattern we term provenance-sensitive over-defense. The asymmetry holds across both fine-tuned architectures and does not diminish as the augmentation pool grows, with the externally-sourced FPR remaining far above the 0.10 target. Whether augmentation matched to the externally-sourced distribution would close this gap is untested; threshold calibration and curated-style augmentation alone do not.

cs.CR↗