arXiv subjects

Emily Huang

Publications and source records attributed to Emily Huang.

6 recordsLinked to original sources

arXiv2026

Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

Multimodal Foundation Models are increasingly used as reasoning agents, making reliability, knowing when a model may hallucinate, critical. A common intuition, which we call the Attention-Confidence Assumption, holds that reliability follows from "structural" visual perception: tight attention on relevant regions should signal a trustworthy answer, while scattered attention signals confusion. We challenge this through the VLM Reliability Probe (VRP), a systematic cross-family study of reliability signals in contemporary Vision-Language Models (VLMs). We introduce structural-attention metrics, cluster counts (C_k) and spatial entropy (H_s), to quantify the visual encoder's gaze, and track its evolution (Delta H_s) across layers. This reveals a "Symbolic Detachment": models often "Early Lock" visual features only to diffuse attention later, severing early perception from final generation. Contrary to the grounding hypothesis, we find a "Cluster Failure": spatial attention has near-zero correlation (R approx 0.001) with accuracy. Instead, reliability is a phenomenon of generation dynamics and internal-state distributions. Self-Consistency, the agreement rate across sampled reasoning paths, is the dominant predictor of truth (R = 0.429). Scaling causal interventions exposes a sharp architectural divergence: LLaVA locks its prediction in a fragile late-stage bottleneck, whereas PaliGemma and Qwen2-VL distribute reliability globally, staying resilient even when ~50% or more of their most predictive layer is destroyed. For current VLMs, reliability signals are detached from visual grounding maps and are best inferred from generation-time dynamics and hidden-state probes.

cs.CV↗

arXiv2026

Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits

A pervasive intuition holds that vision-language models (VLMs) are most trustworthy when their attention maps look sharp: concentrated attention on the queried region should imply a confident, calibrated answer. We test this Attention-Confidence Assumption directly. We instrument three open-weight VLM families (LLaVA-1.5, PaliGemma, Qwen2-VL; 3-7B parameters) with a unified mechanistic pipeline -- the VLM Reliability Probe (VRP) -- that compares attention structure, generation dynamics, and hidden-state geometry against a single correctness label. Three results emerge. (i) Attention structure is a near-zero predictor of correctness (R_pb(C_k,y)=0.001, 95% CI [-0.034,0.036]; R_pb(H_s,y)=-0.012, [-0.047,0.024] on a pooled n=3,090 split), even though attention remains causally necessary for feature extraction (top-30% patch masking drops accuracy by 8.2-11.3 pp, p<0.001). (ii) Reliability becomes legible later in the computation: a single hidden-state linear probe reaches AUROC>0.95 on POPE for two of three families, and self-consistency at K=10 is the strongest behavioral predictor we measure at 10x inference cost (R_pb=0.43). (iii) Causal neuron-level ablations expose a sharp architectural split with direct monitor-design implications: late-fusion LLaVA concentrates reliability in a fragile late bottleneck (-8.3 pp object-identification accuracy after top-5 probe-neuron ablation), whereas early-fusion PaliGemma and Qwen2-VL distribute it widely and absorb destruction of ~50% of their peak-layer hidden dimension with <=1 pp degradation. The takeaway is narrow but consequential: in 3-7B VLMs, reliability is read more reliably off hidden-state geometry, layer-wise margin formation, and sparse late-layer circuits than off attention-map sharpness.

cs.AI↗

arXiv2025

DeckFlow: Iterative Specification on a Multimodal Generative Canvas

Generative AI promises to allow people to create high-quality personalized media. Although powerful, we identify three fundamental design problems with existing tooling through a literature review. We introduce a multimodal generative AI tool, DeckFlow, to address these problems. First, DeckFlow supports task decomposition by allowing users to maintain multiple interconnected subtasks on an infinite canvas populated by cards connected through visual dataflow affordances. Second, DeckFlow supports a specification decomposition workflow where an initial goal is iteratively decomposed into smaller parts and combined using feature labels and clusters. Finally, DeckFlow supports generative space exploration by generating multiple prompt and output variations, presented in a grid, that can feed back recursively into the next design iteration. We evaluate DeckFlow for text-to-image generation against a state-of-practice conversational AI baseline for image generation tasks. We then add audio generation and investigate user behaviors in a more open-ended creative setting with text, image, and audio outputs.

cs.HC↗

arXiv2021

Shuffle Squares and Reverse Shuffle Squares

Let $\mathcal{SS}_k(n)$ be the family of {\it shuffle squares} in $[k]^{2n}$, words that can be partitioned into two disjoint identical subsequences. Let $\mathcal{RSS}_k(n)$ be the family of {\it reverse shuffle squares} in $[k]^{2n}$, words that can be partitioned into two disjoint subsequences which are reverses of each other. Henshall, Rampersad, and Shallit conjectured asymptotic formulas for the sizes of $\mathcal{SS}_k(n)$ and $\mathcal{RSS}_k(n)$ based on numerical evidence. We prove that \[ \lvert \mathcal{SS}_k(n) \rvert=\dfrac{1}{n+1}\dbinom{2n}{n}k^n-\dbinom{2n-1}{n+1}k^{n-1}+O_n(k^{n-2}), \] confirming their conjecture for $\mathcal{SS}_k(n)$. We also prove a similar asymptotic formula for reverse shuffle squares that disproves their conjecture for $\lvert \mathcal{RSS}_k(n) \rvert$. As these asymptotic formulas are vacuously true when the alphabet size is small, we study the binary case separately and prove that $|\mathcal{SS}_2(n)| \ge \binom{2n}{n}$.

math.CO↗

arXiv2021

Combining Accelerometer and Gyroscope Data in Smartphone-Based Activity Recognition using Movelets

Physical activity patterns can be informative about a patient's health status. Traditionally, activity data have been gathered using patient self-report. However, these subjective data can suffer from bias and are difficult to collect over long time periods. Smartphones offer an opportunity to address these challenges. The smartphone has built-in sensors that can be programmed to collect data objectively, unobtrusively, and continuously. Due to their widespread adoption, smartphones are also accessible to most of the population. A main challenge in smartphone-based activity recognition is extracting information optimally from multiple sensors to identify the unique features of different activities. In our study, we analyze data collected by the accelerometer and gyroscope, which measure the phone's acceleration and angular velocity, respectively. We propose an extension to the "movelet method" that jointly incorporates both sensors. We also apply this joint-sensor method to a data set we collected previously. The findings show that combining data from the two sensors can result in more accurate activity recognition than using each sensor alone. For example, the joint-sensor method reduces errors of the gyroscope-only method in differentiating between standing and sitting. It also reduces errors of the accelerometer-only method in classifying vigorous activities.

stat.ME↗

arXiv2019

Activity Classification Using Smartphone Gyroscope and Accelerometer Data

Activities, such as walking and sitting, are commonly used in biomedical settings either as an outcome or covariate of interest. Researchers have traditionally relied on surveys to quantify activity levels of subjects in both research and clinical settings, but surveys are not objective in nature and have many known limitations, such as recall bias. Smartphones provide an opportunity for unobtrusive objective measurement of various activities in naturalistic settings, but their data tends to be noisy and needs to be analyzed with care. We explored the potential of smartphone accelerometer and gyroscope data to distinguish between five different types of activity: walking, sitting, standing, ascending stairs, and descending stairs. We conducted a study in which four participants followed a study protocol and performed a sequence of various activities with one phone in their front pocket and another phone in their back pocket. The subjects were filmed throughout, and the obtained footage was annotated to establish ground truth activity. We applied the so-called movelet method to classify their activity. Our results demonstrate the promise of smartphones for activity detection in naturalistic settings, but they also highlight common challenges in this field of research.

cs.HC↗