Impact of color in computer and human-based object recognition under occlusion

Courtney M. King; Elissa Aminoff; Daniel D. Leeds

Home/Archive/arXiv

arXiv · 2611.08808

Impact of color in computer and human-based object recognition under occlusion

Courtney M. King·Elissa Aminoff·Daniel D. Leeds

View original source ↗Read source document ↗Export metadata ↓

Abstract

Partial occlusions challenge object recognition in both human and computer vision (CV), but the impact is largely dependent on the occlusion and model design. Properties relating to color are particularly important in biological recognition and frequently leveraged for camouflage, however there are still open questions about what makes them effective and how they translate to CV. This work studies partial occlusions that create camouflage, including those that blend in with target objects or with scenes, which alter object representations in distinct ways. Experiments evaluate recognition performance on both human participants and models ranging from convolutional neural networks (CNNs) to transformers, and additional analysis is performed on models to uncover trends in processing and object representations under multiple occlusion types. Novel metrics offer explainability of model behavior under occlusion: non-occlusion precision (NOP) and non-occlusion recall (NOR) measure alignment of model activations with and without occlusion, and activation in occlusion (AIO) measures the percentage of activations that occur in occluded regions. Findings show that human recognition, and most transformers, are negatively impacted by generated camouflaging occluders, with scene-based camouflage performing worse than object-based. Classical CNNs show a strong opposite trend, performing the worst on neutral-colored occlusions, suggesting that contextually-related colors, particularly scene-based, benefit object recognition rather than disrupting shape or boundary cues. Additionally, generated camouflaging occluders for training augmentation improves robustness to unseen camouflage for selected CNN model, and occlusion metrics suggest trends about learned mechanisms. Findings have interesting implications for alignment in human and computer vision, model vulnerabilities, and explainability.

Explore related subjects

cs.CV

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Courtney M. King, Elissa Aminoff, Daniel D. Leeds. 2026-11-06. Impact of color in computer and human-based object recognition under occlusion. https://arxiv.org/abs/2611.08808

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Domain Adaptive Diabetic Retinopathy Grading with Model Absence and Flowing Data

Domain shift (the difference between source and target domains) poses a significant challenge in clinical applications, e.g., Diabetic Retinopathy (DR) grading. Despite considering certain clinical requirements, like source data privacy, conventional transfer methods are predominantly model-centered and often struggle to prevent model-targeted attacks. In this paper, we address a challenging Online Model-aGnostic Domain Adaptation (OMG-DA) setting, driven by the demands of clinical environments. This setting is characterized by the absence of the model and the flow of target data. To tackle the new challenge, we propose a novel approach, Generative Unadversarial ExampleS (GUES), which enables adaptation from a data-centric perspective. Specifically, we first theoretically reformulate conventional perturbation optimization in a generative way--learning a perturbation generation function with a latent input variable. During model instantiation, we leverage a Variational AutoEncoder to express this function. The encoder with the reparameterization trick predicts the latent input, whilst the decoder is responsible for the generation. Furthermore, the saliency map is selected as pseudo-perturbation labels. Because it not only captures potential lesions but also theoretically provides an upper bound on the function input, enabling the identification of the latent variable. Extensive comparative experiments on DR benchmarks with both frozen pre-trained models and trainable models demonstrate the superiority of GUES, showing robustness even with small batch size.

cs.CV↗

arXiv2026

PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model

PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. However, these formats often face issues related to usability, interpretability, and data integrity. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. To address these issues, We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy. PolSAM introduces Microwave Vision Data (MVD), a lightweight and interpretable data representation derived from polarimetric decomposition and semantic correlations. We propose two key components: the Feature-Level Fusion Prompt (FFP), which fuses visual tokens from pseudo-colored SAR images and MVD to address modality incompatibility in the frozen SAM encoder, and the Semantic-Level Fusion Prompt (SFP), which refines sparse and dense segmentation prompts using semantic information. Experimental results on the PhySAR-Seg datasets demonstrate that PolSAM significantly outperforms existing SAM-based and multimodal fusion models, improving segmentation accuracy, reducing data storage, and accelerating inference time. The source code and datasets will be made publicly available at https://github.com/XAI4SAR/PolSAM.

cs.CV↗

arXiv2026

ReefNet: A Large-Scale Dataset and Benchmark for Fine-Grained Coral Reef Recognition

Coral reef monitoring requires scalable recognition of visually similar taxa across heterogeneous underwater surveys. We introduce ReefNet, a public dataset that aligns approximately 925K hard-coral point annotations from 76 CoralNet sources and an independently collected Al-Wajh dataset with the World Register of Marine Species taxonomy. A stratified expert audit of 8,962 patches guides filtering to a 479,027-annotation benchmark with 39 labels: 38 genera and the family Fungiidae. Expert agreement is 92% on the retained audit subset, rather than an exhaustive accuracy estimate. We evaluate pretrained models through zero-shot recognition, few-shot adaptation, and full fine-tuning, each on within-source ReefNet and held-out-source Al-Wajh tests. Biological vision-language pretraining provides the strongest zero-shot results; supervised adaptation substantially improves recognition, with DINOv2 reaching 65.79% macro recall on Al-Wajh. Matched text-description experiments improve all four evaluated MLLMs on Al-Wajh, while a GPT-5.4 reasoning ablation does not improve over direct classification. Long-tail recognition, label uncertainty, and geographic imbalance remain challenges. ReefNet provides a taxonomically grounded resource for studying these challenges; the Al-Wajh test is a regional stress test rather than evidence of global robustness. Public data and resource links are documented in the supplement.

cs.CV↗