Data Annotations as Pedagogical Hints: From Subjective Labels to Critical Thinking
Machine learning courses often use pre-labelled datasets, hiding the subjectivity of human annotation. This produces an overly trusting view of data and AI models in students, at the expense of interpretive diversity and contestability of algorithmic outputs. We investigated whether manual data annotation tasks teach students about subjective labelling. Study Design: An annotation activity was implemented at two universities: Fontys (Netherlands) and IT University Copenhagen (Denmark). Students annotated skin lesion images for hair coverage on a 3-point scale. Surveys were collected from 43 participants, measuring their understanding of annotation ambiguity, data quality, bias, fairness, implementation barriers, and pedagogical effectiveness. Key Findings: Self-reported familiarity with the course content increased substantially across all concepts. Most students recognised that personal interpretation affects annotations. Students rated the activity as more effective than traditional lectures in understanding bias. Participants were motivated to learn more. Main Drawbacks: Emotional discomfort from viewing medical images was the primary issue. Many students still requested clearer guidelines to reduce disagreement, suggesting they had not yet internalised that disagreement arising from different perspectives is a feature, not a bug. Recommendations for Future Iterations: Ensure sufficient interpretive ambiguity in materials. Reduce repetitive annotation workload. Mitigate emotional discomfort from sensitive content. Explicitly frame disagreement as a learning opportunity rather than a problem to solve. Manual data annotations effectively teach students that human judgement shapes model behaviour and that disagreement reflects domain complexity, not just noise.