Molecular notation can act as a shortcut in AI benchmark evaluation
High predictive accuracy does not necessarily mean that an AI model relies on information relevant to its scientific task. Molecular prediction illustrates this concern: the same molecule can be represented by different SMILES strings, potentially altering model behavior. We combined a systematic PubMed literature survey with experiments on chemical language models. SMILES preparation could not be identified from articles or Supplementary Information in 71 of 174 studies (40.8%). Across six benchmarks, 58.7% of identity-preserving input pairs used different SMILES strings. Notation differences changed reconstruction and latent representations, while separately trained classifiers often achieved similar scores. In two datasets, notation was associated with labels and changed predictions of a fixed model, showing how benchmark scores can reward notational shortcuts. Balancing notation during classifier training reduced prediction differences but sometimes lowered performance on original inputs. These findings support evaluating predictive performance alongside consistency under identity-preserving changes in input representation.