Anton S. Petrov | Searcharxiv

arXiv2026

LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling

Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.

cs.LG↗

arXiv2025

Foundation Models for Astrobiology: Paper I -- Workshop and Overview

Advances in machine learning over the past decade have resulted in a proliferation of algorithmic applications for encoding, characterizing, and acting on complex data that may contain many high dimensional features. Recently, the emergence of deep-learning models trained across very large datasets has created a new paradigm for machine learning in the form of Foundation Models. Foundation Models are programs trained on very large and broad datasets with an extensive number of parameters. Once built, these powerful, and flexible, models can be utilized in less resource-intensive ways to build many different, downstream applications that can integrate previously disparate, multimodal data. The development of these applications can be done rapidly and with a much lower demand for machine learning expertise. And the necessary infrastructure and models themselves are already being established within agencies such as NASA and ESA. At NASA this work is across several divisions of the Science Mission Directorate including the NASA Goddard and INDUS Large Language Models and the Prithvi Geospatial Foundation Model. And ESA initiatives to bring Foundation Models to Earth observations has led to the development of TerraMind. A workshop was held by the NASA Ames Research Center and the SETI Institute, in February 2025, to investigate the potential of Foundation Models for astrobiological research and to determine what steps would be needed to build and utilize such a model or models. This paper shares the findings and recommendations of that workshop, and describes clear near-term, and future opportunities in the development of a Foundation Model (or Models) for astrobiology applications. These applications would include a biosignature, or life characterization, task, a mission development and operations task, and a natural language task for integrating and supporting astrobiology research needs.

astro-ph.IM↗