DiviK: Divisive intelligent K-Means for hands-free unsupervised clustering in big biological data
Investigating molecular heterogeneity provides insights about tumor origin and metabolomics. The increasing amount of data gathered makes manual analyses infeasible - therefore, automated unsupervised learning approaches are utilized for discovering heterogeneity. However, automated unsupervised analyses require a lot of experience with setting their hyperparameters and usually an upfront knowledge about the number of expected substructures. Moreover, numerous measured molecules require an additional step of feature engineering to provide valuable results. In this work, we propose DiviK: a scalable stepwise algorithm with local data-driven feature space adaptation for the segmentation of high-dimensional datasets. The combination of three quality indices: Dice Index, Rand Index and EXIMS score are used to assess the quality of unsupervised analyses in 3D space. DiviK was validated on two separate high-throughput datasets acquired by Mass Spectrometry Imaging in 2D and 3D. DiviK could be one of the default choices to consider during the initial exploration of Mass Spectrometry Imaging data. It provides a trade-off between absolute heterogeneity detection and focus on biologically plausible structures, and does not require specifying the number of expected structures before the analysis. With its unique local feature space adaptation, it is robust against dominating global patterns when focusing on the detail. Finally, due to its simplicity, DiviK is easily generalizable to an even more flexible framework, useful for other '-omics' data, or tabular data in general (including medical images after appropriate embedding). A generic implementation is freely available under Apache 2.0 license at https://github.com/gmrukwa/divik.