AI Models CpGPT and MethylGPT Revolutionize DNA Methylation Analysis for Aging and Disease Prediction

Revolutionary AI predicts aging and disease from DNA patterns

By Vijay Kumar MalesuReviewed by Susha Cheriyedath, M.Sc.Nov 11 2024 By integrating DNA sequence and epigenetic context, CpGPT sets new standards for predicting aging-related outcomes, offering unprecedented accuracy in assessing mortality and disease risk across various datasets. Study: CpGPT: a Foundation Model for DNA Methylation. Image Credit: Shutterstock AI *Important notice: bioRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information. In a recent pre-print* study posted to the bioRxiv server, a team of researchers introduced the Cytosine-phosphate-Guanine Pretrained Transformer (CpGPT: a transformer-based foundation model for deoxyribonucleic acid (DNA) methylation) designed to enhance analysis and prediction across diverse tissues and conditions. Background Since the advent of transformer architecture, artificial intelligence has rapidly progressed, especially through foundation models and large language models (LLMs) that utilize self-attention to capture complex patterns. Transformers have significantly impacted biology and medicine, advancing single-cell transcriptomics and revealing previously unknown biology with models like single-cell GPT (scGPT) and Geneformer. Despite progress in aging research, many epigenetic aging clocks still rely on simple linear models using CpG DNA methylation data, often overlooking sequence context and complex interactions. Few predictors, such as AltumAge and DeepMAge, employ deep neural networks. Further research is needed to develop advanced models that better capture the intricate mechanisms of aging. About the study CpGPT tackles missing data: With impressive accuracy, CpGPT can predict missing methylation values, making it a powerful tool for researchers who rely on incomplete datasets for aging and disease studies. To develop the CpGPT model, the researchers curated a comprehensive DNA methylation dataset named "CpGCorpus," aggregating data from more than 1,502 studies and over 106,000 human samples available in the Gene Expression Omnibus. This dataset contained various Illumina methylation array platforms and represented a rich diversity of tissue types, developmental stages, disease conditions, and demographic backgrounds. Raw data were processed using a Single Sample Methylation Analysis pipeline (SeSAMe), while normalized beta value matrices were used for already processed data. Quality control measures and probe harmonization were applied to ensure consistency across the dataset. The data were split into training, validation, and test sets without overlapping samples or studies. The CpGPT model integrated sequence, positional, and epigenetic information. Input representations included "embeddings of the nucleotide sequences" obtained from a pre-trained DNA language model, methylation beta values representing the methylation state of each site, and genomic positional encoding to capture the CpG site's location within the genome. A dual positional encoding strategy was employed, combining absolute and relative positional encodings to capture multi-scale genomic information. Specialized decoders were designed for beta value prediction, condition prediction, and uncertainty estimation. Pretraining was conducted using a multi-task learning approach with tailored loss functions, optimizing the model's ability to reconstruct missing data and learn meaningful sample representations. For fine-tuning, CpG sites associated with mortality were selected based on intra-class correlation coefficients and z-score thresholds. The model was then trained using a modified Cox proportional hazard loss. Predictive performance for mortality and morbidity was evaluated across multiple cohorts using Cox regression models, receiver operating characteristic analyses, and survival analyses, adjusting for age and employing appropriate statistical methods. Study results Beyond human data -- CpGPT's versatility across species: Fine-tuning CpGPT on mammalian data allowed it to predict methylation patterns in other species, hinting at its potential in broader evolutionary and biomedical research. The researchers developed CpGPT, which includes over 100,000 human DNA methylation samples from more than 1,500 studies covering a diverse range of tissue types, developmental stages, and disease conditions. The data were thoroughly preprocessed and harmonized to ensure consistency across various Illumina methylation array platforms, such as the HumanMethylation450 BeadChip (450k), HumanMethylation27 BeadChip (27k), Infinium MethylationEPIC BeadChip (EPIC), EPIC+, and EPICv2 arrays. CpGPT integrates three key types of contextual information: sequence context based on the DNA nucleotides near each CpG site, positional context covering local and global information, and epigenetic state. Sequence context is encoded using embeddings of nucleotide sequences surrounding each CpG site, derived from a pre-trained DNA language model. The model organizes sequence embeddings by genomic positions to capture positional context, groups them by chromosomes, and applies stochastic shuffling to prevent positional biases. Each CpG site's methylation state is transformed into an embedding representing its epigenetic status, and these embeddings are combined to form the model's input. The core architecture of CpGPT is based on the Transformer++ model, an enhanced version of the transformer architecture with modifications for increased training stability and accuracy. The model is trained in an unsupervised manner to predict methylation states (beta values) and their uncertainties, enabling it to generate meaningful sample-level embeddings that encapsulate comprehensive methylation profiles. The training process employs multiple loss functions to optimize various performance aspects and is designed to handle missing data effectively. Evaluations using dimensionality reduction techniques revealed that CpGPT's locus embeddings naturally reflect functional genomic annotations, with CpG sites clustering according to features like island status and chromatin states. Sample embeddings effectively captured biological variations, clustering samples according to tissue types and cell lines. The model demonstrated the ability to perform zero-shot reference mapping, which allows it to transfer labels from reference datasets with known annotations to new target datasets without additional training. CpGPT showed strong performance in imputing missing methylation data, accurately reconstructing beta values for missing probes, and improving the performance of various epigenetic clocks. Through its attention mechanism, CpGPT dynamically weights features, allowing sample-specific interpretation by assigning importance scores to each CpG site. This highlighted biologically relevant genes important for tissue-specific epigenetic regulation. When fine-tuned for mortality prediction, CpGPT exhibited predictive performance across multiple cohorts, effectively stratifying individuals based on their biological aging profiles. It showed significant associations with mortality and morbidity outcomes, including risks for conditions such as neurodegenerative diseases, cardiovascular issues, and physical function measurements. Conclusions To summarize, CpGPT effectively integrates sequence context, positional information, and epigenetic state to learn rich embeddings at both the CpG site and sample levels. The model excels in tasks such as imputing missing methylation values, array conversion, zero-shot reference mapping, and predicting age and mortality. By capturing complex dependencies among CpG sites, CpGPT overcomes the limitations of traditional linear models, enhancing predictive capabilities for aging-related outcomes and disease risks across various datasets. *Important notice: bioRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information. Journal reference: Preliminary scientific report. CpGPT: a Foundation Model for DNA Methylation, Lucas Paulo de Lima Camillo, Raghav Sehgal, Jenel Armstrong, Albert T. Higgins-Chen, Steve Horvath, Bo Wang, bioRxiv 2024.10.24.619766; doi: 10.1101/2024.10.24.619766, https://www.biorxiv.org/content/10.1101/2024.10.24.619766v1

MethylGPT unlocks DNA secrets for age and disease prediction

By Tarun Sai LomteReviewed by Susha Cheriyedath, M.Sc.Nov 11 2024 By harnessing advanced AI, MethylGPT decodes DNA methylation with unprecedented accuracy, offering new paths for age prediction, disease diagnosis, and personalized health interventions. Study: MethylGPT: a foundation model for the DNA methylome. Image Credit: Shutterstock AI *Important notice: bioRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information. In a recent study posted to the bioRxiv preprint* server, researchers developed a transformer-based foundation model, MethylGPT, for the DNA methylome. DNA methylation is a type of epigenetic modification that regulates gene expression via methyl-binding proteins and changes in chromatin accessibility. It also helps maintain genomic stability through transposable element repression. DNA methylation has features of an ideal biomarker, and studies have revealed distinct methylation signatures across pathological states, allowing for molecular diagnostics. Nevertheless, several analytic challenges impede the implementation of diagnostics based on DNA methylation. Current approaches rely on simple statistical and linear models, which are limited in capturing complex, non-linear data. They also fail to account for context-specific effects such as higher-order interactions and regulatory networks. Therefore, a unified analytical framework that can model complex, non-linear patterns in various tissue and cell types is urgently needed. Recent advances in foundation models and transformer architectures have revolutionized analyses of complex biological sequences. Foundation models have also been introduced for various omics layers, such as AlphaFold3 and ESM-3 for proteomics and Evo and Enformer for genomics. The achievements of the foundation models suggest that DNA methylation analyses could be transformed with a similar approach. The study and findings MethylGPT can predict DNA methylation levels with high accuracy, achieving a Pearson correlation of 0.929, showing its precision in capturing complex epigenetic patterns. In the present study, researchers developed MethylGPT, a transformer-based foundation model for the DNA methylome. First, they acquired data on 226,555 human DNA methylation profiles spanning multiple tissue types from the EWAS Data Hub and Clockbase. Following deduplication and quality control, 154,063 samples were retained for pretraining. The model focused on 49,156 CpG sites, which were selected based on their known associations with various traits, as this would maximize their biological relevance. The model was pre-trained using two complementary loss functions: masked language modeling (MLM) loss and profile reconstruction loss, enabling it to accurately predict methylation at masked CpG sites. The model achieved a mean squared error (MSE) of 0.014 and a Pearson correlation of 0.929 between predicted and actual methylation levels, indicating high predictive accuracy. Researchers also evaluated whether the model could capture biologically relevant features of DNA methylation. As such, they analyzed the learned representations of CpG sites in the embedding space. They found that CpG sites clustered based on their genomic contexts, suggesting that the model learned the regulatory features of the methylome. In addition, there was a clear separation between autosomes and sex chromosomes, indicating that MethylGPT also captured higher-order chromosomal features. Next, the team analyzed zero-shot embedding spaces. This showed a clear biological organization, clustering by sex, tissue type, and genomic context. Major tissue types formed well-defined clusters, indicating that the model learned methylation patterns specific to tissues without explicit supervision. Notably, MethylGPT also avoided batch effects, which often confound results in complex datasets. Besides, female and male samples demonstrated consistent separation, reflecting sex-specific differences. Next, the researchers assessed the ability of MethylGPT to predict chronological age from methylation patterns. To this end, they used a dataset of over 11,400 samples from diverse tissue types. Fine-tuning for age prediction led to robust age-dependent clustering. Notably, intrinsic age-related organization was evident even before fine-tuning. Moreover, MethylGPT outperformed existing age prediction methods (e.g., Horvath's clock and ElasticNet), achieving superior accuracy. Its median absolute error for age prediction was 4.45 years, further demonstrating its robustness. MethylGPT was also remarkably resilient to missing data. It exhibited stable performance with up to 70% missing data, outperforming multi-layer perceptron and ElasticNet approaches. MethylGPT's resilience to missing data (up to 70%) surpasses traditional models, proving useful in real-world scenarios with incomplete datasets. Analysis of methylation profiles during induced pluripotent stem cell (iPSC) reprogramming showed a clear rejuvenation trajectory; samples progressively transitioned to a younger methylation state over the course of reprogramming. The model was also able to identify the point during reprogramming (day 20) when cells began showing clear signs of epigenetic age reversal. Finally, the model's ability to predict disease risk was assessed. The pre-trained model was fine-tuned to predict the risk of 60 diseases and mortality. The model achieved an area under the curve of 0.74 and 0.72 on validation and test sets, respectively. In addition, they used this disease risk prediction framework to evaluate the impact of eight interventions on predicted disease incidence. Interventions included smoking cessation, high-intensity training, and the Mediterranean diet, among others, each of which showed varying degrees of effectiveness across disease categories. This showed distinct intervention-specific effects across disease categories, highlighting the potential of MethylGPT in predicting intervention-specific outcomes and optimizing tailored intervention strategies. Conclusions The findings illustrate that transformer architectures could effectively model DNA methylation patterns while preserving biological relevance. The organization of CpG sites based on regulatory features and genomic context suggests that the model captured fundamental aspects without explicit supervision. MethylGPT also demonstrated superior performance in age prediction across different tissues. Moreover, its robust performance in handling missing data (≤ 70%) underscores its potential utility in clinical and research applications. *Important notice: bioRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information. Journal reference: Preliminary scientific report. MethylGPT: a foundation model for the DNA methylome, Kejun Ying, Jinyeop Song, Haotian Cui, Yikun Zhang, Siyuan Li, Xingyu Chen, Hanna Liu, Alec Eames, Daniel L McCartney, Riccardo E. Marioni, Jesse R. Poganik, Mahdi Moqri, Bo Wang, Vadim N. Gladyshev bioRxiv 2024.10.30.621013; doi: 10.1101/2024.10.30.621013, https://www.biorxiv.org/content/10.1101/2024.10.30.621013v2