RNA secondary structure is a key determinant of RNA function. While deep learning models—particularly transformer architectures—have shown strong performance in RNA structure prediction, they often fail to generalise across RNA families, sequence lengths, and global structural motifs such as multiloops. In an attempt to complete our analysis of transformers in predicting RNA secondary structures, this project aims to analyse the internal embedding spaces learned by state-of-the-art deep learning models, e.g., Vanilla and GAT models (our transRNA model), RNAformer, RiNALMo, and BPfold, to understand the origins of these generalisation failures. The embedding layer of a transformer is a prerequisite for most transformer models, and there are a variety of methods for capturing relationships among input tokens. However, in the context of RNA folding, this embedding captures only pairing information, rather than more complex structural features such as multiloops, bulges, and hairpins.
Using both synthetic and biological RNA datasets, we will investigate how different training objectives (classification versus translation) impact the representation of sequence and structural information in transformer models. We aim to provide a systematic characterisation of what transformer embeddings encode for RNAs, how training objectives (e.g., classification vs sequence-to-structure “translation”) shape these representations, and why generalisation breaks on families, lengths, and motifs such as multiloops, bulges, and hairpins. Insights will guide embedding and architecture choices that improve out-of-distribution robustness. We will also pay particular attention to the imbalance between the number of RNA sequences and the number of distinct RNA structures, and how this imbalance affects representation learning.