Representation Learning for Long Sequential Data with Circular Dilated Convolutional Neural Networks

Abstract

Sequential data is prevalent in many application domains. To effectively and efficiently learn meaningful representations from long sequences, neural networks need properties such as a full-receptive field, scalability, broad information flow, and symmetry. However, conventional neural networks often struggle with handling long-range dependencies and maintaining efficiency, making it challenging to process long sequential data effectively. This research project aims to develop scalable neural networks for representation learning from long sequential data. We propose four novel neural networks, with this thesis focusing specifically on Circular Dilated Convolutional Neural Networks (CDIL-CNNs). CDIL-CNNs employ symmetric convolutions with exponentially increasing dilations and circular padding. This thesis demonstrates that each output element of the final layer in CDIL-CNNs has an equal chance to receive information from all input elements, with a computational complexity of O(N log2 N), where N represents the sequence length. Additionally, we incorporate CDIL-CNNs into a self-supervised learning framework to leverage substantial amounts of unlabeled sequences. This thesis employs masked learning, a technique that utilizes unmasked parts of a sequence to predict the masked parts during the pre-training stage. This approach enables our method to learn contextual information within the data and extract meaningful features from unlabeled sequences, thereby enhancing its generalization capability for various downstream tasks. We have tested CDIL-CNNs across various long sequential datasets, including synthetic data, images, text, time series, and DNA sequences. The experimental results demonstrate that our neural network architecture outperforms many state-of-the-art approaches.