Timetable: Dynamic Time Series Data Store Utilizing Managed Cloud Services

Abstract

Internet of Things (IoT) have in the recent years exploded in popularity and usage around the world. This advance have brought about new ways and technologies which enable companies and individuals to extend physical installations into the digital realm, in addition to the the advent of the cloud and managed solutions which has paved the way for storing, computing and analyzing vast amounts of data with ease. These factors have brought about what is now called the industrial internet of things. Cognite has positioned themselves as a global leader in this field and delivers a ready to use, general purpose platform for ingestion, aggregation and analytics of time series data, especially data emitting from IoT. The primary storage solution for this kind of data is a time series database (TSDB), these databases are however often created for a specific use case, and are therefore not general enough to support all needs emanating from Cognite. This research project looks at designing a dynamic time series database that suits a variety of data, especially one that relies on managed cloud services, primarily the Google Cloud Platform, with BigTable and Cloud PubSub. The work done throughout this research project shows that the need for such a dynamic datastore is needed when creating a solution that can deal with highly diversified and fluid time series data. In addition to this the architecture shows how one can leverage existing cloud solution to create a high performant time series data store, with scalability in mind. The prototype presented shows promising results as well boasting the ability of dynamic sharding of time series data, in wide column datastores.