Searching large data volumes with MISD processing

Abstract

Historically, supercomputing has focused on number crunching. Nonnumeric applications, such as information retrieval and analysis, have to a lesser extent been able to exploit the inherent resources of supercomputers. This thesis presents the results from the development of a novel multiple instruction, single data (MISD) architecture, targeting evaluation of complex queries in large data volumes. For such applications, this architecture provides a better price versus performance ratio, better use of the available memory bandwidth, lower power consumption, as well as linear scalability.

The core element of this technology is the Pattern Matching Chip (PMC).

Each chip provides 1024 processing elements, with an accumulated performance of 1010 operations per second. Multiple chips can be run in parallel with linear scalability, either within one computer, or in larger clusters. Up to half a million processing elements have been used in parallel in this project, providing 5 · 1013 operations per second at 48 GB per second data throughput, in a unit smaller than one cubic meter. Even larger systems can be constructed, still with linear scalability.

Through the novelty of this hardware architecture, the performance gained has enabled information processing in a way that would have been cost prohibitive with traditional computers. Such processing has demonstrated the capability of finding nuggets of valuable data in large and complex data volumes. The main effort – thus also the most important practical results – has been in bioinformatics. However, the technology has applicability in numerous other data mining applications.