Improving the Performance of Pipelined Query Processing with Skipping
Original version
Lecture Notes in Computer Science = Lecture notes in artificial intelligence 2012;7651:1-15 10.1007/s11280-013-0260-2Abstract
Web search engines need to provide high throughput and
short query latency. Recent results show that pipelined query processing
over a term-wise partitioned inverted index may have superior throughput.
However, the query processing latency and scalability with respect to
the collections size are the main challenges associated with this method.
In this paper, we evaluate the e ect of inverted index skipping on the
performance of pipelined query processing. Further, we introduce a novel
idea of using Max-Score pruning within pipelined query processing and
a new term assignment heuristic, partitioning by Max-Score. Our current
results indicate a signi cant improvement over the state-of-the-art
approach and lead to several further optimizations, which include dynamic
load balancing, intra-query concurrent processing and a hybrid
combination between pipelined and non-pipelined execution.