Universal Indexes for Highly Repetitive Document Collections

Abstract

Indexing highly repetitive collections has become a relevant problem with the emergence of large repositoriesof versioned documents, among other applications. These collections may reach huge sizes, but are formedmostly of documents that are near-copies of others. Traditional techniques for indexing these collections failto properly exploit their regularities in order to reduce space.We introduce new techniques for compressing inverted indexes that exploit this near-copy regularity.They are based on run-length, Lempel-Ziv, or grammar compression of the differential inverted lists, insteadof the usual practice of gap-encoding them. We show that, in this highly repetitive setting, our compressionmethods significantly reduce the space obtained with classical techniques, at the price of moderate slowdowns.Moreover, our best methods are universal, that is, they do not need to know the versioning structure of thecollection, nor that a clear versioning structure even exists.We also introduce compressed self-indexes in the comparison. These are designed for general strings(not only natural language texts) and represent the text collection plus the index structure (not an invertedindex) in integrated form. We show that these techniques can compress much further, using a small fractionof the space required by our new inverted indexes. Yet, they are orders of magnitude slower.