print lucene in inverted index format

Viewed 669

According to my understanding, Lucene uses inverted indexes. Is there any way to extract/print lucene index (lucene 6) in an inverted index format:

term1   <doc1, doc100, ..., doc555>
term1   <doc1, ..., doc100, ..., do89>
term1   <doc3, doc2, doc5, ...>
.
.
.
termn   <doc10, doc43, ..., dock>
3 Answers

Worked out a version that prints docId:tokenPos for Lucene 6.6.

Directory directory = new RAMDirectory();
Analyzer analyzer = new StandardAnalyzer();
IndexWriterConfig iwc = new IndexWriterConfig(analyzer);
iwc.setOpenMode(OpenMode.CREATE);
IndexWriter writer = new IndexWriter(directory, iwc);

FieldType type = new FieldType();
type.setStoreTermVectors(true);
type.setStoreTermVectorPositions(true);
type.setStoreTermVectorOffsets(true);
type.setIndexOptions(IndexOptions.DOCS);

Field fieldStore = new Field("text", "We hold that proof beyond a reasonable doubt is required.", type);
Document doc = new Document();
doc.add(fieldStore);
writer.addDocument(doc);

fieldStore = new Field("text", "We hold that proof requires reasoanble preponderance of the evidenceb.", type);
doc = new Document();
doc.add(fieldStore);
writer.addDocument(doc);

writer.close();

DirectoryReader reader = DirectoryReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);

MatchAllDocsQuery query = new MatchAllDocsQuery();
TopDocs hits = searcher.search(query, Integer.MAX_VALUE);

Map<String, Set<String>> invertedIndex = new HashMap<>();
BiFunction<Integer, Integer, Set<String>> mergeValue = 
    (docId, pos)-> {TreeSet<String> s = new TreeSet<>(); s.add((docId+1)+":"+pos); return s;};

for ( ScoreDoc scoreDoc: hits.scoreDocs ) {
    Fields termVs = reader.getTermVectors(scoreDoc.doc);
    Terms terms = termVs.terms("text");
    TermsEnum termsIt = terms.iterator();
    PostingsEnum docsAndPosEnum = null;
    BytesRef bytesRef;
    while ( (bytesRef = termsIt.next()) != null ) {
        docsAndPosEnum = termsIt.postings(docsAndPosEnum, PostingsEnum.ALL);
        docsAndPosEnum.nextDoc();
        int pos = docsAndPosEnum.nextPosition();
        String term = bytesRef.utf8ToString();
        invertedIndex.merge(
            term, 
            mergeValue.apply(scoreDoc.doc, pos), 
            (s1,s2)->{s1.addAll(s2); return s1;}
        );
    }
}
System.out.println( invertedIndex);
Related