Lucene99HnswVectorsFormat

Lucene 9.9 vector format, which encodes numeric vector values into an associated graph connecting the documents having values. The graph is used to power HNSW search. The format consists of two files, and requires a FlatVectorsFormat to store the actual vectors:

.vex (vector index)

Stores graphs connecting the documents for each field organized as a list of nodes' neighbours as following:

• For each level:

• For each node:

• vint the number of neighbor nodes

• arrayvint the delta encoded neighbor ordinals

• After all levels are encoded, memory offsets for each node's neighbor nodes are appended to the end of the file. The offsets are encoded by [ ].

.vem (vector metadata) file

For each field:

• int32 field number

• int32 vector similarity function ordinal

• vlong offset to this field's index in the .vex file

• vlong length of this field's index data, in bytes

• vint dimension of this field's vectors

• int the number of documents having values for this field

• vint the maximum number of connections (neighbours) that each node can have

• vint number of levels in the graph

• Graph nodes by level. For each level

• vint the number of nodes on this level

• arrayvint for levels greater than 0 list of nodes on this level, stored as the level 0th delta encoded nodes' ordinals.