-
Notifications
You must be signed in to change notification settings - Fork 25.3k
Wrap ES KNN queries with PatienceKNN query #127223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
🔍 Preview links for changed docs: 🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes. |
Pinging @elastic/es-search-relevance (Team:Search Relevance) |
Hi @tteofili, I've created a changelog YAML for you. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whew, way cleaner! Still some room for simplifying. But the index setting looks much nicer to me :)
server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java
Outdated
Show resolved
Hide resolved
…nseVectorFieldMapper.java Co-authored-by: Benjamin Trent <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So clean! I like it, and the performance numbers are ⚡
This exposes an option to add the patience-based early termination defined in apache/lucene#14094 to KNN queries in ES.
This leverages
PatienceKnnVectorQuery
which wraps ES KNN queries when mapping is one ofhnsw, int8_hnsw, int4_hnsw, bbq_hnsw
andearly_termination
is set totrue
(withinindex_options
).e.g.:
Currently this has the limitation that we can't profile that query because we can't extend the
PatienceKnnQuery
same as https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/vectors/ESKnnByteVectorQuery.java#L28-L38 because of visibility issues. However it's fixed in 10.3 branch via apache/lucene#14838 and we'll incorporate that as soon as 10.3 is released and included in ESmain
branch.