Add cosine similarity support for faiss engine #2376
+603
−42
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
VijayanB
force-pushed
the
faiss-cosine
branch
3 times, most recently
from
d6f16a1 to
6671e64
Compare
|
Adding additional unit and integration test for radial search. Will mark it as ready once i add those tests |
VijayanB
force-pushed
the
faiss-cosine
branch
6 times, most recently
from
eee45d5 to
4658bee
Compare
VijayanB
requested review from
heemin32,
navneet1v,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
martin-gaievski,
ryanbogan,
luyuncheng,
shatejas and
0ctopus13prime
as code owners
jmazanec15
reviewed
Jan 10, 2025
src/main/java/org/opensearch/knn/index/engine/AbstractKNNMethod.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/knn/index/engine/AbstractKNNMethod.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/knn/index/engine/AbstractKNNMethod.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/knn/index/engine/KNNLibraryIndexingContext.java
Show resolved
Hide resolved
src/main/java/org/opensearch/knn/index/mapper/ModelFieldMapper.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/knn/index/mapper/VectorTransformer.java
Outdated
Show resolved
Hide resolved
| private float[] getVectorForCreatingQueryRequest(VectorDataType vectorDataType, KNNEngine knnEngine) { | ||
| private float[] getVectorForCreatingQueryRequest(VectorDataType vectorDataType, KNNEngine knnEngine, SpaceType spaceType) { | ||
|
|
||
| // Cosine similarity is supported as Inner product by FAISS by normalizing input vector, hence, we have to normalize |
There was a problem hiding this comment.
Can we move this check out of this class? This class is already very crowded and I want to avoid adding more checks around engines. Instead, could we investigate either adding it to https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/engine/KNNLibrarySearchContext.java and/or adding a method in KNNVectorFieldType (https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldType.java#L85) that says "should normalize query" or, better yet, transformQuery?
There was a problem hiding this comment.
Moved it to KNNVectorFieldType
| @@ -700,7 +703,8 @@ protected void parseCreateField(ParseContext context, int dimension, VectorDataT | |||
| } | |||
| final float[] array = floatsArrayOptional.get(); | |||
| getVectorValidator().validateVector(array); | |||
| context.doc().addAll(getFieldsForFloatVector(array)); | |||
| final float[] transformedArray = getVectorTransformer().transform(array); | |||
There was a problem hiding this comment.
This will be called before the per-dimension processor too? What should contract around these 2 be? Im wondering if we even need the per-dimension or if we can wrap that in this new full vector transform.
FAISS engine doesn't support cosine similarity natively. However we can use inner product to achieve the same, because, when vectors are normalized then inner product will be same as cosine similarity. Hence, before ingestion and perform search, normalize the input vector and add it to faiss index with type as inner product. Since we will be storing normalized vector in segments, to get actual vectors, source can be used. By saving as normalized vector, we don't have to normalize whenever segments are merged. This will keep force merge time and search at competitive, provided we will face additional latency during indexing (one time where we normalize). We also support radial search for cosine similarity. Signed-off-by: Vijayan Balasubramanian <[email protected]>
Signed-off-by: Vijayan Balasubramanian <[email protected]>
VijayanB
force-pushed
the
faiss-cosine
branch
5 times, most recently
from
ff72fe5 to
5063334
Compare
VijayanB
force-pushed
the
faiss-cosine
branch
from
5063334 to
8764557
Compare
Signed-off-by: Vijayan Balasubramanian <[email protected]>
VijayanB
force-pushed
the
faiss-cosine
branch
from
8764557 to
7829347
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
FAISS engine doesn't support cosine similarity natively. However we can use inner product to achieve the same, because, when vectors are normalized then inner product will be same as cosine similarity. Hence, before ingestion, normalize the input vector, and add it to faiss index with type as inner product, and, before search, normalize query vector if space type is cosine and engine is faiss.
Since we will be storing normalized vector in segments, we don't have to normalize whenever segments are merged. This will keep force merge time and search at competitive, provided we will face additional latency during indexing (one time where we normalize). To avoid this additional latency, customers can normalize their data set and create inner product.
This also adds support to radial search, for both max distance and min score.
Related Issues
#2242
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.