Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Milvus Vector Database Implementation #4751

Merged
merged 33 commits into from
Dec 24, 2024

Conversation

franciscojavierarceo
Copy link
Member

@franciscojavierarceo franciscojavierarceo commented Nov 11, 2024

What this PR does / why we need it:

This PR introduces the Milvus Vector Database into Feast. This integration enhances the capability of Feast to handle and store vector embeddings, along with traditional features. More specifically, this allows for vector retrieval configurable at the feature-level.

  1. .github/workflows/pr_local_integration_tests.yml
  • Updated test cache key
  1. Makefile
  • Added milvus makefile target
  1. docs/how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md
  • Updated to remove path
  1. sdk/python/docs/source/feast.infra.online_stores.milvus_online_store.rst
  • Updated sphinx docs
  1. sdk/python/feast/infra/online_stores/milvus_online_store/milvus.py
  • Added Milvus online store implementation
  1. sdk/python/feast/repo_config.py
  • Added milvus to config
  1. sdk/python/feast/type_map.py
  • Updated _proto_value_to_value_type function
  1. sdk/python/tests/foo_provider.py
  • Updated requested_features as Optional
  1. sdk/python/tests/integration/online_store/test_universal_online.py
  • Added specific Milvus test...will likely remove later

Which issue(s) this PR fixes:

#4364

Misc

Will add docs in subsequent PR

@franciscojavierarceo franciscojavierarceo force-pushed the milvus-vector-db branch 2 times, most recently from a714f85 to 458d434 Compare November 20, 2024 18:05
@franciscojavierarceo franciscojavierarceo force-pushed the milvus-vector-db branch 3 times, most recently from c8a989f to c8f81a5 Compare November 27, 2024 13:42
@tchughesiv
Copy link
Contributor

@franciscojavierarceo as we add support for additional data store types, should we encourage devs to update the operator db store types accordingly as well? or should we treat that as a separate chore since the dev language is different?

// +kubebuilder:validation:Enum=snowflake.online;redis;ikv;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore

var ValidOnlineStoreDBStorePersistenceTypes = []string{
"snowflake.online",
"redis",
"ikv",
"datastore",
"dynamodb",
"bigtable",
"postgres",
"cassandra",
"mysql",
"hazelcast",
"singlestore",
}

looking at the python code, it appears we need to add several more supported types to the operator.

@franciscojavierarceo franciscojavierarceo force-pushed the milvus-vector-db branch 2 times, most recently from 83db9bc to a026729 Compare December 17, 2024 21:28
@franciscojavierarceo
Copy link
Member Author

@franciscojavierarceo as we add support for additional data store types, should we encourage devs to update the operator db store types accordingly as well? or should we treat that as a separate chore since the dev language is different?

// +kubebuilder:validation:Enum=snowflake.online;redis;ikv;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore

var ValidOnlineStoreDBStorePersistenceTypes = []string{
"snowflake.online",
"redis",
"ikv",
"datastore",
"dynamodb",
"bigtable",
"postgres",
"cassandra",
"mysql",
"hazelcast",
"singlestore",
}

looking at the python code, it appears we need to add several more supported types to the operator.

Definitely. We can add an explicit test for it too and update the docs. I can do that in a separate PR after I get this thing wrangled.

I'm still only partially through this and I still have quite a bit of work to do.

Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
…ine retrieval. long term this can all go in the FeatureView class and in get_online_features

Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Copy link
Collaborator

@shuchu shuchu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@franciscojavierarceo franciscojavierarceo merged commit 22c7b58 into master Dec 24, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants