Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Milvus Vector Database Implementation #4751

Merged
merged 33 commits into from
Dec 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
ab7d4fb
merged rebase
franciscojavierarceo Dec 9, 2024
a6c9005
adding configuration
franciscojavierarceo Nov 11, 2024
b61b965
updated
franciscojavierarceo Nov 12, 2024
88bca02
changed things and linted
franciscojavierarceo Nov 15, 2024
81bd84a
adding updated builds
franciscojavierarceo Nov 15, 2024
083fe67
adding repo config
franciscojavierarceo Nov 20, 2024
fcf32d9
updated
franciscojavierarceo Nov 25, 2024
662119a
renaming test and adding milvus to integration test
franciscojavierarceo Nov 27, 2024
4a7edd8
not quite working but have milvus talking
franciscojavierarceo Nov 27, 2024
4c16c0b
updated tests
franciscojavierarceo Dec 1, 2024
fabac18
apply() method now works
franciscojavierarceo Dec 2, 2024
2ee74c5
updated setup
franciscojavierarceo Dec 9, 2024
1859e76
checking in progresss...getting there
franciscojavierarceo Dec 13, 2024
35557ce
making some progress
franciscojavierarceo Dec 14, 2024
c54309a
partially running
franciscojavierarceo Dec 15, 2024
0094e27
checking in progress...finding issues still
franciscojavierarceo Dec 16, 2024
25a065e
have the apply working
franciscojavierarceo Dec 16, 2024
f1a92e0
adjusting some type issues
franciscojavierarceo Dec 16, 2024
3282a05
updated and removed test
franciscojavierarceo Dec 16, 2024
2dee462
have things behaving with the enviornment arg
franciscojavierarceo Dec 17, 2024
be8c5bc
updated milvus
franciscojavierarceo Dec 17, 2024
dbc11e2
almost have retrieval working, having to make a lot of changes to onl…
franciscojavierarceo Dec 18, 2024
37d93f1
almost have deserialization from the search results done
franciscojavierarceo Dec 19, 2024
67ad688
got the retrieval working now too :D
franciscojavierarceo Dec 19, 2024
d1c15cf
updates to fix linter and new signature for all implementations
franciscojavierarceo Dec 20, 2024
6d94d7a
linter
franciscojavierarceo Dec 20, 2024
fe518ed
more linting
franciscojavierarceo Dec 20, 2024
a639013
Removing some unnecessary code
franciscojavierarceo Dec 20, 2024
c91681f
removing change to setup
franciscojavierarceo Dec 20, 2024
bd5ff48
adding sphinx docs
franciscojavierarceo Dec 20, 2024
32c9a9f
adjusting workflow
franciscojavierarceo Dec 23, 2024
22bac8b
adding logging to debug
franciscojavierarceo Dec 23, 2024
134b908
changing to vectordb environment
franciscojavierarceo Dec 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pr_local_integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
uses: actions/cache@v4
with:
path: ${{ steps.uv-cache.outputs.dir }}
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
key: ${{ runner.os }}-${{ matrix.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', matrix.python-version)) }}
- name: Install dependencies
run: make install-python-dependencies-ci
- name: Test local integration tests
Expand Down
18 changes: 15 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ test-python-universal-postgres-online:
not test_snowflake" \
sdk/python/tests

test-python-universal-mysql-online:
test-python-universal-mysql-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.mysql_online_store.mysql_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.mysql \
Expand All @@ -292,7 +292,11 @@ test-python-universal-cassandra:
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.cassandra_online_store.cassandra_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.cassandra \
python -m pytest -x --integration \
sdk/python/tests
sdk/python/tests/integration/offline_store/test_feature_logging.py \
--ignore=sdk/python/tests/integration/offline_store/test_validation.py \
-k "not test_snowflake and \
not test_spark_materialization_consistency and \
not test_universal_materialization"

test-python-universal-hazelcast:
PYTHONPATH='.' \
Expand Down Expand Up @@ -330,7 +334,7 @@ test-python-universal-cassandra-no-cloud-providers:
not test_snowflake" \
sdk/python/tests

test-python-universal-elasticsearch-online:
test-python-universal-elasticsearch-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.elasticsearch_online_store.elasticsearch_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.elasticsearch \
Expand All @@ -349,6 +353,14 @@ test-python-universal-cassandra-no-cloud-providers:
not test_snowflake" \
sdk/python/tests

test-python-universal-milvus-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.milvus_online_store.milvus_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.milvus \
python -m pytest -n 8 --integration \
-k "test_retrieve_online_milvus_ocuments" \
sdk/python/tests --ignore=sdk/python/tests/integration/offline_store/test_dqm_validation.py

test-python-universal-singlestore-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.singlestore_repo_configuration \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ OnlineStore class names must end with the OnlineStore suffix!

### Contrib online stores

New online stores go in `sdk/python/feast/infra/online_stores/contrib/`.
New online stores go in `sdk/python/feast/infra/online_stores/`.

#### What is a contrib plugin?

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ feast.infra.online\_stores.milvus\_online\_store package
Submodules
----------

feast.infra.online\_stores.milvus\_online\_store.milvus module
--------------------------------------------------------------

.. automodule:: feast.infra.online_stores.milvus_online_store.milvus
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.milvus\_online\_store.milvus\_repo\_configuration module
-----------------------------------------------------------------------------------

Expand Down
Loading
Loading