diff --git a/examples/operator-quickstart/.gitignore b/examples/operator-quickstart/.gitignore new file mode 100644 index 0000000000..335ec9573d --- /dev/null +++ b/examples/operator-quickstart/.gitignore @@ -0,0 +1 @@ +*.tar.gz diff --git a/examples/operator-quickstart/01-Install.ipynb b/examples/operator-quickstart/01-Install.ipynb new file mode 100644 index 0000000000..25966e6941 --- /dev/null +++ b/examples/operator-quickstart/01-Install.ipynb @@ -0,0 +1,426 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Install Feast on Kind with the Feast Operator\n", + "## Objective\n", + "\n", + "Provide a reference implementation of a runbook to deploy a Feast development environment on a Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/docs/user/quick-start) and the [Feast Operator](../../infra/feast-operator/)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "* Kubernetes Cluster - e.g. [Kind](https://kind.sigs.k8s.io/) or OpenShift\n", + "* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) Kubernetes CLI tool." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Install Prerequisites\n", + "\n", + "The following commands install and configure all the prerequisites on MacOS environment. You can find the\n", + "equivalent instructions on the offical documentation pages:\n", + "* Install Kind and Container runtime (e.g. [Colima](https://github.com/abiosoft/colima)).\n", + "* Create Kind cluster named `feast`.\n", + "* Install and setup the `kubectl` context.\n", + "```bash\n", + "brew install colima\n", + "colima start\n", + "brew install kind\n", + "kind create cluster --name feast\n", + "kind start\n", + "brew install kubectl\n", + "kubectl config use-context kind-feast\n", + "```\n", + "\n", + "Additionally, we create a `feast` namespace and use it as the default for the `kubectl` CLI:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "namespace/feast created\n", + "Context \"default/api-cluster-lp5xn-lp5xn-sandbox1728-opentlc-com:6443/admin\" modified.\n" + ] + } + ], + "source": [ + "!kubectl create ns feast\n", + "!kubectl config set-context --current --namespace feast" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Validate the cluster setup:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NAME STATUS AGE\n", + "feast Active 3s\n" + ] + } + ], + "source": [ + "!kubectl get ns feast" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deployment Architecture\n", + "The primary objective of this runbook is to guide the deployment of Feast services on a Kubernetes Kind cluster, using the `postgres` template to set up a basic feature store.\n", + "\n", + "In this notebook, we will deploy a distributed topology of Feast services, which includes:\n", + "\n", + "* `Registry Server`: Handles metadata storage for feature definitions.\n", + "* `Online Store Server`: Uses the `Registry Server` to query metadata and is responsible for low-latency serving of features.\n", + "* `Offline Store Server`: Uses the `Registry Server` to query metadata and provides access to batch data for historical feature retrieval.\n", + "\n", + "Each service is backed by a `PostgreSQL` database, which is also deployed within the same Kind cluster." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Install PostgreSQL\n", + "Install the [reference deployment](postgres.yaml) to install and configure a simple PostgreSQL database." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "deployment.apps/postgres created\n", + "service/postgres created\n", + "deployment.apps/redis created\n", + "service/redis created\n", + "deployment.apps/postgres condition met\n", + "deployment.apps/redis condition met\n" + ] + } + ], + "source": [ + "!kubectl apply -f postgres.yaml -f redis.yaml\n", + "!kubectl wait --for=condition=available deployment/postgres --timeout=2m\n", + "!kubectl wait --for=condition=available deployment/redis --timeout=2m" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[33;1mWarning:\u001b[0m apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+\n", + "NAME READY STATUS RESTARTS AGE\n", + "pod/postgres-7ccc445fc7-9qq86 1/1 Running 0 16s\n", + "pod/redis-574b9b57-pwj4h 1/1 Running 0 16s\n", + "\n", + "NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n", + "service/postgres ClusterIP 172.30.151.48 5432/TCP 16s\n", + "service/redis ClusterIP 172.30.16.213 6379/TCP 16s\n", + "\n", + "NAME READY UP-TO-DATE AVAILABLE AGE\n", + "deployment.apps/postgres 1/1 1 1 16s\n", + "deployment.apps/redis 1/1 1 1 16s\n", + "\n", + "NAME DESIRED CURRENT READY AGE\n", + "replicaset.apps/postgres-7ccc445fc7 1 1 1 16s\n", + "replicaset.apps/redis-574b9b57 1 1 1 16s\n" + ] + } + ], + "source": [ + "!kubectl get all" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install the Operator" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/Users/tohughes/workspace/feast/infra/feast-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths=\"./...\" output:crd:artifacts:config=config/crd/bases\n", + "/Users/tohughes/workspace/feast/infra/feast-operator/bin/kustomize-v5.3.0 build config/crd | kubectl apply -f -\n", + "customresourcedefinition.apiextensions.k8s.io/featurestores.feast.dev unchanged\n", + "/Users/tohughes/workspace/feast/infra/feast-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths=\"./...\" output:crd:artifacts:config=config/crd/bases\n", + "cd config/manager && /Users/tohughes/workspace/feast/infra/feast-operator/bin/kustomize-v5.3.0 edit set image controller=quay.io/tchughesiv/feast-operator:0.42.0-dev\n", + "/Users/tohughes/workspace/feast/infra/feast-operator/bin/kustomize-v5.3.0 build config/default | kubectl apply -f -\n", + "namespace/feast-operator-system unchanged\n", + "customresourcedefinition.apiextensions.k8s.io/featurestores.feast.dev unchanged\n", + "serviceaccount/feast-operator-controller-manager unchanged\n", + "role.rbac.authorization.k8s.io/feast-operator-leader-election-role unchanged\n", + "clusterrole.rbac.authorization.k8s.io/feast-operator-featurestore-editor-role unchanged\n", + "clusterrole.rbac.authorization.k8s.io/feast-operator-featurestore-viewer-role unchanged\n", + "clusterrole.rbac.authorization.k8s.io/feast-operator-manager-role unchanged\n", + "clusterrole.rbac.authorization.k8s.io/feast-operator-metrics-reader unchanged\n", + "clusterrole.rbac.authorization.k8s.io/feast-operator-proxy-role unchanged\n", + "rolebinding.rbac.authorization.k8s.io/feast-operator-leader-election-rolebinding unchanged\n", + "clusterrolebinding.rbac.authorization.k8s.io/feast-operator-manager-rolebinding unchanged\n", + "clusterrolebinding.rbac.authorization.k8s.io/feast-operator-proxy-rolebinding unchanged\n", + "service/feast-operator-controller-manager-metrics-service unchanged\n", + "deployment.apps/feast-operator-controller-manager unchanged\n" + ] + } + ], + "source": [ + "#!kubectl apply -f ../../infra/feast-operator/dist/install.yaml\n", + "!make -C ../../infra/feast-operator install\n", + "!make -C ../../infra/feast-operator deploy IMG=quay.io/tchughesiv/feast-operator:0.42.0-dev\n", + "#!make -C ../../infra/feast-operator deploy IMG=quay.io//feast-operator:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Install the Feast services via FeatureStore CR\n", + "We'll use the Operator in this local repository to install the feast services. Install the [reference deployment](feast.yaml) to install and configure Feast." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "secret/feast-data-stores configured\n", + "featurestore.feast.dev/example unchanged\n" + ] + } + ], + "source": [ + "!kubectl apply -f feast.yaml" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Validate deployment\n", + "Validate application and service status:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NAME STATUS AGE\n", + "example Ready 123m\n", + "\u001b[33;1mWarning:\u001b[0m apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+\n", + "NAME READY STATUS RESTARTS AGE\n", + "pod/feast-example-66f8cccd7f-qzz6p 3/3 Running 0 123m\n", + "pod/postgres-7ccc445fc7-9qq86 1/1 Running 0 124m\n", + "pod/redis-574b9b57-pwj4h 1/1 Running 0 124m\n", + "\n", + "NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n", + "service/feast-example-offline ClusterIP 172.30.147.29 443/TCP 123m\n", + "service/feast-example-online ClusterIP 172.30.13.5 443/TCP 123m\n", + "service/feast-example-registry ClusterIP 172.30.73.106 443/TCP 123m\n", + "service/postgres ClusterIP 172.30.151.48 5432/TCP 124m\n", + "service/redis ClusterIP 172.30.16.213 6379/TCP 124m\n", + "\n", + "NAME READY UP-TO-DATE AVAILABLE AGE\n", + "deployment.apps/feast-example 1/1 1 1 123m\n", + "deployment.apps/postgres 1/1 1 1 124m\n", + "deployment.apps/redis 1/1 1 1 124m\n", + "\n", + "NAME DESIRED CURRENT READY AGE\n", + "replicaset.apps/feast-example-66f8cccd7f 1 1 1 123m\n", + "replicaset.apps/postgres-7ccc445fc7 1 1 1 124m\n", + "replicaset.apps/redis-574b9b57 1 1 1 124m\n", + "deployment.apps/feast-example condition met\n" + ] + } + ], + "source": [ + "!kubectl get feast\n", + "!kubectl get all\n", + "!kubectl wait --for=condition=available deployment/feast-example" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify the content of the feast client configuration file (it's generated by the Operator and stored in a `ConfigMap`)." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "project: credit_scoring_local\n", + "provider: local\n", + "offline_store:\n", + " host: feast-example-offline.feast.svc.cluster.local\n", + " type: remote\n", + " port: 443\n", + " scheme: https\n", + " cert: /tls/offline/tls.crt\n", + "online_store:\n", + " path: https://feast-example-online.feast.svc.cluster.local:443\n", + " type: remote\n", + " cert: /tls/online/tls.crt\n", + "registry:\n", + " path: feast-example-registry.feast.svc.cluster.local:443\n", + " registry_type: remote\n", + " cert: /tls/registry/tls.crt\n", + "auth:\n", + " type: no_auth\n", + "entity_key_serialization_version: 3\n" + ] + } + ], + "source": [ + "!kubectl get cm feast-example-client -o jsonpath='{.data.feature_store\\.yaml}'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the DB includes the expected tables." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " List of relations\n", + " Schema | Name | Type | Owner \n", + "--------+-------------------------+-------+----------\n", + " public | data_sources | table | postgres\n", + " public | entities | table | postgres\n", + " public | feast_metadata | table | postgres\n", + " public | feature_services | table | postgres\n", + " public | feature_views | table | postgres\n", + " public | managed_infra | table | postgres\n", + " public | on_demand_feature_views | table | postgres\n", + " public | permissions | table | postgres\n", + " public | projects | table | postgres\n", + " public | saved_datasets | table | postgres\n", + " public | stream_feature_views | table | postgres\n", + " public | validation_references | table | postgres\n", + "(12 rows)\n", + "\n" + ] + } + ], + "source": [ + "!kubectl exec deploy/postgres -- psql -h localhost -U postgres feast -c '\\dt'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, let's verify the `feast` version" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Feast SDK Version: \"0.42.0\"\n" + ] + } + ], + "source": [ + "!kubectl exec deployment/feast-example -c registry -- feast version" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.11" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/examples/operator-quickstart/02-Demo.ipynb b/examples/operator-quickstart/02-Demo.ipynb new file mode 100644 index 0000000000..b0728b4ffe --- /dev/null +++ b/examples/operator-quickstart/02-Demo.ipynb @@ -0,0 +1,501 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Run the Credit Score demo" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Apply the feature store definitions" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify the client `feature_store.yaml`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/.gitignore\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/LICENSE\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/README.md\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/app.py\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/credit_model.py\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/credit_history.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/credit_history_sample.csv\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/loan_table.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/loan_table_sample.csv\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/training_dataset_sample.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/zipcode_table.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/data/zipcode_table_sample.csv\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/credit_history.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/credit_history_sample.csv\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/loan_table.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/loan_table_sample.csv\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/training_dataset_sample.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/zipcode_table.parquet\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/data/zipcode_table_sample.csv\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/feature_store.yaml\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/feature_repo/features.py\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/requirements.txt\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/run.py\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/streamlit.png\n", + "feast-credit-score-local-tutorial-f43b44b245ae2632b582f14176392cfe31f98da9/streamlit_app.py\n" + ] + } + ], + "source": [ + "![ -f f43b44b.tar.gz ] || wget https://github.com/feast-dev/feast-credit-score-local-tutorial/archive/f43b44b.tar.gz\n", + "!kubectl cp f43b44b.tar.gz $(kubectl get pods -l 'feast.dev/name=example' -ojsonpath=\"{.items[*].metadata.name}\"):/feast-data -c registry\n", + "!kubectl exec deploy/feast-example -c registry -- rm -rf /feast-data/feast-credit-score-local-tutorial\n", + "!kubectl exec deploy/feast-example -c registry -- mkdir /feast-data/feast-credit-score-local-tutorial\n", + "!kubectl exec deploy/feast-example -c registry -- tar vfx /feast-data/f43b44b.tar.gz -C /feast-data/feast-credit-score-local-tutorial --strip-components 1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Move `feature_store.yaml` config to demo directory, then verify its contents." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "command terminated with exit code 1\n", + "project: credit_scoring_local\n", + "provider: local\n", + "offline_store:\n", + " type: duckdb\n", + "online_store:\n", + " type: redis\n", + " connection_string: redis.feast.svc.cluster.local:6379\n", + "registry:\n", + " path: postgresql+psycopg://postgres@postgres.feast.svc.cluster.local:5432/feast\n", + " registry_type: sql\n", + " cache_ttl_seconds: 60\n", + " sqlalchemy_config_kwargs:\n", + " echo: false\n", + " pool_pre_ping: true\n", + "auth:\n", + " type: no_auth\n", + "entity_key_serialization_version: 3\n" + ] + } + ], + "source": [ + "!kubectl exec deploy/feast-example -c registry -- [ -f /feast-data/feature_store.yaml ] || kubectl exec deploy/feast-example -c registry -- mv /feast-data/credit_scoring_local/feature_repo/feature_store.yaml /feast-data/feature_store.yaml\n", + "!kubectl exec deploy/feast-example -c registry -- cp -f /feast-data/feature_store.yaml /feast-data/feast-credit-score-local-tutorial/feature_repo/feature_store.yaml\n", + "!kubectl exec deploy/feast-example -c registry -- cat /feast-data/feast-credit-score-local-tutorial/feature_repo/feature_store.yaml" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Apply feature store data." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.11/site-packages/feast/feature_store.py:575: RuntimeWarning: On demand feature view is an experimental feature. This API is stable, but the functionality does not scale well for offline retrieval\n", + " warnings.warn(\n", + "No project found in the repository. Using project name credit_scoring_local defined in feature_store.yaml\n", + "Applying changes for project credit_scoring_local\n", + "Deploying infrastructure for zipcode_features\n", + "Deploying infrastructure for credit_history\n" + ] + } + ], + "source": [ + "!kubectl exec deploy/feast-example -c registry -- feast -c /feast-data/feast-credit-score-local-tutorial/feature_repo apply" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Materializing \u001b[1m\u001b[32m2\u001b[0m feature views to \u001b[1m\u001b[32m2025-01-09 01:51:04+00:00\u001b[0m into the \u001b[1m\u001b[32mredis\u001b[0m online store.\n", + "\n", + "\u001b[1m\u001b[32mzipcode_features\u001b[0m from \u001b[1m\u001b[32m2025-01-08 21:18:23+00:00\u001b[0m to \u001b[1m\u001b[32m2025-01-09 01:51:04+00:00\u001b[0m:\n", + "0it [00:00, ?it/s]\n", + "0it [00:00, ?it/s]\n", + "\u001b[1m\u001b[32mcredit_history\u001b[0m from \u001b[1m\u001b[32m2025-01-08 21:18:23+00:00\u001b[0m to \u001b[1m\u001b[32m2025-01-09 01:51:04+00:00\u001b[0m:\n" + ] + } + ], + "source": [ + "!kubectl exec deploy/feast-example -c registry -- bash -c 'cd /feast-data/feast-credit-score-local-tutorial/feature_repo && feast materialize-incremental $(date -u +\"%Y-%m-%dT%H:%M:%S\")'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Execute feast commands inside the client Pod" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we run the full test suite from the client folder." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NAME ENTITIES TYPE\n", + "zipcode_features {'zipcode'} FeatureView\n", + "credit_history {'dob_ssn'} FeatureView\n", + "total_debt_calc {'dob_ssn'} OnDemandFeatureView\n", + "spec:\n", + " name: total_debt_calc\n", + " features:\n", + " - name: total_debt_due\n", + " valueType: DOUBLE\n", + " sources:\n", + " application_data:\n", + " requestDataSource:\n", + " type: REQUEST_SOURCE\n", + " requestDataOptions:\n", + " schema:\n", + " - name: loan_amnt\n", + " valueType: INT64\n", + " name: application_data\n", + " credit_history:\n", + " featureViewProjection:\n", + " featureViewName: credit_history\n", + " featureColumns:\n", + " - name: credit_card_due\n", + " valueType: INT64\n", + " - name: mortgage_due\n", + " valueType: INT64\n", + " - name: student_loan_due\n", + " valueType: INT64\n", + " - name: vehicle_loan_due\n", + " valueType: INT64\n", + " - name: hard_pulls\n", + " valueType: INT64\n", + " - name: missed_payments_2y\n", + " valueType: INT64\n", + " - name: missed_payments_1y\n", + " valueType: INT64\n", + " - name: missed_payments_6m\n", + " valueType: INT64\n", + " - name: bankruptcies\n", + " valueType: INT64\n", + " timestampField: created_timestamp\n", + " createdTimestampColumn: created_timestamp\n", + " batchSource:\n", + " type: BATCH_FILE\n", + " timestampField: event_timestamp\n", + " createdTimestampColumn: created_timestamp\n", + " fileOptions:\n", + " fileFormat:\n", + " parquetFormat: {}\n", + " uri: data/credit_history.parquet\n", + " name: Credit history\n", + " featureTransformation:\n", + " userDefinedFunction:\n", + " name: total_debt_calc\n", + " body: gASVQQMAAAAAAACMCmRpbGwuX2RpbGyUjBBfY3JlYXRlX2Z1bmN0aW9ulJOUKGgAjAxfY3JlYXRlX2NvZGWUk5QoQxYCCyYCIAEO/wQBDv8EAg7+BAMw/AgFlEsBSwBLAEsCSwNLA0O6lwB0AQAAAAAAAAAAAABqAQAAAAAAAAAApgAAAKsAAAAAAAAAAAB9AXwAZAEZAAAAAAAAAAAAfABkAhkAAAAAAAAAAAB6AAAAfABkAxkAAAAAAAAAAAB6AAAAfABkBBkAAAAAAAAAAAB6AAAAfABkBRkAAAAAAAAAAAB6AAAAoAIAAAAAAAAAAAAAAAAAAAAAAAAAAHQGAAAAAAAAAAAAAKYBAACrAQAAAAAAAAAAfAFkBjwAAAB8AVMAlChOjA9jcmVkaXRfY2FyZF9kdWWUjAxtb3J0Z2FnZV9kdWWUjBBzdHVkZW50X2xvYW5fZHVllIwQdmVoaWNsZV9sb2FuX2R1ZZSMCWxvYW5fYW1udJSMDnRvdGFsX2RlYnRfZHVllHSUKIwCcGSUjAlEYXRhRnJhbWWUjAZhc3R5cGWUjAVmbG9hdJR0lIwLZmVhdHVyZXNfZGaUjAJkZpSGlIxGL2ZlYXN0LWRhdGEvZmVhc3QtY3JlZGl0LXNjb3JlLWxvY2FsLXR1dG9yaWFsL2ZlYXR1cmVfcmVwby9mZWF0dXJlcy5weZSMD3RvdGFsX2RlYnRfY2FsY5RoF0tRQ2uAAPUWAAoMjByJHowegELgCBPQFCXUCCaoG7Be1ClE0QhE2AgT0BQm1Agn8QMBCSjYKjXQNkjUKknxAwEJSgHgCBOQS9QIIPEFAgkh9wYABw2CZo1VgW2EbfAJAAUH0AcX0QQY8AoADA6ASZRDAJQpKXSUUpR9lIwIX19uYW1lX1+UjAhfX21haW5fX5RzaBdOTnSUUpR9lH2UjA9fX2Fubm90YXRpb25zX1+UfZQoaBOMEXBhbmRhcy5jb3JlLmZyYW1llGgPk5SMBnJldHVybpRoJnVzhpRiaBwojAJwZJRoAIwOX2ltcG9ydF9tb2R1bGWUk5SMBnBhbmRhc5SFlFKUjAVmbG9hdJRoAIwKX2xvYWRfdHlwZZSTlGgRhZRSlHUwLg==\n", + " bodyText: \"@on_demand_feature_view(\\n sources=[\\n credit_history,\\n\\\n", + " \\ input_request,\\n ],\\n schema=[\\n Field(name='total_debt_due',\\\n", + " \\ dtype=Float64),\\n ],\\n mode=\\\"pandas\\\",\\n)\\ndef total_debt_calc(features_df:\\\n", + " \\ pd.DataFrame) -> pd.DataFrame:\\n df = pd.DataFrame()\\n df['total_debt_due']\\\n", + " \\ = (\\n features_df['credit_card_due'] + features_df['mortgage_due']\\\n", + " \\ + \\n features_df['student_loan_due'] + features_df['vehicle_loan_due']\\\n", + " \\ + \\n features_df['loan_amnt']\\n ).astype(float)\\n return df\\\n", + " \\ \\n\"\n", + " mode: pandas\n", + " entities:\n", + " - __dummy\n", + " entityColumns:\n", + " - name: __dummy_id\n", + " valueType: STRING\n", + "meta:\n", + " createdTimestamp: '2025-01-08T21:18:14.368352Z'\n", + " lastUpdatedTimestamp: '2025-01-09T01:50:55.122536Z'\n", + "\n", + "NAME DESCRIPTION TYPE\n", + "dob_ssn Date of birth and last four digits of social security number ValueType.STRING\n", + "zipcode ValueType.INT64\n" + ] + } + ], + "source": [ + "!kubectl exec deploy/feast-example -c registry -- feast -c /feast-data/feast-credit-score-local-tutorial/feature_repo feature-views list\n", + "!kubectl exec deploy/feast-example -c registry -- feast -c /feast-data/feast-credit-score-local-tutorial/feature_repo on-demand-feature-views describe total_debt_calc\n", + "!kubectl exec deploy/feast-example -c registry -- feast -c /feast-data/feast-credit-score-local-tutorial/feature_repo entities list" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Run test code inside the client Pod" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we run the full test suite from the client folder." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting streamlit (from -r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading streamlit-1.41.1-py2.py3-none-any.whl.metadata (8.5 kB)\n", + "Collecting shap (from -r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2))\n", + " Downloading shap-0.46.0-cp311-cp311-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (24 kB)\n", + "Requirement already satisfied: pandas in /usr/local/lib/python3.11/site-packages (from -r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 3)) (2.2.3)\n", + "Collecting scikit-learn (from -r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 4))\n", + " Downloading scikit_learn-1.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)\n", + "Collecting matplotlib (from -r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 5))\n", + " Downloading matplotlib-3.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\n", + "Collecting altair<6,>=4.0 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading altair-5.5.0-py3-none-any.whl.metadata (11 kB)\n", + "Collecting blinker<2,>=1.0.0 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading blinker-1.9.0-py3-none-any.whl.metadata (1.6 kB)\n", + "Requirement already satisfied: cachetools<6,>=4.0 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (5.5.0)\n", + "Requirement already satisfied: click<9,>=7.0 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (8.1.8)\n", + "Requirement already satisfied: numpy<3,>=1.23 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (1.26.4)\n", + "Requirement already satisfied: packaging<25,>=20 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (24.2)\n", + "Collecting pillow<12,>=7.1.0 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading pillow-11.1.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (9.1 kB)\n", + "Requirement already satisfied: protobuf<6,>=3.20 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (4.25.5)\n", + "Requirement already satisfied: pyarrow>=7.0 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (17.0.0)\n", + "Requirement already satisfied: requests<3,>=2.27 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (2.32.3)\n", + "Requirement already satisfied: rich<14,>=10.14.0 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (13.9.4)\n", + "Requirement already satisfied: tenacity<10,>=8.1.0 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (8.5.0)\n", + "Requirement already satisfied: toml<2,>=0.10.1 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (0.10.2)\n", + "Requirement already satisfied: typing-extensions<5,>=4.3.0 in /usr/local/lib/python3.11/site-packages (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (4.12.2)\n", + "Collecting watchdog<7,>=2.1.5 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl.metadata (44 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.3/44.3 kB 16.7 MB/s eta 0:00:00\n", + "Collecting gitpython!=3.1.19,<4,>=3.0.7 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading GitPython-3.1.44-py3-none-any.whl.metadata (13 kB)\n", + "Collecting pydeck<1,>=0.8.0b4 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading pydeck-0.9.1-py2.py3-none-any.whl.metadata (4.1 kB)\n", + "Collecting tornado<7,>=6.0.3 (from streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading tornado-6.4.2-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB)\n", + "Collecting scipy (from shap->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2))\n", + " Downloading scipy-1.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.0/62.0 kB 24.0 MB/s eta 0:00:00\n", + "Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.11/site-packages (from shap->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2)) (4.67.1)\n", + "Collecting slicer==0.0.8 (from shap->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2))\n", + " Downloading slicer-0.0.8-py3-none-any.whl.metadata (4.0 kB)\n", + "Collecting numba (from shap->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2))\n", + " Downloading numba-0.60.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)\n", + "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.11/site-packages (from shap->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2)) (3.1.0)\n", + "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/site-packages (from pandas->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 3)) (2.9.0.post0)\n", + "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/site-packages (from pandas->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 3)) (2024.2)\n", + "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/site-packages (from pandas->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 3)) (2024.2)\n", + "Collecting joblib>=1.2.0 (from scikit-learn->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 4))\n", + " Downloading joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)\n", + "Collecting threadpoolctl>=3.1.0 (from scikit-learn->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 4))\n", + " Downloading threadpoolctl-3.5.0-py3-none-any.whl.metadata (13 kB)\n", + "Collecting contourpy>=1.0.1 (from matplotlib->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 5))\n", + " Downloading contourpy-1.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.4 kB)\n", + "Collecting cycler>=0.10 (from matplotlib->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 5))\n", + " Downloading cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)\n", + "Collecting fonttools>=4.22.0 (from matplotlib->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 5))\n", + " Downloading fonttools-4.55.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (165 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 165.1/165.1 kB 15.5 MB/s eta 0:00:00\n", + "Collecting kiwisolver>=1.3.1 (from matplotlib->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 5))\n", + " Downloading kiwisolver-1.4.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)\n", + "Collecting pyparsing>=2.3.1 (from matplotlib->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 5))\n", + " Downloading pyparsing-3.2.1-py3-none-any.whl.metadata (5.0 kB)\n", + "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/site-packages (from altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (3.1.5)\n", + "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.11/site-packages (from altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (4.23.0)\n", + "Collecting narwhals>=1.14.2 (from altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading narwhals-1.21.1-py3-none-any.whl.metadata (10.0 kB)\n", + "Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.19,<4,>=3.0.7->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading gitdb-4.0.12-py3-none-any.whl.metadata (1.2 kB)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 3)) (1.17.0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests<3,>=2.27->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (3.4.1)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests<3,>=2.27->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (3.10)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/site-packages (from requests<3,>=2.27->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (2.3.0)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests<3,>=2.27->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (2024.12.14)\n", + "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/site-packages (from rich<14,>=10.14.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (3.0.0)\n", + "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/site-packages (from rich<14,>=10.14.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (2.19.1)\n", + "Collecting llvmlite<0.44,>=0.43.0dev0 (from numba->shap->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 2))\n", + " Downloading llvmlite-0.43.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)\n", + "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.19,<4,>=3.0.7->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1))\n", + " Downloading smmap-5.0.2-py3-none-any.whl.metadata (4.3 kB)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/site-packages (from jinja2->altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (3.0.2)\n", + "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.11/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (24.3.0)\n", + "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.11/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (2024.10.1)\n", + "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.11/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (0.35.1)\n", + "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.11/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (0.22.3)\n", + "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich<14,>=10.14.0->streamlit->-r /feast-data/feast-credit-score-local-tutorial/requirements.txt (line 1)) (0.1.2)\n", + "Downloading streamlit-1.41.1-py2.py3-none-any.whl (9.1 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.1/9.1 MB 82.7 MB/s eta 0:00:00\n", + "Downloading shap-0.46.0-cp311-cp311-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (540 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 540.2/540.2 kB 80.1 MB/s eta 0:00:00\n", + "Downloading slicer-0.0.8-py3-none-any.whl (15 kB)\n", + "Downloading scikit_learn-1.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 96.6 MB/s eta 0:00:00\n", + "Downloading matplotlib-3.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.6 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 100.2 MB/s eta 0:00:00\n", + "Downloading altair-5.5.0-py3-none-any.whl (731 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.2/731.2 kB 88.2 MB/s eta 0:00:00\n", + "Downloading blinker-1.9.0-py3-none-any.whl (8.5 kB)\n", + "Downloading contourpy-1.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (326 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 326.2/326.2 kB 71.0 MB/s eta 0:00:00\n", + "Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)\n", + "Downloading fonttools-4.55.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 98.0 MB/s eta 0:00:00\n", + "Downloading GitPython-3.1.44-py3-none-any.whl (207 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 207.6/207.6 kB 56.4 MB/s eta 0:00:00\n", + "Downloading joblib-1.4.2-py3-none-any.whl (301 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.8/301.8 kB 68.8 MB/s eta 0:00:00\n", + "Downloading kiwisolver-1.4.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 93.7 MB/s eta 0:00:00\n", + "Downloading pillow-11.1.0-cp311-cp311-manylinux_2_28_x86_64.whl (4.5 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 89.1 MB/s eta 0:00:00\n", + "Downloading pydeck-0.9.1-py2.py3-none-any.whl (6.9 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 98.1 MB/s eta 0:00:00\n", + "Downloading pyparsing-3.2.1-py3-none-any.whl (107 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.7/107.7 kB 37.5 MB/s eta 0:00:00\n", + "Downloading scipy-1.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (40.6 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.6/40.6 MB 73.6 MB/s eta 0:00:00\n", + "Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)\n", + "Downloading tornado-6.4.2-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (437 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 437.2/437.2 kB 77.5 MB/s eta 0:00:00\n", + "Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl (79 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.1/79.1 kB 30.9 MB/s eta 0:00:00\n", + "Downloading numba-0.60.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 98.4 MB/s eta 0:00:00\n", + "Downloading gitdb-4.0.12-py3-none-any.whl (62 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 24.9 MB/s eta 0:00:00\n", + "Downloading llvmlite-0.43.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.9 MB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.9/43.9 MB 72.5 MB/s eta 0:00:00\n", + "Downloading narwhals-1.21.1-py3-none-any.whl (282 kB)\n", + " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 282.7/282.7 kB 63.2 MB/s eta 0:00:00\n", + "Downloading smmap-5.0.2-py3-none-any.whl (24 kB)\n", + "Installing collected packages: watchdog, tornado, threadpoolctl, smmap, slicer, scipy, pyparsing, pillow, narwhals, llvmlite, kiwisolver, joblib, fonttools, cycler, contourpy, blinker, scikit-learn, pydeck, numba, matplotlib, gitdb, shap, gitpython, altair, streamlit\n", + "\n", + "[notice] A new release of pip is available: 24.0 -> 24.3.1\n", + "[notice] To update, run: pip install --upgrade pip\n", + "Successfully installed altair-5.5.0 blinker-1.9.0 contourpy-1.3.1 cycler-0.12.1 fonttools-4.55.3 gitdb-4.0.12 gitpython-3.1.44 joblib-1.4.2 kiwisolver-1.4.8 llvmlite-0.43.0 matplotlib-3.10.0 narwhals-1.21.1 numba-0.60.0 pillow-11.1.0 pydeck-0.9.1 pyparsing-3.2.1 scikit-learn-1.6.0 scipy-1.15.0 shap-0.46.0 slicer-0.0.8 smmap-5.0.2 streamlit-1.41.1 threadpoolctl-3.5.0 tornado-6.4.2 watchdog-6.0.0\n", + "Loan rejected!\n", + "\n", + "Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.\n", + "\n", + "\n", + " You can now view your Streamlit app in your browser.\n", + "\n", + " Local URL: http://localhost:8501\n", + " Network URL: http://10.131.0.79:8501\n", + " External URL: http://18.218.252.97:8501\n", + "\n" + ] + } + ], + "source": [ + "!kubectl exec deploy/feast-example -c registry -- python -m venv --system-site-packages /feast-data/venv\n", + "!kubectl exec deploy/feast-example -c registry -- bash -c 'source /feast-data/venv/bin/activate && pip install -r /feast-data/feast-credit-score-local-tutorial/requirements.txt'\n", + "!kubectl exec deploy/feast-example -c registry -- bash -c 'source /feast-data/venv/bin/activate && cd /feast-data/feast-credit-score-local-tutorial && python run.py'\n", + "!kubectl exec deploy/feast-example -c registry -- bash -c 'source /feast-data/venv/bin/activate && cd /feast-data/feast-credit-score-local-tutorial && streamlit run --server.port 8501 streamlit_app.py'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```bash\n", + "$ kubectl port-forward deploy/feast-example 8501:8501\n", + "```\n", + "Then navigate to the local URL on which Streamlit is being served.\n", + "\n", + "http://localhost:8501\n", + "\n", + "```bash\n", + "$ kubectl port-forward svc/feast-example-online 8888:443\n", + "```\n", + "Then navigate to the local URL on which Streamlit is being served.\n", + "\n", + "https://localhost:8888/docs#/" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.11" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/examples/operator-quickstart/03-Uninstall.ipynb b/examples/operator-quickstart/03-Uninstall.ipynb new file mode 100644 index 0000000000..ad02f1f624 --- /dev/null +++ b/examples/operator-quickstart/03-Uninstall.ipynb @@ -0,0 +1,54 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Uninstall deployment" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl delete -f client.yaml\n", + "!kubectl delete -f feast.yaml\n", + "!kubectl delete -f postgres.yaml\n", + "!kubectl delete -f redis.yaml\n", + "!kubectl delete -f ../../infra/feast-operator/dist/install.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl get all" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.4" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/examples/operator-quickstart/README.md b/examples/operator-quickstart/README.md new file mode 100644 index 0000000000..20fba851d6 --- /dev/null +++ b/examples/operator-quickstart/README.md @@ -0,0 +1,7 @@ +# Install and run Feast with the Operator + +The following notebooks will guide you through an end-to-end journey to install and validate a simple Feast feature store in a +Kind Kubernetes or OpenShift cluster: +* [01-Install.ipynb](./01-Install.ipynb): Install and configure the cluster with the Operator. +* [02-Demo.ipynb](./02-Demo.ipynb): In a kind/k8s cluster, validate the feature store with demo application. +* [03-Uninstall.ipynb](./03-Uninstall.ipynb): Clear the installed deployments. diff --git a/examples/operator-quickstart/feast.yaml b/examples/operator-quickstart/feast.yaml new file mode 100644 index 0000000000..b407e2cfce --- /dev/null +++ b/examples/operator-quickstart/feast.yaml @@ -0,0 +1,49 @@ +--- +apiVersion: v1 +kind: Secret +metadata: + name: feast-data-stores +stringData: + redis: | + connection_string: redis.feast.svc.cluster.local:6379 + sql: | + path: postgresql+psycopg://postgres@postgres.feast.svc.cluster.local:5432/feast + cache_ttl_seconds: 60 + sqlalchemy_config_kwargs: + echo: false + pool_pre_ping: true + +--- +apiVersion: feast.dev/v1alpha1 +kind: FeatureStore +metadata: + name: example +spec: + feastProject: credit_scoring_local + services: + offlineStore: + persistence: + file: + type: duckdb + image: quay.io/tchughesiv/feature-server:0.42.0 + imagePullPolicy: Always + onlineStore: + persistence: + store: + type: redis + secretRef: + name: feast-data-stores + image: quay.io/tchughesiv/feature-server:0.42.0 + imagePullPolicy: Always + registry: + local: + persistence: + store: + type: sql + secretRef: + name: feast-data-stores + image: quay.io/tchughesiv/feature-server:0.42.0 + imagePullPolicy: Always + env: + - name: MPLCONFIGDIR + value: /tmp diff --git a/examples/operator-quickstart/postgres.yaml b/examples/operator-quickstart/postgres.yaml new file mode 100644 index 0000000000..7280053519 --- /dev/null +++ b/examples/operator-quickstart/postgres.yaml @@ -0,0 +1,46 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: postgres +spec: + replicas: 1 + selector: + matchLabels: + app: postgres + template: + metadata: + labels: + app: postgres + spec: + containers: + - name: postgres + image: 'postgres:16-alpine' + ports: + - containerPort: 5432 + env: + - name: POSTGRES_DB + value: feast + - name: POSTGRES_HOST_AUTH_METHOD + value: trust + volumeMounts: + - mountPath: /var/lib/postgresql + name: postgresdata + volumes: + - name: postgresdata + emptyDir: {} + +--- +apiVersion: v1 +kind: Service +metadata: + name: postgres + labels: + app: postgres +spec: + type: ClusterIP + ports: + - port: 5432 + targetPort: 5432 + protocol: TCP + selector: + app: postgres \ No newline at end of file diff --git a/examples/operator-quickstart/redis.yaml b/examples/operator-quickstart/redis.yaml new file mode 100644 index 0000000000..e8dbf1c98e --- /dev/null +++ b/examples/operator-quickstart/redis.yaml @@ -0,0 +1,38 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: redis +spec: + replicas: 1 + selector: + matchLabels: + app: redis + template: + metadata: + labels: + app: redis + spec: + containers: + - name: redis + image: 'bitnami/redis:latest' + ports: + - containerPort: 6379 + env: + - name: ALLOW_EMPTY_PASSWORD + value: "yes" + +--- +apiVersion: v1 +kind: Service +metadata: + name: redis + labels: + app: redis +spec: + type: ClusterIP + ports: + - port: 6379 + targetPort: 6379 + protocol: TCP + selector: + app: redis \ No newline at end of file diff --git a/infra/feast-operator/Makefile b/infra/feast-operator/Makefile index 6984ac66e7..49b86f08a4 100644 --- a/infra/feast-operator/Makefile +++ b/infra/feast-operator/Makefile @@ -111,7 +111,7 @@ vet: ## Run go vet against code. go vet ./... .PHONY: test -test: build-installer fmt vet lint envtest ## Run tests. +test: build-installer vet lint envtest ## Run tests. KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out # Utilize Kind or modify the e2e tests to load the image locally, enabling compatibility with other vendors. @@ -172,7 +172,7 @@ docker-buildx: ## Build and push docker image for the manager for cross-platform rm Dockerfile.cross .PHONY: build-installer -build-installer: manifests generate kustomize ## Generate a consolidated YAML with CRDs and deployment. +build-installer: manifests generate-ref kustomize ## Generate a consolidated YAML with CRDs and deployment. mkdir -p dist cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG} $(KUSTOMIZE) build config/default > dist/install.yaml @@ -203,7 +203,8 @@ undeploy: kustomize ## Undeploy controller from the K8s cluster specified in ~/. ##@ Dependencies ## Location to install dependencies to -LOCALBIN ?= $(shell pwd)/bin +LOCALDIR ?= $(shell pwd) +LOCALBIN ?= $(LOCALDIR)/bin $(LOCALBIN): mkdir -p $(LOCALBIN) @@ -211,12 +212,14 @@ $(LOCALBIN): KUBECTL ?= kubectl KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION) CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION) +CRD_REF_DOCS ?= $(LOCALBIN)/crd-ref-docs-$(CRD_REF_DOCS_VERSION) ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION) GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION) ## Tool Versions KUSTOMIZE_VERSION ?= v5.3.0 CONTROLLER_TOOLS_VERSION ?= v0.14.0 +CRD_REF_DOCS_VERSION ?= v0.1.0 ENVTEST_VERSION ?= release-0.17 GOLANGCI_LINT_VERSION ?= v1.57.2 @@ -271,6 +274,15 @@ OPERATOR_SDK = $(shell which operator-sdk) endif endif +.PHONY: crd-ref-docs +crd-ref-docs: $(CRD_REF_DOCS) ## Download crd-ref-docs locally if necessary. +$(CRD_REF_DOCS): $(LOCALBIN) + $(call go-install-tool,$(CRD_REF_DOCS),github.com/elastic/crd-ref-docs,$(CRD_REF_DOCS_VERSION)) + +.PHONY: generate-ref +generate-ref: generate fmt crd-ref-docs + $(CRD_REF_DOCS) --log-level=WARN --config=$(LOCALDIR)/docs/crd-ref-templates/config.yaml --source-path=$(LOCALDIR)/api/v1alpha1 --renderer=markdown --templates-dir=$(LOCALDIR)/docs/crd-ref-templates/markdown --output-path=$(LOCALDIR)/docs/api/markdown/ref.md + .PHONY: bundle bundle: manifests kustomize operator-sdk ## Generate bundle manifests and metadata, then validate generated files. $(OPERATOR_SDK) generate kustomize manifests -q diff --git a/infra/feast-operator/README.md b/infra/feast-operator/README.md index 3012eb63d4..2abe6e836f 100644 --- a/infra/feast-operator/README.md +++ b/infra/feast-operator/README.md @@ -1,10 +1,12 @@ # Feast Operator This is a K8s Operator that can be used to deploy and manage **Feast**, an open source feature store for machine learning. +### **[FeatureStore CR API Reference](docs/api/markdown/ref.md)** + ## Getting Started ### Prerequisites -- go version v1.21.0+ +- go version v1.21 - docker version 17.03+. - kubectl version v1.11.3+. - Access to a Kubernetes v1.11.3+ cluster. @@ -153,6 +155,3 @@ make test-e2e # delete cluster once you are done. kind delete cluster ``` - - - diff --git a/infra/feast-operator/config/samples/kustomization.yaml b/infra/feast-operator/config/samples/kustomization.yaml index 4869cc7b24..ec9b395f9d 100644 --- a/infra/feast-operator/config/samples/kustomization.yaml +++ b/infra/feast-operator/config/samples/kustomization.yaml @@ -1,4 +1,4 @@ ## Append samples of your project ## resources: -- v1alpha1_featurestore.yaml +- v1alpha1_featurestore_all_services_default.yaml #+kubebuilder:scaffold:manifestskustomizesamples diff --git a/infra/feast-operator/config/samples/v1alpha1_featurestore_all_services_default.yaml b/infra/feast-operator/config/samples/v1alpha1_featurestore_all_services_default.yaml index 1dd156378d..de7d474bcc 100644 --- a/infra/feast-operator/config/samples/v1alpha1_featurestore_all_services_default.yaml +++ b/infra/feast-operator/config/samples/v1alpha1_featurestore_all_services_default.yaml @@ -1,14 +1,9 @@ apiVersion: feast.dev/v1alpha1 kind: FeatureStore metadata: - name: sample-all-default + name: sample-all-services spec: feastProject: my_project services: - onlineStore: - image: 'feastdev/feature-server:0.40.0' - offlineStore: - image: 'feastdev/feature-server:0.40.0' - registry: - local: - image: 'feastdev/feature-server:0.40.0' \ No newline at end of file + onlineStore: {} + offlineStore: {} diff --git a/infra/feast-operator/config/samples/v1alpha1_featurestore_db_persistence.yaml b/infra/feast-operator/config/samples/v1alpha1_featurestore_db_persistence.yaml index fd6feb79f2..93ab10d0c3 100644 --- a/infra/feast-operator/config/samples/v1alpha1_featurestore_db_persistence.yaml +++ b/infra/feast-operator/config/samples/v1alpha1_featurestore_db_persistence.yaml @@ -3,20 +3,32 @@ kind: Secret metadata: name: postgres-secret namespace: test + labels: + app: postgres +stringData: + POSTGRES_DB: feast + POSTGRES_USER: feast + POSTGRES_PASSWORD: feast +--- +apiVersion: v1 +kind: Secret +metadata: + name: feast-data-stores + namespace: test stringData: postgres-secret-parameters: | - path: postgresql+psycopg://postgres:mysecretpassword@127.0.0.1:55001/feast + path: postgresql+psycopg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres.test.svc.cluster.local:5432/${POSTGRES_DB} cache_ttl_seconds: 60 sqlalchemy_config_kwargs: - echo: false - pool_pre_ping: true + echo: false + pool_pre_ping: true postgres: | - host: 127.0.0.1 - port: 55001 - database: feast + host: postgres.test.svc.cluster.local + port: 5432 + database: ${POSTGRES_DB} db_schema: public - user: postgres - password: mysecretpassword + user: ${POSTGRES_USER} + password: ${POSTGRES_PASSWORD} --- apiVersion: feast.dev/v1alpha1 kind: FeatureStore @@ -31,12 +43,44 @@ spec: store: type: postgres secretRef: + name: feast-data-stores + env: + - name: POSTGRES_DB + valueFrom: + secretKeyRef: name: postgres-secret + key: POSTGRES_DB + - name: POSTGRES_USER + valueFrom: + secretKeyRef: + name: postgres-secret + key: POSTGRES_USER + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: postgres-secret + key: POSTGRES_PASSWORD registry: local: persistence: store: type: sql secretRef: - name: postgres-secret + name: feast-data-stores secretKeyName: postgres-secret-parameters # optional, will use store.type by default as the SecretKeyName if none is specified, in this case that's "sql" + env: + - name: POSTGRES_DB + valueFrom: + secretKeyRef: + name: postgres-secret + key: POSTGRES_DB + - name: POSTGRES_USER + valueFrom: + secretKeyRef: + name: postgres-secret + key: POSTGRES_USER + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: postgres-secret + key: POSTGRES_PASSWORD diff --git a/infra/feast-operator/docs/api/markdown/ref.md b/infra/feast-operator/docs/api/markdown/ref.md new file mode 100644 index 0000000000..a03e105f27 --- /dev/null +++ b/infra/feast-operator/docs/api/markdown/ref.md @@ -0,0 +1,507 @@ +# API Reference + +## Packages +- [feast.dev/v1alpha1](#feastdevv1alpha1) + + +## feast.dev/v1alpha1 + +Package v1alpha1 contains API Schema definitions for the v1alpha1 API group + +### Resource Types +- [FeatureStore](#featurestore) + + + +#### AuthzConfig + + + +AuthzConfig defines the authorization settings for the deployed Feast services. + +_Appears in:_ +- [FeatureStoreSpec](#featurestorespec) + +| Field | Description | +| --- | --- | +| `kubernetes` _[KubernetesAuthz](#kubernetesauthz)_ | | +| `oidc` _[OidcAuthz](#oidcauthz)_ | | + + +#### DefaultConfigs + + + +DefaultConfigs k8s container settings that are applied by default + +_Appears in:_ +- [LocalRegistryConfig](#localregistryconfig) +- [OfflineStore](#offlinestore) +- [OnlineStore](#onlinestore) +- [ServiceConfigs](#serviceconfigs) + +| Field | Description | +| --- | --- | +| `image` _string_ | | + + +#### FeatureStore + + + +FeatureStore is the Schema for the featurestores API + + + +| Field | Description | +| --- | --- | +| `apiVersion` _string_ | `feast.dev/v1alpha1` +| `kind` _string_ | `FeatureStore` +| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | +| `spec` _[FeatureStoreSpec](#featurestorespec)_ | | +| `status` _[FeatureStoreStatus](#featurestorestatus)_ | | + + +#### FeatureStoreRef + + + +FeatureStoreRef defines which existing FeatureStore's registry should be used + +_Appears in:_ +- [RemoteRegistryConfig](#remoteregistryconfig) + +| Field | Description | +| --- | --- | +| `name` _string_ | Name of the FeatureStore | +| `namespace` _string_ | Namespace of the FeatureStore | + + +#### FeatureStoreServices + + + +FeatureStoreServices defines the desired feast services. An ephemeral registry is deployed by default. + +_Appears in:_ +- [FeatureStoreSpec](#featurestorespec) + +| Field | Description | +| --- | --- | +| `offlineStore` _[OfflineStore](#offlinestore)_ | | +| `onlineStore` _[OnlineStore](#onlinestore)_ | | +| `registry` _[Registry](#registry)_ | | + + +#### FeatureStoreSpec + + + +FeatureStoreSpec defines the desired state of FeatureStore + +_Appears in:_ +- [FeatureStore](#featurestore) +- [FeatureStoreStatus](#featurestorestatus) + +| Field | Description | +| --- | --- | +| `feastProject` _string_ | FeastProject is the Feast project id. This can be any alphanumeric string with underscores, but it cannot start with an underscore. Required. | +| `services` _[FeatureStoreServices](#featurestoreservices)_ | | +| `authz` _[AuthzConfig](#authzconfig)_ | | + + +#### FeatureStoreStatus + + + +FeatureStoreStatus defines the observed state of FeatureStore + +_Appears in:_ +- [FeatureStore](#featurestore) + +| Field | Description | +| --- | --- | +| `applied` _[FeatureStoreSpec](#featurestorespec)_ | Shows the currently applied feast configuration, including any pertinent defaults | +| `clientConfigMap` _string_ | ConfigMap in this namespace containing a client `feature_store.yaml` for this feast deployment | +| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#condition-v1-meta) array_ | | +| `feastVersion` _string_ | Version of feast that's currently deployed | +| `phase` _string_ | | +| `serviceHostnames` _[ServiceHostnames](#servicehostnames)_ | | + + +#### KubernetesAuthz + + + +KubernetesAuthz provides a way to define the authorization settings using Kubernetes RBAC resources. +https://kubernetes.io/docs/reference/access-authn-authz/rbac/ + +_Appears in:_ +- [AuthzConfig](#authzconfig) + +| Field | Description | +| --- | --- | +| `roles` _string array_ | The Kubernetes RBAC roles to be deployed in the same namespace of the FeatureStore. +Roles are managed by the operator and created with an empty list of rules. +See the Feast permission model at https://docs.feast.dev/getting-started/concepts/permission +The feature store admin is not obligated to manage roles using the Feast operator, roles can be managed independently. +This configuration option is only providing a way to automate this procedure. +Important note: the operator cannot ensure that these roles will match the ones used in the configured Feast permissions. | + + +#### LocalRegistryConfig + + + +LocalRegistryConfig configures the deployed registry service + +_Appears in:_ +- [Registry](#registry) + +| Field | Description | +| --- | --- | +| `image` _string_ | | +| `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core)_ | | +| `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#pullpolicy-v1-core)_ | | +| `resources` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | | +| `persistence` _[RegistryPersistence](#registrypersistence)_ | | +| `tls` _[TlsConfigs](#tlsconfigs)_ | | +| `logLevel` _string_ | LogLevel sets the logging level for the registry service +Allowed values: "debug", "info", "warning", "error", "critical". | + + +#### OfflineStore + + + +OfflineStore configures the deployed offline store service + +_Appears in:_ +- [FeatureStoreServices](#featurestoreservices) + +| Field | Description | +| --- | --- | +| `image` _string_ | | +| `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core)_ | | +| `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#pullpolicy-v1-core)_ | | +| `resources` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | | +| `persistence` _[OfflineStorePersistence](#offlinestorepersistence)_ | | +| `tls` _[TlsConfigs](#tlsconfigs)_ | | +| `logLevel` _string_ | LogLevel sets the logging level for the offline store service +Allowed values: "debug", "info", "warning", "error", "critical". | + + +#### OfflineStoreDBStorePersistence + + + +OfflineStoreDBStorePersistence configures the DB store persistence for the offline store service + +_Appears in:_ +- [OfflineStorePersistence](#offlinestorepersistence) + +| Field | Description | +| --- | --- | +| `type` _string_ | | +| `secretRef` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#localobjectreference-v1-core)_ | Data store parameters should be placed as-is from the "feature_store.yaml" under the secret key. "registry_type" & "type" fields should be removed. | +| `secretKeyName` _string_ | By default, the selected store "type" is used as the SecretKeyName | + + +#### OfflineStoreFilePersistence + + + +OfflineStoreFilePersistence configures the file-based persistence for the offline store service + +_Appears in:_ +- [OfflineStorePersistence](#offlinestorepersistence) + +| Field | Description | +| --- | --- | +| `type` _string_ | | +| `pvc` _[PvcConfig](#pvcconfig)_ | | + + +#### OfflineStorePersistence + + + +OfflineStorePersistence configures the persistence settings for the offline store service + +_Appears in:_ +- [OfflineStore](#offlinestore) + +| Field | Description | +| --- | --- | +| `file` _[OfflineStoreFilePersistence](#offlinestorefilepersistence)_ | | +| `store` _[OfflineStoreDBStorePersistence](#offlinestoredbstorepersistence)_ | | + + +#### OidcAuthz + + + +OidcAuthz defines the authorization settings for deployments using an Open ID Connect identity provider. +https://auth0.com/docs/authenticate/protocols/openid-connect-protocol + +_Appears in:_ +- [AuthzConfig](#authzconfig) + +| Field | Description | +| --- | --- | +| `secretRef` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#localobjectreference-v1-core)_ | | + + +#### OnlineStore + + + +OnlineStore configures the deployed online store service + +_Appears in:_ +- [FeatureStoreServices](#featurestoreservices) + +| Field | Description | +| --- | --- | +| `image` _string_ | | +| `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core)_ | | +| `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#pullpolicy-v1-core)_ | | +| `resources` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | | +| `persistence` _[OnlineStorePersistence](#onlinestorepersistence)_ | | +| `tls` _[TlsConfigs](#tlsconfigs)_ | | +| `logLevel` _string_ | LogLevel sets the logging level for the online store service +Allowed values: "debug", "info", "warning", "error", "critical". | + + +#### OnlineStoreDBStorePersistence + + + +OnlineStoreDBStorePersistence configures the DB store persistence for the offline store service + +_Appears in:_ +- [OnlineStorePersistence](#onlinestorepersistence) + +| Field | Description | +| --- | --- | +| `type` _string_ | | +| `secretRef` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#localobjectreference-v1-core)_ | Data store parameters should be placed as-is from the "feature_store.yaml" under the secret key. "registry_type" & "type" fields should be removed. | +| `secretKeyName` _string_ | By default, the selected store "type" is used as the SecretKeyName | + + +#### OnlineStoreFilePersistence + + + +OnlineStoreFilePersistence configures the file-based persistence for the offline store service + +_Appears in:_ +- [OnlineStorePersistence](#onlinestorepersistence) + +| Field | Description | +| --- | --- | +| `path` _string_ | | +| `pvc` _[PvcConfig](#pvcconfig)_ | | + + +#### OnlineStorePersistence + + + +OnlineStorePersistence configures the persistence settings for the online store service + +_Appears in:_ +- [OnlineStore](#onlinestore) + +| Field | Description | +| --- | --- | +| `file` _[OnlineStoreFilePersistence](#onlinestorefilepersistence)_ | | +| `store` _[OnlineStoreDBStorePersistence](#onlinestoredbstorepersistence)_ | | + + +#### OptionalConfigs + + + +OptionalConfigs k8s container settings that are optional + +_Appears in:_ +- [LocalRegistryConfig](#localregistryconfig) +- [OfflineStore](#offlinestore) +- [OnlineStore](#onlinestore) +- [ServiceConfigs](#serviceconfigs) + +| Field | Description | +| --- | --- | +| `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core)_ | | +| `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#pullpolicy-v1-core)_ | | +| `resources` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | | + + +#### PvcConfig + +_Underlying type:_ `[struct{Ref *k8s.io/api/core/v1.LocalObjectReference "json:\"ref,omitempty\""; Create *PvcCreate "json:\"create,omitempty\""; MountPath string "json:\"mountPath\""}](#struct{ref-*k8sioapicorev1localobjectreference-"json:\"ref,omitempty\"";-create-*pvccreate-"json:\"create,omitempty\"";-mountpath-string-"json:\"mountpath\""})` + +PvcConfig defines the settings for a persistent file store based on PVCs. +We can refer to an existing PVC using the `Ref` field, or create a new one using the `Create` field. + +_Appears in:_ +- [OfflineStoreFilePersistence](#offlinestorefilepersistence) +- [OnlineStoreFilePersistence](#onlinestorefilepersistence) + + + + + +#### Registry + + + +Registry configures the registry service. One selection is required. Local is the default setting. + +_Appears in:_ +- [FeatureStoreServices](#featurestoreservices) + +| Field | Description | +| --- | --- | +| `local` _[LocalRegistryConfig](#localregistryconfig)_ | | +| `remote` _[RemoteRegistryConfig](#remoteregistryconfig)_ | | + + +#### RegistryDBStorePersistence + +_Underlying type:_ `[struct{Type string "json:\"type\""; SecretRef k8s.io/api/core/v1.LocalObjectReference "json:\"secretRef\""; SecretKeyName string "json:\"secretKeyName,omitempty\""}](#struct{type-string-"json:\"type\"";-secretref-k8sioapicorev1localobjectreference-"json:\"secretref\"";-secretkeyname-string-"json:\"secretkeyname,omitempty\""})` + +RegistryDBStorePersistence configures the DB store persistence for the registry service + +_Appears in:_ +- [RegistryPersistence](#registrypersistence) + + + +#### RegistryFilePersistence + +_Underlying type:_ `[struct{Path string "json:\"path,omitempty\""; PvcConfig *PvcConfig "json:\"pvc,omitempty\""; S3AdditionalKwargs *map[string]string "json:\"s3_additional_kwargs,omitempty\""}](#struct{path-string-"json:\"path,omitempty\"";-pvcconfig-*pvcconfig-"json:\"pvc,omitempty\"";-s3additionalkwargs-*map[string]string-"json:\"s3_additional_kwargs,omitempty\""})` + +RegistryFilePersistence configures the file-based persistence for the registry service + +_Appears in:_ +- [RegistryPersistence](#registrypersistence) + + + +#### RegistryPersistence + + + +RegistryPersistence configures the persistence settings for the registry service + +_Appears in:_ +- [LocalRegistryConfig](#localregistryconfig) + +| Field | Description | +| --- | --- | +| `file` _[RegistryFilePersistence](#registryfilepersistence)_ | | +| `store` _[RegistryDBStorePersistence](#registrydbstorepersistence)_ | | + + +#### RemoteRegistryConfig + + + +RemoteRegistryConfig points to a remote feast registry server. When set, the operator will not deploy a registry for this FeatureStore CR. +Instead, this FeatureStore CR's online/offline services will use a remote registry. One selection is required. + +_Appears in:_ +- [Registry](#registry) + +| Field | Description | +| --- | --- | +| `hostname` _string_ | Host address of the remote registry service - :, e.g. `registry..svc.cluster.local:80` | +| `feastRef` _[FeatureStoreRef](#featurestoreref)_ | Reference to an existing `FeatureStore` CR in the same k8s cluster. | +| `tls` _[TlsRemoteRegistryConfigs](#tlsremoteregistryconfigs)_ | | + + +#### SecretKeyNames + + + +SecretKeyNames defines the secret key names for the TLS key and cert. + +_Appears in:_ +- [TlsConfigs](#tlsconfigs) + +| Field | Description | +| --- | --- | +| `tlsCrt` _string_ | defaults to "tls.crt" | +| `tlsKey` _string_ | defaults to "tls.key" | + + +#### ServiceConfigs + + + +ServiceConfigs k8s container settings + +_Appears in:_ +- [LocalRegistryConfig](#localregistryconfig) +- [OfflineStore](#offlinestore) +- [OnlineStore](#onlinestore) + +| Field | Description | +| --- | --- | +| `image` _string_ | | +| `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core)_ | | +| `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#pullpolicy-v1-core)_ | | +| `resources` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | | + + +#### ServiceHostnames + + + +ServiceHostnames defines the service hostnames in the format of :, e.g. example.svc.cluster.local:80 + +_Appears in:_ +- [FeatureStoreStatus](#featurestorestatus) + +| Field | Description | +| --- | --- | +| `offlineStore` _string_ | | +| `onlineStore` _string_ | | +| `registry` _string_ | | + + +#### TlsConfigs + + + +TlsConfigs configures server TLS for a feast service. in an openshift cluster, this is configured by default using service serving certificates. + +_Appears in:_ +- [LocalRegistryConfig](#localregistryconfig) +- [OfflineStore](#offlinestore) +- [OnlineStore](#onlinestore) + +| Field | Description | +| --- | --- | +| `secretRef` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#localobjectreference-v1-core)_ | references the local k8s secret where the TLS key and cert reside | +| `secretKeyNames` _[SecretKeyNames](#secretkeynames)_ | | +| `disable` _boolean_ | will disable TLS for the feast service. useful in an openshift cluster, for example, where TLS is configured by default | + + +#### TlsRemoteRegistryConfigs + + + +TlsRemoteRegistryConfigs configures client TLS for a remote feast registry. in an openshift cluster, this is configured by default when the remote feast registry is using service serving certificates. + +_Appears in:_ +- [RemoteRegistryConfig](#remoteregistryconfig) + +| Field | Description | +| --- | --- | +| `configMapRef` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#localobjectreference-v1-core)_ | references the local k8s configmap where the TLS cert resides | +| `certName` _string_ | defines the configmap key name for the client TLS cert. | + + diff --git a/infra/feast-operator/docs/crd-ref-templates/config.yaml b/infra/feast-operator/docs/crd-ref-templates/config.yaml new file mode 100644 index 0000000000..42d10e08b0 --- /dev/null +++ b/infra/feast-operator/docs/crd-ref-templates/config.yaml @@ -0,0 +1,8 @@ +processor: + ignoreTypes: + - "(FeatureStore)List$" + ignoreFields: + - "TypeMeta$" + +render: + kubernetesVersion: "1.30" \ No newline at end of file diff --git a/infra/feast-operator/docs/crd-ref-templates/markdown/gv_details.tpl b/infra/feast-operator/docs/crd-ref-templates/markdown/gv_details.tpl new file mode 100644 index 0000000000..30ad0d7518 --- /dev/null +++ b/infra/feast-operator/docs/crd-ref-templates/markdown/gv_details.tpl @@ -0,0 +1,19 @@ +{{- define "gvDetails" -}} +{{- $gv := . -}} + +## {{ $gv.GroupVersionString }} + +{{ $gv.Doc }} + +{{- if $gv.Kinds }} +### Resource Types +{{- range $gv.SortedKinds }} +- {{ $gv.TypeForKind . | markdownRenderTypeLink }} +{{- end }} +{{ end }} + +{{ range $gv.SortedTypes }} +{{ template "type" . }} +{{ end }} + +{{- end -}} diff --git a/infra/feast-operator/docs/crd-ref-templates/markdown/gv_list.tpl b/infra/feast-operator/docs/crd-ref-templates/markdown/gv_list.tpl new file mode 100644 index 0000000000..a4d3dadf18 --- /dev/null +++ b/infra/feast-operator/docs/crd-ref-templates/markdown/gv_list.tpl @@ -0,0 +1,15 @@ +{{- define "gvList" -}} +{{- $groupVersions := . -}} + +# API Reference + +## Packages +{{- range $groupVersions }} +- {{ markdownRenderGVLink . }} +{{- end }} + +{{ range $groupVersions }} +{{ template "gvDetails" . }} +{{ end }} + +{{- end -}} diff --git a/infra/feast-operator/docs/crd-ref-templates/markdown/type.tpl b/infra/feast-operator/docs/crd-ref-templates/markdown/type.tpl new file mode 100644 index 0000000000..c0ac2e0353 --- /dev/null +++ b/infra/feast-operator/docs/crd-ref-templates/markdown/type.tpl @@ -0,0 +1,33 @@ +{{- define "type" -}} +{{- $type := . -}} +{{- if markdownShouldRenderType $type -}} + +#### {{ $type.Name }} + +{{ if $type.IsAlias }}_Underlying type:_ `{{ markdownRenderTypeLink $type.UnderlyingType }}`{{ end }} + +{{ $type.Doc }} + +{{ if $type.References -}} +_Appears in:_ +{{- range $type.SortedReferences }} +- {{ markdownRenderTypeLink . }} +{{- end }} +{{- end }} + +{{ if $type.Members -}} +| Field | Description | +| --- | --- | +{{ if $type.GVK -}} +| `apiVersion` _string_ | `{{ $type.GVK.Group }}/{{ $type.GVK.Version }}` +| `kind` _string_ | `{{ $type.GVK.Kind }}` +{{ end -}} + +{{ range $type.Members -}} +| `{{ .Name }}` _{{ markdownRenderType .Type }}_ | {{ template "type_members" . }} | +{{ end -}} + +{{ end -}} + +{{- end -}} +{{- end -}} diff --git a/infra/feast-operator/docs/crd-ref-templates/markdown/type_members.tpl b/infra/feast-operator/docs/crd-ref-templates/markdown/type_members.tpl new file mode 100644 index 0000000000..182fa18216 --- /dev/null +++ b/infra/feast-operator/docs/crd-ref-templates/markdown/type_members.tpl @@ -0,0 +1,8 @@ +{{- define "type_members" -}} +{{- $field := . -}} +{{- if eq $field.Name "metadata" -}} +Refer to Kubernetes API documentation for fields of `metadata`. +{{- else -}} +{{ $field.Doc }} +{{- end -}} +{{- end -}}