Skip to content

Commit

Permalink
Merge Cloud Integrations and create a Vertex AI Example (#693)
Browse files Browse the repository at this point in the history 
* Initial merge of cloud integrations

* Small changes to integration.md

* Merged develop into this branch

* Fixed all imports

* Vertex AI example working with python run.py

* Fixed spelling mistakes

* azure secrets manager flavor renamed

* Reformatted

* Adjusted README

* Added init.pys and shortened import paths

* Reformatted

* Adjusted links and mentions in the docs

* Apply suggestions from code review

Co-authored-by: Barış Can Durak <[email protected]>
Co-authored-by: Michael Schuster <[email protected]>

* Removed backtics and added zenml pull

* No more docs in docs

* Apply suggestions from code review

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Reverted changes to secrets managers flavor names

* Apply suggestions from code review

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Fixed `almost` all linting issues

* Added PWD

* Fixed indentation errors

* Small reformat to trigger action rerun

* Moved tests into gcp folder

* Reformatted

* Add import ignore to pass linting

Co-authored-by: Barış Can Durak <[email protected]>
Co-authored-by: Michael Schuster <[email protected]>
Co-authored-by: Alex Strick van Linschoten <[email protected]>
  • Loading branch information
@AlexejPenner @bcdurak
@schustmi @strickvl
4 people authored Jun 13, 2022
1 parent f4b6e7d commit 68d5b5d
Show file tree
Hide file tree
Showing 50 changed files with 632 additions and 401 deletions.
8 changes: 8 additions & 0 deletions .pyspelling-ignore-words
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ CamelCase
ChatOps
ClassVar
ClickException
CloudSQL
Colab
ComponentWrapper
ComputeTarget
Expand All @@ -75,6 +76,7 @@ ConnectionError
ContainerOp
ContainerOps
Contextmanager
cryptoKeys
Cron
CustomFormatter
DAGs
Expand Down Expand Up @@ -108,6 +110,7 @@ Evidently's
EvidentlyProfileConfig
EvidentlyProfileStep
EvidentlyVisualizer
eu
ExampleSecretSchema
ExamplesRepo
ExecutionStatus
Expand Down Expand Up @@ -177,6 +180,8 @@ KMS
KServe
Keras
KeyError
keyRings
kr
Kubeflow
KubeflowMetadataStore
KubeflowOrchestrator
Expand Down Expand Up @@ -734,6 +739,7 @@ numericaltargetdrift
numpy
odbc
onerror
orch
orchestrator
orchestrators
os
Expand Down Expand Up @@ -764,6 +770,7 @@ probabilisticmodelperformance
profiler
proto
protobuf
PWD
py
pydantic
pyenv
Expand Down Expand Up @@ -805,6 +812,7 @@ seldon
sepc
serializable
serverless
serviceacc
serviceAccountName
setFormatter
setattr
Expand Down
2 changes: 1 addition & 1 deletion docs/book/advanced-guide/execute-pipelines-in-cloud.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ new stack with these components that you have just created.
zenml container-registry register cloud_registry --flavor=<aws/gcp/azure> --uri=$PATH_TO_YOUR_CONTAINER_REGISTRY
zenml orchestrator register cloud_orchestrator --flavor=kubeflow --custom_docker_base_image_name=YOUR_IMAGE
zenml artifact-store register cloud_artifact_store --flavor=<s3/gcp/azure> --path=$PATH_TO_YOUR_BUCKET
zenml secrets-manager register cloud_secrets_manager --flavor=<aws/gcp-secrets-manager>
zenml secrets-manager register cloud_secrets_manager --flavor=<aws/gcp/azure>
# Register the cloud stack
zenml stack register cloud_kubeflow_stack -m cloud_metadata_store -a cloud_artifact_store -o cloud_orchestrator -c cloud_registry -x cloud_secrets_manager
Expand Down
19 changes: 11 additions & 8 deletions docs/book/advanced-guide/integrations.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,21 @@ These are the third-party integrations that ZenML currently supports:
|---------------------------|--------|------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Apache Airflow | ✅ | Orchestrator | Works for local environment. | [airflow_orchestration](https://github.com/zenml-io/zenml/tree/main/examples/airflow_orchestration) |
| Apache Beam | ✅ | Distributed Processing | | |
| AWS | ✅ | Secrets Manager | Use AWS as a secrets manager. | |
| Azure | ✅ | Cloud | Use Azure Blob Storage buckets as ZenML artifact stores and AzureML as a ZenML step operator. | [azureml_step_operator](https://github.com/zenml-io/zenml/tree/main/examples/step_operator_remote_training) |
| BentoML | ⛏ | Cloud | Looking for community implementors. | |
| AWS | ✅ | Container Registry | Use the AWS container registry to store your containers. | |
| AWS | ✅ | Secrets Manager | Use AWS as a secrets manager. | [cloud_secrets_manager](https://github.com/zenml-io/zenml/tree/develop/examples/cloud_secrets_manager) |
| AWS | ✅ | Step Operator | Sagemaker as a ZenML step operator. | [sagemaker_step_operator](https://github.com/zenml-io/zenml/tree/main/examples/step_operator_remote_training) |
| Azure | ✅ | Artifact Store | Use Azure Blob Storage buckets as ZenML artifact stores. | |
| Azure | ✅ | Step Operator | Use AzureML as a step operator to supercharge specific steps. | [azureml_step_operator](https://github.com/zenml-io/zenml/tree/main/examples/step_operator_remote_training) |
| BentoML | ⛏ | Deployment | Looking for community implementors. | |
| Dash | ✅ | Visualizer | For Pipeline and PipelineRun visualization objects. | [lineage](https://github.com/zenml-io/zenml/tree/main/examples/lineage) |
| Evidently | ✅ | Monitoring | Allows for visualization of drift as well as export of a `Profile` object. | [drift_detection](https://github.com/zenml-io/zenml/tree/release/0.5.7/examples/drift_detection) |
| Facets | ✅ | Visualizer | Quickly visualize your datasets using `facets`. | [statistics](https://github.com/zenml-io/zenml/tree/main/examples/statistics) |
| Feast | ✅ | Feature Store | Use Feast with Redis for your online features. | [feature_store](https://github.com/zenml-io/zenml/tree/main/examples/feature_store) |
| GCP | ✅ | Cloud | Use GCS buckets as a ZenML artifact store. | |
| GitHub | ✅ | Orchestrator | Use GitHub Actions to orchestrate your ZenML pipelines. | [github_actions_orchestration](https://github.com/zenml-io/zenml/tree/main/examples/github_actions_orchestration) |
| GCP | ✅ | Artifact Store | Use GCS buckets as a ZenML artifact store. | |
| GCP | ✅ | Step Secrets Manager | Use the GCP Secret Manager. | [cloud_secrets_manager](https://github.com/zenml-io/zenml/tree/develop/examples/cloud_secrets_manager) |
| GCP | ✅ | Step Operator | Vertex AI as a ZenML step operator. | [vertex_step_operator](https://github.com/zenml-io/zenml/tree/main/examples/step_operator_remote_training) |
| GCP | ✅ | Orchestrator | Execute your ZenML pipelines using Vertex AI Pipelines. | |
| Graphviz | ✅ | Visualizer | For Pipeline and PipelineRun visualization objects. | [dag_visualizer](https://github.com/zenml-io/zenml/tree/main/examples/dag_visualizer) |
| Great Expectations | ⛏ | Data Validation | Looking for community implementors. | |
| Hugging Face | ✅ | Materializer | Use Hugging Face tokenizers, datasets and models. | [huggingface](https://github.com/zenml-io/zenml/tree/main/examples/huggingface) |
Expand All @@ -58,16 +64,13 @@ These are the third-party integrations that ZenML currently supports:
| Plotly | ✅ | Visualizer | For Pipeline and PipelineRun visualization objects. | [lineage](https://github.com/zenml-io/zenml/tree/main/examples/lineage) |
| PyTorch | ✅ | Training | | [pytorch](https://github.com/zenml-io/zenml/tree/main/examples/pytorch) |
| PyTorch Lightning | ✅ | Training | | |
| Vertex | ✅ | Step Operator | Vertex AI as a ZenML step operator. | [vertex_step_operator](https://github.com/zenml-io/zenml/tree/main/examples/step_operator_remote_training) |
| Vertex | ✅ | Orchestrator | Execute your ZenML pipelines using Vertex AI Pipelines |
| S3 | ✅ | Artifact Store | Use S3 buckets as ZenML artifact stores. | [caching chapter](https://docs.zenml.io/v/docs/guides/functional-api/caching) |
| Sagemaker | ✅ | Cloud | Sagemaker as a ZenML step operator. | [sagemaker_step_operator](https://github.com/zenml-io/zenml/tree/main/examples/step_operator_remote_training) |
| scikit-learn | ✅ | Training | | [caching chapter](https://docs.zenml.io/v/docs/guides/functional-api/caching) |
| scipy | ✅ | Materializer | Use sparse matrices. | |
| Seldon Core | ✅ | Deployment | Seldon Core as a model deployer. | [seldon](https://github.com/zenml-io/zenml/tree/main/examples/seldon_deployment) |
| Tensorflow | ✅ | Training, Visualizer | TensorBoard support. | [quickstart](https://github.com/zenml-io/zenml/tree/main/examples/quickstart). [kubeflow](https://github.com/zenml-io/zenml/tree/main/examples/kubeflow) |
| Weights & Biases Tracking | ✅ | Experiment Tracking | Tracks your pipelines and your training runs. | [wandb_tracking](https://github.com/zenml-io/zenml/tree/main/examples/wandb_tracking) |
| whylogs | ✅ | logging | Integration fully implemented for data logging. | [whylogs](https://github.com/zenml-io/zenml/tree/main/examples/whylogs) |
| whylogs | ✅ | Logging | Integration fully implemented for data logging. | [whylogs](https://github.com/zenml-io/zenml/tree/main/examples/whylogs) |
| xgboost | ✅ | Training | Support for `Booster` and `DMatrix` materialization. | [xgboost](https://github.com/zenml-io/zenml/tree/main/examples/xgboost) |

✅ means the integration is already implemented.
Expand Down
2 changes: 1 addition & 1 deletion docs/book/advanced-guide/manage-your-secrets.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ then register your secrets manager in the following way:

```shell
zenml integration install azure
zenml secrets-manager register AZURE_SECRETS_MANAGER_NAME -f azure_secrets_manager \
zenml secrets-manager register AZURE_SECRETS_MANAGER_NAME -f azure \
--key_vault_name=<YOUR_KEY_VAULT_NAME>

# update your default stack, for example
Expand Down
12 changes: 7 additions & 5 deletions docs/book/extending-zenml/orchestrators.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,13 @@ Moreover, additional orchestrators can be found in specific `integrations`
modules, such as the `AirflowOrchestrator` in the `airflow` integration and the
`KubeflowOrchestrator` in the `kubeflow` integration.

| | Flavor | Integration |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|-------------|
| [LocalOrchestrator](https://apidocs.zenml.io/latest/api_docs/orchestrators/#zenml.orchestrators.local.local_orchestrator.LocalOrchestrator) | local | `built-in` |
| [AirflowOrchestrator](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.airflow.orchestrators.airflow_orchestrator.AirflowOrchestrator) | airflow | airflow |
| [KubeflowOrchestrator](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.kubeflow.orchestrators.kubeflow_orchestrator.KubeflowOrchestrator) | kubeflow | kubeflow |
| | Flavor | Integration |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------------|
| [LocalOrchestrator](https://apidocs.zenml.io/latest/api_docs/orchestrators/#zenml.orchestrators.local.local_orchestrator.LocalOrchestrator) | local | `built-in` |
| [AirflowOrchestrator](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.airflow.orchestrators.airflow_orchestrator.AirflowOrchestrator) | airflow | airflow |
| [KubeflowOrchestrator](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.kubeflow.orchestrators.kubeflow_orchestrator.KubeflowOrchestrator) | kubeflow | kubeflow |
| [VertexAIOrchestrator](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.gcp.orchestrators.vertex_orchestrator.VertexOrchestrator) | vertex | gcp |


If you would like to see the available flavors for artifact stores, you can
use the command:
Expand Down
17 changes: 9 additions & 8 deletions docs/book/extending-zenml/secrets-managers.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,15 @@ not intended for production use.

For production use cases some more flavors can be found in specific
`integrations` modules, such as the `GCPSecretsManager` in the
`gcp_secrets_manager` integration and the `AWSSecretsManager` in the
`aws` integration.

| | Flavor | Integration |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------|---------------------|
| [LocalSecretsManager](https://apidocs.zenml.io/latest/api_docs/secrets_managers/#zenml.secrets_managers.local.local_secrets_manager.LocalSecretsManager) | local | `built-in` |
| [AWSSecretsManager](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.aws.secrets_managers.aws_secrets_manager.AWSSecretsManager) | aws | aws |
| GCPSecretsManager | gcp_secrets_manager | gcp_secrets_manager |
`gcp` integration, the `AWSSecretsManager` in the
`aws` integration and the `AzureSecretsManager` in the `azure` integration.

| | Flavor | Integration |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|-------------|
| [LocalSecretsManager](https://apidocs.zenml.io/latest/api_docs/secrets_managers/#zenml.secrets_managers.local.local_secrets_manager.LocalSecretsManager) | local | `built-in` |
| [AWSSecretsManager](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.aws.secrets_managers.aws_secrets_manager.AWSSecretsManager) | aws | aws |
| [GCPSecretsManager](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.gcp.secrets_managers.gcp_secrets_manager.GCPSecretsManager) | gcp | gcp |
| [AzureSecretsManager](https://apidocs.zenml.io/latest/api_docs/integrations/#zenml.integrations.azure.secrets_managers.azure_secrets_manager.AzureSecretsManager) | azure | azure |

If you would like to see the available flavors for secret managers, you can
use the command:
Expand Down
Loading

0 comments on commit 68d5b5d

Please sign in to comment.