The shared kafka cluster requires mTLS for authentication and authorization.
Use the tls-app module for granting access for the apps to the cluster.
We have a custom tflint plugin that help enforce best practices and avoid some
common issues. This is run through pre-commit, see the contributing
docs for installation details.
See the plugin repo's README for a list of the rules.
Please also consider the number of partitions for a topic. In Amazon MSK the total number of partitions influences the cost. See RFC
When migrating to the MSK cluster, the number of partitions in MSK needs to be the same as in the source cluster.
Each child folder contains a Terraform module for a team with the shared kafka resources for that team.
For each of these modules there is a terraform applier entry under:
Create a folder with your team name under the current folder and copy everything, with keeping the symbolic links, from the folder _template_team_x.
Then replace in every file _template_team_x with your team name
Example commands:
export TEAM=myteam
mkdir -p "${TEAM}"
cp -a _template_team_x/* "${TEAM}"
for f in "${TEAM}"/*; do sed -i '' "s/_template_team_x/${TEAM}/g" $f; doneIn the kubernetes-manifests repository, under dev or prod
copy the file tf-module-_template_team_x.yaml into one with your team name and then replace team_x with your team name in the file.
Example commands:
export TEAM=myteam
cat tf-module-_template_team_x.yaml| sed "s/team_x/${TEAM}/g" > tf-module-${TEAM}.yaml After the PR with the new file will be merged in the main branch, the terraform-applier will pick up this new CRD and an entry for your team will appear in the UI in dev or prod.
Please note that this module is created in plan-only mode, meaning it will only issue the terraform plan command without applying.
After you check that the configuration for the module is ok and the plan is as expected, you can enable the apply mode by setting in this file planOnly: false.
See instructions