Is there a reason to not use the "kubernetes" tfstate backend?

[docteurklein]

Hi,
I love the idea behind a terraform CRD (I'm watching this space and just discovered your project).

I'm wondering if there is a reason for not chosing https://www.terraform.io/docs/backends/types/kubernetes.html as a default backend? Is this because of the secret-size limit (1MB) ?

Thanks!

[isaaguilar]

I believe that the kubernetes backend is only available with Terraform >= 0.13.0. This project supports >= 0.11.9 (or something like that) so it would make the default backend incompatible with earlier versions.

There is a customBackend feature to allow users to specify their own backends. I just tried, however, and it looks like if "kubernetes" backend is selected, the role for the tf-runner needs to be updated to this:

# rbac role rules for the tf-runner pod
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets # needs to be added
  verbs:
  - '*'
# and leases needs to be added
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - '*'

limitations

I could see some limitations in the kubernetes backend since it requires Secrets permissions to access tfstate. Hypothetically, If state data needs to be read from another namespace for a certain module, there would be an issue getting access to it.

solution?

I think I might make some changes to read for backend "kubernetes" to add RBAC permissions to the tf-runner. That would allow someone to use "kubernetes" backend by adding this to the terraform spec:

  customBackend: |-
    terraform {
      backend "kubernetes" {
        secret_suffix    = "state"
        in_cluster_config  = true
      }
    }

[asaintsever]

Hello @isaaguilar and kudos for this nice project!

This project supports >= 0.11.9 (or something like that)

This is an information I failed to find. I must be missing something here since it is possible to specify the Terraform version in the CRD (https://github.com/isaaguilar/terraform-operator/blob/master/deploy/crds/tf.isaaguilar.com_terraforms_crd.yaml#L333).

Edit: ok, just found out you are building tf runners by iterating on available HashiCorp's Terraform image tags (https://github.com/isaaguilar/terraform-operator/blob/master/docker/terraform/build.sh#L11)

[jstrachan]

@asaintsever FWIW we're migrating our BDD terraform testing on the Jenkins X project over to using this most excellent operator - totally loving it so far.

We're currently using the kubernetes back end - as it was the simplest thing to do to start with. The only thing to be careful of is make sure the backend namespace is the same as the namespace you are creating the Terraform resources inside otherwise I hit RBAC issues.

e.g. here's our templated Terraform we're using: https://github.com/jstrachan/jxr-versions/blob/tf-operator/.lighthouse/jenkins-x/bdd/terraform.yaml#L37-L44

Though the main downside is k8s clusters ideally would be cattle not pets; so using (say) a bucket is easier to backup + survives your k8s cluster being recreated (e.g. if you change region or something)

[asaintsever]

@jstrachan interesting to know you are more than giving a try to this operator. I am also on the verge of starting a PoC to assess it. Thanks for your template. I will likely also make use of the k8s backend as a starting point but will quickly change to S3.