Vault as an External Secret Store

This guide walks through the steps required to configure Crossplane and its Providers to use Vault as an External Secret Store (ESS) with ESS Plugin Vault.

Warning

External Secret Stores are an alpha feature.

They’re not recommended for production use. Crossplane disables External Secret Stores by default.

Crossplane uses sensitive information including Provider credentials, inputs to managed resources and connection details.

The Vault credential injection guide details using Vault and Crossplane for Provider credentials.

Crossplane doesn’t support for using Vault for managed resources input. Crossplane issue #2985 tracks support for this feature.

Supporting connection details with Vault requires a Crossplane external secret store.

Prerequisites

This guide requires Helm version 3.11 or later.

Install Vault

Note

Detailed instructions on installing Vault are available from the Vault documentation.

Add the Vault Helm chart

Add the Helm repository for hashicorp.

1helm repo add hashicorp https://helm.releases.hashicorp.com --force-update

Install Vault using Helm.

1helm -n vault-system upgrade --install vault hashicorp/vault --create-namespace

Unseal Vault

If Vault is sealed unseal Vault using the unseal keys.

Get the Vault keys.

1kubectl -n vault-system exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json
2VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")

Unseal the vault using the keys.

 1kubectl -n vault-system exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
 2Key             Value
 3---             -----
 4Seal Type       shamir
 5Initialized     true
 6Sealed          false
 7Total Shares    1
 8Threshold       1
 9Version         1.13.1
10Build Date      2023-03-23T12:51:35Z
11Storage Type    file
12Cluster Name    vault-cluster-df884357
13Cluster ID      b3145d26-2c1a-a7f2-a364-81753033c0d9
14HA Enabled      false

Configure Vault Kubernetes authentication

Enable the Kubernetes auth method for Vault to authenticate requests based on Kubernetes service accounts.

Get the Vault root token

The Vault root token is inside the JSON file created when unsealing Vault.

1cat cluster-keys.json | jq -r ".root_token"

Enable Kubernetes authentication

Connect to a shell in the Vault pod.

1kubectl -n vault-system exec -it vault-0 -- /bin/sh
2/ $

From the Vault shell, login to Vault using the root token.

 1vault login # use the root token from above
 2Token (will be hidden):
 3Success! You are now authenticated. The token information displayed below
 4is already stored in the token helper. You do NOT need to run "vault login"
 5again. Future Vault requests will automatically use this token.
 6
 7Key                  Value
 8---                  -----
 9token                hvs.TSN4SssfMBM0HAtwGrxgARgn
10token_accessor       qodxHrINVlRXKyrGeeDkxnih
11token_duration       ∞
12token_renewable      false
13token_policies       ["root"]
14identity_policies    []
15policies             ["root"]

Enable the Kubernetes authentication method in Vault.

1vault auth enable kubernetes
2Success! Enabled kubernetes auth method at: kubernetes/

Configure Vault to communicate with Kubernetes and exit the Vault shell

1vault write auth/kubernetes/config \
2        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
3        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
4        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
5Success! Data written to: auth/kubernetes/config
6/ $ exit

Configure Vault for Crossplane integration

Crossplane relies on the Vault key-value secrets engine to store information and Vault requires a permissions policy for the Crossplane service account.

Enable the Vault kv secrets engine

Enable the Vault KV Secrets Engine.

Important

Vault has two versions of the KV Secrets Engine. This example uses version 2.

1kubectl -n vault-system exec -it vault-0 -- vault secrets enable -path=secret kv-v2
2Success! Enabled the kv-v2 secrets engine at: secret/

Create a Vault policy for Crossplane

Create the Vault policy to allow Crossplane to read and write data from Vault.

1kubectl -n vault-system exec -i vault-0 -- vault policy write crossplane - <<EOF
2path "secret/data/*" {
3    capabilities = ["create", "read", "update", "delete"]
4}
5path "secret/metadata/*" {
6    capabilities = ["create", "read", "update", "delete"]
7}
8EOF
9Success! Uploaded policy: crossplane

Apply the policy to Vault.

1kubectl -n vault-system exec -it vault-0 -- vault write auth/kubernetes/role/crossplane \
2    bound_service_account_names="*" \
3    bound_service_account_namespaces=crossplane-system \
4    policies=crossplane \
5    ttl=24h
6Success! Data written to: auth/kubernetes/role/crossplane

Install Crossplane

Important

Crossplane v1.12 introduced the plugin support. Make sure your version of Crossplane supports plugins.

Install the Crossplane with the External Secrets Stores feature enabled.

1helm upgrade --install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace --set args='{--enable-external-secret-stores}'

Install the Crossplane Vault plugin

The Crossplane Vault plugin isn’t part of the default Crossplane install. The plugin installs as a unique Pod that uses the Vault Agent Sidecar Injection to connect the Vault secret store to Crossplane.

First, configure annotations for the Vault plugin pod.

1cat > values.yaml <<EOF
2podAnnotations:
3  vault.hashicorp.com/agent-inject: "true"
4  vault.hashicorp.com/agent-inject-token: "true"
5  vault.hashicorp.com/role: crossplane
6  vault.hashicorp.com/agent-run-as-user: "65532"
7EOF

Next, install the Crossplane ESS Plugin pod to the crossplane-system namespace and apply the Vault annotations.

1helm upgrade --install ess-plugin-vault oci://xpkg.upbound.io/crossplane-contrib/ess-plugin-vault --namespace crossplane-system -f values.yaml

Configure Crossplane

Using the Vault plugin requires configuration to connect to the Vault service. The plugin also requires Providers to enable external secret stores.

With the plugin and providers configured, Crossplane requires two StoreConfig objects to describe how Crossplane and the Providers communicate with vault.

Enable external secret stores in the Provider

Note

This example uses Provider GCP, but the ControllerConfig is the same for all Providers.

Create a ControllerConfig object to enable external secret stores.

1echo "apiVersion: pkg.crossplane.io/v1alpha1
2kind: ControllerConfig
3metadata:
4  name: vault-config
5spec:
6  args:
7    - --enable-external-secret-stores" | kubectl apply -f -

Install the Provider and apply the ControllerConfig.

1echo "apiVersion: pkg.crossplane.io/v1
2kind: Provider
3metadata:
4  name: provider-gcp
5spec:
6  package: xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.23.0-rc.0.19.ge9b75ee5
7  controllerConfigRef:
8    name: vault-config" | kubectl apply -f -

Connect the Crossplane plugin to Vault

Create a VaultConfig resource for the plugin to connect to the Vault service:

 1echo "apiVersion: secrets.crossplane.io/v1alpha1
 2kind: VaultConfig
 3metadata:
 4  name: vault-internal
 5spec:
 6  server: http://vault.vault-system:8200
 7  mountPath: secret/
 8  version: v2
 9  auth:
10    method: Token
11    token:
12      source: Filesystem
13      fs:
14        path: /vault/secrets/token" | kubectl apply -f -

Create a Crossplane StoreConfig

Create a StoreConfig object from the secrets.crossplane.io group. Crossplane uses the StoreConfig to connect to the Vault plugin service.

The configRef connects the StoreConfig to the specific Vault plugin configuration.

 1echo "apiVersion: secrets.crossplane.io/v1alpha1
 2kind: StoreConfig
 3metadata:
 4  name: vault
 5spec:
 6  type: Plugin
 7  defaultScope: crossplane-system
 8  plugin:
 9    endpoint: ess-plugin-vault.crossplane-system:4040
10    configRef:
11      apiVersion: secrets.crossplane.io/v1alpha1
12      kind: VaultConfig
13      name: vault-internal" | kubectl apply -f -

Create a Provider StoreConfig

Create a StoreConfig object from the Provider’s API group, gcp.crossplane.io. The Provider uses this StoreConfig to communicate with Vault for Managed Resources.

The configRef connects the StoreConfig to the specific Vault plugin configuration.

 1echo "apiVersion: gcp.crossplane.io/v1alpha1
 2kind: StoreConfig
 3metadata:
 4  name: vault
 5spec:
 6  type: Plugin
 7  defaultScope: crossplane-system
 8  plugin:
 9    endpoint: ess-plugin-vault.crossplane-system:4040
10    configRef:
11      apiVersion: secrets.crossplane.io/v1alpha1
12      kind: VaultConfig
13      name: vault-internal" | kubectl apply -f -

Create Provider resources

Check that Crossplane installed the Provider and the Provider is healthy.

1kubectl get providers
2NAME           INSTALLED   HEALTHY   PACKAGE                                                                     AGE
3provider-gcp   True        True      xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.23.0-rc.0.19.ge9b75ee5   10m

Create a CompositeResourceDefinition

Create a CompositeResourceDefinition to define a custom API endpoint.

 1echo "apiVersion: apiextensions.crossplane.io/v1
 2kind: CompositeResourceDefinition
 3metadata:
 4  name: compositeessinstances.ess.example.org
 5  annotations:
 6    feature: ess
 7spec:
 8  group: ess.example.org
 9  names:
10    kind: CompositeESSInstance
11    plural: compositeessinstances
12  claimNames:
13    kind: ESSInstance
14    plural: essinstances
15  connectionSecretKeys:
16    - publicKey
17    - publicKeyType
18  versions:
19  - name: v1alpha1
20    served: true
21    referenceable: true
22    schema:
23      openAPIV3Schema:
24        type: object
25        properties:
26          spec:
27            type: object
28            properties:
29              parameters:
30                type: object
31                properties:
32                  serviceAccount:
33                    type: string
34                required:
35                  - serviceAccount
36            required:
37              - parameters" | kubectl apply -f -

Create a Composition

Create a Composition to create a Service Account and Service Account Key inside GCP.

Creating a Service Account Key generates connectionDetails that the Provider stores in Vault using the publishConnectionDetailsTo details.

 1echo "apiVersion: apiextensions.crossplane.io/v1
 2kind: Composition
 3metadata:
 4  name: essinstances.ess.example.org
 5  labels:
 6    feature: ess
 7spec:
 8  publishConnectionDetailsWithStoreConfigRef: 
 9    name: vault
10  compositeTypeRef:
11    apiVersion: ess.example.org/v1alpha1
12    kind: CompositeESSInstance
13  mode: Pipeline
14  pipeline:
15  - step: patch-and-transform
16    functionRef:
17      name: function-patch-and-transform
18    input:
19      apiVersion: pt.fn.crossplane.io/v1beta1
20      kind: Resources
21      resources:
22        - name: serviceaccount
23          base:
24            apiVersion: iam.gcp.crossplane.io/v1alpha1
25            kind: ServiceAccount
26            metadata:
27              name: ess-test-sa
28            spec:
29              forProvider:
30                displayName: a service account to test ess
31        - name: serviceaccountkey
32          base:
33            apiVersion: iam.gcp.crossplane.io/v1alpha1
34            kind: ServiceAccountKey
35            spec:
36              forProvider:
37                serviceAccountSelector:
38                  matchControllerRef: true
39              publishConnectionDetailsTo:
40                name: ess-mr-conn
41                metadata:
42                  labels:
43                    environment: development
44                    team: backend
45                configRef:
46                  name: vault
47          connectionDetails:
48            - name: publicKey
49              type: FromConnectionSecretKey
50              fromConnectionSecretKey: publicKey
51            - name: publicKey
52              type: FromConnectionSecretKey
53              fromConnectionSecretKey: publicKeyType" | kubectl apply -f -

Create a Claim

Now create a Claim to have Crossplane create the GCP resources and associated secrets.

Like the Composition, the Claim uses publishConnectionDetailsTo to connect to Vault and store the secrets.

 1echo "apiVersion: ess.example.org/v1alpha1
 2kind: ESSInstance
 3metadata:
 4  name: my-ess
 5  namespace: default
 6spec:
 7  parameters:
 8    serviceAccount: ess-test-sa
 9  compositionSelector:
10    matchLabels:
11      feature: ess
12  publishConnectionDetailsTo:
13    name: ess-claim-conn
14    metadata:
15      labels:
16        environment: development
17        team: backend
18    configRef:
19      name: vault" | kubectl apply -f -

Verify the resources

Verify all resources are READY and SYNCED:

1kubectl get managed
2NAME                                                      READY   SYNCED   DISPLAYNAME                     EMAIL                                                            DISABLED
3serviceaccount.iam.gcp.crossplane.io/my-ess-zvmkz-vhklg   True    True     a service account to test ess   my-ess-zvmkz-vhklg@testingforbugbounty.iam.gserviceaccount.com
4
5NAME                                                         READY   SYNCED   KEY_ID                                     CREATED_AT             EXPIRES_AT
6serviceaccountkey.iam.gcp.crossplane.io/my-ess-zvmkz-bq8pz   True    True     5cda49b7c32393254b5abb121b4adc07e140502c   2022-03-23T10:54:50Z

View the claims

1kubectl -n default get claim
2NAME     READY   CONNECTION-SECRET   AGE
3my-ess   True                        19s

View the composite resources.

1kubectl get composite
2NAME           READY   COMPOSITION                    AGE
3my-ess-zvmkz   True    essinstances.ess.example.org   32s

Verify Vault secrets

Look inside Vault to view the secrets from the managed resources.

1kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/default
2Keys
3----
4ess-claim-conn

The key ess-claim-conn is the name of the Claim’s publishConnectionDetailsTo configuration.

Check connection secrets in the crossplane-system Vault scope.

1kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/crossplane-system
2Keys
3----
4d2408335-eb88-4146-927b-8025f405da86
5ess-mr-conn

The key d2408335-eb88-4146-927b-8025f405da86 comes from

and the key ess-mr-conn comes from the Composition’s publishConnectionDetailsTo configuration.

Check contents of Claim’s connection secret ess-claim-conn to see the key created by the managed resource.

 1kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/default/ess-claim-conn
 2======= Metadata =======
 3Key                Value
 4---                -----
 5created_time       2022-03-18T21:24:07.2085726Z
 6custom_metadata    map[environment:development secret.crossplane.io/ner-uid:881cd9a0-6cc6-418f-8e1d-b36062c1e108 team:backend]
 7deletion_time      n/a
 8destroyed          false
 9version            1
10
11======== Data ========
12Key              Value
13---              -----
14publicKey        -----BEGIN PUBLIC KEY-----
15MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzsEYCokmYEsZJCc9QN/8
16Fm1M/kTPp7Gat/MXLTP3zFyCTBFVNLN79MbAKdinWi6ePXEb75vzB79IdZcWj8lo
178trnS64QjNB9Vs4Xk5UvDALwleFN/bZeperxivDPwVPvT9Aqy/U9kohoS/LHyE8w
18uWQb5AuMeVQ1gtCTnCqQZ4d2MSVhQXYVvAWax1spJ9LT7mHub5j95xDdYIcOV3VJ
19l9CIo4VrWIT8THFN2NnjTrGq9+0TzXY0bV674bjJkfBC6v6yXs5HTetG+Uekq/xf
20FCjrrDi1+2UR9Mu2WTuvl8qn50be+mbwdJO5wE32jewxdYrVVmj19+PkaEeAwGTc
21vwIDAQAB
22-----END PUBLIC KEY-----
23publicKeyType    TYPE_RAW_PUBLIC_KEY

Check contents of managed resource connection secret ess-mr-conn. The public key is identical to the public key in the Claim since the Claim is using this managed resource.

 1kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/crossplane-system/ess-mr-conn
 2======= Metadata =======
 3Key                Value
 4---                -----
 5created_time       2022-03-18T21:21:07.9298076Z
 6custom_metadata    map[environment:development secret.crossplane.io/ner-uid:4cd973f8-76fc-45d6-ad45-0b27b5e9252a team:backend]
 7deletion_time      n/a
 8destroyed          false
 9version            2
10
11========= Data =========
12Key               Value
13---               -----
14privateKey        {
15  "type": "service_account",
16  "project_id": "REDACTED",
17  "private_key_id": "REDACTED",
18  "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
19  "client_email": "ess-test-sa@REDACTED.iam.gserviceaccount.com",
20  "client_id": "REDACTED",
21  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
22  "token_uri": "https://oauth2.googleapis.com/token",
23  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
24  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ess-test-sa%40REDACTED.iam.gserviceaccount.com"
25}
26privateKeyType    TYPE_GOOGLE_CREDENTIALS_FILE
27publicKey         -----BEGIN PUBLIC KEY-----
28MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzsEYCokmYEsZJCc9QN/8
29Fm1M/kTPp7Gat/MXLTP3zFyCTBFVNLN79MbAKdinWi6ePXEb75vzB79IdZcWj8lo
308trnS64QjNB9Vs4Xk5UvDALwleFN/bZeperxivDPwVPvT9Aqy/U9kohoS/LHyE8w
31uWQb5AuMeVQ1gtCTnCqQZ4d2MSVhQXYVvAWax1spJ9LT7mHub5j95xDdYIcOV3VJ
32l9CIo4VrWIT8THFN2NnjTrGq9+0TzXY0bV674bjJkfBC6v6yXs5HTetG+Uekq/xf
33FCjrrDi1+2UR9Mu2WTuvl8qn50be+mbwdJO5wE32jewxdYrVVmj19+PkaEeAwGTc
34vwIDAQAB
35-----END PUBLIC KEY-----
36publicKeyType     TYPE_RAW_PUBLIC_KEY

Remove the resources

Deleting the Claim removes the managed resources and associated keys from Vault.

1kubectl delete claim my-ess