Vault as an External Secret Store

This document is for an unreleased version of Crossplane.

This document applies to the Crossplane master branch and not to the latest release v1.16.

This guide walks through the steps required to configure Crossplane and its Providers to use Vault as an External Secret Store (ESS) with ESS Plugin Vault.


External Secret Stores are an alpha feature.

They’re not recommended for production use. Crossplane disables External Secret Stores by default.

Crossplane uses sensitive information including Provider credentials, inputs to managed resources and connection details.

The Vault credential injection guide details using Vault and Crossplane for Provider credentials.

Crossplane doesn’t support for using Vault for managed resources input. Crossplane issue #2985 tracks support for this feature.

Supporting connection details with Vault requires a Crossplane external secret store.


This guide requires Helm version 3.11 or later.

Install Vault

Detailed instructions on installing Vault are available from the Vault documentation.

Add the Vault Helm chart

Add the Helm repository for hashicorp.

1helm repo add hashicorp --force-update 

Install Vault using Helm.

1helm -n vault-system upgrade --install vault hashicorp/vault --create-namespace

Unseal Vault

If Vault is sealed unseal Vault using the unseal keys.

Get the Vault keys.

1kubectl -n vault-system exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json
2VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")

Unseal the vault using the keys.

 1kubectl -n vault-system exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
 2Key             Value
 3---             -----
 4Seal Type       shamir
 5Initialized     true
 6Sealed          false
 7Total Shares    1
 8Threshold       1
 9Version         1.13.1
10Build Date      2023-03-23T12:51:35Z
11Storage Type    file
12Cluster Name    vault-cluster-df884357
13Cluster ID      b3145d26-2c1a-a7f2-a364-81753033c0d9
14HA Enabled      false

Configure Vault Kubernetes authentication

Enable the Kubernetes auth method for Vault to authenticate requests based on Kubernetes service accounts.

Get the Vault root token

The Vault root token is inside the JSON file created when unsealing Vault.

1cat cluster-keys.json | jq -r ".root_token"

Enable Kubernetes authentication

Connect to a shell in the Vault pod.

1kubectl -n vault-system exec -it vault-0 -- /bin/sh
2/ $

From the Vault shell, login to Vault using the root token.

 1vault login # use the root token from above
 2Token (will be hidden):
 3Success! You are now authenticated. The token information displayed below
 4is already stored in the token helper. You do NOT need to run "vault login"
 5again. Future Vault requests will automatically use this token.
 7Key                  Value
 8---                  -----
 9token                hvs.TSN4SssfMBM0HAtwGrxgARgn
10token_accessor       qodxHrINVlRXKyrGeeDkxnih
11token_duration       ∞
12token_renewable      false
13token_policies       ["root"]
14identity_policies    []
15policies             ["root"]

Enable the Kubernetes authentication method in Vault.

1vault auth enable kubernetes
2Success! Enabled kubernetes auth method at: kubernetes/

Configure Vault to communicate with Kubernetes and exit the Vault shell

1vault write auth/kubernetes/config \
2        token_reviewer_jwt="$(cat /var/run/secrets/" \
3        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
4        kubernetes_ca_cert=@/var/run/secrets/
5Success! Data written to: auth/kubernetes/config
6/ $ exit

Configure Vault for Crossplane integration

Crossplane relies on the Vault key-value secrets engine to store information and Vault requires a permissions policy for the Crossplane service account.

Enable the Vault kv secrets engine

Enable the Vault KV Secrets Engine.

Vault has two versions of the KV Secrets Engine. This example uses version 2.
1kubectl -n vault-system exec -it vault-0 -- vault secrets enable -path=secret kv-v2
2Success! Enabled the kv-v2 secrets engine at: secret/

Create a Vault policy for Crossplane

Create the Vault policy to allow Crossplane to read and write data from Vault.

1kubectl -n vault-system exec -i vault-0 -- vault policy write crossplane - <<EOF
2path "secret/data/*" {
3    capabilities = ["create", "read", "update", "delete"]
5path "secret/metadata/*" {
6    capabilities = ["create", "read", "update", "delete"]
9Success! Uploaded policy: crossplane

Apply the policy to Vault.

1kubectl -n vault-system exec -it vault-0 -- vault write auth/kubernetes/role/crossplane \
2    bound_service_account_names="*" \
3    bound_service_account_namespaces=crossplane-system \
4    policies=crossplane \
5    ttl=24h
6Success! Data written to: auth/kubernetes/role/crossplane

Install Crossplane

Crossplane v1.12 introduced the plugin support. Make sure your version of Crossplane supports plugins.

Install the Crossplane with the External Secrets Stores feature enabled.

1helm upgrade --install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace --set args='{--enable-external-secret-stores}'

Install the Crossplane Vault plugin

The Crossplane Vault plugin isn’t part of the default Crossplane install. The plugin installs as a unique Pod that uses the Vault Agent Sidecar Injection to connect the Vault secret store to Crossplane.

First, configure annotations for the Vault plugin pod.

1cat > values.yaml <<EOF
3 "true"
4 "true"
5 crossplane
6 "65532"

Next, install the Crossplane ESS Plugin pod to the crossplane-system namespace and apply the Vault annotations.

1helm upgrade --install ess-plugin-vault oci:// --namespace crossplane-system -f values.yaml

Configure Crossplane

Using the Vault plugin requires configuration to connect to the Vault service. The plugin also requires Providers to enable external secret stores.

With the plugin and providers configured, Crossplane requires two StoreConfig objects to describe how Crossplane and the Providers communicate with vault.

Enable external secret stores in the Provider

This example uses Provider GCP, but the ControllerConfig is the same for all Providers.

Create a ControllerConfig object to enable external secret stores.

1echo "apiVersion:
2kind: ControllerConfig
4  name: vault-config
6  args:
7    - --enable-external-secret-stores" | kubectl apply -f -

Install the Provider and apply the ControllerConfig.

1echo "apiVersion:
2kind: Provider
4  name: provider-gcp
6  package:
7  controllerConfigRef:
8    name: vault-config" | kubectl apply -f -

Connect the Crossplane plugin to Vault

Create a VaultConfig resource for the plugin to connect to the Vault service:

 1echo "apiVersion:
 2kind: VaultConfig
 4  name: vault-internal
 6  server: http://vault.vault-system:8200
 7  mountPath: secret/
 8  version: v2
 9  auth:
10    method: Token
11    token:
12      source: Filesystem
13      fs:
14        path: /vault/secrets/token" | kubectl apply -f -

Create a Crossplane StoreConfig

Create a StoreConfig object from the group. Crossplane uses the StoreConfig to connect to the Vault plugin service.

The configRef connects the StoreConfig to the specific Vault plugin configuration.

 1echo "apiVersion:
 2kind: StoreConfig
 4  name: vault
 6  type: Plugin
 7  defaultScope: crossplane-system
 8  plugin:
 9    endpoint: ess-plugin-vault.crossplane-system:4040
10    configRef:
11      apiVersion:
12      kind: VaultConfig
13      name: vault-internal" | kubectl apply -f -

Create a Provider StoreConfig

Create a StoreConfig object from the Provider’s API group, The Provider uses this StoreConfig to communicate with Vault for Managed Resources.

The configRef connects the StoreConfig to the specific Vault plugin configuration.

 1echo "apiVersion:
 2kind: StoreConfig
 4  name: vault
 6  type: Plugin
 7  defaultScope: crossplane-system
 8  plugin:
 9    endpoint: ess-plugin-vault.crossplane-system:4040
10    configRef:
11      apiVersion:
12      kind: VaultConfig
13      name: vault-internal" | kubectl apply -f -

Create Provider resources

Check that Crossplane installed the Provider and the Provider is healthy.

1kubectl get providers
2NAME           INSTALLED   HEALTHY   PACKAGE                                                                     AGE
3provider-gcp   True        True   10m

Create a CompositeResourceDefinition

Create a CompositeResourceDefinition to define a custom API endpoint.

 1echo "apiVersion:
 2kind: CompositeResourceDefinition
 4  name:
 5  annotations:
 6    feature: ess
 8  group:
 9  names:
10    kind: CompositeESSInstance
11    plural: compositeessinstances
12  claimNames:
13    kind: ESSInstance
14    plural: essinstances
15  connectionSecretKeys:
16    - publicKey
17    - publicKeyType
18  versions:
19  - name: v1alpha1
20    served: true
21    referenceable: true
22    schema:
23      openAPIV3Schema:
24        type: object
25        properties:
26          spec:
27            type: object
28            properties:
29              parameters:
30                type: object
31                properties:
32                  serviceAccount:
33                    type: string
34                required:
35                  - serviceAccount
36            required:
37              - parameters" | kubectl apply -f -

Create a Composition

Create a Composition to create a Service Account and Service Account Key inside GCP.

Creating a Service Account Key generates connectionDetails that the Provider stores in Vault using the publishConnectionDetailsTo details.

 1echo "apiVersion:
 2kind: Composition
 4  name:
 5  labels:
 6    feature: ess
 8  publishConnectionDetailsWithStoreConfigRef: 
 9    name: vault
10  compositeTypeRef:
11    apiVersion:
12    kind: CompositeESSInstance
13  resources:
14    - name: serviceaccount
15      base:
16        apiVersion:
17        kind: ServiceAccount
18        metadata:
19          name: ess-test-sa
20        spec:
21          forProvider:
22            displayName: a service account to test ess
23    - name: serviceaccountkey
24      base:
25        apiVersion:
26        kind: ServiceAccountKey
27        spec:
28          forProvider:
29            serviceAccountSelector:
30              matchControllerRef: true
31          publishConnectionDetailsTo:
32            name: ess-mr-conn
33            metadata:
34              labels:
35                environment: development
36                team: backend
37            configRef:
38              name: vault
39      connectionDetails:
40        - fromConnectionSecretKey: publicKey
41        - fromConnectionSecretKey: publicKeyType" | kubectl apply -f -

Create a Claim

Now create a Claim to have Crossplane create the GCP resources and associated secrets.

Like the Composition, the Claim uses publishConnectionDetailsTo to connect to Vault and store the secrets.

 1echo "apiVersion:
 2kind: ESSInstance
 4  name: my-ess
 5  namespace: default
 7  parameters:
 8    serviceAccount: ess-test-sa
 9  compositionSelector:
10    matchLabels:
11      feature: ess
12  publishConnectionDetailsTo:
13    name: ess-claim-conn
14    metadata:
15      labels:
16        environment: development
17        team: backend
18    configRef:
19      name: vault" | kubectl apply -f -

Verify the resources

Verify all resources are READY and SYNCED:

1kubectl get managed
2NAME                                                      READY   SYNCED   DISPLAYNAME                     EMAIL                                                            DISABLED   True    True     a service account to test ess
5NAME                                                         READY   SYNCED   KEY_ID                                     CREATED_AT             EXPIRES_AT   True    True     5cda49b7c32393254b5abb121b4adc07e140502c   2022-03-23T10:54:50Z

View the claims

1kubectl -n default get claim
3my-ess   True                        19s

View the composite resources.

1kubectl get composite
2NAME           READY   COMPOSITION                    AGE
3my-ess-zvmkz   True   32s

Verify Vault secrets

Look inside Vault to view the secrets from the managed resources.

1kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/default

The key ess-claim-conn is the name of the Claim’s publishConnectionDetailsTo configuration.

Check connection secrets in the “crossplane-system” Vault scope.

1kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/crossplane-system

The key d2408335-eb88-4146-927b-8025f405da86 comes from

and the key ess-mr-conn comes from the Composition’s publishConnectionDetailsTo configuration.

Check contents of Claim’s connection secret ess-claim-conn to see the key created by the managed resource.

 1kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/default/ess-claim-conn
 2======= Metadata =======
 3Key                Value
 4---                -----
 5created_time       2022-03-18T21:24:07.2085726Z
 6custom_metadata    map[environment:development team:backend]
 7deletion_time      n/a
 8destroyed          false
 9version            1
11======== Data ========
12Key              Value
13---              -----
14publicKey        -----BEGIN PUBLIC KEY-----
22-----END PUBLIC KEY-----
23publicKeyType    TYPE_RAW_PUBLIC_KEY

Check contents of managed resource connection secret ess-mr-conn. The public key is identical to the public key in the Claim since the Claim is using this managed resource.

 1kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/crossplane-system/ess-mr-conn
 2======= Metadata =======
 3Key                Value
 4---                -----
 5created_time       2022-03-18T21:21:07.9298076Z
 6custom_metadata    map[environment:development team:backend]
 7deletion_time      n/a
 8destroyed          false
 9version            2
11========= Data =========
12Key               Value
13---               -----
14privateKey        {
15  "type": "service_account",
16  "project_id": "REDACTED",
17  "private_key_id": "REDACTED",
18  "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
19  "client_email": "",
20  "client_id": "REDACTED",
21  "auth_uri": "",
22  "token_uri": "",
23  "auth_provider_x509_cert_url": "",
24  "client_x509_cert_url": ""
27publicKey         -----BEGIN PUBLIC KEY-----
35-----END PUBLIC KEY-----
36publicKeyType     TYPE_RAW_PUBLIC_KEY

Remove the resources

Deleting the Claim removes the managed resources and associated keys from Vault.

1kubectl delete claim my-ess