Vault as an External Secret Store

On this page

This guide walks through the steps required to configure Crossplane and its Providers to use Vault as an External Secret Store. For the sake of completeness, we will also include steps for Vault installation and setup, however, you can skip those and use your existing Vault.

External Secret Stores are an alpha feature. They are not yet recommended for production use, and are disabled by default.

Crossplane consumes and also produces sensitive information to operate which could be categorized as follows:

Provider credentials: These are the credentials required for Providers to authenticate against external APIs. For example, AWS Access/Secret keys, GCP service account json, etc.
Connection Details: Once an infrastructure provisioned, we usually need some connection data to consume it. Most of the time, this information includes sensitive information like usernames, passwords or access keys.
Sensitive Inputs to Managed Resources: There are some Managed resources which expect input parameters that could be sensitive. Initial password of a managed database is a good example of this category.

It is already possible to use Vault for the 1st category (i.e. Provider Credentials) as described in the previous guide. The 3rd use case is relatively rare and being tracked with this issue.

In this guide we will focus on the 2nd category, which is storing Connection Details for managed resources in Vault.

Steps

Some steps in this guide duplicates the previous guide on Vault injection. However, for convenience, we put them here as well with minor changes/improvements.

At a high level we will run the following steps:

Install and Unseal Vault.
Configure Vault with Kubernetes Auth.
Install and Configure Crossplane by enabling the feature.
Install and Configure Provider GCP by enabling the feature.
Deploy a Composition and CompositeResourceDefinition.
Create a Claim.
Verify all secrets land in Vault as expected.

For simplicity, we will deploy Vault into the same cluster as Crossplane, however, this is not a requirement as long as Vault has Kubernetes auth enabled for the cluster where Crossplane is running.

Prepare Vault

Install Vault Helm Chart

Add the Vault Helm chart.

1helm repo add hashicorp https://helm.releases.hashicorp.com --force-update

Install Vault.

1helm -n vault-system --create-namespace upgrade --install vault hashicorp/vault

Unseal Vault

Get the Vault keys.

1kubectl -n vault-system exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json
2VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")

Unseal the vault using the keys.

1kubectl -n vault-system exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

Configure Vault with Kubernetes Auth.

In order for Vault to be able to authenticate requests based on Kubernetes service accounts, the Kubernetes auth method must be enabled. This requires logging in to Vault and configuring it with a service account token, API server address, and certificate. Because we are running Vault in Kubernetes, these values are already available via the container filesystem and environment variables.

Get Vault Root Token:

1cat cluster-keys.json | jq -r ".root_token"

1kubectl -n vault-system exec -it vault-0 -- /bin/sh

1vault login # use root token from above

Enable the Kubernetes authentication method.

1vault auth enable kubernetes

Configure Vault to talk to Kubernetes.

1vault write auth/kubernetes/config \
2        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
3        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
4        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Enable Vault Key Value Secret Engine

There are two different versions of Vault KV Secrets Engine, v1 and v2, which you can find more details in the linked documentation page. We will use v2 in this guide as an example, however, both versions are supported as an external secret store.

1kubectl -n vault-system exec -it vault-0 -- vault secrets enable -path=secret kv-v2

Create a Vault Policy and Role for Crossplane

1kubectl -n vault-system exec -i vault-0 -- vault policy write crossplane - <<EOF
2path "secret/data/*" {
3    capabilities = ["create", "read", "update", "delete"]
4}
5path "secret/metadata/*" {
6    capabilities = ["create", "read", "update", "delete"]
7}
8EOF

Apply the vault policy.

1kubectl -n vault-system exec -it vault-0 -- vault write auth/kubernetes/role/crossplane \
2    bound_service_account_names="*" \
3    bound_service_account_namespaces=crossplane-system \
4    policies=crossplane \
5    ttl=24h

Install and Configure Crossplane

Install Crossplane by:

Enabling External Secret Stores feature.
Annotating for Vault Agent Sidecar Injection

1helm repo add crossplane-stable https://charts.crossplane.io/stable --force-update

Create the Vault configuration settings.

1cat << EOF > values.yaml
2args:
3- --enable-external-secret-stores
4customAnnotations:
5  vault.hashicorp.com/agent-inject: "true"
6  vault.hashicorp.com/agent-inject-token: "true"
7  vault.hashicorp.com/role: "crossplane"
8  vault.hashicorp.com/agent-run-as-user: "65532"
9  EOF

Apply the settings to the Crossplane installation.

1helm upgrade --install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace -f values.yaml

Create a Secret StoreConfig for Crossplane to be used by Composition types, i.e. Composites and Claims:

 1echo "apiVersion: secrets.crossplane.io/v1alpha1
 2kind: StoreConfig
 3metadata:
 4  name: vault
 5spec:
 6  type: Vault
 7  defaultScope: crossplane-system
 8  vault:
 9    server: http://vault.vault-system:8200
10    mountPath: secret/
11    version: v2
12    auth:
13      method: Token
14      token:
15        source: Filesystem
16        fs:
17          path: /vault/secrets/token" | kubectl apply -f -

Install and Configure Provider GCP

Similar to Crossplane, install Provider GCP by:

Enabling External Secret Stores feature.
Annotating for Vault Agent Sidecar Injection

 1echo "apiVersion: pkg.crossplane.io/v1alpha1
 2kind: ControllerConfig
 3metadata:
 4  name: vault-config
 5spec:
 6  args:
 7    - --enable-external-secret-stores
 8  metadata:
 9    annotations:
10      vault.hashicorp.com/agent-inject: \"true\"
11      vault.hashicorp.com/agent-inject-token: \"true\"
12      vault.hashicorp.com/role: crossplane
13      vault.hashicorp.com/agent-run-as-user: \"2000\"
14---
15apiVersion: pkg.crossplane.io/v1
16kind: Provider
17metadata:
18  name: provider-gcp
19spec:
20  package: xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.22.0
21  controllerConfigRef:
22    name: vault-config" | kubectl apply -f -

Create a Secret StoreConfig for Provider GCP to be used by GCP Managed Resources:

 1echo "apiVersion: gcp.secrets.crossplane.io/v1alpha1
 2kind: StoreConfig
 3metadata:
 4  name: vault
 5spec:
 6  type: Vault
 7  defaultScope: crossplane-system
 8  vault:
 9    server: http://vault.vault-system:8200
10    mountPath: secret/
11    version: v2
12    auth:
13      method: Token
14      token:
15        source: Filesystem
16        fs:
17          path: /vault/secrets/token" | kubectl apply -f -

Deploy and Test

Prerequisite: You should have a working default ProviderConfig for GCP available.

Create a CompositeResourceDefinition:

 1echo "apiVersion: apiextensions.crossplane.io/v1
 2kind: CompositeResourceDefinition
 3metadata:
 4  name: compositeessinstances.ess.example.org
 5  annotations:
 6    feature: ess
 7spec:
 8  group: ess.example.org
 9  names:
10    kind: CompositeESSInstance
11    plural: compositeessinstances
12  claimNames:
13    kind: ESSInstance
14    plural: essinstances
15  connectionSecretKeys:
16    - publicKey
17    - publicKeyType
18  versions:
19  - name: v1alpha1
20    served: true
21    referenceable: true
22    schema:
23      openAPIV3Schema:
24        type: object
25        properties:
26          spec:
27            type: object
28            properties:
29              parameters:
30                type: object
31                properties:
32                  serviceAccount:
33                    type: string
34                required:
35                  - serviceAccount
36            required:
37              - parameters" | kubectl apply -f -

Create a Composition.

 1echo "apiVersion: apiextensions.crossplane.io/v1
 2kind: Composition
 3metadata:
 4  name: essinstances.ess.example.org
 5  labels:
 6    feature: ess
 7spec:
 8  publishConnectionDetailsWithStoreConfigRef: 
 9    name: vault
10  compositeTypeRef:
11    apiVersion: ess.example.org/v1alpha1
12    kind: CompositeESSInstance
13  resources:
14    - name: serviceaccount
15      base:
16        apiVersion: iam.gcp.crossplane.io/v1alpha1
17        kind: ServiceAccount
18        metadata:
19          name: ess-test-sa
20        spec:
21          forProvider:
22            displayName: a service account to test ess
23    - name: serviceaccountkey
24      base:
25        apiVersion: iam.gcp.crossplane.io/v1alpha1
26        kind: ServiceAccountKey
27        spec:
28          forProvider:
29            serviceAccountSelector:
30              matchControllerRef: true
31          publishConnectionDetailsTo:
32            name: ess-mr-conn
33            metadata:
34              labels:
35                environment: development
36                team: backend
37            configRef:
38              name: vault
39      connectionDetails:
40        - fromConnectionSecretKey: publicKey
41        - fromConnectionSecretKey: publicKeyType" | kubectl apply -f -

Create a Claim:

 1echo "apiVersion: ess.example.org/v1alpha1
 2kind: ESSInstance
 3metadata:
 4  name: my-ess
 5  namespace: default
 6spec:
 7  parameters:
 8    serviceAccount: ess-test-sa
 9  compositionSelector:
10    matchLabels:
11      feature: ess
12  publishConnectionDetailsTo:
13    name: ess-claim-conn
14    metadata:
15      labels:
16        environment: development
17        team: backend
18    configRef:
19      name: vault" | kubectl apply -f -

Verify all resources SYNCED and READY:

View the managed resources.

1kubectl get managed
2# Example output:
3# NAME                                                      READY   SYNCED   DISPLAYNAME                     EMAIL                                                            DISABLED
4# serviceaccount.iam.gcp.crossplane.io/my-ess-zvmkz-vhklg   True    True     a service account to test ess   my-ess-zvmkz-vhklg@testingforbugbounty.iam.gserviceaccount.com
5
6# NAME                                                         READY   SYNCED   KEY_ID                                     CREATED_AT             EXPIRES_AT
7# serviceaccountkey.iam.gcp.crossplane.io/my-ess-zvmkz-bq8pz   True    True     5cda49b7c32393254b5abb121b4adc07e140502c   2022-03-23T10:54:50Z

View the claims

1kubectl -n default get claim
2# Example output:
3# NAME     READY   CONNECTION-SECRET   AGE
4# my-ess   True                        19s

View the composite resources.

1kubectl get composite
2# Example output:
3# NAME           READY   COMPOSITION                    AGE
4# my-ess-zvmkz   True    essinstances.ess.example.org   32s

Verify the Connection Secrets landed to Vault

1kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/default
2# Example output:
3# Keys
4# ----
5# ess-claim-conn

Check connection secrets in the “crossplane-system” scope (namespace).

1kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/crossplane-system
2# Example output:
3# Keys
4# ----
5# d2408335-eb88-4146-927b-8025f405da86
6# ess-mr-conn

Check contents of claim connection secret

 1kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/default/ess-claim-conn
 2# Example output:
 3# ======= Metadata =======
 4# Key                Value
 5# ---                -----
 6# created_time       2022-03-18T21:24:07.2085726Z
 7# custom_metadata    map[environment:development secret.crossplane.io/owner-uid:881cd9a0-6cc6-418f-8e1d-b36062c1e108 team:backend]
 8# deletion_time      n/a
 9# destroyed          false
10# version            1
11# 
12# ======== Data ========
13# Key              Value
14# ---              -----
15# publicKey        -----BEGIN PUBLIC KEY-----
16# MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzsEYCokmYEsZJCc9QN/8
17# Fm1M/kTPp7Gat/MXLTP3zFyCTBFVNLN79MbAKdinWi6ePXEb75vzB79IdZcWj8lo
18# 8trnS64QjNB9Vs4Xk5UvDALwleFN/bZeperxivDPwVPvT9Aqy/U9kohoS/LHyE8w
19# uWQb5AuMeVQ1gtCTnCqQZ4d2MSVhQXYVvAWax1spJ9LT7mHub5j95xDdYIcOV3VJ
20# l9CIo4VrWIT8THFN2NnjTrGq9+0TzXY0bV674bjJkfBC6v6yXs5HTetG+Uekq/xf
21# FCjrrDi1+2UR9Mu2WTuvl8qn50be+mbwdJO5wE32jewxdYrVVmj19+PkaEeAwGTc
22# vwIDAQAB
23# -----END PUBLIC KEY-----
24# publicKeyType    TYPE_RAW_PUBLIC_KEY

Check contents of managed resource connection secret.

 1kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/crossplane-system/ess-mr-conn
 2# Example output:
 3# ======= Metadata =======
 4# Key                Value
 5# ---                -----
 6# created_time       2022-03-18T21:21:07.9298076Z
 7# custom_metadata    map[environment:development secret.crossplane.io/owner-uid:4cd973f8-76fc-45d6-ad45-0b27b5e9252a team:backend]
 8# deletion_time      n/a
 9# destroyed          false
10# version            2
11# 
12# ========= Data =========
13# Key               Value
14# ---               -----
15# privateKey        {
16#   "type": "service_account",
17#   "project_id": "REDACTED",
18#   "private_key_id": "REDACTED",
19#   "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
20#   "client_email": "ess-test-sa@REDACTED.iam.gserviceaccount.com",
21#   "client_id": "REDACTED",
22#   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
23#   "token_uri": "https://oauth2.googleapis.com/token",
24#   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
25#   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ess-test-sa%40REDACTED.iam.gserviceaccount.com"
26# }
27# privateKeyType    TYPE_GOOGLE_CREDENTIALS_FILE
28# publicKey         -----BEGIN PUBLIC KEY-----
29# MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzsEYCokmYEsZJCc9QN/8
30# Fm1M/kTPp7Gat/MXLTP3zFyCTBFVNLN79MbAKdinWi6ePXEb75vzB79IdZcWj8lo
31# 8trnS64QjNB9Vs4Xk5UvDALwleFN/bZeperxivDPwVPvT9Aqy/U9kohoS/LHyE8w
32# uWQb5AuMeVQ1gtCTnCqQZ4d2MSVhQXYVvAWax1spJ9LT7mHub5j95xDdYIcOV3VJ
33# l9CIo4VrWIT8THFN2NnjTrGq9+0TzXY0bV674bjJkfBC6v6yXs5HTetG+Uekq/xf
34# FCjrrDi1+2UR9Mu2WTuvl8qn50be+mbwdJO5wE32jewxdYrVVmj19+PkaEeAwGTc
35# vwIDAQAB
36# -----END PUBLIC KEY-----
37# publicKeyType     TYPE_RAW_PUBLIC_KEY

The commands above verifies using the cli, however, you can also connect to the Vault UI and check secrets there.

1kubectl -n vault-system port-forward vault-0 8200:8200

Now, you can open http://127.0.0.1:8200/ui in browser and login with the root token.

Cleanup

Delete the claim which should clean up all the resources created.

1kubectl -n default delete claim my-ess