Crossplane logo
Crossplane logo
master v1.11
Latest
v1.10 v1.9

Vault as an External Secret Store

On this page
This document is for an older version of Crossplane.

This document applies to Crossplane version v1.9 and not to the latest release v1.11.

This guide walks through the steps required to configure Crossplane and its Providers to use Vault as an External Secret Store. For the sake of completeness, we will also include steps for Vault installation and setup, however, you can skip those and use your existing Vault.

External Secret Stores are an alpha feature. They are not yet recommended for production use, and are disabled by default.

Crossplane consumes and also produces sensitive information to operate which could be categorized as follows:

  1. Provider credentials: These are the credentials required for Providers to authenticate against external APIs. For example, AWS Access/Secret keys, GCP service account json, etc.
  2. Connection Details: Once an infrastructure provisioned, we usually need some connection data to consume it. Most of the time, this information includes sensitive information like usernames, passwords or access keys.
  3. Sensitive Inputs to Managed Resources: There are some Managed resources which expect input parameters that could be sensitive. Initial password of a managed database is a good example of this category.

It is already possible to use Vault for the 1st category (i.e. Provider Credentials) as described in the previous guide. The 3rd use case is relatively rare and being tracked with this issue.

In this guide we will focus on the 2nd category, which is storing Connection Details for managed resources in Vault.

Steps

Some steps in this guide duplicates the previous guide on Vault injection. However, for convenience, we put them here as well with minor changes/improvements.

At a high level we will run the following steps:

  • Install and Unseal Vault.
  • Configure Vault with Kubernetes Auth.
  • Install and Configure Crossplane by enabling the feature.
  • Install and Configure Provider GCP by enabling the feature.
  • Deploy a Composition and CompositeResourceDefinition.
  • Create a Claim.
  • Verify all secrets land in Vault as expected.

For simplicity, we will deploy Vault into the same cluster as Crossplane, however, this is not a requirement as long as Vault has Kubernetes auth enabled for the cluster where Crossplane is running.

Prepare Vault

  1. Install Vault Helm Chart
1kubectl create ns vault-system
2
3helm repo add hashicorp https://helm.releases.hashicorp.com --force-update
4helm -n vault-system upgrade --install vault hashicorp/vault
  1. Unseal Vault
1kubectl -n vault-system exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json
2VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")
3kubectl -n vault-system exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
  1. Configure Vault with Kubernetes Auth.

In order for Vault to be able to authenticate requests based on Kubernetes service accounts, the Kubernetes auth method must be enabled. This requires logging in to Vault and configuring it with a service account token, API server address, and certificate. Because we are running Vault in Kubernetes, these values are already available via the container filesystem and environment variables.

Get Vault Root Token:

1cat cluster-keys.json | jq -r ".root_token"

Login as root and enable/configure Kubernetes Auth:

 1kubectl -n vault-system exec -it vault-0 -- /bin/sh
 2
 3vault login # use root token from above
 4
 5vault auth enable kubernetes
 6vault write auth/kubernetes/config \
 7        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 8        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
 9        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
10        
11exit # exit vault container
  1. Enable Vault Key Value Secret Engine

There are two different versions of Vault KV Secrets Engine, v1 and v2, which you can find more details in the linked documentation page. We will use v2 in this guide as an example, however, both versions are supported as an external secret store.

1kubectl -n vault-system exec -it vault-0 -- vault secrets enable -path=secret kv-v2
  1. Create a Vault Policy and Role for Crossplane
 1kubectl -n vault-system exec -i vault-0 -- vault policy write crossplane - <<EOF
 2path "secret/data/*" {
 3    capabilities = ["create", "read", "update", "delete"]
 4}
 5path "secret/metadata/*" {
 6    capabilities = ["create", "read", "update", "delete"]
 7}
 8EOF
 9
10kubectl -n vault-system exec -it vault-0 -- vault write auth/kubernetes/role/crossplane \
11    bound_service_account_names="*" \
12    bound_service_account_namespaces=crossplane-system \
13    policies=crossplane \
14    ttl=24h

Install and Configure Crossplane

  1. Install Crossplane by:
 1kubectl create ns crossplane-system
 2
 3helm repo add crossplane-stable https://charts.crossplane.io/stable --force-update
 4
 5helm upgrade --install crossplane crossplane-stable/crossplane --namespace crossplane-system \
 6  --set 'args={--enable-external-secret-stores}' \
 7  --set-string customAnnotations."vault\.hashicorp\.com/agent-inject"=true \
 8  --set-string customAnnotations."vault\.hashicorp\.com/agent-inject-token"=true \
 9  --set-string customAnnotations."vault\.hashicorp\.com/role"=crossplane \
10  --set-string customAnnotations."vault\.hashicorp\.com/agent-run-as-user"=65532
  1. Create a Secret StoreConfig for Crossplane to be used by Composition types, i.e. Composites and Claims:
 1echo "apiVersion: secrets.crossplane.io/v1alpha1
 2kind: StoreConfig
 3metadata:
 4  name: vault
 5spec:
 6  type: Vault
 7  defaultScope: crossplane-system
 8  vault:
 9    server: http://vault.vault-system:8200
10    mountPath: secret/
11    version: v2
12    auth:
13      method: Token
14      token:
15        source: Filesystem
16        fs:
17          path: /vault/secrets/token" | kubectl apply -f -

Install and Configure Provider GCP

  1. Similar to Crossplane, install Provider GCP by:
 1echo "apiVersion: pkg.crossplane.io/v1alpha1
 2kind: ControllerConfig
 3metadata:
 4  name: vault-config
 5spec:
 6  args:
 7    - --enable-external-secret-stores
 8  metadata:
 9    annotations:
10      vault.hashicorp.com/agent-inject: \"true\"
11      vault.hashicorp.com/agent-inject-token: \"true\"
12      vault.hashicorp.com/role: crossplane
13      vault.hashicorp.com/agent-run-as-user: \"2000\"
14---
15apiVersion: pkg.crossplane.io/v1
16kind: Provider
17metadata:
18  name: provider-gcp
19spec:
20  package: xpkg.upbound.io/crossplane-contrib/provider-gcp:v0.22.0
21  controllerConfigRef:
22    name: vault-config" | kubectl apply -f -
  1. Create a Secret StoreConfig for Provider GCP to be used by GCP Managed Resources:
 1echo "apiVersion: gcp.crossplane.io/v1alpha1
 2kind: StoreConfig
 3metadata:
 4  name: vault
 5spec:
 6  type: Vault
 7  defaultScope: crossplane-system
 8  vault:
 9    server: http://vault.vault-system:8200
10    mountPath: secret/
11    version: v2
12    auth:
13      method: Token
14      token:
15        source: Filesystem
16        fs:
17          path: /vault/secrets/token" | kubectl apply -f -

Deploy and Test

Prerequisite: You should have a working default ProviderConfig for GCP available.

  1. Create a Composition and a CompositeResourceDefinition:
 1echo "apiVersion: apiextensions.crossplane.io/v1
 2kind: CompositeResourceDefinition
 3metadata:
 4  name: compositeessinstances.ess.example.org
 5  annotations:
 6    feature: ess
 7spec:
 8  group: ess.example.org
 9  names:
10    kind: CompositeESSInstance
11    plural: compositeessinstances
12  claimNames:
13    kind: ESSInstance
14    plural: essinstances
15  connectionSecretKeys:
16    - publicKey
17    - publicKeyType
18  versions:
19  - name: v1alpha1
20    served: true
21    referenceable: true
22    schema:
23      openAPIV3Schema:
24        type: object
25        properties:
26          spec:
27            type: object
28            properties:
29              parameters:
30                type: object
31                properties:
32                  serviceAccount:
33                    type: string
34                required:
35                  - serviceAccount
36            required:
37              - parameters" | kubectl apply -f -
38              
39echo "apiVersion: apiextensions.crossplane.io/v1
40kind: Composition
41metadata:
42  name: essinstances.ess.example.org
43  labels:
44    feature: ess
45spec:
46  publishConnectionDetailsWithStoreConfigRef: 
47    name: vault
48  compositeTypeRef:
49    apiVersion: ess.example.org/v1alpha1
50    kind: CompositeESSInstance
51  resources:
52    - name: serviceaccount
53      base:
54        apiVersion: iam.gcp.crossplane.io/v1alpha1
55        kind: ServiceAccount
56        metadata:
57          name: ess-test-sa
58        spec:
59          forProvider:
60            displayName: a service account to test ess
61    - name: serviceaccountkey
62      base:
63        apiVersion: iam.gcp.crossplane.io/v1alpha1
64        kind: ServiceAccountKey
65        spec:
66          forProvider:
67            serviceAccountSelector:
68              matchControllerRef: true
69          publishConnectionDetailsTo:
70            name: ess-mr-conn
71            metadata:
72              labels:
73                environment: development
74                team: backend
75            configRef:
76              name: vault
77      connectionDetails:
78        - fromConnectionSecretKey: publicKey
79        - fromConnectionSecretKey: publicKeyType" | kubectl apply -f -
  1. Create a Claim:
 1echo "apiVersion: ess.example.org/v1alpha1
 2kind: ESSInstance
 3metadata:
 4  name: my-ess
 5  namespace: default
 6spec:
 7  parameters:
 8    serviceAccount: ess-test-sa
 9  compositionSelector:
10    matchLabels:
11      feature: ess
12  publishConnectionDetailsTo:
13    name: ess-claim-conn
14    metadata:
15      labels:
16        environment: development
17        team: backend
18    configRef:
19      name: vault" | kubectl apply -f -
  1. Verify all resources SYNCED and READY:
 1kubectl get managed
 2# Example output:
 3# NAME                                                      READY   SYNCED   DISPLAYNAME                     EMAIL                                                            DISABLED
 4# serviceaccount.iam.gcp.crossplane.io/my-ess-zvmkz-vhklg   True    True     a service account to test ess   my-ess-zvmkz-vhklg@testingforbugbounty.iam.gserviceaccount.com
 5
 6# NAME                                                         READY   SYNCED   KEY_ID                                     CREATED_AT             EXPIRES_AT
 7# serviceaccountkey.iam.gcp.crossplane.io/my-ess-zvmkz-bq8pz   True    True     5cda49b7c32393254b5abb121b4adc07e140502c   2022-03-23T10:54:50Z
 8
 9kubectl -n default get claim
10# Example output:
11# NAME     READY   CONNECTION-SECRET   AGE
12# my-ess   True                        19s
13
14kubectl get composite
15# Example output:
16# NAME           READY   COMPOSITION                    AGE
17# my-ess-zvmkz   True    essinstances.ess.example.org   32s

Verify the Connection Secrets landed to Vault 

 1# Check connection secrets in the "default" scope (namespace). 
 2kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/default
 3# Example output:
 4# Keys
 5# ----
 6# ess-claim-conn
 7
 8# Check connection secrets in the "crossplane-system" scope (namespace).
 9kubectl -n vault-system exec -i vault-0 -- vault kv list /secret/crossplane-system
10# Example output:
11# Keys
12# ----
13# d2408335-eb88-4146-927b-8025f405da86
14# ess-mr-conn
15
16# Check contents of claim connection secret
17kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/default/ess-claim-conn
18# Example output:
19# ======= Metadata =======
20# Key                Value
21# ---                -----
22# created_time       2022-03-18T21:24:07.2085726Z
23# custom_metadata    map[environment:development secret.crossplane.io/owner-uid:881cd9a0-6cc6-418f-8e1d-b36062c1e108 team:backend]
24# deletion_time      n/a
25# destroyed          false
26# version            1
27# 
28# ======== Data ========
29# Key              Value
30# ---              -----
31# publicKey        -----BEGIN PUBLIC KEY-----
32# MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzsEYCokmYEsZJCc9QN/8
33# Fm1M/kTPp7Gat/MXLTP3zFyCTBFVNLN79MbAKdinWi6ePXEb75vzB79IdZcWj8lo
34# 8trnS64QjNB9Vs4Xk5UvDALwleFN/bZeperxivDPwVPvT9Aqy/U9kohoS/LHyE8w
35# uWQb5AuMeVQ1gtCTnCqQZ4d2MSVhQXYVvAWax1spJ9LT7mHub5j95xDdYIcOV3VJ
36# l9CIo4VrWIT8THFN2NnjTrGq9+0TzXY0bV674bjJkfBC6v6yXs5HTetG+Uekq/xf
37# FCjrrDi1+2UR9Mu2WTuvl8qn50be+mbwdJO5wE32jewxdYrVVmj19+PkaEeAwGTc
38# vwIDAQAB
39# -----END PUBLIC KEY-----
40# publicKeyType    TYPE_RAW_PUBLIC_KEY
41
42# Check contents of managed resource connection secret
43kubectl -n vault-system exec -i vault-0 -- vault kv get /secret/crossplane-system/ess-mr-conn
44# Example output:
45# ======= Metadata =======
46# Key                Value
47# ---                -----
48# created_time       2022-03-18T21:21:07.9298076Z
49# custom_metadata    map[environment:development secret.crossplane.io/owner-uid:4cd973f8-76fc-45d6-ad45-0b27b5e9252a team:backend]
50# deletion_time      n/a
51# destroyed          false
52# version            2
53# 
54# ========= Data =========
55# Key               Value
56# ---               -----
57# privateKey        {
58#   "type": "service_account",
59#   "project_id": "REDACTED",
60#   "private_key_id": "REDACTED",
61#   "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
62#   "client_email": "ess-test-sa@REDACTED.iam.gserviceaccount.com",
63#   "client_id": "REDACTED",
64#   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
65#   "token_uri": "https://oauth2.googleapis.com/token",
66#   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
67#   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ess-test-sa%40REDACTED.iam.gserviceaccount.com"
68# }
69# privateKeyType    TYPE_GOOGLE_CREDENTIALS_FILE
70# publicKey         -----BEGIN PUBLIC KEY-----
71# MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzsEYCokmYEsZJCc9QN/8
72# Fm1M/kTPp7Gat/MXLTP3zFyCTBFVNLN79MbAKdinWi6ePXEb75vzB79IdZcWj8lo
73# 8trnS64QjNB9Vs4Xk5UvDALwleFN/bZeperxivDPwVPvT9Aqy/U9kohoS/LHyE8w
74# uWQb5AuMeVQ1gtCTnCqQZ4d2MSVhQXYVvAWax1spJ9LT7mHub5j95xDdYIcOV3VJ
75# l9CIo4VrWIT8THFN2NnjTrGq9+0TzXY0bV674bjJkfBC6v6yXs5HTetG+Uekq/xf
76# FCjrrDi1+2UR9Mu2WTuvl8qn50be+mbwdJO5wE32jewxdYrVVmj19+PkaEeAwGTc
77# vwIDAQAB
78# -----END PUBLIC KEY-----
79# publicKeyType     TYPE_RAW_PUBLIC_KEY

The commands above verifies using the cli, however, you can also connect to the Vault UI and check secrets there.

1kubectl -n vault-system port-forward vault-0 8200:8200

Now, you can open http://127.0.0.1:8200/ui in browser and login with the root token.

Cleanup

Delete the claim which should clean up all the resources created.

kubectl -n default delete claim my-ess