Managed Resources

A Managed Resource (MR) is Crossplane’s representation of a resource in an external system - most commonly a cloud provider. Managed Resources are opinionated, Crossplane Resource Model (XRM) compliant Kubernetes Custom Resources that are installed by a Crossplane provider.

For example, RDSInstance in the AWS Provider corresponds to an actual RDS Instance in AWS. There is a one-to-one relationship and the changes on managed resources are reflected directly on the corresponding resource in the provider. Similarly, the Database types in the SQL provider represent a PostgreSQL or MySQL database.

Managed Resources are the building blocks of Crossplane. They’re designed to be composed into higher level, opinionated Custom Resources that Crossplane calls Composite Resources or XRs - not used directly. See the Composition documentation for more information.

Syntax

Crossplane API conventions extend the Kubernetes API conventions for the schema of Crossplane managed resources. Following is an example of a managed resource:

• AWS
• GCP
• Azure

 1apiVersion: database.aws.crossplane.io/v1beta1
 2kind: RDSInstance
 3metadata:
 4  name: rdspostgresql
 5spec:
 6  forProvider:
 7    region: us-east-1
 8    dbInstanceClass: db.t2.small
 9    masterUsername: masteruser
10    allocatedStorage: 20
11    engine: postgres
12    engineVersion: "12"
13    skipFinalSnapshotBeforeDeletion: true
14  writeConnectionSecretToRef:
15    namespace: crossplane-system
16    name: aws-rdspostgresql-conn

1kubectl apply -f https://raw.githubusercontent.com/crossplane/crossplane/release-1.10/docs/snippets/provision/aws.yaml

1kubectl get rdsinstance rdspostgresql

1kubectl describe secret aws-rdspostgresql-conn -n crossplane-system

1kubectl delete rdsinstance rdspostgresql

 1apiVersion: database.gcp.crossplane.io/v1beta1
 2kind: CloudSQLInstance
 3metadata:
 4  name: cloudsqlpostgresql
 5spec:
 6  forProvider:
 7    databaseVersion: POSTGRES_12
 8    region: us-central1
 9    settings:
10      tier: db-custom-1-3840
11      dataDiskType: PD_SSD
12      dataDiskSizeGb: 10
13  writeConnectionSecretToRef:
14    namespace: crossplane-system
15    name: cloudsqlpostgresql-conn

1kubectl apply -f https://raw.githubusercontent.com/crossplane/crossplane/release-1.10/docs/snippets/provision/gcp.yaml

1kubectl get cloudsqlinstance cloudsqlpostgresql

1kubectl describe secret cloudsqlpostgresql-conn -n crossplane-system

1kubectl delete cloudsqlinstance cloudsqlpostgresql

 1apiVersion: azure.crossplane.io/v1alpha3
 2kind: ResourceGroup
 3metadata:
 4  name: sqlserverpostgresql-rg
 5spec:
 6  location: West US 2
 7---
 8apiVersion: database.azure.crossplane.io/v1beta1
 9kind: PostgreSQLServer
10metadata:
11  name: sqlserverpostgresql
12spec:
13  forProvider:
14    administratorLogin: myadmin
15    resourceGroupNameRef:
16      name: sqlserverpostgresql-rg
17    location: West US 2
18    sslEnforcement: Disabled
19    version: "9.6"
20    sku:
21      tier: GeneralPurpose
22      capacity: 2
23      family: Gen5
24    storageProfile:
25      storageMB: 20480
26  writeConnectionSecretToRef:
27    namespace: crossplane-system
28    name: sqlserverpostgresql-conn

1kubectl apply -f https://raw.githubusercontent.com/crossplane/crossplane/release-1.10/docs/snippets/provision/azure.yaml

1kubectl get postgresqlserver sqlserverpostgresql

1kubectl describe secret sqlserverpostgresql-conn -n crossplane-system

1kubectl delete postgresqlserver sqlserverpostgresql
2kubectl delete resourcegroup sqlserverpostgresql-rg

In Kubernetes, spec top field represents the desired state of the user. Crossplane adheres to that and has its own conventions about how the fields under spec should look like.

• writeConnectionSecretToRef: A reference to the secret that you want this managed resource to write its connection secret that you’d be able to mount to your pods in the same namespace. For RDSInstance, this secret would contain endpoint, username and password.

• providerConfigRef: Reference to the ProviderConfig resource that will provide information regarding authentication of Crossplane to the provider. ProviderConfig resources refer to Secret and potentially contain other information regarding authentication. The providerConfigRef is defaulted to a ProviderConfig named default if omitted.

• deletionPolicy: Enum to specify whether the actual cloud resource should be deleted when this managed resource is deleted in Kubernetes API server. Possible values are Delete (the default) and Orphan.

• managementPolicy: Enum to specify the level of control Crossplane has over the external resource. Possible values are FullControl (the default) and ObserveOnly.

  Important

  managementPolicy is an experimental feature, see the management policies section below for further details.

• forProvider: While the rest of the fields relate to how Crossplane should behave, the fields under forProvider are solely used to configure the actual external resource. In most of the cases, the field names correspond to the what exists in provider’s API Reference.

  The objects under forProvider field can get huge depending on the provider API. For example, GCP ServiceAccount has only a few fields while GCP CloudSQLInstance has over 100 fields that you can configure.

Versioning

Crossplane closely follows the Kubernetes API versioning conventions for the CRDs that it deploys. In short, for vXbeta and vX versions, you can expect that either automatic migration or instructions for manual migration will be provided when a new version of that CRD schema is released.

In practice, we suggest the following guidelines to provider developers:

• Every new kind has to be introduced as v1alpha1 with no exception.
• Breaking changes require a version change, i.e. v1alpha1 needs to become v1alpha2.
  • Alpha resources don’t require automatic conversions or manual instructions but it’s recommended that manual instructions are provided.
  • Beta resources require at least manual instructions but it’s recommended that conversion webhooks are used so that users can upgrade without any hands-on operation.
  • Stable resources require conversion webhooks.
• As long as the developer feels comfortable with the guarantees above, they can bump the version to beta or stable given that the CRD shape adheres to the Crossplane Resource Model (XRM) specifications for managed resources here.
• It’s suggested that the bump from Alpha to Beta or from Beta to Stable happen after a bake period which includes at least one release.

Grouping

In general, managed resources are high fidelity resources meaning they will provide parameters and behaviors that are provided by the external resource API. This applies to grouping of resources, too. For example, Queue appears under sqs API group in AWS,so, its APIVersion and Kind look like the following:

1apiVersion: sqs.aws.crossplane.io/v1beta1
2kind: Queue

Behavior

As a general rule, managed resource controllers try not to make any decision that is not specified by the user in the desired state since managed resources are the lowest level primitives that operate directly on the cloud provider APIs.

Continuous Reconciliation

Crossplane providers continuously reconcile the managed resource to achieve the desired state. The parameters under spec are considered the one and only source of truth for the external resource. This means that if someone changed a configuration in the UI of the provider, like AWS Console, Crossplane will change it back to what’s given under spec.

Connection Details

Some Crossplane resources support writing connection details - things like URLs, usernames, endpoints, and passwords to a Kubernetes Secret. You can specify the secret to write by setting the spec.writeConnectionSecretToRef field. Note that while all managed resources have a writeConnectionSecretToRef field, not all managed resources actually have connection details to write - many will write an empty Secret.

Which managed resources have connection details and what connection details they have is currently undocumented. This is tracked in this issue.

Immutable Properties

There are configuration parameters in external resources that cloud providers do not allow to be changed. For example, in AWS, you cannot change the region of an RDSInstance.

Some infrastructure tools such as Terraform delete and recreate the resource to accommodate those changes but Crossplane does not take that route. Unless the managed resource is deleted and its deletionPolicy is Delete, its controller never deletes the external resource in the provider.

Kubernetes does not yet support immutable fields for custom resources. This means Crossplane will allow immutable fields to be changed, but will not actually make the desired change. This is tracked in this issue.

Pausing Reconciliations

If a managed resource being reconciled by the managed reconciler, has the crossplane.io/paused annotation with its value set to true as in the following example, then further reconciliations are paused on that resource after emitting an event with the type Synced, the status False, and the reason ReconcilePaused:

1apiVersion: ec2.aws.upbound.io/v1beta1
2kind: VPC
3metadata:
4  name: paused-vpc
5  annotations:
6    crossplane.io/paused: "true"
7...

Reconciliations on the managed resource will resume once the crossplane.io/paused annotation is removed or its value is set to anything other than true.

External Name

By default the name of the managed resource is used as the name of the external cloud resource that will show up in your cloud console. To specify a different external name, Crossplane has a special annotation to represent the name of the external resource. For example, I would like to have a CloudSQLInstance with an external name that is different than its managed resource name:

1apiVersion: database.gcp.crossplane.io/v1beta1
2kind: CloudSQLInstance
3metadata:
4  name: foodb
5  annotations:
6    crossplane.io/external-name: my-special-db
7spec:
8  ...

When you create this managed resource, you will see that the name of CloudSQLInstance in GCP console will be my-special-db.

If the annotation is not given, Crossplane will fill it with the name of the managed resource by default. In cases where provider doesn’t allow you to name the resource, like AWS VPC, the controller creates the resource and sets external annotation to be the name that the cloud provider chose. So, you would see something like vpc-28dsnh3 as the value of crossplane.io/external-name annotation of your AWS VPC resource even if you added your own custom external name during creation.

Late Initialization

For some of the optional fields, users rely on the default that the cloud provider chooses for them. Since Crossplane treats the managed resource as the source of the truth, values of those fields need to exist in spec of the managed resource. So, in each reconciliation, Crossplane will fill the value of a field that is left empty by the user but is assigned a value by the provider. For example, there could be two fields like region and availabilityZone and you might want to give only region and leave the availability zone to be chosen by the cloud provider. In that case, if the provider assigns an availability zone, Crossplane gets that value and fills availabilityZone. Note that if the field is already filled, the controller won’t override its value.

Deletion

When a deletion request is made for a managed resource, its controller starts the deletion process immediately. However, the managed resource is kept in the Kubernetes API (via a finalizer) until the controller confirms the external resource in the cloud is gone. So you can be sure that if the managed resource is deleted, then the external cloud resource is also deleted. Any errors that happen during deletion will be added to the status of the managed resource, so you can troubleshoot any issues.

Dependencies

In many cases, an external resource refers to another one for a specific configuration. For example, you could want your Azure Kubernetes cluster in a specific Virtual Network. External resources have specific fields for these relations, however, they usually require the information to be supplied in different formats. In Azure MySQL, you might be required to enter only the name of the Virtual Network while in Azure Kubernetes, it could be required to enter a string in a specific format that includes other information such as resource group name.

In Crossplane, users have 3 fields to refer to another resource. Here is an example from Azure MySQL managed resource referring to an Azure Resource Group:

1spec:
2  forProvider:
3    resourceGroupName: foo-res-group
4    resourceGroupNameRef:
5      name: resourcegroup
6    resourceGroupNameSelector:
7      matchLabels:
8        app: prod

In this example, the user provided only a set of labels to select a ResourceGroup managed resource that already exists in the cluster via resourceGroupNameSelector. Then after a specific ResourceGroup is selected, resourceGroupNameRef is filled with the name of that ResourceGroup managed resource. Then in the last step, Crossplane fills the actual resourceGroupName field with whatever format Azure accepts it. Once a dependency is resolved, the controller never changes it.

Users are able to specify any of these three fields:

• Selector to select via labels
• Reference to point to a determined managed resource
• Actual value that will be submitted to the provider

It’s important to note that in case a reference exists, the managed resource does not create the external resource until the referenced object is ready. In this example, creation call of Azure MySQL Server will not be made until referenced ResourceGroup has its status.condition named Ready to be true.

Management Policies

Crossplane offers a set of management policies that allow you to define the level of control it has over external resources. You can configure these policies using the spec.managementPolicy field in the managed resource definition. The available policies include:

• FullControl (Default): With this policy, Crossplane fully manages and controls the external resource.
• ObserveOnly: With the ObserveOnly policy, Crossplane only observes the external resource without making any changes or deletions.

To use management policies, you must enable them with the --enable-management-policies flag when starting the provider controller.

Importing Existing Resources

If you have some resources that are already provisioned in the cloud provider, you can import them as managed resources and let Crossplane manage them. What you need to do is to enter the name of the external resource as well as the required fields on the managed resource. For example, let’s say I have a GCP Network provisioned from GCP console and I would like to migrate it to Crossplane. Here is the YAML that I need to create:

 1apiVersion: compute.gcp.crossplane.io/v1beta1
 2kind: Network
 3metadata:
 4  name: foo-network
 5  annotations:
 6    crossplane.io/external-name: existing-network
 7spec:
 8  forProvider: {}
 9  providerConfigRef:
10    name: default

Crossplane will check whether a GCP Network called existing-network exists, and if it does, then the optional fields under forProvider will be filled with the values that are fetched from the provider.

Note that if a resource has required fields, you must fill those fields or the creation of the managed resource will be rejected. So, in those cases, you will need to enter the name of the resource as well as the required fields.

Alternative Import Procedure: Start with ObserveOnly

Directly importing existing managed resources approach has the following caveats:

1. You must provide all the required fields in the spec of the resource with correct values even though they’re not used for importing the resource. A wrong value for a required field result in a configuration update which isn’t desired.
2. Any typos in the external name annotation or mistakes in the identifying arguments, such as the region, results in the creation of a new resource instead of importing the existing one.

Instead of manually creating resources you can import the resource with an ObserveOnly management policy.

Crossplane imports ObserveOnly resources but never changes or deletes the resource.

To configure an ObserveOnly resource:

1. Create a new resource with an ObserveOnly management policy.
2. With the crossplane.io/external-name annotation set to the external name of the resource to import.
3. Only provide the identifying arguments (for example, region ) in the spec of the resource.

 1apiVersion: sql.gcp.upbound.io/v1beta1
 2kind: DatabaseInstance
 3metadata:
 4  annotations:
 5    crossplane.io/external-name: existing-database-instance
 6  name: existing-database-instance
 7spec:
 8  managementPolicy: ObserveOnly
 9  forProvider:
10    region: "us-central1"

Crossplane discovers the managed resource and populates the status.atProvider with the observed state.

 1apiVersion: sql.gcp.upbound.io/v1beta1
 2kind: DatabaseInstance
 3metadata:
 4  annotations:
 5    crossplane.io/external-name: existing-database-instance
 6  name: existing-database-instance
 7spec:
 8  managementPolicy: ObserveOnly
 9  forProvider:
10    region: us-central1
11status:
12  atProvider:
13    connectionName: crossplane-playground:us-central1:existing-database-instance
14    databaseVersion: POSTGRES_14
15    deletionProtection: true
16    firstIpAddress: 35.184.74.79
17    id: existing-database-instance
18    publicIpAddress: 35.184.74.79
19    region: us-central1
20    <truncated-for-brevity>
21    settings:
22    - activationPolicy: ALWAYS
23      availabilityType: REGIONAL
24      diskSize: 100
25      <truncated-for-brevity>
26      pricingPlan: PER_USE
27      tier: db-custom-4-26624
28      version: 4
29  conditions:
30  - lastTransitionTime: "2023-02-22T07:16:51Z"
31    reason: Available
32    status: "True"
33    type: Ready
34  - lastTransitionTime: "2023-02-22T07:16:51Z"
35    reason: ReconcileSuccess
36    status: "True"
37    type: Synced

To allow Crossplane to control and change the ObserveOnly resource, edit the policy.

Change the ObserveOnly field to FullControl .

Copy any required parameter values from status.atProvider and provide them in spec.forProvider .

 1apiVersion: sql.gcp.upbound.io/v1beta1
 2kind: DatabaseInstance
 3metadata:
 4  annotations:
 5    crossplane.io/external-name: existing-database-instance
 6  name: existing-database-instance
 7spec:
 8  managementPolicy: Full
 9  forProvider:
10    databaseVersion: POSTGRES_14
11    region: us-central1
12    settings:
13    - diskSize: 100
14      tier: db-custom-4-26624
15status:
16  atProvider:
17    <truncated-for-brevity>
18  conditions:
19    - lastTransitionTime: "2023-02-22T07:16:51Z"
20      reason: Available
21      status: "True"
22      type: Ready
23    - lastTransitionTime: "2023-02-22T11:16:45Z"
24      reason: ReconcileSuccess
25      status: "True"
26      type: Synced

Backup and Restore

Crossplane adheres to Kubernetes conventions as much as possible and one of the advantages we gain is backup & restore ability with tools that work with native Kubernetes types, like Velero.

If you’d like to backup and restore manually, you can simply export them and save YAMLs in your file system. When you reload them, as we’ve discovered in import section, their crossplane.io/external-name annotation and required fields are there and those are enough to import a resource. The tool you’re using needs to store annotations and spec fields, which most tools do including Velero.