Provider Development Guide

Crossplane allows you to manage infrastructure directly from Kubernetes. Each infrastructure API resource that Crossplane orchestrates is known as a “managed resource”. This guide will walk through the process of adding support for a new kind of managed resource to a Crossplane Provider.

You can watch TBS Episode 18 to follow along the live implementation of GCP PubSub managed resource.

If there is a corresponding Terraform Provider, please consider generating a Crossplane Provider with Terrajet by following the Generating a Crossplane Provider guide.

If you plan to implement a managed resource for AWS, please see the code generation guide.

What Makes a Crossplane Infrastructure Resource

Crossplane builds atop Kubernetes’s powerful architecture in which declarative configuration, known as resources, are continually ‘reconciled’ with reality by one or more controllers. A controller is an endless loop that:

  1. Observes the desired state (the declarative configuration resource).
  2. Observes the actual state (the thing said configuration resource represents).
  3. Tries to make the actual state match the desired state.

Typical Crossplane managed infrastructure consists of two configuration resources and one controller. The GCP Provider’s support for Google Cloud Memorystore illustrates this. First, the configuration resources:

  1. A managed resource. Managed resources are cluster scoped, high-fidelity representations of a resource in an external system such as a cloud provider’s API. Managed resources are non-portable across external systems (i.e. cloud providers); they’re tightly coupled to the implementation details of the external resource they represent. Managed resources are defined by a Provider. The GCP Provider’s CloudMemorystoreInstance resource is an example of a managed resource.
  2. A provider. Providers enable access to an external system, typically by indicating a Kubernetes Secret containing any credentials required to authenticate to the system, as well as any other metadata required to connect. Providers are cluster scoped, like managed resources and classes. The GCP ProviderConfig is an example of a provider. Note that provider is a somewhat overloaded term in the Crossplane ecosystem - it’s also used to refer to the controller manager for a particular cloud, for example provider-gcp.

A managed resource is powered by a controller. This controller is responsible for taking instances of the aforementioned high-fidelity managed resource kind and reconciling them with an external system. The CloudMemorystoreInstance controller watches for changes to CloudMemorystoreInstance resources and calls Google’s Cloud Memorystore API to create, update, or delete an instance as necessary.

Crossplane does not require controllers to be written in any particular language. The Kubernetes API server is our API boundary, so any process capable of watching the API server and updating resources can be a Crossplane controller.

Getting Started

At the time of writing all Crossplane Services controllers are written in Go, and built using crossplane-runtime. While it is possible to write a controller using any language and tooling with a Kubernetes client this set of tools are the “golden path”. They’re well supported, broadly used, and provide a shared language with the Crossplane community. This guide targets crossplane-runtime v0.9.0. It assumes the reader is familiar with the Kubernetes API Conventions and the kubebuilder book.

If you are building a new provider from scratch, instead of adding new resources to an already existing one, please use provider-template repository as a template by hitting the Use this template button in GitHub UI. It codifies most of the best practices used by the Crossplane community so far and is the easiest way to start a new provider.

Defining Resource Kinds

Let’s assume we want to add Crossplane support for your favourite cloud’s database-as-a-service. Your favourite cloud brands these instances as “Favourite DB instances”. Under the hood they’re powered by the open source FancySQL engine. We’ll name the new managed resource kind FavouriteDBInstance.

The first step toward implementing a new managed service is to define the code level schema of its configuration resources. These are referred to as resources, (resource) kinds, and objects interchangeably. The kubebuilder scaffolding is a good starting point for any new Crossplane API kind.

Note that while Crossplane was originally derived from kubebuilder scaffolds its patterns have diverged somewhat. It is possible to use kubebuilder to scaffold a resource, but the author must be careful to adapt said resource to Crossplane patterns. It may often be quicker to copy and modify a v1beta1 or above resource from the same provider repository, rather than using kubebuilder.

1kubebuilder create api \
2    --group example --version v1alpha1 --kind FavouriteDBInstance \
3    --resource=true --controller=false --namespaced=false

The above command should produce a scaffold similar to the below example:

 1type FavouriteDBInstanceSpec struct {
 2    // INSERT ADDITIONAL SPEC FIELDS - desired state of infrastructure
 3    // Important: Run "make" to regenerate code after modifying this file
 4}
 5
 6// FavouriteDBInstanceStatus defines the observed state of FavouriteDBInstance
 7type FavouriteDBInstanceStatus struct {
 8    // INSERT ADDITIONAL STATUS FIELD - define observed state of infrastructure
 9    // Important: Run "make" to regenerate code after modifying this file
10}
11
12// +kubebuilder:object:root=true
13
14// FavouriteDBInstance is the Schema for the favouritedbinstance API
15// +kubebuilder:resource:scope=Cluster
16type FavouriteDBInstance struct {
17    metav1.TypeMeta   `json:",inline"`
18    metav1.ObjectMeta `json:"metadata,omitempty"`
19
20    Spec   FavouriteDBInstanceeSpec  `json:"spec,omitempty"`
21    Status FavouriteDBInstanceStatus `json:"status,omitempty"`
22}

Crossplane requires that these newly generated API type scaffolds be extended with a set of struct fields, getters, and setters that are standard to all Crossplane resource kinds. The getters and setter methods required to satisfy crossplane-runtime interfaces are omitted from the below examples for brevity. They can be added by hand, but new services are encouraged to use angryjet to generate them automatically using a //go:generate comment per the angryjet documentation.

Note that in many cases a suitable provider will already exist. Frequently adding support for a new managed service requires only the definition of the managed resource itself.

Managed Resource Kinds

Managed resources must:

The Parameters struct should be a high fidelity representation of the writeable fields of the external resource’s API. Put otherwise, if your favourite cloud represents Favourite DB instances as a JSON object then FavouriteDBParameters should marshal to a something as close to that JSON object as possible while still complying with Kubernetes API conventions.

For example, assume the external API object for Favourite DB instance was:

1{
2    "id": 42,
3    "name": "mycoolinstance",
4    "fanciness_level": 100,
5    "version": "2.3",
6    "status": "ONLINE",
7    "hostname": "cool.fcp.example.org"
8}

Further assume the id, status, and hostname fields were output only, and the version field was optional. The FavouriteDBInstance managed resource should look as follows:

 1// FavouriteDBInstanceParameters define the desired state of an FavouriteDB
 2// instance. Most fields map directly to an Instance:
 3// https://favourite.example.org/api/v1/db#Instance
 4type FavouriteDBInstanceParameters struct {
 5
 6    // We're still working on a standard for naming external resources. See
 7    // https://github.com/crossplane/crossplane/issues/624 for context.
 8
 9    // Name of this instance.
10    Name string `json:"name"`
11
12    // Note that fanciness_level becomes fancinessLevel below. Kubernetes API
13    // conventions trump cloud provider fidelity.
14
15    // FancinessLevel specifies exactly how fancy this instance is.
16    FancinessLevel int `json:"fancinessLevel"`
17
18    // Version specifies what version of FancySQL this instance will run.
19    // +optional
20    Version *string `json:"version,omitempty"`
21}
22
23// A FavouriteDBInstanceSpec defines the desired state of a FavouriteDBInstance.
24type FavouriteDBInstanceSpec struct {
25    xpv1.ResourceSpec  `json:",inline"`
26    ForProvider FavouriteDBInstanceParameters `json:"forProvider"`
27}
28
29// A FavouriteDBInstanceStatus represents the observed state of a
30// FavouriteDBInstance.
31type FavouriteDBInstanceStatus struct {
32    xpv1.ResourceStatus `json:",inline"`
33
34    // Note that we add the three "output only" fields here in the status,
35    // instead of the parameters. We want this representation to be high
36    // fidelity just like the parameters.
37
38    // ID of this instance.
39    ID int `json:"id,omitempty"`
40
41    // Status of this instance.
42    Status string `json:"status,omitempty"`
43
44    // Hostname of this instance.
45    Hostname string `json:"hostname,omitempty"`
46}
47
48// A FavouriteDBInstance is a managed resource that represents a Favourite DB
49// instance.
50// +kubebuilder:subresource:status
51type FavouriteDBInstance struct {
52    metav1.TypeMeta   `json:",inline"`
53    metav1.ObjectMeta `json:"metadata,omitempty"`
54
55    Spec   FavouriteDBInstanceSpec   `json:"spec"`
56    Status FavouriteDBInstanceStatus `json:"status,omitempty"`
57}

Note that Crossplane uses the GoDoc strings of API kinds to generate user facing API documentation. Document all fields and prefer GoDoc that assumes the reader is running kubectl explain, or reading an API reference, not reading the code. Refer to the Managed Resource API Patterns one pager for more detail on authoring high fidelity managed resources.

Provider Kinds

You’ll typically only need to add a new Provider kind if you’re creating an infrastructure provider that adds support for a new infrastructure provider.

Providers must:

  • Be named exactly ProviderConfig.
  • Embed a ProviderSpec struct in their Spec struct.
  • Use the +kubebuilder:resource:scope=Cluster comment marker.

The Favourite Cloud ProviderConfig would look as follows. Note that the cloud to which it belongs should be indicated by its API group, i.e. its API Version would be favouritecloud.crossplane.io/v1alpha1 or similar.

 1// A ProviderSpec defines the desired state of a Provider.
 2type ProviderSpec struct {
 3    xpv1.ProviderSpec `json:",inline"`
 4
 5    // Information required outside of the Secret referenced in the embedded
 6    // xpv1.ProviderSpec that is required to authenticate to the provider.
 7    // ProjectID is used as an example here.
 8    ProjectID string `json:"projectID"`
 9}
10
11// A Provider configures a Favourite Cloud 'provider', i.e. a connection to a
12// particular Favourite Cloud project using a particular Favourite Cloud service
13// account.
14type Provider struct {
15    metav1.TypeMeta   `json:",inline"`
16    metav1.ObjectMeta `json:"metadata,omitempty"`
17
18    Spec ProviderSpec `json:"spec"`
19}

Finishing Touches

At this point we’ve defined the managed resource necessary to start building controllers. Before moving on to the controllers:

  • Add any kubebuilder comment markers that may be useful for your resource. Comment markers can be used to validate input, or add additional columns to the standard kubectl get output, among other things.
  • Run make reviewable to generate Custom Resource Definitions and additional helper methods for your new resource kinds.
  • Make sure any package documentation (i.e. // Package v1alpha1... GoDoc, including package level comment markers) are in a file named doc.go. kubebuilder adds them to groupversion_info.go, but several code generation tools only check doc.go.

Finally, add convenience GroupVersionKind variables for each new resource kind. These are typically added to either register.go or groupversion_info.go depending on which version of kubebuilder scaffolded the API type:

1// FavouriteDBInstance type metadata.
2var (
3    FavouriteDBInstanceKind             = reflect.TypeOf(FavouriteDBInstance{}).Name()
4    FavouriteDBInstanceKindAPIVersion   = FavouriteDBInstanceKind + "." + GroupVersion.String()
5    FavouriteDBInstanceGroupVersionKind = GroupVersion.WithKind(FavouriteDBInstanceKind)
6)

Consider opening a draft pull request and asking a Crossplane maintainer for review before you start work on the controller!

Adding Controllers

Crossplane controllers, like those scaffolded by kubebuilder, are built around the controller-runtime library. controller-runtime flavoured controllers encapsulate most of their domain-specific logic in a reconcile.Reconciler implementation. Most Crossplane controllers are one of the three kinds mentioned under What Makes a Crossplane Infrastructure Resource. Each of these controller kinds are similar enough across implementations that crossplane-runtime provides ‘default’ reconcilers. These reconcilers encode what the Crossplane community has learned about managing external systems and narrow the problem space from reconciling a Kubernetes resource kind with an arbitrary system down to Crossplane-specific tasks.

crossplane-runtime provides the following reconcile.Reconcilers:

  • The managed.Reconciler reconciles managed resources with external systems by instantiating a client of the external API and using it to create, update, or delete the external resource as necessary.

Crossplane controllers typically differ sufficiently from those scaffolded by kubebuilder that there is little value in using kubebuilder to generate a controller scaffold.

Managed Resource Controllers

Managed resource controllers should use managed.NewReconciler to wrap a managed-resource specific implementation of managed.ExternalConnecter. Parts of managed.Reconciler’s behaviour is customisable; refer to the managed.NewReconciler GoDoc for a list of options. The following is an example controller for the FavouriteDBInstance managed resource we defined earlier:

  1import (
  2    "context"
  3    "fmt"
  4    "strings"
  5
  6    "github.com/pkg/errors"
  7    corev1 "k8s.io/api/core/v1"
  8    "k8s.io/apimachinery/pkg/types"
  9    ctrl "sigs.k8s.io/controller-runtime"
 10    "sigs.k8s.io/controller-runtime/pkg/client"
 11
 12    // An API client of the hypothetical FavouriteDB service.
 13    "github.com/fcp-sdk/v1/services/database"
 14
 15    xpv1 "github.com/crossplane/crossplane-runtime/apis/common/v1"
 16    "github.com/crossplane/crossplane-runtime/pkg/meta"
 17    "github.com/crossplane/crossplane-runtime/pkg/resource"
 18    "github.com/crossplane/crossplane-runtime/pkg/reconciler/managed"
 19
 20    "github.com/crossplane/provider-fcp/apis/database/v1alpha3"
 21    fcpv1alpha3 "github.com/crossplane/provider-fcp/apis/v1alpha3"
 22)
 23
 24type FavouriteDBInstanceController struct{}
 25
 26// SetupWithManager instantiates a new controller using a managed.Reconciler
 27// configured to reconcile FavouriteDBInstances using an ExternalClient produced by
 28// connecter, which satisfies the ExternalConnecter interface.
 29func (c *FavouriteDBInstanceController) SetupWithManager(mgr ctrl.Manager) error {
 30    return ctrl.NewControllerManagedBy(mgr).
 31        Named(strings.ToLower(fmt.Sprintf("%s.%s", v1alpha3.FavouriteDBInstanceKind, v1alpha3.Group))).
 32        For(&v1alpha3.FavouriteDBInstance{}).
 33        Complete(managed.NewReconciler(mgr,
 34            resource.ManagedKind(v1alpha3.FavouriteDBInstanceGroupVersionKind),
 35            managed.WithExternalConnecter(&connecter{client: mgr.GetClient()})))
 36}
 37
 38// Connecter satisfies the resource.ExternalConnecter interface.
 39type connecter struct{ client client.Client }
 40
 41// Connect to the supplied resource.Managed (presumed to be a
 42// FavouriteDBInstance) by using the Provider it references to create a new
 43// database client.
 44func (c *connecter) Connect(ctx context.Context, mg resource.Managed) (managed.ExternalClient, error) {
 45    // Assert that resource.Managed we were passed in fact contains a
 46    // FavouriteDBInstance. We told NewControllerManagedBy that this was a
 47    // controller For FavouriteDBInstance, so something would have to go
 48    // horribly wrong for us to encounter another type.
 49    i, ok := mg.(*v1alpha3.FavouriteDBInstance)
 50    if !ok {
 51        return nil, errors.New("managed resource is not a FavouriteDBInstance")
 52    }
 53
 54    // Get the Provider referenced by the FavouriteDBInstance.
 55    p := &fcpv1alpha3.Provider{}
 56    if err := c.client.Get(ctx, meta.NamespacedNameOf(i.Spec.ProviderReference), p); err != nil {
 57        return nil, errors.Wrap(err, "cannot get Provider")
 58    }
 59
 60    // Get the Secret referenced by the Provider.
 61    s := &corev1.Secret{}
 62    n := types.NamespacedName{Namespace: p.Namespace, Name: p.Spec.Secret.Name}
 63    if err := c.client.Get(ctx, n, s); err != nil {
 64        return nil, errors.Wrap(err, "cannot get Provider secret")
 65    }
 66
 67    // Create and return a new database client using the credentials read from
 68    // our Provider's Secret.
 69    client, err := database.NewClient(ctx, s.Data[p.Spec.Secret.Key])
 70    return &external{client: client}, errors.Wrap(err, "cannot create client")
 71}
 72
 73// External satisfies the resource.ExternalClient interface.
 74type external struct{ client database.Client }
 75
 76// Observe the existing external resource, if any. The managed.Reconciler
 77// calls Observe in order to determine whether an external resource needs to be
 78// created, updated, or deleted.
 79func (e *external) Observe(ctx context.Context, mg resource.Managed) (managed.ExternalObservation, error) {
 80    i, ok := mg.(*v1alpha3.FavouriteDBInstance)
 81    if !ok {
 82        return managed.ExternalObservation{}, errors.New("managed resource is not a FavouriteDBInstance")
 83    }
 84
 85    // Use our FavouriteDB API client to get an up to date view of the external
 86    // resource.
 87    existing, err := e.client.GetInstance(ctx, i.Spec.Name)
 88
 89    // If we encounter an error indicating the external resource does not exist
 90    // we want to let the managed.Reconciler know so it can create it.
 91    if database.IsNotFound(err) {
 92        return managed.ExternalObservation{ResourceExists: false}, nil
 93    }
 94
 95    // Any other errors are wrapped (as is good Go practice) and returned to the
 96    // managed.Reconciler. It will update the "Synced" status condition
 97    // of the managed resource to reflect that the most recent reconcile failed
 98    // and ensure the reconcile is reattempted after a brief wait.
 99    if err != nil {
100        return managed.ExternalObservation{}, errors.Wrap(err, "cannot get instance")
101    }
102
103    // The external resource exists. Copy any output-only fields to their
104    // corresponding entries in our status field.
105    i.Status.Status = existing.GetStatus()
106    i.Status.Hostname = existing.GetHostname()
107    i.Status.ID = existing.GetID()
108
109    // Update our "Ready" status condition to reflect the status of the external
110    // resource. Most managed resources use the below well known reasons that
111    // the "Ready" status may be true or false, but managed resource authors
112    // are welcome to define and use their own.
113    switch i.Status.Status {
114    case database.StatusOnline:
115        resource.SetBindable(i)
116        i.SetConditions(xpv1.Available())
117    case database.StatusCreating:
118        i.SetConditions(xpv1.Creating())
119    case database.StatusDeleting:
120        i.SetConditions(xpv1.Deleting())
121    }
122
123    // Finally, we report what we know about the external resource. In this
124    // hypothetical case FancinessLevel is the only field that can be updated
125    // after creation time, so the resource does not need to be updated if
126    // the actual fanciness level matches our desired fanciness level. Any
127    // ConnectionDetails we return will be published to the managed resource's
128    // connection secret if it specified one.
129    o := managed.ExternalObservation{
130        ResourceExists:   true,
131        ResourceUpToDate: existing.GetFancinessLevel == i.Spec.FancinessLevel,
132        ConnectionDetails: managed.ConnectionDetails{
133            xpv1.ResourceCredentialsSecretUserKey:     []byte(existing.GetUsername()),
134            xpv1.ResourceCredentialsSecretEndpointKey: []byte(existing.GetHostname()),
135        },
136    }
137
138    return o, nil
139}
140
141// Create a new external resource based on the specification of our managed
142// resource. managed.Reconciler only calls Create if Observe reported
143// that the external resource did not exist.
144func (e *external) Create(ctx context.Context, mg resource.Managed) (managed.ExternalCreation, error) {
145    i, ok := mg.(*v1alpha3.FavouriteDBInstance)
146    if !ok {
147        return managed.ExternalCreation{}, errors.New("managed resource is not a FavouriteDBInstance")
148    }
149    // Indicate that we're about to create the instance. Remember ExternalClient
150    // authors can use a bespoke condition reason here in cases where Creating
151    // doesn't make sense.
152    i.SetConditions(xpv1.Creating())
153
154    // Create must return any connection details that are set or returned only
155    // at creation time. The managed.Reconciler will merge any details
156    // with those returned during the Observe phase.
157    password := database.GeneratePassword()
158    cd := managed.ConnectionDetails{xpv1.ResourceCredentialsSecretPasswordKey: []byte(password)}
159
160    // Create a new instance.
161    new := database.Instance{Name: i.Name, FancinessLevel: i.FancinessLevel, Version: i.Version}
162    err := e.client.CreateInstance(ctx, new, password)
163
164    // Note that we use resource.Ignore to squash any error that indicates the
165    // external resource already exists. Create implementations must not return
166    // an error if asked to create a resource that already exists. Real managed
167    // resource controllers are advised to avoid unintentially 'adoptign' an
168    // existing, unrelated external resource, per
169    // https://github.com/crossplane/crossplane-runtime/issues/27
170    return managed.ExternalCreation{ConnectionDetails: cd}, errors.Wrap(resource.Ignore(database.IsExists, err), "cannot create instance")
171}
172
173// Update the existing external resource to match the specifications of our
174// managed resource. managed.Reconciler only calls Update if Observe
175// reported that the external resource was not up to date.
176func (e *external) Update(ctx context.Context, mg resource.Managed) (managed.ExternalUpdate, error) {
177    i, ok := mg.(*v1alpha3.FavouriteDBInstance)
178    if !ok {
179        return managed.ExternalUpdate{}, errors.New("managed resource is not a FavouriteDBInstance")
180    }
181
182    // Recall that FancinessLevel is the only field that we _can_ update.
183    new := database.Instance{Name: i.Name, FancinessLevel: i.FancinessLevel}
184    err := e.client.UpdateInstance(ctx, new)
185    return managed.ExternalUpdate{}, errors.Wrap(err, "cannot update instance")
186}
187
188// Delete the external resource. managed.Reconciler only calls Delete
189// when a managed resource with the 'Delete' deletion policy (the default) has
190// been deleted.
191func (e *external) Delete(ctx context.Context, mg resource.Managed) error {
192    i, ok := mg.(*v1alpha3.FavouriteDBInstance)
193    if !ok {
194        return errors.New("managed resource is not a FavouriteDBInstance")
195    }
196    // Indicate that we're about to delete the instance.
197    i.SetConditions(xpv1.Deleting())
198
199    // Delete the instance.
200    err := e.client.DeleteInstance(ctx, i.Spec.Name)
201
202    // Note that we use resource.Ignore to squash any error that indicates the
203    // external resource does not exist. Delete implementations must not return
204    // an error when asked to delete a non-existent external resource.
205    return errors.Wrap(resource.Ignore(database.IsNotFound, err), "cannot delete instance")
206}

Wrapping Up

Once all your controllers are in place you’ll want to test them. Note that most projects under the crossplane org favor table driven tests that use Go’s standard library testing package over kubebuilder’s Gingko based tests. Please do not add or proliferate Gingko based tests.

Finally, don’t forget to plumb any newly added resource kinds and controllers up to your controller manager. Simple providers may do this for each type within within main(), but most more complicated providers take an approach in which each package exposes an AddToScheme (for resource kinds) or SetupWithManager (for controllers) function that invokes the same function within its child packages, resulting in a main.go like:

 1import (
 2    "time"
 3
 4    "sigs.k8s.io/controller-runtime/pkg/client/config"
 5    "sigs.k8s.io/controller-runtime/pkg/manager"
 6    "sigs.k8s.io/controller-runtime/pkg/manager/signals"
 7
 8    crossplaneapis "github.com/crossplane/crossplane/apis"
 9
10    fcpapis "github.com/crossplane/provider-fcp/apis"
11    "github.com/crossplane/provider-fcp/pkg/controller"
12)
13
14func main() {
15    cfg, err := config.GetConfig()
16    if err != nil {
17        panic(err)
18    }
19
20    mgr, err := manager.New(cfg, manager.Options{SyncPeriod: 1 * time.Hour})
21    if err != nil {
22        panic(err)
23    }
24
25    if err := crossplaneapis.AddToScheme(mgr.GetScheme()); err != nil {
26        panic(err)
27    }
28
29    if err := fcpapis.AddToScheme(mgr.GetScheme()); err != nil {
30        panic(err)
31    }
32
33    if err := controller.SetupWithManager(mgr); err != nil {
34        panic(err)
35    }
36
37    panic(mgr.Start(signals.SetupSignalHandler()))
38}

In Review

In this guide we walked through the process of defining the resource kinds and controllers necessary to build support for new managed infrastructure; possibly even a completely new infrastructure provider. Please do not hesitate to reach out to the Crossplane maintainers and community for help designing and implementing support for new managed services. We would highly value any feedback you may have about the development process!