Operations

An Operation runs a function pipeline once to completion to perform operational tasks that don’t fit the typical resource creation pattern. Unlike compositions that continuously reconcile desired state, Operations focus on tasks like backups, rolling upgrades, configuration validation, and scheduled maintenance.

How operations work

Operations are like Kubernetes Jobs - they run once to completion rather than continuously reconciling. Like compositions, Operations use function pipelines to implement their logic, but they’re designed for operational workflows instead of resource composition.

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: backup-database
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: create-backup
 9    functionRef:
10      name: function-database-backup
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: DatabaseBackupInput
14      database: production-db
15      retentionDays: 30

When you create this Operation, Crossplane:

1. Validates the operation and its function dependencies
2. Executes the function pipeline step by step
3. Applies any resources the functions create or change
4. Updates the Operation status with results and completion state

Key characteristics

• Runs once to completion (like Kubernetes Jobs)
• Uses function pipelines (like Compositions)
• Can create or change any Kubernetes resources
• Provides detailed status and output from each step
• Supports retry on failure with configurable limits

Operation functions vs composition functions

Operations and compositions both use function pipelines, but with important differences:

Composition Functions:

• Purpose: Create and maintain resources
• Lifecycle: Continuous reconciliation
• Input: Observed composite resources
• Output: Desired composed resources
• Ownership: Creates owner references

Operation Functions:

• Purpose: Perform operational tasks
• Lifecycle: Run once to completion
• Input: Required resources only
• Output: Any Kubernetes resources
• Ownership: Force applies without owners

Functions can support both modes by declaring the appropriate capabilities in their package metadata. Function authors declare this in the crossplane.yaml file when building the function package:

1apiVersion: meta.pkg.crossplane.io/v1
2kind: Function
3metadata:
4  name: my-function
5spec:
6  capabilities:
7  - composition
8  - operation

This allows Crossplane to know which modes the function supports and avoid trying to use a composition-only function for operations.

Common use cases

Rolling upgrades

Use Operations for controlled rolling upgrades:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: cluster-upgrade
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: rolling-upgrade
 9    functionRef:
10      name: function-cluster-upgrade
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: ClusterUpgradeInput
14      targetVersion: "1.28"
15      batches: [0.25, 0.5, 1.0]  # 25%, 50%, then 100%
16      healthChecks: [Synced, Ready]

One-time maintenance

Use Operations for specific maintenance tasks:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: certificate-rotation
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: rotate-certificates
 9    functionRef:
10      name: function-cert-rotation
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: CertRotationInput
14      targetCertificates:
15        matchLabels:
16          rotate: "true"

Advanced configuration

Retry behavior

Operations automatically retry when they fail. Configure the retry limit to control how often attempts occur:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: resilient-operation
 5spec:
 6  retryLimit: 10  # Try up to 10 times before giving up (default: 5)
 7  mode: Pipeline
 8  pipeline:
 9  - step: flaky-task
10    functionRef:
11      name: function-flaky-task
12    input:
13      apiVersion: fn.crossplane.io/v1beta1
14      kind: FlakyTaskInput
15      # Task that might fail due to temporary issues
16      timeout: "30s"

Retry behavior:

• Each retry resets the entire pipeline - if step 2 of 3 fails, the retry starts from step 1
• Operations use exponential backoff: 1 s, 2 s, 4 s, 8 s, 16 s, 32 s, then 60 s max
• Operations track the number of failures in status.failures
• After reaching retryLimit, the Operation becomes Succeeded=False

Credentials

Operations can provide credentials to functions through Secrets:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: secure-backup
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: backup-with-credentials
 9    functionRef:
10      name: function-backup
11    credentials:
12    - name: backup-creds
13      source: Secret
14      secretRef:
15        namespace: crossplane-system
16        name: backup-credentials
17        key: api-key
18    - name: database-creds
19      source: Secret
20      secretRef:
21        namespace: crossplane-system
22        name: database-credentials
23        key: connection-string
24    input:
25      apiVersion: fn.crossplane.io/v1beta1
26      kind: BackupInput
27      destination: s3://my-backup-bucket

Multiple pipeline steps

Complex operations can use multiple pipeline steps:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: multi-step-deployment
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: validate-config
 9    functionRef:
10      name: function-validator
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: ValidatorInput
14      configName: app-config
15  - step: backup-current
16    functionRef:
17      name: function-backup
18    input:
19      apiVersion: fn.crossplane.io/v1beta1
20      kind: BackupInput
21      target: current-deployment
22  - step: deploy-new-version
23    functionRef:
24      name: function-deploy
25    input:
26      apiVersion: fn.crossplane.io/v1beta1
27      kind: DeployInput
28      image: myapp:v2.0.0
29      strategy: rollingUpdate
30  - step: verify-health
31    functionRef:
32      name: function-health-check
33    input:
34      apiVersion: fn.crossplane.io/v1beta1
35      kind: HealthCheckInput
36      timeout: 300s
37      healthEndpoint: /health

RBAC permissions

If your Operation needs to access resources that Crossplane doesn’t have permissions for by default, create a ClusterRole that aggregates to Crossplane:

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: ClusterRole
 3metadata:
 4  name: operation-additional-permissions
 5  labels:
 6    rbac.crossplane.io/aggregate-to-crossplane: "true"
 7rules:
 8# Additional permissions for Operations
 9- apiGroups: ["networking.k8s.io"]
10  resources: ["ingresses"]
11  verbs: ["get", "list", "patch", "update"]
12- apiGroups: [""]
13  resources: ["persistentvolumes"]
14  verbs: ["get", "list"]
15# Add other resources your Operations need to access

This ClusterRole is automatically aggregated to Crossplane’s main ClusterRole, giving the Crossplane service account the permissions needed for your Operations.

Function response cache

Operations can use function response caching to improve performance for operations that:

• Call the same functions often with identical inputs
• Use functions that perform expensive computations or external API calls
• Run frequently through CronOperation or WatchOperation

The cache works the same way as for Compositions - function responses with time to live values cache and reuse identical requests until they expire.

Function response caching helps Operations that:

• Validate configurations using expensive checks
• Query external systems for status information
• Perform complex calculations that don’t change frequently

For cache configuration details, see the Function response cache documentation.

Required resources

Operations can preload resources for functions to access:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: resource-aware-operation
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: process-deployment
 9    functionRef:
10      name: function-processor
11    requirements:
12      requiredResources:
13      - requirementName: app-deployment
14        apiVersion: apps/v1
15        kind: Deployment
16        name: my-app
17        namespace: production
18      - requirementName: app-service
19        apiVersion: v1
20        kind: Service
21        name: my-app-service
22        namespace: production
23    input:
24      apiVersion: fn.crossplane.io/v1beta1
25      kind: ProcessorInput
26      action: upgrade

Functions access these resources through the standard request structure:

 1from crossplane.function import request, response
 2
 3def operate(req, rsp):
 4    # Access required resources
 5    deployment = request.get_required_resource(req, "app-deployment")
 6    service = request.get_required_resource(req, "app-service")
 7    
 8    if not deployment or not service:
 9        response.set_output(rsp, {"error": "Required resources not found"})
10        return
11    
12    # Process the resources
13    new_replicas = deployment["spec"]["replicas"] * 2
14    
15    # Return updated resources with full GVK and metadata for server-side apply
16    rsp.desired.resources["app-deployment"].resource.update({
17        "apiVersion": "apps/v1",
18        "kind": "Deployment",
19        "metadata": {
20            "name": deployment["metadata"]["name"],
21            "namespace": deployment["metadata"]["namespace"]
22        },
23        "spec": {"replicas": new_replicas}
24    })

Status and monitoring

Operations provide rich status information:

 1status:
 2  conditions:
 3  - type: Synced
 4    status: "True"
 5    reason: ReconcileSuccess
 6  - type: Succeeded
 7    status: "True"
 8    reason: PipelineSuccess
 9  - type: ValidPipeline
10    status: "True"
11    reason: ValidPipeline
12  failures: 1  # Number of retry attempts
13  pipeline:
14  - step: create-backup
15    output:
16      backupId: "backup-20240115-103000"
17      size: "2.3GB"
18  appliedResourceRefs:
19  - apiVersion: "v1"
20    kind: "Secret"
21    namespace: "production"
22    name: "backup-secret"
23  - apiVersion: "apps/v1" 
24    kind: "Deployment"
25    name: "updated-deployment"

Key status fields:

• conditions: Standard Crossplane conditions (Synced) and Operation-specific conditions:
  • Succeeded: True when the operation completed successfully, False when it failed
  • ValidPipeline: True when all functions have the required operation capability
• failures: Number of times the operation has failed and retried
• pipeline: Output from each function step for tracking progress
• appliedResourceRefs: References to all resources the Operation created or modified

Events

Operations emit Kubernetes events for important activities:

• Function run results and warnings
• Resource apply failures
• Operation lifecycle events (creation, completion, failure)

Troubleshooting operations

Check operation status:

1kubectl get operation my-operation -o wide

View detailed information:

1kubectl describe operation my-operation

Common failure scenarios:

1. Operations do nothing - Operations feature not enabled:

   1# Operation exists but has no status conditions and never progresses
   2status: {}
   

   Solution: enable Operations by adding --enable-operations to Crossplane’s startup arguments.

2. ValidPipeline condition is False - Function doesn’t support operations:

   1conditions:
   2- type: ValidPipeline
   3  status: "False"
   4  reason: InvalidFunctionCapability
   5  message: "Function function-name doesn't support operations"
   

   Solution: use a function that declares operation capability.

3. Succeeded condition is False - Function run failed:

   1conditions:
   2- type: Succeeded
   3  status: "False" 
   4  reason: PipelineFailure
   5  message: "Function returned error: connection timeout"
   

   Solution: view function logs and fix the underlying issue.

4. Resource apply failures - View events for details:

   1kubectl get events --field-selector involvedObject.name=my-operation
   

Debug function runs:

1# View function logs
2kubectl logs -n crossplane-system deployment/function-python
3
4# Check operation events
5kubectl get events --field-selector involvedObject.kind=Operation
6
7# Inspect operation status in detail
8kubectl get operation my-operation -o jsonpath='{.status.pipeline}' | jq '.'

Resource management

Operations can create or change any Kubernetes resources using server-side apply with force ownership. This means:

What Operations can do:

• Create new resources of any kind
• Change existing resources by taking ownership of specific fields
• Apply changes that may conflict with other controllers

What Operations can’t do:

• Delete resources (current limitation of alpha implementation)
• Establish owner references (resources aren’t garbage collected)
• Continuously maintain desired state (they run once)

Test an operation

You can preview the output of any Operation using the Crossplane CLI. You don’t need a Crossplane control plane to do this. The Crossplane CLI uses Docker Engine to run functions.

Provide an operation, composition functions, and any required resources to render the output locally.

1crossplane alpha render op operation.yaml functions.yaml --required-resources=ingress.yaml

crossplane alpha render op prints the Operation status and any resources the operation functions created or modified. It shows what would happen if you applied the Operation to a cluster.

 1---
 2# Operation status showing function results
 3apiVersion: ops.crossplane.io/v1alpha1
 4kind: Operation
 5metadata:
 6  name: ingress-cert-monitor
 7status:
 8  conditions:
 9  - type: Succeeded
10    status: "True"
11    reason: PipelineSuccess
12  pipeline:
13  - step: check-ingress-certificate
14    output:
15      certificateExpires: "Sep 29 08:34:02 2025 GMT"
16      daysUntilExpiry: 53
17      hostname: google.com
18      ingressName: example-app
19      status: ok
20---
21# Modified Ingress resource with certificate annotations
22apiVersion: networking.k8s.io/v1
23kind: Ingress
24metadata:
25  annotations:
26    cert-monitor.crossplane.io/expires: Sep 29 08:34:02 2025 GMT
27    cert-monitor.crossplane.io/days-until-expiry: "53"
28    cert-monitor.crossplane.io/status: ok
29  name: example-app
30  namespace: default
31spec:
32  # ... ingress spec unchanged

Use --required-resources to provide resources that your operation functions need access to. You can specify multiple files or use glob patterns:

1# Multiple specific files
2crossplane alpha render op operation.yaml functions.yaml \
3  --required-resources=deployment.yaml,service.yaml,configmap.yaml
4
5# Glob pattern for all YAML files in a directory
6crossplane alpha render op operation.yaml functions.yaml \
7  --required-resources="resources/*.yaml"

Best practices

Operation-specific practices

1. Plan for rollback - Design operations to be reversible when possible, because Operations don’t auto rollback like Compositions
2. Make operations idempotent - Operations should be safe to retry if they fail partway through
3. Use required resources - Prepopulate functions with needed resources for efficiency rather than requesting them during running

Function development

1. Declare capabilities - Explicitly declare operation capability in function metadata to enable Operations support
2. Return meaningful output - Use the output field to track what the operation accomplished for monitoring and debugging

Next steps

• Get started with Operations to create your first Operation
• Learn about CronOperation for scheduled operations
• Learn about WatchOperation for reactive operations