Watch Operations

This is an alpha feature. Crossplane may change or drop this feature at any time.

This feature was introduced in v2.

For more information read the Crossplane feature lifecycle.

This document is for an unreleased version of Crossplane.

This document applies to the Crossplane master branch and not to the latest release v2.0.

A WatchOperation creates Operations when watched Kubernetes resources change. Use WatchOperations for reactive operational workflows such as backing up databases before deletion, validating configurations after updates, or triggering alerts when resources fail.

How WatchOperations work

WatchOperations watch specific Kubernetes resources and create new Operations whenever those resources change. The changed resource is automatically injected into the Operation for the function to process.

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: config-validator
 5spec:
 6  watch:
 7    apiVersion: v1
 8    kind: ConfigMap
 9    matchLabels:
10      validate: "true"
11  concurrencyPolicy: Allow
12  operationTemplate:
13    spec:
14      mode: Pipeline
15      pipeline:
16      - step: validate
17        functionRef:
18          name: function-config-validator
19        input:
20          apiVersion: fn.crossplane.io/v1beta1
21          kind: ConfigValidatorInput
22          rules:
23          - required: ["database.url", "database.port"]
24          - format: "email"
25            field: "notification.email"
26      - step: notify
27        functionRef:
28          name: function-slack-notifier
29        input:
30          apiVersion: fn.crossplane.io/v1beta1
31          kind: SlackNotifierInput
32          channel: "#alerts"
33          severity: "warning"

Important

WatchOperations are an alpha feature. You must enable Operations by adding --enable-operations to Crossplane’s arguments.

Key features

Watches any Kubernetes resource type - Not limited to Crossplane resources
Supports namespace and label filtering - Target specific resources
Automatically injects changed resources - Functions receive the triggering resource
Configurable concurrency policies - Control operation creation

Resource watching

WatchOperations can watch any Kubernetes resource with flexible filtering:

Watch all resources of a type

1spec:
2  watch:
3    apiVersion: apps/v1
4    kind: Deployment

Watch resources in a specific namespace

1spec:
2  watch:
3    apiVersion: v1
4    kind: ConfigMap
5    namespace: production

Watch resources with specific labels

1spec:
2  watch:
3    apiVersion: example.org/v1
4    kind: Database
5    matchLabels:
6      backup: "enabled"
7      environment: "production"

Watch cluster-scoped resources

1spec:
2  watch:
3    apiVersion: v1
4    kind: Node
5    matchLabels:
6      node-role.kubernetes.io/worker: ""

Resource injection

When a WatchOperation creates an Operation, it automatically injects the changed resource using the special requirement name ops.crossplane.io/watched-resource. Functions can access this resource without explicitly requesting it.

For example, when a ConfigMap with label validate: "true" changes, the WatchOperation creates an Operation like this:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: config-validator-abc123
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: validate
 9    functionRef:
10      name: function-config-validator
11    requirements:
12      requiredResources:
13      - requirementName: ops.crossplane.io/watched-resource
14        apiVersion: v1
15        kind: ConfigMap
16        name: my-config
17        namespace: default
18    # ... other pipeline steps from operationTemplate

The watched resource is automatically available to functions in req.required_resources under the special name ops.crossplane.io/watched-resource.

Concurrency policies

WatchOperations support the same concurrency policies as CronOperations:

Allow (default): Multiple Operations can run simultaneously. Use this when operations don’t interfere with each other.
Forbid: New Operations don’t start if previous ones are still running. Use this for operations that can’t run concurrently.
Replace: New Operations stop running ones before starting. Use this when you always want the latest operation to run.

Common use cases

Note

The following examples use hypothetical functions for illustration. At launch, only function-python supports operations.

Configuration validation

Validate ConfigMaps when they change:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: config-validator
 5spec:
 6  watch:
 7    apiVersion: v1
 8    kind: ConfigMap
 9    matchLabels:
10      validate: "true"
11  operationTemplate:
12    spec:
13      mode: Pipeline
14      pipeline:
15      - step: validate-config
16        functionRef:
17          name: function-config-validator
18        input:
19          apiVersion: fn.crossplane.io/v1beta1
20          kind: ConfigValidatorInput
21          rules:
22          - required: ["database.host", "database.port"]
23          - format: "email"
24            field: "notification.email"

Database backup on deletion

Backup databases before they’re deleted:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: backup-on-deletion
 5spec:
 6  watch:
 7    apiVersion: rds.aws.m.upbound.io/v1beta1
 8    kind: Instance
 9    # Note: Watching for deletion requires function logic
10    # to check deletion timestamp
11  operationTemplate:
12    spec:
13      mode: Pipeline
14      pipeline:
15      - step: create-backup
16        functionRef:
17          name: function-rds-backup
18        input:
19          apiVersion: fn.crossplane.io/v1beta1
20          kind: RDSBackupInput
21          retentionDays: 30

Resource failure alerting

Alert when resources enter a failed state:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: failure-alerts
 5spec:
 6  watch:
 7    apiVersion: example.org/v1
 8    kind: App
 9    matchLabels:
10      alert: "enabled"
11  operationTemplate:
12    spec:
13      mode: Pipeline
14      pipeline:
15      - step: check-status
16        functionRef:
17          name: function-status-checker
18        input:
19          apiVersion: fn.crossplane.io/v1beta1
20          kind: StatusCheckerInput
21          alertConditions:
22          - type: "Ready"
23            status: "False"
24      - step: send-alert
25        functionRef:
26          name: function-alertmanager
27        input:
28          apiVersion: fn.crossplane.io/v1beta1
29          kind: AlertInput
30          severity: "critical"

Advanced configuration

Advanced watch patterns

Complex resource watching with multiple conditions:

 1# Watch Deployments in specific namespaces with multiple label conditions
 2apiVersion: ops.crossplane.io/v1alpha1
 3kind: WatchOperation
 4metadata:
 5  name: multi-condition-watcher
 6spec:
 7  watch:
 8    apiVersion: apps/v1
 9    kind: Deployment
10    namespace: production  # Only production namespace
11    matchLabels:
12      app.kubernetes.io/managed-by: "crossplane"
13      environment: "prod"
14      backup-required: "true"
15  operationTemplate:
16    spec:
17      mode: Pipeline
18      pipeline:
19      - step: backup-deployment
20        functionRef:
21          name: function-deployment-backup

 1# Watch custom resources across all namespaces
 2apiVersion: ops.crossplane.io/v1alpha1
 3kind: WatchOperation
 4metadata:
 5  name: database-lifecycle-manager
 6spec:
 7  watch:
 8    apiVersion: database.example.io/v1
 9    kind: PostgreSQLInstance
10    # No namespace specified = watch all namespaces
11    matchLabels:
12      lifecycle-management: "enabled"
13  operationTemplate:
14    spec:
15      mode: Pipeline
16      pipeline:
17      - step: lifecycle-check
18        functionRef:
19          name: function-database-lifecycle
20        input:
21          apiVersion: fn.crossplane.io/v1beta1
22          kind: DatabaseLifecycleInput
23          checkDeletionTimestamp: true
24          autoBackup: true

Cross-resource workflows

WatchOperations can watch one resource type and dynamically fetch related resources. Here’s a WatchOperation that watches Ingresses and manages certificates:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: ingress-certificate-manager
 5spec:
 6  watch:
 7    apiVersion: networking.k8s.io/v1
 8    kind: Ingress
 9    matchLabels:
10      auto-cert: "enabled"
11  operationTemplate:
12    spec:
13      mode: Pipeline
14      pipeline:
15      - step: manage-certificates
16        functionRef:
17          name: function-cert-manager
18        input:
19          apiVersion: fn.crossplane.io/v1beta1
20          kind: CertManagerInput
21          issuer: "letsencrypt-prod"
22          renewBefore: "720h"  # 30 days

The function examines the watched Ingress and dynamically requests related resources:

 1from crossplane.function import request, response
 2
 3def operate(req, rsp):
 4    # Access the watched Ingress resource
 5    ingress = request.get_required_resource(req, "ops.crossplane.io/watched-resource")
 6    if not ingress:
 7        response.fatal(rsp, "No watched resource found")
 8        return
 9    
10    # Extract the service name from the Ingress backend
11    rules = ingress.get("spec", {}).get("rules", [])
12    if not rules:
13        response.fatal(rsp, "Could not extract service name from ingress")
14        return
15        
16    backend = rules[0].get("http", {}).get("paths", [{}])[0].get("backend", {})
17    service_name = backend.get("service", {}).get("name")
18    if not service_name:
19        response.fatal(rsp, "Could not extract service name from ingress")
20        return
21        
22    ingress_namespace = ingress.get("metadata", {}).get("namespace", "default")
23    
24    # CRITICAL: Always request the same resources to ensure requirement
25    # stabilization. Crossplane calls the function repeatedly until 
26    # requirements don't change.
27    response.require_resources(
28        rsp, 
29        name="related-service",
30        api_version="v1",
31        kind="Service",
32        match_name=service_name,
33        namespace=ingress_namespace
34    )
35    
36    # Check if the service is available and process accordingly
37    service = request.get_required_resource(req, "related-service")
38    if service:
39        # Success: Both resources available
40        response.set_output(rsp, {
41            "status": "success",
42            "message": "Certificate management completed",
43            "ingress_host": ingress.get("spec", {}).get("rules", [{}])[0].get("host"),
44            "service_name": service.get("metadata", {}).get("name")
45        })
46        return
47        
48    # Waiting: Service not available yet
49    response.set_output(rsp, {
50        "status": "waiting", 
51        "message": f"Waiting for service '{service_name}' to be available"
52    })

Important

Critical resource stabilization pattern: functions must return the same requirements in each iteration to signal completion. The function in the preceding example always calls response.require_resources() regardless of whether the service exists. This ensures Crossplane knows when to stop calling the function.

Common mistake: only requesting resources when missing breaks the stabilization contract and causes timeout errors.

This pattern allows functions to:

Examine the watched resource (injected automatically)
Dynamically determine what other resources the function needs
Request those resources consistently using response.require_resources()
Process all resources when available, or provide status when waiting

Status and monitoring

WatchOperations provide status information about watching:

 1status:
 2  conditions:
 3  - type: Synced
 4    status: "True"
 5    reason: ReconcileSuccess
 6  - type: Watching
 7    status: "True"
 8    reason: WatchActive
 9  watchingResources: 12
10  runningOperationRefs:
11  - name: config-validator-anjda
12  - name: config-validator-f0d92

Key status fields:

Conditions: Standard Crossplane conditions (Synced) and WatchOperation-specific conditions:
- Watching: True when the WatchOperation is actively watching resources, False when paused or failed
watchingResources: Number of resources under watch
runningOperationRefs: Running Operations created by this WatchOperation

Events

WatchOperations emit events for important activities:

EstablishWatched (Warning) - Watch establishment failures
TerminateWatched (Warning) - Watch termination failures
GarbageCollectOperations (Warning) - Operation cleanup failures
CreateOperation (Warning) - Operation creation failures
ReplaceRunningOperation (Warning) - Operation replacement failures

Monitoring

Monitor WatchOperations using:

 1# Check WatchOperation status
 2kubectl get watchoperation my-watchop
 3
 4# View recent Operations created by the WatchOperation
 5kubectl get operations -l crossplane.io/watchoperation=my-watchop
 6
 7# Check watched resource count
 8kubectl describe watchoperation my-watchop
 9
10# Check events
11kubectl get events --field-selector involvedObject.name=my-watchop

Best practices

Resource selection

Use specific label selectors - Prevent unnecessary Operations with precise filtering
Avoid high-churn resources - Be careful watching frequently changing resources
Start small - Begin with narrow selectors and expand as needed

Event handling

Implement event filtering - Check generation, deletion timestamp, and status conditions to avoid processing irrelevant changes
Monitor operation volume - Popular resources can create numerous Operations

Concurrency policies

Choose appropriate concurrency policies:
- Allow for independent processing that can run in parallel
- Forbid for operations that must complete before processing new changes
- Replace for status-checking or monitoring where only latest state matters

History management

Like CronOperations, WatchOperations automatically clean up completed Operations:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: config-validator
 5spec:
 6  watch:
 7    apiVersion: v1
 8    kind: ConfigMap
 9  successfulHistoryLimit: 10  # Keep 10 successful Operations (default: 3)
10  failedHistoryLimit: 5       # Keep 5 failed Operations (default: 1)
11  operationTemplate:
12    # Operation template here

Watched resource injection

WatchOperations automatically inject the changed resource into the created Operation using a special requirement name ops.crossplane.io/watched-resource:

 1from crossplane.function import request, response
 2
 3def operate(req, rsp):
 4    # Access the resource that triggered this Operation
 5    watched_resource = request.get_required_resource(req, "ops.crossplane.io/watched-resource")
 6    if not watched_resource:
 7        response.set_output(rsp, {"error": "No watched resource found"})
 8        return
 9    
10    # Process based on the watched resource
11    if watched_resource["kind"] == "ConfigMap":
12        config_data = watched_resource["data"]
13        # Validate configuration...

The watched resource is available in the function’s required_resources map without needing to declare it in the Operation template.

For general Operations best practices including function development and operational considerations, see Operation best practices.

Troubleshooting

WatchOperation not creating Operations

Verify the WatchOperation has Watching=True condition
Check that watched resources exist and match the selector
Ensure resources are actually changing
Look for events indicating watch establishment failures

Too many Operations created

Refine label selectors to match fewer resources
Consider using Forbid or Replace concurrency policy
Check if resources are changing more frequently than expected
Review function logic to ensure it’s not causing resource updates

Operations failing to process watched resources

Verify function capabilities include operation
Check that functions handle the ops.crossplane.io/watched-resource
Review function logs for processing errors
Ensure functions can handle the specific resource types under watch

Next steps

Learn about Operation for one-time operational tasks
Learn about CronOperation for scheduled operations
Get started with Operations to create your first reactive operation