Testing and Debugging

Overview

A Resource Plugin runs only when a sandbox asks for the resource it provides, so developing one is a loop of publishing a version, creating a sandbox that uses it, and reading the logs of the steps that ran. This page describes that loop and where to look when a step fails.

What apply validates

signadot resourceplugin apply validates the plugin specification when it is submitted. The following are rejected at apply time:

• a malformed spec, or a runner.podTemplateOverlay that does not parse as a pod template
• an overlay that conflicts with runner.image or runner.namespace, sets a restartPolicy other than Never, or uses a reserved container name (see the spec reference)
• a step that consumes an output no step produces, or a cycle of valueFromStep inputs between steps
• re-publishing a (name, version) pair that already exists

Nothing in a step script is executed at apply time. There is no --dry-run mode for resourceplugin apply; a plugin is exercised by creating a sandbox that uses it.

Iterating on a plugin

Published plugin versions are immutable, so a second apply of the same name: is rejected:

Error: resource plugin "my-plugin" default version (0.0.0) already exists; versions are immutable — add an @<semver> suffix to publish a new revision (e.g. my-plugin@0.0.1)

While developing, bump the @<semver> suffix on each revision and have the test sandbox refer to the version under test explicitly, so that fixing the plugin never means deleting the version a working sandbox is pinned to.

# plugin.yaml
name: my-plugin@0.0.3
spec:
  runner:
    image: alpine:3
  create:
  - name: provision
    script: "@{embed:./provision.sh}"

# sandbox.yaml — declares only the resource, no forked workloads
name: my-plugin-test
spec:
  cluster: "@{cluster}"
  resources:
  - name: test
    plugin: my-plugin@0.0.3

signadot resourceplugin apply -f plugin.yaml
signadot sandbox apply -f sandbox.yaml --set cluster=my-cluster

A sandbox that declares a resource and nothing else is enough to run the create workflow, and deleting it runs the delete workflow. Bare plugin: my-plugin resolves to the highest published version at sandbox-creation time; see Versioning.

Step scripts are ordinary shell, so @{embed: <file>} lets you keep them in separate files and iterate with shellcheck or by running them locally against the same image (docker run --rm -it <runner.image> sh) before publishing. See the CLI templating reference.

How steps run

Each step in create and delete runs in its own pod, created from runner.image and runner.podTemplateOverlay in runner.namespace. Steps that do not consume each other's outputs are scheduled in parallel; a step that has a valueFromStep input waits for the producing step to complete. Nothing else is shared between steps: a file written in one step is not visible to the next unless it is exported as an output and consumed as an input. If a workflow needs to fetch something once and use it in several commands, do that inside a single step rather than across several.

A pod's restartPolicy is always Never. A step whose script exits non-zero is marked Failed, the resource is marked failed, and the sandbox reports NotReady. Failed steps are not retried; to re-run the workflow, fix the plugin (publishing a new version), then delete and re-create the sandbox.

Pods are left in place after they finish, whether they succeeded or failed, until Kubernetes' terminated-pod garbage collection removes them or the sandbox is deleted. This is what makes the logs available for debugging.

Finding out why a step failed

Sandbox status

signadot sandbox get <name> reports which resource failed, but not which step:

Status:   Not Ready (ResourceFailed: Resource "test" has failed)

The step and its output are available through Signadot without cluster access, as described next. The kubectl views further down show the same information from inside the cluster.

Step logs

signadot logs reads the logs of a resource's step pods using the names in the sandbox spec. Without --step it returns every create step; --step selects a single step from either workflow:

signadot logs --sandbox my-plugin-test --resource test
signadot logs --sandbox my-plugin-test --resource test --step provision

The step's script runs in the container named main, and its output is what usually explains a failure. Each step pod also contains Signadot's input and output containers (signadot-execunit-input-manager, signadot-execunit-output-manager); their logs are only relevant if inputs were not injected or outputs were not exported, for example because valueFromPath named a file the script never wrote. When a pod has more than one container each line is prefixed with the container name; pass -c main to see only the script's output.

The same logs are in the dashboard: open the sandbox, choose the Logs tab, and select the resource and step.

Two conditions return an error instead of logs:

• no logs available for step "<step>" ... it has no recorded pod yet: the step's pod does not exist yet. If the sandbox has stayed Not Ready with PendingResourceCreation for longer than the steps should take, see Operator logs.
• 403 ... pod log access is disabled for cluster ...: the operator in that cluster was installed with controlPlane.podLogAccess.enabled: false, which withholds pod log access from Signadot. Use kubectl as described below.

Sandbox logs need v1.8.0 or later of the CLI; see the signadot logs reference for all flags.

Runner pod logs with kubectl

Every step pod carries two labels: signadot.com/olm-name, the name of the in-cluster object that tracks the resource's lifecycle (of the form <sandbox-name>-<resource-name>-<hash>), and signadot.com/olm-workunit-name, the step name from the plugin spec. To list the pods for a sandbox's resources in the runner namespace:

kubectl get pods -n <runner-namespace> -l signadot.com/olm-name \
  -L signadot.com/olm-name,signadot.com/olm-workunit-name | grep <sandbox-name>

Pod names have the form <olm-name>-<step-name>-<hash>, with the leading component truncated when it is long. Then read the step's output from its main container:

kubectl logs -n <runner-namespace> <pod> -c main

Per-step status

The operator tracks each resource's workflows in a cluster-scoped SignadotObjectLifecycleMethods object (short name solm), named after the resource as described above. Its status lists every step of the create workflow (preStart) and the delete workflow (postStop) with a phase and the pod that ran it:

kubectl get solm | grep <sandbox-name>
kubectl get solm <name> -o yaml

status:
  phase: Failed
  failedPhase: PreStart
  preStart:
  - name: provision
    phase: Failed
    execUnit:
      name: <sandbox-name>-test-<hash>-provision-<hash>
      namespace: default

phase: Failed on a step points at the pod whose logs explain the failure. Steps still in Pending were waiting on a failed or unfinished step's outputs.

Operator logs

If no pod was created at all, the reason is usually in the operator's controller manager, for example an overlay that admission control rejected or a runner namespace it cannot create pods in. This is the only case on this page that needs access to the operator's namespace rather than the runner namespace:

kubectl logs -n <operator-namespace> deploy/<controller-manager-deployment> -c manager --since=1h \
  | grep <sandbox-name>

Common failures

Symptom	Likely cause
apply fails with already exists; versions are immutable	Re-publishing a version. Bump the @<semver> suffix.
Sandbox stays Not Ready with PendingResourceCreation	A step is still running, or is Pending behind a slow step it depends on. Check pod status.
Step Failed, its main logs show a command not found	runner.image lacks the tool. Use an image that has it or install it at the top of the script.
Step Failed, main succeeded but signadot-execunit-output-manager logged an error	An outputs[_].valueFromPath file was not written, or was written to a different path.
Step Failed with a permissions error from kubectl/a cloud CLI in the script	The runner pod has no service account or credentials. Add serviceAccountName or a secret mount via runner.podTemplateOverlay.
Step Failed on delete, sandbox stuck deleting	The delete workflow's script errors. Read its logs with signadot logs ... --step <delete-step>; once the cause is understood, signadot sandbox delete --force removes the sandbox without running the workflow.
signadot logs reports the step has no recorded pod yet	No pod was created. Check operator logs; the pod spec produced from the overlay may have been rejected by the cluster.