Skip to content

Add semantic unit tests for CEL rules#14878

Draft
Vincent056 wants to merge 2 commits into
ComplianceAsCode:masterfrom
Vincent056:cmp-4484-cel-unit-tests
Draft

Add semantic unit tests for CEL rules#14878
Vincent056 wants to merge 2 commits into
ComplianceAsCode:masterfrom
Vincent056:cmp-4484-cel-unit-tests

Conversation

@Vincent056

Copy link
Copy Markdown
Contributor

Description

CEL rules (applications/**/cel/shared.yml) had no semantic unit tests: nothing verified that an expression returns the expected PASS/FAIL for given inputs. Two of the five shipped OpenShift Virtualization rules carry bugs that such tests would have caught (the nonroot rule false-fails on OCP 4.18+, and the overcommit rule iterates a list input without .items so it always errors).

This PR adds:

  • Per-rule fixtures at <rule-dir>/cel/tests/cases.yaml{name, expect, inputs} cases plus a provenance block (fetch date, OpenShift/Kubernetes versions, the apiVersion actually served — versions only, nothing cluster-identifiable). Fixtures were seeded from real API objects on an OCP 4.22 + CNV cluster with celctl cac scaffold --from-cluster, so the resource shapes match what the API actually serves (schemas drift across versions: HyperConverged featureGates is an object on v1beta1 but an array on v1; nonRoot was removed in 4.18+).
  • tests/unit/kubernetes/test_cel_rules.py, which enforces that every CEL rule ships fixtures covering at least one compliant and one non-compliant case (or is allowlisted with a reason), lints each rule, and evaluates its fixtures through celctl — cel-go with the Compliance Operator scanner's environment (parseJSON/parseYAML, operator binding semantics, no such key mapping to FAIL), so results here match the operator.
  • known_broken_cel_rules.yml allowlist — one entry: kubevirt-no-vms-overcommitting-guest-memory (the .items bug, tracked in CMP-4451; its fix will remove the entry and add fixtures). Allowlist hygiene is itself tested.
  • CMake registration as python-unit-kubernetes (label quick), so ctest runs it in the existing gates.

celctl-dependent tests skip when the binary is absent (CELCTL env var or PATH), so contributors without Go are not blocked — the fixture-presence policy still runs. CI can enable full evaluation with one line: go install github.com/Vincent056/cel-rule-skill/celctl@v0.1.4 (follow-up; happy to add it to the gate workflow here if preferred).

Edge-case fixture expectations encode current rule behavior and reference their tracking issues in the case names (CMP-4447, CMP-4448, CMP-4449, CMP-4450).

Rationale

Unit-testing CEL in a different engine than production gives false confidence; evaluating through cel-go + the scanner environment means a rule that passes here behaves the same in the Compliance Operator. Authoring flow for new rules: celctl cac scaffold <rule-dir> --from-cluster, fill the TODO cases, celctl cac test <rule-dir>.

Review Hints

  • With celctl on PATH: python3 -m pytest tests/unit/kubernetes/test_cel_rules.py -v — 18 passed, 4 skipped (the allowlisted rule).
  • Without celctl: 10 passed, 12 skipped (policy checks still enforced).
  • Via the build system: ctest -R python-unit-kubernetes --output-on-failure.

Jira: https://issues.redhat.com/browse/CMP-4484

🤖 Generated with Claude Code

CEL rules had no semantic unit tests: nothing verified that an expression
returns the expected PASS/FAIL for given inputs, and two of the five shipped
OpenShift Virtualization rules carry bugs that such tests would have caught.

Add per-rule fixtures at <rule-dir>/cel/tests/cases.yaml ({name, expect,
inputs} cases with a provenance block recording fetch date, OpenShift and
Kubernetes versions, and the apiVersion actually served - versions only,
nothing cluster-identifiable; seeded from real API objects on an OCP 4.22 +
CNV cluster via 'celctl cac scaffold --from-cluster').

Add tests/unit/kubernetes/test_cel_rules.py, which:
- requires every CEL rule to ship fixtures covering at least one compliant
  and one non-compliant case, unless allowlisted with a reason in
  known_broken_cel_rules.yml (allowlist hygiene is itself tested)
- lints each rule and evaluates its fixtures through celctl, which uses
  cel-go and the scanner's environment (parseJSON/parseYAML, operator
  binding semantics), so results match the Compliance Operator
- requires a provenance block on fixture files
- skips the celctl-dependent tests when celctl is absent (CELCTL env var or
  PATH), so contributors without Go are not blocked; CI installs it with
  'go install github.com/Vincent056/cel-rule-skill/celctl@<version>'

Register the suite in CMake as python-unit-kubernetes (label quick) so ctest
runs it in the existing gates.

kubevirt-no-vms-overcommitting-guest-memory is allowlisted: its expression
iterates the list input without .items so evaluation always errors; the fix
and its fixtures are tracked in CMP-4451. The fixture expectations for the
nonroot (CMP-4448), persistent-reservation (CMP-4449), host-devices
(CMP-4447) and tls-registries (CMP-4450) edge cases encode current behavior
and reference the tracking issues in the case names.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Jul 14, 2026
@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Used by openshift-ci bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant