Using Severity Levels

[sapritchard]

I wanted to make use of severity levels within the custom rules I'm writing so a pipeline can make a decision at the end of the run to fail, pass or pass with warnings. Obviously using -Level Error works as I expect it to. Using a value of Warning is what I'm not understanding.
When my rule with a Warning level fails it still shows as FAIL on the console and also within a result variable.
I do get the output at the end WARNING: One or more rules reported failure.

My question is how to leverage this programmatically as I would have thought in a result variable the test wouldn't be a failure but in a Warning result so you could look for it. In this example my Assert-PSRule is using a ResultVariable of validation and I can look in the variable to make a decision:

if (($validation | Where-Object {$_.Outcome -eq 'Fail'}).Count -eq 0) {
    # Success
}
else {
    # Failed tests
}

Rule example:

Rule 'common.found.schema' -Ref 'vaf.config.0002' -Level Warning {
    $Assert.HasField($TargetObject, '$schema')
}

Output example:

    [PASS] common.valid.json.configuration (vaf.config.0001)
    [FAIL] common.found.schema (vaf.config.0002)
    | File: rule_module/test_configs/cluster.version_3_0_0.json:2:125

    | REASON:
    | - Path $schema: Does not exist.

    [PASS] common.found.version (vaf.config.0003)

I've referenced - testing-infrastructure

In the end I'm trying to replicate behavior I would have used with Pester to set a test result as Inconclusive to generate that soft failure in a validation.

[BernieWhite]

Hi @sapritchard. Thanks for the question. I'm not sure I fully understand the how you are integrating with Pester, but let me try to answer the question. Additionally see: https://microsoft.github.io/PSRule/v2/scenarios/validation-pipeline/validation-pipeline/#calling-from-pester

If the object did not pass the condition the rule outcome is Fail regardless of the level. It can be Error if there was an error generated by the rule with Write-Error/ or a bug in the rule that caused an unexpected error.

There are a few other properties that would be helpful for your case.

• The Level property on the result shows is the level of the rule, which can be combined with Outcome. i.e. the rule failed and it was a warning level rule.
• The OutcomeReason property would normally be Processed but could be Inconclusive if the rule didn't return a pass or fail for some reason such as a bug.

---

The original intent of the rule level is to allow breaking tests based on severity. In PSRule v2, PSRule will generate an error/ warning/ information event to allow the pipeline or tests to terminate. Error being the normal cases when the pipeline would break.

This is more configurable in PSRule v3 which allows configuration for breaking the pipeline at a lower level like warning.

However this functionality is mostly for the purpose of Assert-PSRule. If you are wrapping in Pester you can read the properties directly and make your own logic.

---

Currently the level is not output specifically in the text output of Assert-PSRule although if it something that useful we can look at adding it.

[sapritchard]

Ok thanks @BernieWhite appreciate your response and I'll look into using those properties to see if I can achieve what I'm after.

Apologies for the confusion around Pester. My current rule testing engine so to speak is based around Pester and it's handy to allow a configuration through the validation cycle if something that isn't going to prevent a build is just flagged instead of hard failing.