Difference in handling apply_manifest(pp, :expect_failures => true) about beaker

[electrical]

did some checking together with 'ashp' on IRC.
Puppet 2.7 and 3.1 are returning exit code 2. Puppet 3.2 and up returning exit code 6.
Any idea's on resolving this?

[electrical]

Solved it for now by limiting tests based on the puppetversion fact.
Im guessing that's the only way for now?

[anodelman]

From the apply_manifest code:
if opts[:catch_changes]
args << '--detailed-exitcodes'

      # We're after idempotency so allow exit code 0 only.
      on_options[:acceptable_exit_codes] |= [0]
    elsif opts[:catch_failures]
      args << '--detailed-exitcodes'

      # We're after only complete success so allow exit codes 0 and 2 only.
      on_options[:acceptable_exit_codes] |= [0, 2]
    elsif opts[:expect_failures]
      args << '--detailed-exitcodes'

      # We're after failures specifically so allow exit codes 1, 4, and 6 only.
      on_options[:acceptable_exit_codes] |= [1, 4, 6]
    elsif opts[:expect_changes]
      args << '--detailed-exitcodes'

      # We're after changes specifically so allow exit code 2 only.
      on_options[:acceptable_exit_codes] |= [2]
    else
      # Either use the provided acceptable_exit_codes or default to [0]
      on_options[:acceptable_exit_codes] |= [0]
    end

From our code, a return value on failure is 1,4,6 while a return value of 2 indicates a change was made to the machine configuration.