open-feature / flagd-testbed Goto Github PK

Two of the rules in evaluation.feature have repeating "then"s.

For example:

Then then the resolved string response should be "INTERNAL"

instead of

Then the resolved string response should be "INTERNAL"

We inject a few properties. We should have some simple gherkin tests (probably added here) which assert that:

• $flagd.flagKey and $flagd.timestamp can be used as expected in rules

The current evaluation.feature suite does a good job covering the basic behavior concerning providers / evaluation API, but things like context merging precedence and hook behavior aren't really tested here.

We should create a new suite that tests some of the above.

We should add some flags here and associated tests here for common edge cases and mis-configurations, including:

• targeting rule returns null should fall through to defaultVariant and return reason STATIC
• invalid/erroneous JSON logic (should error, and cause use of application (SDK) default)
• targeting returns a variant which is not in variant definition (should error, and cause use of application (SDK) default)

We need a test suite that can be used to test events in all implementing SDKs. Just like our existing suites, it should use the flagd-testbed docker image. It must implement tests for the PROVIDER_READY and PROVIDER_CONFIGURATION_CHANGED events. To simplify things, the container might have one flag which is changed by a simple bash script at a fixed interval automatically. This would allow us to easily subscribe to flag changes without the need to share a volume with the container:

{
  "flags": {
    "changing-flag": {
      "state": "ENABLED",
      "variants": {
        "flip": "FLIP",
        "flop": "FLOP"
      },
      "defaultVariant": "flip" // this value will change every 3 seconds
    }
  }
}

Definition of done:

• new gherkin suite testing events happy path
• supporting changes in flagd-testbed container

Blocked by:

• flagd providers updated to use events

As caching is being implemented in flagd providers it'd be good to have integration tests for asserting the behaviour is as expected when wired up.

Consider e2e test failures in Go SDK testing ¹. This seems to be caused by the sync server and we need to investigate the root cause.

AC

Improve the sync server so that e2e tests become stable

Currently, the test-harness builds, tags as latest and pushes the flagd-testbed image.
This practice has caused PR workflow processes to fail due to not absorbing test changes required by a later update of flagd.
While it is a good thing to be notified that the tests are failing against an upgraded flagd, having it apparent in unrelated PRs is cumbersome and confusing.

Imo, a better practice would be to version the flagd-testbed image (same version as the base flagd image) and introduce renovate config into the relevant repos to check for flagd-testbed image upgrades. This way any test failures against an upgraded flagd is only apparent in the renovate PR and it's immediately clear to see what's going wrong.

• Version flagd-testbed image
• Implement renovate config that watches test-harness version in go-sdk
• Implement renovate config that watches test-harness version in java-sdk
• Implement renovate config that watches test-harness version in js-sdk

flagd releasing ought to trigger an upgrade of the flagd testbed image to avoid the integration tests running against a stale image.

flagd itself now implements the sync.proto. This means the sync-server in this repo (only used for testing) can be replaced with flagd itself.

• update https://github.com/open-feature/flagd-testbed/blob/main/sync/Dockerfile and https://github.com/open-feature/flagd-testbed/blob/main/sync/Dockerfile.unstable to make use of flagd instead of the small implementation here
• delete https://github.com/open-feature/flagd-testbed/tree/main/sync/pkg

Currently, the suite asserts that a flag has been removed from cache but not that other flags are still present in cache.
This means an implementation could hypothetically be purging all cache on update of a flag, this ought to be checked in tests.

s/gerkin/gherkin/

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update golang.org/x/exp digest to fc45aab
• chore(deps): update ghcr.io/open-feature/flagd docker tag to v0.10.3
• fix(deps): update module buf.build/gen/go/open-feature/flagd/protocolbuffers/go to v1.34.2-20240215170432-1e611e2999cc.2
• fix(deps): update module buf.build/gen/go/open-feature/flagd/grpc/go to v1.4.0-20240215170432-1e611e2999cc.1
• chore(deps): update google-github-actions/release-please-action action to v4
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Currently, a flagd upgrade is classed as a chore(deps), therefore not considered enough for a release.
In order for the integration tests to run against the latest flagd a release of test-harness is needed for each flagd release.
Automate this process.