Enable QIT to run performance tests with k6 locally.

[alopezari]

This PR introduces the initial performance testing infrastructure to QIT, enabling developers to run k6-based performance tests against WooCommerce extensions. The feature integrates with QIT's existing local testing infrastructure just like activation tests do. So far it runs a set of dummy tests and generate a simple HTML report over which we will iterate in upcoming tasks.

The implementation uses k6 running in Docker containers to perform HTTP-based performance tests against WordPress/WooCommerce sites with extensions installed. Tests can be configured with different load patterns, thresholds, and scenarios to validate extension performance under various conditions.

This implementation was built with QIT-CLI v1.0 in mind, making sure it is self contained for easier integration once we release v1.0. This is the main reason why you will see that this implementation uses the QIT_ENVIRONMENT_TYPE env variable for figuring out if we're running E2E or Performance environment, for example. This should make it easier to migrate the code to the new architecture once v1.0 is out.

Changes added in this PR

Core Infrastructure

• New Command: Added qit run:performance command to execute performance tests.
• Performance Environment: Created PerformanceEnvironment and PerformanceEnvInfo classes with performance-specific optimizations.
• K6 Integration: Implemented K6Runner and K6DockerConfig for k6 test execution and Docker container management.
• Test Management: Added PerformanceTestManager for orchestrating test execution and result collection.

Other non self-contained changes

• Bootstrap Integration: Updated bootstrap.php to register the new RunPerformanceTestCommand with the application.
• Environment Type Detection: Modified EnvConfigLoader.php to detect performance environment type via QIT_ENVIRONMENT_TYPE environment variable.
• Base Environment Factory: Enhanced EnvInfo.php to support performance environment instantiation and domain configuration for performance tests.
• Theme Activation Support: Extended ThemeActivation.php to handle both E2E and Performance environments with unified theme activation logic.
• Notification System Integration: Updated LocalTestRunNotifier.php to:
  • Handle performance test notifications and lifecycle events.
  • Support performance test result processing and status determination.
  • Process k6 exit codes for performance test failure detection.
• Docker Infrastructure: Added k6/Dockerfile with grafana/k6 base image and QIT helper integration for k6 test execution.
• Cross-Environment Compatibility: Minor updates to E2EEnvInfo.php to ensure compatibility with the expanded environment system.

Testing instructions

1. Make sure you're using a Compatibility Dashboard version that contains the changes from this PR and you're running it locally.
2. Regarding QIT-CLI, checkout to this branch and make sure you've connected it to the local environment with qit switch local.
3. Run the default performance test with qit run:performance automatewoo (you can use any plugin you'd like as long as it's available in your local environment).
4. Ensure the test passes.
5. Ensure the test is now found in the Test Runs page with test type = performance and status success.
6. Now tweak the thresholds to make them smaller in order to make the tests fail, for example:

        'http_req_duration': ['p(95)<2000'], // 95th percentile under 2 seconds
        'http_req_duration{expected_response:true}': ['avg<1000'], // Average response time under 1 second for successful requests
        'http_req_failed': ['rate<0.1'], // Error rate under 10%
        'checks': ['rate>0.9'], // At least 90% of checks should pass

7. Run the performance tests again with qit run:performance automatewoo.
8. Ensure the test has failed now because the threshold condition isn't met.
9. Verify the you can see the test in the Test Runs page with test type = performance and status failed.
10. Test around this feature (play with different arguments from qit run:performance, etc).
11. Make sure the activation tests keep working as usual (`qit run:activation automatewoo).

[zhongruige]

Just to confirm this point @alopezari:

you're running it locally

Does this mean we don't expect this to work in staging?

[alopezari]

Does this mean we don't expect this to work in staging?

Good question! It should work if you connect the QIT-CLI to staging and deploy this PR on staging. Haven't tested it though (I tend to work locally whenever possible, but that's a good call).

[zhongruige]

Thanks for confirming! Wanted to check since I tried to run this on staging but the test got stuck in dispatched: https://stagingcompatibilitydashboard.wpcomstaging.com/wp-admin/post.php?post=1407831&action=edit

Doesn't look like it got picked up by an action workflow, so we might need to add that piece in (hadn't dug into the code changes yet).

[alopezari]

Yes! The CD PR includes the changes for communicating the local test run with the CD so, without it, we won't get any test updates in the CD.

[zhongruige]

Hmm okay looks like it may not be quite working, since that PR is on staging and I pointed my CLI to it as well:

CI Runners are updated with the latest remote commit from the current checked out branch.
Deploy done.
➜  compatibility-dashboard git:(add/new-performance-test-type)

qit backend:current
staging

[alopezari]

Hmmm thanks for checking @zhongruige!

I'll take a look after the call 👍

[alopezari]

Hi @zhongruige!

It worked on my local pointing to staging:

• Before running the performance tests:

id	test_type	cd_status	created_at
44856	activation	running		2025-07-14 15:07:19

• Right after running the tests:

id	test_type	cd_status	created_at
44857	performance	running		2025-07-14 15:09:19

• Once the performance tests finished running locally:

id	test_type	cd_status	created_at
44857	performance	success		2025-07-14 15:09:19

Out of curiosity, did you pull the latest change from 24a4007?

[zhongruige]

Hm should have been the latest but I'll give it a go on it!

[zhongruige]

Ah sorry @alopezari I realized what the issue was, I needed to build the binary to include these changes. Usually we'll build the binary as part of a PR introducing changes to the CLI, here's an example from a recent PR: https://github.com/woocommerce/qit-cli/pull/316/files

I did get it to start, but then I hit this error:

➜  qit-cli git:(add/perf-tests-infra) ✗ ./qit run:performance automatewoo

[...]

No specific performance tests configured. Running default performance test.
Performance test failed: Default performance test file not found: phar:///qit-cli/qit/src/LocalTests/Performance/Runner/../tests/default-performance.k6.js

[alopezari]

Usually we'll build the binary as part of a PR introducing changes to the CLI

Oh TIL, thanks for letting me know!

I did get it to start, but then I hit this error

Hmm interesting, it should run this file:
https://github.com/woocommerce/qit-cli/pull/315/files#diff-c6e8a1970b3f966d0f6ce88ede4dd08b5792dc399af5d19c19611992194ca89e

Let me see if I can reproduce the error

[alopezari]

Hi @zhongruige!

The issue was that the default test file wasn't bundled with the QIT binary and therefore QIT couldn't find that file when running directly the binary. I didn't spot this because I tend to run the QIT with php src/qit-cli.php.

I've updated the K6Runner class to generate the file dynamically (see 3580884). This is a temporary file until we create a proper test management system with proper tags, etc, similar to how custom tests work. This will be handled in QIT-707: Implement performance tests for WooCommerce Core.

[zhongruige]

Nice, thanks @alopezari! Works great for me now:

HTML report looks good too:

I did notice these do store in my local home/woo-qit-cli directory, is that expected at this point?

[alopezari]

Good catch!

I've updated it in 9255e2d.

Now it will store the artifacts in the var path, just like activation tests do:

✓ k6 performance test passed

Artifacts saved to: /var/folders/j8/1djrt0mj0p56ph_dlrxghbz00000gn/T/qit-results-6876083fec883
HTML report: /var/folders/j8/1djrt0mj0p56ph_dlrxghbz00000gn/T/qit-results-6876083fec883/report.html
Average response time: 71.86ms

[zhongruige]

Also been getting some weird environment issues, it's possible mine was a bit messed up from the previous implementation but using QIT commands is giving me this all the time now:

qit switch prod


 [ERROR] Invalid environment type.

[alopezari]

Ah yes! I also found this yesterday after switching to staging, but I thought it was a glitch on my end.

I had to delete and clone again the project, couldn't find out why that happened 🤔

My guess is that it happened after switching from an environment where performance tests were defined (i.e. staging) to one where performance tests weren't available (staging after switching back to trunk when the CD PR wasn't merged yet and therefore the performance test type wasn't available). Do you think this was the case on your end?

[alopezari]

Hi @zhongruige!

Also been getting some weird environment issues, it's possible mine was a bit messed up from the previous implementation but using QIT commands is giving me this all the time now:

I thought that was a glitch but it was actually caused by this PR. We've always had one environment type (e2e) and now we added a 2nd one (performance). Every time we run an activation or performance test, some data is persisted in the QIT cache (~/woo-qit-cli/.env-local.json). That caused that this error appeared if you ran a performance test, the data was persisted there and then if you switched to a branch or version that didn't contain that performance type.

I've fixed this in 434f472, in order to fallback to e2e environment if the cache ever stores an unknown environment. That should unblock this flow.

As a consequence, you might see these warnings when reproducing the issue to check that it was fixed:

<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.
<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.
<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.
<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.
<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.
<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.
<warning>Warning: Unknown environment type "performance" found in cache. Falling back to "e2e" environment type.</warning>
Warning: Key "test_tag" not found in environment info.

However, that's non-blocking, and I think it makes sense to show them in order to flag there is weird data in the cache. The reason why you will see it repeated multiple times it's because these are flagged by the environment_monitor and it's run multiple times during the execution.

[alopezari]

Ah yes! Bear in mind that if you try to replicate the issue to see if the fix works, it will likely fail unless you include the changes from 434f472 in the branch you switch to.

[alopezari]

Adding @MrJnrman and @Luc45 to the review in case they'd like to take a peak, in order to be able to merge this PR, which is blocking other work.

[MrJnrman]

Great job here @alopezari! Tested and worked fine both locally and in staging. I see the point of this PR is to create the infra to be able to run locally, so I'll approve based on that notion instead of diving too deep into every bit of the code.

[zhongruige]

Went through this again and it's working great for me, excited to see this foundation work in place!