TEST SUITES

Going forward, all CartePlus testing will be categorized into the following three “suites”:

Unit Test Suite

Tests individual components in isolation, uses mocks for any dependencies.

  • Will be run as part of the deployment pipeline (deployments will be blocked if any tests fail).
  • Should be runnable in local dev environment.
  • Should run to completion in a couple of seconds.
  • Misc requirements
    • Use a standard test framework supported by github actions.
    • During CI, the test results should be attached to the workflow run (using actions/upload-artifact action).

E2E (End to End) Test Suite

Full system tests (but uses Emulate3D instead of the actual bots, uses auto-acknowledged operator actions instead of operator UI or PTLs).

  • Will be scheduled and run automatically in a dedicated environment (say E3D) by our new test harness.
  • Will run completely unattended, with no manual intervention.
  • Will run to completion in a couple of minutes.

Because of the sheer number of deployed microservices needed, these E2E tests cannot be run locally on a dev machine. Also not runnable in regular CI/CD (GitHub hosted runner) as it’ll heavily consume minutes. However a self-hosted GitHub runner is a viable option as it’s free, and there is no usage cap.

Exploratory Test Suite

E2E tests, but with variations in algorithms, and configurations.

The CartePlus* environment variables will be used to direct the config/algo changes.

This test suite can potentially house the benchmarking tests (which are executed using Emulate3D).

❌ DISCARDED SUITES

We considered, but ultimately rejected these suite categories:

❌ Integration Test Suite

Tests integration between two components

There may not be much value to these if we manage to run E2E (full system tests) frequently.

❌ Fault Injection / Chaos Test Suite

This is not a test suite per se, but more of a test mode.

We can just run E2E tests with fault-mode turned on: CartePlusFaultInjection=1. As such, it can be considered a part of the exploratory test suite.

All components will honor CartePlusFaultInjection=1 and randomly inject faults.

Tracked by #1168.