// SPDX-FileCopyrightText: 2014 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
// SPDX-FileCopyrightText: 2024-2025 Kienan Stewart <kstewart@efficios.com>
//
// SPDX-License-Identifier: CC-BY-SA-4.0

// Render with Asciidoctor
:check: ✔
:ex: ✗

= LTTng-tools Tests

:toc:

== Test harness

The full tests are currently powered via `make check`. Individual tests may
be run directly by calling their respective script. For example:

[,bash]
----
./tests/regression/tools/live/test_ust
----

New tests must be written using the python framework (see `tests/utils/lttngtest`).
The purpose of the `lttngtest` framework is to more reliably control and clean-up
the environment for each test with the eventual long-term goal to be able to
parallelize some of the tests.

The file tests/template_test.py can be copied as a starting point for new tests
that are being written in Python.

Legacy tests are principally written in bash, and some are in C/C++. C/C++ tests
must be limited to those that actually require interacting with loaded libraries
or internal testpoints.

=== Test-related configuration options

 * `PRECIOUS_TESTS`: (Default:{nbsp}yes) When set to  `yes`, test logs will not be deleted after a test failure. This stops tests from being able to automatically rerun with `make check`

=== TAP drivers options ===

. Options supported by the various TAP drivers
[cols="1,1,1,1,1"]
|===
| Option | Description | Python tap-driver | BASH tap-driver | C/C++ tap driver

| `LTTNG_TESTS_TAP_AUTOTIME`
| Automatically outputs test step timing information
| {check} | {check} | {check}

| `LTTNG_TEST_LOG_DIR`
| Sets the default log dir to `<testname>.log.d`
| {check} | {check} | {ex}

|===

=== Test environment variables

. Environment variables for controlling which tests are run
[cols="1,1,1,1,1"]
|===
| Environment Variable | Description | Python Framework | Bash Framework | C/C++ Framework
| `LTTNG_ENABLE_DESTRUCTIVE_TESTS`
| When set to `will-break-my-system`, tests that change global state (e.g. clock settings, creating and destroying users, ...) will be run. These tests typically also require being run as the root user.
| {check} | {check} | {ex}

| `LTTNG_TEST_ABORT_ON_MISSING_PLATFORM_REQUIREMENTS`
| When set to a non-empty, non-'0' value, tests should bail out instead of skipping. This is useful in certain CI scenarios to avoid hiding errors behind skipped tests.
| {check} | {check} | {ex}

| `LTTNG_TOOLS_DISABLE_KERNEL_TESTS`
| Do not run kernel (LTTng-modules) tests
| {check} | {check} | {ex}

| `LTTNG_TOOLS_RUN_TESTS_LONG_REGRESSION`
| When set to non-empty string other than '0', run long regression tests
| {check} | {check} | {ex}

| `LTTNG_TOOLS_BUILD_PROFILE_DIR`
| Directory of build profiles (e.g. `lttng-build-profile.json` files, generated by `configure`) describing builds other than the in-tree one, so that their daemons, client and test applications can be exercised by cross-ABI tests. Discovered profiles are added to the in-tree build's own. Multi-ABI tests run against every matching profile that make sense for the test and skip unless profiles of at least two ABI word sizes (e.g. a 32-bit build alongside a 64-bit one) are available.
| {check} | {ex} | {ex}
|===

. Environment variables for controlling test output
[cols="1,1,1,1,1"]
|===
| Environment Variable | Description | Python Framework | Bash Framework | C/C++ Framework

| `LTTNG_TEST_LOG_DIR`
| Define a directory to which the verbose logs will be written to disk.
`-` may be used to indicate stdout/stderr respectively.
Some invocations of programs during the test may override this value to perform the test.
| {check} | {check} | {ex}

| `LTTNG_TEST_BABELTRACE_VERBOSITY`
| Define the level of verbosity for `babeltrace2`. Default:{nbsp}`I`
| {ex} | {check} | {ex}

| `LTTNG_TEST_VERBOSE_BABELTRACE`
| Run `babeltrace2` in it's verbose mode when not "0" or empty
| {check} | {check} | {ex}

| `LTTNG_TEST_VERBOSE_CLIENT`
| Run `lttng-client` in it's verbose mode when not "0" or empty
| {check} | {check} | {ex}

| `LTTNG_TEST_VERBOSE_RELAYD`
| Run `lttng-relayd` in it's verbose and debug mode when not "0" or empty
| {check} | {check} | {ex}

| `LTTNG_TEST_VERBOSE_SESSIOND`
| Run `lttng-sesiond in it's verbose and debug mode when not "0" or empty
| {check} | {check} | {ex}
|===

. Environment variables for debugging during tests
[cols="1,1,1,1,1"]
|===
| Environment Variable | Description | Python Framework | Bash Framework | C/C++ Framework

| `LTTNG_RELAYD_ENV_VARS`
| Environment variables to forward into test environment
| {check} | {ex} | {ex}

| `LTTNG_SESSIOND_ENV_VARS`
| Environment variables to forward into test environment
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_KERNEL
| Launch `gdbserver` and attach to newly spawned `lttng-consumerd` process
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_KERNEL_PORT`
| The TCP port for the gdbserver instance. Default:{nbsp}`1026`
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_KERNEL_WAIT`
| When not empty, the test script will wait for user input after starting lttng-consumerd before continuing.
Useful to connect and insert breakpoints before other commands are run.
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_UST32`
| Launch `gdbserver` and attach to newly spawned `lttng-consumerd` UST32 process
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_UST32_PORT`
| The TCP port for the gdbserver instance. Default:{nbsp}`1027`
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_UST32_WAIT`
| When not empty, the test script will wait for user input after starting lttng-consumerd before continuing.
Useful to connect and insert breakpoints before other commands are run.
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_UST64`
| Launch `gdbserver` and attach to newly spawned `lttng-consumerd` UST64 process
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_UST64_PORT`
| The TCP port for the gdbserver instance. Default:{nbsp}`1028`
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_CONSUMERD_UST64_WAIT`
| When not empty, the test script will wait for user input after starting lttng-consumerd before continuing.
Useful to connect and insert breakpoints before other commands are run.
| {check} | {ex} | {ex}

| `LTTNG_TEST_GDBSERVER_RELAYD`
| Launch `gdbserver` and attach to newly spawned `lttng-relayd` processes
| {check} | {check} | {ex}

| `LTTNG_TEST_GDBSERVER_RELAYD_PORT`
| The TCP port for the gdbserver instance. Default:{nbsp}`1025`
| {check} | {check} | {ex}

| `LTTNG_TEST_GDBSERVER_RELAYD_WAIT`
| When not empty, the test script will wait for user input after starting lttng-relayd before continuing.
Useful to connect and insert breakpoints before other commands are run.
| {check} | {check} | {ex}

| `LTTNG_TEST_GDBSERVER_SESSIOND`
| Launch `gdbserver` and attach to newly spawned `lttng-sessiond` processes
| {check} | {check} | {ex}

| `LTTNG_TEST_GDBSERVER_SESSIOND_PORT`
| The TCP port for the gdbserver instance. Default:{nbsp}`1024`
| {check} | {check} | {ex}

| `LTTNG_TEST_GDBSERVER_SESSIOND_WAIT`
| When not empty, the test script will wait for user input after starting lttng-sessiond before continuing.
Useful to connect and insert breakpoints before other commands are run.
| {check} | {check} | {ex}

| `LTTNG_TEST_NO_RELAYD`
| Do not spawn `lttng-relayd` in test environment
Used to manually launch one during test troubleshooting
| {check} | {ex} | {ex}

| `LTTNG_TEST_PRESERVE_TEST_ENV`
| Do not delete test environment folders on tear-down
| {check} | {ex} | {ex}

| `LTTNG_TEST_PRESERVE_TEST_ENV_DIR`
| Move the preserved test environments to the given path. The path must exist and be a directory.
| {check} | {ex} | {ex}

| `LTTNG_TEST_PRESERVE_TEST_ENV_ON_FAILURE`
| Set `LTTNG_TEST_PRESERVE_TEST_ENV` when the tap generator encounters a failure
| {check} | {ex} | {ex}
|===

. Environment variables for managing test suite behaviour
[cols="1,1,1,1,1"]
|===
| `LTTNG_TEST_SERIAL_TEST_POLL_PERIOD_SECONDS`
| The time in seconds that the serial test runner should read stdout and check
the test timeout condition. (Default: 60)
| {check} | N/A | N/A

| `LTTNG_TEST_SERIAL_TEST_TIMEOUT_MINUTES`
| The time in minutes after which a test run by the serial test runner will be killed. (Default: 30)
| {check} | N/A | N/A

| `LTTNG_TEST_TEARDOWN_TIMEOUT`
| A timeout in seconds after which waiting for lttng-consumerd, lttng-relayd, or lttng-sessiond
will stop and a kill attempted during the clean-up of the test environment. (Default: 60)
| {check} | {check} | N/A

| `LTTNG_TOOLS_TESTS_DISABLE_WARN_LTTNG_PROCESSES`
| Skip checking for existing LTTng processes when starting `make check`
| N/A | N/A | N/A
|===

== Test Anti-Patterns

OK, there are a few patterns that have been found over and over in the
testing code base which makes the tests flaky. Here is an incomplete
list. Don't do that.


=== Taskset

In some containerised environments, such as Incus, the affinities CPUs
allocated to the container may change during runtime. When using taskset,
there may be a race between selecting a CPU that is currently online and then
running the task itself.

It is important to check for taskset's specific failure, use retries where
appropriate (e.g. `retry_anycpu_taskset`), or have another method to mitigate
potential failures.

This type of issue is rarely seen in local testing, but happens in our CI
cluster.

=== Using pidof to wait for a background application (by name) to disappear

Why is it flaky ?

The application may be delayed after being forked, but not executed yet.
Therefore, pidof will not find it. Use "wait" instead.

=== Using sleep as delay-based optimistic synchronization technique

Why is it flaky ?

Everything that needs to happen before/after other things need to
be explicitly synchronized using e.g. a file (used as a flag).
Sleep is just an indicator of a minimum arbitrary delay, but
machine load and scheduling can actually mess up the real delay
between applications. Use explicit synchronization points. Never
sleep.

=== Using killall on a background application

Why is it flaky ?

Similarly to pidof, killall may run before the background application
executes, thus failing to find it. Store the application PID after it
it launched in background into a temporary variable for later use
by kill and wait.

=== Using wait ${!} to wait for completion of many background applications

Why is it flaky ?

It just waits for the last application put in background. Track the PIDs
of the applications of interest in an array and use a construction similar
to the following to wait for them:

```
## Avoid running `wait ` when the array is empty
if [[ -n "${pids[*]}" ]]; then
   wait "${pids[@]}"`
fi
```

Avoid `wait` without arguments as it will wait for other background
prcesses such as lttng-relayd or lttng-sessiond when they are started
without daemonization.

=== Forgetting wait at the end (or error return path) of a test phase that has background applications

Why is it flaky ?

Those application may interact with the following testing phases,
thus skewing the results.

=== Not grepping into the entire code base for similar patterns

When you find a problematic coding pattern, chances are it appears
elsewhere in the testing code base. Please fix it everywhere!

=== Introducing a utility abstraction without changing all open coded similar code path

When an abstraction for e.g. starting and stopping the session daemon
is introduced as a utility (e.g. utils.sh), future changes will
assume that all the testing code base is using this abstraction.
Leaving a few custom open-coded sites of duplicated code around is a
good way to make it a pain to update the abstraction in the future.
