Skip to content

Test definitions

A test definition says which two nodes to test between, what traffic to send, and which criteria decide pass or fail. You can write it as a YAML document or as a Robot Framework suite. The file suffix decides which: .yaml/.yml or .robot.

The two formats are peers, not a primary format and a fallback. No criterion can be written in one and not in the other.

The same test, twice

apiVersion: dokimi.io/v1alpha1
kind: NetworkTest

metadata:
  name: worker-01-to-worker-02-udp

placement:
  mode: explicit
  source: worker-01
  target: worker-02

test:
  type: iperf2
  protocol: udp
  duration: 30s
  targetBandwidth: 1G

monitoring:
  mode: managed

validation:
  - metric: throughput
    comparison: atLeast
    threshold: 950
    unit: megabits per second
  - metric: packetLoss
    comparison: atMost
    threshold: 0
    unit: percent

results:
  directory: ./results
*** Settings ***
Documentation       worker-01-to-worker-02-udp
Resource            resources/Dokimi.resource

*** Variables ***
${SOURCE_NODE}         worker-01
${TARGET_NODE}         worker-02
${TEST_TYPE}           iperf2
${PROTOCOL}            udp
${TEST_DURATION}       30s
${TARGET_BANDWIDTH}    1G
${MONITORING_MODE}     managed
${RESULTS_DIRECTORY}   ./results

*** Test Cases ***
worker-01-to-worker-02-udp
    Run Network Test
    ...    type=${TEST_TYPE}
    ...    protocol=${PROTOCOL}
    ...    source=${SOURCE_NODE}
    ...    target=${TARGET_NODE}
    ...    bandwidth=${TARGET_BANDWIDTH}
    ...    duration=${TEST_DURATION}

    Throughput Should Be     At Least    950    megabits per second
    Packet Loss Should Be    At Most     0      percent

What a definition contains

Field Accepted values
metadata.name the test's name: letters, digits, single spaces and a little punctuation
placement.mode explicit: you name both nodes
placement.source, placement.target two different Kubernetes node names
test.type iperf2
test.protocol udp or tcp
test.duration a number with ms, s, m or h, for example 30s
test.targetBandwidth a number with an optional K, M or G, for example 1G. The prefixes are decimal: 1G is 10⁹ bits per second
monitoring.mode managed, existing or auto. Defaults to auto. See Monitoring modes
validation at least one pass criterion. See Pass criteria
results.directory where the run's reports go on your machine, under the run id. Defaults to ./results

These are the values the parser accepts today. Images are not part of a definition: they are chosen when you submit, so the same test runs unchanged in every cluster. Installation shows how.

Monitoring modes

Mode What it means Status
managed The run starts its own Prometheus, which scrapes the run's exporters, and keeps it after the run works today
auto Use a compatible Prometheus the cluster already has, or fall back to managed. Detecting one is not built yet, so today auto always falls back to managed, and the log says so falls back to managed today
existing Use a Prometheus the cluster already has, and modify nothing else planned

validate and submit refuse existing today, with exit code 3, so a run that could only fail is never created. Runs and exit codes describes what a managed run deploys.

Strict, on purpose

What eventually runs is never the file you wrote. It is a suite rendered from what dokimi parsed. Anything the parser did not understand would silently not run, and a criterion that silently does not run is a test that passes without checking anything. So dokimi refuses instead:

  • in YAML, any field the schema does not define;
  • in a Robot suite, anything dokimi would not itself produce: any keyword other than Run Network Test and the criterion keywords, control structures such as IF and FOR, [Setup], [Teardown], Suite Teardown, other imports, a *** Keywords *** section, a second test case, or a variable it does not know.

Every refusal names the line and the reason:

dokimi: line 26: unrecognised keyword 'Packet Loss Shuld Be'. A test case may
call only 'Run Network Test' and the criterion keywords, because anything else would not
be carried into the suite that runs.

Keyword names are matched the way Robot matches them, so packet loss should be and Packet_Loss_Should_Be both read as Packet Loss Should Be.

A malformed definition exits 3, before anything is created.