Testing Pipes

It is useful to test pipes - or sections of them - before deploying them to actual hardware.

Consider this basic pipe which has a context expansion:

# test.yml
name: test
context:
  msg: hello
input:
  exec:
    command: 'echo {{msg}}'
output:
  write: console

We can run it using the Hotrod CLI:

$ hotrod pipes run --file test.yml
{"_raw":"hello"}

To experiment with overriding the context, define a little file containing some context:

# context.yml
context:
  msg: goodbye

And use this to override the pipe context:

$ hotrod pipes run --file test.yml --context-files context.yml
{"_raw":"goodbye"}

Playing with pipes in this way builds confidence. For more complicated pipes, it is always easier to troubleshoot problems on a local machine.

The following pipe consumes a file times.txt containing times in Unix epoch format (seconds since 1970). The pipe must detect any gaps in this time record.

The time comes in as a string with the default fied _raw, and we rename and convert into a number.

name: stream
input:
  exec:
    command: cat times.txt
actions:
- rename:
  - _raw: epoch
- convert:
  - epoch: num
output:
  write: console

Running:

$ hotrod pipes run -f stream.yml
{"epoch":1592928057}
{"epoch":1592928058}
{"epoch":1592928059}
{"epoch":1592928060}
{"epoch":1592928061}
{"epoch":1592928062}
{"epoch":1592928063}
{"epoch":1592928064}
{"epoch":1592928065}
{"epoch":1592928066}

To find the difference between times, use the stream action just after convert

- stream:
    operation: delta
    watch: epoch

We now get:

{"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928058,"delta":1,"elapsed":0}
{"epoch":1592928059,"delta":1,"elapsed":0}
{"epoch":1592928060,"delta":1,"elapsed":0}
{"epoch":1592928061,"delta":1,"elapsed":0}
{"epoch":1592928062,"delta":1,"elapsed":0}
{"epoch":1592928063,"delta":1,"elapsed":0}
{"epoch":1592928064,"delta":1,"elapsed":0}
{"epoch":1592928065,"delta":1,"elapsed":0}
{"epoch":1592928066,"delta":1,"elapsed":0}

How to detect gaps? Only pass through events where delta isn't 1, and it is not the first event:

    - filter:
        condition: delta != 1 and count() > 1

The pipe will output nothing, but we can now provoke output by putting gaps into time.yml. This would be an example of a pipe monitoring data and detecting anomalies.

So the suggested way to build non-trivial pipes is step-by-step in this fashion.

It is possible to override the inputs of a pipe, which is useful for testing pipes which get their input from some other source like a live HTTP request, TCP. and so forth. For instance, the version of stream.yml that ends with stream can be fed a single timestamp:

$ hotrod pipes run -f stream.yml --input 1592928057
{"epoch":1592928057,"delta":1592928057,"elapsed":0}

Using --input @times.txt gives us what we had before - i.e. if the argument starts with '@' it is considered a file containing the data (this is same behaviour as curl).

Use --output <filename> to save the output to a file.

Tracing a pipe can show you how each step transforms the data. Here are our steps:

name: stream
input:
  exec: # step 0
    command: cat times.txt
actions:
- rename:  # step 1
  - _raw: epoch
- convert: # step 2
  - epoch: num
- stream: # step 3
    operation: delta
    watch: epoch
output:
  write: console  # step 4

Run the last example with tracing enabled:

$ hotrod pipes run -f stream.yml --input 1592928057 --trace
[TRACE]  action-rename step 1
LINE: {"_raw":"1592928057"} -> [{"epoch":"1592928057"}]
[TRACE]  action-convert step 2
LINE: {"epoch":"1592928057"} -> [{"epoch":1592928057}]
[TRACE]  action-stream step 3
LINE: {"epoch":1592928057} -> [{"epoch":1592928057,"delta":1592928057,"elapsed":0}]
[TRACE]  output-exec step 4
LINE: {"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928057,"delta":1592928057,"elapsed":0}