Testing Pipes

It is useful to test pipes - or sections of them - before deploying them to actual hardware.

Consider this basic pipe which has a context expansion:

# test.yml
name: test
context:
  msg: hello
input:
  exec:
    command: 'echo {{msg}}'
output:
  write: console

We can run it using the Hotrod CLI:

$ hotrod pipes run --file test.yml
{"_raw":"hello"}

To experiment with overriding the context, define a little file containing some context:

# context.yml
context:
  msg: goodbye

And use this to override the pipe context:

$ hotrod pipes run --file test.yml --context-files context.yml
{"_raw":"goodbye"}

Playing with pipes in this way builds confidence. For more complicated pipes, it is always easier to troubleshoot problems on a local machine.

The following pipe consumes a file times.txt containing times in Unix epoch format (seconds since 1970). The pipe must detect any gaps in this time record.

1592928057
1592928058
1592928059
1592928060
1592928061
1592928062
1592928063
1592928064
1592928065
1592928066

The time comes in as a string with the default fied _raw, and we rename and convert into a number.

name: stream
input:
  exec:
    command: cat times.txt
actions:
- rename:
  - _raw: epoch
- convert:
  - epoch: num
output:
  write: console

Running:

$ hotrod pipes run -f stream.yml
{"epoch":1592928057}
{"epoch":1592928058}
{"epoch":1592928059}
{"epoch":1592928060}
{"epoch":1592928061}
{"epoch":1592928062}
{"epoch":1592928063}
{"epoch":1592928064}
{"epoch":1592928065}
{"epoch":1592928066}

To find the difference between times, use the stream action just after convert

- stream:
    operation: delta
    watch: epoch

We now get:

{"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928058,"delta":1,"elapsed":0}
{"epoch":1592928059,"delta":1,"elapsed":0}
{"epoch":1592928060,"delta":1,"elapsed":0}
{"epoch":1592928061,"delta":1,"elapsed":0}
{"epoch":1592928062,"delta":1,"elapsed":0}
{"epoch":1592928063,"delta":1,"elapsed":0}
{"epoch":1592928064,"delta":1,"elapsed":0}
{"epoch":1592928065,"delta":1,"elapsed":0}
{"epoch":1592928066,"delta":1,"elapsed":0}

How to detect gaps? Only pass through events where delta isn't 1, and it is not the first event:

    - filter:
        condition: delta != 1 and count() > 1

The pipe will output nothing, but we can now provoke output by putting gaps into time.yml. This would be an example of a pipe monitoring data and detecting anomalies.

So the suggested way to build non-trivial pipes is step-by-step in this fashion.

It is possible to override the inputs of a pipe, which is useful for testing pipes which get their input from some other source like a live HTTP request, TCP. and so forth. For instance, the version of stream.yml that ends with stream can be fed a single timestamp:

$ hotrod pipes run -f stream.yml --input 1592928057
{"epoch":1592928057,"delta":1592928057,"elapsed":0}

Using --input @times.txt gives us what we had before - i.e. if the argument starts with '@' it is considered a file containing the data (this is same behaviour as curl).

Use --output <filename> to save the output to a file.

Tracing a pipe can show you how each step transforms the data. Here are our steps:

name: stream
input:
  exec: # step 0
    command: cat times.txt
actions:
- rename:  # step 1
  - _raw: epoch
- convert: # step 2
  - epoch: num
- stream: # step 3
    operation: delta
    watch: epoch
output:
  write: console  # step 4    

Run the last example with tracing enabled:

$ hotrod pipes run -f stream.yml --input 1592928057 --trace
[TRACE]  action-rename step 1
LINE: {"_raw":"1592928057"} -> [{"epoch":"1592928057"}]
[TRACE]  action-convert step 2
LINE: {"epoch":"1592928057"} -> [{"epoch":1592928057}]
[TRACE]  action-stream step 3
LINE: {"epoch":1592928057} -> [{"epoch":1592928057,"delta":1592928057,"elapsed":0}]
[TRACE]  output-exec step 4
LINE: {"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928057,"delta":1592928057,"elapsed":0}