Testing Pipes
It is useful to test pipes - or sections of them - before deploying them to actual hardware.
Consider this basic pipe which has a context expansion:
# test.yml
name: test
context:
msg: hello
input:
exec:
command: 'echo {{msg}}'
output:
write: console
We can run it using the Hotrod CLI:
$ hotrod pipes run --file test.yml
{"_raw":"hello"}
To experiment with overriding the context, define a little file containing some context:
# context.yml
context:
msg: goodbye
And use this to override the pipe context:
$ hotrod pipes run --file test.yml --context-files context.yml
{"_raw":"goodbye"}
Playing with pipes in this way builds confidence. For more complicated pipes, it is always easier to troubleshoot problems on a local machine.
The following pipe consumes a file times.txt
containing times in Unix epoch format (seconds since 1970).
The pipe must detect any gaps in this time record.
1592928057
1592928058
1592928059
1592928060
1592928061
1592928062
1592928063
1592928064
1592928065
1592928066
The time comes in as a string with the default fied _raw
, and we rename and convert into a number.
name: stream
input:
exec:
command: cat times.txt
actions:
- rename:
- _raw: epoch
- convert:
- epoch: num
output:
write: console
Running:
$ hotrod pipes run -f stream.yml
{"epoch":1592928057}
{"epoch":1592928058}
{"epoch":1592928059}
{"epoch":1592928060}
{"epoch":1592928061}
{"epoch":1592928062}
{"epoch":1592928063}
{"epoch":1592928064}
{"epoch":1592928065}
{"epoch":1592928066}
To find the difference between times, use the stream
action just after convert
- stream:
operation: delta
watch: epoch
We now get:
{"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928058,"delta":1,"elapsed":0}
{"epoch":1592928059,"delta":1,"elapsed":0}
{"epoch":1592928060,"delta":1,"elapsed":0}
{"epoch":1592928061,"delta":1,"elapsed":0}
{"epoch":1592928062,"delta":1,"elapsed":0}
{"epoch":1592928063,"delta":1,"elapsed":0}
{"epoch":1592928064,"delta":1,"elapsed":0}
{"epoch":1592928065,"delta":1,"elapsed":0}
{"epoch":1592928066,"delta":1,"elapsed":0}
How to detect gaps? Only pass through events where delta
isn't 1, and it is not the first event:
- filter:
condition: delta != 1 and count() > 1
The pipe will output nothing, but we can now provoke output by putting gaps into time.yml
.
This would be an example of a pipe monitoring data and detecting anomalies.
So the suggested way to build non-trivial pipes is step-by-step in this fashion.
It is possible to override the inputs of a pipe, which is useful for testing pipes which get their
input from some other source like a live HTTP request, TCP. and so forth. For instance, the version
of stream.yml
that ends with stream
can be fed a single timestamp:
$ hotrod pipes run -f stream.yml --input 1592928057
{"epoch":1592928057,"delta":1592928057,"elapsed":0}
Using --input @times.txt
gives us what we had before - i.e. if the argument starts with '@' it is
considered a file containing the data (this is same behaviour as curl
).
Use --output <filename>
to save the output to a file.
Tracing a pipe can show you how each step transforms the data. Here are our steps:
name: stream
input:
exec: # step 0
command: cat times.txt
actions:
- rename: # step 1
- _raw: epoch
- convert: # step 2
- epoch: num
- stream: # step 3
operation: delta
watch: epoch
output:
write: console # step 4
Run the last example with tracing enabled:
$ hotrod pipes run -f stream.yml --input 1592928057 --trace
[TRACE] action-rename step 1
LINE: {"_raw":"1592928057"} -> [{"epoch":"1592928057"}]
[TRACE] action-convert step 2
LINE: {"epoch":"1592928057"} -> [{"epoch":1592928057}]
[TRACE] action-stream step 3
LINE: {"epoch":1592928057} -> [{"epoch":1592928057,"delta":1592928057,"elapsed":0}]
[TRACE] output-exec step 4
LINE: {"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928057,"delta":1592928057,"elapsed":0}