Pipes Language

A pipe consists of an input which generates data that passes through actions and finally gets sent to the desired destination with an output.

In addition, some inputs may be used as actions, as well as some outputs

Inputs

exec run a command, either continuously or scheduled

amqp read from a RabbitMQ queue

http-poll do a GET or POST

udp listens for UDP connections

tcp listens for TCP connections

http-server events from a general web hook

redis read from the Redis in-memory key-value store

files read files specified with wildcard patterns

Outputs

write write to file or device

amqp write to a RabbitMQ queue

exec run a command, either continuously or per-event

http-post bulk HTTP POST

elastic bulk Elasticsearch insert

udp write to UDP port

tcp connect to TCP server

http-server serve up data on HTTP

redis write to the Redis in-memory key-value store

Actions

raw: manipulate non-JSON data

  • extract extract and replace text with a pattern
  • replace similar, except do not insist on a match
  • discard-until ignore lines until they match a pattern
  • to-json quote raw text as JSON

expand convert fields containing data into JSON

  • csv CSV (with specified delimiter)
  • key-value key-value pairs like "a=1 b=2"
  • json expand quoted JSON in a field (merges)
  • events

collapse convert JSON into other formats (reverse of expand)

  • csv
  • key-value
  • array

extract extract fields using a pattern

convert convert fields from a type into another type

  • JSON types str, num, bool
  • Size kiB(K), MiB(M), GiB(G), GB,MB,KB
  • Time s, ms, us

time convert and write out time fields

  • time when classifies current time (e.g. 'business hours')

add add fields to an event with text field expansions (${field})

  • output-fields
  • template

script add calculated fields to an event using Lua (see add)

  • (condition) conditionally add the fields (optional)
  • let set fields to value of expressions
  • set set fields to constants

remove remove fields from an event

rename rename fields in an event

filter filter events

  • patterns a set of fields must each match a pattern to pass through event
  • exclude event is dropped if the fields match
  • condition a Lua expression

stream can calculate new events based on value of last event

  • delta difference between new value of field and the old value

transition detect when data changes

stalled detect when data stops flowing

transaction collect matching events together into a larger transaction event

exec run a shell command on event fields

enrich enrich data by matching columns in a CSV database

generate generate events by tracking event history

Action Inputs
Action Outputs
Inputs
Actions
Outputs