Pipes Language


exec run a command, either continuously or scheduled

amqp read from a RabbitMQ queue

http-poll do a GET or POST

udp listens for UDP connections

tcp listens for TCP connections

http-server events from a general web hook

redis read from the Redis in-memory key-value store


write write to file or device

amqp write to a RabbitMQ queue

exec run a command, either continuously or per-event

http-post bulk HTTP POST

elastic bulk Elasticsearch insert

udp write to UDP port

tcp connect to TCP server

http-server serve up data on HTTP

redis write to the Redis in-memory key-value store


raw: manipulate non-JSON data

  • extract extract and replace text with a pattern
  • replace similar, except do not insist on a match
  • discard-until ignore lines until they match a pattern
  • to-json quote raw text as JSON

expand convert fields containing data into JSON

  • csv CSV (with specified delimiter)
  • key-value key-value pairs like "a=1 b=2"
  • json expand quoted JSON in a field (merges)
  • events

collapse convert JSON into other formats (reverse of expand)

  • csv
  • key-value
  • array

extract extract fields using a pattern

convert convert fields from a type into another type

  • JSON types str, num, bool
  • Size kiB(K), MiB(M), GiB(G), GB,MB,KB
  • Time s, ms, us

time convert and write out time fields

  • time when classifies current time (e.g. 'business hours')

add add fields to an event with text field expansions (${field})

  • output-fields
  • template

script add calculated fields to an event using Lua (see add)

  • (condition) conditionally add the fields (optional)
  • let set fields to value of expressions
  • set set fields to constants

remove remove fields from an event

rename rename fields in an event

filter filter events

  • patterns a set of fields must each match a pattern to pass through event
  • exclude event is dropped if the fields match
  • condition a Lua expression

stream can calculate new events based on value of last event

  • delta difference between new value of field and the old value

transition detect when data changes

stalled detect when data stops flowing

transaction collect matching events together into a larger transaction event

exec run a shell command on event fields

enrich enrich data by matching columns in a CSV database

generate generate events by tracking event history