Between the input and the output is a set of processing steps which are applied in order, each consuming the output of the last one. These are called actions.

As with inputs and soutputs, inputs to these steps can contain {{var}} context expansions.

Generally, these steps work with JSON data but there are a few which convert to and from other formats which are used as 'bookends' in the processing chain.

The inputs usually generate JSON data unless otherwise specified.

The processing steps belong to these categories:

  • filtering out unneeded lines
  • extracting raw data and converting to JSON
  • converting fields
  • adding extra fields, perhaps conditionally
  • removing unneeded fields
  • generating new events, such as alerts

With JSON data, we call the records events and the keys fields. It will be clear whether we are dealing with an input or an output field.

Field names must start with a letter and otherwise consist only of letters, digits, and underscores. So status_result is fine, status-result is not. (This restriction may be removed in future.)

Expressions involve field names directly, like a + 1 or throughput/1024. Conditions are similar, but involve comparisons like a > 0.

The expression syntax is based on Lua, so e.g a > 0 and b > 0 but previous notation is supported, e.g a > 0 && b > 0 where && means 'and', || means 'or', and == means 'equals'.

By pattern, we mean a regular expression.

By default, missing fields do not cause an error; the data is passed through unaffected. To force warnings, switch on debug mode.

This tolerant mode means that actions ignore events they do not recognize. For instance, this gives you a way to do conditional execution of scripts. This exec action will pass the value of the payload field through the standard input of the command, but only if the field exists:

- exec:
    input-field: payload
      command: cat >> /path/to/payload-file

The exec action can also be triggered by the existence of a field - it need not be text. For instance, the end-marker-field is set by batch in the http-poll input to be true and can be used to trigger a command on the last line read.

extract (action)Extract data from plain text, using a pattern
convert (action)Converts data types of values
raw (action)Operations on raw (non-JSON) data
filter (action)Removes events, based on some given conditions
script (action)Set fields to computed values, perhaps conditionally
stream (action)Create a new field calculated on historical data
add (action)Add *new* fields to an event
remove (action)Remove fields
rename (action)Rename fields
time (action)Time{stamp} manipulation
transaction (action)Collects events together based on some condition to make a single new event
stalled (action)Reports when a stream has stopped getting events for a given duration
expand (action)Converts simple separated data into JSON
collapse (action)Converts JSON records to another format, like CSV or key-value pairs
exec (action)Execute arbitrary commands
generate (action)Create new events, specifically for alerts
transition (action)Performs various actions based on a changed field
enrich (action)Allows using CSV lookup to enrich data