Between the input and the output is a set of processing steps which are applied in order, each consuming the output of the last one. These are called actions.

As with inputs and soutputs, inputs to these steps can contain {{var}} context expansions.

Generally, these steps work with JSON data but there are a few which convert to and from other formats which are used as 'bookends' in the processing chain.

The inputs usually generate JSON data unless otherwise specified.

The processing steps belong to these categories:

  • filtering out unneeded lines
  • extracting raw data and converting to JSON
  • converting fields
  • adding extra fields, perhaps conditionally
  • removing unneeded fields
  • generating new events, such as alerts

With JSON data, we call the records events and the keys fields. It will be clear whether we are dealing with an input or an output field.

Field names must start with a letter and otherwise consist only of letters, digits, and underscores. So status_result is fine, status-result is not. (This restriction may be removed in future.)

Expressions involve field names directly, like a + 1 or throughput/1024. Conditions are similar, but involve comparisons like a > 0.

In general the syntax is similar to that of JavaScript, so e.g a > 0 && b > 0 where && means 'and', || means 'or', and == means 'equals'.

By pattern, we mean a regular expression.

By default, inputs generate JSON. In these docs we will say '# raw:' or '# json:' for unchanged input, and '# input:' for the usual mode where lines become value of a _raw field.

extract (action)Extract data from plain text, using a pattern
convert (action)Converts data types of values
raw (action)Operations on raw (non-JSON) data
filter (action)Removes events, based on some given conditions
script (action)Set fields to computed values, perhaps conditionally
stream (action)Create a new field calculated on historical data
add (action)Add *new* fields to an event
remove (action)Remove fields
rename (action)Rename fields
time (action)Time{stamp} manipulation
transaction (action)Collects events together based on some condition to make a single new event
stalled (action)Reports when a stream has stopped getting events for a given duration
expand (action)Converts simple separated data into JSON
collapse (action)Converts JSON records to another format, like CSV or key-value pairs
exec (action)Execute arbitrary commands
generate (action)Create new events, specifically for alerts
transition (action)Performs various actions based on a changed field
enrich (action)Allows using CSV lookup to enrich data