raw

Operations on raw (non-JSON) data

Sub ActionDescription
raw-to-jsonWrap plain text as a JSON field
raw-discard-untilA pattern that must match before data starts to be processed
raw-extractExtract data from plain text, using a pattern
raw-replaceReplace data from plain text, using a pattern
raw-multilinecombine lines into multi-line events

raw to-json

Wrap plain text as a JSON field

Type: field

Example

input:

some text

action:

raw:
  to-json: some-field

output:

{"some-field":"some text"}

raw discard-until

A pattern that must match before data starts to be processed

All data before then will be discarded. An example of where this is useful is where the data would start with comments. Use begin-marker-field to reset

Type: regex

Example

input:

# some comment
# some other comment
num=1
num=2
num=3

action:

raw:
  discard-until: '^num'

output:

num=1
num=2
num=3

Example

input:

{"a":"# comment","first":true}
{"a":"num=1"}
{"a":"num=2"}
{"a":"# comment","first":true}
{"a":"# another"}                  
{"a":"num=10"}
{"a":"num=11"}

action:

raw:
  discard-until:
    pattern: '^[^#]'
    input-field: a
    begin-marker-field: first

output:

{"a":"num=1"}
{"a":"num=2"}
{"a":"num=10"}
{"a":"num=11"}

raw extract

Extract data from plain text, using a pattern

Field NameDescriptionTypeDefault
input-fieldField containing datafield-
patternThe pattern to match onregex-
replaceReplacement textstring-

input-field

Field containing data

Type: field

Example

input:

{"one":"1"}

action:

raw:
  extract:
    input-field: one
    replace: we got $0

output:

{"one":"we got 1"}

pattern

The pattern to match on

If replace is not provided, the first match is the output

Type: regex

Example

input:

num=1
num=2
num=3

action:

raw:
  extract:
    pattern: 'num=(\d+)'

output:

1
2
3

replace

Replacement text

If pattern is not provided, the whole input is still available as $0.

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Type: string

Example

input:

num=1
num=2
num=3

action:

raw:
  extract:
    pattern: 'num=(\d+)'
    replace: we got $1

output:

we got 1
we got 2
we got 3

raw replace

Replace data from plain text, using a pattern

The difference between this and extract is that it will never fail

Field NameDescriptionTypeDefault
input-fieldField containing datafield-
patternThe pattern to match onregex-
substitutionReplacement textstring-

input-field

Field containing data

Type: field

Example

input:

{"one":"hey 1"}

action:

raw:
  replace:
    input-field: one
    pattern: 'hey (\d+)'
    substitution: we got $1

output:

{"one":"we got 1"}

pattern

The pattern to match on

Type: regex

Example

input:

{"num":"1"}
{"text":"one"}

action:

raw:
  replace:
    pattern: one
    substitution: we got $0

output:

{"num":"1"}
{"text":"we got one"}

Example

input:

{"num":"1"}
{"text":"one"}

action:

raw:
  replace:
    pattern: (\d+)
    substitution: prefix of $1

output:

{"num":"prefix of 1"}
{"text":"one"}

substitution

Replacement text

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Type: string

raw multiline

combine lines into multi-line events

Similar to how Logstash does it, except that 'pattern' is used instead of 'matches' and 'merge' is used instead of 'what'; we say 'not-matches' instead of 'pattern' plus 'negate'

Field NameDescriptionTypeDefault
input-fieldoptional field containing line, otherwise use whole linefield-
mergedirection to try merge'previous' or 'next'-
delimhow to join lines of an event togetherstring\n
matchesthe line must match this patternpattern-
not-matchesthe line must NOT match this patternpattern-

input-field

optional field containing line, otherwise use whole line

Type: field

merge

direction to try merge

Type: 'previous' or 'next'

Example: keep merging matching lines with previous

input:

hello dolly
foo this gets merged
foo this as well
hello jane
foo that gets merged
foo that as well

action:

raw:
  multiline:
    matches: '^foo'
    merge: previous
    delim: '|'

output:

hello dolly|foo this gets merged|foo this as well
hello jane|foo that gets merged|foo that as well

delim

how to join lines of an event together

Type: string

matches

the line must match this pattern

Type: pattern

Example: merge matching lines with next line

input:

foo again
this gets merged
this does not      <---- what happens to stray lines?
foo two
we get this second line
foo three
we get this third line

action:

raw:
  multiline:
    matches: '^foo'
    merge: next
    delim: '|'

output:

foo again|this gets merged
this does not      <---- what happens to stray lines?
foo two|we get this second line
foo three|we get this third line

not-matches

the line must NOT match this pattern

must either have 'matches' or 'not-matches'

Type: pattern

Example: merge not-matching lines with previous line

input:

FOO header line
a merged line
another merged line
FOO header line 2
a merged line 2
another merged line 2
FOO header line 3
a merged line 3
another merged line 3

action:

raw:
  multiline:
    not-matches: '^FOO'
    merge: previous
    delim: '|'

output:

FOO header line|a merged line|another merged line
FOO header line 2|a merged line 2|another merged line 2