expand

Converts simple separated data into JSON

This is called expand because it pulls JSON data (values, key-value pairs or JSON text) from a field.

Field NameDescriptionTypeDefault
input-fieldField containing datafield_raw
delimThe delimiter (separator) to be usedstring,
removeWhether to remove field after its value is usedboolfalse
document-startwith 'expand array' collapseinput-field between doc-start and doc-end into one text eventboolfalse
Sub ActionDescription
expand-csvConverts CSV rows into JSON records
expand-key-valueConverts data in key=value format
expand-jsonTreat incoming data as already in JSON format
expand-eventsConverts a single event into multiple events
expand-multilinecollect lines into fields

expand csv

Converts CSV rows into JSON records

Field NameDescriptionTypeDefault
headerWhether to treat first line as a CSV headerboolfalse
begin-marker-fieldForce a line to be considered a 'first' header line--
autoconvertcontrol auto-conversion when reading in headersbooltrue
gen-headersWhether to generate automatic CSV headersboolfalse
fieldsSpecify fields and their types (str, num, bool, num)array of key-value pairs-
field-fileSpecify fields and their types (str, num, bool, num)File containing "name:type" pairs-
header-fieldField containing header (CSV column names)--
header-field-typesField containing header has types specified (with name:type format)-false
relaxed-schemaAllow picking just the first given number fields, and giving those names-false

Whether to treat first line as a CSV header

Type: bool

Example

input:

text,number
one,1
two,2

action:

expand:
  input-field: _raw
  remove: true
  csv:
    header: true

output:

{"text":"one","number":1}
{"text":"two","number":2}

begin-marker-field

Force a line to be considered a 'first' header line

see the begin-marker-field of input exec batch

autoconvert

control auto-conversion when reading in headers

Type: bool

gen-headers

Whether to generate automatic CSV headers

Type: bool

Example

input:

one,1

action:

expand:
  input-field: _raw
  remove: true
  csv:
    gen-headers: true

output:

{"_0":"one","_1":1}

fields

Specify fields and their types (str, num, bool, num)

Type: array of key-value pairs

Example

input:

{"data":"one,1"}

action:

expand:
  input-field: data
  csv:
    fields:
      - text: str
      - number: num

output:

{"data":"one,1","text":"one","number":1}

field-file

Specify fields and their types (str, num, bool, num)

Type: File containing "name:type" pairs

Example

file: some-file

text:str
number:num

input:

{"data":"one,1"}

action:

expand:
  input-field: data
  csv:
    field-file: some-file

output:

{"data":"one,1","text":"one","number":1}

header-field

Field containing header (CSV column names)

Example

input:

{"header":"text,number","row":"one,1"}

action:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header

output:

{"text":"one","number":1}

header-field-types

Field containing header has types specified (with name:type format)

Example

input:

{"header":"text:str,number:str","row":"one,1"}

action:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header
    header-field-types: true

output:

{"text":"one","number":"1"}

relaxed-schema

Allow picking just the first given number fields, and giving those names

Example

input:

{"row":"1,2,3,4,5"}

action:

expand:
  input-field: row
  remove: true
  csv:
    relaxed-schema: true
    fields:
      - first: num
      - second: str

output:

{"first":1,"second":"2"}

expand key-value

Converts data in key=value format

Example: Something basic

input:

one=1 two=2

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

output:

{"one":"1","two":"2"}

Example: Quoted strings

input:

male="john doe" female="jane doe"

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

output:

{"male":"john doe","female":"jane doe"}
Field NameDescriptionTypeDefault
autoconvertAutomatically convert to numbers if possibleboolfalse
key-value-delimSet the delimiter between the key and the valuebool=
multipleIn the case of repeated, choose whether to choose first, last, or to place all in an arraystring-

autoconvert

Automatically convert to numbers if possible

Type: bool

Example

input:

one=1 two=2.0

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    autoconvert: true

output:

{"one":1,"two":2}

key-value-delim

Set the delimiter between the key and the value

Type: bool

Example

input:

one:1 two:2

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: ':'

output:

{"one":"1","two":"2"}

multiple

In the case of repeated, choose whether to choose first, last, or to place all in an array

Type: string

Example: first

input:

name=john name=jane

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: first

output:

{"name":"john"}

Example: last

input:

name=john name=jane

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: last

output:

{"name":"jane"}

Example: array

input:

name=john name=jane

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: array

output:

{"name":["john","jane"]}

expand json

Treat incoming data as already in JSON format

Type: bool

Example

input:

{"row":"{\"one\":1,\"two\":2}"}

action:

expand:
  input-field: row
  remove: true
  json: true

output:

{"one":1,"two":2}

expand events

Converts a single event into multiple events

Field NameDescriptionTypeDefault
output-split-fieldA field where resulting split data is to be placedfield-
skip-listAn array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"array of JSON Pointers-

output-split-field

A field where resulting split data is to be placed

Type: field

Example

input:

{"names":"john jane","family":"doe"}

action:

expand:
  input-field: names
  remove: true
  delim: ' '
  events:
    output-split-field: name

output:

{"family":"doe","name":"john"}
{"family":"doe","name":"jane"}

skip-list

An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"

Type: array of JSON Pointers

Example

input:

{ "array": [{"name": "jane","family":"doe" }, { "name":"john","family":"doe" }] }

action:

expand:
  events:
    skip-list: []

output:

{"array":{"name":"jane","family":"doe"}}
{"array":{"name":"john","family":"doe"}}

expand multiline

collect lines into fields

Currently the patterns allowed are '\n' (line end) and '\n\n' (blank line)

Type: array of field-pattern pairs

Example: collecting pairs of lines into fields x and y

input:

one 1
one 2
two 1
two 2

action:

expand:
  input-field: _raw
  multiline:
    - x: '\n'
    - y: '\n'

output:

{"x":"one 1","y":"one 2"}
{"x":"two 1","y":"two 2"}

Example: map first line to x and rest to y upto empty line

input:

one 1
one 2
one 3
one 4

two 1
two 2

action:

expand:
  input-field: _raw
  delim: "\n"
  multiline:
    - x: "\n"
    - y: "\n\n"

output:

{"x":"one 1","y":"one 2\none 3\none 4"}
{"x":"two 1","y":"two 2"}

input-field

Field containing data

Type: field

Example

input:

{"good":"fast,safe","bad":"slow,dangerous"}

action:

expand:
  input-field: good
  remove: true
  csv:
    fields:
    - speed: str
    - trust: str

output:

{"bad":"slow,dangerous","speed":"fast","trust":"safe"}

delim

The delimiter (separator) to be used

Type: string

Example

input:

{"row":"1 2"}

action:

expand:
  delim: " "
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

output:

{"row":"1 2","one":1,"two":"2"}

remove

Whether to remove field after its value is used

Type: bool

Example

input:

{"row":"1 2"}

action:

expand:
  delim: " "
  remove: true
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

output:

{"one":1,"two":"2"}

document-start

with 'expand array' collapseinput-field between doc-start and doc-end into one text event

Type: bool