Apache NiFi RecordPath Guide

When we reference an Array field, the value of the field may be an array that contains several elements, but we may want only a few of those elements. For example, we may want to reference only the first element; only the last element; or perhaps the first, second, third, and last elements. We can reference a specific element simply by using the index of the element within square brackets (the index is 0-based). So let us consider a modified version of the Record above:

We can now reference the first element in the addresses array by using the RecordPath /addresses[0]. We can access the second element using the RecordPath /addresses[1]. There may be times, though, that we don’t know how many elements will exist in the array. So we can use negative indices to count backward from the end of the array. For example, we can access the last element as /addresses[-1] or the next-to-last element as /addresses[-2]. If we want to reference several elements, we can use a comma-separated list of elements, such as /addresses[0, 1, 2, 3]. Or, to access elements 0 through 8, we can use the range operator (..), as in /addresses[0..8]. We can also mix these, and reference all elements by using the syntax /addresses[0..-1] or even /addresses[0, 1, 4, 6..-1]. Of course, not all of the indices referenced here will match on the Record above, because the addresses array has only 2 elements. The indices that do not match will simply be skipped.

Similar to an Array field, a Map field may actually consist of several different values. RecordPath gives us the ability to select a set of values based on their keys. We do this by using a quoted String within square brackets. As an example, let’s re-visit our original Record from above:

Now, though, let’s consider that the Schema that is associated with the Record indicates that the address field is not a Record but rather a Map field. In this case, if we attempt to reference the zip using the RecordPath /details/address/zip the RecordPath will not match because the address field is not a Record and therefore does not have any Child Record named zip. Instead, it is a Map field with keys and values of type String. Unfortunately, when looking at JSON this may seem a bit confusing because JSON does not truly have a Type system. When we convert the JSON into a Record object in order to operate on the data, though, this distinction can be important.

In the case laid out above, we can still access the zip field using RecordPath. We must now use the a slightly different syntax, though: /details/address['zip']. This is telling the RecordPath that we want to access the details field at the highest level. We then want to access its address field. Since the address field is a Map field we can use square brackets to indicate that we want to specify a Map Key, and we can then specify the key in quotes.

Further, we can select more than one Map Key, using a comma-separated list: /details/address['city', 'state', 'zip']. We can also select all of the fields, if we want, using the Wildcard operator (*): /details/address[*]. Map fields do not contain any sort of ordering, so it is not possible to reference the keys by numeric indices.

Thus far, we have discussed two different types of filters. Each of them allows us to select one or more elements out from a field that allows for many values. Often times, though, we need to apply a filter that allows us to restrict which Record fields are selected. For example, what if we want to select the zip field but only for an address field where the state is not New York? The above examples do not give us any way to do this.

RecordPath provides the user the ability to specify a Predicate. A Predicate is simply a filter that can be applied to a field in order to determine whether or not the field should be included in the results. Like other filters, a Predicate is specified within square brackets. The syntax of the Predicate is <Relative RecordPath> <Operator> <Expression>. The Relative RecordPath works just like any other RecordPath but must start with a . (to reference the current field) or a .. (to reference the current field’s parent) instead of a slash and references fields relative to the field that the Predicate applies to. The Operator must be one of:

The Expression can be a literal value such as 50 or Hello or can be another RecordPath.

To illustrate this, let’s take the following Record as an example:

Now we can use a Predicate to choose only the fields where the state is not New York. For example, we can use /*[./state != 'NY']. This will select any Record field that has a state field if the state does not have a value of "NY". Note that the details Record will not be returned because it does not have a field named state. So in this example, the RecordPath will select only the homeAddress field. Once we have selected that field, we can continue on with our RecordPath. As we stated above, we can select the zip field: /*[./state != 'NY']/zip. This RecordPath will result in selecting the zip field only from the homeAddress field.

We can also compare the value in one field with the value in another field. For example, we can select the address that is in the person’s preferred state by using the RecordPath /*[./state = /details/preferredState]. In this example, this RecordPath will retrieve the workAddress field because its state field matches the value of the preferredState field.

Additionally, we can write a RecordPath that references the "city" field of any record whose state is "NJ" by using the parent operator (..): /*/city[../state = 'NJ'].

The substring function returns a portion of a String value. The function requires 3 arguments: The value to take a portion of, the 0-based start index (inclusive), and the 0-based end index (exclusive). The start index and end index can be 0 to indicate the first character of a String, a positive integer to indicate the nth index into the string, or a negative integer. If the value is a negative integer, say -n, then this represents the n`th character for the end. A value of `-1 indicates the last character in the String. So, for example, substring( 'hello world', 0, -1 ) means to take the string hello, and return characters 0 through the last character, so the return value will be hello world.

Returns the portion of a String value that occurs after the first occurrence of some other value.

Returns the portion of a String value that occurs after the last occurrence of some other value.

Returns the portion of a String value that occurs before the first occurrence of some other value.

Returns the portion of a String value that occurs before the last occurrence of some other value.

Replaces all occurrences of a String with another String.

Evaluates a Regular Expression against the contents of a String value and replaces any match with another value. This function requires 3 arguments: the String to run the regular expression against, the regular expression to run, and the replacement value. The replacement value may optionally use back-references, such as $1 and ${named_group}

Concatenates all the arguments together.

Joins together multiple values with a separator.

Allows evaluating a RecordPath while anchoring the root context to a child record.

Additionally, this can be used in conjunction with arrays. For example, if we have the following record:

We can evaluate hte following Record paths:

Normally, when a path is given to a particular field in a Record, what is returned is the value of that field. It can sometimes be useful, however, to obtain the name of the field instead of the value. To do this, we can use the fieldName function.

In the above example, the first RecordPath returns two separate field names: "workAddress" and "homeAddress". The second RecordPath, in contrast, returns the value of a "city" field and uses the fieldName function as a predicate. The second RecordPath finds a "city" field whose parent does not have a name that begins with "work". This means that it will return the value of the "city" field whose parent is "homeAddress" but not the value of the "city" field whose parent is "workAddress".

Converts a String to a date. For example, given a schema such as:

and a record such as:

The following record path would parse the eventDate field into a Date:

toDate( /eventDate, "yyyy-MM-dd’T’HH:mm:ss’Z'")

toDate( /eventDate, "yyyy-MM-dd’T’HH:mm:ss’Z'", "GMT+8:00")

Converts a value to a String, using the given character set if the input type is "bytes". For example, given a schema such as:

and a record such as:

The following record path would parse the bytes field into a String:

toString( /bytes, "UTF-8")

Converts a String to byte[], using the given character set. For example, given a schema such as:

and a record such as:

The following record path would convert the String field into a byte array using UTF-16 encoding:

toBytes( /s, "UTF-16")

Returns the first value from the given arguments that is non-null. For example, given a record such as:

The following record path would return "John Doe":

coalesce(/id, /name)

Given the record:

The same record path would return "1234".

Given the record:

The record path would return null.

Given the record:

The record path would return the String "null". Note here the very important difference in that the id field does not have a null value but rather the value of the field is the literal string "null".

Given the record:

The record path would return null. Given that the id field is not present, it is treated as a null value.

Given the record:

The record path would return "1234". However, the record path coalesce(/name, /id) would return "John Doe" because both fields given are non-null, so the coalesce function returns the first value that is referenced in its arguments, not the first value that is encountered in the Record itself.

Converts a Date to a String in the given format with an optional time zone. The function defaults to the system local time zone when the second argument is not provided.

The first argument to this function must be a Date or a Number, and the second argument must be a format String that follows the Java DateTimeFormatter, and the third argument, optional, must be a format String that either an abbreviation such as "PST", a full name such as "America/Los_Angeles", or a custom ID such as "GMT-8:00"

For example, given a schema such as:

and a record such as:

The following record path expressions would format the date as a String:

In the case where the field is declared as a String, the toDate function must be called before formatting.

For example, given a schema such as:

and a record such as:

The following record path expression would re-format the date String:

Removes whitespace from the start and end of a string.

and a record such as:

The following record path expression would remove extraneous whitespace:

Change the entire String to upper case

and a record such as:

The following record path expression change lower case letters to upper case:

Changes the entire string to lower case.

and a record such as:

The following record path expression change upper case letters to lower case:

Converts a String or byte[] using Base64 encoding, using the UTF-8 character set. For example, given a schema such as:

and a record such as:

The following record path expression would encode the String using Base64:

Decodes a Base64-encoded String or byte[]. For example, given a schema such as:

and a record such as:

The following record path expression would decode the String using Base64:

JSON Stringifies a Record, Array or simple field (e.g. String), using the UTF-8 character set. For example, given a schema such as:

and a record such as:

The following record path expression would convert the record into an escaped JSON String:

Converts a stringified JSON element to a Record, Array or simple field (e.g. String), using the UTF-8 character set.

This function takes up to three arguments:

Note: RecordSetWriters will not be able to serialise Maps, so fields may be omitted if not converted to Records. For example, using unescapeJson to replace the root ("/") of a Record in UpdateRecord without using the Record conversion, would result in an empty output because the RecordSetWriter is unable to match the resultant Map content to a RecordSchema.

For example, given a schema such as:

and a record such as:

The following record path expression would populate the record with unescaped JSON fields:

Given a record such as:

The following record path expression would return:

Given a record such as:

The following record path expression would return:

Given a record such as:

The following record path expression would return:

Note that the target schema must be pre-defined if the unescaped JSON is to be set in a Record’s fields - Infer Schema will not currently do this automatically.

Converts a String using a hash algorithm. For example, given a schema such as:

and a record such as:

The following record path expression would hash the String using one of these, [SHA-384, SHA-224, SHA-256, MD2, SHA, SHA-512, MD5] algorithms.

Prepends characters to the input String until it reaches the desired length.

and a record such as:

The following record path expression would prepend '@' characters to the input String:

Appends characters to the input String until it reaches the desired length.

and a record such as:

The following record path expression would append '@' characters to the input String:

Inserts a UUID v5 into the target field.

There are two ways to use this function: with or without a namespace. Given this schema:

and a record such as:

There are two options for using this function:

uuid5(/input)

uuid5(/input, /id_ns)

The first option will generate a simple UUID v5 that does not use a namespace in the generation process. The second will take the value of the supplied record path and use it as the namespace.

Please note that the namespace must always be a valid UUID string. An empty string, another data type, etc. will result in an error. This is by design because the most common use case for UUID v5 is to uniquely identify records across data sets.

Returns the count of the number of elements. This is commonly used in conjunction with arrays. For example, if we have the following record:

We could determine the number of elements by using the count function. Using:

Would yield a value of 2. We could also use this as a filter, such as:

Which will return the id element with a value of 1234.

Creates a map of Strings with the given parameters. For example, if we have the following record:

We could use the UpdateRecord processor with

And that would give us something like:

This function requires an even number of arguments and the record paths must represent simple field values.

Returns true if a String value contains the provided substring, false otherwise

Evaluates a Regular Expression against the contents of a String value and returns true if the Regular Expression exactly matches the String value, false otherwise. This function requires 2 arguments: the String to run the regular expression against, and the regular expression to run.

Returns true if a String value starts with the provided substring, false otherwise

Returns true if a String value ends with the provided substring, false otherwise

Inverts the value of the function or expression that is passed into the not function.

Returns true if the provided value is either null or is an empty string.

Returns true if the provided value is either null or is an empty string or a string that consists only of white space (spaces, tabs, carriage returns, and new-line characters).