Skip to main content Skip to footer

Analytics Module Overview

This document presents the analytics modules available in the Crosser Module library. Analytics modules are used to transform and harmonize message formats from different sources, as well as operate on the actual message content. The modules are presented in groups with related functionality.

Data Transformation

Property Mapper

The Property Mapper module is the Swiss army knife for modifying message structure.

Use it to:

  • Rename properties
  • Move properties to new hierarchy levels
  • Remove properties
  • Add new properties with static values
  • Copy existing property values to new properties

Working with Arrays

Arrays are common both on inputs and outputs, but may not be the optimal internal format when applying streaming processing. For example, to apply processing on individual array elements in a streaming pipeline you need to split up the array into individual messages. Another example is when you want to write data into columns in a database, then you need an object with key/value pairs that can be mapped against the database columns.

There are several modules in the Crosser Module library that help you convert back and forth between these different formats. In this environment an array is typically an array of objects, i.e. each element contains multiple values and even hierarchical message structures.

Names in bold indicate the most common array modules.

 

Name

Description

Input

Output

Array Split

Breaks up an array into individual messages

Array

Messages

Array Join

Combine a stream of messages into an array, by time or message count (opposite of Array Split)

Messages

Array

Array to Object

Convert an array into an object with key/value pairs. Each item in the array must have a property holding the key value and another property with the value for that key

Array

Message

Object to Array

Convert an object with key/value pairs into an array. Each element in the array will have a property containing the name of the key and another with the value belonging to that key (opposite of Array To Object)

Message

Array

Array Filter

Filter out some elements from an array by matching against one ore several conditions.

Array

Array

Array GroupBy

Split an array into sub-arrays by grouping elements based on the value of a property.

Array

Arrays

Array Property Pick

Selects some of the properties available in each array element. A list of properties to pick must be provided.

Array

Array

Array Property Omit

Remove some of the properties available in each array element. A list of properties to remove must be provided.

Array

Array

Array Sort By List

Sort array elements by comparing the values on a specified property against an ordered list of values provided. Typically used together with Array Property Get to create lists of values where the position in the array is used to identify a source.

Array

Array

Array Property Get

Select the values from a selected property. The output is an array with only values. Useful when feeding e.g. ML models where the position in the array is used to identify a source, rather than key/value mappings.

Array

Array of Values

Join

This is a variant of the Array Join module that ensures that data for a specified number of sources are always available in each output. If no data has been received for a specific source within the given time period an output will be generated based on the strategy selected. This module is more complex to set up but is useful in cases where you must ensure that each output has a value for each source, e.g. when feeding an ML model.

Messages

Array

Working with Strings

Name

Description

Input

Output

Text Template

Generates text messages based on a template provided in the settings. Template syntax can be used to insert data from the message together with static text.

Message

String

String Replace

Replace a substring with another substring.

String

String

String Substring

Select a substring based on start and stop positions

String

String

CSV Line Parser

Break up a line of text (string) into substrings based on a delimiter, e.g. “,”. The output is an object with one property per substring. These properties are named based on the position in the string, e.g. col1, col2...

String

Object with substrings

CSV Text Parser

Same operation as the CSV Line Parser but operating on multi-line strings. The output is an array with one element per line. Each element has the same format as with the CSV Line Parser.

String

Array of objects with substrings

Format Conversions

Messages sent between modules use a custom .NET data type (FlowMessage) that supports dynamic structures with hierarchies of objects and arrays which can contain basic .NET data types. Sometimes, e.g. in the debug window, messages may look like Javascript objects (JSON) but that is only for presentation purposes. JSON is never used internally. When communicating with external systems you may get data in JSON or XML format, or the output is expected to be in one of these formats. To work with these types of data you must convert between the internal format and the external formats. To do that you use these conversion modules.

 

Name

Description

JSON

Converts to and from JSON strings. If the input is a string it will be interpreted as JSON and the output is a FlowMessage.

If the input is a FlowMessage it will be converted into a JSON string.

This module operates on a selected property of the incoming message and the result can be assigned to a new property.

XML

Same as the JSON module but working with XML text.

Time Stamp

Convert DateTime values into another format. Predefined formats are ISO8601 and  Unix timestamp in seconds or milliseconds. Custom output formats can also be defined.

If no input is specified the module will add the current system time on the output.

Base64 Encode

Convert a binary array or a string into a base64 encoded string. For string inputs the output can be set to use base64url format, to create URL safe strings.

Base64 Decode

Decode a base64 encoded string into a byte array or string

Intelligent Logic

Operating on message Data

The following modules will create new data by processing the values in messages. They all work on numerical values and the result can either be assigned to a new property or the original value can be overwritten.

Name

Description

Aggregate

Calculate average, min and max on a selected property from a group of messages. The group of messages to use can either be selected by specifying a time period, or a number of messages. Calculations can be performed independently for messages coming from different sources by specifying a property that indicates the source. At the end of each interval a message is delivered for each source with the calculated statistics.

Math

Execute generic mathematical expressions on message data. The expressions are specified using template syntax where message data can be inserted into the mathematical expression, e.g. ‘Abs({data.cur_temp} - {data.prev_temp})’.

Range Classifier

Add a text classification based on ranges of values on a selected property, e.g. ‘Low’, ‘Med’, ‘High’

Scale

Scale a value on a selected property ‘(value*scale)-offset’, e.g. to convert temperature values from Farenheit to Celsius. Note: This operation can also be performed with the Math Expression module.

Smooth

Apply an exponential smoothing filter on a property value, e.g. to remove noise.

Statistics

Same as Aggregate but operating on a rolling window of messages, i.e. an output message  is generated for each input message. This module will also calculate the standard deviation of the values.

Toggle

Toggle the value of a boolean property, or toggle the value of an internal variable and add to the output message

Code Modules

With the code modules you can apply custom processing of messages using either C# or Python code. The code can either be entered directly in the settings UI, or code files can be uploaded to the Resource library and then referenced from within the modules.

 

Name

Description

Csharp

Execute C# code. The code is compiled at runtime. Standard .NET libraries can be used but not 3rd party libraries.

IronPython

Run Python code using the IronPython interpreter which runs in .NET. Standard libraries can be used but not 3rd party libraries. Supports Python 2.7 code.

Python Bridge

Run python code in a standard Python 3.7 environment outside of .NET. On docker this module is only supported with images that includes the external python environment (tagged with ‘latest-python’). On Windows the python environment must be installed separately and the local node configuration must be updated accordingly. Standard and 3rd party libraries can be used, including ML frameworks.

Counter Modules

Name

Description

Message Counter

Counts the values seen in a selected message property over a time period. At the end of the period the number of occurrences of each value is listed together with the relative count of each value. Useful for KPI calculations, such as Yield.

If no source is specified the total number of messages received is reported.

Time Counter

Counts the time spent in different states by looking at the values received on a specified property and the time between value changes. At the end of the period the total time spent in each state (value) is reported together with the relative times. Useful for KPI calculations such as  Availability.

Timeout

Measures the time since the last update on a selected property. If no update is seen within a specified timeout period a message is sent out. Use this module to monitor expected traffic patterns. For example, if you know that data should arrive every second this module can be used to trigger an alert if no update is seen within 2 seconds.

By keeping track of individual property values the module can operate on multi-source streams and trigger an alert as soon as one a value is missing from one of the sources.

Storage Modules

Sometimes you need to store message data temporarily and then these modules will come in handy.

Name

Description

Key Value Get/Set/Delete

The Node has internal key/value stores which can be accessed using the Key Value modules. You can have multiple named stores and they can either be kept in memory, or on disk (persistent). If stored on disk these stores can also be shared between Flows.

Memory Buffer

This module will keep messages until acknowledged by an external signal. It is designed to work with any output module to keep messages until successfully delivered. Output modules produce a ‘crosser.success’ value indicating if the external delivery was successful or not. This value can be fed back to the Memory Buffer module. The size of the buffer as well as retry strategies can be defined.

Note: All modules have built-in store-and-forward capabilities similar to what the Memory Buffer module can do. You only need to use this module when custom logic is required for controlling the resend operation.

Condition Logic

Message filters

In addition to the below modules each module has a configurable message filter, which can be found on the “Common” tab. By default modules will process every message they receive. By adding a message filter the module will only process messages matching some criteria. Messages not matching the filter criteria can either be dropped, or passed on untouched.

Message filters can be set to match conditions on numeric property values (=, <, >), string property values (contains, does not contain, equal/not equal, isNull, isNotNull, regex) and boolean values (isTrue, isFalse). Multiple conditions can be used.

Filtering modules

In addition to the message filters above the following modules can be used to control which messages that are passed on to the next module(s), based on different criteria.

 

Name

Description

Range Filter

Only let through messages where a selected value is either within or outside of a specified range. The same effect can be achieved by using message filters.

Report By Exception

Only let through messages when the value of a specified property changes. Can be used to remove duplicates.

Split

Split a stream of messages into multiple paths by defining conditions on message data. Multiple conditions can be specified for each path. Each path will get a separate output on the module.

If Else Condition

Split a stream of messages based on whether a condition is true (if) or false (else).

Throttle

Limit the rate of messages, e.g. only let through a maximum of one message per second. Can also be used to spread out messages in time by enforcing a minimum delay between messages, which can be used to smooth out bursty traffic.