FlowApps - Reducing Raw Machine Data for Efficient Processing

Handling large volumes of raw machine data can quickly become inefficient and costly if not properly managed. Data reduction is a key technique to ensure that only relevant and meaningful information is processed and transmitted.

Within the Crosser Control Center, we have a FlowApp that demonstrates how to reduce data from multiple machine sources using different techniques. For example, data from an OPC UA Subscriber is refined using a Property Mapper to extract only relevant values, while a Modbus Reader is paired with a Report By Exception module to pass through only changed values. Similarly, sensor data is received via MQTT Sub Client module and is filtered using a Deadband module to ignore minor fluctuations. These approaches ensure that only significant updates are retained from each source.

The reduced data from all sources is then combined into a single compact message, using Array Join and Array To Object modules. By standardizing and renaming sensor data, the flow avoids conflicts and maintains clarity across different sources. This approach highlights how data volume can be significantly reduced while still delivering meaningful, real-time insights.