Uploading data to AWS S3 with the S3 Bucket Writer Module
Amazon S3 is a common choice for scalable cloud storage in data integration, analytics, and industrial IoT projects. In practice, though, uploading data can get tricky when you need to handle multiple formats, keep files organized, and build dynamic folder structures as solutions grow. The S3 Bucket Writer module is designed to make this easier by offering several upload options and flexible key generation, so you can store data in S3 in a consistent and structured way.
The module supports different upload modes depending on what you’re sending. MultipleFiles uploads several files based on a list of file paths, while SingleFile uploads one file at a time. If your data isn’t already stored as a file, you can use ByteArray to upload raw binary content directly, or String to upload text content such as logs, JSON messages, or generated reports. There’s also an option to delete local files after a successful upload, which can help reduce disk usage on the machine running the flow.
To keep uploads organized, the module builds the S3 object key using a combination of Prefix and Key. This lets you create a folder-like structure in the bucket (even though S3 stores objects in a flat hierarchy). You can also use dynamic templates in the prefix and key, so uploads can automatically be grouped by things like date, username, device ID, or other values coming from the message.
