Description:
This step groups identical values into summary rows during the transformation.
The Group By Transform step groups records within an ETL dataset using one or more selected columns and applies aggregation operations to one or more value columns. This transformation enables the creation of summarized datasets by consolidating records that share common values.
Grouping criteria are configured through the Select Column property, while aggregation operations are defined through the Select Operation property. Common aggregation functions include operations such as Sum, Count, Average, Minimum, and Maximum, depending on the configured value column options.
The Group By Transform step is commonly used after data has been cleansed and validated, but before reporting, exporting, or loading summarized information into data warehouses or analytical systems.
The step supports:
- Single-column grouping
- Multi-column grouping
- Configurable aggregation operations
- Summary dataset generation
- Business reporting
- Data warehousing
- Integration with FlowWright ETL pipelines
This step can be used for:
- Sales summaries
- Financial reporting
- Inventory analysis
- Customer analytics
- Departmental reporting
- Business intelligence
- Enterprise ETL automation
Inputs
- Select Column – Select one or more columns that define how records should be grouped. Records with identical values in the selected columns are consolidated into a single group.
- Select Operation – Configure one or more aggregation operations for the selected value columns. Each operation specifies the target column and the aggregation method to apply.
Returns
- The Group By Transform step does not define explicit return paths.
Note: Once the grouping and aggregation operations have completed successfully, ETL processing automatically continues to the next connected component. Configuration or execution errors are recorded in the ETL execution log.
Usage:
The Group By Transform step is typically inserted after validation, cleansing, sorting, or duplicate removal operations, and before reporting or exporting summarized results.
During execution, the transformation groups records according to the configured grouping columns. Aggregation operations are then applied to each group, producing a summarized dataset containing one record per group.
A typical ETL pipeline might look like this:
Typical grouping scenarios include:
- Summarizing sales by region
- Counting employees by department
- Calculating monthly revenue
- Averaging customer order values
- Determining inventory totals
- Aggregating support tickets by priority
- Producing executive reporting datasets
The resulting summarized dataset can then be exported, reported, or used by downstream analytical processes.
Prerequisite:
The ETL process engine service should be running to execute the ETL definitions.
Example:
Let’s build and execute the “clsGroupByTransformDef” example.
- Create a new ETL definition named “clsGroupByTransformDef” and open it in designer mode.
- Drag “clsInputCSV, clsGroupByTransform, and clsOutputCSV” steps to the canvas.
- Connect the dots between the “clsInputCSV” and other steps, as shown above.
- Define a variable or a global to hold the virtual output path.
- Click the "clsInputCSV" step to configure its "Required" properties. Provide a name for the step. Specify the path to the input CSV file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. Click here to learn about the ETL Data Schema Designer.
- Click the "clsInputCSV" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as shown below. This function helps to understand the schema at a glance.
- Click the "clsGroupByTransform" step to configure its "Required" properties. Provide a name for the step. Click the button to configure the columns to transform. Then click the Save button.
- Click the "clsGroupByTransform" step to configure its "Required" properties. Click the column button to open the transform list. A pop-up window will appear for configuration. Click the Add Row (+) button to insert an empty row. Select the “Group By” column from the drop-down list. Click the Save button. You may add multiple columns for transformation by using the Add Row button.
- Click the "clsGroupByTransform" step to configure its "Required" properties. Click the operation button to open the transform list. A pop-up window will appear for configuration. Click the Add Row (+) button to insert an empty row. Select the “Value” column from the drop-down list. Select the “Operator” to summarize similar rows. Select the “Out” column from the drop-down list. You may also type a new Out column name and press Enter to add it. Click the Save button. You may add multiple columns for transformation by using the Add Row button.
- Click the "clsOutputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output CSV file on the application server. Click the Save button.
- Click the "clsOutputCSV" step to configure its "Optional" properties. Provide a name for the step. Provide a variable or global reference to store the virtual output path. Click the Save button.
- The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.
- Save the ETL definition, create a new ETL instance, and execute. Render the ETL instance. Click the ETL step to view its properties. The “clsGroupByTransform” step should trim the text as configured and write it to the output file. The summarized dataset is then available for reporting, dashboards, exporting, or additional ETL transformations. The variable/global holds the output file’s virtual path as shown below.
Tips:
- Validate and cleanse data before performing grouping operations.
- Select grouping columns that represent meaningful business entities.
- Use aggregation functions appropriate for the type of data being summarized.
- Remove duplicate records before grouping when duplicate values should not affect the aggregated results.
- Sort grouped output afterward if a specific presentation order is required.
- Test grouping logic using representative datasets before deploying to production.
- Combine this transform with Validate Data, Sorting Transform, Remove Duplicates, Pivot Transform, Aggregate Transform, Output Column Transform, JSON Output File, or SQL Output components to build comprehensive ETL reporting solutions.
Notes:
- Records sharing identical values in the selected grouping columns are consolidated into a single output record.
- Multiple grouping columns can be configured to produce hierarchical summaries.
- Multiple aggregation operations may be configured for different value columns.
- The selected grouping and value columns must exist within the ETL schema.
- Large datasets with complex grouping operations may require additional processing time and memory.
- Invalid aggregation configurations or schema mappings may result in ETL execution errors.
- This transform is commonly used for financial reporting, operational dashboards, business intelligence, inventory management, customer analytics, and enterprise reporting.
- Verify aggregated results before loading summarized datasets into production reporting systems.
Definition Sample:
You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.
Note: Verify and complete any missing configuration after importing the sample, including:
- Grouping column selection
- Aggregation operation configuration
- Value column mappings
- ETL schema validation
- Downstream destination mappings
- Environment-specific settings
After verifying the configuration, save the ETL Definition before execution.