Description:

The Remove Columns Transform step removes one or more selected columns from an ETL dataset during execution. This transformation is useful for reducing dataset size, eliminating unwanted or sensitive information, simplifying downstream processing, and ensuring that only required fields are included in the final output.

The columns to be removed are configured through the Configure Columns List property. During execution, the selected columns are excluded from the dataset, while all remaining columns and their corresponding data continue through the ETL pipeline unchanged.

The Remove Columns Transform step is typically used after importing data or after intermediate transformations have completed, but before validation, aggregation, reporting, or loading data into destination systems.

The step supports:

• Removing one or more columns
• Schema simplification
• Dataset optimization
• Sensitive data exclusion
• Configurable column selection
• Enterprise ETL workflows
• Integration with FlowWright ETL pipelines

This step can be used for:

• Removing temporary processing columns
• Eliminating confidential fields
• Preparing datasets for external systems
• Simplifying reporting datasets
• Reducing the exported file size
• Data migration projects
• Enterprise data integration

Usage:

The Remove Columns Transform step is typically inserted after importing or transforming data and before downstream components that consume the finalized dataset.

During execution, the transformation removes the configured columns from the ETL schema and passes the updated dataset to subsequent ETL components. Only the selected columns are removed; all remaining column values and records remain unchanged.

A typical ETL pipeline might look like this:

Typical usage scenarios include:

• Removing internal processing fields
• Excluding confidential information before export
• Eliminating unused columns from imported datasets
• Preparing simplified datasets for reporting
• Reducing the schema before database loading
• Removing obsolete legacy fields
• Optimizing data exchanged with external applications

After the transformation completes, downstream ETL components process only the remaining columns.

Prerequisite:

The ETL process engine service should be running to execute the ETL definitions.

Example:

Let’s build and execute the “clsRemoveColumnsTransformDef” example.          

• Create a new ETL definition named “clsRemoveColumnsTransformDef” and open it in designer mode. 
• Drag “clsInputCSV, clsRemoveColumnsTransform, and clsOutputCSV” steps to the canvas. 
• Connect the dots between the “clsInputCSV” and other steps, as shown above.
• Define a variable or a global to hold the virtual path. 
• Click the "clsInputCSV" step to configure its "Required" properties. Provide a name for the step. Specify the path to the input CSV file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. Click here to learn about the ETL Data Schema Designer.

• Click the "clsInputCSV" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as follows. This function helps to understand the schema at a glance.  

• Click the "clsRemoveColumnsTransform" step to configure its "Required" properties. Provide a name for the step. Click the button to specify which columns to remove during the transformation. Click the Save button. 

• Click the "clsRemoveColumnsTransform" step to configure its "Required" properties. Click the button to open the transform list. A pop-up window will appear for configuration. Click the Add Row (+) button to insert an empty row. Select the column from the drop-down list. Click the Save button. You may add multiple columns for transformation by using the Add Row button. 

• Click the "clsOutputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output CSV file on the application server. Click the Save button. 

• Click the "clsOutputCSV" step to configure its "Optional" properties. Provide a name for the step and specify the variable or global reference to hold the virtual output path. Click the Save button. 

• The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

• Save the ETL Definition. Execute the ETL process. During execution, the specified columns are removed from the dataset. The resulting dataset is then passed to downstream ETL components for validation, transformation, reporting, or export.

Tips:

• Remove unnecessary columns as early as practical to reduce memory usage and simplify downstream processing.
• Ensure removed columns are not required by later ETL transformations.
• Remove confidential or personally identifiable information before exporting data to external systems.
• Review downstream mappings whenever columns are removed from the schema.
• Use descriptive ETL schemas to make column selection easier during configuration.
• Test the ETL definition after modifying the dataset structure.
• Combine this transform with Rename Column, Set Field Value, Validate Data, Sorting Transform, Remove Duplicates, Aggregate Transform, or JSON Output File components to build efficient ETL workflows.

Notes:

• This transformation removes only the selected columns and does not modify the remaining data values.
• Multiple columns can be removed within a single transformation.
• Columns selected for removal must exist in the ETL schema.
• Downstream ETL components cannot reference columns that have been removed.
• Removing required columns may cause subsequent transformations or destination mappings to fail.
• Configuration errors or invalid schema references are reported in the ETL execution log.
• This transform is commonly used in data migration, reporting, system integration, business intelligence, regulatory compliance, and enterprise data preparation workflows.
• Validate the resulting schema before deploying ETL definitions to production environments.

Definition Sample:

You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.

Note: Verify and complete any missing configuration after importing the sample, including:

• Column removal configuration
• ETL schema validation
• Downstream transformation mappings
• Destination component configuration
• Environment-specific settings

After verifying the configuration, save the ETL Definition before execution.

Click here to download the sample file.