Description:

The Duplicate Column Value step creates one or more duplicate columns by copying the values from existing source columns into newly configured destination columns. The original columns remain unchanged, allowing downstream ETL transformations to modify the duplicated values without affecting the source data.

The duplication is configured via the Configure Column property, where ETL designers specify the source and corresponding destination columns for each duplication operation.

This transform is commonly used when the same data must be reused for multiple business purposes. For example, one copy of a column may be retained in its original format while the duplicated column is reformatted, masked, concatenated, converted, or used for reporting.

The Duplicate Column Value step is typically placed after data validation and before additional transformation steps that modify the duplicated columns.

The step supports:

• Column value duplication
• Multiple source-to-destination mappings
• Non-destructive data transformation
• Schema enrichment
• Parallel data processing
• Enterprise ETL workflows
• Integration with FlowWright ETL pipelines

This step can be used for:

• Creating backup copies of important fields
• Preparing data for multiple destinations
• Preserving original values before formatting
• Supporting auditing requirements
• Data migration projects
• Business intelligence preparation
• Enterprise system integration

Usage:

The Duplicate Column Value step is typically inserted after validation and schema preparation, but before transformations that modify data values.

During execution, the configured source column values are copied into their corresponding destination columns. Both the original and duplicated columns remain available to downstream ETL components.

A typical ETL pipeline might look like this:

Typical duplication scenarios include:

• Preserving original customer names before formatting
• Creating backup copies of account numbers before masking
• Duplicating dates for multiple formatting requirements
• Preparing separate reporting and operational fields
• Maintaining original identifiers during data enrichment
• Supporting audit and compliance requirements
• Creating alternate values for downstream integrations

The duplicated columns can then be independently modified without affecting the original source data.

Prerequisite:

The ETL process engine service should be running to execute the ETL definitions.

Example:

Let’s build and execute the “clsDuplicateColumnTransformDef” example.          

• Create a new ETL definition named “clsDuplicateColumnTransformDef” and open it in designer mode.
• Drag “clsInputCSV, clsDuplicateColumnTransform, and clsOutputCSV” steps to the canvas.
• Connect the dots between the “clsInputCSV” step and other steps, as shown above.
• Define a variable or a global to hold the file path.
• Click the "clsInputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the input CSV file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. 
   

• Click the "clsInputCSV" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as shown below. This function helps to understand the schema at a glance.  

• Click the "clsDuplicateColumnTransform" step to configure its "Required" properties. Provide a name for the step. Click the button to configure the columns to transform. Then click the Save button. 

• Click the "clsDuplicateColumnTransform" step to configure its "Required" properties. Click the button to open the transform list. A pop-up window will appear for configuration. Click the Add Row (+) button to insert an empty row. Select the “In” text data type column from the drop-down list. Select the “Out” text data type column from the drop-down list. You may also type a new Out column name and press Enter to add it. Click the Save button. You may add multiple columns for transformation using the Add Row button. 

• Click the "clsOutputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output CSV file on the application server. Click the Save button. 

• Click the "clsOutputCSV" step to configure its "Optional" properties. Provide a name for the step. Provide a variable or global reference to store the virtual output path. Click the Save button. 

• The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

• Save the ETL definition, create a new ETL instance, and execute. Render the ETL instance. Click the ETL step to view its properties. The “clsDuplicateColumnTransform” step should duplicate column values as configured for the output file. The duplicated columns are now available for additional transformations while the original data remains unchanged. The variable or global holds the output file’s virtual path, as shown below. 

Tips:

• Duplicate columns before applying formatting, masking, or conversion operations if the original values must be preserved.
• Use meaningful destination column names to distinguish original and duplicated values.
• Avoid creating unnecessary duplicate columns, as they increase dataset size and memory usage.
• Verify that destination column names do not conflict with existing schema definitions.
• Review downstream mappings after introducing duplicated columns.
• Test duplication logic using representative datasets before deploying to production.
• Combine this transform with Rename Column, Set Field Value, Transform Using Regular Expression, Validate Data, Output Column Transform, JSON Output File, or SQL Output components to build flexible ETL workflows.

Notes:

• The original source columns remain unchanged after duplication.
• Multiple source-to-destination mappings can be configured within a single transformation.
• Each duplicated column contains an exact copy of the source column values at the time the transformation executes.
• The duplicated columns are included in the ETL schema and are available to all downstream components.
• Invalid source column references or duplicate destination column names may result in ETL execution errors.
• Duplicating large numbers of columns may increase memory consumption during ETL execution.
• This transform is commonly used for auditing, data migration, reporting, data masking, system integration, business intelligence, and enterprise data preparation.
• Verify downstream destination mappings whenever additional columns are introduced into the ETL schema.

Definition Sample:

You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.

Note: Verify and complete any missing configuration after importing the sample, including:

• Source-to-destination column mappings
• Destination column naming
• ETL schema validation
• Downstream transformation mappings
• Destination component configuration
• Environment-specific settings

After verifying the configuration, save the ETL Definition before execution.

Click here to download the sample file.