Description:

The MSSQL Input File step is an ETL source component that imports relational data from Microsoft SQL Server into a FlowWright ETL definition.

Instead of reading data from files, this step connects directly to a SQL Server database, executes a user-defined SQL statement, and converts the query results into an ETL dataset. This enables organizations to extract live business data from operational systems for reporting, migration, integration, cleansing, and analytics.

The step is configured by specifying:

• The SQL Server connection
• The SQL query to execute
• The column schema used to map query results into the ETL dataset

During execution, the ETL engine establishes a connection to SQL Server, executes the configured SQL statement, retrieves the result set, applies the selected column schema, and creates an in-memory dataset that becomes the source for downstream ETL components.

Because the SQL statement is fully configurable, the step supports simple queries, filtered datasets, joins, aggregations, stored views, and complex reporting queries.

The MSSQL Input File step is typically the first component in an ETL pipeline.

The step supports:

• Microsoft SQL Server connectivity
• Custom SQL queries
• Column schema mapping
• Enterprise database integration
• Live data extraction
• Integration with FlowWright ETL pipelines

Typical business uses include:

• Customer master extraction
• ERP reporting
• CRM synchronization
• Financial reporting
• Inventory analysis
• Data warehouse staging
• Business intelligence data preparation

Usage:

The MSSQL Input File step is typically the first component in an ETL definition.

During execution:

1. The configured SQL Server connection is established.
2. The SQL statement is executed.
3. The returned result set is retrieved.
4. The selected column schema is applied.
5. The resulting ETL dataset is passed to downstream ETL components.

A typical ETL pipeline might look like this:

Typical usage scenarios include:

• Importing customer master records
• Extracting ERP transaction data
• Synchronizing CRM information
• Preparing operational reporting datasets
• Migrating legacy system data
• Creating business intelligence datasets
• Feeding downstream enterprise integrations

The imported dataset can immediately be validated, transformed, filtered, grouped, aggregated, sorted, or exported to files and databases.

Prerequisite:

The ETL process engine service should be running to execute the ETL definitions.

Example:

Let’s build and execute the “clsInputMSSQLDBDef” example.          

• Create a new ETL definition named “clsInputMSSQLDBDef” and open it in designer mode. 
• Drag “clsInputMSSQLDB, clsTransformCase, and clsOutputMSSQLDB” steps to the canvas. 
• Connect the dots between the “clsInputMSSQLDB” and other steps, as shown above.
• Click the "clsInputMSSQLDB" step to configure its "Required" properties. Provide a name for the step. Select the SQL connection from the drop-down list. Enter the SELECT SQL statement. Select the data schema from the drop-down list that matches the SQL result set. Click the Save button. Click here to learn about ETL Data Schema Designer.

• The data schema is provided here for reference. 

• Click the "clsTransformCase" step to configure its "Required" properties. Enter the step name. Click the button to configure the case transform list. Click the Save button. 

• Click the button to configure the case transform list. The pop-up window opens for configuration. Click the Add Row (+) button to insert an empty row. Select the column and case type from the drop-down list. Click the Save button. Click the Add Row button to insert multiple columns for case transform. 

• Click the "clsOutputMSSQLDB" step to configure its "Required" properties. Provide a name for the step. Select the SQL connection from the drop-down list. Specify the table name to create and store in the connection's database. Select “Yes” to create a new table. Select the primary key column from the drop-down list. Select the database operation from the drop-down list. Click the Save button. 

• Select the database operation from the drop-down list. You may configure the step to perform Insert, Update, Insert and Update, Delete, or Replace All. Click the Save button. 

• The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

• Save the ETL definition, create a new instance, and execute it. Render the ETL instance. Click the process step to view its properties. The “clsInputMSSQLDB” step should input the MSSQL DB file for transformation. 

• The source SQL record set used for case transformation is provided here for reference.  

• The case-transformed SQL result set is provided here for reference.  

Tips:

• Use SQL queries that return only the required columns to improve ETL performance.
• Apply filtering in the SQL statement whenever possible to reduce unnecessary data transfer.
• Ensure the selected column schema matches the query result set.
• Use database views or stored procedures when business logic is maintained within SQL Server.
• Validate imported data before performing downstream transformations.
• Test SQL statements independently before incorporating them into production ETL definitions.
• Combine this step with Validate Data, Transform Text Case, Transform Date by Format, Sorting Transform, Filter with Conditions, Group By Transform, MSSQL Output File, CSV Output File, or Excel Output File to build complete enterprise ETL workflows.

Notes:

• A valid Microsoft SQL Server connection must be configured before using this step.
• The SQL statement determines the structure and contents of the generated ETL dataset.
• The selected column schema must correspond to the columns returned by the SQL query.
• Database permissions must allow execution of the configured SQL statement.
• SQL syntax errors, connectivity issues, or schema mismatches may prevent successful execution.
• Because this is an ETL source component, it has no incoming connections and serves as the starting point of the ETL pipeline.
• Query execution statistics, connection information, and any database errors are recorded in the ETL execution log.

MSSQL Input File vs. CSV Input File:

Both components import data into FlowWright ETL, but they retrieve information from different sources.

As a general guideline:

• Use MSSQL Input File when extracting live operational data directly from Microsoft SQL Server.
• Use CSV Input File when importing externally supplied files or datasets generated by other systems.

Definition Sample:

You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.

Note: Verify and complete any missing configuration after importing the sample, including:

• SQL Server connection
• SQL statement
• Column schema selection
• Database permissions
• Query validation
• Downstream transformation mappings
• Destination component configuration

After verifying the configuration, save the ETL Definition before execution.

Click here to download the sample file.