Workflows with a Dynamic Destination Path

Workflows are designed to orchestrate an automated and iterative execution of ordered tasks. When designing a workflow to orchestrate a process, it requires a source object that can provide access to a directory containing bulk load of source files of the same layout and format. A workflow is designed such that each source file in that particular directory goes through the same data processing.

This requires a source file to be replaced in the run-time until all the source files (of the same layout and format) within a directory are processed and loaded to respective destination files. Now, this further involves the associated destination files to be replaced in the run-time as well, hence making it essential to provide a dynamic destination path. For example, a source directory containing 100 excel source files in this workflow will create 100 destination files after going through the process.

In this document, we will learn to create workflows with a dynamic destination path that directs Centerprise to create a unique destination file for each source file.

In Centerprise 8, dynamic destination path is achieved in two steps:

1. Parameterizing the dataflow through Variables object.

2. Providing a run-time destination path through Expression Transformation object in the workflow.

Parameterizing the Dataflow

In this case, we are orchestrating a simple dataflow that contains customer account details coming in through an Excel Workbook Source. The source data is passed through an Aggregate Transformation and loaded to an Excel Workbook Destination.

Now, there are 100 Excel source files in a directory containing customer account details of 100 customers. Each of these customer account files must go through the same aggregate transformation and loaded to a distinctive Excel destination file.

For this purpose, we will add a Variables object and parameterize the source file path and destination file path in the dataflow.

1. Go to Toolbox > Resources > Variables, and drag and drop the Variables object onto the dataflow designer.

The Variables object is currently empty and needs to be configured.

2. Right-click on the object’s header and select Properties from the context menu. A properties window will open.

3. On the Variable Properties window, create two new fields:

• Excel_Source
• Excel_Destination

Set both fields as Input. For their Default Value, paste the path for customer account file source and customer account file destination in each field, respectively.

Click OK.

4. Double-click on the header of Customer_Account source. An Excel Source Properties window will open. Click Next and you will arrive at the Layout Builder screen. Click Next again and Config Parameters screen will open.

5. On the Config Parameters window, under the Value column for File Path, provide the name of the field enclosed in {….} that points to the source file path in the Variables object in step 3.

6. Repeat step 4 and step 5 now for ExcelDestination object.

Out dataflow has been parameterized. The next step is to create a workflow with a dynamic destination path.

Creating a Workflow

1. Open a new workflow by going to File > New > Workflow.

2. To provide access to the local source directory containing 100 customer account files, go to Toolbox > Sources > File System Item Source and drag and drop the File System Item Source object onto the workflow designer.

3. Right-click on the object’s header and select Properties from the context menu. A configuration window will open up for the FileSystem object.

4. Provide the local directory path in File Location. Apply a filter ***.xls ** and click OK.

5. Right-click on the FileSystem source object’s header and select Loop from the context menu. This will direct Centerprise to run the task in a loop, replacing the source file paths in real-time, ultil all the source files have been processed through it.

6. Go to Toolbox > Workflow Tasks > Run Dataflow and drag and drop the Run Dataflow object onto the workflow designer. Map the header output port of FileSystem source object onto the header input port of Run Dataflow object.

7. Right-click on the Run Dataflow object’s header and select Properties from the context menu. Here, provide the Job Info path for the dataflow that is being orchestrated and click OK.

8. Go to Toolbox > Transformation > Constant Value, drag and drop the Constant Value transformation object onto the workflow designer. Double-click on its header, a Constant Value Map Properties window will open.

9. Paste the path of the folder where Centerprise will load the destination files, in the Constant Value box as shown below. Click OK.

.png)

10. Go to Toolbox > Transformation > Expression, drag and drop the Expression transformation object onto the workflow designer. Double-click on its header, a Layout Builder screen will open.

11. On this Layout Builder screen, create four new fields and set the data type accordingly.

• FullPath
• DestinationDirectory
• FileName
• DestinationFilePath

Click OK.

12. Define field mappings as follows:

• FullPath field under FileSystem source object onto the FullPath field under Expression transformation object.
• FullPath field under FileSystem source object onto the Excel_Source field under RunDataflow task object.
• Value field under Constant Value transformation object onto the DestinationDirectory field under Expression transformation object.
• DestinationFilePath field under Expression transformation object onto the Excel_Destination field under RunDataflow task object.

13. Double-click on the header of Expression transformation object, its Layout Builder screen will open. Write the following expressions for these fields:

• FileName : LSplit(RSplit(FullPath,2,”\”,0),2,”.”,0)
• DestinationFilePath : DestinationDirectory+FileName+”.xls”

Click OK. A dynamic destination file path has been created.

14. Click on Start Workflow icon placed in the toolbar at the top to execute this workflow.

This will create a uqniue destination file for each source file.