Using Workflows for Orchestration

The workflow component in Astera Data Warehouse Builder can be used to orchestrate the process of loading and updating the dimension and fact tables present in a data warehouse. In essence, a workflow allows you to orchestrate a set of tasks either in parallel or in a particular sequence. For example, you can execute a bunch of dataflows through a single workflow.

To learn more about workflows in general, click here.

Use Case

Here, we have a sample dimensional model that we’ve deployed on the server.

The entities present in this model represent the tables in a data warehouse. These tables need to be populated with the source data via dataflows. We’ve designed separate dataflows to populate each entity in this model. As an example, here is the dataflow that has been designed to populate the Customer table.

Similarly, we’ve designed separate dataflows for each of the other entities in the dimensional model.

In this use case, we’ll be designing a workflow to orchestrate the process of loading the fact and dimension tables.

Workflow

Here is the final workflow that we’ve designed to orchestrate the process of loading the dimension and fact tables in the dimensional model that we saw earlier.

The boxes present in the workflow are Run Dataflow objects that you can find in the Workflow Tasks section of the workflow toolbox.

To learn more about this object, click here.

An easier way to get all of the required Run Dataflow objects onto the workflow designer would be to drag and drop each respective dataflow directly from the data warehousing project, which is visible in the Project Explorer.

Once all of the dataflows are on the workflow designer, you can designate a sequence for their execution by creating links between the objects through their output and input ports. In case of dimensions and facts, we know that the dimension tables need to be populated before the fact table(s). Therefore, in this instance, we’ll connect the output ports of each dimension table dataflow to the input port of the fact table dataflow.

Once everything is in place, we can run the workflow via the Start Workflow icon the main toolbar.

An alternative to running this workflow manually is to schedule it for execution via the Job Scheduler. We’ll discuss this in great detail in the next article of this section.

This concludes our discussion using workflows for orchestration in Astera Data Warehouse Builder.