Using Data Quality Rules

Data Quality Rules

A Data Quality Rules transformation can be used to apply one or more conditions, which are called data quality rules, against incoming records. Records that do not meet the data quality rule criteria will be assigned the ‘Error’ status, and may be optionally excluded from processing by the downstream objects.

To add a Data Quality Rules transformation, drag the Data Quality Rules object from the Transformations group in the Flow toolbox and drop it on the dataflow.

An example of what a Data Quality Rules object might look like is shown below.

To configure the properties of a Data Quality Rules object after it was added to the dataflow, right-click on it and select Properties from the context menu. The following properties are available:

Meta Object Builder screen:

Meta Object Builder screen allows you to add or remove fields in the field layout, as well as select their data type.

Note: To quickly add fields to the layout, drag and drop the node Output port of the object whose layout you wish to replicate into the node Input port of the Data Quality Rules object. The fields added this way show in the list of fields inside the node and as well as in the Meta Object Builder.

Data Quality Rules screen provides the interface to manage your data quality rules.

Click the icon to create a new data quality rule.

Type a descriptive name for the rule in the Description field.

Using the Attach Rule to the Field dropdown, select which field will be returned in the transfer Profile, should the rule fail.

In the Expression input, enter the expression making the rule. For example, LTV > 60 and LTV <= 80. Or click … to open Expression Builder, a tool that allows you to visually build your rule using Record tree and Intellisense.

Click Compile to check for any syntax errors in your rule. The Status should read “Successful” for a successful compilation.

If your data quality rule has compiled successfully, enter a short message in the Show Message input. This custom message is displayed in the transfer Profile when a record fails the expression criteria. (If your data quality rule has not compiled successfully, correct the errors and click Compile again).

To activate the rule, check the Active switch.

• If the ‘Do not process record if rule fails’ switch is checked, a record that fails the data quality rule will be filtered out by this object and will not make it to downstream objects on the dataflow.
• If the ‘Do not process record if rule fails’ switch is cleared, a record that fails the data quality rule will be assigned the “Warning” status, but will not be filtered out by this object.

Add other data quality rules if necessary. To delete an existing data quality rule, select it and click the icon.

Usage