Record Level Log

A Record Level Log captures the status (Success, Error, Warning, or Skip) for each of the records transferred and includes snapshots of the source record and the destination record, as well as provides additional details, such as error messages.

You can have any number of record level logs on the dataflow. Each record level log will collect the status of the records in the object that it is connected to.

In this document, we will learn how to use Record Level Log object in Astera Centerprise.

Using record level log

In this case we have a simple dataflow performing a data quality check process. It contains Customers data coming in from an Excel Workbook Source. Then a Data Quality Rule is applied to validate data for error and warning checks and finally data is written to a Database Table Destination.

If you Preview Output for Customers data, you will see that some of the records for Region field and Fax field are empty.

A Data Quality Rule is applied to identify null records in Region field as errors and empty records in Fax field as warnings. Upon previewing its output you will see that the records that failed to match the rule have returned an error, denoted by a red warning sign.

If you hover on these warning signs, it will show you the error message.

Now, when we run this dataflow we want to know which records passed the validation check, which records failed the validation check, which records had errors and which records ended in only warnings.

For this purpose, we will use Record Level Log.

Configuring a Record Level Log Object

1. To get a Record Level Log object from the Toolbox, go to Toolbox > Data Profiling > Record Level Log. If you’re unable to see the toolbox, go to View > Toolbox or press Ctrl + Alt + X.

2. Drag-and-drop the Record Level Log object onto the dataflow designer.

You can see that the dragged Log1 object is empty right now. This is because we haven’t mapped any fields on it yet.

3. Auto-map the fields from the source object onto the record log object.

4. To configure the Record Level Log object, right-click on its header and select Properties from the context menu.

A configuration window will open. First screen is Layout Builder screen. This is where we can create or delete fields, change its name and data type.

5. Click Next and you will be directed to a Properties window where you have to configure settings for creating the log file.

6. Specify the File Path where Centerprise will save this log file. Log files are saved with .prof extension.

7. Specify the Log Level Type from the dropdown list.

All – all records (including Success records) are logged

Errors – only error records are logged

Warnings – only warning records are logged

Errors and Warnings – both error and warning records are logged

Off – no logging

In this case, we will select Errors and Warnings as our log type.

Stop Logging After … Records with Errors – allows you to limit excessive logging by setting a cap on the maximum number of errors to be logged. The logging stops after the cap has been reached.

The default value is 1000 errors.

Click OK.

Executing The Task

1. After configuring settings for Log object, click on Start Dataflow icon from the toolbar at the top. A Job Progress window will open at this instant and will show you the trace of the job.

2. Click on the log file link provided in the Job Progress window.

Record Level Log will open in Centerprise showing you the status of logged records.

Centerprise stores the error logs in XML format and if you expand each record, it will give you the Field Name to which the error/warning message was attached, the Processing Step of data check that resulted in the errors, as well as the error Message.

If you click on View Record, Centerprise will show you the field values of the record failing the data quality rule.