Database Table Destination¶
Database Table¶
Centerprise Database Table Destination provides the functionality to write data to a database table. This component features a great deal of control over how data is written to the database table. Centerprise supports a wide range of on-premise and cloud databases including SQL Server, Oracle, DB2, Sybase, MySql, database.com from Salesforce, and Microsoft Dynamics CRM and more. Centerprise delivers highly optimized implementations for these database connectors including high-performance bulk insert, set based updates and transaction management. This, combined with Centerprise’s parallel processing architecture, delivers industrial strength performance and scalability.
Transaction management¶
Database Table Destination provides full support for transaction management including rollback on job level or record level errors. Moreover, using the Shared Connection object, you can link multiple destinations to a single connection and transaction. This gives you control over transaction rollback based on errors in any of the destinations.
Database Write Strategies¶
Centerprise Database Write Strategies transformations work in conjunction with Database Table Destination to provide change data capture (CDC) functionality as well as provide specialized transformations such as Type 1 and Type 2 slowly changing dimension management. These transformations enable you to create powerful database integrations with a straightforward user interface. For information, please see Database Write Strategies.
Steps¶
Adding a database table destination object allows you to write to a database table. An example of what a database table destination object looks like is shown below.
To configure the properties of a Database Table Destination object after it was added to the dataflow, right-click on it and select Properties from the context menu. The following properties are available:
Destination Connection screen – allows you to enter the connection information for your destination, such as server name, database and schema, as well as credentials for connecting to the selected destination.
Pick Table screen:
Database Transaction Management: Enable Transaction Management if you want to wrap your transfer inside a transaction. Depending on your database settings, this can give you performance improvements during the transfer. When Transaction Management is enabled, you should choose between always committing transaction at the end of the transfer, or only committing it if there were no errors. Any errors would result in the entire transaction being rolled back.
Preserve System Generated Key Values From Source. This option is only available if you assigned at least one field in your destination layout as System Generated field. If enabled, Centerprise will pass the incoming value from the source to the system generated field. Otherwise, the incoming source value will be ignored, and the system will write auto-increasing values to the destination System Generated field.
Database Load Options: specify the type of insert of your records into a destination database. The available types are Single Record Insert, Bulk Insert with Batch Size, and Bulk Insert with All Records in One Batch.
These types allow you to customize your transfer to balance performance vs. logging needs. Bulk inserts typically result in a better performance as they require less logging.
Single Record Insert - Records are inserted into a destination table one-by-one. Performance is the slowest among the three insert types. However, any errors or warnings during the transfer are displayed to you immediately as the transfer progresses.
Bulk Insert with All Records in One Batch – typically a fast method of transferring large amounts of data. But keep in mind that, should there be any database specific errors in your transfer, they won’t show until the end of the transfer at which time the entire batch were to be written to the destination database. Note: Not all database providers support this type of insert.
Bulk Insert with Batch Size – a good tradeoff between performance and logging needs. Records are inserted in batches of the specified size. Typically, larger batch sizes result in better transfer speeds; however performance gains may be less with relatively large batch sizes. Note: Not all database providers support this type of insert.
Note: Bulk insert may not be available if there are certain data types in a destination table. In this case the transfer will proceed as “single insert”.
Field Layout¶
The Field Layout screen is available in the properties of most objects on the dataflow to help you specify the fields making up the object. The table below explains the attributes you can set in the Field Layout screen.
| Attribute | Description |
|---|---|
| Name | The system pre-fills this item for you based on the field header. Field names do not allow spaces. Field names are used to refer to the fields in the Expression Builder or tools where a field is used in a calculation formula. |
| Header | Represents the field name specified in the header row of the file. Field headers may contain spaces. |
| Data Type | Specifies the data type of a field, such as Integer, Real, String, Date, or Boolean. |
| Format | Specifies the format of the values stored in that field, depending on the field’s data type. For example, for dates you can choose between DD-MM-YY, YYYY-MM-DD, or other available formats. |
| Start Position | Specifies the position of the field’s first character relative to the beginning of the record.Note: This option is only available for fixed length layout type. |
| Length | Specifies the maximum number of characters allotted for a value in the field. The actual value may be shorter than what is allowed by the Length attribute.Note: This option is only available for fixed length and database layout types. |
| Column Name | Specifies the column name of the database table.Note: This option is only available in database layout. |
| DB Type | Specifies the database specific data type that the system assigns to the field based on the field's data. Each database (Oracle, SQL, Sybase, etc) has its own DB types. For example, Long is only available in Oracle for data type string.Note: This option is only available in database layout. |
| Decimal Places | Specifies the number of decimal places for a data type specified as real.Note: This option is only available in database layout. |
| Allows Null | Controls whether the field allows blank or NULL values in it. |
| Default Value | Specifies the value that is assigned to the field in any one of the following cases:- The source field does not have a value- The field is not found in the source layout- The destination field is not mapped to a source fieldNote: This option is only available in destination layout. |
| Sequence | Represents the column order in the source file. You can change the column order of the data being imported by simply changing the number in the sequence field. The other fields in the layout will then be reordered accordingly. |
| Description | Contains information about the field to help you remember its purpose. |
| Alignment | Specifies the positioning of the field’s value relative to the start position of the field. Available alignment modes are LEFT, CENTER, and RIGHT.Note: This option is only available for fixed length layout type. |
| Primary Key | Denotes the primary key field (or part of a composite primary key) for the table.Note: This option is only available in database layout. |
| System Generated | Indicates that the field will be automatically assigned an increasing Integer number during the transfer.Note: This option is only available in database layout. |
The table below provides a list of all the attributes available for a particular layout type.
| Layout Type | Attributes Available |
|---|---|
| Source Delimited file and Excel worksheet | Name, Header, Data type, Format |
| Source Fixed Length file | Name, Header, Data type, Format, Start position, Length |
| Source Database Table and SQL query | Column name, Name, Data type, DB type, Length, Decimal places, Allows null |
| Destination Delimited file and Excel worksheet | Name, Header, Data type, Format, Allows null, Default value |
| Destination Fixed Length file | Sequence, Name, Header, Description, Data type, Format, Start position, Length, Allows null, Default value, Alignment |
| Destination Database Table | Column name, Name, Data type, DB type, Length, Decimal places, Allows null, Primary key, System generated |