Transformations > Source transformation > Partitions
  

Partitions

If your organization has the Partitioning license, you can use partitions to optimize performance for mapping tasks.
If a mapping task processes large data sets or includes transformations that perform complicated calculations, the task can take a long time to process. When you use multiple partitions, the mapping task divides data into partitions and processes the partitions concurrently, which can optimize performance. Not all source types support partitioning.
Enable partitioning when you configure the Source transformation in the Mapping Designer. When you configure partitions in the Source transformation, partitioning occurs throughout the mapping.
To enable partitioning for a source, select a partitioning method on the Partitions tab. The partitioning methods that you can select vary based on the source type. For more information about partitioning different types of sources, see the help for the appropriate connector.
You can select one of the following partitioning methods based on the source type:
None
The mapping task processes all data in a single partition. This is the default option.
Fixed
The mapping task distributes rows of data based on the number of partitions that you specify. You can specify up to 64 partitions.
Use this method for a source type that does not allow key range partitioning such as a flat file source, or when the mapping includes a transformation that does not support key range partitioning.
Consider the number of records to be passed in the mapping to determine an appropriate number of partitions for the mapping. For a small number of records, partitioning might not be advantageous.
If the mapping includes multiple sources, specify the same number of partitions for each source.
Key range
The mapping task distributes rows of data based on a field that you define as a partition key. You select one field in the source as the partition key, and then you define a range of values for the partition key.
You can use this method for tabular sources such as relational, Cassandra, Google BigQuery, and JDBC V2 sources.
Key ranges can be of the following data types:
If the mapping includes multiple sources, use the same number of key ranges for each source.
Pass through
The mapping task processes data without redistributing rows among partitions. All rows in a single partition stay in the partition. Choose pass-through partitioning when you want to create additional partitions to improve performance, but do not want to change the distribution of data across partitions.
You can use this method for sources such as Amazon S3, Netezza, and Teradata.
Dynamic
The mapping task determines the optimal number of partitions to create at runtime based on the source size.
You can specify dynamic partitioning in certain sources in elastic mappings.
You cannot partition a mapping in the following situations:
When you configure partitions, be sure to save and run the mapping in the Mapping Designer to validate the partition settings.

Partitioning rules and guidelines

Before you partition a mapping, note the following rules and guidelines:

Partitioning examples

The following examples show how you can configure partitioning in a mapping.

Partitioning with a Flat File Source

You have a mapping task that uses a large, 1GB flat file source. You want to specify two partitions in the Source transformation to optimize performance.
On the Partitions tab for the Source transformation, you select fixed partitioning and enter the number of partitions, as shown in the following image:
On the Partitions tab of the Source transformation, the partitioning type is "Fixed" and the number of partitions is set to "2."

Key Range Partitioning with a Relational Database Source

You have customer names, addresses, and purchasing history in a relational database source. You decide to partition the source data into three partitions based on postal codes, using the following ranges:
On the Partitions tab for the Source transformation, you select key range partitioning and choose the BILLINGPOSTALCODE field as the partition key. You add three key ranges to create three partitions, as shown in the following image:
On the Partitions tab for the Source transformation, the partitioning type is "Key Range" and the BILLINGPOSTALCODE column is selected as the partition key. The Start Range and End Range columns for each partition define the range of values for each partition. In the first partition, the start range is blank, so the minimum value is used as the starting value. In the third partition, the end range is blank, so the maximum value is used as the ending value.
Note that for the first partition, you leave the start value blank for the minimum value. In the last partition, you leave the end value blank for the maximum value.
Using these values, records with a postal code of 0 up to 30000 are processed in partition #1, records with a postal code of 30001 to 50000 are processed in partition #2, and records with a postal code of 50001 or higher are processed in partition #3.
After you configure the mapping, you save and run the mapping to validate the partitions.