Amazon S3 V2 Connector > Mappings and mapping tasks with Amazon S3 V2 > Amazon S3 V2 objects in mappings
  

Amazon S3 V2 objects in mappings

When you create a mapping, you can configure a Source or Target transformation to represent an Amazon S3 V2 object.

Amazon S3 V2 sources in mappings

In a mapping, you can configure a Source transformation to represent an Amazon S3 V2 object as the source to read data from Amazon S3.
Specify the name and description of the Amazon S3 V2 source. Configure the Amazon S3 V2 source and advanced properties for the source object.
The following table describes the Amazon S3 V2 source properties that you can configure in a Source transformation:
Property
Description
Connection Name
Name of the Amazon S3 V2 source connection. Select a source connection or click New Parameter to define a new parameter for the source connection.
Source Type
Source type. Select one of the following types:
  • - Single Object
  • - Parameter. Select Parameter to define the source type when you configure the mapping task.
Object
Name of the source object.
When you select an object, you can also select a .manifest file object when you want to read from multiple files.
Parameter
Select an existing parameter for the source object or click New Parameter to define a new parameter for the source object. The Parameter property appears only if you select Parameter as the source type. If you want to overwrite the parameter at runtime, select the Overwrite Parameter option.
Format
Specifies the file format that the Amazon S3 V2 Connector uses to read data from Amazon S3.
You can select the following file format types:
  • - None1
  • - Delimited
  • - Avro
  • - ORC
  • - Parquet
  • - JSON2
  • - Discover Structure2
Default is None. If you select None as the format type, the Secure Agent reads data from Amazon S3 files in binary format.
You cannot use parameterized sources when you select the discover structure format.
Open the Formatting Options dialog box to define the format of the file.
For more information about format options, see Amazon S3 V2 file formatting options.
Intelligent Structure Model2
Applicable to Discover Structure format type. Select the intelligent structure model.
For more information, see Components.
1Applies only to mappings.
2Applies only to elastic mappings.
The remaining properties are applicable for both mappings and elastic mappings.
Note: When you parameterize a Delimited source file in an elastic mapping, the FileName field appears in the source fields.
As a workaround, add a rule in the incoming fields of the target to exclude the FileName field.
The following table describes the Amazon S3 V2 advanced source properties that you can configure in a Source transformation:
Property
Description
Source Type
Type of the source from which you want to read data.
You can select the following source types:
  • - File
  • - Directory
Default is File.
For more information about the source type, see Source types in Amazon S3 V2 sources.
Folder Path
Overwrites the bucket name or folder path of the Amazon S3 source file.
If applicable, include the folder name that contains the source file in the <bucket_name>/<folder_name> format.
If you do not provide the bucket name and specify the folder path starting with a slash (/) in the /<folder_name> format, the folder path appends with the folder path that you specified in the connection properties.
For example, if you specify the /<dir2> folder path in this property and <my_bucket1>/<dir1> folder path in the connection property, the folder path appends with the folder path that you specified in the connection properties in <my_bucket1>/<dir1>/<dir2> format.
If you specify the <my_bucket1>/<dir1> folder path in the connection property and <my_bucket2>/<dir2> folder path in this property, the Secure Agent reads the file in the <my_bucket2>/<dir2> folder path that you specify in this property.
File Name
Overwrites the Amazon S3 source file name.
Allow Wildcard Characters2
Indicates whether you want to use wildcard characters for the directory source type.
If you select this option, you can use the question mark (?) and asterisk (*) wildcard characters in the folder path or file name. For details, see Wildcard characters.
Enable Recursive Read2
Indicates whether you want to read Delimited, Avro, JSON, ORC, or Parquet files recursively from the specified folder and its subfolders and files. Applicable when you select the directory source type.
Encryption Type
Method you want to use to decrypt data.
You can select one of the following encryption types:
  • - None
  • - Informatica encryption
Default is None.
Note: You cannot select client-side encryption, server-side encryption, and server-side encryption with KMS encryption types.
Staging Directory1
Path of the local staging directory.
Ensure that the user has write permissions on the directory. In addition, ensure that there is sufficient space to enable staging of the entire file. Default staging directory is the /temp directory on the machine that hosts the Secure Agent.
When you specify the directory path, the Secure Agent create folders depending on the number of partitions that you specify in the following format: InfaS3Staging<00/11><timestamp>_<partition number> where, 00 represents read operation and 11 represents write operation.
For example, InfaS3Staging000703115851268912800_0
The temporary files are created within the new directory.
The staging directory source property does not apply to Avro, ORC, and Parquet files.
Hadoop Performance Tuning Options
This property is not applicable for Amazon S3 V2 Connector.
Compression Format
Decompresses data when you read data from Amazon S3.
You can choose to decompress the data in the following formats:
  • - None
  • - Bzip22
  • - Gzip
  • - Lzo
Default is None.
You can decompress data in an elastic mapping if the mapping reads data from a JSON file in Bzip2 format.
Note: Amazon S3 V2 Connector does not support the Lzo compression format even though the option appears in this property.
For more information about the compression format, see Data compression in Amazon S3 V2 sources and targets.
Download Part Size1
Downloads the part size of an Amazon S3 object in bytes.
Default is 5 MB. Use this property when you run a mapping to read a file of flat format type.
Multiple Download Threshold1
Minimum threshold size to download an Amazon S3 object in multiple parts.
To download the object in multiple parts in parallel, ensure that the file size of an Amazon S3 object is greater than the value you specify in this property. Default is 10 MB.
Temporary Credential Duration
The time duration during which an IAM user can use the dynamically generated temporarily credentials to access the AWS resource. Enter the time duration in seconds.
Default is 900 seconds.
If you require more than 900 seconds, you can set the time duration maximum up to 12 hours in the AWS console and then enter the same time duration in this property.
Tracing Level
This property is not applicable for Amazon S3 V2 Connector.
1 Applies only to mappings.
2Applies only to elastic mappings.
The remaining properties are applicable for both mappings and elastic mappings.

Amazon S3 V2 targets in mappings

In a mapping, you can configure a Target transformation to represent an Amazon S3 V2 object as the target to write data to Amazon S3.
Specify the name and description of the Amazon S3 V2 target. Configure the Amazon S3 V2 target and advanced properties for the target object.
The following table describes the Amazon S3 V2 target properties that you can configure in a Target transformation:
Property
Description
Connection
Name of the Amazon S3 V2 target connection. Select a target connection or click New Parameter to define a new parameter for the target connection.
Target Type
Target type. Select one of the following types:
  • - Single Object
  • - Parameter: Select Parameter to define the target type when you configure the mapping task.
Object
Name of the target object.
You can select an existing object or create an object at runtime.
Parameter
Select an existing parameter for the source object or click New Parameter to define a new parameter for the target object. The Parameter property appears only if you select Parameter as the target type. If you want to overwrite the parameter at runtime, select the Overwrite Parameter option.
Create Target
Creates a target.
Enter a name and path for the target object. You can use parameters defined in a parameter file in the target name.
Format
Specifies the file format that the Amazon S3 V2 Connector uses to write data Amazon S3.
You can select the following file format types:
  • - None 1
  • - Delimited
  • - Avro
  • - ORC
  • - Parquet
  • - JSON2
Default is None. If you select None is as the format type, the Secure Agent writes data to Amazon S3 files in binary format.
Open the Formatting Options dialog box to define the format of the file.
For more information about format options, see Amazon S3 V2 file formatting options.
Operation
Type of the target operation.
You can perform only insert operation on an Amazon S3 V2 target.
1Applies only to mappings.
2Applies only to elastic mappings.
The remaining properties are applicable for both mappings and elastic mappings.
The following table describes the Amazon S3 V2 advanced target properties that you can configure in a Target transformation:
Property
Description
Overwrite File(s) If Exists
Overwrites an existing target file.
Default is true. For more information about overwriting the existing files, see Overwriting existing files.
Folder Path
Bucket name or folder path where you want to write the Amazon S3 target file.
If applicable, include the folder name that contains the target file in the <bucket_name>/<folder_name> format.
If you do not provide the bucket name and specify the folder path starting with a slash (/) in the /<folder_name> format, the folder path appends with the folder path that you specified in the connection properties.
For example, if you specify the /<dir2> folder path in this property and <my_bucket1>/<dir1> folder path in the connection property, the folder path appends with the folder path that you specified in the connection properties in <my_bucket1>/<dir1>/<dir2> format.
If you specify the <my_bucket1>/<dir1> folder path in the connection property and <my_bucket2>/<dir2> folder path in this property, the Secure Agent writes the file in the <my_bucket2>/<dir2> folder path that you specify in this property.
File Name
Creates a new file name or overwrites an existing target file name.
Encryption Type
Method you want to use to encrypt data.
Select one of the following encryption types:
  • - None
  • - Client Side Encryption1
  • - Server Side Encryption
  • - Server Side Encryption with KMS
  • - Informatica Encryption
Default is None.
For more information about the encryption type, see Data encryption in Amazon S3 V2 targets.
Staging Directory1
Enter the path of the local staging directory.
Ensure that the user has write permissions on the directory. In addition, ensure that there is sufficient space to enable staging of the entire file. Default staging directory is the /temp directory on the machine that hosts the Secure Agent.
When you specify the directory path, the Secure Agent create folders depending on the number of partitions that you specify in the following format: InfaS3Staging<00/11><timestamp>_<partition number> where, 00 represents read operation and 11 represents write operation.
For example, InfaS3Staging000703115851268912800_0
The temporary files are created within the new directory.
The staging directory target property does not apply to Avro, ORC, and Parquet files.
File Merge
This property is not applicable for Amazon S3 V2 Connector.
Hadoop Performance Tuning Options
This property is not applicable for Amazon S3 V2 Connector.
Compression Format
Compresses data when you write data to Amazon S3.
You can compress the data in the following formats:
  • - None
  • - Bzip22
  • - Deflate
  • - Gzip
  • - Lzo
  • - Snappy
  • - Zlib
Default is None.
Note: Amazon S3 V2 Connector does not support the Lzo compression format even though the option appears in this property.
For more information about the compression format, see Data compression in Amazon S3 V2 sources and targets.
Object Tags
The key value pairs to add single or multiple tags to the objects stored on the Amazon S3 bucket.
You can either enter the key value pairs or specify the file path that contains the key value pairs.
Use this property when you run a mapping to write a file of flat format type. For more information about the object tags, see Object tag.
TransferManager Thread Pool Size1
The number of threads to write data in parallel.
Default is 10. Use this property when you run a mapping to write a file of flat format type.
Amazon S3 V2 Connector uses the AWS TransferManager API to upload a large object in multiple parts to Amazon S3.
When the file size is more than 5 MB, you can configure multipart upload to upload object in multiple parts in parallel. If you set the value of TransferManager Thread Pool Size to greater than 50, the value reverts to 50.
Merge Partition Files1
Determines whether the Secure Agent must merge the number of partition files as a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 V2 targets.
Temporary Credential Duration
The time duration during which an IAM user can use the dynamically generated temporarily credentials to access the AWS resource. Enter the time duration in seconds.
Default is 900 seconds.
If you require more than 900 seconds, you can set the time duration maximum up to 12 hours in the AWS console and then enter the same time duration in this property.
Part Size1
Uploads the part size of an Amazon S3 object in bytes.
Default is 5 MB. Use this property when you run a mapping to write a file of flat format type.
Forward Rejected Rows
This property is not applicable for Amazon S3 V2 Connector.
1Applies only to mappings.
2 Applies only to elastic mappings.
The remaining properties are applicable for both mappings and elastic mappings.
Note: When you read from or write to the ORC files on Cloudera 6.1 distribution, the unicode characters are parsed incorrectly.
When you create a mapping and the column name in the Amazon S3 source or target object contains special characters, the Secure Agent replaces the special characters with an underscore (_) and the mapping fails.

Amazon S3 V2 lookups

You can use Amazon S3 V2 objects in a connected and an unconnected cached Lookup transformation.
For more information about the Lookup transformation, see Transformations.