Amazon S3 V2 Connector > Data type reference > Avro Amazon S3 file data types and transformation data types
  

Avro Amazon S3 file data types and transformation data types

Avro Amazon S3 file data types map to transformation data types that the Secure Agent uses to move data across platforms.
The following table lists the Avro Amazon S3 file data types that the Secure Agent supports and the corresponding transformation data types:
Avro Amazon S3 File Data Type
Transformation Data Type
Range and Description
Array
Array
Unlimited number of characters
Boolean
Integer
TRUE (1) or FALSE (0)
Date
Date/Time
January 1, 0001 to December 31, 9999.
Decimal
Decimal
For mappings- Precision 18 and 28 digits. Scale 0 to 28.
If you specify a precision less than 18 or 28 digits, 18 or 28 is considered as the precision.
For elastic mappings- Precision 18, 28, and 38 digits. Scale 0 to 38.
Double
Double
Precision 15
Float
Double
Precision 15
Int
Integer
-2,147,483,648 to 2,147,483,647 Precision 10, scale 0
Long
Bigint
-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
Precision 19, scale 0
Map
Map
Unlimited number of characters
Null
Integer
-2,147,483,648 to 2,147,483,647
Precision 10, scale 0
Record
Struct
Unlimited number of characters
String
String
-1 to 104,857,600 characters
Union
Corresponding data type in a union of ["primitive_type|complex_type", "null"] or ["null", "primitive_type|complex_type"].
Dependent on primitive or complex data type.
The following table lists the Timestamp data type support for Avro file formats:
Timestamp Data type
Mapping
Elastic Mapping
Timestamp_micros
Yes
Yes
Timestamp_millis
Yes
No
Time_millis
Yes
No
Time_micros
Yes
No

Rules and guidelines for Avro data types

Consider the following rules and guidelines when you use Avro Amazon S3 file data types and transformation data types:

Enabling Date, Decimal, and Timestamp types

Perform the following steps to use the Date, Decimal, and Timestamp types for Avro data types before you run a mapping:
  1. 1. In Administrator, select the Secure Agent listed on the Runtime Environments tab.
  2. 2. Click Edit.
  3. 3. In the System Configuration Details section, select Data Integration Service as the service.
  4. 4. Edit the INFA_DEBUG property, and enter -DEnableNewAvroDataTypes=true.
  5. Note: To enter multiple flags, separate the flags with spaces.
  6. 5. Click Save.
  7. 6. Restart the Secure Agent.
Perform the following steps to use the Date, Decimal, and Timestamp types for Avro data types before you run an elastic mapping:
  1. 1. In Administrator, select Elastic Clusters.
  2. 2. Go to the existing configuration or create a new configuration.
  3. 3. In the Advanced Configuration section, go to Custom Properties.
  4. 4. Enter -DEnableNewAvroDataTypes=true.
  5. 5. Click Save.