Data Integration Elastic Administration > Elastic configurations > AWS properties
  

AWS properties

To configure properties in an elastic configuration, click New Elastic Configuration or click the name of the configuration that you want to edit on the Elastic Clusters page.
The basic properties describe the elastic configuration and define the cloud platform to host the elastic cluster. To configure the cluster, configure the platform, advanced, and runtime properties.

Basic configuration

The following table describes the basic properties:
Property
Description
Name
Name of the elastic configuration.
Description
Description of the elastic configuration.
Runtime Environment
Runtime environment to associate with the elastic configuration. The runtime environment can contain only one Secure Agent. A runtime environment cannot be associated with more than one configuration.
If you don't select a runtime environment, the validation process can't validate the communication link to the Secure Agent and that the Secure Agent has the minimum runtime requirements to start a cluster.
Cloud Platform
Cloud platform that hosts the elastic cluster.
Select Amazon Web Services (AWS).

Platform configuration

The following table describes the platform properties:
Property
Description
Region
Region in which to create the cluster. Use the drop-down menu to view the regions that you can use.
Master Instance Type
Instance type to host the master node. Use the drop-down menu to view the instance types that you can use.
The instance types that you can use depend on the availability zones that you select and your AWS account.
For information to verify that the instance type that you select from the drop-down menu is supported in the selected availability zones and your AWS account, refer to the AWS documentation.
Master Instance Profile
Instance profile to be attached to the master node. The name must consist of alphanumeric characters with no spaces. You can also include any of the following characters: _+=,.@-
If you specify the master instance profile, you must also specify the worker instance profile.
Worker Instance Type
Instance type to host the worker nodes. Use the drop-down menu to view the instance types that you can use.
The instance types that you can use depend on the availability zones that you select and your AWS account.
For information to verify that the instance type that you select from the drop-down menu is supported in the selected availability zones and your AWS account, refer to the AWS documentation.
Worker Instance Profile
Instance profile to be attached to the worker nodes. The name must consist of alphanumeric characters with no spaces. You can also include any of the following characters: _+=,.@-
If you specify the worker instance profile, you must also specify the master instance profile.
Number of Worker Nodes
Number of worker nodes in the cluster. Specify the minimum and maximum number of worker nodes.
Enable Spot Instances
Indicates whether to use Spot Instances for worker nodes.
Spot Instance Price Ratio
Maximum percentage of On-Demand Instance price to pay for Spot Instances. Specify an integer value between 1 and 100.
Required if you enable Spot Instances. If you do not enable Spot Instances, this property is ignored.
Enable High Availability
Indicates whether the cluster is highly available. An odd number of master nodes will be created based on the number of availability zones or subnets that you provide. You must provide at least three availability zones or subnets.
For example, if you provide six availability zones, five master nodes are created with each master node in a different availability zone.
Note: When you provide multiple availability zones or subnets, worker nodes are highly available. Worker nodes are created across the availability zones or subnets regardless of whether high availability is enabled.
For more information about high availability, refer to the Kubernetes documentation.
Availability Zones
List of AWS availability zones where cluster nodes are created. The master node is created in the first availability zone in the list. If multiple zones are specified, the cluster nodes are created across the specified zones.
If you specify availability zones, the zones must be unique and be within the specified region.
The availability zones that you can use depend on your AWS account. To check which zones are available for your account, refer to the AWS documentation.
Required if you do not specify a VPC. If you specify a VPC, you cannot provide availability zones. You must provide subnets instead of availability zones.
EBS Volume Type
Type of Amazon EBS volumes to attach to Amazon EC2 instances as local storage. You can use only EBS General Purpose SSD (gp2).
EBS Volume Size
Size of the EBS volume to attach to a worker node for temporary storage during data processing. The volume size scales between the minimum and maximum based on job requirements. The range must be between 50 GB and 16 TB.
By default, the minimum and maximum volume sizes are 100 GB.
This configuration property does not apply to Graviton-enabled clusters.
Note: When the volume size scales down, the jobs that are currently running on the cluster might take longer to complete.
Cluster Shutdown
Cluster shutdown method. You can select one of the following cluster shutdown methods:
  • - Smart shutdown. The Secure Agent stops the cluster when no job is expected during the defined idle timeout, based on historical data.
  • - Idle timeout. The Secure Agent stops the cluster after the amount of idle time that you define.
Mapping Task Timeout
Amount of time to wait for a mapping task to complete before it is terminated. By default, a mapping task does not have a timeout.
If you specify a timeout, a value of at least 10 minutes is recommended. The timeout begins when the mapping task is submitted to the Secure Agent.
Staging Location
Location on Amazon S3 for staging data.
You can use a path that includes the folders in the bucket, such as <bucket name>/<folder name>. Specify an S3 bucket in the same region as the cluster to improve latency.
Log Location
Location on Amazon S3 to store logs that are generated when you run an elastic job.
You can use a path that includes the folders in the bucket, such as <bucket name>/<folder name>. Specify an S3 bucket in the same region as the cluster to improve latency.

Advanced configuration

The following table describes the advanced properties:
Property
Description
VPC
Amazon Virtual Private Cloud (VPC) in which to create the cluster. The VPC must be in the specified region.
If you do not specify a VPC, the agent creates a VPC on your AWS account based on the region and the availability zones that you select.
Note: If you plan to use the Sequence Generator transformation, you must specify a VPC and subnets.
Subnets
Subnets in which to create cluster nodes. Use a comma-separated list to specify the subnets.
Required if a VPC is specified. Each subnet must be in a different availability zone within the specified VPC.
If you do not specify a VPC, you cannot specify subnets. You must provide availability zones instead of subnets.
Note: If you plan to use the Sequence Generator transformation, you must specify a VPC and subnets.
Initialization Script Path
Amazon S3 file path of the initialization script to run on each cluster node when the node is created. Use the format: <bucket name>/<folder name>. The script can reference other init scripts in the same folder or in a subfolder.
The script must be a bash script.
ELB Security Group
Defines the inbound rules between the Kubernetes API server and clients that are external to the elastic cluster. Also defines the outbound rules between the Kubernetes API server and the cluster nodes. This security group attaches to the load balancer that the Secure Agent provisions for the elastic cluster.
When you specify a security group, VPC and subnet information are required.
For more information about this security group, see .
Master Security Group
Defines the inbound rules between master nodes and worker nodes in the elastic cluster, ELB security group, Secure Agent, and outbound rules to other nodes. This security group attaches to all master nodes of the cluster.
When you specify a security group, VPC and subnet information are required.
For more information about this security group, see .
Worker Security Group
Defines the inbound and outbound rules between worker nodes in the elastic cluster and other nodes. This security group is attached to all worker nodes of the cluster.
When you specify a security group, VPC and subnet information are required.
For more information about this security group, see .
AWS Tags
AWS tags to apply to cluster nodes. Each tag has a key and a value.
You can list a maximum of 30 tags.
The Secure Agent also assigns default tags to cloud resources. The default tags do not contribute to the limit of 30 tags.
Note: Issues can occur when you override default tags. Do not override the following default tags:
  • - Name
  • - KubernetesCluster
  • - k8s.io/cluster-autoscaler/enabled
  • - k8s.io/cluster-autoscaler/<cluster instance ID>.k8s.local

Runtime configuration

The following table describes the runtime properties:
Property
Description
Encrypt Data
Indicates whether temporary data on the cluster is encrypted.
Note: Encrypting temporary data might slow down job performance.
Runtime Properties
Custom properties to customize the cluster and the jobs that run on the cluster.

Validating the configuration

You can validate the information needed to create or update an elastic configuration before you save the configuration properties.
The validation process performs the following validations:
If you encounter errors related to context keys when you validate an elastic configuration, then add key ccs.k8s.policy.context.key to the runtime property in the elastic configuration. You can use the following value structure to add the context keys:
"ContextKeyName-'keyName1',ContextKeyValues-'keyValue1',ContextKeyType-(string|stringList|numeric|numericList|boolean|booleanList|ip|ipList|binary|binaryList|date|dateList)&infaContextKeyName-'keyName2',ContextKeyValues-'keyValue2',ContextKeyType-(string|stringList|numeric|numericList|boolean|booleanList|ip|ipList|binary|binaryList|date|dateList)"
For example:
ccs.k8s.policy.context.key=ContextKeyName-'aws:username',ContextKeyValues-'kops',ContextKeyType-string&infaContextKeyName-'ec2:ResourceTag/CREATED_BY',ContextKeyValues-'SFA-TDS',ContextKeyType-string
For more information about context keys, contact Informatica Global Customer Support.

GPU-enabled clusters

When you configure the worker instance type for the elastic configuration, you can select a GPU-enabled instance type. Selecting a GPU-enabled instance type creates a GPU-enabled cluster. GPUs use a massive parallel architecture to accelerate concurrent processing, offering performance benefits in many cases.
To create a GPU-enabled cluster, select a worker instance type in the g4 and p3 instance families. For more information about these instance types, refer to AWS documentation.
When you use a GPU-enabled cluster, all mappings submitted to the cluster that can run on GPU will run on GPU. Spark tasks in the mapping that cannot run on GPU run on CPU instead. To see which Spark jobs run on GPU and which jobs run on CPU, check the Spark event log after the job completes.
Note: The output of a task that runs on GPU might be different than the output that runs on CPU. For example, floating-point values might be rounded differently. For more information about processing differences, refer to Spark RAPIDS documentation.
For rules and guidelines for mappings that run on GPU-enabled clusters, refer to Mappings in the Data Integration help.

Graviton worker instance type

You can select an AWS Graviton 2 as a worker instance type to run elastic mappings. Graviton is a CPU-based instance type that uses Advanced RISC Machines (ARM) Neoverse N1 cores to deliver computational technology.
You can select one of the following worker instance types:
For more information about these instance types, refer to AWS documentation.

Graviton guidelines and limitations

The following guidelines and limitations apply to Graviton worker instance types:

Spot Instances

You can configure an elastic cluster in an AWS environment to use Spot Instances to host worker nodes.
Spot Instances are spare EC2 capacity that AWS offers at a lower price than On-Demand Instances. Spot Instances are not always available and AWS can interrupt running Spot Instances to reclaim the capacity.
To use Spot Instances, select Enable Spot Instances and set the Spot Instance price ratio. The Spot Instance price ratio is the maximum price you will pay for Spot Instances as a percentage of On-Demand Instance price. For example, if On-Demand Instances cost $0.68 an hour and you set the Spot Instance price ratio to 50, you will pay up to $0.34 an hour for Spot Instances.
The Secure Agent always creates a number of On-Demand worker nodes equal to the minimum number of worker nodes that you configure. When you enable Spot Instances and the cluster scales up, the agent tries to create additional worker nodes on Spot Instances up to the maximum number of worker nodes. If Spot Instances are not available or cost more than the maximum price you set, the cluster uses On-Demand Instances for the worker nodes.
For example, if you set the minimum number of worker nodes to 5 and the maximum to 8, the agent creates 5 nodes on On-Demand Instances and tries to create 3 nodes on Spot Instances. If you set the maximum number of worker nodes equal to the minimum, the cluster uses only On-Demand Instances.
If AWS interrupts a Spot node that is running an elastic job, the agent will use On-Demand nodes to complete the job.

High availability

An elastic cluster can become highly available to eliminate a single point of failure when the master node goes down. If you enable high availability and one master node goes down, other master nodes will be available and jobs on the cluster can continue running.
When a cluster is highly available, watch out for job failures in the following scenarios:

Propagating tags to cloud resources

The Secure Agent propagates tags to cloud resources based on the AWS tags that you specify in an elastic configuration.
The agent propagates tags to the following resources:
*Tags are propagated to EBS volumes only if you use user-defined master and worker roles.
Note: The Secure Agent propagates tags only to cloud resources that the agent creates. If you create a VPC and subnets and specify the resources in an elastic configuration, the agent does not propagate AWS tags to the VPC and subnets.
If your enterprise follows a tagging policy, make sure to manually assign tags to the following resources:

Default tags for cloud resources

In addition to the cloud platform tags that you specify in an elastic configuration, the Secure Agent assigns several default tags to resources. The default tags assist Kubernetes Operations (kops), services on the cloud platform, and data governance. Do not override the default tags.
The following table describes tags that the agent also assigns to cluster nodes to report information about the cluster:
Cloud platform tag
Description
infa:ccs:hostname
The host name of the Secure Agent machine that started the cluster.
If the Secure Agent machine stops unexpectedly and the Secure Agent restarts on a different machine, the host name is the original Secure Agent machine.
infa:k8scluster:configname
Name of the elastic configuration that is used to create the cluster.
infa:k8scluster:workdir
Staging directory that the cluster uses.
Some default tags do not have a namespace and can conflict with the user-defined tags that you specify in an elastic configuration. For example, kops automatically adds the Name and KubernetesCluster tags to all resources, but the tags do not have a namespace. If you specify a user-defined tag with the same name, such as KubernetesCluster, kops will override the user-defined tag with the default tag.
Note: Issues can occur when you override default tags. Do not override the following default tags:

Initialization scripts

Cluster nodes can run an initialization script based on an init script path that you specify in an elastic configuration. Each node runs the script when the node is created, and the script can reference other init scripts.
You might want to run an init script to install additional software on the cluster. For example, your enterprise policy might require each cluster node to contain monitoring and anti-virus software to protect your data.
Consider the following guidelines when you create the init script:
The init script path must be in cloud storage. You can place the scripts in a unique path on the cloud storage system, or you can place the scripts in the staging location.

Initialization script failures

When an initialization script fails on a cluster node, it can have a significant impact on the elastic cluster. An init script failure can prevent the cluster from scaling up or cause the Secure Agent to terminate the cluster.
Note the impact that an init script failure can have in the following situations:
Failure during cluster creation
If the init script fails on any node during cluster creation, the Secure Agent terminates the cluster.
Resolve the issues with the init script before running a job to start the cluster again.
Failure during a scale up event
If the init script fails on a node that is added to the cluster during a scale up event, the node fails to start and the cluster fails to scale up. If the cluster attempts to scale up again and the node continues to fail to start, it adds to the number of accumulated node failures until the Secure Agent terminates the cluster.
Failure while recovering a master node
If you enable high availability in an AWS environment and the init script fails on a recovered master node, the node fails to start and contributes to the number of accumulated node failures over the cluster lifecycle.
Accumulated failures over the cluster lifecycle
During the cluster lifecycle, the Secure Agent tracks the number of accumulated node failures that occur due to an init script within a certain time frame. If the number of failures is too high, the agent terminates the cluster.
Find the log files for the nodes where the init script failed and use the log files to resolve the failures before running a job to start the cluster again.

Updating the runtime environment or the staging location

Update the runtime environment or the staging location in an elastic configuration based on the status of the Secure Agent and the elastic cluster.
To update the runtime environment or the staging location, perform one of the following tasks based on the status of the Secure Agent and the elastic cluster:
The Secure Agent and the elastic cluster are running.
If the agent and the cluster are running, complete the following tasks:
  1. 1. Update the runtime environment or the staging location in the elastic configuration.
  2. 2. Stop the cluster when you save the configuration.
The Secure Agent is unavailable or the elastic cluster cannot be reached.
If the agent is unavailable or the cluster cannot be reached, complete the following tasks:
  1. 1. Run the command to delete the cluster or make sure that all cluster resources are deleted by logging in to your account on the cloud platform. For information about commands, see Command reference.
  2. 2. Update the runtime environment or the staging location in the elastic configuration.
  3. 3. Disable the cluster when you save the configuration.
Note: If you update the runtime environment, the new Secure Agent will create a new elastic cluster with a different cluster ID.

Accessing a new staging location

If you plan to use a new staging location, you must first change the staging location in the elastic configuration and then change the permissions to the staging location on AWS.
If you use role-based security, you must also change the permissions to the staging location on the Secure Agent machine.
If you change the permissions before changing the staging location in the configuration, elastic jobs fail with the following error:
Error while executing mapping. ExecutionId '<execution ID>'. Cause: [Failed to start cluster for [01000D25000000000005]. Error reported while starting cluster [Cannot apply cluster operation START because the cluster is in an error state.].].
To fix the error, perform the following tasks:
  1. 1. Revert the changes to the permissions for the staging location.
  2. 2. Edit the elastic configuration to revert the staging location.
  3. 3. Stop the cluster when you save the configuration.
  4. 4. Update the staging location in the configuration, and then change the permissions to the staging location on AWS.