Data Integration Elastic Administration > Data Integration Elastic on AWS > Task-based access to resources
  

Task-based access to resources

To process data, the Secure Agent and the elastic cluster access the resources that are part of an elastic job, including resources on the cloud platform, source and target data, and staging and log locations.
Resources are accessed differently based on the task that is performed:

Designing an elastic mapping

Designing an elastic mapping is similar to designing a non-elastic mapping in Data Integration. When you design a mapping, the Secure Agent accesses sources and targets so that you can read and write data.
For example, when you add a Source transformation to a mapping, the Secure Agent accesses the source to display the fields that you can use in the rest of the mapping. The Secure Agent also accesses the source when you preview data.
The Secure Agent accesses sources and targets based on the type of connectors that the job uses:
Connectors with direct access to Amazon data sources
If the mapping uses a connector with direct access to Amazon data sources, the Secure Agent uses credential-based security or role-based security to access the source or target. For credential-based security, the Secure Agent accesses the source or target through connection-level AWS credentials. For role-based security, the Secure Agent uses the permissions in the Secure Agent role to access the source or target.
Connectors without direct access to Amazon data sources
If the mapping does not use a connector with direct access to Amazon data sources, the Secure Agent uses the connection properties to access the source or target. For example, the Secure Agent might use the user name and password that you provide in the connection properties to access a database.

Creating an elastic cluster

To create an elastic cluster, the Secure Agent accesses the staging and log locations before sending requests to AWS to create cluster resources.
The Secure Agent uses the permissions in the kops role and the worker role to perform the following tasks:
The following image shows the sequence of events when the Secure Agent creates a cluster:
The diagram shows the sequence of events in AWS when the Secure Agent creates an elastic cluster. An elastic job initiates the creation process. The agent receives the job and uses the Secure Agent role to assume the kops role. As the kops role, the agent assumes the worker role to verify that the cluster can access the necessary staging and log locations. Then, the agent stores cluster information in the staging location and creates the elastic cluster.
  1. 1. You run an elastic job.
  2. 2. The Secure Agent obtains elevated privileges on AWS using the Secure Agent role to assume the kops role.
  3. 3. If you configure user-defined master and worker roles, the Secure Agent uses the kops role to assume the worker role. Using the permissions in the worker role, the Secure Agent validates that the cluster has access to staging and log locations.
  4. 4. The Secure Agent stores cluster information in the staging location.
  5. 5. The Secure Agent creates the elastic cluster.

Running a job with direct access to Amazon data sources

When an elastic job uses a connector with direct access to Amazon data sources, the job accesses Amazon resources using credential-based security or role-based security.
Amazon resources are accessed in the following ways based on the security type:
Credential-based security
If you implement credential-based security, the connection-level AWS credentials are used to access Amazon resources, including Amazon data sources and the staging location. The worker role is used to access the log location.
Role-based security
If you implement role-based security, either the user-defined or the default worker role accesses Amazon resources, including Amazon data sources, the staging location, and the log location.
Note: If you use default master and worker roles, the policies that are attached to the Secure Agent role are passed to the worker role. The policies that are passed to the worker role can grant the worker role access to Amazon resources.
When the job is complete, the Secure Agent accesses the log location to upload the agent job log. The Secure Agent accesses the log location through the Secure Agent role.
The following image shows how resources are accessed when the job runs on an elastic cluster:
The diagram shows the sequence of events in AWS when you run an elastic job. The worker nodes access source and target data. During the job, the Secure Agent stores job dependencies in the staging location. Then, the worker nodes get the job dependencies, stage data in the staging location, and store logs in the log location. At the end of the job, the Secure Agent uploads the agent job log to the log location.
  1. 1. To run the job, the worker nodes access Amazon resources through either role-based security or credential-based security. If you use credential-based security, the worker nodes use the connection-level AWS credentials. If you use role-based security, the worker nodes use the worker role.
  2. 2. The worker nodes use the connection-level AWS credentials or the worker role to access source and target data.
  3. 3. The Secure Agent stores job dependencies in the staging location using the kops role.
  4. 4. The worker nodes use the connection-level AWS credentials or the worker role to get job dependencies and to stage data in the staging location. The worker nodes use the worker role to store logs in the log location.
  5. 5. The Secure Agent uploads the agent job log to the log location using the Secure Agent role.

Running a job without direct access to Amazon data sources

When an elastic job does not use a connector with direct access to Amazon data sources, the job accesses job resources using the connection properties and the permissions in the worker role.
The following table describes how each resource is accessed:
Resource
Accessed using...
Sources and targets
Connection properties.
For example, a user name and password that you provide in the connection properties might be used to log in to a database on Amazon Aurora.
Staging location
User-defined or default worker role.
Log location
User-defined or default worker role when the job is running.
Secure Agent role when the job is complete.
The Secure Agent requires access to the log location to upload the agent job log. To access the log location, the agent uses the Secure Agent role.
The following image shows how resources are accessed when the job runs on an elastic cluster:
The diagram shows the sequence of events in AWS when you run an elastic job. The worker nodes access source and target data. During the job, the Secure Agent stores job dependencies in the staging location. Then, the worker nodes get the job dependencies, stage data in the staging location, and store logs in the log location. At the end of the job, the Secure Agent uploads the agent job log to the log location.
  1. 1. The worker nodes use the connection properties to access source and target data.
  2. 2. The Secure Agent stores job dependencies in the staging location using the kops role.
  3. 3. The worker nodes use the worker role to get job dependencies from the staging location, to stage data in the staging location, and to store logs in the log location.
  4. 4. The Secure Agent uploads the agent job log to the log location through the Secure Agent role.
Note: If any connector in the job uses AWS credentials to directly access a source or target, the connection-level AWS credentials override the worker role to gain access to the staging location.

Polling logs

When you use Monitor, the Secure Agent accesses the log location to poll logs.
The Secure Agent polls logs based on the type of connectors that the job uses:
Connectors with direct access to Amazon data sources
If the job uses a connector with direct access to Amazon data sources, the Secure Agent uses either credential-based security or role-based security to access the log location. For credential-based security, the Secure Agent polls logs through the connection-level AWS credentials. For role-based security, the Secure Agent polls logs through the permissions in the Secure Agent role.
Connectors without direct access to Amazon data sources
If the job does not use a connector with direct access to Amazon data sources, the Secure Agent polls logs through the permissions in the Secure Agent role.