Data Integration Elastic Administration > Data Integration Elastic on Google Cloud > Task-based access to resources
  

Task-based access to resources

To process data, the Secure Agent and the elastic cluster access the resources that are part of an elastic job, including resources on the cloud platform, source and target data, and staging and log locations.
Resources are accessed differently based on the task that is performed:

Designing an elastic mapping

Designing an elastic mapping is similar to designing a non-elastic mapping in Data Integration. When you design a mapping, the Secure Agent accesses sources and targets so that you can read and write data.
For example, when you add a Source transformation to a mapping, the Secure Agent accesses the source to display the fields that you can use in the rest of the mapping. The Secure Agent also accesses the source when you preview data.
To access a source or target, the Secure Agent uses the permissions in the Secure Agent service account.

Creating an elastic cluster

When you run an elastic job, the Secure Agent creates an elastic cluster. The agent stores cluster information in the staging location and accesses the same resources to start the elastic cluster.
The following image shows the sequence of events when the Secure Agent creates a cluster:
The diagram shows the sequence of events in Google Cloud when the Secure Agent creates an elastic cluster. The elastic job initiates the creation process. The agent receives the job and uses the Secure Agent service account to store cluster information in the staging location. Then the agent creates the elastic cluster.
  1. 1. You run an elastic job.
  2. 2. The Secure Agent stores cluster information in the staging location using the permissions from the Secure Agent service account.
  3. 3. The Secure Agent creates cluster resources and starts the elastic cluster using the permissions in the Secure Agent service account. If you configure user-defined master and worker node roles, the Secure Agent attaches the roles to the cluster nodes.

Running a job

To run an elastic job, the Secure Agent and the worker nodes access sources and targets, the staging location, and the log location.
The following image shows how resources are accessed when a job runs on an elastic cluster:
The diagram shows the sequence of events in Google Cloud when you run an elastic job. The worker nodes access source and target data. During the job, the Secure Agent stores job dependencies in the staging location. Then, the worker nodes get the job dependencies, stage data in the staging location, and store logs in the log location. At the end of the job, the Secure Agent uploads the agent log to the log location.
  1. 1. Worker nodes use the worker node service account to access source and target data.
  2. 2. The Secure Agent uses the Secure Agent service account to store job dependencies in the staging location.
  3. 3. The worker nodes use the worker node service account to access the staging and log locations to get job dependencies from the staging location, to stage data in the staging location, and to store logs in the log location.
  4. 4. The Secure Agent uses the Secure Agent service account to upload the agent job log to the log location.

Polling logs

When you use Monitor, the Secure Agent accesses the log location to poll logs.
To poll logs from the log location, the Secure Agent uses the permissions in the Secure Agent service account.