Data Integration Elastic Administration > Data Integration Elastic on Microsoft Azure > Task-based access to resources
  

Task-based access to resources

To process data, the Secure Agent and the elastic cluster access the resources that are part of an elastic job, including resources on the cloud platform, source and target data, and staging and log locations.
Resources are accessed differently based on the task that is performed:

Designing an elastic mapping

Designing an elastic mapping is similar to designing a non-elastic mapping in Data Integration. When you design a mapping, the Secure Agent accesses sources and targets so that you can read and write data.
For example, when you add a Source transformation to a mapping, the Secure Agent accesses the source to display the fields that you can use in the rest of the mapping. The Secure Agent also accesses the source when you preview data.
To access a source or target, the Secure Agent uses the connection properties. For example, the Secure Agent might use the user name and password that you provide in the connection properties to access a database.

Creating an elastic cluster

To create an elastic cluster, the Secure Agent stores cluster information in the staging location and accesses the same resources to create the cluster.
The following image shows the sequence of events when the Secure Agent creates a cluster:
This diagram shows the sequence of events in Microsoft Azure when the Secure Agent creates an elastic cluster. An elastic job initiates the creation process. The agent receives the job and stores cluster information in the staging location. Then, the Secure Agent creates the elastic cluster on the Azure cloud.
  1. 1. You run an elastic job.
  2. 2. The Secure Agent stores cluster information in the staging location using the permissions in the managed identity.
  3. 3. The Secure Agent creates cluster resources and starts the elastic cluster using the permissions in the managed identity.

Running a job

To run an elastic job, the Secure Agent and the worker nodes access sources and targets, the staging location, and the log location. Meanwhile, the worker nodes and Azure disks auto-scale as necessary.
The following image shows how resources are accessed when a job runs on an elastic cluster:
This diagram shows the sequence of events in Microsoft Azure when you run an elastic job. The Secure Agent gets the access keys to the staging and log storage accounts so that the worker nodes can access the staging and log locations during the job. The worker nodes access the sources and targets using the connection properties. The Secure Agent also accesses the staging location to store job dependencies. The worker nodes and Azure disks auto-scale as necessary. At the end of the job, the Secure Agent accesses the log location.
  1. 1. The Secure Agent uses the managed identity to gather the access keys to the staging and log storage accounts and uses a secure channel to make the keys available to the worker nodes.
  2. 2. The Secure Agent makes the service principal credentials available to the cluster.
  3. 3. The worker nodes use the connection properties to access source and target data.
  4. 4. The Secure Agent uses the managed identity to store job dependencies in the staging location.
  5. 5. The worker nodes use the access keys to the staging and log storage accounts to get job dependencies from the staging location, to stage data in the staging location, and to store logs in the log location.
  6. 6. Worker nodes and Azure disks auto-scale using the permissions in the service principal.
  7. 7. The Secure Agent uses the managed identity to upload the agent job log to the log location.

Polling logs

When you use Monitor, the Secure Agent accesses the log location to poll logs.
To poll logs from the log location, the Secure Agent uses the permissions in the managed identity that is assigned to the Secure Agent machine.