Data Integration Elastic Administration > Introduction to Data Integration Elastic administration > Lifecycle of an elastic cluster
  

Lifecycle of an elastic cluster

A cluster lifecycle is the sequence of events that occurs on the elastic cluster.
The cluster lifecycle includes the following events:
  1. 1. The agent creates an elastic cluster.
  2. 2. Jobs run on the cluster.
  3. 3. The agent stops the cluster.

The agent creates a cluster

To create an elastic cluster, the Secure Agent uses the elastic configuration that you associate with a job's runtime environment.
The agent performs the following tasks:
  1. 1. Creates a cluster configuration that includes configuration information about the elastic cluster. The configuration is stored using YAML files that the Secure Agent populates.
  2. 2. Provisions the necessary resources to create an elastic cluster. The agent deletes the resources when the cluster stops.
Note: Informatica uses a secure pathway to fetch job-related container images for cluster nodes from the Informatica-specific JFrog repository. It also accesses the public internet to fetch files that are required to create the logical cluster layer on cluster nodes.

Jobs run on the cluster

After an elastic cluster is created, the agent pushes the elastic job to the cluster and leverages the Serverless Spark engine to process the data logic in the job.
When you run an elastic job, the agent generates an execution plan which is a set of instructions for the Spark engine. The execution plan divides the data logic into multiple Spark tasks, and Spark drivers and executors are launched to process the Spark tasks simultaneously.
As developers run additional elastic jobs over the course of the cluster lifecycle, the cluster adapts to the size and number of jobs as resources are provisioned and deprovisioned, such as cluster nodes and cluster storage.
For each job, the agent generates a session log, a Spark driver log, and Spark executor logs. The agent also generates an agent job log for each Spark task in the job.

The agent stops the cluster

After all jobs are complete, the Secure Agent determines when to stop the cluster based on the cluster shutdown method that you select in the elastic configuration.
The agent can perform a smart shutdown that is based on historical data or stop the cluster based on an idle timeout that you specify.
The Secure Agent also stops the cluster in the following situations:
After the Secure Agent stops the cluster, the agent verifies that all cluster resources are deleted, except for some Informatica binaries that remain in the staging location in the infa_rpm.tar file. The binaries are required in order to run jobs on the cluster, and the file is reused the next time that the agent starts the cluster.
The agent deletes the infa_rpm.tar file in the following situations:
The agent restarts the cluster when you run another elastic job.