Data Integration Elastic Administration > AWS integration tasks > Define master and worker roles
  

Define master and worker roles

Define master and worker IAM roles to pass permissions to the master and worker nodes in an elastic cluster. You can create user-defined roles or use default roles.
You can define one of the following types of master and worker roles:
User-defined roles
If you create user-defined roles, you run a script to generate policy content that includes the permissions that the master and worker roles require. You can edit certain elements in the policy content to restrict a role's access to cloud resources.
Default roles
If you use default roles, the Secure Agent creates default IAM roles for the master and worker nodes.
The following table shows a comparison of each option:
Area
User-defined roles
Default roles
Creation of master and worker roles
You have greater visibility of the master and worker roles and the policies that are attached to each role.
Roles are created automatically and it is more difficult to monitor the policies that are attached to each role.
Ability to edit policies
You can restrict some resources in the policies.
You cannot edit the policies.
Number of IAM permissions that the kops role requires
Fewer IAM permissions are required.
More IAM permissions are required.
Credential-based security for direct access to Amazon data sources
No impact on master and worker roles.
No impact on master and worker roles.
Role-based security for direct access to Amazon data sources
You must manually verify that the worker role and the Secure Agent role can both access the data sources that you use in an elastic job.
You can also configure cross-account access to S3 buckets in multiple Amazon accounts.
You must only verify that the Secure Agent role can access the data sources that you use in an elastic job. The worker role can always access the same data sources as the Secure Agent role because the policies that are attached to the Secure Agent role are automatically attached to the worker role.
You cannot configure cross-account access to S3 buckets in multiple Amazon accounts.
Role-sharing
You can use the same master and worker roles across multiple elastic configurations.
Separate master and worker roles are created for each elastic configuration. You cannot reuse roles.
Modifying staging and log locations
You must manually update the staging and log locations in the policies.
Policies are automatically updated.
Product upgrades
A product upgrade might change the policies that the master and worker roles require. If the policies change, you must regenerate the policy content and restrict access to resources again.
Policies are automatically updated.
For more information about the master and worker roles, see Task-based access to resources.

Create user-defined master and worker roles

Create user-defined master and worker roles to define policies for the master and worker nodes in an elastic cluster.
To create user-defined roles, complete the following tasks:
  1. 1. Generate policy content for the master and worker roles.
  2. 2. Restrict the generated policy content.
  3. 3. Use IAM to create the master and worker roles.
  4. 4. Allow the kops role to assume the worker role.
  5. 5. Optionally, allow the kops role to assume the master role.
The master and worker roles, the instance profiles, and the kops role must be defined under the same AWS account.
After completing these tasks, you can specify the master and worker instance profiles in an elastic configuration.
When the Secure Agent starts the elastic cluster, the agent uses the kops role to validate whether the instance profiles exist and whether the master and worker roles have access to required cluster directories, such as staging and log locations. If validation fails, the cluster fails to be created.

Step 1. Generate policy content for master and worker roles

Generate the policy content for the master and worker roles.
To generate the policy content, run the generate-policies-for-userdefined-roles.sh command. For more information about the command, see generate-policies-for-userdefined-roles.sh.
The command creates the output file my-userdefined-master-worker-role-policies.json.
The following table lists the policies in the output file:
Policy
Attach to which roles?
When is the policy required?
Minimal requirements for the master role
Master
Always.
Minimal requirements for the worker role
Worker
Always.
Auto-scale EBS volumes
Worker
Required only if EBS volumes auto-scale.
Access to staging and log locations
Master
Worker
Always required for the worker role.
Required for the master role only if you use an initialization script.
Access to the initialization script path and the location that stores init script and cloud-init logs
Master
Worker
Required for the master and worker roles only if you use an initialization script.

Step 2. Restrict the generated policy content

Restrict elements in the generated policy content to restrict the resources that nodes in the elastic cluster can access.
You can restrict the following elements depending on their values:
Resource elements with the value *
If the value for a Resource element is the wildcard *, you cannot restrict the resources.
For example, the generated policy for the master node might have the following statement:
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes"
],
"Resource": [
"*"
]
},
Because the value for the Resource element is the wildcard *, you cannot edit the Resource element.
If you edit a Resource element whose value is the wildcard *, the Secure Agent might fail to identify the required resources to start an elastic cluster and the cluster might not start properly.
If you encrypt staging data and log files using SSE-KMS, you can edit the resources in the statement that contains actions on AWS Key Management Service (KMS) even though the Resource element is the wildcard *. For more information, see Encrypt staging data and log files at rest.
Resource elements without the value *
If the value for a Resource element is not the wildcard *, you can restrict the Resource element to specify the resources that the statement covers.
For example, a generated policy for the worker node might have the following statement:
{
"Effect": "Allow",
"Action": [
"s3:Get*"
],
"Resource": [
"arn:aws:s3:::<cluster-staging-dir1>/*",
"arn:aws:s3:::<cluster-staging-dir2>/*"
]
},
Because the value for the Resource element is not the wildcard *, you can edit the resources in the statement. In this example, you can restrict the Resource element to the S3 resources that define one or more staging locations.
You can provide staging and log locations for multiple elastic clusters to share the same policy content between clusters that use different elastic configurations.
To avoid cross-region data-transfer costs, use S3 buckets that are in the same region. To help you manage each bucket, use different buckets for staging locations, log locations, and data sources.

Step 3. Use IAM to create the master and worker roles

Use IAM to create the master and worker roles and attach each policy to the appropriate role.
You can define each policy as an inline policy or a managed policy.
Note: Do not copy and paste the comments in the JSON file into the policy documents.
When you create the master and worker roles, AWS automatically generates an instance profile for each role. You can specify the profiles in an elastic configuration.
If the policy content provides access to staging and log locations for multiple elastic clusters, you can reuse the same instance profiles across different elastic configurations.

Step 4. Allow the kops role to assume the worker role

The kops role must be able to assume the worker role so that the Secure Agent can validate access to required cluster directories.
To allow the kops role to assume the worker role, add the kops role to the trust relationship of the worker role.
For example, edit the trust relationship of the worker role and specify the following policy:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"AWS":[
"arn:aws:iam::<AWS account>:role/<kops role name>"
],
"Service":"ec2.amazonaws.com"
},
"Action":"sts:AssumeRole"
}
]
}

Step 5. Allow the kops role to assume the master role

If you use an initialization script, the kops role must be able to assume the master role so that the Secure Agent can validate access to required cluster directories.
To allow the kops role to assume the master role, add the kops role to the trust relationship of the master role.
For example, edit the trust relationship of the master role and specify the following policy:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"AWS":[
"arn:aws:iam::<AWS account>:role/<kops role name>"
],
"Service":"ec2.amazonaws.com"
},
"Action":"sts:AssumeRole"
}
]
}

Use default master and worker roles

If you choose to use default mater and worker roles, the Secure Agent automatically creates the master and worker roles when the agent starts an elastic cluster.
The Secure Agent creates a master role for the master node and a worker role for the worker nodes. The agent attaches policies to the master and worker roles to add permissions that are required by Kubernetes services.
If you use role-based security for direct access to Amazon data sources, the agent also identifies the policies that are attached to the Secure Agent role and passes the policies to the worker role.
If you use default roles, you must add the following IAM permissions to the minimal policy for the kops role:
"Action": [
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:DeleteInstanceProfile",
"iam:DeleteRole",
"iam:DeleteRolePolicy",
"iam:GetInstanceProfile",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:GetUser",
"iam:ListAttachedRolePolicies",
"iam:ListInstanceProfiles",
"iam:ListInstanceProfilesForRole",
"iam:ListRolePolicies",
"iam:ListRoles",
"iam:PassRole",
"iam:PutRolePolicy",
"iam:RemoveRoleFromInstanceProfile",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:CreateServiceLinkedRole",
],
"Resource": "*"
}
]
}