Informatica® Big Data Management™ allows users to build big data pipelines that can be seamlessly ported on to any big data ecosystem such as Amazon AWS, Azure HDInsight and so on. A pipeline built in the Big Data Management (BDM) is known as a mapping and typically defines a data flow from one or more sources to one or more targets with optional transformations in between. The mappings and other associated data objects are stored in a Model Repository via a Model Repository Service (MRS). In design-time environment, mappings are often organized into folders within projects. A mapping can refer to objects across projects and folders. Mappings can be grouped together into a workflow for orchestration. Workflow defines the sequence of execution of various objects including mappings.
Deployment process overview
For mappings and workflows to be deployed and executed in the run-time, they are grouped into applications. Application is a container that holds executable objects such as mappings and workflows. Applications are defined in the Developer and deployed to a Data Integration Service for execution. Once deployed, Data Integration Service persists a copy of the Application. Application can also be deployed to a file known as Informatica Application Archive (.iar) file, which can subsequently be deployed to a Data Integration Service in same or different domain. The overall process flow for deployment in BDM is as shown here:
The process of deploying a design-time application to an Informatica application archive (.iar) file can be executed via a infacmd CLI with Object Import Export (oie) plugin. A sample of the deploy application command is as follows:
infacmd.sh oie deployApplication -dn $infaDomainName -un $infaUserName -pd $infaPassword -sdn $infaSecurityDomain -rs $designTimeMRSName -ap $applicationPath -od $Output_Directory
The above example uses several user-defined environment variables. They can be named as per the individual organization standards. The password provided is case sensitive. Alternatively, an encrypted password string can be stored in the predefined environment variable INFA_DEFAULT_DOMAIN_PASSWORD. When an encrypted password is used, -pd option is not required. This command is documented in detail in Informatica documentation at Command Reference Guide → infacmd OIE Command Reference → Deploy Application
Once the application archive file is created, it can be optionally checked into GIT or other version control system for audit and tracking purposes.
Subsequently, the application archive file can be deployed to Data Integration Service of the same or different domain. Typically the application archive file is created out of a development domain and is eventually deployed into QA, UAT and Production domains. This can be achieved via infacmd CLI with Data Integration Service (dis) plugin. A sample of such deployment command is as follows:
infacmd.sh dis deployApplication -dn $infaDomainName -un $infaUserName -pd $infaPassword -sdn $infaSecurityDomain -sn $dataIntegrationServiceName -a $applicationName -f $applicationArchiveFileName
This command is documented in detail in Informatica documentation at Command Reference Guide → infacmd DIS Command Reference → Deploy Application. Once deployment is successful the listApplications and listApplicationObjects in the DIS plugin can be used to get a list of the deployed applications and their contents respectively. This information can be used for post-deployment verification / sanity checks.
Integration with Jenkins
The CLI described above can be used to initiate the deployment process from within a Jenkins task. A "Build Step" of type "Execute Shell" can be added to the Jenkins. The step can be configured to execute one of the infacmd commands as shown in the example below
A sample template file for Jenkins is attached (Jenkins-Template-App-Deployment) . The template contains the commands to:
- Create an Informatica Application Archive (.iar) file
- Commit the application archive file to GIT
- Deploy the application into DIS
Informatica BDM jobs can be deployed using Jenkins without any need for 3ʳᵈ party plugins. infacmd CLI commands can be directly used in Jenkins just as they can be used in an enterprise scheduling tool.
- Keshav Vadrevu, Principal Product Manager
- Paul Siddal, Big Data Presales Specialist