What are we announcing?
The release of Informatica 10.4
Who would benefit from this release?
This release is for all customers and prospects who want to take advantage of the latest PowerCenter, Data Engineering Integration, Data Engineering Quality, Data Engineering Streaming, Enterprise Data Catalog, and Enterprise Data Preparation capabilities.
What’s in this release?
This update provides the latest ecosystem & connectivity support, security enhancements, cloud support, and performance enhancements while improving the user experience. Also, the Big Data product family is renamed to Data Engineering.
The following product names have changed:
- Big Data Management has changed to Data Engineering Integration.
- Big Data Quality has changed to Data Engineering Quality.
- Big Data Streaming has changed to Data Engineering Streaming.
- Big Data Masking has changed to Data Engineering Masking.
Enterprise Data Catalog and Enterprise Data Preparation are aligned within the Data Catalog product family.
Data Engineering Integration (DEI)
- CI/CD & REST initiatives: Use REST APIs to deploy, update, and query objects and to compare mappings that you develop in a CI/CD pipeline.
- CLAIRE® recommendations and insights: Provides best practices recommendations for mappings during design time. It also gives insights into mapping design patterns.
- Debugging enhancements: Collect aggregated cluster logs for a mapping in the Monitoring tool or by using an infacmd ms command.
- Blockchain support: Connect to a blockchain to use blockchain sources and targets in mappings that run on the Spark engine. (Technical Preview)
- Data Processor on Spark: Process unstructured and semi-structured file formats using the Data Processor transformation on the Spark engine.
- Profiling on Spark: Run profiles and choose sampling options on the Spark engine. You can perform data domain discovery and run scorecards on the Spark engine.
- Hierarchical Data Processing enhancements:
- Midstream hierarchical data parsing: Parse hierarchical JSON and XML data in a midstream string port using intelligent structure models and complex functions.
- Data preview: Preview hierarchical data using the Spark Jobserver in the Amazon EMR, Cloudera CDH, and Hortonworks HDP environments. Spark Jobserver allows for faster data preview jobs.
- Intelligent Structure Discovery improvements: Processes additional input types such as ORC, Avro, and Parquet, creates an intelligent structure model from a sample file at design time and arranges unidentified input data in the sample file as a structured JSON format in the output model.
- Security: Support for cross-realm Kerberos authentication and multiple LDAP servers in a single Informatica domain
- Leverage cloud infrastructure: Installing PowerCenter on AWS is easier with added support for Azure Linux as server OS and PostgreSQL as a repository database
- Transformation enhancements: Enhanced ability to consume REST web services using an HTTP transformation with added support for PUT, PATCH and DELETE methods
- Productivity enhancements: New refresh option to refresh metadata in the PowerCenter Designer and Workflow Manager without requiring you to log in again
- New connectors: PowerExchange for DB2 Warehouse, PowerExchange for PostgreSQL, PowerExchange for Dynamics 365 Sales (Rest API Based)
- New Advanced Edition: New paid offering, Operational Insights Advanced Edition on top of Operational Insights Base Edition, which is free
- New Operator Console: Operational analytics console with a projects view for effectively viewing PowerCenter, Data Engineering Integration, Data Engineering Quality, and Data Quality operational data with permissions management with Advanced Edition
- Claire alerts: Auto-detection and alerts for job anomalies in PowerCenter workflows based on the elapsed time and data processed with Advanced Edition
· Database support added:
- Oracle 19c
- SQL Server 2019
· Operating system support added:
- Amazon Linux2
- Windows 2019 Server
· Operating system support dropped:
· Java support
- Azul OpenJDK 1.8.0_222
- IBM JDK 184.108.40.206
- Tomcat 7.0.96
- Browser support
- Microsoft Edge Browser (Win 10) 44.18
- Internet Explorer 11.x
- Google Chrome 75.x
- Safari 12.1.1
- Support for cross realm Kerberos authentication to allow Informatica nodes, application services, and users to belong to different Kerberos realms.
- Support for Smart card-based Kerberos single sign-on.
- Support for multiple LDAP servers in a single Informatica domain.
- Support for Oracle Universal Directory for LDAP authentication.
- Support for defining custom LDAP types.
- Support for Microsoft Azure Active Directory for secure LDAP authentication.
- Support for Microsoft Active Directory Federation Services 4.0 and PingFederate as Security Assertion Markup Language (SAML) identity providers.
- Version Control System Support added:
- Collabnet Subversion Edge 5.2.4
- Bitbucket Server 6.4
- Perforce 2019.1
- Visual SVN 4.0.2
- Version Control System Support dropped:
- Collabnet Subversion Edge 5.2.2
- Perforce 2014.2
- Visual SVN - 3.6, 3.7
Informatica Container Utility
- A quick and easy process to install the Informatica domain configured with Docker and Kubernetes for PowerCenter and Data Engineering Integration with a new UI-based utility. You can create the standard Docker images for PowerCenter and Data Engineering Integration and then can create containers using Docker or Kubernetes. It also provides an option for one-click deployment by downloading and running the standard pre-built image of PowerCenter and Data Engineering Integration.
- To download: https://marketplace.informatica.com/listings/on-premise/solutions/informatica_powercenter_docker_utility.html and https://marketplace.informatica.com/listings/on-premise/solutions/informatica_big_data_management_docker_utility.html
Data Engineering Streaming (DES)
Streaming Data Integration
- Data Quality transformations in streaming mappings: Apply real-time Data Quality transformations to streaming data.
- Lineage for streaming mappings: Ability to view the lineage for streaming mappings in Enterprise Data Catalog.
- Enhanced OS support: Data Engineering Streaming is now supported on SuSE Linux.
- Dynamic mapping support: Ability to run streaming mappings with dynamic mapping support. (Technical Preview)
- CLAIRE®integration for industry standard data parsing: Ability to parse HL7 messages using Intelligent Structure Discovery integration. (Technical Preview)
- Enhanced support for CDC ingestion to Hadoop.
- Enhanced monitoring capabilities for streaming jobs.
- Latest Spark and Hadoop distribution support.
- Confluent schema registry support: Ability to parse complex messages from Kafka using schema from schema registry.
- SSL support with Kafka: Ability to connect to secure Kafka cluster.
Cloud streaming support
- Ephemeral cluster support: State support in cloud repositories and workflow support.
- Serverless streaming in Azure Databricks: Ability to run streaming jobs in an Azure Databricks cluster.
Enterprise Data Preparation (EDP)
- Upload files directly to the data lake: Data analysts and data scientists can now initiate the data preparation process by uploading files directly to the data lake, without waiting for IT to fulfill their request.
- Microsoft Excel support: Excel files are now supported for data preparation. CLAIRE® technology helps in automatic table structure discovery used for data preparation and publication. (Technical Preview)
- Publish as files: File-based data assets can now be prepared and published to the data lake as files without dependence on hive or relational layer.
- ADLS Gen2 support: Support for file-based data preparation on ADLS Gen2.
- NULL value handling: Intelligent handling of NULL values during data preparation and publication.
- Privilege control: You can now enforce privilege control on various user activities.
- Stabilization and performance: Stabilization and performance improvements of data preparation projects from intermittent failures and slowness.
Enterprise Data Catalog (EDC)
- Snowflake: New scanner that can extract object and lineage metadata. Lineage metadata includes view to table lineage.
- Cassandra: New scanner to extract metadata Cassandra keyspaces, tables and views. Profiling is not supported.
- AWS Glue: New scanner to extract metadata from the Glue catalog. The Glue scanner can extract metadata from sources in the AWS environment (S3, Redshift, DynamoDB, RDS) as reference objects. Scanner draws lineage from Glue objects to source objects. Lineage for ETL jobs is not supported.
- Informatica Data Quality:
- Extract rules, scorecards definitions and results as well as profiling and data domain discovery statistics from an IDQ or a BDQ model repository service and profiling warehouse. Users who have built Data Quality processes in IDQ/BDQ can now extract the quality scores and visualize them in EDC.
- Extract profiling and data domain discovery statistics from an IDQ or a BDQ profiling warehouse. Users who have already run profiling and enterprise discovery in IDQ/BDQ can now extract these profiling results and visualize them in EDC.
- Azure Power BI: New scanner to extract metadata for Workspaces, Dashboard, Reports, Datasets and Dataflows as well as lineage between them.
- Google Cloud Storage: New scanner to extract metadata from files and folders. Refer to the PAM for supported file formats. Profiling is not supported.
- SAP BW: New scanner to extract metadata, lineage, and relationships between SAP Business Warehouse objects. Profiling is not available. (Technical Preview)
- SAP BW/4HANA: New scanner to extract metadata, lineage, and relationships between SAP BW/4HANA objects. Profiling is not available. (Technical Preview)
- Informatica Data Engineering Streaming: The Informatica platform scanner now supports extracting metadata from streaming mappings, including streaming sources. Streaming sources are created as reference objects.
- Google Big Query (Profiling): Support for column profiling and data domain discovery.
- SAP HANA Database (Profiling): In addition to the extraction of the metadata, EDC is now capable of profiling the SAP HANA database tables and views to extract column profiling and data domain discovery statistics.
- Reference Objects: Extract data lineage and referred objects directly from ETL, BI, and catalogs. Users can search, annotate, govern and view lineage for reference objects.
- Offline scanner support added for:
- Amazon Redshift
- Amazon S3
- Azure Data Lake Store
- Azure Microsoft SQL Data Warehouse
- Azure Microsoft SQL Server
- Google BigQuery
- Microsoft Azure Blob Storage
- Business Glossary
- Custom Lineage
- Database Scripts
- Informatica Intelligent Cloud Services
- SAP PowerDesigner
- IBM Cognos
- QlikView Business Intelligence
- SAP HANA
- AWS Glue
- Google Cloud Storage
- SAP BW
- SAP BW/HANA
- Custom Scanner Enhancements:
- Support for profiling of sample files: Author of custom metadata can now provide sample data files that will be used to compute profiling statistics.
Business User Experience
- Summarized lineage view: Users can view lineage at any level from the highest, system-wide view to the granular, field-level view. (Technical Preview)
- Lineage filters: Users can now apply filters on the lineage view to hide object types and associated links from the view for better clarity.
- Control lineage: Object dependency generated by SQL statement Where clauses or lookup are considered as control lineage and now reported in EDC in the tabular summary view of the lineage diagram.
- Resource-level attributes: Administrators can create custom attributes assigned to specific resources instead of the class type of any resource.
- Export from search results: Users can now export a list of objects from search results. Import is now possible at a global level containing objects from multiple sources. Import and export jobs can be monitored in a central UI.
Data Provisioning (GA)
- Data Provisioning: After discovery, users can now move data to a target where it can be analyzed. EDC works with IICS to provision data for end-users. Credentials are supplied by the users for both the source and the target.
- Databases: Oracle, SQL Server, Teradata, Hive, Redshift, Azure SQLDB, Azure SQL DW, JDBC
- Data Lakes: S3, ADLS, Blob, HDFS
- Applications: Salesforce
- BI: Tableau Server, Qlik,
- Data Lakes: S3, ADLS, Blob, HDFS, Google Cloud Storage
- Databases: Hive, Redshift, Google Big Query, Azure SQLDB, Azure SQL DW, JDBC, Teradata, Oracle, SQL Server
- Live Data Preview: Users can now preview source data at the table level by providing source credentials.
- Unique Key inference: The derived key (PK) information from datasets allows users better understand the characteristics of datasets.
- Data Domain Discovery in Text (CLOB) fields: Data Domain discovery is now applied to CLOB fields during database source profiling.
- Profiling on spark: Administrators can run profiles using the Spark engine for selected sources.
- Deployment Support Added
- Hortonworks HDP 3.1 GA
- Source Support
- Hive, HDFS on CDH 6.1, 6.2
- Hive, HDFS on HDP 3.1
- Informatica Data Quality 10.2HF2, 10.4
- SAP BW 7.4 and 7.5
- Oracle 19c
- MS SQL Server 2019
- Cassandra 3.11
- Informatica Platform 10.2 HF2, 10.2.2HF1, 10.4
- Informatica PowerCenter 10.2 HF2, 10.4
- Azure Power BI
- Google Cloud Storage
- AWS Glue
Data Engineering Quality (DEQ)
- Profiling on Spark: Administrators can now run profiles using the Spark engine for selected sources.
- Address Verification: Update to Address Verification Engine to 5.15
(PAM and Platform updates as per DEI)
Cloud Ecosystems and Connectivity
- S3 deferred policy check
- Support for Aurora Postgres
- Filename ports for both batch and streaming
- Microsoft/ Azure:
- ADLS Gen 2: New connector for the native and Spark environments including streaming across HDInsight and Databricks
- SQL DW: VNET SE authentication support, PDO support via ODBC, Proxy support
- Blob: Support for Shared Access Signature (SAS) authentication
- Azure SQL DB: Support for managed instance
- SQL Server always encrypted support
- New Dynamics 365 Sales (common data service) support
- Dynamic mapping support for Snowflake
- Performance improvement
- Database Push Down Optimization (PDO)
- Support for Snowflake target in streaming mappings. (Technical Preview)
- BigQuery connector optimized in performance and scale for Sparkexecution
- Support for global regions
- Folders support for Google cloud storage
- Databricks is now supported on both Azure and AWS ecosystems as well as for Snowflake
- Databricks Delta Lake support with additional transformations including Streaming pipeline support (Technical Preview)
- Support for API 47
- Salesforce Marketing Cloud support
- Core Data Services (CDS) support
- Calculation and Analytics Views
- Additional datatypes support including: HANA DB LTRIM RTRIM
- Oracle 19c RAC support
- Essbase 220.127.116.11 Certification
- JD Edwards Enterprise One 9.2 Certification
- Enterprise Data Warehouses
- Greenplum 5.1 support
- DB2 Warehouse on Cloud support
- Minor enhancements to Teradata connector
- New Kafka connector for PowerCenter and PowerCenter real-time.
- PowerExchange for JDBC V2: Support for Spark and Databricks to connect to Aurora PostgreSQL, Azure SQL Database, or any database that supports the Type 4 JDBC driver.
- Enhancements to Complex File, Teradata, MongoDB, Cassandra connectors
- Enhanced ability to consume a REST service using an HTTP transformation with added support for the PUT, PATCH, and DELETE methods
- Support for MQ 9.1, JMS 2.0
PowerExchange (PWX Mainframe & CDC)
- Database version support added:
- Db2 for i 7.4
- Db2 LUW V11.5
- Oracle 18c
- MySQL 8.0
- PostgreSQL 10.x and 11.4
- Windows 2019 for source & target
- Dropped support for CICS/TS 4.1
- Database version support dropped:
- z/OS Adabas 8.1 and 8.2
- z/OS Datacom 12 and 14
- z/OS Db2 9.1 and 10
- z/OS IDMS 17 and 18
- z/OS IMS 10, 11, and 12
- Db2 for I 7.1
- LUW Oracle 11g R2 & 12c R1
- Operating system support dropped:
- z/OS 1.11, 1.12, and 1.13
- i5/OS 7.1
New and Extended Utilities
- A new IBM I installer that runs on Windows guides users through installing and configuring PowerExchange on i5/OS. It can perform full and upgrade installations.
- The z/OS Installation Assistant now supports IBM PDSE data sets.
- The new PWXUMAP utility provides additional reporting capabilities for data maps, extraction maps, and source schemas.
- Performance of the DTLURDMO utility when processing a large number of data maps has been improved.
General Enhancement or Feature Updates
- PostgreSQL has been added as a CDC source.
- PowerExchange can now read DBD information from the IBM IMS catalog when you create data maps or at CDC or IMS unload run-time.
- Oracle 18c has been certified for operation with PowerExchange. Oracle Express CDC to run as per previous versions of PowerExchange for Oracle CDC capabilities.
Intelligent Structure Discovery Enhancements for Data Engineering Integration and Data Integration Streaming:
- Ability to process additional input types: ORC, Avro, Parquet.
- Ability to process HL7 messages. (Technical Preview)
- Users can create a model based on a sample file in Data Engineering Integration at design time, without having to go through the Informatica Cloud design flow first.
- The output of unidentified data is structured in a JSON format.
- Informatica 10.4: https://docs.informatica.com/big-data-management/shared-content-for-data-engineering/10-4-0/release-notes/abstract.html
- PowerExchange Adapters for Informatica:https://docs.informatica.com/data-integration/powerexchange-adapters-for-informatica/10-4-0/powerexchange-adapters-for-informatica-release-notes.html
- PowerExchange Adapters for PowerCenter:https://docs.informatica.com/data-integration/powerexchange-adapters-for-powercenter/10-4-0/powerexchange-adapters-for-powercenter-release-notes.html