What are we announcing?

The release of Informatica 10.4.1

 

Who would benefit from this release?

This release is for all customers and prospects who want to take advantage of the latest PowerCenter, Data Engineering Integration, Data Engineering Quality, Data Engineering Streaming, Data Privacy Management, Enterprise Data Catalog,  Enterprise Data Preparation and Test Data Management capabilities.

 

What’s in this release?

This Release provides new features, latest ecosystem & connectivity support, security enhancements, cloud support, and performance enhancements while improving the user experience.

 

In addition, Data Privacy Management and Test Data Management are now included in the Informatica Services installer to improve product compatibility.

Data Engineering Integration (DEI)

New Features:

  • Macro Transformation
    Provides dynamic functionality to transformation logic contained in a mapplet. You can add a transformation that does not support dynamic functionality to a mapplet and assign the mapplet to a Macro transformation.
  • Hierarchical Data on Hive Sources and Targets
    Read from and write to Hive tables with hierarchical data in a mapping that runs on the Spark engine. When you use Hive tables with hierarchical data, you can perform schema synchronization on the mapping.
  • SAML Authentication for Mass Ingestion
    Configure SAML authentication to log in to the Mass Ingestion tool. SAML authentication allows you to use a third-party identity provider to manage user credentials and authorization.

PAM Update

  • CDP-DC 7.1.x (Technical Preview)
  • CDH 5.13
  • WANdisco on HDP 3.14
  • WANdisco on CDH 6.3
  • Isilon support on HDP 3.1.4 with Single Zone

PowerCenter

  • Internet Protocol version 6 support:Informatica supports Internet Protocol version 6 (IPv6) address format in addition to the Internet Protocol version 4 (IPv4) address format.
  • Improved session error logging with new timezone and process ID information.
  • Oracle instant client support:  Enabled for PowerCenter Repository Service, PowerCenter Integration Service, and source and target connections.
  • Security: Update to third party libraries to improve overall security.   

Platform

  • Internet Protocol version 6 (IPv6) support: Deploy the PowerCenter domain, services, repositories, PowerCenter clients, and source and target connections on IPv6. All existing features of PowerCenter continue to work on IPv6.  IPv4 continues to be the default mode. See the support statement for more details  -  https://network.informatica.com/docs/DOC-16182 > Informatica Support Statement for IPv6
  • Domain authentication support: Added for F5 Networks BIG-IP SAML SSO and NetScaler IdP SAML SSO
  • Platform PAM
    • Database support added:  PostgreSQL 11.7
    • Java support updated - IBM JDK 8.0.6.10
    • Tomcat updated- 7.0.103
    • Browser support updated -
      • Microsoft Edge Chromium - Version 81.0.416.53
      • Chrome Version - 80.x
      • Microsoft edge - v44
      • Internet Explorer - 11.10x
  • Model Repository

Added support for the following version control systems:

  • GitHub Enterprise Server
  • GitLab
  • Bitbucket Server
  • SVN
  • Informatica Container Utility
    • Enterprise Data Catalog integration with embedded and existing clusters
    • Data Profiling and Content Management Service
    • Persistent volume support

 

Data Engineering Streaming (DES)

  • Streaming Data Integration
    • Parquet data format for Complex File targets: Persist streaming data into targets in Parquet formats.
    • Dynamic mapping enhancements: Run streaming mappings with dynamic mapping enhanced by refresh-schema-enabled mapping flow. Create dynamic ports in a transformation to receive new or changed columns from an upstream transformation.
    • Bulk ingestion of streaming data: Ingest CDC data from multiple Kafka topics into HDFS and partitioned Hive tables.
    • Improved Hadoop certifications: Cluster security solution certifications.
  • Cloud Streaming Support
    • Databricks on AWS: Run streaming mappings on the AWS Databricks service in AWS cloud ecosystems.
    • Additional connectivity on Databricks: Amazon Kinesis, Amazon Kinesis Firehose, and Amazon S3.
    • JDBC V2 lookup transformation on Databricks cluster.
  • Connectivity Enhancements
    • Apache Kafka and Confluent Kafka support on Databricks.
    • Filename port support for Microsoft Azure Data Lake Store Gen2.
    • Rollover size and time parameters added for ADLS Gen2 and S3 targets (Technical Preview).

Enterprise Data Preparation (EDP)

New Features:

  • Filtered data import into the lake
  • User interface performance improvements
  • Functional improvements of data-preparation capabilities on Amazon S3 data lake

PAM Update

  • CDH 5.13
  • Isilon support on HDP 3.1.4 with Single Zone

 

Enterprise Data Catalog (EDC)

  • Data Asset Analytics: Optimize the data asset value in your data catalog by using analytics on data asset inventory, usage, enrichment, and user collaboration. You can either generate reports or directly connect to event data assets in Data Asset Analytics from business intelligence tools. The event data assets include information on user logins, searches, asset inventory over time, asset changes, asset configuration scan history, user collaboration, asset enrichment, asset lineage, and asset impact.
    You can also use the real-time out of the box dashboards in Data Asset Analytics for storytelling on user adoption, asset inventory, asset enrichment, collaboration, and data value.  Each dashboard comes with real-time, pertinent key metrics, trend charts, and filters for quick analysis.
  • Enhanced change summary: Identify changes to assets more efficiently by using the filter and search options. You can also export the change summary results as a CSV file.
  • Contextual walkthroughs: Accelerate onboarding of new catalog users with in-application tutorials like:
    • Introduction to Catalog Homepage
    • Introduction to Search Results
    • Introduction to Table Overview
    • Introduction to Lineage
    • Introduction to Column Overview
    • Enhancing Asset Credibility
  • Resource configuration permission: Grant resource management permissions to user groups, such as catalog administrators. You can also grant resource management permissions to users while creating a resource.
  • Service and Resource Log Collection: Collect logs for Catalog service and the corresponding Yarn applications, specific resources based on resource name (Technical Preview)
  • Uninterrupted Catalog Backup: Back up the catalog without disabling the Catalog Service (Technical Preview)
  • Discovery
    • Override rules for data domain discovery: You can control the data domain behavior if metadata and data rules conflict with each other.  Conflict resolution reduces the number of false positive data domain assignments in the catalog.
    • Auto assignment of Business Glossary terms
      • Handle cases that are not in order.
      • Factor word-based weight calculation into the overall match score to improve accuracy and reduce false positive cases.
  • Scanners
    • IBM InfoSphere DataStage is now GA.
    • SAP BW is now GA.
    • SAP BW/4HANA is now GA.
    • SAP ECC and S/4HANA: New scanner to extract SAP objects, attributes, descriptions, relationships, transaction codes, programs, and function modules. You can use the SAP Central Instance or SAP Load Balancer connection type. (Technical Preview)
    • ETL scanners improvements: View field-level lineage for flat files.
    • Informatica Cloud Services: View file lineage for cloud storage.
    • Microsoft SSIS: View detailed lineage for transformations, extract metadata from the SSIS database, view the field-level lineage for flat files, and view the control summary of SSIS assets.
    • MicroStrategy: Support for incremental loading and viewing lineage at the report level.
    • Informatica Master Data Management (MDM): Extract metadata from the MDM data source through APIs and view detailed lineage.
    • File partition detection for file system scanners: Identify and publish partitioned files as a single file in the Catalog.
    • Parquet file profiling on Amazon S3 and ADLS Gen2: Support for column profiling and data domain discovery of parquet files on Amazon S3 and ADLS Gen2.
    • ADLS Gen 2 (Profiling): Support for column profiling and data domain discovery for structured files, unstructured file types, and extended unstructured formats.
    • Cassandra (Profiling): Profile Cassandra database tables and views to extract column profiling and data domain discovery statistics.
    • Contextual lineage support for the custom metadata framework: Allows representation of lineage for the same object with multiple execution instances.
    • Hive and Microsoft SQL Server scanner improvements: Reference objects enabled for external database objects allowing to link object across resources through connection assignments.

PAM Update

Scanners:

  • Informatica Platform 10.4.1
  • Informatica PowerCenter 10.4.1
  • Informatica Business Glossary 10.4.1
  • Informatica Data Quality 10.4.1
  • Teradata 16.20
  • DB2 z/OS 12
  • SSIS 2012 to 2016
  • Informatica MDM 10.4

Deployment:

  • EDC (External Clusters)
    • CDH 5.13.x
    • HDP 3.1.4 on Isilon
    • Data Asset Analytics:
      • Oracle DB 18c, 19c
      • MS SQL Server 2017, 2019
      • PostgreSQL 10.6

Informatica Metadata Manager

  • New HTML based data lineage diagramming, replacing Adobe Flash based lineage diagrams.
  • Support for IPv6 naming convention for nodes

 

Data Privacy Management (DPM)

  • New model for domain discovery of unstructured sources: 

A new approach to scanning unstructured data sources for domain discovery with the following benefits:

  • 4x performance improvements with a Remote Agent option that is deployable closer to data sources.
  • Processing of scans on a server separate from the DPM hosted server.
  • Includes natural language processing, keywords, and file tags to improve accuracy.
  • Capture of additional file metadata to provide greater visibility into data  context supporting data minimization and privacy and security operations.
  • An algorithm for calculating confidence scores.
  • Delta scans based on file metadata.
  • A new flat view for listing all unstructured files to support large lists

Users can choose to use existing scan mechanisms or the new Remote Agent-based scans for supported sources. Refer to the DPM Product Availability Matrix (PAM) for a list of unstructured sources supported by the Remote Agent.

  • Privacy Dashboard:

A new Privacy Dashboard for expanded insights into privacy status and privacy-oriented metrics with the following advantages:

  • Provides a quick one-stop view of the status of privacy operations.
  • Call-to-action alerts for critical time-bound subject-related tasks.

Users can easily switch between the Privacy and Security dashboards if they are entitled to the required privileges.

  • DPM Installer Updates:

Data Privacy Management is now included in the Informatica Services installer to improve product compatibility. The Informatica installer includes an option to install Data Privacy Management. This provides clarity on supported versions of Informatica Services and Enterprise Data Catalog when you plan scheduled updates for DPM. Refer to the DPM section in Platform PAM for Supported Operating Systems and Databases for repositories on which DPM can be installed.

  • PAM Updates:
    • Added support for the following repositories:
      • Microsoft SQL Server 2019
      • Oracle 19
  • Added connectivity support for domain discovery and subject registry:
    • Microsoft SQL Server 2019
    • Oracle 19c
    • SAP Hana
    • Snowflake (native)

Test Data Management

Installer:

The Test Data Management installer is now merged with the Informatica Services installer. To install Test Data Management, install Informatica domain services and choose the appropriate install option. The latest version of Test Data Management will now be available with the latest Informatica platform version.

Enhancements:

  • Support for IPv6 naming convention for nodes
  • Fixed high-risk vulnerabilities in application code
  • Optimize dictionary value usage when using substitution masking
  • Improved logging for substitution masking
  • Support for Test data Warehouse in a multi-realm Kerberos environment

Connectivity Updates:

  • Support for new versions of the following databases and distributions: MongoDB, Cassandra, Teradata
  • Support for new version of Cloudera distribution
  • Support for new versions of Mainframe operating (iOS and zOS) and the corresponding IBM DB2 versions and IMS versions have been updated.

See the Product Availability Matrix (PAM) for details.

Data Engineering Quality (DEQ)

  • PAM Update
    • Support Oracle Connection Manager (OCM) for reference tables and exception management

Ecosystems and Connectivity

  • Amazon:

PowerExchange for Amazon Redshift (Data Engineering Integration, Data Quality)      

  • The Amazon S3 bucket to create staging files can be in a different region than the Amazon Redshift cluster.
  • Support to use Amazon Resource Name (ARN) key for SSE-KMS enabled buckets.

PowerExchange for Amazon S3 (Data Engineering Integration, Data Quality)

  • Use a manifest file to read data from Amazon S3 in the native environment.
  • Support to read data from and write data to the Hong Kong region.
  • Directory read support for the binary file format.
  • Support to use the Amazon Resource Name (ARN) key for SSE-KMS enabled buckets.
  • Support to update proxy server settings in the "developer.ini" file.
  • Microsoft Azure:

PowerExchange for Microsoft Azure Data Lake Storage Gen2 (Data Engineering Integration, Data Quality):

  • Read and write ORC flat files in the native environment, on the Spark engine, and on the Databricks engine.
  • Read and write JSON flat files in the native environment.
  • Configure the Azure Government endpoints in mappings in the native environment and on the Spark engine.
  • Configure the authenticated proxy server settings for the Data Integration Service to connect to Microsoft Azure Data Lake Storage Gen2.

PowerExchange for Microsoft Azure SQL Data Warehouse (Data Engineering Integration, Data Quality):

  • Read data from or write data to a Microsoft Azure SQL Data Warehouse endpoint that resides in a virtual network (VNet).
  • Snowflake
    PowerExchange for Snowflake (Data Engineering Integration, Data Quality):
    • Support for external tables as Snowflake sources.
    • Support for materialized views as Snowflake sources.

PowerExchange for Snowflake (PowerCenter):

  • Support for target update overrides in the Snowflake target.
  • Support for "on error abort" for Snowflake target sessions.
  • Support to use external tables as Snowflake sources.
  • Support to use materialized views as Snowflake sources.
  • Google
    PowerExchange for Google BigQuery (Data Engineering Integration, Data Quality):
    • Support for connected and unconnected lookups.
    • Support for Google BigQuery regions available on Google Cloud Platform.

PowerExchange for Google BigQuery (PowerCenter)

  • Support for CDC as a target for Google BigQuery.
  • Support for Google BigQuery regions available on Google Cloud Platform.
  • Support for connected and unconnected lookups.
  • Pushdown optimization enhancements
  • SFDC
    PowerExchange for Salesforce (PowerCenter, Data Quality)
    • Support for Salesforce API version 48.0.
  • SAP

PowerExchange for HANA (PowerCenter)

      • Requires a license (since 10.4).
      • HANA database support for calculation and analytical views.

PowerExchange for NetWeaver (PowerCenter)

      • Dropped support for non-unicode SAP systems (since 10.4).
      • Support to read from SAP tables, views (including CDS views) in HTTP mode of execution using PowerExchange for SAP Dynamic ABAP Table Extractor.
      • Enhancements in SAP Table Reader performance.

PowerExchange for NetWeaver (Data Quality)

      • Support for SAP Table Reader CDS views.

Relational:

    • IPv6 certification for DB2, JDBC, Microsoft SQL Server, ODBC, and Oracle in Data Quality.
    • IPv6 certification for DB2, Microsoft SQL Server, ODBC, Oracle, Sybase ASE and Sybase IQ in PowerCenter.
    • Support to connect to Oracle by using Oracle Connection Manager (OCM) in PowerCenter and Data Quality.
  • No-SQL:
    PowerExchange for MongoDB JDBC (PowerCenter)
    • Introduced a new native adapter "PowerExchange for MongoDB JDBC", with MongoDB 4.2 certification on PowerCenter.
  • Enterprise Data Warehouse

PowerExchange for Greenplum (PowerCenter)

  • Certification for Greenplum 5.8.x GPLoad connectivity on Windows 2012 R2 platform.

PowerExchange for Db2 Warehouse (PowerCenter)

  • Certification for PowerExchange for Db2 Warehouse on AIX and Windows using v11.5 IBM Data Server Driver Package (DS Driver).
  • Hadoop:
    PowerExchange for Hive (Data Engineering Integration)
    • Support to read and write Hierarchical (Htype) data type from Hive tables in a mapping that runs on the Spark engine.
  • Technology:

PowerExchange for Kafka (PowerCenter)

  • Support for additional security properties, such as SASL_PLAINTEXT authentication.

 

PowerExchange CDC and Mainframe

 

  • PAM Changes

Added support for database versions and editions:

  • Adabas Version 8.5.1 on z/OS
  • PostgreSQL Version 12.1 on Linux, Unix, and Windows for CDC
  • MySQL Community Edition on Linux and Windows for CDC, and Amazon Relational Database Service (RDS) implementations of MySQL for CDC with certain prerequisites

Note: Informatica has tested PowerExchange CDC for MySQL Community Edition using the native MySQL ODBC driver. PowerExchange does not provide this driver.

  • IBM i Installer Improvements
    • The IBM i Installer interface includes improvements based on early user feedback, including menu options to save installation parameters to a file, generate an encrypted password, and view the installation log file,  For more information, see the PowerExchange documentation.
  • Security and Connectivity Enhancements
    • For Db2 for z/OS sources, PowerExchange now detects and supports the use of IBM Db2 Huffman compression algorithms. While this support is mostly transparent to users, PowerExchange produces messages to indicate where Db2 Huffman compression has been detected.
  • Performance Enhancement for PowerExchange Express CDC for Oracle in ASM Environments
    • You can optionally write chunks of Oracle redo log to a staging file to improve CDC performance, significantly increase data throughput, and reduce CPU usage in an ASM environment. This feature, previously provided under controlled availability, is now generally available.

 

B2B

 

Intelligent Structure Discovery Enhancements for Data Engineering Integration and Data Integration Streaming:

  • Data type inference for Avro, Parquet and ORC files now uses the same data types that the native connectors use, resulting in unified output across platforms.
  • You can select which element in an array structure is the root node to prevent data loss in the output. For example, when you use an intelligent structure model in midstream in a mapping.

 

Release Notes & Product Availability Matrix (PAM)

 

PAM:

Informatica Platform 10.4.1: https://network.informatica.com/docs/DOC-18676

Test Data Management 10.4.1: https://network.informatica.com/docs/DOC-18677

 

Release Notes:

You can download the Hotfixes from: https://network.informatica.com/downloadsView.jspa.