What's new in IICS Winter ‘19?

<This document is currently in draft mode. please check back at Winter'19 GA (Mar '2019) for final version>

The Informatica Intelligent Cloud Services (IICS) Winter 2019 release offers several new capabilities that address key data challenges that businesses are facing today. Highlights are listed below.

Data Integration

  • Data discovery in Cloud Data Integration with Enterprise Data Catalog (EDC) integration - Customers can now search and discover enterprise-wide metadata from within Data Integration, import connection & object metadata, and use that information to more easily create new or enrich existing mappings and tasks by connecting with an existing EDC installation.
  • “Smart match” recommendations for field mappings increases the frequency of field matches in mappings and tasks. Expanding on the existing automatch, smart match looks for common patterns in field names (prefixes, suffixes, abbreviations, etc.) based on 6 additional matchers and fuzzy match techniques for recommending field mappings.
  • API access to taskflows for externalized scheduling & execution. With this enhancement, customers now can call taskflows on demand via an API call and dynamically provide input parameters for the tasks that it orchestrates to fully leverage Data Integration’s parameterization capabilities. Taskflows have also been enhanced to allow them to invoke other taskflows (a.k.a sub-taskflows), allowing for increased taskflow reuse.
  • Change data capture has been expanded to include additional sources for Db2 on Linux, Unix, Windows, and iSeries (AS400, i5/OS) platforms, which further enables near real-time changed data propagation capabilities.
  • Mass ingestion is extending connector support, adding Google Storage & Google Big Query as targets and HDFS as both a source and target. Additional enhancements expose CRUD-focused APIs.

API & Application Integration

  • Support for Kafka Messaging – Messaging is at the core of many publish-subscribe (Pub/Sub) based applications as a means of decoupling producers and consumers of data.  The addition of Kafka for application integration significantly increases current message-based Pub/Sub interactions between data and applications that today are fulfilled using JMS, AMQP, Amazon SNS/SQS, and Azure Service Bus based “topics”. The ability to bridge these message-based events with Cloud Integration Hub Pub/Sub style of data propagation provides additional integration pattern options making Informatica unique in the flexibility and capabilities it providesits customers.
  • JSON Web Token (JWT) based authentication –API and Application Integration services now support JSON Web Token (JWT) based authentication, an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between API consumers and REST web services. This provides IICS users with another and more secure means of API authentication.
  • OData Provider services for Impala and Hive – The OData Provider adds support for big data sources. Impala and Hive join MySQL, DB2, SQL Server, and Oracle databases as data platforms that can now be accessed via the OData v4 standard data access API. This gives customers the ability to better leverage the data they have amassed as part of their big data initiatives.
  • Japanese language support for API and Application Integration services – In addition to Japanese language support by the Data Integration service, Japanese customers can now have access to the API and Application Integration servicesuser interface and documentation in Japanese.
  • REST and SOAP Service API-based “service connectors” – distributed via GitHub.

Today 55% of Application Integration customer’s connectivity needs are met using service connectors. A service connector allows customers to define REST (XML/JSON), JSON/RPC, or SOAP service integration using simple a web form, with no coding required. The Application Integration service takes care of the rest. If the service offers a WSDL or Swagger interface document, the service connector can be automatically generated by importing the interface document. By creating service connectors, customers can import and configure pre-built business and data service definitions as reusable assets that they can share and/or move from one environment to another. 

This capability, unique to Informatica, provides customers with unparalleled value. Service connectors avoid lock-in or an inability to make updates as you need it to take advantage of new capabilities or built-in extensibility that an API platform may offer. 

To provide this flexibility to customers and encourage community contribution by customers, partners and other practitioners, Informatica is establishing a GitHub repository where it will publish service connectors it has created and which it will share with its customers and partners. Customers and partners are free to use these definitions without restriction, including the rights to use, copy, modify, merge, publish, and distribute these under an MIT license. Informatica will also encourage contributions back to the community. Our goal is simple: drive  innovation and reduce perceived barriers to adoption.

  • The DiscoveryIQ Service now offers additional usage reports specifically targeted to Application Integration users.

 

Integration Hub

  • Informatica has improved the search experience for Hub events and support for the CLOB data type on topics.

 

B2B Gateway

  • ICS B2B Gateway customers will be migrated to the IICS platform as part of the release upgrade and will benefit from all IICS platform capabilities.

 

Intelligent Structure Discovery (ISD) - Note that the below features of ISD are not available in pre-release but are only available from GA.

  • Intelligent Structure Discovery expanded its parsing capabilities to handle ORC format files and Excel multiple sheet files – the user can now design the structure model based on multiple sheet structures and then use the model in runtime to parse Excel files in their entirety.
  • With R31 Structure Parser transformation asset can be positioned in midstream to enable a more flexible mapping usage and chaining. In addition the Intelligent Structure Model detected datatypes are now propagated to the Structure Parser output ports.
  • The ISD design time user interface is enhanced with a "Find" functionality which allows the user to search for a specific string in the discovered tree fields and get a list of results showing the path and visually correlated with the model representation. The user can also perform actions on multiple elements chosen from the result list such as include, exclude, replace and even change of element type. The ability to perform action on multiple elements significantly improves the usability and productivity.
  • A new vertical folder view mode will be available in R31 for handling complex hierarchies files.

 

IICS Platform

  • Common Explore productivity enhancements – Improved copy functionality with overwrite & rename conflict resolution options to copy assets within and across folders. “Created by” and “Last updated by” attributes as columns for all asset types in the common Explore page.
  • Export/import capability for sub-organizations which enables asset migration across environments that use an organization hierarchy. More control and flexibility with enable/disable checksum validation options during export and import.
  • Improved export/import error logging along with ability to access and download export/import logs through the UI and the API.
  • API to search, list, and filter assets in projects and folders using a variety of conditions such as timestamp, location, “last updated by,” and tags. This API can also be leveraged along with export APIs to export objects.
  • Improvements to the RunAJob utility – Support for projects and folders to invoke tasks by task name.
  • Search capability for connections and job monitoring to quickly and easily find needed information.
  • Ability to enable and disable a service for agents in a Secure Agent group to effectively deploy workloads and efficiently utilize computing resources.
  • Secure Agent registration using tokens (instead of passwords) for increased security and enabling SAML single sign-on.

 

Operational Insights

  • Operational Insights makes its debut on IICS! Application Integration will be the first service to get the benefits of Operational Insights. This will add the ability to obtain valuable usage information that will drive a variety of reports and analytics which over time will give IICS customers unique visibility into the operational aspects of their integrations.
  • Operational Insights extends support to the on-premises Data Quality product, in addition to BDM and PowerCenter, with capabilities such as domain health, job run analytics, resource utilization, and alerts.
  • Click-through analytics journey from cross-domain to the individual job level and enhancements to job run analytics for on-premises products (PowerCenter, BDM). Plus, enhancements to job run analytics to report on data loaded and data processed.
  • PowerCenter & BDM infrastructure e-mail alert enhancements such as Secure Agent unavailability and Operational Insights data collection failures.
  • Operational Insights is available in EMEA from this release onwards.

 

Connectivity Enhancements

New AWS Quickstart for Cloud Analytics Modernization, an end-to-end solution for self-service cloud analytics with Informatica (IICS, Enterprise Data Catalog), Tableau Server, and AWS Services.

 

Several new connectors and enhancements to existing connectors across ecosystems as listed below. New connectors introduced are highlighted in bold:

 

    • Azure: ADLS Gen 2 Preview, Azure DW V3, Azure Data Lake Store V3, Azure Blob V3
    • Google: Google Cloud Storage V2, Google Analytics, Google Big Query V2, Google Big Query
    • Amazon: Amazon S3 V2, Amazon Aurora, Amazon Redshift V2
    • Salesforce: Salesforce Marketing Cloud (SFMC), SFDC (Sales and Service)
    • SAP: SAP Connector, SAP HANA Cloud Platform (DB)
    • Adobe: Adobe Cloud Platform
    • Analytics: CDM Folders connector, Tableau V3, Tableau V2
    • Databases and DW: MySQL Relational, Hive, Greenplum, DashDB, Snowflake
    • Tech: REST V2, WSconsumer, Complex File Processor
    • Microsoft Apps: MS Dynamics 365 Operations, MS Dynamics 365 CRM (SOAP), Sharepoint Online
    • Oracle: Netsuite V1, Oracle Relational

 

     Connectivity enhancements include the following:

     AWS

    • Support for file names longer than 250 characters with S3
    • Support for custom JDBC URL for Redshift
    • Support for ORC files with S3

    

     Snowflake

    • Custom Query metadata fetching without having to run the query

    

     Google

    • Custom Query Support for the Google Big Query V2 connector
    • Pushdown support for Google Big Query thru ODBC
    • Google Analytics
    • Google Big Query Mass Ingestion

     

     Azure

    • Preview of ADLS Gen2 connector: support create target, configurable escape character and text qualifier in R/W scenarios, create and rename directory, rename file, headerless files, support RBAC for all types of AAD authentication, append data to an existing file, support parameterization
    • Azure DW V3: support TARGET NAME OVERRIDE and TARGET SCHEMA NAME OVERRIDE in the writer, support SOURCE NAME OVERRIDE and SOURCE SCHEMA OVERRIDE with the reader, support custom query and multiple objects in CMD and MCT

 

     Microsoft Apps

    • MS Dynamics 365 Operations: delete and write support
    • Sharepoint online: support for agents running on Linux

 

     Analytics

    • Preview of CDM Folders connector: new connector, with the ability to write to ADLS Gen 2 in CDM format, and then access the data from Power BI as a dataflow
    • Tableau V2: upgrade connector to the latest Tableau SDK

 

     Databases & DW

    • MySQL Relational: array insert support
    • Greenplum: native reader and writer

 

     Netsuite V1

    • Address two-factor authentication

 

     Salesforce

    • SFDC Sales, Service connector: support latest SFDC API
    • Salesforce Marketing Cloud SFMC: insert/update operation for “Non-contact Linked Data Extensions”