The Informatica Intelligent Cloud Services (IICS) Winter 2019 release offers several new capabilities that address key data challenges that businesses are facing today. Highlights are listed below.
- Data discovery in Cloud Data Integration with Enterprise Data Catalog (EDC) integration - Customers can now search and discover enterprise-wide metadata from within Data Integration, import connection & object metadata, and use that information to more easily create new or enrich existing mappings and tasks by connecting with an existing EDC installation.
- “Smart match” recommendations for field mappings increases the frequency of field matches in mappings and tasks. Expanding on the existing automatch, smart match looks for common patterns in field names (prefixes, suffixes, abbreviations, etc.) based on six additional matchers and fuzzy match techniques for recommending field mappings.
- Taskflows can be invoked via APIs for externalized scheduling and execution. With this enhancement, customers now can invoke taskflows on-demand via an API call and provide input parameters for the tasks it orchestrates, allowing customers to fully leverage Data Integration’s parameterization capabilities. Please refer to the Taskflow as a Service FAQ.
- Taskflows have also been enhanced to allow them to embed other taskflows to promote reuse.
- Change data capture has been expanded to include additional sources for DB2 on Linux, Unix, Windows, and iSeries (also known as AS400, i5/OS) platforms, which further enables near real-time changed data propagation capabilities.
- Mass ingestion is extending connector support, adding Google Storage & Google Big Query as targets and HDFS as both a source and target. Additional enhancements expose CRUD-focused APIs.
API and Application Integration
- Support for Kafka Messaging – Messaging is at the core of many publish-subscribe (Pub/Sub) based applications as a means of decoupling producers and consumers of data. The addition of Kafka for application integration significantly increases current message-based Pub/Sub interactions between data and applications that today are fulfilled using JMS, AMQP, Amazon SNS/SQS, and Azure Service Bus based “topics.” The ability to bridge these message-based events with the Cloud Integration Hub Pub/Sub style of data propagation provides additional integration pattern options making Informatica unique in the flexibility and capabilities it provides for its customers.
- JSON Web Token (JWT) based authentication – The API and Application Integration services now support JSON Web Token (JWT) based authentication, an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between API consumers and REST web services. This provides IICS users that use API Manager with another and more secure means of API authentication.
- API grouping – To better manage the use of JWT tokens and associate their use to multiple API endpoints, a new API “application” grouping capability is being introduced in API Manager. This capability will provide API consumers with fewer tokens to deal with, and API producers will now more easily manage or revoke a consumer’s access to multiple APIs.
- Japanese language support for the API and Application Integration services – In addition to Japanese language support for the Data Integration service, Japanese customers now have access to the API and Application Integration services user interface and documentation in Japanese.
- REST and SOAP Service API-based “service connectors” – distributed via GitHub.
Today 55% of Application Integration customer’s connectivity needs are met using service connectors. A service connector allows customers to define REST (XML/JSON), JSON/RPC, or SOAP service integration using a simple web form, with no coding required. The Application Integration service takes care of the rest. If the service offers a WSDL or Swagger interface document, the service connector can be automatically generated by importing the interface document. By creating service connectors, customers can import and configure pre-built business and data service definitions as reusable assets that they can share and/or move from one environment to another.
This capability, unique to Informatica, provides customers with unparalleled value. Service connectors avoid lock-in or an inability to make updates as you need them to take advantage of new capabilities or built-in extensibility that an API platform can offer.
To provide this flexibility to customers and encourage community contribution by customers, partners, and other practitioners, Informatica is establishing a GitHub repository where it will publish the service connectors it has created and which it will share with its customers and partners. Customers and partners are free to use these definitions without restriction, including the rights to use, copy, modify, merge, publish, and distribute these under an MIT license. Informatica will also encourage contributions back to the community. Our goal is simple: drive innovation and reduce perceived barriers to adoption.
- Informatica has improved the search experience for Hub events and support for the CLOB data type on topics.
- ICS B2B Gateway customers will be migrated to the IICS platform as part of the release upgrade and will benefit from all IICS platform capabilities.
Intelligent Structure Discovery
- Intelligent Structure Discovery expanded its parsing capabilities to handle ORC format files and Excel multiple sheet files. The user can now design the structure model based on multiple sheet structures and then use the model at run time to parse Excel files in their entirety.
- With R31, a Structure Parser transformation can be positioned mid-stream to enable a more flexible mapping usage and chaining. In addition, the Intelligent Structure Model detected datatypes are now propagated to the Structure Parser output ports.
- The ISD design time user interface is enhanced with a "find" functionality which allows the user to search for a specific string in the discovered tree fields and get a list of results showing the path and visually correlated with the model representation. The user can also perform actions on multiple elements chosen from the result list such as include, exclude, replace, and even change of element type. The ability to perform actions on multiple elements significantly improves the usability and productivity.
- A new vertical folder view mode will be available in R31 for handling complex hierarchy files.
- Common Explore productivity enhancements – Improved copy functionality with overwrite & rename conflict resolution options to copy assets within and across folders. “Created by” and “last updated by” attributes as columns for all asset types in the common Explore page.
- Export/import capability for sub-organizations which enables asset migration across environments that use an organization hierarchy. More control and flexibility with enable/disable checksum validation options during export and import.
- Improved export/import error logging along with the ability to access and download export/import logs through the UI and the API.
- API to search, list, and filter assets in projects and folders using a variety of conditions such as timestamp, location, “last updated by,” and tags. This API can also be leveraged along with export APIs to export objects.
- Improvements to the RunAJob utility – Support for projects and folders to invoke tasks by task name.
- Usability improvements – Ability to copy table cell data in the common Explore page, Monitor service, and Administrator service for use in other user interfaces like search boxes and filter conditions for easier task completion.
- Search capability for connections and job monitoring to quickly and easily find needed information.
- Ability to enable and disable a service for agents in a Secure Agent group to effectively deploy workloads and efficiently utilize computing resources.
- Secure Agent registration using tokens (instead of passwords) for increased security and enabling SAML single sign-on.
- Operational Insights extends support to the on-premises Data Quality product, in addition to BDM and PowerCenter, with capabilities such as domain health, job run analytics, resource utilization, and alerts.
- Click-through analytics journey from cross-domain to the individual job level and enhancements to job run analytics for on-premises products (PowerCenter, BDM). Plus, enhancements to job run analytics to report on data loaded and data processed.
- Power Center & BDM infrastructure e-mail alert enhancements such as Secure Agent unavailability and Operational Insights data collection failures.
New AWS Quickstart for Cloud Analytics Modernization, an end-to-end solution for self-service cloud analytics with Informatica (IICS, Enterprise Data Catalog), Tableau Server, and AWS Services.
Several new connectors and enhancements to existing connectors across ecosystems as listed below. New connectors introduced are highlighted in bold:
- Azure: ADLS Gen 2 Preview, Azure DW V3, Azure Data Lake Store V3, Azure Blob V3
- Google: Google Cloud Storage V2, Google Analytics, Google Big Query V2, Google Big Query
- Amazon: Amazon S3 V2, Amazon Aurora, Amazon Redshift V2
- Salesforce: Salesforce Marketing Cloud (SFMC), SFDC (Sales and Service)
- SAP: SAP Connector, SAP HANA Cloud Platform (DB)
- Adobe: Adobe Cloud Platform
- Analytics: CDM Folders connector preview, Tableau V3, Tableau V2
- Databases: MySQL Relational, Hive, Greenplum, DashDB, Snowflake
- Tech: REST V2, WSconsumer, Complex File Processor
- Microsoft Apps: Microsoft SharePoint Online
- Oracle: Oracle Netsuite V1, Oracle Relational
Summary of some of the connectivity enhancements are as follows:
- Supporting file names longer than 250 characters with S3
- Support for custom JDBC URL for Redshift
- Support for ORC files with S3
- Custom Query metadata fetching without having to run the query
- Custom Query Support for Google Big Query V2 connector
- Pushdown support for Google Big Query thru ODBC
- Google Analytics - Enhancement to fetch fields based on multiple Views IDs from GA
- Google Big Query Mass Ingestion - Direct load Cloud Storage->Big Query
- Preview of ADLS Gen2 connector: support create target, configurable escape character and text qualifier in R/W scenarios, create and rename directory, rename file, header-less files, support RBAC for all types of AAD Authentication, append data to an existing file, support parameterization
- Azure DW V3: support TARGET NAME OVERRIDE and TARGET SCHEMA NAME OVERRIDE in the writer, support SOURCE NAME OVERRIDE and SOURCE SCHEMA OVERRIDE with the reader, support custom query and multiple objects in CMD and MCT
- Microsoft SharePoint Online: support for agents running on Linux
- Preview of CDM Folders connector: new connector, with the ability to write to ADLS Gen 2 in CDM format, and then access the data from Power BI as a dataflow
- Tableau V2: upgrade connector to the latest Tableau SDK
Databases & DW
- MySQL Relational: array insert support
- Greenplum: native reader and writer
NetSuite V1: address 2-factor authentication
- SFDC Sales, Service connector: support latest SFDC API
- Salesforce Marketing Cloud SFMC: insert/update operation for “Non Contact Linked Data Extensions”