The Spring 2020 release of Informatica Intelligent Cloud Services includes many changes. Please review the attached document for more details. We will provide individual pod upgrade dates within another week.
Cloud Data Integration
Continued investments to the current leader in integration for data warehousing to make it more scalable, flexible and dynamic.
- Data warehousing support for transaction control transform and dynamic file names to enhance the control over the writing of target data.
- Highly requested capabilities of mid-stream data preview for a better debugging experience.
- Roll-out of Operational Insights for Cloud Data Integration and improved visualizations for new operational analytics.
- Expansion of the user, task, and sessions variables for enhanced expression creation.
- Extending Change Data Capture (CDC) sourcing capabilities to include z/OS VSAM file capture.
- Databricks Delta connector as a source and target for mass ingestion tasks.
Cloud Data Integration Elastic
- Support for hierarchical datatypes. While applications are collecting massive amounts of data, including IoT data, storage costs become a concern and we begin looking at columnar databases and binary formats such as JSON, Avro, Parquet, and ORC. To enable BI applications to use the binary data, developers can pass the data into an elastic mapping and convert it into a structured format.
- Serverless. A new modern cloud architecture that removes to need to manage servers, virtual machines (VMs), and containers. Developers can run mappings in response to events without weighing down administrators to maintain infrastructure. Auto-scaling, high availability, and recovery are built in and enabled automatically.
- Auto-tuning. Traditional approaches to manually tune the Spark engine are an inefficient and time-consuming process too often characterized by trial and error. Attempts to tune are error-prone because one parameter generally impacts another. CLAIRE, Informatica’s AI engine, now automatically tunes mapping tasks based on a number of heuristics including the size of the data, transformation logic, the size of the cluster, and the size of cluster nodes.
- Node bootstrapping. Cluster nodes can run initialization scripts to perform bootstrap actions that install additional software or customize cluster instances.
- Support for the Sequence Generator transformation.
- Auto-scaling enhancements for an elastic cluster through a custom scheduler.
- Support for high availability. You can configure an elastic cluster to become highly available so that jobs continue running smoothly during hiccups.
- New platform capabilities: continuous availability through upgrades, job recovery, high availability, and disaster recovery.
Taskflows for Cloud Data Integration
Continued investments of Cloud Data Integration taskflow support of ingestion use cases. Taskflows now support:
- Mass ingestion tasks: Provides the means to pre-process files before they are mass ingested into a data lake.
- Inline file listener tasks: Makes it possible to wait for the arrival of a file before proceeding to further processing steps.
- Unique API name override: Customers can now override the unique API name assigned to taskflows. This makes it easier to put new versions in production without the need to update consumers.
Note: You can find a summary of taskflow features introduced in 2019 in the Taskflows folder of the Cloud Data Integration community and specifically in this article: Cloud Data Integration - Taskflows - 2019 Highlights
Cloud Data Quality
Cloud Data Quality continues its evolution of new data quality services. New with the Spring 2020 release are:
- Deduplicate asset: Identity matching (single-source) functionality to compare records on a field-by-field basis and generate a numerical score that indicates the degree of similarity between the records.
- Parse asset: Parse discrete words or strings from an input field using regular expressions and/or dictionaries.
- Rule specification asset: Additional function expressions are available in rule statements: Convert to Date, Convert to Float, Convert to Integer, Is Spaces, Substring, and To Char.
- Profiling: Integration with the Operational Insights service.
Cloud Integration Hub
- Ability to subscribe using API to partial published data to allow lower amounts of data to be consume per API subscription call.
- Subscription API does not have a size limit when using a private publication repository.
- Ability to create a synchronization task-based subscription from the Cloud Integration Hub subscription menu for better self-service.
- Ability to create a topic based on an existing connection schema.
- Explore functionality now supports folders and tags.
- Usability enhancements for topics with topic-related publications and subscription tables and visual alignment across all pages.
- Performance enhancements when using a private publication repository.
- Onboard Cloud Integration Hub to Azure POD.
Intelligent Structure Discovery (ISD)
- Integration with CDI Elastic source transformation to support hierarchical datatypes and data drift handling.
- Ability to use the Structure Parser transformation in Data Integration with real time connectors.
- Ability to use the Structure Parser transformation structure parser in Data Integration in passthrough mode.
- Ability to set the Structure Parser transformation in Data Integration for non-relational output (serialize to JSON, Parquet, Avro & ORC).
- B2B Partners portal: Take partners community management to the next level of empowering business partners. Partners can use the portal to track their file exchanges with the organization and to send and receive files to and from the organization, leveraging a secure HTTPs protocol.
- SFTP server: The new SFTP service provides customers with the ability to manage and use SFTP servers to exchange files with partners.
- OAuth 2.0 support – The API Gateway and Informatica Intelligent Cloud Services platform are delivering a new authorization option for API access. Initially available to Cloud Application Service API consumers, OAuth 2.0 used in conjunction with Client ID and Secrets extends authorization choices that today include basic authentication, and JSON-Web Token (JWT)-based authentication.
- Personal Identifiable Information (PII) Privacy Policies have been extended. The PII policy not only reports on incoming and outgoing PII transfer, but now also provides the ability to block requests/response that contain sensitive PII data.
Cloud Application Integration
Spring 2020 April
This release includes the following updates:
- Support for an Operational Insights Dashboard for Application Integration depicting API, process, and connector execution metrics.
- Application Integration API endpoint authorization.
- Will now be able to make use of the API Gateway’s OAuth service. This extends support for OAuth 2.0-based authorization in addition to the current basic authentication and JSON Web Token (JWT) based authentication capabilities.
- Will now be able to restrict incoming requests from the API Gateway. This ensures that monitoring and policy enforcement are applied consistently at the gateway.
The January 2020 release represents a major release of Application Integration. See the following link for information about the new features: Announcing the January 2020 Release of Cloud Application Integration
- Making it easier to implement by creating process object schema simply by importing WSDL, XSD, or Swagger interface documents from a file, a zip or a URL
- Making it easier to debug:
- To help you debug processes, the Assignment step now gives you access to field values controlled by a process’s tracing level. To turn up tracing without redeploying a process, a new X-Debug HTTP header has been introduced.
- Tools like Postman, SOAPUI, or RESTclient are great but require you to leave Process Designer to initiate a process. You can now create and associate with a process one or more JSON or XML process inputs and run a process with one or all inputs. You can then use the new process instance debugging capabilities to better analyze and identify the root cause of errors.
- Making it easier to consume databases as fully-fledged REST-enabled APIs. Not only can you enable your database with OData v4 with a single click, you can now expose it as a fully-fledged REST API-enabled database. Just download the auto-generated Swagger interface and you’re good to go.
- Making it easier for developers to consume JSON and XML content-type responses and work with attachments and multipart responses
- Unique API name override for processes and taskflows
- Making it possible for operators to restart processes from the Application Integration Console to recover from unhandled errors communicating with end system.
Cloud Mass Ingestion Service
Enhanced capabilities for data ingestion from a variety of sources, using a simple and unified user experience with extensive connectivity, to address mass ingestion use cases for Cloud data warehouses and Cloud data lakes.
Mass Ingestion Databases
Continuing to build usability and resilience into the service while adding new functionality. New features include:
- Schema drift support, which enables database ingestion tasks that are running to recognize when the schema of source tables within a task change and to dynamically process the changes through to the CDC-supported target.
- Alerts to identify when a schema change occurs
- Asset import and export functionality for database ingestion tasks
- GitHub source control for database ingestion tasks
- Ability to drill down on a task to view object dependencies.
Mass Ingestion Streaming
- Continuing to enable ingestion from variety of streaming sources with real time monitoring and lifecycle management.
- New streaming connectivity and enhancements, new streaming sources and targets:
- New connectivity: Amazon Kinesis Streams source and Microsoft Azure Data Lake Storage Gen2 target.
- Connectivity enhancements: Flat file source.
- Test connection for Mass Ingestion Streaming connectors.
- New transformations during ingestion:
- Python transformation support
- Splitter transformation support (technical preview).
- Real-time monitoring and lifecycle management:
- Real-time refresh of Mass Ingestion Streaming job statistics.
- Stop and Resume support for Mass Ingestion Streaming tasks.
- Enterprise readiness:
- Sub-organization support to represent different business environments.
- Integration with GitHub for source control on Mass Ingestion Streaming tasks
- Deployment of the Mass Ingestion Streaming service in Kubernetes for autoscaling and high availability.
Runtime continuity (more details here)
Runtime continuity (zero downtime for runtime) to enable uninterrupted execution of scheduled jobs and processes at all times including during Informatica Intelligent Cloud Services upgrade windows.
- Undo checkout capability for administrators on behalf of any user and any asset.
- Bulk unlink capability.
- Multi-project pull capability to enable pull across multiple projects with cross-project dependencies.
- Automatically deploy upstream dependent objects upon save to be consistent with the behavior of non-source-controlled Cloud Data Integration assets.
- Inline Git actions in dependency view.
- Ability to export/import agent configurations to automate agent provisioning with the option to either restore or tune agent configuration settings.
- Ability to export/import schedule objects across environments.
- Informatica Intelligent Cloud Services availability on Japan POD (Azure).
- Informatica Intelligent Cloud Services availability on GCP POD and Marketplace.
- Global availability of Operational Insights across all AWS PODs.
- Generate alerts based on resource usage by individual agent services.
- Generate alerts on disk utilization of Secure Agents.
- Take preventive actions using custom scripts based on generated alerts.
- In-app notification of infrastructure alert messages.
Ecosystems and Connectivity
- Expanded coverage in terms of functional depth and breadth for cloud ecosystems
- SQL DW: Polybase, parameterization overrides (for schema, database, and table), performance enhancements, and unconnected lookup
- ADLS Gen2: Performance enhancements, WLM, source/target parameterization, Azure Gov Cloud, FileName port, Parquet Decimal, Date/Time data type support, User authenticated proxy
- S3: Hierarchical data types, ISD, multi-region, Parquet Decimal, Date/Time data type support, KMS (other accounts), source/ target parameterization
- RedShift: Ecosystem PDO (S3 to RedShift), JDBC driver update, KMS (other accounts), multi-region, source/ target parameterization
- Snowflake DW
- Database PDO enhancements
- Unconnected lookups on PDO
- Snowflake on GCP
- gzip compression
- IICS on Google Cloud Platform
- BigQuery: CDC support, lookup
- Google Cloud Storage: read from directory
- Microsoft Business Apps:
- Common Data Model: Schema update (0.9)
- Dynamics 365 Operations: Certificate, update retry
- CRM: Client secrets
- SAP BW Reader: supports dates before the year 1753.
- SAP HANA (New private connector): Read from tables and modelling views (analytical, attribute, and calculation views.
- Sales and Service Cloud: API updates; DRS enhancements to support deleted records;
- Commerce Cloud: Cycle dependencies.
- Oracle Apps
- Oracle HCM: Writer updates, BI Publisher
- NetSuite: API update (2019.2)
- Adobe: XDM connector enhancements
- Support for new patterns and use cases
- S3: Hierarchical data type and ISD support
- Azure SQL DW and ADLS Gen2: Elastic mappings
- JDBC: Scala v2, partitioning, AWS and Azure runtime support
- MI files:
- Databricks Delta
- Snowflake create target
- ADLS Gen2: Performance improvement
- Cloud DQ: CDM
- Kafka Pub/Sub Connector in CDI
- Improved connectivity across horizontal technology, systems, and applications
- OAuth 2.0 JWT Support
- REST V2: real-time transaction support, minor enhancements Hadoop 3.1 for Cloudera, 4.1 for HDInsights;
- Database and EDW
- ODBC: Unconnected lookup;
- Oracle: Blob/Clob datatype, RAC failover, Bulk, Upsert
- MySQL: advanced runtime properties;
- SQL Server: advanced runtime properties;
- Azure SQL DB: Bulk, Upsert
- PostgreSQL: schema name enhancements;
- Teradata: SQL Transform, Lookup transform, PDO;
- MongoDB: BSon and Json document support, partitioning, CreateTarget with schema-less support
- Workday: Hierarchy parameterization support
- ServiceNow: Upsert, OAuth 2.0
- Cloud Apps
- Cvent: child object
- Coupa: Proxy
- Zuora: Pagination
- Zuora Aqua: OAuth
- New Add-on Connectors page to access add-on connectors on Marketplace.
IICS_Spring2020April_WhatsNew_en.pdf 969.1 KB