The Spring 2020 release of Informatica Intelligent Cloud Services includes many changes. Please review the attached document for more details.
Cloud Data Integration
Continued investments to the current leader in integration for data warehousing to make it more scalable, flexible, and dynamic.
- Data warehousing support for transaction control transform and dynamic file names to enhance the control over the writing of target data.
- Highly requested capabilities of mid-stream data preview for a better debugging experience.
- Expansion of the user, task, and sessions variables for enhanced expression creation.
- Enhancements to parameterization capabilities to access parameter files from cloud stores like Amazon S3, Microsoft Azure Data Lake Storage Gen2 (ADLS Gen2), or Google Cloud Storage.
- Improvements to the Union transformation to support more than two input groups.
- Ability to specify the data flow run order to load targets in a mapping in a .
- Extending Change Data Capture (CDC) sourcing capabilities to include z/OS VSAM file capture.
- Databricks Delta connector as a source and target for mass ingestion tasks.
- Roll-out of Operational Insights for Cloud Data Integration and improved visualizations for new operational analytics.
Cloud Data Integration Elastic
- Support for hierarchical datatypes. While applications are collecting massive amounts of data, including IoT data, storage costs become a concern and we begin looking at columnar databases and binary formats such as JSON, Avro, Parquet, and ORC. To enable BI applications to use the binary data, developers can pass the data into an elastic mapping and convert it into a structured format.
- Serverless. A new modern cloud architecture that removes the need to manage servers, virtual machines (VMs), and containers. Developers can run mappings in response to events without weighing down administrators to maintain infrastructure or install a Secure Agent. Auto-scaling, high availability, and recovery are built in and enabled automatically. This feature is in preview.
- Auto-tuning. Traditional approaches to manually tune the Spark engine are an inefficient and time-consuming process too often characterized by trial and error. Attempts to tune are error-prone because one parameter generally impacts another. CLAIRE, Informatica’s AI engine, now automatically tunes mapping tasks based on heuristics including the size of the data, transformation logic, the size of the cluster, and the size of cluster nodes.
- Node bootstrapping. Cluster nodes can run initialization scripts to perform bootstrap actions that install additional software or customize cluster instances.
- Support for the Sequence Generator transformation.
- Auto-scaling enhancements for an elastic cluster through a custom scheduler.
- Support for high availability. You can configure an elastic cluster to become highly available so that jobs continue running smoothly without interruption.
- New platform capabilities: Continuous availability through upgrades, job recovery, high availability, and disaster recovery.
Taskflows for Cloud Data Integration
Continued investments of Cloud Data Integration support of ingestion use cases. now support:
- Mass ingestion tasks: Provides the means to apply any post or business logic before or after ingesting the data to/from a data lake.
- Inline file listener tasks: Makes it possible to wait for the arrival of a file before proceeding to further processing steps.
- Unique API name override: Customers can now override the unique API name assigned to . This makes it easier to put new versions in production without the need to update consumers.
Note: You can find a summary of features introduced in 2019 in the Taskflow folder of the Cloud Data Integration community and specifically in this article: Cloud Data Integration - Taskflows - 2019 Highlights
Cloud Data Quality
Cloud Data Quality continues its evolution of data quality services. New with the Spring 2020 release are:
- Deduplicate asset:
- Identity matching (single-source) functionality to compare records on a field-by-field basis and generate a numerical score that indicates the degree of similarity between the records.
- Optional consolidation of the records that the deduplication process identifies as duplicates. Consolidation evaluates each set of duplicate records and creates a preferred version of the records in the set according to criteria that the user defines.
- Parse asset: Parse discrete words or strings from an input field using regular expressions and/or dictionaries.
- Rule specification asset: Additional function expressions are available in rule statements: Convert to Date, Convert to Float, Convert to Integer, Is Spaces, Substring, and To Char.
- Data Profiling:
- Perform What-If scenarios by profiling outputs of Data Quality assets: rule specification, verifier, and cleanse.
- Extension of source support for Salesforce, Microsoft Common Data Model, and Google Big Query.
- Auto-Assignment of Data Quality assets to source columns of Microsoft Common Data Model.
- Ability to modify connection and source object from existing profiling tasks.
- Option to query records that have data quality issues into delimited file.
- Integration of Data Profiling with the Operational Insights service.
Cloud Integration Hub
- Ability to subscribe using API to partial published data to allow lower amounts of data to be consume per API subscription call.
- Subscription API does not have a size limit when using a private publication repository.
- Ability to create a synchronization task-based subscription from the Cloud Integration Hub subscription menu for better self-service.
- Ability to create a topic based on an existing connection schema.
- Explore functionality now supports folders and tags.
- Usability enhancements for topics with topic-related publications and subscription tables and visual alignment across all pages.
- Performance enhancements when using a private publication repository.
- Onboard Cloud Integration Hub to Azure POD.
Intelligent Structure Discovery (ISD)
- Ability to use the Structure Parser transformation in Data Integration with real time connectors.
- Ability to use the Structure Parser transformation structure parser in Data Integration in passthrough mode.
- Ability to set the Structurer Parse transformation in Data Integration for non-relational output (serialize to JSON, XML, Parquet, Avro & ORC).
- B2B Partners portal: Take partners community management to the next level of empowering business partners. Partners can use the portal to track their file exchanges with the organization and to send and receive files to and from the organization, leveraging a secure HTTPs protocol.
- SFTP server: The new SFTP service provides customers with the ability to manage and use SFTP servers to exchange files with partners.
- OAuth 2.0 support: The API Gateway and Informatica Intelligent Cloud Services platform are delivering a new authorization option for API access. Initially available to Cloud Application Service API consumers, OAuth 2.0 used in conjunction with Client ID and Secrets extends authorization choices that today include basic authentication, and JSON-Web Token (JWT)-based authentication.
- Personal Identifiable Information (PII) Privacy Policies have been extended. The PII policy not only reports on incoming and outgoing PII transfer, but now also provides the ability to block requests/response that contain sensitive PII data.
Cloud Application Integration
Spring 2020 April
This release includes the following updates:
- Support for an Operational Insights Dashboard for Application Integration depicting API, process, and connector execution metrics.
- Application Integration API endpoint authorization.
- Will now be able to make use of the API Gateway’s OAuth service. This extends support for OAuth 2.0-based authorization in addition to the current basic authentication and JSONWeb Token (JWT)based authentication capabilities.
- Will now be able to restrict incoming requests from the API Gateway. This ensures that monitoring and policy enforcement are applied consistently at the gateway.
The January 2020 release represents a major release of Application Integration. See the following link for information about the new features: Announcing the January 2020 release of Cloud Application Integration
- Making it easier to implement by creating process object schemas simply by importing WSDL, XSD, or Swagger interface documents from a file, a zip or a URL.
- Making it easier to debug:
- To help you debug processes, the Assignment step now gives you access to field values controlled by a process’s tracing level. To turn up tracing without redeploying a process, a new X-Debug HTTP header has been introduced.
- Tools such as Postman, SOAPUI, or are great but require you to leave Process Designer to initiate a process. You can now create and associate with a process one or more JSON or XML process inputs and run a process with one or all inputs. You can then use the new process instance debugging capabilities to better analyze and identify the root cause of errors.
- Making it easier to consume databases as fully-fledged REST-enabled APIs. Not only can you enable your database with OData v4 with a single click, you can now expose it as a fully-fledged REST API-enabled database. Just download the auto-generated Swagger interface and you’re good to go.
- Making it easier for developers to consume JSON and XML content-type responses and work with attachments and multipart responses
- Unique API name override for processes and
- Making it possible for operators to restart processes from the Application Integration Console to recover from unhandled errors communicating with the end system.
Cloud Mass Ingestion Service
Enhanced capabilities for data ingestion from a variety of sources, using a simple and unified user experience with extensive connectivity, to address mass ingestion use cases for Cloud data warehouses and Cloud data lakes.
Mass Ingestion Databases
Continuing to build usability and resilience into the service while adding new functionality. New features include:
- Schema drift support, which enables running database ingestion tasks to recognize when the schema of source tableswithin the task change (column changes only) and to dynamically process the changes through to the CDC-supported target.
- Asset import and export functionality for database ingestion tasks.
- GitHub source control for database ingestion tasks.
Mass Ingestion Streaming
- Continuing to enable ingestion from variety of streaming sources with real time monitoring and lifecycle management.
- New streaming connectivity & enhancements,new streaming sources and targets:
- New connectivity: Amazon Kinesis Streams source and Microsoft Azure Data Lake Storage Gen2 target.
- Connectivity enhancements: Flat file source.
- Test connection for Mass Ingestion Streaming connectors.
- New transformations during ingestion:
- Python transformation support
- Splitter transformation support (technical preview).
- Real-time monitoring and lifecycle management:
- Real-time refresh of Mass Ingestion Streaming job statistics.
- Stop and Resume support for Mass Ingestion Streaming tasks.
- Enterprise readiness:
- Sub-organization support to represent different business environments.
- Integration with GitHub for source control onMass Ingestion Streaming tasks
- Deployment of the Mass Ingestion Streaming service in Kubernetes for autoscaling and high availability.
Mass Ingestion Files
- Continuing to enable mass ingestion of files from variety of sources to cloud repositories with real time monitoring and different scheduling capabilities.
- New connectivity: Databricks Delta Lake as source & target.
- integration for Mass Ingestion Files tasks to support complex file ingestion flows that require orchestration of multiple tasks.
MDM - Reference 360
- Delta export:
- Retrieve values that have changed time period.
- Configure attribute-level validation rules to be executed when creating or editing code values.
- Receive validation errors on create and update of individual code values.
- Improved loading of larger data sets:
- Reduce the loading time for data sets and hierarchies by loading a subset of code values at a time.
- Export enhancements:
- Choose which attributes to export both from UI and API.
- Export data in JSON format with the REST API.
- Workflow email notifications:
- Receive email confirmations at each step of an approval workflow.
- Navigate to your task in Reference 360 through a link in the notification email.
Runtime Continuity (more details here)
Runtime continuity (zero downtime for runtime) to enable uninterrupted execution of scheduled jobs and processes including during Informatica Intelligent Cloud Services upgrade windows.
- Undo checkout capability for administrators on behalf of any user and any asset.
- Bulk unlink capability.
- Multi-project pull capability to enable pull across multiple projects with cross-project dependencies.
- Automatically deploy upstream dependent objects upon save to be consistent with the behavior of non-source-controlled Cloud Data Integration assets.
- Inline Git actions in dependency view
Export/import environment properties and schedules
- Ability to export/import Secure Agent configurations to automate agent provisioning with the option to either restore or tune agent configuration settings through export/import APIs. This enables users to tune runtime environment properties such as agent configurations.
- Ability to export/import schedule objects across environments through export/import APIs and in the Informatica Intelligent Cloud Services user interface.
Ability to reassign asset ownership to enable uninterrupted job execution when the asset owners are disabled in the organization.
- Informatica Intelligent Cloud Services availability on Japan POD (Azure).
- Informatica Intelligent Cloud Services availability on Google Cloud Platform (GCP)POD and Marketplace.
- Informatica Intelligent Cloud Services availability in Canada region.
- Global availability of Operational Insights across all Amazon Web Services (AWS) PODs.
- Generate alerts based on resource usage by individual agent services.
- Generate alerts on disk utilization of Secure Agents.
- Take preventive actions using custom scripts based on generated alerts.
- In-app notification of infrastructure alert messages.
Ecosystems and Connectivity
- Expanded coverage in terms of functional depth and breadth for cloud ecosystems:
- SQL Data Warehouse V3: Unconnected lookup, ADLS Gen2 as optional staging area for , improved exception handling, source/target parameterization overrides with parameter files, parameterization overrides (for schema, database, and table) in PRE SQL/POST SQL/SQL Override, and performance enhancements.
- ADLS Gen2:Source/target parameterization overrides with parameter files, Azure Gov Cloud, port, New data type (Date/Time) support for Parquet files, Parquet Decimal, Date/Time data type support, User authenticated proxy, and performance enhancements.
- Blob Storage:Source/target parameterization overrides with parameter files.
- CDM Folders (available forpreview only): Support for new CDM schema (v0.9) and character support.
- S3 V2: Hierarchical data types, ISD, multi-region, Parquet Decimal, Date/Time datatype support, KMS (other accounts), source/target parameterization.
- RedShift: Ecosystem pushdown optimization (S3 to RedShift), JDBC driver update, KMS (other accounts), multi-region, source/target parameterization.
- Snowflake DW:
- Database pushdown optimization enhancements.
- Unconnected lookups with pushdown optimization.
- Snowflake on Google Cloud Platform.
- IICS on Google Cloud Platform.
- : CDC support, lookup.
- Google Cloud Storage: Read from directory.
- Microsoft Business Apps:
- Common Data Model (CDM) folders: Schema update (0.9).
- Dynamics 365 Operations: Certificate, update retry.
- CRM: Client secrets.
- SAP BW Reader: Supports dates before the year 1753.
- SAP HANA (new private connector): Read from tables and modelling views (analytical, attribute, and calculation views).
- Sales and Service Cloud: API updates, DRS enhancements to support deleted records.
- Commerce Cloud: Cycle dependencies.
- Oracle Apps:
- Oracle HCM: Writer updates.
- NetSuite: API update (2019.2).
- Adobe: XDM connector enhancements.
- Support for new patterns and use cases:
- Cloud Data Integration Elastic:
- Amazon S3 V2: Hierarchical data type and ISD support.
- Azure SQL Data Warehouse V3: Elastic mappings.
- ADLS Gen2: Elastic mappings (available for preview only).
- JDBC V2: Scala v2, partitioning, AWS and Azure runtime support.
- Cloud Mass Ingestion:
- Databricks Delta
- ADLS Gen2: Direct load from ADLS Gen2 and performance improvements.
- Cloud Data Quality: CDM
- Kafka Pub/Sub Connector in Cloud Data Integration
- Cloud Data Integration Elastic:
- Improved connectivity across horizontal technology, systems, and applications:
- OAuth 2.0 JWT support.
- REST V2: Real-time transaction support, minor enhancements, Hadoop 3.1 for Cloudera, 4.1 for .
- Database and EDW:
- ODBC: Unconnected lookup.
- Oracle: Blob/ datatype.
- MySQL: Advanced runtime properties.
- SQL Server: Advanced runtime properties.
- Azure SQL DB: Bulk, .
- PostgreSQL: Schema name enhancements.
- MongoDB: BSON and JSON document support, partitioning, create target with schema-less support.
- Workday: Hierarchy parameterization support.
- ServiceNow: .
- Cloud Apps:
- : Child object
- : Proxy
- New Add-on Connectors page to access add-on connectors on Marketplace.