Notification of an upcoming change of behavior in Amazon Redshift V2 Connector

Product: Informatica Intelligent Cloud Services

Connector: Amazon Redshift V2

Description of the change

With the upcoming July 2021 release, when you migrate a Redshift V2 connection related mapping you can run the imported mapping using the target org’s connection, without any changes to the mapping attributes.

For example, you develop a mapping in the IICS development organization and you then migrate the mapping to the IICS production organization. When the connection endpoint or object path in the production organization differs, you can update the Redshift V2 connection to reflect the production endpoint. You do not have to modify the mapping again. Previously, you had to manually update the mapping and create the object in the production organization.

Impact of this change

After July Major release, an Amazon Redshift V2 mapping will fail when the Target transformation includes all of the following configurations: 

• The target operation selected is update, upsert, or delete.
• The column in the Update Columns field is not selected.
• Target table name specified as an override in the target advanced properties does not have a primary key.

The following image shows the configurations in the Target transformation that you need to consider:

Error Scenario

If the target table specified in the Redshift V2 mapping at the object level has a primary key, but the table name specified as an override in the target advanced properties does not have a primary key defined, Data Integration considers the table name specified in the advanced properties. The mapping fails with the following error message:

[ERROR] <Target_table_name> No key is selected for Operation Type

Solution

When the mapping includes an update, upsert, or delete operation in the Target transformation, perform one of the following tasks before you move to the July 2021 release and migrate mappings from one environment to another:

• In the Update Column field, select the column that you want to use as the primary key.
• Ensure that a primary key is defined in the Redshift target table if you override Target Table Name in the target advanced properties.

As a result of the recent separation of the United Kingdom from the European Union, Informatica has made the decision to relocate our disaster recovery site for our EMEA IICS cloud services. You are receiving this notice as an administrator of an Informatica service located in our primary EMEA site, Dublin Ireland (“eu-west-1"). We will move the Disaster Recovery site to Paris, France (“eu-west-3") on January 30, 2021 to help our customers keep our European Union (EU) instances in the EU. All Informatica services currently running in the primary EMEA site (“eu-west-1") will remain unchanged.

As a result of this change, we request that you verify and if necessary, update the allowed domains & IP address ranges in your firewalls (as you had previously for our existing site) as documented in the following KB articles.

• Customers using Cloud Data Integration (CDI): - 628338
• Customer using API Manager (APIM): - 94569
• Customers using Cloud Integration Hub (CIH): - 570388
• Customers using Cloud Application Integration (CAI): - 535281

We do not expect any downtime, impact or risk from this change. The disaster recovery IP addresses are for disaster recovery situations only and will be invoked if a DR situation were to occur.

With the coming Informatica Fall '20 release, File Mass Ingestion task is moving from Data Integration service to the unified Mass Ingestion service, introducing a common and unified user experience to all ingestion types: Steaming, Database and File.

Please read the attached short deck to get some early insight into the coming change and learn what’s new in the coming release for File Mass Ingestion.

For any questions you can contact the product management team:

Belur, Vishwanath  (vbelur@informatica.com)

Afriat, Etty  (eafriat@Informatica.com )

The Spring 2020 release of Informatica Intelligent Cloud Services includes many changes. Please review the attached document for more details. We will provide individual pod upgrade dates within another week.

·

Cloud Data Integration

Continued investments to the current leader in integration for data warehousing to make it more scalable, flexible, and dynamic.

• Data warehousing support for transaction control transform and dynamic file names to enhance the control over the writing of target data.
• Highly requested capabilities of mid-stream data preview for a better debugging experience.
• Expansion of the user, task, and sessions variables for enhanced expression creation.

• Enhancements to parameterization capabilities to access parameter files from cloud stores like Amazon S3, Microsoft Azure Data Lake Storage Gen2 (ADLS Gen2), or Google Cloud Storage.
• Improvements to the Union transformation to support more than two input groups.
• Ability to specify the data flow run order to load targets in a mapping in a particular order.
• Extending Change Data Capture (CDC) sourcing capabilities to include z/OS VSAM file capture.
• Databricks Delta connector as a source and target for mass ingestion tasks.

• Roll-out of Operational Insights for Cloud Data Integration and improved visualizations for new operational analytics.

Cloud Data Integration Elastic

• Support for hierarchical datatypes. While applications are collecting massive amounts of data, including IoT data, storage costs become a concern and we begin looking at columnar databases and binary formats such as JSON, Avro, Parquet, and ORC. To enable BI applications to use the binary data, developers can pass the data into an elastic mapping and convert it into a structured format.
• Serverless. A new modern cloud architecture that removes the need to manage servers, virtual machines (VMs), and containers. Developers can run mappings in response to events without weighing down administrators to maintain infrastructure or install a Secure Agent. Auto-scaling,high availability, and recovery arebuilt in and enabled automatically. This feature is in preview.
• Auto-tuning. Traditional approaches to manually tune the Spark engineare an inefficient and time-consuming process too often characterized by trial and error. Attempts to tune areerror-prone because one parameter generally impacts another. CLAIRE, Informatica’s AI engine, now automatically tunes mapping tasks based on a number of heuristics including the size of the data, transformation logic, the size of the cluster, and the size ofcluster nodes.
• Node bootstrapping. Cluster nodes can run initialization scripts to perform bootstrap actions that install additional software or customize cluster instances.
• Support for the Sequence Generator transformation.
• Auto-scaling enhancements for an elastic cluster through a custom scheduler.
• Support for high availability. You can configure an elastic cluster to become highly available so that jobs continue running smoothly without interruption.
• New platform capabilities: Continuous availability through upgrades, job recovery, high availability, and disaster recovery.

Taskflows for Cloud Data Integration

Continued investments of Cloud Data Integration taskflow support of ingestion use cases. Taskflows now support:

• Mass ingestion tasks: Provides the means to apply any post or pre process business logic before or after ingesting the data to/from a data lake.
• Inline file listener tasks: Makes it possible to wait for the arrival of a file before proceeding to further processing steps.
• Unique API name override: Customers can now override the unique API name assigned to taskflows. This makes it easier to put new versions in production without the need to update consumers.

Note: You can find a summary of taskflow features introduced in 2019 in the Taskflow folder of the Cloud Data Integration community and specifically in this article: Cloud Data Integration - Taskflows - 2019 Highlights

Cloud Data Quality

Cloud Data Quality continues its evolution of data quality services. New with the Spring 2020 release are:

• Deduplicate asset:
  • Identity matching (single-source) functionality to compare records on a field-by-field basis and generate a numerical score that indicates the degree of similarity between the records.
  • Optional consolidation of the records that the deduplication process identifies as duplicates. Consolidation evaluates each set of duplicate records and creates a preferred version of the records in the set according to criteria that the user defines.
• Parse asset: Parse discrete words or strings from an input field using regular expressions and/or dictionaries.
• Rule specification asset: Additional function expressions are available in rule statements: Convert to Date, Convert to Float, Convert to Integer, Is Spaces, Substring, and To Char.
• Data Profiling:
  • Perform What-If scenarios by profiling outputs of Data Quality assets: rule specification, verifier, and cleanse.
  • Extension of source support for Salesforce, Microsoft Common Data Model, and Google Big Query.
  • Auto-Assignment of Data Quality assets to source columns of Microsoft Common Data Model.
  • Ability to modify connection and source object from existing profiling tasks.
  • Option to query records that has data quality issues into delimited file.
  • Integration of Data Profiling with the Operational Insights service.

Cloud Integration Hub

• Ability to subscribe using API to partial published data to allow lower amounts of data to be consume per API subscription call.
• Subscription API does not have a size limit when using a private publication repository.
• Ability to create a synchronization task-based subscription from the Cloud Integration Hub subscription menu for better self-service.
• Ability to create a topic based on an existing connection schema.
• Explore functionality now supports folders and tags.
• Usability enhancements for topics with topic-related publications and subscription tables and visual alignment across all pages.
• Performance enhancements when using a private publication repository.
• Onboard Cloud Integration Hub to Azure POD.

Intelligent Structure Discovery (ISD)

• Ability to use the Structure Parser transformation in Data Integration with real time connectors.
• Ability to use the Structure Parser transformation structure parser in Data Integration in passthrough mode.
• Ability to set the Structurer Parse transformation in Data Integration for non-relational output (serialize to JSON, XML, Parquet, Avro & ORC).

B2B Gateway

• B2B Partners portal: Take partners community management to the next level of empowering business partners. Partners can use the portal to track their file exchanges with the organization and to send and receive files to and from the organization, leveraging a secure HTTPs protocol.
• SFTP server: The new SFTP service provides customers with the ability to manage and use SFTP servers to exchange files with partners.

API Gateway

• OAuth 2.0 support: The API Gateway and Informatica Intelligent Cloud Services platform are delivering a new authorization option for API access. Initially available to Cloud Application Service API consumers, OAuth 2.0 used in conjunction with Client ID and Secrets extends authorization choices that today include basic authentication, and JSON-Web Token (JWT)-based authentication.
• Personal Identifiable Information (PII) Privacy Policies have been extended. The PII policy not only reports on incoming and outgoing PII transfer, but now also provides the ability to block requests/response that contain sensitive PII data.

Cloud Application Integration

Spring 2020 April

This release includes the following updates:

• Support for an Operational Insights Dashboard for Application Integration depicting API, process, and connector execution metrics.
• Application Integration API endpoint authorization.
• Will now be able to make use of the API Gateway’s OAuth service. This extends support for OAuth 2.0-based authorization in addition to the current basic authentication and JSON Web Token (JWT) based authentication capabilities.
• Will now be able to restrict incoming requests from the API Gateway. This ensures that monitoring and policy enforcement are applied consistently at the gateway.

January 2020

The January 2020 release represents a major release of Application Integration. See the following link for information about the new features: Announcing the January 2020 release of Cloud Application Integration

Summarizing:

• Making it easier to implement by creating process object schemas simply by importing WSDL, XSD, or Swagger interface documents from a file, a zip, or a URL.
• Making it easier to debug:
  • To help you debug processes, the Assignment step now gives you access to field values controlled by a process’s tracing level. To turn up tracing without redeploying a process, a new X-Debug HTTP header has been introduced.
  • Tools such as Postman, SOAPUI, or RESTclient are great but require you to leave Process Designer to initiate a process. You can now create and associate with a process one or more JSON or XML process inputs and run a process with one or all inputs. You can then use the new process instance debugging capabilities to better analyze and identify the root cause of errors.
• Making it easier to consume databases as fully-fledged REST-enabled APIs. Not only can you enable your database with OData v4 with a single click, you can now expose it as a fully-fledged REST API-enabled database. Just download the auto-generated Swagger interface and you’re good to go.
• Making it easier for developers to consume JSON and XML content-type responses and work with attachments and multipart responses.
• Unique API name override for processes and taskflows.
• Making it possible for operators to restart processes from the Application Integration Console to recover from unhandled errors communicating with the end system.

Cloud Mass Ingestion Service

Enhanced capabilities for data ingestion from a variety of sources, using a simple and unified user experience with extensive connectivity, to address mass ingestion use cases for Cloud data warehouses and Cloud data lakes.

Mass Ingestion Databases

Continuing to build usability and resilience into the service while adding new functionality. New features include:

• Schema drift support, which enables running database ingestion tasks to recognize when the schema of source tableswithin the task change (column changes only) and to dynamically process the changes through to the CDC-supported target.
• Asset import and export functionality for database ingestion tasks.
• GitHub source control for database ingestion tasks.

Mass Ingestion Streaming

• Continuing to enable ingestion from variety of streaming sources with real time monitoring and lifecycle management.
• New streaming connectivity and enhancements, new streaming sources and targets:
  • New connectivity: Amazon Kinesis Streams source and Microsoft Azure Data Lake Storage Gen2 target.
  • Connectivity enhancements: Flat file source.
  • Test connection for Mass Ingestion Streaming connectors.
• New transformations during ingestion:
  • Python transformation support
  • Splitter transformation support (technical preview).
• Real-time monitoring and lifecycle management:
  • Real-time refresh of Mass Ingestion Streaming job statistics.
  • Stop and Resume support for Mass Ingestion Streaming tasks.
• Enterprise readiness:
  • Sub-organization support to represent different business environments.
  • Integration with GitHub for source control on Mass Ingestion Streaming tasks.
  • Deployment of the Mass Ingestion Streaming service in Kubernetes for autoscaling and high availability.

Mass Ingestion Files

• Continuing to enable mass ingestion of files from variety of sources to cloud repositories with real time monitoring and different scheduling capabilities.
• New connectivity: Databricks Delta Lake as source and target.
• Taskflow integration for Mass Ingestion Files tasks to support complex file ingestion flows that require orchestration of multiple tasks.

MDM - Reference 360

• Delta export:
  • Retrieve values that have changed in a given time period.
• Validation:
  • Configure attribute-level validation rules to be executed when creating or editing code values.
  • Receive validation errors on create and update of individual code values.
• Improved loading of larger data sets:
  • Reduce the loading time for data sets and hierarchies by loading a subset of code values at a time.
• Export enhancements:
  • Choose which attributes to export both from UI and API.
  • Export data in JSON format with the REST API.
• Workflow email notifications:
  • Receive email confirmations at each step of an approval workflow.
  • Navigate to your task in Reference 360 through a link in the notification email.

IICS Platform

Runtime Continuity (more details here)

Runtime continuity (zero downtime for runtime) to enable uninterrupted execution of scheduled jobs and processes at all times including during Informatica Intelligent Cloud Services upgrade windows.

GitHub Integration

• Undo checkout capability for administrators on behalf of any user and any asset.
• Bulk unlink capability.
• Multi-project pull capability to enable pull across multiple projects with cross-project dependencies.
• Automatically deploy upstream dependent objects upon save to be consistent with the behavior of non-source-controlled Cloud Data Integration assets.
• Inline Git actions in dependency view.

Export/import Environment Properties and Schedules

• Ability to export/import Secure Agent configurations to automate agent provisioning with the option to either restore or tune agent configuration settings through export/import APIs. This enables users to tune runtime environment properties such as agent configurations.
• Ability to export/import schedule objects across environments through export/import APIs and in the Informatica Intelligent Cloud Services user interface.

Asset Management

• Ability to reassign asset ownership to enable uninterrupted job execution when the asset owners are disabled in the organization.

New PODs

• Informatica Intelligent Cloud Servicesavailability on Japan POD (Azure).
• Informatica Intelligent Cloud Services availability on Google Cloud Platform (GCP)POD and Marketplace.
• Informatica Intelligent Cloud Services availability in Canada region.

Operational Insights

• Global availability of Operational Insights across all Amazon Web Services (AWS) PODs.
• Generate alerts based on resource usage by individual agent services.
• Generate alerts on disk utilization of Secure Agents.
• Take preventive actions using custom scripts based on generated alerts.
• In-app notification of infrastructure alert messages.

Ecosystems and Connectivity

• Expanded coverage in terms of functional depth and breadth for cloud ecosystems:
  • Azure:
    • SQL Data Warehouse V3: Unconnected lookup, ADLS Gen2 as optional staging area for Polybase, improved exception handling, source/target parameterization overrides with parameter files, parameterization overrides (for schema, database, and table) in PRE SQL/POST SQL/SQL Override, and performance enhancements.
    • ADLS Gen2: Source/target parameterization overrides with parameter files,  Azure Gov Cloud, FileName port, New data type (Date/Time) support for Parquet files, Parquet Decimal, Date/Time data type support, User authenticated proxy, and performance enhancements.
    • Blob Storage: Source/target parameterization overrides with parameter files.
    • CDM Folders (available forpreview only): Support for new CDM schema (v0.9) and unicode character support.
  • AWS:
    • S3 V2: Hierarchical data types, ISD, multi-region, Parquet Decimal, Date/Time datatype support, KMS (other accounts), source/target parameterization.
    • RedShift: Ecosystem pushdown optimization (S3 to RedShift), JDBC driver update, KMS (other accounts), multi-region, source/target parameterization.
  • Snowflake DW:
    • Database pushdown optimization enhancements.
    • Unconnected lookups with pushdown optimization.
    • Snowflake on Google Cloud Platform.
    • OAuth2.
    • gzip compression.
  • Google:
    • IICS on Google Cloud Platform.
    • BigQuery: CDC support, lookup.
    • Google Cloud Storage: Read from directory.
  • Microsoft Business Apps:
    • Common Data Model (CDM) folders: Schema update (0.9).
    • Dynamics 365 Operations:  Certificate, update retry.
    • CRM:  Client secrets.
  • SAP:
    • SAP BW Reader: Supports dates before the year 1753.
    • SAP HANA (new private connector): Read from tables and modelling views (analytical, attribute, and calculation views).
  • Salesforce:
    • Sales and Service Cloud: API updates,  DRS enhancements to support deleted records.
    • Commerce Cloud: Cycle dependencies.
  • Oracle Apps:
    • Oracle HCM:  Writer updates.
    • NetSuite: API update (2019.2).
  • Adobe:  XDM connector enhancements.
• Support for new patterns and use cases:
  • Cloud Data Integration Elastic: 
    • Amazon S3 V2:  Hierarchical data type and ISD support.
    • Azure SQL Data Warehouse V3: Elastic mappings.
    • ADLS Gen2: Elastic mappings (available for preview only).
    • JDBC V2:  Scala v2,partitioning, AWS and Azure runtime support.
  • Cloud Mass Ingestion:
    • Databricks Delta
    • ADLS Gen2: Direct Polybase load from ADLS Gen2 and performance improvements.
  • Cloud Data Quality: CDM
  • Kafka Pub/Sub Connector in Cloud Data Integration.
• Improved connectivity across horizontal technology, systems, and applications:
  • Technology:
    • OAuth 2.0 JWT support.
    • REST V2:  Real-time transaction support, minor enhancements,  Hadoop 3.1 for Cloudera, 4.1 for HDInsights.
  • Database and EDW:
    • ODBC:  Unconnected lookup.
    • Oracle:  Blob/Clob datatype.
    • MySQL:  Advanced runtime properties.
    • SQL Server:  Advanced runtime properties.
    • Azure SQL DB: Bulk, Upsert.
    • PostgreSQL:  Schema name enhancements.
  • MongoDB:  BSON and JSON document support, partitioning, create target with schema-less support.
  • Workday:  Hierarchy parameterization support.
  • ServiceNow:  Upsert.
  • Cloud Apps:
    • Cvent:  Child object
    • Coupa: Proxy
• New Add-on Connectors page to access add-on connectors on Marketplace.

We are hosting a webinar with Kelly Services on September 4, 2019. Ravi Ginjupalli reviews the process to master 4 critical domains, starting with customer, and connecting Informatica MDM with Salesforce through IICS. The webinar is co-presented with Capgemini. This is also a Teradata to Azure migration at Kelly Services. Hear about the story, challenges, and lessons learned at Kelly Services as Ravi Ginjupalli and Venkat Gupta of Capgemini discuss the approach and technology behind their effort.

You can register for the webinar on Sept 4, 2019, at 11 AM PDT here: https://www.informatica.com/about-us/webinars.html?commid=367064

They realize:

• 99.99% accuracy in identifying duplicates using MDM match rules
• 1000 Average number of Accounts weekly (Includes both prospects and new clients)
• 500k Number of customer loaded during the initial load
• Eliminated 18% duplicate Customer records in Salesforce
• 6 month Time to deliver; Salesforce + D&B integration + Hierarchies; 180 tables; 5218 Attributes; 288 mapping

ALERT:  for customers using Informatica Cloud

To better serve our customers, we place older connectors and unused connectors in End-of-Life (EOL) or Maintenance Mode. If you need a new connector enabled, per the customer action below, please create a shipping case and request we add the new connector to your org(s). The differences between Maintenance Mode and EOL are summarized in table below:

Am I impacted?

Refer to the list below to determine if you are using one of these connectors.  As necessary, emails are sent to "Subscription" customers for these connectors in advance of EOL.  For EOL connectors, customers are given 6 months notice to address their existing mappings; maintenance mode connectors will move to EOL as appropriate.

How do I address the issue?

Please reference Customer Action in table above; and Notes to Customers and Alternative connector columns in table below.

The following table shows the connectors in end-of-life (EOL) or maintenance mode

The table below is continually updated.

Updates

• 6/2019: added MemSQL and Eloqua
• 7/2019: added Ariba
• 11/2019: added BigMachines
• 2/2020: added MSD AX v3, MSD NAV 2009-13
• 5/2020: added Amazon Aurora, Hadoop Files, FileList, FileProcessor, JDBC
• 6/2020: revised alternative connector for Intuit Quickbooks Online

Here is an orientation video for the upcoming Informatica Cloud to Informatica Intelligent Cloud Services (IICS) migration process. The video provides an introduction to IICS for customers that are migrating from Informatica Cloud.

We will be migrating organizations between April and August. You will receive notification when YOUR organization(s) is going to be migrated and when your sandbox environment (pre-release) is available.

Video link (moved to YouTube so that it doesn't have to be downloaded):

Introducing Informatica Intelligent Cloud Services - YouTube

Full FAQ:

Informatica Intelligent Cloud Services Migration FAQ