Authors: inuser445331 Skip navigation

Cloud Data Integration

3 Posts authored by: dilip poluru

<This community article will be periodically updated, as needed, until IICS Summer 2019 (R32) GA>.

 

The IICS Summer 2019 release offers several new capabilities that include integration patterns, expanded connectivity, and platform functions. Highlights are listed below.

 

Data Integration

  • Machine learning-based transformation recommendations allow you to get suggestions for the next best transformation based on the mapping you are working on.
  • Improved support for custom, local parameter file locations, support for fully parameterized SQL queries, ability to overwrite connection and Taskflows support invocation through the RunAJob command line utility as well as through the new file listeners. These can be passed as parameters to data tasks for listener-driven events.
  • A new and improved taskflow canvas with a more compact layout improving design experience
  • The SQL transformation now supports ad hoc SQL queries (in addition to Stored Procedures).
  • Intelligent Structure Discovery
    • Enhancements to existing ISD models with data from additional sample files (JSON & XML) to quickly adapt to data drift
    • The Structure Parser transformation can now update or change the associated intelligent structure model without breaking unchanged ports mappings which enhances mapping flexibility and allows for fast adaptation to changes in incoming files.
    • Simplified design time experience with the ability to add a prefix or a suffix to existing node names using the bulk action functionality
    • Enhanced parsing engine support with the ability to discover and parse AVRO files
  • Files Mass Ingestion  
    • Additional sources and targets:  ADLS GEN1 as source, ADLS GEN2 as source and target, Snowflake as source and target
    • Support for custom actions during file transfer:  compress/decompress, encrypt/decrypt
    • Integration with Enterprise Data Catalog (EDC): select Mass Ingestion source from EDC
    • Filter files to transfer by file size and date
    • Optimization for large file handling – resume from point of failure

 

Application Integration

  • Developer productivity enhancements: new compact canvas, improved diagnostics and debugging 
    • Improved Process and Guide canvas compact layout improving design experience
    • New validation panels help diagnose service connectors and connection errors
    • Fault details are now available with every step to diagnose runtime errors and improve debugging. 
  • Guides: Improved page layout and Salesforce Lightning theme
    • Guide Screen step designer has been revamped to provide more real estate to design guides
    • Salesforce Lightning theme support is now available for guides and the Guide Launcher
  • Source control and deployment automation 
    • New CLI utility automates export and extract of design assets to any source control system. The package, import and publish capabilities of the CLI provides the ability to deploy full deployments, packages and patches across environments.
    • Sample Jenkins pipeline to help developers adopt continuous integration practices for Application and Data Integration, and help operations staff with the automation of continuous deployments
  • Expanded pub/sub and messaging capabilities with Salesforce Platform Events and RabbitMQ to compliment the current support for AMQP, JMS, AWS SNS/SQS, Azure Service Bus/Event Hub and Kafka
    • New support for Salesforce Platform Event and Push Topic events enabling customers to deliver secure, scalable and customizable event notifications within Salesforce or from external sources 
    • New RabbitMQ native connector providing support for durable queues 

 

API Management

  • CLAIRE-enabled privacy data leak identification - Privacy data identification in API content, masking or blocking of sensitive personally identifiable information. (This feature is in preview mode).
  • Ability to customize the structure of API URLs.
  • Support for the internationalization of API, group and organization names.

 

Integration Hub  

  • Governance enhancement - Support for predefined and custom roles with granular privileges that promote governance.

 

B2B Gateway

  • Exchange files with your partners using Cloud AS2.
  • Get insights into your daily gateway activities and statuses with the new dashboard.

 

Platform

  • Asset Dependency feature providing “used by” and “uses” relationship information across all asset types and services for impact analysis.
  • Common Explore enhancements: changes made by the user to the explore column settings are now automatically persisted for that user for various filtered views such as “all assets,” “projects & folders,” etc.
  • More control and flexibility in managing users: Grant/deny service level access to a user through assign services capability. Disable users to prevent user logins for both native and SAML authentication mechanisms. REST APIs for user management – users, groups and roles.
  • Ability to start/stop Secure Agent services to optimize computing resources and isolate the need to restart a single service from others.

 

Connectivity Enhancements  

  • New native connectors to connect to CDM/Power BI, Snowflake (V2), MongoDB, Adobe Experience Platform, Ariba (V2), and Cassandra
  • Updates to existing connectors: Microsoft (MSD 365 Operations, Cosmos DB, ADLS Gen2 and Gen 1, SQL DW), Google (Google Big Query, Google Storage), Amazon (S3, Redshift), Oracle NetSuite, Salesforce, SAP (Concur V2, S/4 HANA) and Greenplum

 

New IICS Services

The IICS Summer 2019 release offers the following new services:

 

Streaming Ingestion (included in Data Ingestion Service)

  • Ingestion of streaming data from logs, Apache Kafka and IoT sources
  • Supports AWS & Azure ecosystems as targets of Ingestion: Amazon Kinesis Streams, Kinesis Firehose, Amazon S3 and Azure EventHub
  • Supports lightweight transformations at the edge
  • Real-time monitoring of streaming ingestion jobs

 

Integration at Scale (included in Data Integration Service)

  • Run data integration jobs at scale on a fully managed spark serverless cluster.
  • Support for AWS ecosystem: S3 & Redshift
  • Auto-tune and scale fully-managed Kubernetes clusters.

 

Operational Insights

  • Operational Insights is now available for IICS cloud services to help with efficient infrastructure monitoring and alerting. IICS cloud services users can monitor the health of a) Secure Agents, b) services running on each Secure Agent, c) Runtime environments and d) subscribed cloud services.
  • Email Alerts can be set on services and Secure Agents for unavailability or excessive resource consumption

 

Reference 360  

  • Enables business users to manage enterprise reference data in an easy-to-use, configurable, business-friendly user interface.
  • Provides versioning, collaboration and complete life-cycle management of reference data.
  • REST APIs for managing data and meta-data, enabling automation and system integration.
  • Improved stability and performance.

 

 

Data Quality

  • Enable business users to create rule specifications to perform a variety of Data Quality actions including completeness, validation and standardization.
  • Create and manage value lists for data quality operations using dictionaries. Dictionaries can be used to identify, validate and standardize data as part of a rule specification.
  • Embed Data Quality rules in Data Integration mappings to support Data Quality activities in typical data integration scenarios such as data warehousing, data migrations and more.

 

Thank you, 

The IICS Team

Today, we are incredibly excited to announce the launch of Informatica Intelligent Cloud Services - Winter 2017 December Release. At Informatica, our mission is to accelerate data driven digital transformation of businesses. At the heart of the mission is the innovation in enterprise cloud data management space to meet growing needs of enterprises. To realize this goal, we built Informatica Intelligent Cloud Services (IICS) from ground up leveraging cloud native frameworks.

 

Based on a micro services architecture and a modern user interface, the IICS platform is built for the future to provide complete end-to-end data management in a uniform, non-siloed approach. IICS unifies existing Informatica cloud service offerings and expands into a full suite of cloud data management services over time. IICS will be comprised of four clouds: Integration Cloud, Data Quality & Governance Cloud, Master Data Management Cloud, and Data Security Cloud, powered by a common core platform and the Informatica CLAIRE metadata intelligence engine.

 

The Winter 2017 December Release is the initial offering of Informatica Intelligent Cloud Services and covers Cloud Data Integration within Integration Cloud. This release offers a rich set of new capabilities to our customers.

 

Below are some highlights of this release:

 

New & Modern User Interface Experience

First, the new user interface design provides a consistent look and feel across all intelligent cloud services and experiences tailored to user roles: Designer, Operator and Administrator using a common user interface shell and service switcher.

 

Next, the new design introduces the concept of workspaces which allows users to keep multiple tabs open within a cloud service. With this capability, users don’t lose the state of assets they are working on when they navigate from one screen to another or open multiple assets for additional context to complete their workflow.

 

Finally, the Home Page provides actionable insights like runtime/job health, recent organization-wide activity as well as easy access to Marketplace and Community.

 

Template Driven Development

Productivity is a key theme for this release. As part of this theme, we added template driven development capability. Customers can now create a new integration asset either from scratch or by choosing a template from dozens of templates packaged as part of the Data Integration service. Templates include pre-built logic and cover data integration areas like data preparation, cleansing and data warehousing.

 

In addition to providing templates to our customers, we want to continue to empower customers to create their own templates/solutions and share those within their organization and even publish to Informatica Marketplace. Towards that end, we expanded bundles capability to support mapping tasks in addition to mappings, Visio templates and mapplets.

 

Templates and bundles together provide reusability and promote best practice design, thus boosting productivity and simplifying the development experience for organizations.

 

Enterprise Orchestration

The advanced taskflow capability of this release enables users to design complex taskflows by orchestrating mapping tasks and synchronization tasks in a non-linear fashion. Users can define custom logic that involves actions like parallel tasks, loops, conditions, decisions, wait time, exception and error handling, thus achieving much more complex orchestrations than before. The advanced taskflow designer has a similar user interface as the mapping designer, and integrates seamlessly with the Monitor service.

 

File Mass Ingestion

The file mass ingestion capability of this release enables customers to transfer enterprise data assets in a flat file format from on-premise to Amazon S3 datastores and Amazon Redshift data warehouses in cloud using standard protocols – FTP, SFTP and FTPS. Developers can easily author mass ingestion tasks using a wizard-based approach and monitor task executions both at job and file granularity using the same Monitor service used for other data integration jobs. File mass Ingestion is robust and is designed to handle thousands of files a day in a single process.

 

Integrated Asset Management

As data complexity and the scope of integration work grows, asset management becomes a challenge for customers. Projects and folders in the Explore user interface help address this challenge and enable customers to organize their integration assets using taxonomy suitable for their business needs, applying proper security controls.

 

Administrators can now manage project and folder structures and grant access to users through role-based security. Administrators also have flexibility to set fine grained access control at the individual project, folder or asset level on top of role based security.

Once security is granted, developers can easily store, discover, search and manage their assets through actions exposed via Explore. As business requirements or team structures change, developers can also reorganize their assets using copy/move operations which ensure asset dependencies are taken care of.

 

Design-time assets organized in projects and folders can continue to be easily attached to runtime assets like schedules, agents and connections. Through proper design of role based and instance based security for both design-time and runtime assets, enterprises can realize the right level of isolation for their organizations.

 

APIs that enable Continuous Delivery

With the IICS platform, we are promoting DevOps practice that includes iterative development throughout the lifecycle of the project with close collaboration between developers and IT operations. To enable DevOps practices, we are surfacing project/folder/asset export and import APIs that facilitate continuous delivery through automation with external version control systems, release and deployment pipelines. In addition to these capabilities that enable continuous delivery, IICS provides automated monitoring capabilities that completes the agile team based development model between developers and operations.

 

Connectivity Enhancements

Informatica provides extensive connectivity to on-premises and cloud applications and services. We have extended connectivity options in this release by adding several new connectors, most of which are co-engineered with our ecosystem partners. New connectors in this release include: Snowflake, AWS Redshift Spectrum, Microsoft Azure Blob Storage (version 2), Oracle HCM, Microsoft Dynamics 365 for Sales, SAP SuccessFactors, SAP Ariba, MemSQL, CallidusCloud Badgeville, CallidusCloud Litmos. To see a complete list of available connectors, click here

 

Cloud data warehouse connectors such as Snowflake, AWS Redshift and Microsoft Azure Data Warehouse are optimized for efficient data loads and includes features such as pushdown optimization and partitioning.

 

We have also made significant enhancements to over 40+ connectors across application and SaaS connectors such as NetSuite, Microsoft Dynamics and Salesforce; analytics connectors such as Tableau and Cloud platforms such as Google, Amazon AWS and Microsoft Azure.

 

Support for popular data formats such as Avro, Parquet, JSON was being added to cloud object store connectors. Performance enhancements include leveraging the latest APIs, optimizing for bulk loads and adding additional partitioning options. Technology connector enhancements include a new connector for OData, support for additional Hadoop distributions, and enhancements to REST and file connectors.

 

Informatica is committed to offering universal connectivity across the IICS Platform. This release enables Informatica to deliver new and enhanced connectors at more rapid pace by introducing a connectivity micro service in the platform. The connectivity micro service also enables Informatica to address any issues more seamlessly providing an overall superior experience.

 

Conclusion

These are just some of the highlights of this release. There are also other useful capabilities in this release like custom roles support, self-service non-administrator user registration, in-service notifications, improved job monitoring to isolate job issues, PowerCenter to cloud conversion utility and more.

 

Informatica Intelligent Cloud Services – Winter 2017 December Release is a significant milestone. At Informatica, we are laser focused to deliver the next generation of iPaaS and Data Management, and this is just the beginning of a new journey for us. We can’t wait to see how customers around the world use Informatica Intelligent Cloud Services to unleash the power of data!

 

Learn More

iPaaS Reimagined

journey.jpg

 

Earlier this month, the Informatica Cloud virtual summit event showcased how Informatica is uniquely positioned to help customers in their journey to the cloud, whether it is a cloud first or hybrid cloud journey. Today, I’m excited to announce the preview program of Informatica Cloud Spring 2017 which will further help customers to more efficiently address their complex data management needs particularly cloud data warehousing.

 

Informatica Cloud Spring 2017 delivers significant capabilities across all application areas – Cloud Application and Data Integration, B2B Gateway and Cloud Integration Hub along with underlying Platform elements like Connectivity and Secure Agent that enable all integration patterns for different deployment models.

 

Here is a sneak peek into key capabilities of Informatica Cloud Spring 2017:

 

Secure Agent Platform

Customers always wanted minimal downtime and component level isolation during Informatica Cloud upgrades or customer-driven maintenance activities. The Spring 2017 release made this a reality. Customers can now take advantage of the Secure Agent platform to run multiple services independently. Built on microservices architecture, the Secure Agent platform can intelligently run multiple pluggable engines and auto-update itself with uninterrupted job execution during quarterly upgrades or monthly patches.

 

Secure Agent Audit Filtering

Customers can easily filter the Secure Agent audit information on the Agent Audit page by date range, component type and message type.

 

Sub-organization Secure Agent Group Sharing

In addition to high availability and scalability, the release provides agent deployment flexibility and resource optimization through Sub-organization Secure Agent group sharing. Using this capability, OEMs can now setup a single Secure Agent group for the parent organization and use that Secure Agent group for more than one customer, where each customer uses a sub-organization. Similarly, enterprises can now use Secure Agent group resources of the parent organization across multiple departments, where each department runs its jobs in a sub-organization.

 

Cloud Application Integration

Cloud Application Integration customers also benefit with the introduction of new Secure Agent platform capabilities offered in this release. The Process Server service is a new engine of the Secure Agent platform. It is the component of the Informatica Cloud Real Time service responsible to execute processes deployed to the Secure Agent. This version of the agent-based Process Server service offers:

• PostgreSQL as a replacement to the H2 Database. The PostgreSQL database will handle larger amounts of data than the H2 database previously-bundled could. You will also observe performance improvements.

• HTTPS Listener Support. This version now provides the ability to instantiate agent-based REST and SOAP processes using an HTTPS transport in addition to JMS and AMQP queues and topics.

• Secure Agent Group Awareness. This release provides the ability to employ the Secure Agent group feature to logically group agent-based Process Servers and distribute the load across each member of the group. This provides the ability to scale.

• Agent Clustering. This release provides you with the ability to cluster two or more Secure Agent group members into a single logical Process Server instance to benefit of high-availability features such as process instance failover in the case of a node failure.

 

 

Cloud API Management

This release provides Cloud Application Integration customers with new API Management capabilities. Customers can not only API-enable services and data sources, and create composite service and data APIs using OData, REST/XML, JSON or SOAP/WSDL using the Informatica Cloud Real Time Application Integration Service, they can now expose managed APIs to partners, customers and internal consumers through the Informatica Cloud API Gateway. In conjunction with API Manager, API gateway provides the means by which to control and secure access to the APIs you expose.

 

 

Cloud Application Integration customers can now take advantage of new API management features delivered with Informatica Cloud Real Time Advanced and Informatica Cloud Premium editions.

 

 

Cloud B2B Gateway

Artificial Intelligence is now embedded inside Cloud B2B Gateway. Using Intelligent Structure Discovery functionality to discover, parse, and map non-EDI messages, Cloud B2B Gateway expedites the on-boarding of partners that do not use EDI files. This release also provides end to end order-to-cash and procure-to-pay processes on EDI and EDIFACT standards, with out-of-the-box EDI solutions with full visibility and tracking to files exchanged with the partner community.

 

 

Cloud Integration Hub

Cloud Integration Hub simplifies data integration and increases user efficiency by allowing users to publish data once and consume data from multiple subscribing applications. Cloud Integration Hub Salesforce Accelerator is available with this release. The accelerator automatically creates the entities that are required to connect Salesforce to Cloud Integration Hub, so users can publish data from salesforce and have it ready for other applications to consume.

 

Cloud Integration Hub is now offered as a free trial to all Informatica Cloud Premium and Advanced edition customers. Customers will find the user interface modern, clean and simple to use.

 

Power Exchange for Cloud Applications

Existing PowerCenter customers can now take advantage of PowerExchange capability made available in this release to accomplish hybrid integration scenarios.Customers can continue to use the PowerCenter Designer interface that they are already familiar with and seamlessly access all Informatica Cloud connectors associated with the Informatica Cloud account directly from PowerCenter. Informatica Cloud thus acts as an agile connectivity service layer for PowerCenter installations in enabling the above scenario.

 

Connectivity

Informatica continues to provide increased support to all leading cloud PaaS providers (AWS, Microsoft, Google) and SaaS vendors (like Salesforce, NetSuite, Workday, Marketo, etc.) so customers can easily light up their cloud data warehouse and analytics use cases.

The following connectors are made available in this release:

• JIRA Cloud

• Teradata Parallel Transporter API

• IBM DashDB via ODBC Connector

• Zuora REST API

• SWAGGER definition generation for REST API connector

• Oracle bulk insert support via REST API connector

 

Connector enhancements to cloud ecosystems in this release include advanced configurations to the user to customize integrations with various sources/targets suitable for their needs, performance and security benefits. Below are some examples.

 

Amazon Web Service

Customers can now manage source/target connection strings effectively and securely by passing Amazon KMS-generated keys for connections associated with Amazon S3 & Redshift services.

 

Customers can now take advantage of multi-part upload option for improved performance and increased throughput when moving data using Amazon S3 & Redshift services.

 

Customers can now read and write multiple files from/to Amazon S3 through configuration settings.

 

 

Microsoft Cloud

From this release, customers can use Informatica hosted secure agent to connect to DocumentDB. The new Microsoft Azure SQL Data Warehouse V2 connector enables customers to read data from and write to SQL Data Warehouse. The connector supports insert, upsert and delete operations. Microsoft Azure Data Lake Store V2 connector is coming soon in May.

 

Dynamics CRM connector is enhanced to connect to Dynamics 365.

 

 

Google Cloud

Customers who use Google data storage services for their analytics scenarios can now leverage Informatica connector enhancements to Google Big Query and Google Cloud Storage.

 

It’s now simple and easy to perform read, write, and update operations against Google Big Query.

 

Customers can create standard Informatica asset types like mappings, mapping configuration tasks, and data synchronization tasks against Google Cloud Storage and read data in binary or string formats in single or multi-threaded fashion. Customers can also use Google Cloud Storage for staging purposes to write to Google Big Query.

 

Support for New Regions

This release introduces new connector code pages to all Informatica supported relational connectors. The support enables customers to manage their data in key Asian and middle-eastern languages.

 

Customers using AWS services for their data integrations can now leverage Informatica support to read and write to S3 buckets / Redshift data warehouses located in US East (Ohio), Asia Pacific (Mumbai) and Canada regions.

 

 

Learn More

You can see demos of some of the capabilities above in action in the upcoming webinar on wednesday, Apr 12, 9:00am. More videos in cloud first / hybrid cloud jouney series:

 

General Availability of this release is expected to be on Apr 22nd 2017. For questions about the Spring 2017 pre-release, send an email to pre-release@informatica.com.

 

Informatica Cloud Spring 2017 is an important milestone. We are incredibly excited to put this into the hands of our customers to accelerate their journey to the cloud. We can’t wait to see how customers around world use Informatica Cloud Spring 2017 to increase their productivity and unlock insights.