Databricks Delta Connector > Introduction to Databricks Delta Connector > Databricks Delta Connector overview
  

Databricks Delta Connector overview

You can use Databricks Delta Connector to securely read data from or write data to Databricks Delta.
You can create a Databricks Delta connection and use the connection in mappings, elastic mappings, and mapping tasks. The following section explains how does the Secure Agent communicate with Databricks Delta during the design time and the run time:
Design-time data flow for mappings and elastic mappings
During the mapping design, the Secure Agent communicates with the Databricks all-purpose or analytics cluster for metadata-related operations.
Run-time data flow for mappings
The following image shows how the Secure Agent connects to Databricks Delta to read or write data in Cloud Data Integration mappings:
The Secure Agent uses Amazon S3 in AWS environment or Azure Data Lake Storage Gen2 in Azure environment for staging the data. The Secure Agent creates Databricks data engineering cluster to read data from or write data to Databricks Delta tables. When Databricks Delta connector is used as a source, the Secure Agent runs a spark job in the Databricks data engineering cluster to read data from a Databricks Delta table and write to the staging location. When Databricks Delta connector is used as a target, the Secure Agent runs a spark job in the Databricks data engineering cluster to read data from the staging location and write to a Databricks Delta table.
Run-time data flow for elastic mappings
Elastic mappings make use of the elastic cluster to run a spark job and process data.