This guide walks you through the end-to-end process of creating a Databricks source in the Integrate module, from selecting the source category to verifying a successful integration. For detailed information about the underlying components and prerequisites, refer to Create and Implement a Databricks Source.Documentation Index
Fetch the complete documentation index at: https://docs.zeotap.com/llms.txt
Use this file to discover all available pages before exploring further.
Step 1: Open the Sources Tab and Click Create Source
Navigate to Integrate → Sources in the Zeotap CDP App. Click Create Source in the top-right corner of the Sources listing page.
Step 2: Select Data Warehouse as the Category
In the Create Source dialog, you will be presented with a list of source categories. Select Data Warehouse to proceed with a Databricks source.
Step 3: Fill in Source Details
The Source Details step collects the configuration needed to connect to your Databricks workspace. Fill in the following fields:- Source Name — Enter a short, descriptive name for the source.
- Sequence — Specify the sequence number for data ordering.
- Sync Frequency — Set how often data should be synchronised (for example, hourly, daily, or weekly).
- Host — Enter your Databricks workspace URL.
- Catalog Name — Provide the catalog that contains your target data.
- Schema Name — Specify the schema within the catalog.
- Table Name — Enter the name of the table to ingest.
- Data Entity — Choose whether you are ingesting Customer Data or Non Customer Data.
- Delta Queries Selection — Set to true if you want to fetch only new and updated records based on a timestamp column. Set to false to fetch all records on every run.
Step 4: Choose the Connection Type and Authentication
Under Type, select the mechanism used to pull data from Databricks:- JDBC — The standard connection method.
- Job Based — The recommended option for large data volumes.
NoteIf you are ingesting more than 1 million records, use the Job Based approach. JDBC may encounter performance issues with large data volumes.
Step 5: Configure JDBC Authentication
If you selected JDBC, enter the following credentials:| Field | Description |
|---|---|
| Client ID | The OAuth client ID from your Databricks service principal. |
| Client Secret | The OAuth client secret associated with your service principal. |
| HTTP Path | The HTTP path of your Databricks SQL warehouse. Found under SQL Warehouses → Connection details. |
| Partition Column | Optional. A column used to split large datasets for parallel reads. Use a unique column if no partition column exists. |
Step 6: Configure Job-Based Authentication
If you selected Job Based, choose the Auth Type and enter the corresponding credentials: PACKTOKEN| Field | Description |
|---|---|
| Pack Token | A personal access token generated from your Databricks account. Navigate to Settings → User Settings → Developer → Generate new token. |
| Field | Description |
|---|---|
| Client ID | The client ID of the Databricks service principal. |
| Client Secret | The client secret associated with the service principal. |
Step 7: Finalise and Verify the Source
After entering all required fields, click Next to proceed to column selection. Select the columns you want to ingest, then click Create Source.
NoteThe initial data transfer from Databricks to Zeotap CDP may take time depending on data volume. For assistance with Databricks source setup, contact the Zeotap support team at support@zeotap.com.