databricks_pipelines Data Source

Retrieves a list of all databricks_pipeline (Lakeflow Declarative Pipelines) ids deployed in a workspace, or those matching the provided search term. Maximum 100 results.

-> This data source can only be used with a workspace-level provider!

Example Usage

Get all Lakeflow Declarative Pipelines:

data "databricks_pipelines" "all" {}

output "all_pipelines" {
  value = data.databricks_pipelines.all.ids
}

Filter Lakeflow Declarative Pipelines by name (exact match):

data "databricks_pipelines" "this" {
  pipeline_name = "my_pipeline"
}

output "my_pipeline" {
  value = data.databricks_pipelines.this.ids
}

Filter Lakeflow Declarative Pipelines by name (wildcard search):

data "databricks_pipelines" "this" {
  pipeline_name = "%pipeline%"
}

output "wildcard_pipelines" {
  value = data.databricks_pipelines.this.ids
}

Argument Reference

This data source exports the following attributes:

• pipeline_name - (Optional) Filter Lakeflow Declarative Pipelines by name for a given search term. % is the supported wildcard operator.
• provider_config - (Optional) Configure the provider for management through account provider. This block consists of the following fields:
  • workspace_id - (Required) Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.

Attribute Reference

This data source exports the following attributes:

• ids - List of ids for Lakeflow Declarative Pipelines pipelines matching the provided search criteria.

Related Resources

The following resources are used in the same context:

• End to end workspace management guide.
• databricks_pipeline to deploy Lakeflow Declarative Pipelines.
• databricks_cluster to create Databricks Clusters.
• databricks_job to manage Databricks Jobs to run non-interactive code in a databricks_cluster.
• databricks_notebook to manage Databricks Notebooks.