Automations

Pipelines vs. Automations

When to reach for which tool.

Last updated June 29, 2026
Reading time 2 min read

Both pipelines and automations run on a cron schedule. Both can do data work. So when should you reach for which?

The 30-second answer

  • Pipeline: one source → one target, maybe a transform in between, on a schedule.
  • Automation: anything with multiple steps, or work that mixes pipelines, SQL, semantic models, or Databricks jobs.

If you find yourself saying "and then, and then, and then" describing a workflow — it's an automation.

Side-by-side

Question Pipeline Automation
Is it just moving data? ✅ Can, but overkill
Multiple steps in sequence? ❌ ✅
Run a SQL script after a sync? ❌ ✅
Build or redeploy a semantic model? ❌ ✅
Trigger a Databricks job or notebook? ❌ ✅
Email the team on success / failure? ✅ Built in ✅ Built in
Cron schedule? ✅ ✅

Examples that help

Clearly a pipeline

  • "Sync Salesforce Accounts to Lakehouse every hour."
  • "Pull yesterday's CSV drop from S3 into a staging table."
  • "Copy Postgres to Databricks overnight."

Single source, single target, simple schedule. Pipeline.

Clearly an automation

  • "Every Monday at 8am, run three pipelines and then a SQL rollup."
  • "Every morning, run the Salesforce pipeline, then redeploy the Finance semantic model."
  • "Nightly: run a Databasin pipeline, then trigger a Databricks job that builds the gold tables."

Multiple steps, multiple systems. Automation.

The fuzzy middle

  • "Sync Salesforce, then refresh a materialized view that depends on it."

This is where people pause. Two reasonable choices:

  • Pipeline with a post-sync SQL step. Good if the SQL is small and always runs on the same target. Simplest to operate.
  • Automation with two tasks. Better if the SQL depends on multiple sources, or you want to split the triggers cleanly ("refresh view" becomes its own reusable building block).
When in doubt, start with a pipeline

It's cheaper to upgrade a pipeline into an automation later than to build an over-engineered automation on day one. A single-pipeline-with-a-post-step handles 80% of cases.

Don't split hair, split work

A useful rule: each pipeline should do one conceptually clean sync. Each automation should do one conceptually clean workflow. If you're trying to cram three things into one of either, the shape is wrong.

Small, focused primitives compose. Giant everything-pipelines break.

Where to go next

Automations overviewThe full picture of triggers and tasks. Pipeline schedulingSchedules, overlap protection, pauses. Monitoring and alertsWhere failures surface.