Prefect

Prefect is a Python workflow orchestration framework for turning ordinary scripts into production data pipelines. It's aimed at data engineers, data scientists, and ML engineers who need scheduling, retries, and monitoring without rewriting their code into a rigid DAG format.

At its core, Prefect wraps existing Python functions with @flow and @task decorators. This keeps the orchestration layer close to your code instead of forcing you into a separate configuration language or a fixed pipeline structure. You get dynamic branching, conditional logic, and loops, because it's just Python.

Key features

• Decorator-based API: add @flow and @task to existing Python functions to get orchestration, logging, and observability with minimal code changes.
• Scheduling: turn any flow into a deployment with a cron schedule (or other schedule types) using .serve().
• Retries and caching: built-in retry logic and caching reduce the amount of custom error-handling code you have to write.
• Event-based automations: trigger deployments in response to events, not just fixed schedules.
• Dynamic workflows: because flows are Python functions, you can use standard control flow (loops, conditionals, branching) instead of a static DAG definition.
• Local UI: prefect server start spins up a dashboard on localhost to inspect flow runs, task states, and logs.
• Managed option: Prefect Cloud offers a hosted dashboard and orchestration layer if you don't want to run your own server.
• Lightweight client: prefect-client provides SDK access for ephemeral environments that only need to talk to Prefect Cloud or a remote server, without the full package.

Ideal use cases

Prefect fits teams that already write data pipelines in Python and want to add reliability features (retries, scheduling, alerting) without switching languages or adopting a heavyweight orchestration platform. It works well for ETL/ELT jobs, ML training pipelines, and any recurring script that currently runs via cron with no visibility into failures.

It's also a good fit if you need workflows that react to events, not just fixed time schedules, or if you want a local dashboard to debug why a task failed last night. Teams moving off ad-hoc cron jobs or brittle bash scripts get immediate value from the retry and caching primitives.

It's less of a fit if your pipelines are simple, run once, and never fail in ways that need automated recovery. A plain script or a lightweight task runner may be enough in that case. Prefect also assumes a Python-first environment. Teams standardized on other languages, or needing a purely declarative config-driven orchestrator (like some YAML-based CI tools), may find the decorator-based approach a mismatch. Very large, static, dependency-heavy DAGs with strict scheduling requirements across many teams might be better served by tools built specifically for that shape of problem, though Prefect can handle a lot of that too.

Installation

Prefect requires Python 3.10 or later. Install it with pip or uv:

pip install -U prefect

uv add prefect

Write a flow using the @task and @flow decorators:

from prefect import flow, task
import httpx


@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")


@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
    for repo in repos:
        get_stars(repo)


if __name__ == "__main__":
    github_stars(["PrefectHQ/prefect"])

Start a local Prefect server to view flow runs in the UI:

prefect server start

The UI runs at http://localhost:4200. To schedule the flow to run every minute, replace the __main__ block with:

if __name__ == "__main__":
    github_stars.serve(
        name="first-deployment",
        cron="* * * * *",
        parameters={"repos": ["PrefectHQ/prefect"]}
    )

This starts a local process that picks up scheduled runs, and you can also trigger runs manually from the UI or CLI, or in response to events.