Mastering Apache Airflow, Part 5: Executors and Auth Managers

Unpacking the Power Behind Task Execution: How Executors and Auth Backends Shape Airflow's Runtime and Security Ecosystem

Giving Context🌟

Hello everyone!!!

In the previous chapters, we explored the intricacies of DAG lifecycles, task orchestration, and execution dynamics. Now it’s time to dive deeper into the operational backbone of Airflow.

In this issue, we’ll focus on three essential components that shape how Airflow runs and scales in production environments:

• Executors – the core abstraction behind how tasks are actually executed, whether locally, on a Celery worker, or inside a Kubernetes pod. We’ll explore the built-in executors, how they differ, and how you can even use multiple at once.

• Auth Managers – Airflow’s flexible, pluggable authentication and authorization layer, crucial for securing the web UI and API.

• Object Storage Integration – how Airflow handles large artifacts, task outputs, and remote logs through external object stores like S3 or GCS.

Understanding these systems will allow you to build more robust, secure, and scalable Airflow deployments tailored to your team’s infrastructure and operational needs.

Let’s get started with executors—the engine behind task execution.🚀

---

🔧 Executors: The Task Runners Behind the Curtain

In Apache Airflow, executors are essentially the engines that actually run your tasks. Think of them as the muscle behind the orchestration brain, each executor implements a pluggable interface that the Scheduler delegates work to.

You set the executor in your airflow.cfg under the [core] section:

[core]
executor = KubernetesExecutor

You can also specify a custom one via module path:

[core]
executor = my.custom.module.ExecutorClass

Want to check what executor is currently set?

$ airflow config get-value core executor

---

🧱 Executor Categories

Airflow ships with multiple executors, each with distinct tradeoffs. Let’s break them down in points, so that you can actually understand what’s going on in simpler terms:

🖥️ Local Executors

Local executors run tasks in the same process or host as the Scheduler.

• Pros: Simple to set up, low latency, perfect for dev or small prod installs.

• Cons: Limited scalability. They share resources with the Scheduler, which can hurt performance under load.

🔹 Example: LocalExecutor

📦 Remote Executors

Remote executors delegate task execution to external worker nodes, offering better scalability.

⏱️ Queued/Batch Executors

Tasks are pushed into a queue and consumed by workers.

• Pros: Workers are decoupled from Scheduler; good for throughput and concurrency.

• Cons: You may run into the “noisy neighbor” problem if multiple tasks fight over the same worker resources.

🔹 Examples:

• CeleryExecutor

• BatchExecutor

• EdgeExecutor (still experimental 🚧)

🐳 Containerized Executors

Each task spins up its own container/pod, fully isolated.

• Pros: Full task isolation. Fine-grained control over runtime environment and resources. Great for heterogeneous workloads.

• Cons: Startup latency from launching containers. Overhead for short-lived tasks.

🔹 Examples:

• KubernetesExecutor

• EcsExecutor

⚠️ Tip: The executor runs within the Scheduler, not as a standalone process. You don’t need to start a separate service unless you're using something like Celery workers.

---

🤹 Multi-Executor Mode (Airflow 2.10+)

Airflow now supports multiple executors at once 🎉, I believe it’s a REAL game-changer for hybrid workloads. Let’s briefly see what I mean with that:

[core]
executor = LocalExecutor,CeleryExecutor

• The first listed executor becomes the default.

• Others can be targeted per task or DAG using the executor= argument.

You can even alias executors for full readability:

[core]
executor = LocalExecutor,ShortName:my.custom.module.ExecutorClass

Specifying Executors in Code

At the task level we have:

BashOperator(
    task_id="hello_world",
    bash_command="echo 'hi'",
    executor="LocalExecutor"
)

Or with TaskFlow:

@task(executor="CeleryExecutor")
def hello():
    print("hi from celery")

At the DAG level:

with DAG(
    dag_id="example",
    default_args={"executor": "KubernetesExecutor"}
) as dag:
    ...

🧠 Useful Note: Tasks store their executor in the metadata DB. Changes apply on the next DAG parse.

---

📊 Monitoring and Metrics

Executors might be the workhorses behind your DAGs….But without visibility into how they’re performing, it’s like flying completely blind.

Airflow exposes a rich set of executor-level metrics that give you deep insights into task scheduling, resource saturation, and bottlenecks. These are particularly useful when integrating with observability stacks like Prometheus + Grafana.

Here are a few key metrics you’ll want to track:

• executor.open_slots.<executor_name> – The number of available slots for task execution. If this frequently drops to zero, your executor may be under-provisioned.

• executor.queued_slots.<executor_name> – How many tasks are waiting to be picked up. High values here indicate a backlog—either due to insufficient workers or tasks being scheduled faster than they can run.

• executor.running_tasks.<executor_name> – The number of tasks actively executing. This tells you how "busy" your system is and is a good indicator of runtime load.

🔎 Why it matters: Tracking these metrics lets you answer critical operational questions:

• Is my executor underutilized or maxed out?

• Do I have enough workers?

• Why are tasks taking so long to start?

• Am I hitting limits in Kubernetes or Celery queues?

⚙️ Best Practices:

• Use Grafana dashboards to plot open vs. queued slots over time.

• Set alerts when queued_slots grows consistently or open_slots hits zero.

• Combine executor metrics with task-level metrics (like task duration or retries) for full-stack visibility.

When pipelines grow in scale or sensitivity, proactive monitoring becomes essential—not just to debug issues, but to keep SLAs and team trust intact ✅.

---

⚠️ Avoid Statically-Coded Hybrid Executors

Old-school hybrid executors like CeleryKubernetesExecutor abuse the queue field to simulate executor selection. These are now discouraged due to maintainability issues and broken semantics.

Use multi-executor mode instead. Cleaner, scalable, and natively supported 🧼.

✨ Build Your Own Executor

Airflow executors implement the BaseExecutor interface. If none of the existing options fit, build your own!

You might want to if:

• You use a custom job runner or compute backend

• You have an in-house task queue or environment

• You want to tightly integrate with a cloud-native platform

Key Methods to Implement

Method Purpose execute_async Launches a task (sync or enqueue) sync Updates task states on heartbeat start, end, terminate Lifecycle hooks get_event_buffer Reports task results to scheduler has_task Avoids duplicate runs send_callback Fires post-execution hooks

You can also implement:

• get_cli_commands() → for custom CLI support

• get_task_log() → to inject external logs into Airflow UI

🎯 Be sure to set compatibility attributes like supports_pickling, is_local, is_production, etc.

---

🛠️ CLI & Logging

If your executor needs additional command-line interface (CLI) commands, you can define a method that returns a custom command group. This allows users to interact with your executor using the Airflow CLI, making it more accessible and easier to manage.

In terms of logging, you can customize how your executor contributes to task logs by implementing a method that returns both executor-specific messages and the standard task output. This gives more transparency during execution and aids debugging, especially in distributed or custom environments.

Together, these capabilities make your executor more powerful and user-friendly, enhancing both operability and observability.

How to extend task logs:

def get_task_log(self, ti, try_number):
    return ["executor log message"], ["task log content"]

✅ Registering Your Executor

Once implemented, register it in the config:

[core]
executor = my_company.executors.MyExecutor

📚 Consider to read Modules Management and Public Interfaces for integration details and additional insights.

---

Auth Manager: Pluggable Authentication and Authorization 🔐

In Apache Airflow, the Auth Manager is the pluggable component responsible for user authentication (sign-in) and authorization (access control). Its modular design allows users to switch between different auth implementations based on their deployment needs.

⚙️ Configuration

Airflow supports only one auth manager at a time, configured via the auth_manager option in the [core] section of airflow.cfg.

$ airflow config get-value core auth_manager
airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager

📘 Note: For more on configuring Airflow, see Setting Configuration Options in the official docs.

---

🔄 Why Use a Pluggable Auth Manager?

Different Airflow setups serve different audiences—from solo users to enterprise-scale teams. The pluggable auth manager design enables administrators to choose or build a user management strategy that fits their unique requirements.

By default, Airflow uses the Flask AppBuilder (FAB) Auth Manager.

Available auth managers:

• SimpleAuthManager (included with Airflow)

• FABAuthManager (via the Flask AppBuilder provider)

• AWSAuthManager (via the AWS provider)

📌 Warning: Switching auth managers is a major operation. It changes the login/logout flow, and user data must be migrated manually.

---

🧩 Writing a Custom Auth Manager

All auth managers must implement the BaseAuthManager interface, which defines the public contract used by Airflow's core and UI components for auth-related operations.

Use cases for a custom manager:

• Integrate with an external identity provider (e.g., OAuth, SSO, LDAP).

• Use organization-specific user/group databases.

• Extend or replace existing permission logic.

🔑 Authentication Methods

Defined in BaseAuthManager:

• get_user() → Returns the currently signed-in user.

• get_url_login() → Returns the login redirect URL.

---

🔒 Authorization Methods

All follow a common pattern:

is_authorized_<resource>(
    method="GET" | "POST" | "PUT" | "DELETE" | "MENU",
    user=current_user,
    details=...  # depends on resource
)

Implemented methods include:

• is_authorized_configuration

• is_authorized_connection

• is_authorized_dag

• is_authorized_dataset

• is_authorized_pool

• is_authorized_variable

• is_authorized_view

• is_authorized_custom_view

Example:

auth_manager.is_authorized_dag(
    method="GET",
    access_entity=DagAccessEntity.Run,
    details=DagDetails(id="dag-1")
)

---

🪪 JWT Token Management

Auth managers must support JWT-based access for the Airflow REST API. This includes:

• Exposing a POST /auth/token endpoint that generates JWT tokens.

• Setting the token in a _token cookie for the Airflow UI to read and store.

from airflow.api_fastapi.auth.managers.base_auth_manager import COOKIE_NAME_JWT_TOKEN

response = RedirectResponse(url="/")
secure = conf.has_option("api", "ssl_cert")
response.set_cookie(COOKIE_NAME_JWT_TOKEN, token, secure=secure)
return response

⚠️ Do not set httponly=True on the cookie—Airflow UI must access it via JavaScript.

⚡ Performance Optimization (Optional)

Override these batch methods for performance:

• batch_is_authorized_dag

• batch_is_authorized_connection

• batch_is_authorized_pool

• batch_is_authorized_variable

Also, implement:

• get_authorized_dag_ids() → Returns all DAGs visible to the user.

---

💻 Extending the CLI

Auth managers can expose custom airflow CLI commands via get_cli_commands().

Here’s the actual code template to do that:

@staticmethod
def get_cli_commands() -> list[CLICommand]:
    sub_commands = [
        ActionCommand(
            name="command_name",
            help="Description of command",
            func=lazy_load_command("path.to.command"),
            args=(),
        ),
    ]
    return [
        GroupCommand(
            name="my_auth",
            help="Group of custom auth commands",
            subcommands=sub_commands,
        )
    ]

📘 Use unique command names to avoid namespace collisions.

---

🧩 Wrapping Up: Build for Scale, Not Just Functionality

As we’ve seen, scaling Airflow isn’t just about throwing more resources at your DAGs—it’s about understanding the infrastructure backbone that makes it all work.

Executors determine how your tasks run across environments, while Auth Managers define who gets access—and how securely. Together, they shape how Airflow performs, scales, and protects your workflows.

Whether you're running a single DAG on your laptop or orchestrating thousands of tasks across containers and clouds, these core components let you build a secure, scalable, and production-grade orchestration platform.

In the next chapter, we’ll take this even further—digging into secrets backends, connection management, and deployment architectures that turn your Airflow setup from dev to prod, the right way.

Until then, keep iterating, keep learning—and always, keep shipping ✈️