11,986 questions
0
votes
0
answers
58
views
MWAA Airflow task_instance_mutation_hook not working
I have added a airflow_local_setting.py file to my mwaa s3 bucket at the root of dags folder. I have a dag_policy defined which is being picked up and works as expected.
In the same file I have added ...
- 13
-3
votes
0
answers
37
views
Why KubernetesPodOperator with do_xcom_push=True does not push container logs to XCom?
I'm trying to run a task using KubernetesPodOperator in Airflow 3.0.3 and retrieve the logs generated by my container via XCom.
Here is a simplified DAG:
from airflow import DAG
from airflow.providers....
0
votes
1
answer
89
views
Unable to use Airflow variable with function in Jinja template
I'm trying to pass Airflow logical_date to dbt model so that I can use it in the model (sql). I'm using Airflow 2.11.0. I'm doing below, but the dag couldn't get constructed, with an error says:
'...
- 367
0
votes
0
answers
159
views
How can I dynamically refresh a JWT token in Airflow with Microsoft SSO using RSA?
I’ve configured Microsoft SSO with Apache Airflow using RSA-based authentication. The setup involves uploading the public key to the Azure App Registration, while Airflow holds the private key to ...
- 19
-1
votes
1
answer
25
views
How to correctly use retry_delay or retry_delay_sec in Airflow YAML DAG?
I'm currently working with Apache Airflow and trying to configure retries for my DAG tasks using the retry_delay (or retry_delay_sec) parameter in a YAML-based DAG definition.
However, I'm running ...
1
vote
1
answer
132
views
DeprecationWarning 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead
I'm working on AWS EC2. I created a virtual environment and installed all the necessary dependencies. I connected to it via VSCode, and when I try to start the server using the airflow standalone ...
- 23
0
votes
0
answers
33
views
Why do my BigQueryInsertJobOperator tasks in Cloud Composer show long queued durations before running?
I’m using Cloud Composer (Airflow) and have two BigQuery operators like this:
run_aggregation = BigQueryInsertJobOperator(
task_id='aggregation_task',
configuration={
"query":...
0
votes
0
answers
20
views
apache airflow client connections are failing to connect to pgbouncer sporadically
We are facing occassional errors while airflow workers and scheduler connecting to pgbouncer. We have tried using external postgres server and bitnami postgres available from airflow helm chart, in ...
- 125
0
votes
0
answers
67
views
Unable to establish a connection with Azure Blob using Apache Airflow
I have created a DAG in Airflow that will detect a file in Azure blob storage. This is a sensor that will detect the existence of a blob in Azure blob storage.
But it is failing repeatedly with the ...
0
votes
1
answer
34
views
Apache airflow 3.0.6 oralce connection type is not found
I am using Apache airflow 3.0.6. I am using docker desktop in windows pc. I have oracle connection but it shows below error:
Timestamp: 2025-10-09, 12:05:43
Traceback (most recent call last):
File &...
- 5,095
0
votes
0
answers
22
views
Airflow tasks for data without timestamps
I would like to use Airflow for some ETL operations, where the source data does not have (indexed) timestamp columns. The source data is a database table where new records of events are appended ...
- 5,766
1
vote
1
answer
87
views
How to migrate metadata from Airflow 2.0.x to 3.0.x?
How to migrate the metadata from Airflow version 2.5.1 to Airflow version 3.0.x in MWAA (Managed Workflows for Apache Airflow)? Direct database access has been removed in Airflow 3.0.x. I am not able ...
0
votes
0
answers
56
views
Tasks stuck in queued
I’m running into an issue with Airflow 3.1 using CeleryExecutor inside Docker Compose, and I can’t figure out what’s wrong.
This is for a small project I’m building for my business. The idea is to use ...
1
vote
1
answer
106
views
How can I run airflow db clean command using BashOperator with Airlfow 3+
I run airflow using official docker compose file.
I would like to run a simple command with Airflow BashOperator to clean my logs:
clean_database = BashOperator(
task_id="clean-...
- 11.6k
1
vote
0
answers
52
views
Airflow 3.1. + CeleryExecutor in Docker Compose: DAGs stuck in `queued`, tasks never sent to workers
I’m trying to run Apache Airflow 3.1. with the CeleryExecutor using Docker Compose (together with Redis + Postgres).
My problem is, when I trigger a DAG (directly over the command line or how it's ...
- 1,661