Hi guys,

New to airflow. I have tried for hours to figure this out and think I have come to the correct process up until Celery Executor. Please see below

When running Airflow in cluster mode, we have workers on various machines that connect back to the scheduler. Airflow uses RabbitMQ to get the messages in a Queue from which the workers can retrieve the message. I just don’t completely understand why we need celery? Does it act as a load balancer?

Any help would be really appreciated.


Source link

No tags for this post.


Please enter your comment!
Please enter your name here