Airflow Worker - Connection Broken: Incompleteread(0 Bytes Read)

Using Airflow worker and webserver/scheduler as a Docker images running on Kubernetes Engine on EC2 We have a task which has KubernetesPodOperator which is resource intensive and r

Solution 1:

so, better late than never

it is because of known bug in KubernetesPodOperator. to avoid this behavior you have to set operators get_logs parameter to False. default value is True.

details here https://issues.apache.org/jira/browse/AIRFLOW-3534https://issues.apache.org/jira/browse/AIRFLOW-5571

Solution 2:

Well, it means that thge program logic didn't get the data it expected to receive from some socket. This can mean anything, from an intermittent network problem to the data simply not arriving in time and the logic not programmed to wait for it. IF the task is automatically retried, you may not even need to worry about intermittent problems.

If you wish to diagnose this further, you need to gather some diagnostic information. Problems are always diagnosed by the same scenario:

1. Identify the exact place in the program when the problem manifests itself.
2. Examine the program's state at that moment and find out which of the values is wrong.
3. Trace the erroneous value back to its origin.

The first can be identified by the stack trace and/or searching the codebase for relevant logic. The second -- debugging or debug printing. The 3rd is usually done by rerunning the program with breakpoints set earlier, at the step in the logic that produces the erroneous value; in your case, you can only do that very slowly, by waiting for the problem to happen again, so you are forced to do educated guesses from the codebase.