Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Porting signal processing to the task monitor condition for 2.10.1 #151

Merged
merged 1 commit into from
Oct 3, 2024
Merged

Porting signal processing to the task monitor condition for 2.10.1 #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions images/airflow/2.10.1/BillOfMaterials/2.10.1-dev/Python-Packages-BOM.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6162,10 +6162,10 @@ Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3-1.35.7.dist-info/NOTICE

boto3-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/mypy_boto3_builder
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2024 Vlad Emelianov
Expand Down Expand Up @@ -6437,10 +6437,10 @@ one at http://mozilla.org/MPL/2.0/.
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore-1.35.7.dist-info/NOTICE

botocore-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/botocore-stubs
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2022 Vlad Emelianov
Expand Down
8 changes: 4 additions & 4 deletions images/airflow/2.10.1/BillOfMaterials/2.10.1-explorer-dev/Python-Packages-BOM.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6162,10 +6162,10 @@ Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3-1.35.7.dist-info/NOTICE

boto3-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/mypy_boto3_builder
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2024 Vlad Emelianov
Expand Down Expand Up @@ -6437,10 +6437,10 @@ one at http://mozilla.org/MPL/2.0/.
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore-1.35.7.dist-info/NOTICE

botocore-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/botocore-stubs
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2022 Vlad Emelianov
Expand Down
8 changes: 4 additions & 4 deletions images/airflow/2.10.1/BillOfMaterials/2.10.1-explorer-privileged-dev/Python-Packages-BOM.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6162,10 +6162,10 @@ Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3-1.35.7.dist-info/NOTICE

boto3-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/mypy_boto3_builder
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2024 Vlad Emelianov
Expand Down Expand Up @@ -6437,10 +6437,10 @@ one at http://mozilla.org/MPL/2.0/.
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore-1.35.7.dist-info/NOTICE

botocore-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/botocore-stubs
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2022 Vlad Emelianov
Expand Down
8 changes: 4 additions & 4 deletions images/airflow/2.10.1/BillOfMaterials/2.10.1-explorer-privileged/Python-Packages-BOM.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6162,10 +6162,10 @@ Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3-1.35.7.dist-info/NOTICE

boto3-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/mypy_boto3_builder
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2024 Vlad Emelianov
Expand Down Expand Up @@ -6437,10 +6437,10 @@ one at http://mozilla.org/MPL/2.0/.
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore-1.35.7.dist-info/NOTICE

botocore-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/botocore-stubs
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2022 Vlad Emelianov
Expand Down
8 changes: 4 additions & 4 deletions images/airflow/2.10.1/BillOfMaterials/2.10.1-explorer/Python-Packages-BOM.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6162,10 +6162,10 @@ Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3-1.35.7.dist-info/NOTICE

boto3-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/mypy_boto3_builder
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2024 Vlad Emelianov
Expand Down Expand Up @@ -6437,10 +6437,10 @@ one at http://mozilla.org/MPL/2.0/.
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore-1.35.7.dist-info/NOTICE

botocore-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/botocore-stubs
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2022 Vlad Emelianov
Expand Down
8 changes: 4 additions & 4 deletions images/airflow/2.10.1/BillOfMaterials/2.10.1/Python-Packages-BOM.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6162,10 +6162,10 @@ Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3-1.35.7.dist-info/NOTICE

boto3-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/mypy_boto3_builder
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/boto3_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2024 Vlad Emelianov
Expand Down Expand Up @@ -6437,10 +6437,10 @@ one at http://mozilla.org/MPL/2.0/.
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore-1.35.7.dist-info/NOTICE

botocore-stubs
1.35.21
1.35.29
MIT License
https://github.com/youtype/botocore-stubs
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.21.dist-info/LICENSE
/usr/local/airflow/.local/lib/python3.11/site-packages/botocore_stubs-1.35.29.dist-info/LICENSE
MIT License

Copyright (c) 2022 Vlad Emelianov
Expand Down
3 changes: 3 additions & 0 deletions images/airflow/2.10.1/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ x-airflow-common: &airflow-common
MWAA__LOGGING__AIRFLOW_WORKER_LOGS_ENABLED: ${MWAA__LOGGING__AIRFLOW_WORKER_LOGS_ENABLED}
MWAA__LOGGING__AIRFLOW_WORKER_LOG_GROUP_ARN: ${MWAA__LOGGING__AIRFLOW_WORKER_LOG_GROUP_ARN}
MWAA__LOGGING__AIRFLOW_WORKER_LOG_LEVEL: ${MWAA__LOGGING__AIRFLOW_WORKER_LOG_LEVEL}
MWAA__CORE__TASK_MONITORING_ENABLED: ${MWAA__CORE__TASK_MONITORING_ENABLED}
MWAA__CORE__TERMINATE_IF_IDLE: ${MWAA__CORE__TERMINATE_IF_IDLE}
MWAA__CORE__MWAA_SIGNAL_HANDLING_ENABLED: ${MWAA__CORE__MWAA_SIGNAL_HANDLING_ENABLED}

volumes:
- ./dags:/usr/local/airflow/dags
Expand Down
4 changes: 2 additions & 2 deletions images/airflow/2.10.1/python/mwaa/celery/sqs_broker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,8 +400,8 @@ def __init__(self, *args, **kwargs):

self.hub = kwargs.get("hub") or get_event_loop()

# Dynamic workers have the MWAA__CORE__TASK_MONITORING_ENABLED set to 'true'.
# This will be used to determine if idle worker checks are to be enabled.
# MWAA__CORE__TASK_MONITORING_ENABLED is set to 'true' for workers where we want to monitor count of tasks currently getting
# executed on the worker. This will be used to determine if idle worker checks are to be enabled.
self.idle_worker_monitoring_enabled = (
os.environ.get("MWAA__CORE__TASK_MONITORING_ENABLED", "false") == "true"
)
Expand Down
Loading
Loading