Generate per-subprocess CPU and memory metrics

[rafidka]

Overview

In the scheduler container, we host multiple Airflow subprocesses, namely the scheduler, the DAG processor, and the triggerer. We need to generate per-subprocess metrics so we know when/if some processeses start others.

Acceptance Criteria

• For each sub-process, generate CPU and memory metrics.
• This should be as automated as possible, so any time we create multiple sub-processes in a single container, it is easy or no-effort to generate metrics for all sub-processes.

Additional Info

Context on this: #59 (comment)

[mateus-saraiva21]

Hi @shubham22, hope you don't mind me tagging you, I saw you suggested this change here. Is there a prediction on when something like this would be implemented?
My team is looking for a workaround to have better monitoring on Task-level and DAG-level metrics like CPU/Memory for debugging purposes in MWAA, and we wouldn't have to go down that route if there were any plans to pick this up in the near future.

Thank you in advance 🙏