You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I may want to use taskfiles as an ETL pipeline API; it should be local filesystem aware, so it knows where to write out outputs if it's a "local" processing job (e.g. data extraction with parsing, jq, yq), however in some cases it makes sense to collect only the output directly (e.g. pipe workflow). Using CombinedOutput doesn't give us access to the stdout separately;
May want to reuse task-ui for this but adjust to capture the outputs separately, so i can distribute a job runner similar to a github matrix job, but sort of in a structured ETL framework.
Filing this in the idea pile, likely going to skip the distributed nature of some tasks 🗡️ ; but i'll need to fan out pretty soon as a single threaded job runner won't work for the use case.
The text was updated successfully, but these errors were encountered:
I may want to use taskfiles as an ETL pipeline API; it should be local filesystem aware, so it knows where to write out outputs if it's a "local" processing job (e.g. data extraction with parsing, jq, yq), however in some cases it makes sense to collect only the output directly (e.g. pipe workflow). Using CombinedOutput doesn't give us access to the stdout separately;
May want to reuse task-ui for this but adjust to capture the outputs separately, so i can distribute a job runner similar to a github matrix job, but sort of in a structured ETL framework.
Filing this in the idea pile, likely going to skip the distributed nature of some tasks 🗡️ ; but i'll need to fan out pretty soon as a single threaded job runner won't work for the use case.
The text was updated successfully, but these errors were encountered: