Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ProtocolUnit log preservation to compute services and object store, with reference in state store; expose via AlchemiscaleClient #349

Open
dotsdl opened this issue Jan 8, 2025 · 0 comments

Comments

@dotsdl
Copy link
Member

dotsdl commented Jan 8, 2025

It's currently the case that ProtocolUnits for many Protocols emit logs using the standard gufe logging mechanism, using their own ProtocolUnit.logger handler. However, within alchemiscale we neither capture nor preserve these logs in execution.

We should empower alchemiscale users to examine the logs emitted by a Task's ProtocolDAGResults by allowing them to retrieve the logs via:

  • AlchemiscaleClient.get_result_logs(pdrr: ScopedKey, order='unit') : get logs for all ProtocolUnitResults in the ProtocolDAGResult, ordered by 'time' or by 'unit'
  • AlchemiscaleClient.get_unit_logs(purr: ScopedKey) : get the logs for a single ProtocolUnitResult
  • AlchemiscaleClient.get_result_units(pdrr: ScopedKey) : get ScopedKeys for all ProtocolUnitResultRefs given the ScopedKey of their ProtocolDAGResultRef

This will require that our compute services capture logs from each ProtocolUnit they execute, then upload those logs to the object store alongside the ProtocolDAGResult itself, creating a ProtocolUnitResultRef and a ProtocolUnitResultLogRef in the state store to enable retrieval. The ProtocolUnitResultRef for each ProtocolUnitResult in the ProtocolDAGResult will need to preserve dependency relationships to the other ProtocolUnitResultRefs in order to enable dependency-order reporting. The same ProtocolUnitResultRefs would likely be used for STDERR, STDOUT, and result file retrieval implemented under a similar pattern later.

Dependent somewhat on #348, since we assume ProtocolDAGResultRef ScopedKeys are now exposed to users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment