MultihostCheckpoint counter error in CLU

[marcvanzee]

Original question by @salayatana66:

I got a CLU related error after training for a while. Essentially mulihost checkpont failed randomly with:

RuntimeError: Expected save_counter=0 to match latest_checkpoint=... Make sure the checkpoint is initialized via .restore_or_initialize() before it's stored and that no other process writes to the same checkpoint directory.

If I try to restart the job from the same checkpoint I get the same error.

[marcvanzee]

Answer by @andsteing:

It seems MultihostCheckpoint could not find overlapping checkpoint numbers in the tasks checkpoint directories. When the task tries to save a checkpoint, the discrepancy is discovered (save_counter is zero if no checkpoint is loaded, but latest_checkpoint is read by the tf checkpoint manager from the directory), and the process fails.

I'm not quite sure what has happened, but maybe you have had different processes that wrote to the same directory?