no batch job logs on Terrascope due to missing log.offset

[bossie]

Recent log documents lack the log.offset property which our query relies on.

Was fixed on CDSE by adding a random one at write time

This is not k8s so maybe it can simply be enabled: https://vector.dev/docs/reference/configuration/sources/file/#offset_key: TBC

[JeroenVerstraelen]

@bossie is this still an issue?

[bossie]

Yes, because the short-term fix was to simply roll back the changes that broke the logs (= move from filebeat to vector on Terrascope).

The actual fix is to get log.offset in the ES log documents but that's a devops issue.

[bossie]

@JeroenVerstraelen: @tcassaert and @pvbouwel are better suited to take on this issue.