Hang with Ohio small animals subset dataset

[khsa1]

I'm running into an issue with my subset of ohio small animals dataset on the k80 gpus. It looks like the terminating event is somehow causing one of the ImageStoredEvents to be lost, and the oracle plugin is waiting for a decision on that image before it can exit.

Here is the log of the oracle plugin:

2025-01-06 17:00:27,205 INFO: Got a message from the event socket - Oracle monitor check [in /oracle_plugin.py:218]
2025-01-06 17:00:27,205 INFO: Image received 27b59b57-1b60-5e52-9dc1-30007d5cd71d 2025-01-06T17:00:27.204806854+00:00 [in /oracle_plugin.py:224]
2025-01-06 17:00:27,206 INFO: Fetching 27b59b57-1b60-5e52-9dc1-30007d5cd71d from /output/uuid_image_mapping.json [in /oracle_plugin.py:117]
2025-01-06 17:00:27,214 INFO: <ctevents.gen_events.ImageReceivedEvent.ImageReceivedEvent object at 0x7ff071add370> [in /oracle_plugin.py:283]
top of get_next_msg; making request for next message
request made; waiting for reply
got reply, returning data
2025-01-06 17:00:27,215 INFO: Got a message from the event socket - Oracle monitor check [in /oracle_plugin.py:218]
2025-01-06 17:00:27,216 INFO: Received Terminating signal from image generating plugin [in /oracle_plugin.py:254]
2025-01-06 17:00:27,216 INFO: Total images processed: 50 [in /oracle_plugin.py:260]
2025-01-06 17:00:27,217 INFO: Oracle has processed: 48 out of 50 [in /oracle_plugin.py:265]
2025-01-06 17:00:27,218 INFO: <ctevents.gen_events.PluginTerminatingEvent.PluginTerminatingEvent object at 0x7ff071add2e0> [in /oracle_plugin.py:283]
top of get_next_msg; making request for next message
request made; waiting for reply
got reply, returning data
2025-01-06 17:00:27,817 INFO: Got a message from the event socket - Oracle monitor check [in /oracle_plugin.py:218]
2025-01-06 17:00:27,818 INFO: Inside scoring 27b59b57-1b60-5e52-9dc1-30007d5cd71d [{'label': 'human', 'probability': 0.011800000444054604}, {'label': 'animal', 'probability': 0.9290000200271606}] 2025-01-06T17:00:27.816305 [in /oracle_plugin.py:235]
2025-01-06 17:00:27,827 INFO: Oracle has processed: 48 out of 50 [in /oracle_plugin.py:265]
2025-01-06 17:00:27,827 INFO: <ctevents.gen_events.ImageScoredEvent.ImageScoredEvent object at 0x7ff071acd5b0> [in /oracle_plugin.py:283]
top of get_next_msg; making request for next message
request made; waiting for reply
got reply, returning data
2025-01-06 17:00:27,829 INFO: Got a message from the event socket - Oracle monitor check [in /oracle_plugin.py:218]
2025-01-06 17:00:27,829 INFO: Image stored 27b59b57-1b60-5e52-9dc1-30007d5cd71d 2025-01-06T17:00:27.817595744+00:00 Save [in /oracle_plugin.py:242]
2025-01-06 17:00:27,870 INFO: Oracle has processed: 49 out of 50 [in /oracle_plugin.py:265]
2025-01-06 17:00:27,870 INFO: <ctevents.gen_events.ImageStoredEvent.ImageStoredEvent object at 0x7ff071add370> [in /oracle_plugin.py:283]
top of get_next_msg; making request for next message
request made; waiting for reply

[khsa1]

Fixed by #48.