Remove alertID when alert not found

[carlyjiang]

close

Could you share the solution in high level?

• If user manually deletes alert in wavefront, alert-manager will fall into infinite reconcile as it couldn't find missing alert.
  • If wavefront return 404, will remove alert-id from alertsconfig status, and then requeue the event.
  • In following retry cycle, alert-manager will find the alert doesn't have alert-id, will try to create new one.

Could you share the test results?

• Create testing case that server return 404
  • Create alert thru alert-manager.
  • Delete alert thru wavefront dashboard.
  • Restart alert-manager, alert-manager will complain 404 error on deleted alert. remove alert-id from alert-status.
  • After retry the event, alert-manager is able to re-create alert again.