NC | Online upgrade improvements

[romayalon]

Environment info

• NooBaa Version: 5.18
• Platform: NC

Missing small functionalities in NC online upgrade

• 1. Add a check for not allowing multiple ongoing config dir upgrades - Fixed by #NC | Online Upgrade | Fail config directory upgrade if upgrade is already in progress #8613
• 2. Although the user is not supposed to do it - Parallel config dir upgrades can have a race condition, this can be improved by using lock - lockfd.fcntllock().
• 3. Add to system json a property that says that the config dir per hostname. - Will be fixed by NC | Online Upgrade | add host config dir version to system.json | Health - add blocked hosts check #8627.
• 4. Based on bullet 2 - add to health monitoring a check that shows which hosts are blocked for updates to the config directory. - Will be fixed by NC | Online Upgrade | add host config dir version to system.json | Health - add blocked hosts check #8627
• 5. One of the config directory upgrade tests creates a directory inside nc_upgrade_scripts and not deleting it appropriately - Will be fixed by NC | Fix test_nc_upgrade_manager cleanup #8636.
• 6. create ENUM based on CONFIG_DIR_UNLOCKED and CONFIG_DIR_LOCKED, see NC | Online Upgrade | Health CLI update config directory and upgrade checks #8532 (comment) - Fixed by NC | Online Upgrade | Short refactoring #8615
• 7. Remove the condition for failing config dir upgrade if the --expected_hosts list does not match the hosts registered in system.json. This change is needed because NooBaa is not cluster aware and we can not take system.json as the source of truth of the current hosts status in the cluster, therefore we will rely on CES to send us the expected hosts list that should contain only hosts the upgraded their noobaa RPM. - Fixed by #NC | Online Upgrade | Replace failure if system.json contains hosts that do not exist in --expected_hosts with a warning #8612
• 8. In configFS - Replace this.config_dir_version ='1.0.0' with a constant so it can't be changed. Fixed by NC | Online Upgrade | Short refactoring #8615
• 9. remove old errors from system.json after upgrade has finished successfully? (finish_update()) currently, we don't and this is the behavior seen in containerized upgrade process as well. - should be fixed by NC | Online upgrade improvements #8638.
• 10. add to the backup directory name the old version - will be fixed by NC | Online upgrade improvements #8638
• 11. Add tests to test_config_dir_restructure.test.j - Fixed by NC | Online Upgrade | Tests | Config directory restructure upgrade script unit tests #8654.
• 12. fix container terms in upgrade_utils.should_upgrade() - Will be fixed by Upgrade utils | change variables, errors and log prints to be general to both Containerized and NC environments #8696

Open questions -

1. Do the RPM changes are still needed?
2. Should we remove old accounts/ backup directory that was created during the restructure after finishing? currently I left it as it is

Expected behavior

Steps to reproduce

More information - Screenshots / Logs / Other output