refactor: restructure code into a package with modules

[rmoreas]

This refactoring:

1. Splits code into logical modules
2. Moves global variables into appropriate modules
3. Creates proper package structure
4. Improves code organization and maintainability
5. Makes testing easier by separating concerns

Here is a summary of the specific responsibilities for each of the modules in the mango_ingest package:

1. mango_ingest.cli
   • Handles command-line interface (CLI) interactions.
   • Parses command-line arguments and options.
   • Initiates the main workflow based on user input.
2. mango_ingest.irods
   • Manages interactions with the iRODS (Integrated Rule-Oriented Data System).
   • Handles file synchronization, uploading, and metadata management within iRODS.
   • Implements logic for filtering and verifying files before transfer.
3. mango_ingest.metadata
   • Manages metadata operations related to the ingest process.
   • Handles extraction, transformation, and storage of metadata for ingested files.
4. mango_ingest.watchdog
   • Monitors file system changes and triggers synchronization processes.
   • Uses watchdog observers to detect and respond to file events.
   • Ensures continuous synchronization of files based on specified criteria.
5. mango_ingest.utils.config
   • Manages configuration settings for the mango_ingest package.
   • Loads and validates configuration files or environment variables.
   • Provides configuration data to other modules.
6. mango_ingest.utils.logging
   • Configures and manages logging for the mango_ingest package.
   • Defines custom logging levels and handlers.
   • Ensures consistent and informative logging throughout the package.
7. mango_ingest.utils.results
   • Manages the results of the ingest process.
   • Tracks and reports the status of file transfers and operations.
   • Generates reports and summaries of the ingest activities.

The functionality remains the same but is now better organized and more maintainable.

[rmoreas]

@paulborgermans : could you have a look a this? Everything should work as before.

I would also like to add a new feature to this PR: a "iinit" subcommand to generate the irods environment file with simple user input, without the need to have icommands installed or run another Python script. I've it ready, but maybe better to do add in another PR?

[paulborgermans]

@paulborgermans : could you have a look a this? Everything should work as before.

I would also like to add a new feature to this PR: a "iinit" subcommand to generate the irods environment file with simple user input, without the need to have icommands installed or run another Python script. I've it ready, but maybe better to do add in another PR?

Hi Ronny,

Ok, yes this is more or less along the lines I wanted to refactor it later on myself too, thanks for this!

There are a few more changes and new features + refactoring in the development branch and one of the additional refactorings is to split the filtering in its own module, which gained a more general "plugin" mechanism. I also upgraded to using a toml file for installing and eventually packaging.

On iinit, there are upcoming changes in irds prc which will make this even more seamless. I also have an uncomitted iinit like command based on what we developed for other projects at KU Leuven. So maybe first show it in a new issue what you have?

I will continue in another branch based on the current development and incorporate most changes you propose here

And we can this discuss all this on site

Thanks again!

[rmoreas]

Oh, I didn't notice there is a development branch. This will be difficult one to merge. Wouldn't it be better to merge your change from develop into this branch? I will be manual work anyway, but there are less changes to merge from develop into here, than the other way around. I can help with that. What do you think?

[rmoreas]

Pushed to branch refactor/split_modules_ronny