Trigger Events on Record Import

[skamensky]

Hey, I saw that there's some interest in triggers for import events.
This is the related forum.

I was curious if there's any interest on the corteza side of outside contribution, and if so, if this is viewed as a reasonable first issue.

After a few minutes of browsing the source code my first stab would be to add a couple events here that correspond to the ImportInit,ImportInit, and ImportRun functions (or make a new grouping in this file)

https://github.com/cortezaproject/corteza-server/blob/57721291b270d4ed9b9e625d7a17fca2cc6c27eb/system/service/event/events.yaml#L101-L111

And then call the events using Service.eventbus.WaitFor like we do here
https://github.com/cortezaproject/corteza-server/blob/57721291b270d4ed9b9e625d7a17fca2cc6c27eb/system/service/application.go#L158-L161

The relevant functions I found after a glance were these:

https://github.com/cortezaproject/corteza-server/blob/57721291b270d4ed9b9e625d7a17fca2cc6c27eb/compose/rest/record.go#L319

https://github.com/cortezaproject/corteza-server/blob/57721291b270d4ed9b9e625d7a17fca2cc6c27eb/compose/rest/record.go#L428

https://github.com/cortezaproject/corteza-server/blob/57721291b270d4ed9b9e625d7a17fca2cc6c27eb/compose/rest/record.go#L302

I'm sure there's more involved. Please let me know whether or not there's any interest and if so if this is the right direction.

[darh]

What problem would you be solving with handlers for these events? Cancelation, notification, mutation?
How often would be onProgress fired?

I do not see a problem with implementing this. Please not though, this importer is for records only. And we desperately need ability to import any type of resource. Record importer already uses pkg/envoy underneath that can do much more than just importing resources of compose-record type from CSV or JSON files.

[skamensky]

What problem would you be solving with handlers for these events? Cancelation, notification, mutation?

Notification+Mutation. It's essentially a workaround to the fact that standard record events aren't fired on import.
In the current implementation if I wanted to compute the value of a record field using workflows I'd need to use a time-based workflow, calculate which records haven't had their values populated, and operate on those records. What we have today is essentially polling for imported records. I want to support event-based record mutation the same way the rest of the system works.

How often would be onProgress fired?

We can set a sane default like the one used in web-compose for polling

https://github.com/cortezaproject/corteza-webapp-compose/blob/9af28f269ae4752c036522273ec9c1af6a14efd2/src/components/Public/Record/Importer/Progress.vue#L102

And expose a config variable to allow for power users to set a different value.

I do not see a problem with implementing this. Please not though, this importer is for records only.

Sorry, I don't understand, you do or don't want this implimented? Or did you mean don't add via the importer interface, but rather through the envoy interface in order to plan for the future?

[skamensky]

@darh ,

Any thoughts on this?

[darh]

Sure, go for it.