Skip to content

Latest commit

 

History

History
80 lines (72 loc) · 4.2 KB

20230718-minutes.md

File metadata and controls

80 lines (72 loc) · 4.2 KB

Metadata Templates Working Group Minutes

Tuesday, 2023/07/18

10:00 am ET

Attendees:

Terrell Russell, Kory Draughn, Alan King, Martin Golasowski (IT4I), Ingrid Barcena Roig (KU Leuven), Paul Borgermans (KU Leuven)

Minutes

Update/Discussion:

  • Goal: Draft paper for circulation among group

    • 6 years of what we've learned
    • Need to add Namespacing/Leuven to paper

  • Goal: develop/release microservices for attach/detach and validation

    • Can define these before next meeting - see if we can agree on naming, parameters/signatures
    • Perhaps also an export/collapse/rasterize msi
    • Reference counter / SQL to know if a schema is 'used'

  • Possible future - MT gets its own iRODS database table, first class object

    • GenQuery
    • Permissions
    • API for management
    • Reference external schema? Keep the text IN iRODS itself?
    • A much bigger amount of effort, probably not in the near term

  • Leuven - need to validate schemas against JSON-Schema eventually

    • Currently schema is used to present the form, rather than validate the data
    • Possibly separate / combine / complement two schemas?
      • A wrapper for the form/display information?
      • Data types are a bit richer than vanilla JSON-Schema at the moment
    • Active phase doesn't 'require' all the metadata, but then on export/repository, all the required metadata must be present
      • Allow the tool/display to aid/assist the user to 'build' / 'pull' existing metadata from others into this 'new' data association
      • Then there is less work during the 'export' step, as most of the earlier metadata is already valid/good.
    • JSON-Schema also would be useful for export of metadata itself
      • Want to compare/validate against a foreign reference / schema
      • Possible mapping to other names, but same ideas
    • Inheritance from 'above', where does this get defined? What happens when it updates 'up there'? Is it recursive 'all the way down'?
      • Maybe only on 'export'. Checkbox per metadata item?
      • Since it is only 'applied' to the parent, not ALL the children
        • Otherwise, too much duplicate data in the database
      • When exported.. Does it include the reference to the parent schema? Or just get combined/included/collapsed to the exported data. Just flat on export.
      • Reference counter to prevent deletion/loss of connected/referenced schemas upon 'removal' - this is hard. Perhaps should be in the app layer... or perhaps the schema machinery in the server should do this natively. Probably best to 'just be SQL', rather than a cached/outofdate/counter. Learned this lesson with the object_count and rmresc.
    • Composite objects - should these be individual AVUs, or a unit of multiple… multiple isn't as searchable
      • Used AVU unit as the glue/composite information

  • IT4I - currently forcing users to only use a single schema, simple for now

    • exporting/syncing with elasticsearch for external apps/search, from multiple zones

  • Next Meeting

    • Aug 2023

  • After meeting (Terrell with Kory)...

    • Paper
      • 6y (yes, no, scope, examples)
      • namespacing
      • inheritance
      • msi (3-4) (just AVUs)
    • Later
      • New MT table(s)
        • Template definitions
        • Join table for associations with data_id/coll_id
    • Microservices
      • Attach (type, schema, object_id)
        • Initially, type will just be 'url'
        • Could later be 'irods_schema' and store the id from the new table
        • Or 'form' for ManGO wrapper/form information
      • Detach (type, schema, object_id)
      • Validate (object_id, recursive)
        • Run gather (below) to build the effective json schema
        • Get and build json payload with all current AVUs
        • Run payload and schema through validator
        • Return result (OK or failure/explanation)
      • Export/Collapse/Rasterize/Gather/Dump (object_id, recursive)
        • Find all associated schemas and construct effective schema
        • Recursive would check/gather all parents up to root
    • JSON Schema Only
      • Could have ManGO form information in a different schema 'type'
        • Could reference 'real' JSON Schema with anchors/names
        • Apps would compose when needed (to draw a form)
      • Gather/Validate would ignore the form schemas