assembly-authoring-tracked

Multi-Session Search Enabled Assembly Authoring

Investigate the ability to externalize PropertyDDS data in domain specific materialized views.

It is our experience that most of the data management use cases are typically I/O bound (however differently on the write and query side) and there is always a denormalization strategy which addresses the performance problems. In a CQRS like pattern the PropertyDDS / SharedPropertyTree (ST) would be the write model, but query cases, for efficiency reasons, would have to be served inevitably by MaterializedViews (MV) derived / projected from the data stored in collections of STs.

Now such MVs would reside outside Fluid and have very different SLAs. There are 3 design aspects we believe to be ubiquitous:

• Fluid Relay (FR) high availability (HA) should be sufficient and the only prerequisite to ensure convergence to (eventual) consistency across collections of STs and including multiple MVs. This needs to happen independent of network outages and even software bugs to be corrected, meaning extremely variable MV availability
• Efficient (reduced I/O, ideally not diff based) MV incremental updates. One possible pattern is that MVs would own/track internally the version of a synchronized ST state and be able to request the delta/change between its known/old ST state and current ST state. Even more, some data management use-cases need to capture multiple revisions of the data and it is very good possible that a particular MV will have to track ALL changes (rather than the squashed change).
• Highly scalable and resilient MV indexing (million of STs or partial ST fragments) by design. A (persistent) notification mechanism, accounting for the fact that MVs may be unavailable at the notification time. Only when notified the MVs will request the deltas from a particular ST

Getting Started

Needed dependencies

$ node --version
v16.13.1

$ npm --version
8.1.2

Using NVM for node version management

nvm use v16.13.1

Build

cd FluidPatterns
npm run clean
npm install --legacy-peer-deps
npm run build

Configure

Add elastic, elasticvue entries to /etc/hosts

127.0.0.1       localhost
127.0.1.1       elastic elasticvue

Run

Note: Starting order is relevant.

1. Start Tinylicious

npx tinylicious

2. Start Elastic Search Database

cd FluidPatterns/apps/assembly-search-agent
docker-compose up

3. Start Plexus Nameservice

cd FluidPatterns/apps/plexus-nameservice-agent
npm start

4. Start Assembly Indexing/Search Service

cd FluidPatterns/apps/assembly-search-agent
npm start

5. Start Assembly Authoring Application

cd FluidPatterns/apps/assembly-authoring-tracked
npm start

6. Start Assembly Search Application

cd FluidPatterns/apps/assembly-search
npm start

Requirement Notes

• Once initialized, any MV be updated incrementally (only). Refreshing full tree data is suboptimal and therefore not desired.
• The solution has to provide a way to preserve all changes associated with multiple ST containers for a long and deterministic time window.
• FR high-availability should be sufficient to guarantee the overall availability and zero data loss, including multiple materialized views
• Guaranteed eventual consistency. It is expected that MVs can go offline for longer (eg. deserialization bugs or upgrade procedures) or shorter (eg. kubernetes restarts) time frames. When returning online, they should catch-up efficiently.
• The granularity of the changes stored with the change log should be configurable, for instance to match custom data revising policies
• The emerging system should be fully auditable, for instance able to observe runtime operations, queries or results from production systems

Design Notes

• MVs are synchronized via an ST based change log
• Change log offsets associated with individual MVs (ie last read offset) is tracked using the ST infrastructure - the Tracker
• Queries and results are also regarded as messages transmitted over an ST based query and result log - Plexus

Legend

What	Symbol
ST / Shared Tree / PropertyDDS

Authoring

Search

Components

• Apps

  • Plexus Nameservice Agent
  • Assembly Indexing/Search Agent
  • Assembly Authoring
  • Assembly Search

• Libs

  • Assembly Util
  • Plexus Util
  • Fluid Util 2

Disclaimer: This is POC level software

Authoring Workflow

Edit above diagram

Search Workflow

Edit above diagram

Visual Paradigm - Session 1

Collaborative authoring session outcome is continuously persisted.

Visual Paradigm - Session n

Search Language

Search is possible when sessions are active (interactive search) and also long after their completion (analytics, forensics, etc. - assembly / component versioning is for sure the next important ingredient but out of scope for the current investigation). For simplicity and expressiveness our demo offers however a mini search language allowing:

Range filters for x, y, width, height, eg.:

width:100-200

Exact matches for fill, id, eg.:

id:rect1

Full text search for annotation field, eg.:

green beam

Combined syntax, eg.:

width:100-110 height:400-500 beam

Visual Paradigm - Search

Visual Paradigm - Search Result Detail 1

From the search results list the user can rehydrate the full session context (assembly) of the selected component. The resulted component can be visually highlighted (notice the dark frame below) in its assembly context.

Visual Paradigm - Search Result Detail 2

Just another detail page corresponding to a different result (third in the list above)

Disclaimer

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services. Use of these trademarks or logos must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.