Updating to Madoc 2.0

[stephenwf]

With the stable 2.0 release comes some important changes to the services that comprise Madoc as a platform. All subsequent releases will be versioned with a log of changes that you can read when upgrading going forward.

Patch releases - For patch releases 2.0.x we will commit to only include fixes to existing functionality.
Minor releases - For minor releases 2.x we may introduce new features, project templates or page blocks. These will be turned off in the configuration by default. We may also introduce deprecations for features, pages or APIs that will be removed.
Major releases - For the next major release 3.0.0 we will remove any features previously deprecated and possibly change initial configurations to enable new features by default. Other breaking changes will be listed and a migration path outlined.

What has changed?

While it would be impossible to create a granular changelog, we can highlight some of the changes since Madoc 1.x and also since the prereleases that have been running up to this point.

We started work on this 2.0 milestone around October 2019, with some foundational goals:

• Reduce vendor lock-in - we aimed to utilise services like AWS without being locked into their proprietary APIs and services. Where an installation of Madoc - and our wider enrichment tools that integrate with madoc - could be deployed anywhere.
• Scalability - We aimed to use the same underlying database (Postgres) for each service, both to reduce infrastructure requirements, backup overheads and complexity - and to offer a single point of scalability of data. We also wanted a queue that could handle the volumes of data that Madoc would be expected to import and process.
• Exporting to standards - We wanted to maintain our compatibility with various community standards (IIIF, W3C) without it impacting or limiting how we store, edit and manage content or annotations. We opted to instead ensure we can export our own data models to these standards, instead of storing the models outlined in the standards directly.
• Sandboxing - we wanted to ensure when you created a new site, it was completely independent from other sites. We aimed to bake this in from the start.
• Extensibility - as a digital agency, it's important for us and our customers that Madoc can be extended and customised to meet specific project goals. We wanted to build this extensibility in from the start.

We have largely reached these goals now, while also building on the user-end features of the platform.

Service structure

Madoc is comprised of 7 services supported by a Postgres database, an nginx and a redis instance.

• Madoc - The IIIF APIs, Admin interface and User interface.
• Tasks - Stores, manages and queues tasks, which are used throughout madoc (crowdsourcing, importing, enriching)
• Capture Models - A store for customisable annotation data models.
• Config - A store for cascading configuration - allowing for granular configuration merged with inherited config.
• Search - A IIIF-based search API with full text and facetted search, built on-top of Postgres.
• Storage - A backend-agnostic file-management API, configurable to S3-compatible backends.
• Okra - A stateless service to convert various file formats (METS/hOCR/Capture models)

The following services have been removed since 1.x and our prereleases.

• Omeka (and MySQL)
• Sorting room
• Elucidate annotation server
• Elasticsearch
• Search indexer
• Tyk gateway (now just nginx)

Updating from 1.x

If you have a Madoc 1.x installation you will be able to migrate your users to a new 2.0 installation. However, existing projects and IIIF content imported will have to be re-imported. This can be done by first updating to madocs pre-release: EC2 + docker-compose examples running the migration helper - visible from the admin dashboard - and then following the instructions below to update to 2.0.

Contact us if you want help migrating existing annotations to the new capture model format. This will be largely dependant on the previous models set up in 1.x. Our recommendation would be to use a fresh 2.0 for new sites and projects - and migrate the users.

Updating from pre-release

If you have been running Madoc recently, it's likely that you have been running our pre-release of 2.0, with Omeka and MySQL still integrated. The primary part of the migration is the migration of sites, users and roles from the existing MySQL database to the new Postgres.

The first thing you need to do is unpin your madoc-ts image version. This will ensure you have the migration tool that we provide. When you log into the Admin interface, you should see the following message.

Click on "Migration guide" and read through the page. This will guide you through the changes you will need to make to your environment variables. You should prepare a new docker-compose.yaml that will replace your existing one.

You should also run the Database integrity check at the bottom of the page. This will ensure that all of the users in Omeka can be migrated successfully to Postgres. If this step fails, please contact us and forward the errors that you run into. You should be able to fix these manually from either Madoc admin interface or from the Omeka interface.

If you require SMTP for email support you can also check the extra environment variables are set up correctly before migrating.

Once you are ready to migrate you should verify your docker-compose.yaml. We have built some [https://github.com/digirati-co-uk/madoc-config/tree/feature/2.0.0](reference docker compose files here). If you are unsure of the changes you have made, please contact us and we can step through and verify.

You should take a backup of your previous docker-compose. Once you are ready you can run:

docker-compose pull

To pull down the new images (this will also verify the format of your compose). And then you can bring everything up:

docker-compose up -d

You should check that your site is loading as it had previously.

If you run into a problem, copy the contents of your backup docker-compose back and then run:

docker-compose up -d

Once you are happy, you can run:

docker-compose up -d  --remove-orphans

This will remove the extra containers.

You have upgraded to 2.0 🎉

Troubleshooting

These are some troubleshooting steps to take, but should also be read before migrating as an additional check.

Changing storage directory

Manifests and canvases imported prior to 2.0 store the location of the original manifest JSON on disk. This path is stored in the database and will not automatically update if you change the value of STORAGE_FILE_DIRECTORY or if you decide to change this in the future.

We may change this behaviour in the future to make it easier to migrate and switch the storage file directory.

Broken images

If you are seeing broken images in your Madoc installation it's likely the path to your storage directory.
In the madoc-ts container, you need to set:

      - STORAGE_FILE_DIRECTORY=/home/node/app/files

So that it matches the volume that you may have set up.

volumes:
  - /opt/data/files:/home/node/app/files:Z

Ensure that /opt/data/files (or whatever you have your volume set to) is the correct directory on your host, and that the target (/home/node/app/files) matches the STORAGE_FILE_DIRECTORY environment variable.

Frozen imports

If you are importing IIIF and no manifest or canvases are appearing you may be missing an environment variable on the tasks-api definition.

      - QUEUE_LIST=tasks-api,madoc-ts

This tells the Tasks API which queues (stored in redis) it should broadcast changes to.

Crashing / "something went wrong"

The most likely cause of this is the permissions on the volumes that have been mounted. Ensure that on your host the owners has been set to node:node or the permissions allow read/write. There are 3 volumes that may require this change:

• Madoc Files - madoc-ts.volumes - /home/node/app/files
• Madoc JWT responses - madoc-ts.volumes - /home/node/app/service-jwt-responses
• Storage API - storage-api.volumes - /home/node/app/files