RFC: adopt monorepo architecture to support more packages

[HipsterBrown]

TL;DR Use pnpm workspaces to enable a broader ecosystem of xs-dev packages while adopting some complexity for managing the core feature set.

---

Background

The project architecture of xs-dev has shifted and matured to meet the needs of the features being developed and maintained. It started out as a single zx script for automating the set up of embedded tooling for Moddable projects. To enable more commands and functionality, like scaffolding new projects, scanning for devices, and running example programs, xs-dev adopted Gluegun as a relatively lightweight toolkit for CLI applications using TypeScript with a lot of useful out-of-the-box utilities. This reduced the amount of works required to create a custom toolkit of various dependencies and undocumented conventions. While this has been great for productivity for the last couple of years, Gluegun is no longer actively maintained by the original creators and has some user experience limitations that may not be addressed by the community, i.e. per command help text, command argument validation, or stricter typing around the various utilities from the "toolbox". At the same time, I have envisioned a larger ecosystem of packages to support embedded JS developers who like to interact with tools outside of the command line, like through VSCode or desktop GUIs.

Proposal

Before releasing 1.0 of xs-dev, the repo should adopt a monorepo architecture made up of the documentation site and two packages: xs-dev CLI and @xs-dev/core library. The monorepo will be structured as a pnpm workspace, as demonstrated by popular projects like Astro and Vite. I've also set up this style of monorepo in another project.

Migration

Because this repo is already using pnpm has the package manager, the migration should be implemented with each of the following steps:

• create the required pnpm-workspace.yaml file
• create a packages directory and move the required CLI package files under packages/xs-dev
• add a package.json to the docs/ and packages/xs-dev directories with the necessary info for each project
• update the root package.json with the minimal required information
• update .github/workflows/ actions to use new file paths for the monorepo structure, especially the automated releases

Once this initial migration is complete, the next steps will be deconstructing the the CLI package to move the core feature logic (most of the src/toolbox modules) to a new packages/core library package to be consumed by the CLI package, which will contain the concerns of I/O from the command line.

Downsides

This will add complexity to new contributors who have never worked in a monorepo and may need more help to get started.

Follow Up

With this new architecture in place, the CLI package can experiment with a new toolkit to improve the performance and user experience without affecting the core logic. LIkewise, the core logic could be consumed by new packages for VSCode, desktop applications, or server-side utilities.

[phoddie]

Before releasing 1.0 of xs-dev...

Happy to see this coming into focus. ;)

The overall direction makes sense. I'm not familiar with pnpm but it doesn't appear to be an insurmountable obstacle for future contributions.

It isn't clear to me what happens to Gluegun -- does it continue to be used or does it go away as part of the repo restructuring?

[HipsterBrown]

It isn't clear to me what happens to Gluegun -- does it continue to be used or does it go away as part of the repo restructuring?

That may have been too much background for this proposal since it will not be affect by the monorepo implementation. I will have a follow up RFC for replacing Gluegun once the core lib + CLI packages have been established. In general, I am considering a few replacement packages such as oclif, yargs, or clipanion; although none of these replace the toolbox of utilities from Gluegun, so they may need to be vendored in some way as part of the migration.

Happy to see this coming into focus. ;)

There is light at the end of the tunnel. 😄

[phoddie]

I was reading more into your notes than was there. I get it now, thanks. I have no suggestions / opinions on what might be a suitable replacement for Gluegun. As someone who needs to learn the runtime to contribute, Gluegun was just one additional thing beyond Node (and TypeScript...) to deal with. Keeping the number of API suites used could help.

It seems like a chore to convert, but the motivation seems good. I suppose most of commands aren't that much code, so it should be manageable.

There is light at the end of the tunnel. 😄

Ever the optimist!

[ScreamZ]

I'm very interested in helping in development of moddable SDK. This is an incredible technology, close to what React Native do for mobile development.

But currently developing on it requires a lot of hacks and things. xs-dev bring first blocks for easier development but still have issues on some install (hello macos and homebrew).

Also this is really complicated to explore the API, and the notion of module etc get tricky.
Also hard to start with typescript.

I would be pleased to help documenting and help on Javascript (or even better Typescript) part of this project.

[HipsterBrown]

Thanks for sticking with the experience despite the speed bumps @ScreamZ 😅

Any issue labeled as "good first issue" is a great place to get started and should contain enough information about how to contribute with a fix. Otherwise, feel free to start poking around the codebase and creating pull requests. 😄