Original installation channel must be visible, updates listed accordingly

[snomos]

Taskcluster is working for all lang-xxx repos, and signing and notarization is working again. That is, we have now a bunch of new nightlies available to us. Except not:

Expected: available for installation

Divvun Manager 2.0.1.

[Eijebong]

Doesn't that show that it's already been updated?

[snomos]

Yes, that would be the expected behavior. But the installed speller is this:

That is, a more than one year old speller (I assume the latest stable release).

Cf also #10 .

[Eijebong]

I wonder if the manager thinks it's the same version because 4.3.0 matches

[snomos]

I think that I was fooled by having installed the original package using the "stable" channel, which makes any newer releases in the nightly channel unavailable.

I will reformulate the bug as follows:

Original installation channel must be visible, updates listed accordingly.

[snomos]

More on this:

• as updates are installed based on the active channel when a package was first installed, it must be possible to see which channel is active for a given package
• listed updates should be listed based on the active channel (updates in other channels could be listed as well, in gray or something, to make it clear that they exist, but are not available via the active channel)

[snomos]

@dylanhand Now that DM is building as it should again, and there is no further direct action to be taken on #41 (except adding logging), it would be good to take a look at this issue.

[dylanhand]

@snomos will do. I'll see if there are obvious places to add logging and then look into how we can indicate from which channel a package is installed.

[snomos]

I think it would be useful to have at least the following information directly readable in the list view in Divvun Manager:

• installed version (with active channel)
• we need column headers in the list view, so that we can tell users exactly what they are seeing, and also make the list view sortable by clicking the column headers (cf Add "Sort" option to "All Repositories" view #44 )

[dylanhand]

Taskcluster is working for all lang-xxx repos, and signing and notarization is working again. That is, we have now a bunch of new nightlies available to us. Except not:

Expected: available for installation

Divvun Manager 2.0.1.

The reason no updates are listed in this situation is because the nightly version number is equal to (or less than) the current stable release version.

One solution is to bump the version number once a release has made it to the stable channel. It could be argued that this is what should have been done all along since, once version 1.0.0 is released, you're no longer building nightly versions of 1.0.0, but rather nightly versions of forthcoming versions such as 1.0.1, 1.1.0, etc.

These steps were taken to test this:

1. Install Inari Sami spellchecker from the stable channel

2. Switch to nightly channel, where current version is 1.0.5-nightly.20240517T030637371Z. It shows "No Updates" because the version, 1.0.5, is the latest version, and it's installed (the current implementation of DM ignores the "-nightly.2024..." part.

3. Bump the speller patch version to produce a new nightly version, then refresh DM. This now shows that an update is available since 1.0.6 > 1.0.5:

---

TLDR

If you want nightly versions to be installable over stable versions, the easy fix that requires no changes is to bump the nightly's version number. The user will still have the option not to update if they choose: they simply don't update when switched to the nightly channel.

---

This solution however does not address what this issue has morphed into, which is to display the channel a package was installed from.

So now, given this new information, the question is: is the current implementation acceptable (i.e., bump the nightly version number to make them available to install over stable versions)?

Let me know if you'd still like a column that lists the active channel from which a package is installed. As discussed in our meeting, I've worked on an experimental feature that lets us get the non-truncated package version (e.g. 1.0.0-nightly... instead of just 1.0.0) from the Info.plist, which would help us determine from which channel a package is installed.

One unknown is whether we can accurately determine whether a package was installed from the beta channel. Packages from the nightly channel always have "nightly" in their version string, so that's easy. If there's a similar requirement for beta packages (e.g. they always contain something like -rc1), that too will be easy. If not, we may have to figure out a way to identify packages installed from the beta channel.

[snomos]

@dylanhand before we can answer the question of whether we need an active channel column, we need to clarify the following:

when do automatic updates occur? The intended behaviour is that within the same channel, new releases should be installed automatically (the installed update will be available at latest after computer restart, due to macOS caching speller lexicons in memory, so an app restart might not be enough). Do automatic updates also occur across channels if version numbers are changed as suggested above?

At least within one and the same channel, automatic updates do happen — the SMN update mentioned above got installed without me doing anything:

$ ls -l /Library/Services/smn.bundle/Contents/Resources/smn.zhfst  
-rw-r--r--  1 root  wheel  10185021 May 20 16:42 /Library/Services/smn.bundle/Contents/Resources/smn.zhfst

$ echo nuvviDspeller | divvunspell suggest -a /Library/Services/smn.bundle/Contents/Resources/smn.zhfst
Reading from stdin...
Input: nuvviDspeller		[INCORRECT]
Divvun speller for Inari Sami		1
smn, desktop, version 1.0.6, 20.05.2024-1341		2
Built using HFST 3.16.0		3

I have now bumped the FAO speller in a similar manner; I have it installed from the stable channel ATM, let's see whether it is updated automatically to nightly if the truncated, semver version number is bumped.

The plan has all along been that automatic updates do occur as follows:

• always when new releases are available in the same channel as previously installed
• never across release channels

The confusion arises for people when they don't remember/see which channel a package was installed from, and expect it to be automatically updated, but then it does not.

[snomos]

Faroese update out (1.1.2-nightly), but not installed, and not listed as installable:

I have this installed:

$ ls -l /Library/Services/*.bundle/Contents/Resources/*.zhfst  
-rw-r--r--  1 root  wheel   5276913 Oct  8  2021 /Library/Services/fo.bundle/Contents/Resources/fo.zhfst

$ echo nuvviDspeller | divvunspell suggest -a /Library/Services/fo.bundle/Contents/Resources/fo.zhfst 
Reading from stdin...
Input: nuvviDspeller		[INCORRECT]
Divvun speller for Faroese		1
fo, desktop, version 1.1.1, 08.10.2021-0722		2
Built using HFST 3.15.5		3

That is, for Faroese, the update is not installed automatically, and it is also not listed as manually installable, despite having a higher (truncated/semver) version number.

[snomos]

That is, for Faroese, the update is not installed automatically, and it is also not listed as manually installable, despite having a higher (truncated/semver) version number.

I was too quick to draw conclusions. Now, an hour or so later, it has been automatically installed:

➜ ls -l /Library/Services/*.bundle/Contents/Resources/*.zhfst                                             
-rw-r--r--  1 root  wheel   5336900 May 21 09:40 /Library/Services/fo.bundle/Contents/Resources/fo.zhfst
[...]

➜ echo nuvviDspeller | divvunspell suggest -a /Library/Services/fo.bundle/Contents/Resources/fo.zhfst
Reading from stdin...
Input: nuvviDspeller		[INCORRECT]
Divvun speller for Faroese		1
fo, desktop, version 1.1.2, 21.05.2024-0640		2
Built using HFST 3.16.0		3

That is, it seems you are right @dylanhand that updates happen irrespective of channel, and based on the truncated version number only.

[dylanhand]

Got it. This is what appears to have happened for me as well -- namely, the nightly Inari Sami spellchecker (1.0.6-nightly) was installed over the stable version (1.0.5).

So then it sounds like the requirements are:

1. Divvun Manager (and the RPC) need to be updated to keep track of which channel a package was installed from
2. That channel should be listed in the DM UI
3. Ensure automatic updates only update packages from the same channel from which a package was initially installed

I know it will be possible to differentiate stable/nightly versions since nightly versions always have "nightly" in the version number.

Is there a similar convention for beta versions? I see that some are of the form 0.1.1-rc1. Will this always be the case?

If not we'll need a different way of identifying packages installed via the beta channel.

[snomos]

So then it sounds like the requirements are:

1. Divvun Manager (and the RPC) need to be updated to keep track of which channel a package was installed from
2. That channel should be listed in the DM UI
3. Ensure automatic updates only update packages from the same channel from which a package was initially installed

Yes.

The nightly/beta/regular release algorithm is very simple:

• nightly: contains nightly in the version string
• beta: does not contain nightly, and version number is < 1.0.0
• standard release: does not contain nightly, and version number is ≥ 1.0.0

That is, we don't do betas after the first public, full release (the 1.0.0 release).

[dylanhand]

Ok great, thanks for clarifying the release details. I'll implement it that way.

[dylanhand]

FYI, here's a mock-up of what I have in mind for listing used channel. Packages that are installed will show what channel they were installed from. Packages that are not installed show nothing in this column.

Please let me know if you agree or see it differently 🙂

[snomos]

Looks good, but also amplifies the need for column headers. What do these labels mean?

[dylanhand]

Agreed. I'll look into adding column headers at the same time.

[dylanhand]

As discussed offline, this is a wontfix.

Divvun Manager was built to have one channel for everything. The main point of confusion was the need to bump nightly version numbers for them to supersede stable versions.

Changes to Pahkat have been made that fix #48.

@snomos can you close this issue if you agree?

[snomos]

There is still at least one case where things are not working for me:

I have updated to the latest nightly of Divvun Manager, and that seems to have fixed the update of most languages, according to the nightly version update fix. Still, for Greenlandic, things are not updating.

This is the list view in Divvun Manager:

Based on the listings, it is clear that I have enabled the nightly channel.

And this is what I see in the Terminal:

$ ls -l /Library/Services/*.bundle/Contents/Resources/*.zhfst
-rw-r--r--  1 root  wheel   5337206 May 29 02:52 /Library/Services/fo.bundle/Contents/Resources/fo.zhfst
-rw-r--r--  1 root  wheel   3165554 Nov  8  2023 /Library/Services/kl.bundle/Contents/Resources/kl.zhfst
[...]
$ echo nuvviDspeller | divvunspell suggest -a /Library/Services/kl.bundle/Contents/Resources/kl.zhfst
Reading from stdin...
Input: nuvviDspeller		[INCORRECT]
Divvun speller for Kalaallisut		1
kl, desktop, version 1.0.0, 08.11.2023-1413		2
Built using HFST 3.16.0		3

That is, despite Greenlandic nightly having a semver string of 1.0.1, and the installed stable version is 1.0.0, the nightly version is not installed as it should. This is in contrast with most other languages I have installed, they are in most cases updated to the latest nightly available:

$ ls -l /Library/Services/*.bundle/Contents/Resources/*.zhfst
-rw-r--r--  1 root  wheel   5337206 May 29 02:52 /Library/Services/fo.bundle/Contents/Resources/fo.zhfst
-rw-r--r--  1 root  wheel   3165554 Nov  8  2023 /Library/Services/kl.bundle/Contents/Resources/kl.zhfst
-rw-r--r--  1 root  wheel    103140 May 29 01:05 /Library/Services/lut.bundle/Contents/Resources/lut.zhfst
-rw-r--r--  1 root  wheel   3998043 May 29 02:05 /Library/Services/mhr.bundle/Contents/Resources/mhr.zhfst
-rw-r--r--  1 root  wheel   2253551 May 29 02:31 /Library/Services/mns.bundle/Contents/Resources/mns.zhfst
-rw-r--r--  1 root  wheel    989850 May 28 22:13 /Library/Services/mrj.bundle/Contents/Resources/mrj.zhfst
-rw-r--r--  1 root  wheel   1499132 Sep 21  2022 /Library/Services/nno.bundle/Contents/Resources/nn-Runr.zhfst
-rw-r--r--  1 root  wheel   1515258 Sep 21  2022 /Library/Services/nno.bundle/Contents/Resources/nn.zhfst
-rw-r--r--  1 root  wheel   1370916 May 28 21:49 /Library/Services/olo.bundle/Contents/Resources/olo.zhfst
-rw-r--r--  1 root  wheel  18770386 May 29 17:05 /Library/Services/se.bundle/Contents/Resources/se.zhfst
-rw-r--r--  1 root  wheel   4737921 Sep 25  2023 /Library/Services/sma.bundle/Contents/Resources/sma.zhfst
-rw-r--r--  1 root  wheel  15322281 May 15 18:47 /Library/Services/smj.bundle/Contents/Resources/smj.zhfst
-rw-r--r--  1 root  wheel  15393132 May 15 18:47 /Library/Services/smj.bundle/Contents/Resources/smj_NO.zhfst
-rw-r--r--  1 root  wheel  15390775 May 15 18:48 /Library/Services/smj.bundle/Contents/Resources/smj_SE.zhfst
-rw-r--r--  1 root  wheel  10386438 May 29 04:47 /Library/Services/smn.bundle/Contents/Resources/smn.zhfst
-rw-r--r--  1 root  wheel  23508463 May 28 23:53 /Library/Services/sms.bundle/Contents/Resources/sms.zhfst

In the two other cases where an update has not been installed, it is either because of build issues (nno) or version number identity for stable and nightly (sma). So only KAL is not behaving as expected.

[dylanhand]

Thanks for the info - I'll look into what's causing lang-kal to behave differently from the others.

[dylanhand]

I was unable to reproduce the issue with lang-kal.

These were my steps:

1. Install the latest nightly of Divvun Manager Mac
2. Restart
3. Use DM to install the speller for lang-kal
4. Restart
5. Take note of the installed version (1.0.1-nightly.20240530T215916935Z)
6. Deploy new version of lang-kal
7. Go to lunch
8. Force-refresh DM UI by switching from "All Repositories" to "Divvun Languages", and back again

Result:
The new speller was installed while I was away at lunch.

Running your command from above also shows that the kl.zhfst file was updated:

ls -l /Library/Services/*.bundle/Contents/Resources/*.zhfst
.rw-r--r-- root wheel  25 KB Thu Feb  1 09:58:43 2024  /Library/Services/aka.bundle/Contents/Resources/ak.zhfst
...
.rw-r--r-- root wheel  18 MB Fri May 31 12:21:09 2024  /Library/Services/kl.bundle/Contents/Resources/kl.zhfst
...
.rw-r--r-- root wheel  34 KB Thu Feb  1 11:43:42 2024  /Library/Services/zul-x-exp.bundle/Contents/Resources/zu.zhfst

[snomos]

I notice an inconsistency in the file listing above (last comment) regarding the bundle names:

• aka and zul is using the full BCP-47 tag including ISO 639-3 codes in the bundle names (ie the same as used for the repository name)
• kl is using the ISO 639-1 code both for bundle name and zhfst file

Could this inconsistency (ie not the same bundle naming conventions everywhere) be causing trouble for DM in finding or detecting the bundle or the bundle metadata/timestamp?

It makes sense that the bundle name should be the full BCP-47 string from the repo name, as that makes it possible to have parallel installs for the same language but different repos installed at the same time. There are several languages with multiple (two) repos, for various reasons, and being able to install spellers from them all/both makes it possible to test both spellers or choose which one to use.