Dashboard no longer reports accurate scores for Cascade Layers

[jensimmons]

Cascade Layers used to report: Chrome 100 / Firefox 96.3 / Safari 100 (preview scores)
Now it reports: Chrome 96.7 / Firefox 93.1 / Safari 96.7 (preview scores)

If you click through to see the tests & the results, you can see that the earlier numbers are the correct numbers. So currently, the dashboard is wrong.

This may be happening for other focus areas, too.

[robnyman]

@DanielRyanSmith If you know why this could be?

[gsnedders]

The dashboard matches https://github.com/web-platform-tests/results-analysis/blob/cc161045215088212e0eb4c678024cd221e7d07e/data/interop-2022/interop-2022-experimental-v2.csv, so it's not a dashboard problem per-se.

The Cascade Layers runs today don't show this; I presume this is the exact set of runs being scored. And that still doesn't show these scores.

Could we somehow have fetched web-platform-tests/results-analysis-cache in some intermediary state?

[gsnedders]

So the following columns have changes in the last two rows:

In Interop 2023:

chrome-interop-2022-cascade
firefox-interop-2022-cascade
firefox-interop-2022-dialog
safari-interop-2022-cascade
safari-interop-2023-contain
safari-interop-2023-flexbox
safari-interop-2023-offscreencanvas

In Interop 2022:

chrome-interop-2022-cascade
firefox-interop-2022-cascade
firefox-interop-2022-dialog
safari-interop-2021-flexbox
safari-interop-2022-cascade

[foolip]

Could we somehow have fetched web-platform-tests/results-analysis-cache in some intermediary state?

I have no theory of what's gone wrong, but I don't think so. Each run gets a tag only once the tree is built/committed, so there should be no intermediary state to fetch.

It's possible a broken conversion has happened, but that then isn't going to fix itself.

@DanielRyanSmith what do you think?

[gsnedders]

I have no theory of what's gone wrong, but I don't think so. Each run gets a tag only once the tree is built/committed, so there should be no intermediary state to fetch.

The alternative is it gets written to the cache in an intermediary state, depending on when wpt.fyi starts serving it?

[gsnedders]

git diff run/5128159591137280/results..run/5192142658469888/results -- css/css-cascade shows quite a lot of changes (that should be from yesterday's to the current run today, for Chrome), which is basically: https://wpt.fyi/results/css/css-cascade?view=subtest&diff&filter=ADC&run_id=5128159591137280&run_id=5192142658469888

[gsnedders]

Ohhh, @DanielRyanSmith: this is because all-prop-revert-layer-noop.html is labelled, and thus considered part of the total number of tests, but we don't have any results for it because it's been split into variants.

[DanielRyanSmith]

Ohhh, @DanielRyanSmith: this is because all-prop-revert-layer-noop.html is labelled, and thus considered part of the total number of tests, but we don't have any results for it because it's been split into variants.

I think you're right. This is a comparison of 3 recent runs today as opposed to 3 yesterday, filtering for the interop-2022-cascade label.

Is this something that requires changing the wpt-metadata to remove the old labeled tests? This is a new scenario that I don't think we've prepared for, and just changing the test name/path could change the historical data overall, as they're listed in older runs with a different path.

[gsnedders]

Is this something that requires changing the wpt-metadata to remove the old labeled tests? This is a new scenario that I don't think we've prepared for, and just changing the test name/path could change the historical data overall, as they're listed in older runs with a different path.

Yeah, this has been the problem anything like this happens, which is always unfortunate. This is was probably most apparent in 2021 where the tests were being revised pretty much throughout the year.

I've filed web-platform-tests/wpt-metadata#3769 to change the metadata, and filed web-platform-tests/results-analysis#149 to try and make this easier to debug.

Note that a change like this splitting a test into variants also would impact the score if it weren't 100%, because it changes the total number of tests we're scoring against. See #256 for some discussion about similar variant scoring.

[foolip]

I've filed #284 about the generic problem.

@nt1m unlabeled the test in web-platform-tests/wpt-metadata#3750 which should restore the Cascade Layers scores to 100 where they were before.

Leaving this issue open to decide what we do about the split test, which is now 8 individual tests. I'll comment on #256 about what I think makes sense in general.

[nt1m]

@nt1m unlabeled the test in web-platform-tests/wpt-metadata#3750 which should restore the Cascade Layers scores to 100 where they were before.

I also merged Sam's PR: web-platform-tests/wpt-metadata#3769 which relabeled the tests, the change that unlabeled the test was just temporary in order to merge the WPT manifest PR.

[foolip]

OK, so the scores are now back to 100 for both 2022 and 2023 dashboards. But historical scores before the new tests existing will be lower. We should work this out in #283 + #286.

[foolip]

And #284 actually, that's the issue directly about this problem.