Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use SSIM as progress without normalization #52

Open
patrickhulce opened this issue May 12, 2017 · 3 comments
Open

Use SSIM as progress without normalization #52

patrickhulce opened this issue May 12, 2017 · 3 comments

Comments

@patrickhulce
Copy link
Collaborator

A followup to #48, we should follow @pahammad's advice and do what visualmetrics does, using SSIM as the progress once the first paint has occurred and do not normalize.

The normalization causes problems on sites like www.chromestatus.com where a white page has a SSIM of 60 (!!!) with the target. This greatly exaggerates the small differences in SSIM later in the page load since the remaining 40% is remapped to 100%.

PSI is ~2600 under the flawed approach when its realistically more like ~1600.

@patrickhulce
Copy link
Collaborator Author

patrickhulce commented May 12, 2017

Not normalizing at all also isn't ideal either, the PSI of twoframes-blank_content.json gets cut in half because the first paint of white with a very thin black border has 83 SSIM with the content :/

image

@pahammad
Copy link

This example you showed reminds me of a video Pat Meenan sent me a while ago, where the webpage just had the text "Hello World!" on a blank page, right after the firstPaint. Such a webpage loading video's PSI should be very close to firstPaint (plus some delta); and this is the correct behavior to expect in my opinion. firstPaint can be replaced with startRender (or firstVisualChange) since these metrics tend to be very close; just different terms used in different repositories.

@patrickhulce
Copy link
Collaborator Author

The problem here is that in this case, the text is not right after firstPaint, it's substantially farther out, so PSI in this case being very close to firstPaint (which from the user's point of view is indistinguishable from a blank white page) is not a very good outcome. All of the progress from the user's perspective here is when the first text loads, not from a blank white frame with an imperceptible border being painted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants