Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reliability of parallel Lin Out_channel tests #360

Closed
jmid opened this issue Jun 7, 2023 · 1 comment · Fixed by #370
Closed

Reliability of parallel Lin Out_channel tests #360

jmid opened this issue Jun 7, 2023 · 1 comment · Fixed by #370

Comments

@jmid
Copy link
Collaborator

jmid commented Jun 7, 2023

Currently we are disabling Out_channel tests on macOS because their output proved too unreliable:

if Sys.getenv_opt "OCAML_SYSTEM" = Some "macosx"
then (
Printf.printf "Lin DSL Out_channel test with Domain disabled under macOS\n\n%!";

I expect that this will also benefit the test signal coming from the new shiny arm64 macOS runners from Multicoretests-CI.

However the test unfortunately doesn't reliably fail on other targets.

Here's a Windows bytecode run with 5.1.0~alpha2 from 2 days ago that failed to trigger:
https://github.com/ocaml-multicore/multicoretests/actions/runs/5173787089/jobs/9319393810

random seed: 217324792
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain (generating)
[✓]   13    0    1   12 / 1000    15.0s Lin DSL In_channel test with Domain

[ ]    0    0    0    0 / 1000     0.0s Lin DSL Out_channel test with Domain
[ ]    3    0    0    3 / 1000   306.4s Lin DSL Out_channel test with Domain
[ ]   20    0    0   20 / 1000   372.0s Lin DSL Out_channel test with Domain
[ ]   43    0    0   43 / 1000   434.1s Lin DSL Out_channel test with Domain
[ ]   76    0    0   76 / 1000   495.0s Lin DSL Out_channel test with Domain
[ ]  108    0    0  108 / 1000   558.8s Lin DSL Out_channel test with Domain
[ ]  129    0    0  129 / 1000   619.0s Lin DSL Out_channel test with Domain
[ ]  168    0    0  168 / 1000   684.1s Lin DSL Out_channel test with Domain
[ ]  217    0    0  217 / 1000   744.5s Lin DSL Out_channel test with Domain
[ ]  262    0    0  262 / 1000   805.0s Lin DSL Out_channel test with Domain
[ ]  311    0    0  311 / 1000   867.0s Lin DSL Out_channel test with Domain
[ ]  387    0    0  387 / 1000   928.1s Lin DSL Out_channel test with Domain
[ ]  444    0    0  444 / 1000   990.5s Lin DSL Out_channel test with Domain
[ ]  525    0    0  525 / 1000  1051.7s Lin DSL Out_channel test with Domain
[ ]  608    0    0  608 / 1000  1113.5s Lin DSL Out_channel test with Domain
[ ]  698    0    0  698 / 1000  1177.1s Lin DSL Out_channel test with Domain
[ ]  794    0    0  794 / 1000  1237.4s Lin DSL Out_channel test with Domain
[ ]  839    0    0  839 / 1000  1305.8s Lin DSL Out_channel test with Domain
[ ]  915    0    0  915 / 1000  1375.6s Lin DSL Out_channel test with Domain
[ ]  945    0    0  945 / 1000  1490.0s Lin DSL Out_channel test with Domain
[✗] 1000    0    0 1000 / 1000  1519.5s Lin DSL Out_channel test with Domain

[...]

--- Failure --------------------------------------------------------------------

Test Lin DSL Out_channel test with Domain failed:

Negative test Lin DSL Out_channel test with Domain succeeded but was expected to fail
================================================================================
failure (1 tests failed, 0 tests errored, ran 2 tests)

and here's a recent Linux arm64 failure to trigger on 5.1:
https://ocaml-multicoretests.ci.dev:8100/job/2023-06-05/132328-ci-ocluster-build-4724e4

random seed: 255390210
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain (generating)
[✓]    4    0    1    3 / 1000     5.6s Lin DSL In_channel test with Domain

[ ]    0    0    0    0 / 1000     0.0s Lin DSL Out_channel test with Domain
[ ]   61    0    0   61 / 1000    63.4s Lin DSL Out_channel test with Domain
[ ]  219    0    0  219 / 1000   212.7s Lin DSL Out_channel test with Domain
[ ]  560    0    0  560 / 1000   273.9s Lin DSL Out_channel test with Domain
[ ]  746    0    0  746 / 1000   334.4s Lin DSL Out_channel test with Domain
[ ]  858    0    0  858 / 1000   395.4s Lin DSL Out_channel test with Domain
[✗] 1000    0    0 1000 / 1000   418.3s Lin DSL Out_channel test with Domain

[...]

--- Failure --------------------------------------------------------------------

Test Lin DSL Out_channel test with Domain failed:

Negative test Lin DSL Out_channel test with Domain succeeded but was expected to fail
================================================================================
failure (1 tests failed, 0 tests errored, ran 2 tests)
@jmid jmid changed the title Reliability of parallel Out_channel tests Reliability of parallel Lin Out_channel tests Jun 7, 2023
@jmid
Copy link
Collaborator Author

jmid commented Jul 10, 2023

Fixed in #370

@jmid jmid closed this as completed Jul 10, 2023
@jmid jmid linked a pull request Jul 10, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant