Ifpack2/KokkosKernels? Performance regression with 4.3.1 in AdditiveSchwarz initialization with overlap

[brian-kelley]

Between 45d800 and 53b714, the "remainder" component of preconditioner initialization (AdditiveSchwarz + ILUk, overlap level 1) slowed down significantly on eclipse and amber (for example, 14.2->17.9s on amber for one abnormal energy problem). The remainder is supposed to not include any of the expensive things like RILUK setup, filter construction or overlapping row matrix construction.

[github-actions]

Automatic mention of the @trilinos/ifpack2 team

[github-actions]

Automatic mention of the @trilinos/ifpack2 team

[csiefer2]

@kliegeois @brian-kelley We saw substantial increases in read off disk time for the sparc problem as well, though we couldn't isolate that to the Kokkos/KK promotion.

Kim reworked the test to remove the disk read from Remainder

[brian-kelley]

@csiefer2 Sounds like that should be a separate issue, if it's reproducible and caused by a code change.
Good that we're not measuring that in the stacked timer anymore though.

[kliegeois]

@csiefer2 to be precise, the disk read was not in the remainder before #12997.
The PR removed the allocation of one vector (and its initialization to zero), computations of initial norms, conversion of line_info vector to parts arrays, and a block spmv used to make sure that the data moved from host to device.

[brian-kelley]

@csiefer2 @kliegeois I think i figured both of these things out (but they're not related)

• This issue appears to be 100% due to software environment changes on amber and eclipse. On amber we had to explicitly chang the compiler we use to intel 24 because intel 21 went away, and this happened on the exact day of the spike. Not sure about the details on eclipse. I wasn't able to replicate the old (good) time on either machine. Both the good and bad SHAs above with the current environment give the (bad) time on both machines.
• The sparc single_gpu remainder slowdown was due to cusparse BSR spmv being extremely slow on the first call. Before this didn't use the TPL at all. I'm talking to NVIDIA about what we could be done about it, but going back to our native impl on Cuda seems like a reasonable workaround.

[brian-kelley]

BTW, sparc was not because host-device copying either, because replacing cusparse with native (also on GPU obviously) decreases the time of that warmup apply from 2.36s -> 0.0038s. So the matrix was already synced to device.

[brian-kelley]

But I'll close this one as "not a code issue"