Removal of SuperLU from Trilinos config and build results in poor Continuity solves about nalu

[srajama1]

@jhux2 : I believe you were going to check this first. Let me know if I can help.

[jhux2]

@srajama1 Yes, I will.

[spdomin]

Is there an update on this issue?

[spdomin]

I am going to again ask about an update. Perhaps the best question is simply: "when is this issue expected to be resolved"

[jhux2]

@spdomin Sorry about the delay. I'll shoot to have this resolved by end of the month.

[jhux2]

Here are the tests that fail on my local machine when I have MueLu use KLU as the coarsest grid solve, instead of SuperLU:

The following tests FAILED:
	  1 - ablForcingEdge (Failed)
	  2 - ablStableElem (Failed)
	  3 - ablUnstableEdge (Failed)
	  7 - cvfemHexHC_P3 (Failed)
	 10 - dgNonConformal3dFluids (Failed)
	 16 - dgNonConformalFluids (Failed)
	 17 - dgNonConformalFluidsEdge (Failed)
	 18 - dgNonConformalThreeBlade (Failed)
	 19 - ductElemWedge (Failed)
	 20 - ductWedge (Failed)
	 23 - ekmanSpiral (Failed)
	 24 - elemBackStepLRSST (Failed)
	 25 - elemClosedDomain (Failed)
	 26 - elemHybridFluids (Failed)
	 27 - elemHybridFluidsShift (Failed)
	 28 - elemPipeCHT (Failed)
	 31 - fluidsPmrChtPeriodic (Failed)
	 33 - heatedWaterChannelEdge (Failed)
	 34 - heatedWaterChannelElem (Failed)
	 35 - heliumPlume (Failed)
	 37 - hoVortex (Failed)
	 40 - kovasznay_P7 (Failed)
	 41 - milestoneRun (Failed)
	 42 - milestoneRunConsolidated (Failed)
	 44 - movingCylinder (Failed)
	 51 - oversetFluids (Failed)
	 52 - oversetFluidsEdge (Failed)
	 54 - periodic3dElemNp4 (Failed)
	 55 - periodic3dElemNp8 (Failed)
	 57 - periodic3dEdgeNp4 (Failed)
	 61 - steadyTaylorVortex_P4 (Failed)
	 62 - hoVortex_P2 (Failed)
	 65 - variableDensNonUniform_P5 (Failed)
	 68 - oversetHybrid (Failed)
	 69 - uqSlidingMeshDG (Failed)
	 70 - waleElemXflowMixFrac3.5m (Failed)

[jhux2]

I see differences in linear iteration counts for some simulations, no differences for others.

ablForcingEdge (SuperLU) : linear iterations --  	avg: 8.775 	min: 8 	max: 11
ablForcingEdge (KLU    ) : linear iterations --  	avg: 8.775 	min: 8 	max: 11
ablStableElem (SuperLU) : linear iterations --  	avg: 11.325 	min: 8 	max: 12
ablStableElem (KLU    ) : linear iterations --  	avg: 11.325 	min: 8 	max: 12
ablUnstableEdge (SuperLU) : linear iterations --  	avg: 10.8 	min: 9 	max: 11
ablUnstableEdge (KLU    ) : linear iterations --  	avg: 10.8 	min: 9 	max: 11
awk: fatal: cannot open file `cvfemHexHC_P3/cvfemHexHC_P3.log' for reading (No such file or directory)
awk: fatal: cannot open file `test-logs-failing/cvfemHexHC_P3.log' for reading (No such file or directory)
dgNonConformal3dFluids (SuperLU) : linear iterations --  	avg: 13.58 	min: 8 	max: 17
dgNonConformal3dFluids (KLU    ) : linear iterations --  	avg: 28.75 	min: 21 	max: 35
dgNonConformalFluids (SuperLU) : linear iterations --  	avg: 9.375 	min: 7 	max: 11
dgNonConformalFluids (KLU    ) : linear iterations --  	avg: 9.375 	min: 7 	max: 11
dgNonConformalFluidsEdge (SuperLU) : linear iterations --  	avg: 8.97 	min: 8 	max: 11
dgNonConformalFluidsEdge (KLU    ) : linear iterations --  	avg: 8.97 	min: 8 	max: 11
dgNonConformalThreeBlade (SuperLU) : linear iterations --  	avg: 9.49 	min: 8 	max: 11
dgNonConformalThreeBlade (KLU    ) : linear iterations --  	avg: 9.49 	min: 8 	max: 11
ductElemWedge (SuperLU) : linear iterations --  	avg: 15.2143 	min: 12 	max: 17
ductElemWedge (KLU    ) : linear iterations --  	avg: 23.8571 	min: 20 	max: 26
ductWedge (SuperLU) : linear iterations --  	avg: 10 	min: 8 	max: 11
ductWedge (KLU    ) : linear iterations --  	avg: 14.2143 	min: 12 	max: 16
ekmanSpiral (SuperLU) : linear iterations --  	avg: 10.225 	min: 10 	max: 11
ekmanSpiral (KLU    ) : linear iterations --  	avg: 10.225 	min: 10 	max: 11
elemBackStepLRSST (SuperLU) : linear iterations --  	avg: 5.6875 	min: 5 	max: 9
elemBackStepLRSST (KLU    ) : linear iterations --  	avg: 32.875 	min: 32 	max: 37
elemClosedDomain (SuperLU) : linear iterations --  	avg: 5.025 	min: 5 	max: 6
elemHybridFluids (SuperLU) : linear iterations --  	avg: 32.9 	min: 26 	max: 34
elemHybridFluids (KLU    ) : linear iterations --  	avg: 32.8833 	min: 26 	max: 34
elemHybridFluidsShift (SuperLU) : linear iterations --  	avg: 43.38 	min: 30 	max: 91
elemHybridFluidsShift (KLU    ) : linear iterations --  	avg: 43.38 	min: 30 	max: 91
elemPipeCHT (SuperLU) : linear iterations --  	avg: 9.25 	min: 8 	max: 11
elemPipeCHT (KLU    ) : linear iterations --  	avg: 11.45 	min: 11 	max: 14
fluidsPmrChtPeriodic (SuperLU) : linear iterations --  	avg: 10.7143 	min: 8 	max: 13
fluidsPmrChtPeriodic (KLU    ) : linear iterations --  	avg: 10.7143 	min: 8 	max: 13
heatedWaterChannelEdge (SuperLU) : linear iterations --  	avg: 10.5125 	min: 8 	max: 18
heatedWaterChannelEdge (KLU    ) : linear iterations --  	avg: 10.5125 	min: 8 	max: 18
heatedWaterChannelElem (SuperLU) : linear iterations --  	avg: 15.6 	min: 10 	max: 23
heatedWaterChannelElem (KLU    ) : linear iterations --  	avg: 15.6 	min: 10 	max: 23
hoVortex (SuperLU) : linear iterations --  	avg: 7.695 	min: 7 	max: 8
hoVortex (KLU    ) : linear iterations --  	avg: 20.485 	min: 16 	max: 30
milestoneRun (SuperLU) : linear iterations --  	avg: 17.5 	min: 15 	max: 30
milestoneRun (KLU    ) : linear iterations --  	avg: 17.5 	min: 15 	max: 30
milestoneRunConsolidated (SuperLU) : linear iterations --  	avg: 17.5 	min: 15 	max: 30
milestoneRunConsolidated (KLU    ) : linear iterations --  	avg: 17.5 	min: 15 	max: 30
movingCylinder (SuperLU) : linear iterations --  	avg: 18.875 	min: 16 	max: 23
movingCylinder (KLU    ) : linear iterations --  	avg: 18.875 	min: 16 	max: 23
oversetFluids (SuperLU) : linear iterations --  	avg: 24.655 	min: 19 	max: 28
oversetFluids (KLU    ) : linear iterations --  	avg: 50 	min: 50 	max: 50
oversetFluidsEdge (SuperLU) : linear iterations --  	avg: 11.885 	min: 11 	max: 14
oversetFluidsEdge (KLU    ) : linear iterations --  	avg: 49.84 	min: 45 	max: 50
awk: fatal: cannot open file `periodic3dElemNp4/periodic3dElemNp4.log' for reading (No such file or directory)
periodic3dElemNp4 (KLU    ) : linear iterations --  	avg: 20.3333 	min: 18 	max: 25
awk: fatal: cannot open file `periodic3dElemNp8/periodic3dElemNp8.log' for reading (No such file or directory)
periodic3dElemNp8 (KLU    ) : linear iterations --  	avg: 21.0833 	min: 19 	max: 26
awk: fatal: cannot open file `periodic3dEdgeNp4/periodic3dEdgeNp4.log' for reading (No such file or directory)
periodic3dEdgeNp4 (KLU    ) : linear iterations --  	avg: 37.0833 	min: 33 	max: 43
awk: fatal: cannot open file `steadyTaylorVortex_P4/steadyTaylorVortex_P4.log' for reading (No such file or directory)
awk: fatal: cannot open file `test-logs-failing/steadyTaylorVortex_P4.log' for reading (No such file or directory)
awk: fatal: cannot open file `hoVortex_P2/hoVortex_P2.log' for reading (No such file or directory)
awk: fatal: cannot open file `test-logs-failing/hoVortex_P2.log' for reading (No such file or directory)
oversetHybrid (SuperLU) : linear iterations --  	avg: 22.95 	min: 13 	max: 33
oversetHybrid (KLU    ) : linear iterations --  	avg: 22.95 	min: 13 	max: 33
uqSlidingMeshDG (SuperLU) : linear iterations --  	avg: 16.675 	min: 14 	max: 18
uqSlidingMeshDG (KLU    ) : linear iterations --  	avg: 16.675 	min: 14 	max: 18
waleElemXflowMixFrac3.5m (SuperLU) : linear iterations --  	avg: 36 	min: 29 	max: 41
waleElemXflowMixFrac3.5m (KLU    ) : linear iterations --  	avg: 36 	min: 29 	max: 41

[spdomin]

Yes. This is consistent with what I originally noted that prompted me to ask the question. In many cases the average iteration count is 2x greater. This would be non-ideal to accept without more understanding. It might be best to choose one and dig deeper. OversetFluids or the elemBackStep might be good choices.

What is your opinion on these diffs?

[jhux2]

I have an idea of why this is happening. I've looked closely at elemBackStepLRSST. The coarsest grid matrix is still distributed over all of the ranks because repartitioning is disabled. MueLu passes this to Amesos2; however, KLU2 is fundamentally a serial solver. I suspect there are four instances of KLU solving just the local parts of the matrix. I'll work with Siva to decide how MueLu and Amesos2 should interact.

[spdomin]

Great. Many thanks.

…

On Wed, Aug 2, 2017 at 7:26 PM Jonathan Hu ***@***.***> wrote: I have an idea of why this is happening. I've looked closely at elemBackStepLRSST. The coarsest grid matrix is still distributed over all of the ranks because repartitioning is disabled. MueLu passes this to Amesos2; however, KLU2 is fundamentally a serial solver. I suspect there are four instances of KLU solving just the local parts of the matrix. I'll work with Siva to decide how MueLu and Amesos2 should interact. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIS9vC2Xv-BA9JNHs-LI2pkNHtLMgAdMks5sUSG_gaJpZM4MVw9q> .

[jhux2]

I'll work with Siva to decide how MueLu and Amesos2 should interact.

@ndellingwood has fixed an issue in an Amesos2 interface, and pushed it to the Trilinos develop branch (commit 14b51f1). For the elemBackStepLRSST test, the patch fixed the linear iteration differences that I was observing.

With the patch applied, the Nalu reg_tests are now diffing with what appear to be very small tolerances, but I'm not in a position to say whether these are acceptable.

(~/.../naluBuilds/build-nalu-relwithdebuginfo-2017-July-31) egrep FAILED ctest-klu-2.log 
24: ..elemBackStepLRSST........... FAILED:  2.0387  s  max diff:  .00000000000059059999729313417127
18: ..dgNonConformalThreeBlade........... FAILED:  151.3890  s  max diff:  .00000000017276256437000051170314
40:   what():  Requirement( databaseIO != nullptr && databaseIO->ok(true) ) FAILED
25: ..elemClosedDomain........... FAILED:  1.9887  s  max diff:  .00000001326820419999359332297040
2: ..ablStableElem........... FAILED:  59.3860  s  max diff:  .00000000001975873227000018823458
42: ..milestoneRunConsolidated........... FAILED:  93.8787  s  max diff:  .00000000000000199991932736676148
16: ..dgNonConformalFluids........... FAILED:  47.0089  s  max diff:  .00000000000021299975672128823590
17: ..dgNonConformalFluidsEdge........... FAILED:  37.9532  s  max diff:  .00000000000136000065376062906352
33: ..heatedWaterChannelEdge........... FAILED:  9.4906  s  max diff:  .00000003312360000173875107520871
3: ..ablUnstableEdge........... FAILED:  27.6248  s  max diff:  .00000000004555002659999903992703
41: ..milestoneRun........... FAILED:  87.7597  s  max diff:  .00000000000000200002774758401002
1: ..ablForcingEdge........... FAILED:  23.8524  s  max diff:  .00000000002062048345634400008503
23: ..ekmanSpiral........... FAILED:  7.1169  s  max diff:  .00000000000000260399993194247899
68: ..oversetHybrid........... FAILED:  693.2446  s  max diff:  .00000000000025874800027326236643
31: ..fluidsPmrChtPeriodic........... FAILED:  76.1854  s  max diff:  .00000000005873000002987183165715
27: ..elemHybridFluidsShift........... FAILED:  58.2611  s  max diff:  .00000000000203366000051823959948
51: ..oversetFluids........... FAILED:  14.6238  s  max diff:  .00000000000000227999585357407941
52: ..oversetFluidsEdge........... FAILED:  9.4396  s  max diff:  .00000000000000204000092643083497

[jhux2]

@spdomin Do you prefer to wait until the patch hits master before evaluating the impact on the Nalu regression tests?

[spdomin]

My understanding is that we set a trilinos configure flag to deactivate SuperLU. As such I was thinking that this change can proceed and the nalu tests can still be using SuperLU. I was hoping that our Q3 runs could be replicated to assess the final path forward. Stephen can probably help with that subject his availability.

…

On Wed, Aug 16, 2017 at 12:21 PM Jonathan Hu ***@***.***> wrote: @spdomin <https://github.com/spdomin> Do you prefer to wait until the patch hits master before evaluating the impact on the Nalu regression tests? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIS9vFbx2BptY749sR8I9thGtwq52b_8ks5sYzMxgaJpZM4MVw9q> .

[srajama1]

The flag works the other way - enabling Superlu. Nalu has this ON and can continue to have it ON as long as someone can verify the diffs.

[jhux2]

@spdomin wrote:

My understanding is that we set a trilinos configure flag to deactivate
SuperLU. As such I was thinking that this change can proceed and the nalu
tests can still be using SuperLU.

As @srajama1 indicated, Nalu can continue to have SuperLU enabled in the Trilinos configure/build. Whether to use it is a runtime decision, controllable through the MueLu XML input deck. Including the following option will force MueLu to use KLU:
<Parameter name="coarse: type" type="string" value="klu2"/>

I'll ask @sthomas61 whether he can help rebaseline.

[jhux2]

I was hoping that our Q3 runs could be replicated to assess the final path
forward. Stephen can probably help with that subject his availability.

@sthomas61 said he can help with this.

[spdomin]

Pruning old issues. I verified that the latest Nalu build runs with the klu2 option with very small diffs. The simulations looked slower, however, no real statistics captured.

Is KLU preferred over SuperLU? If I change to KLU exclusively, do we have confidence of this at the extreme scale Nalu runs?

[jhux2]

@spdomin KLU is only ever used on the coarsest grid on a single MPI process. I'm pretty confident, especially given that it's used in apps like EMPIRE.

[spdomin]

Great!