sandialabs / lcm

Laboratory for Computational Mechanics

Albany-LCM

Albany-LCM (Laboratory for Computational Mechanics) is spin-off (fork) from Albany, an implicit, unstructured grid, finite element code for the solution and analysis of multiphysics problems. The Albany-LCM repository on the GitHub site contains almost 200 regression tests and examples that demonstrate the code's capabilities on a wide variety of problems, with a strong focus on computational solid mechanics and thermo-mechanics.

Features

Analysis of complex multiphysics problems

Illustration of solid mechanics simulation of a notched cylinder, simulated using the alternating Schwarz-based multi-scale coupling capability in Albany-LCM.

Software architecture

Albany-LCM heavily leverages the Trilinos Framework, available at:

git clone https://github.com/trilinos/Trilinos.git

Albany-LCM supports the solution of very large problems (those over 2.1 billion degrees of freedom) using MPI. It relies on automatic differentiation from the Sacado library of Trilinos, which makes it straightforward to add new PDEs/physics to the code. Albany-LCM contains a wide variety of constitutive models for solid mechanics, ranging from simple linear elasticity to sophisticated nonlinear micro-structure models with plasticity (e.g., J2 plasticity, crystal plasticity), and the Schwarz alternating method for multi-scale coupling in solid mechanics. It also houses the terrestrial component of the Arctic Coastal Erosion (ACE) model, a coupled thermo-mechanical model with some novel permafrost constitutive models currently under development as part of the InterFACE project.

Building Albany

To get started with Albany-LCM it is helpful to consult the build instructions for both Trilinos and Albany located on the Albany wiki at https://github.com/sandialabs/Albany/wiki/Building-Albany-and-supporting-tools. For help with building the code, please contact Irina Tezaur ([email protected]) or Alejandro Mota ([email protected]).

Nightly Build and Test Results

The Albany-LCM repository is tested nightly on several CPU-based architectures, with the results posted to an internal-to-Sandia CDash site.

The regression test suite is contained within the Albany-LCM repository in the directories:

/tests

These tests are stand-alone and also serve as nice examples about how to describe various multiphysics problems. They also serve as a template for developing new simulations.

Once Albany-LCM is built, the default test suite is executed by typing ctest within the build directory. Any individual test can be executed by moving into its sub-directory, and executing ctest in that sub-directory. Many Albany-LCM tests run in parallel using up to 4 MPI ranks.

Documentation

Unfortunately, we do not have up-to-date documentation of Albany-LCM; the interested user may wish to consult the (out-of-date) HTML user guide inside the Albany-LCM repository at:

/doc/user-guide/guide.html

The LaTeX Developer's Guide (also out-of-date) is located at:

/doc/developersGuide

Note on Legacy and Unsupported Code

When Albany-LCM was first created by forking from the main Albany code, the idea was to use this code primarily for analyses involving mechanics and thermo-mechanics, as the name of the code suggests. To facilitate development of the code, a decision was made to remove PDEs and capabilities that were no longer funded, including PDEs not relevant for solid mechanics modeling, Kokkos kernels and mesh adaptation. Users interested in these capability should check out the main Albany repository, which houses a performance-portable land-ice model known as MPAS-Albany Land Ice (MALI) and the Albany-SCOREC repository, which focuses on developing capabilities for additive manufacturing and includes adaptive mesh refinement (AMR) via the Parallel Unstructured Mesh Interface (PUMI).

lcm's Issues

Failing MiniErosion serial tests

The following serial MiniErosion tests are failing:

ACE_MiniErosion_Denudation_Serial
ACE_MiniErosion_NewWavePressNBC_Serial

It looks like the problem started on 3/31:

https://sems-cdash-son.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=46927

My guess is it is due to changes to Trilinos that happened around that time...

Failing Tempus GERK test in nightlies

The TransientHeat2D_Tempus_GERK test in Albany started failing due to diffs:

https://sems-cdash-son.sandia.gov/cdash/testDetails.php?test=1052769&build=15330

his is due to changes in Tempus. I was hoping to understand how these changes affected this test before the relevant PR was merged in, but did not have time to do this, so it will have to be looked into after the fact.

Throw exception when running a dynamic LCM problem with no density (and other material parameters) specified

The default value of the density in the code is 1.0 which is non-physical. Exception should be thrown when not specifying density for dynamic problem to prevent user from running with density = 1.0 accidentally.

Nonphysical configurations sometimes obtained with Tempus time-step control when dt is augmented

Some nonphysical configurations were obtained using Tempus time-step control when dt is allowed to be augmented. This does not appear to be due to any recent changes to Tempus or Albany - the issue shows up in an older Albany dating back to Aug. 2018 (I did not go further back), and seems to have slipped through the cracks b/c it does not show up all the time.

The attached figure gives an example of the incorrect configuration obtained for a simple cubes problem (https://github.com/SNLComputation/Albany/blob/master/tests/small/LCM/DynamicsTempus/CubesSDBC/larger_cube.yaml ).

Remove deprecated STK constructor calls from LCM

The following is an email from @alanw0 👍

For codes that directly construct instances of the stk::mesh::BulkData class, we are introducing a MeshBuilder class which will be used as a factory to obtain instances of BulkData.

MeshBuilder is already in the Trilinos version of stk. See packages/stk/stk_mesh/stk_mesh/base/MeshBuilder.hpp.

The public BulkData constructor will be deprecated in coming weeks (exact date not specified, but “soon”).
Once we add the deprecation macro, we will wait approximately 6 weeks before removing the public constructor.

Please see the attached slides for details, and don’t hesitate to send questions and concerns to the stk team at [[email protected]](mailto:[email protected]).
Also, unit-tests may be informative, which are located at stk/stk_unit_tests/stk_mesh/UnitTestMeshBuilder.cpp

The following PR for Panzer may be relevant: trilinos/Trilinos#10573 .

Floating point exceptions in Albany LCM material models showing up in Clang build

This is a spin-off issue from sandialabs/Albany#398.

I have been working on cleaning up the Albany dashboard, including the debug / FPE check builds. Things should be almost entirely clean once sandialabs/Albany#399 is resolved, with the exception of the AlbanyClang build (http://cdash.sandia.gov/CDash-2-3-0/viewTest.php?onlyfailed&buildid=79234). All the tests failing are LCM tests. I have been digging into it and the problem are FPEs deep within the inelastic material models, specifically the computeState() routine. Here is some example output from gdb identifying the location:

SNLComputation/Albany#2  0x0000000003821d22 in LCM::ConstitutiveModelInterface<PHAL::AlbanyTraits::Jacobian, PHAL::AlbanyTraits>::evaluateFields(PHAL::Workset&) ()
    at /ascldap/users/ikalash/glensBuilds/nightlies/repos/Albany/src/./LCM/models/ConstitutiveModelInterface_Def.hpp:240

(https://github.com/gahansen/Albany/blob/master/src/LCM/models/ConstitutiveModelInterface_Def.hpp#L240). If you look at computeState() for various material models, there are numerous places where there could be division by zero which could lead to the FPEs.

My theory is the inelastic material models are the issue. Indeed if I take one of the failing problems and change the material model to Neohookean, the problem runs fine.

This issue goes deeper than I am willing to dive with this clean up effort (and also beyond my knowledge of the material models in Albany). Nonetheless we need to figure out how to proceed b/c it is imperative to have a clean dashboard in my opinion.

I believe from talking to @lxmota that the FPEs in the material models are known of, and there is some disagreement about whether they should be fixed. Perhaps @jwfoulk, @jtostie and/or @calleman21 , you can comment on this.

Regardless, since fixing this would possibly be a major effort, I propose to do one of 2 things:

1.) Turn off LCM in the AlbanyClang build so it can be monitored for real failures that crop up beyond these FPEs. There are many other builds with LCM one on a variety of platforms including debug builds.
2.) Turn off FPE checking in the AlbanyClang build.

I would vote for 1.). I think a clang build is potentially important for other applications, where we want to check for FPEs, so this should continue. We need to make a decision b/c we cannot keep having tests with dozens of failures for months at a time, especially with a major refactor in progress.

What is a bit of a mystery is why the FPEs started some months back. I don't think the material models were changed. Perhaps the compiler or build options were modified? @gahansen , any thoughts, as the owner of the build?

CDash reporting errors when build completes

Starting on 4/20, the clang LCM build started reporting build errors to CDash when in fact there are no errors and a working executable is created. Here is what CDash says (reports just warnings, not errors): https://sems-cdash-son.sandia.gov/cdash/viewBuildError.php?type=0&buildid=48137. I am attaching the full log from the build. I'm puzzled as to what is causing this. @bartlettroscoe , as my go-to cmake/cdash contact, have you seen this sort of thing before? Any tips for how to circumvent it? We could modify the cmake/cdash scripts to run tests even if the build fails, but then the false failed build will still be reported to the CDash site.

lcm-serial-clang-release.log.txt.log

Issue with IP to Nodal Field response writing out all 0 field in first time-step for dynamics problems

Earlier this week, @lxmota and I discovered (yet) an(other) issue with the IP to Nodal Field response. For dynamic problems, when "Exodus Write Interval" is set to 1 (writing solution / responses out at every time step) and requesting the IP to Nodal Field Response, the nodal_* fields are all 0 in the first time-step even though the qp (non-projected to nodes) analogs of these fields are non-zero. Here is an example where the problem manifests itself:

https://github.com/gahansen/Albany/blob/master/tests/small/LCM/TensionSpecimen/tension_dynamic_impl.yaml

If one sets "Exodus Write Interval" to > 1 (e.g., 2), the first nodal_* field is non-zero, so projected fields get calculated / written out just fine. The problem does not seem to occur for quasistatics: https://github.com/gahansen/Albany/blob/master/tests/small/LCM/TensionSpecimen/tension_quasistatic.yaml .

Fix calls to deprecated Tpetra functions

Tpetra is deprecating some functionality used in Albany and hence LCM - please see the following issue in Albany for details: sandialabs/Albany#763 . The deprecated code will be removed at the beginning of February. We need to get the code to work w/o the deprecated code by then.

ACE MiniErosion test with denudation failing on Proxima/Algol post Fedora 35 update

The ACE MiniErosion test with denudation began failing after Algol were upgraded to Fedora 35 when the code is built with a Clang compiler. FPEs are encountered in the test:

https://sems-cdash-son.sandia.gov/cdash/test/1879062

-- Nonlinear Solver Step 0 -- 
||F|| = 2.557e+03  step = 0.000e+00  dx = 0.000e+00
************************************************************************

[algol:2823662:0:2823662] Caught signal 8 (Floating point exception: floating-point invalid operation)
==== backtrace (tid:2823662) ====
 0  /usr/lib64/libucs.so.0(ucs_handle_error+0x2a4) [0x7fe851d05864]
 1  /usr/lib64/libucs.so.0(+0x2a41d) [0x7fe851d0941d]
 2  /usr/lib64/libucs.so.0(+0x2a6fa) [0x7fe851d096fa]
 3  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libifpack2.so.13(_ZN7Ifpack27Details9ChebyshevIdN6Tpetra11MultiVectorIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS4_6SerialENS4_9HostSpaceEEEEEE7computeEv+0x41e) [0x7fe85a3354ae]
 4  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libifpack2.so.13(_ZN7Ifpack29ChebyshevIN6Tpetra9RowMatrixIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS3_6SerialENS3_9HostSpaceEEEEEE7computeEv+0x173) [0x7fe85a231843]
 5  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu15Ifpack2SmootherIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE14SetupChebyshevERNS_5LevelE+0xa52) [0x7fe85bc98aa2]
 6  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu15Ifpack2SmootherIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5SetupERNS_5LevelE+0xcee) [0x7fe85bc92e8e]
 7  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu16TrilinosSmootherIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5SetupERNS_5LevelE+0xd5) [0x7fe85bf888d5]
 8  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZNK5MueLu15SmootherFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE13BuildSmootherERNS_5LevelENS_9PreOrPostE+0x6e3) [0x7fe85bf05693]
 9  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu-adapters.so.13(_ZNK5MueLu22SingleLevelFactoryBase9CallBuildERNS_5LevelE+0x38c) [0x7fe85c8f491c]
10  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu-adapters.so.13(_ZN5MueLu5Level3GetIN7Teuchos3RCPINS_12SmootherBaseIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_6SerialENS5_9HostSpaceEEEEEEEEERT_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKNS_11FactoryBaseE+0x13f) [0x7fe85c8eea5f]
11  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZNK5MueLu18TopSmootherFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelE+0x185) [0x7fe85bf80c15]
12  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu9HierarchyIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5SetupEiN7Teuchos3RCPIKNS_18FactoryManagerBaseEEESC_SC_+0x2877) [0x7fe85bc4ef87]
13  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu-interface.so.13(_ZNK5MueLu16HierarchyManagerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE14SetupHierarchyERNS_9HierarchyIdixS6_EE+0x16bc) [0x7fe85c4f76bc]
14  /home/lcm/LCM/lcm-build-serial-clang-release/src/libalbanyLib.so(_ZN5MueLu26CreateXpetraPreconditionerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEEEN7Teuchos3RCPINS_9HierarchyIT_T0_T1_T2_EEEENS8_IN6Xpetra6MatrixISA_SB_SC_SD_EEEERKNS7_13ParameterListE+0xd46) [0x7fe860a11956]
15  /home/lcm/LCM/lcm-build-serial-clang-release/src/libalbanyLib.so(_ZNK5Thyra26MueLuPreconditionerFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE14initializePrecERKN7Teuchos3RCPIKNS_18LinearOpSourceBaseIdEEEEPNS_18PreconditionerBaseIdEENS_16ESupportSolveUseE+0x43c6) [0x7fe860a08bc6]
16  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libnox.so.13(_ZNK3NOX5Thyra5Group10updateLOWSEv+0x4e4) [0x7fe85b016b44]

Curiously, the problem does not show up in the gcc build.

@lxmota : would you be able to provide instructions for how I can recreate the nightly clang build on algol to try to debug the problem in the serial run?

Element technology

Higher order surface elements and different integration strategies (48 point) for the composite tet.

ACE: add ACE_Cumulative_Time input/output field to Mechanics problem

Feature request from @jennifer-frederick . Behavior will be similar to ACE_bluff_salinity but within mechanics problem in ACE.

LCM/ACE: Neumann BCs are not set correctly for 'T' and 'all' variables for ThermoMechanics problems

When attempting to set Neumann BCs for temperature, Neumann BCs are actually set for the 2nd component of the displacement.

Something wonky is happening also when setting a Neumann BC for 'all' the variables. Lets say we want to set a Neumann BC of [a, b, c, d] for [sig_x, sig_y, sig_z, T] using the 'all' string. What will happen in the code is the following Neumann BC will be set: [sig_x, sig_y, sig_z, T] = [a, 0, 0, 0]. This is wrong!

Finally, there is no logic in the Mechanics problem (and perhaps other problems) to throw an error when trying to specify Neumann BCs on a mesh with no sidesets. Thus users may think they are specifying a Neumann BC b/c they have that section in their input file, but nothing is being done in the code.

I'm working on fixing all these issues now. Opening an issue to document the problem, in case others experience it / have experienced it.

Discuss a two-branch workflow

There are stale branches that have not been touched in years. These should be deleted so they do not clutter the repository.

In addition, mirroring what is done for Trilinos, a develop branch should be created that eventually incorporates all new code.

The master branch should be more stable so that Albany users can pull from it and have more confidence that the code will compile and run.

Failures in nightly builds following upgrade to Fedora 33

A handful of Albany/LCM tests began filing last night following an upgrade of algol/proxima to Fedora 33. Here are the tests:

BoreDemo
ClampedSDBC1DCt_AnalyticMass_NewmarkExplicitAForm_CompositeTet10_LumpedMass ClampedSDBC1DCt_AnalyticMass_NewmarkImplicitDForm_CompositeTet10_AnalyticMass
ClampedSDBC1DCt_AnalyticMass_NewmarkImplicitDForm_CompositeTet10_AnalyticMass
DTKInterp_Error
DTKInterp_InterpVolToNs
Dynamic_ClampedSDBC_NewmarkExplicitAForm_Tempus
Parallel_Dynamic_Cube_NewmarkExplicitAForm_LumpedMass_Tempus
ReactDiffSystem
Static_Body_Force

https://sems-cdash-son.sandia.gov/cdash/index.php?project=Albany_LCM&date=2020-11-11&filtercount=1&showfilters=1&field1=groupname&compare1=61&value1=Nightly

I suspect this is due to some incompatibility of the libraries, since checking out an older Trilinos or DTK leads to the same problem (tagging @gahansen who made changes recently to DTK even though I think that's not causing these errors) on these machines, and running the tests on a Fedora 32 machine does not have the failures. It is not the compiler b/c the errors happen with clang and gcc and gcc did not change when going from Fedora 32 to Fedora 33. Another useful piece of information is that the errors seem to happen only in parallel. Perhaps parallel-netcdf is the culprit? Tagging @lxmota who will help me investigate.

Here is how one of the errors looks like for the tests that outright fail:

p=1: *** Caught standard std::exception of type 'std::logic_error' :

 /home/lcm/LCM/Trilinos/packages/tpetra/core/src/Tpetra_DirectoryImpl_def.hpp:877:
 
 Throw number = 1
 
 Throw test that evaluated to true: curLID == LINVALID
 
 Tpetra::Details::DistributedNoncontiguousDirectory<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > constructor: Incoming global index 0 does not have a corresponding local index in the Directory Map.  Please report this bug to the Tpetra developers.

Update Albany-LCM wiki homepages

Wade Burgess from the SEMS team called to my attention that the Albany-LCM home page is out of date... we should clean it up to make it up-to-date. It should not take a tone of work.

Investigate roadblocks to agile LCM development

1. Proliferation of template parameters inhibits non experts from development
2. How to manage code complexity?

ACE: write parallel MiniErosion test in which Erosion occurs

Per email discussion with @jennifer-frederick . Opening issue so we don't forget!

Revive CEE nightly test

Requires moving test to one of cee compute nodes, since @lxmota 's dedicated workstation is no more.

Failing builds due to stk changes

Nightly LCM builds are failing due to apparent changes to STK:

/home/lcm/LCM/LCM/src/disc/stk/percept/stk_rebalance/GeomDecomp.cpp:54:58: error: no matching function for call to ‘stk::mesh::Field<double, stk::mesh::Cartesian3d>::max_size(const EntityRank&) const’
         const unsigned ndim(nodal_coor.max_size(NODE_RANK));  // TODO - is there a better way to get this info?
                                                          ^
In file included from /home/lcm/LCM/trilinos-install-serial-gcc-release/include/stk_mesh/base/Field.hpp:40,
                 from /home/lcm/LCM/LCM/src/./disc/stk/percept/stk_rebalance/Partition.hpp:44,
                 from /home/lcm/LCM/LCM/src/./disc/stk/percept/stk_rebalance/GeomDecomp.hpp:15,
                 from /home/lcm/LCM/LCM/src/disc/stk/percept/stk_rebalance/GeomDecomp.cpp:12:
/home/lcm/LCM/trilinos-install-serial-gcc-release/include/stk_mesh/base/FieldBase.hpp:147:12: note: candidate: ‘unsigned int stk::mesh::FieldBase::max_size() const’
   unsigned max_size() const;

Seems this was fixed by the following PR: sandialabs/Albany#967 . We'll need something similar in LCM.

I will try to fix this soon.

Make "Scale BC Dofs: true" option with scaling computed on the fly default for LCM?

This is a follow on issue to sandialabs/Albany#307 . Now that "Scale BC Dofs: true" is working correctly and I have added a version of it that computes / adapts the scaling on the fly, should we make this option default for LCM? One thing to consider is this sort of change may require rebaselining a lot of the tests that use exodiff (e.g., if the gold file has a residual field, which will change with the scaling).

Output variables are declared and stored on every element block

It seems that all output variables are stored on every element block. This is handled through an element block truth table in Exodus. This is not required and can lead to significantly larger output files if particular models (read crystal plasticity) have an order of magnitude more element variables than other models (read J2). I don't know how to fix it. I just stumbled onto this problem while debugging a failed test.

The solution would be to have a truth table in exodus that represents a block by block representation of the variables for that particular block. Many thanks.

FPEs in proxima/algol builds

Please don't shoot the messenger...!

The seacas issue that was causing the proxima/algol tests to be broken for awhile has been resolved, but now there are FPEs in a bunch of the tests: https://sems-cdash-son.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=1935 . Have any changes been made to the material models recently that would cause this? I don't think so but I thought I'd double check. If not, I'm not sure how we want to proceed with this...

P.S. again, please don't shoot the messenger! I know this is really frustrating!!

Failing LCM/ATO tests on Mayer (ARM machine)

The Mayer nightly tests might (fingers crossed...) be stable finally. It looks like some tests are failing there:

http://cdash.sandia.gov/CDash-2-3-0/viewTest.php?onlyfailed&buildid=73330

Schwarz_Cubes | Failed | 3s | Completed (Failed)
ATO:RegHeaviside_3D | Failed | 34s 650ms | Completed (Failed)
ATOT:RegHeaviside_3D | Failed | 42s 560ms | Completed (Failed)
CrystalPlasticity_MinisolverStep_Newton | Failed | 3s 720ms | Completed (Failed)
CrystalPlasticity_MinisolverStep_NewtonLineSearch | Failed | 3s 710ms | Completed (Failed)
CrystalPlasticity_MinisolverStep_TrustRegion | Failed | 3s 470ms | Completed (Failed)
CrystalPlasticity_ThermallyActivatedSlip | Failed | 4s 880ms | Completed (Failed)
CrystalPlasticity_SchwarzBar_modified_np1 | Failed | 3s 60ms | Completed (Failed)

The CrystalPlasticity ones might be just due to the tolerances being so tight (@lxmota , @jwfoulk : can you please check?) . Not sure what is going on with the ATO tests. The Schwarz tests throw the following exception

p=0: *** Caught standard std::exception of type 'std::invalid_argument' :

 /home/projects/albany/nightlyCDashTrilinos/repos/Trilinos/packages/tpetra/core/src/Tpetra_MultiVector_def.hpp:3679:
 
 Throw number = 1
 
 Throw test that evaluated to true: (j >= numCols)
 
 Tpetra::MultiVector<double,int,long long,Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>>::MultiVector(const MultiVector&, const size_t): Input index j (== 1) exceeds valid column index range [0, 1 - 1].
CMake Error at /home/projects/albany/nightlyCDashAlbany/repos/Albany/tests/large/LCM/Schwarz/runtest.cmake:14 (message):
  Albany didn't run: test failed
`````````````````````````````````````
I'll investigate this when I have time. 

Create lean and simplified LCM main.

Create an LCM main that has no Epetra or other application code to make it as simple and lean as possible. The complexity of the main files interferes with agile development, as was pointed out to me in a recent meeting with potential LCM developers. It's imperative that we remove this barriers for wider LCM adoption. Related to https://github.com/gahansen/Albany/issues/22, but deserves its own issue.

Use Cn instead of C2 basis that will be deprecated + cleaning

More detail found here: sandialabs/Albany#800 . We need to do the same thing for LCM as Mauro did for Albany.

Albany LCM CDash issues

Due to a change in the SEMS-supported CDash sites, CEE can no longer post to the site due to firewall issues. So one will not see the cee nightly build on the CDash site for now.

A consequence of the change is that the post-processing script for querying the SON CDash site and sending an email is broken too. There is a SEMS ticket out on this, which @bartlettroscoe is working on.

Failing tests in nightly LCM builds

See https://sems-cdash-son.sandia.gov/cdash/viewTest.php?onlydelta&buildid=48998 . It appears it's due to changes in syntax w.r.t. Stratimikos in the input file. I will fix this by looking at how we deal with this in Albany master.

Fracture in parallel

It would be great to have adaptive insertion of cohesive elements for fracture in parallel. More than an issue this is a feature request. I could transfer all my research activities to LCM if this feature was available.

Robin (convection) boundary conditions not working for LCM Mechanics 3D problem

The current implementation of Mechanics 3D problem in LCM does not includes robin boundary conditions. I would like to have ideas on how to remove this limitation. At this moment we cannot solve thermo-mechanics problems with convection boundary conditions.

ACE: add writing / reading of bluff_salinity_ field to ACE Thermal problem

Feature request from @jennifer-frederick . Field needs to be read/modified in ACE_ThermalParameters_Def.hpp.

LCM test failures on some platforms due to exodiff

A few LCM tests have been failing on some of the testing platforms for months now due to the exodiff comparison at the end of the run "failing". The culprits are:

CrystalPlasticity_MinisolverStep_Newton - fails on Mayer, AlbanyIntel (CEE)
CrystalPlasticity_MinisolverStep_NewtonLineSearch - fails on Mayer, AlbanyIntel (CEE)
CrystalPlasticity_MinisolverStep_TrustRegion - fails on Mayer, AlbanyIntel (CEE)
CrystalPlasticity_ThermallyActivatedSlip - fails on Mayer, Blake (Serial and OpenMP build),
HeliumODEs_HeBubblesDecay - fails on Blake (Serial and OpenMP build)

I would like to see the dashboard clean, as having tests failing consistently for months makes it more difficult to spot new failures / monitor code health. Can one of the LCM folks (@lxmota , @calleman21 , @jwfoulk ) please have a look at the output from the tests to determine if the failures are trivial or not? I think some of the failures may be due to having a very small tolerance passed to exodiff, which may not be quite satisfied for all platforms. The following are ways we could move forward:

1.) Increase the exodiff tolerance so that the tests pass everywhere (if it is determined that the results on the platforms where the tests fail are OK).
2.) Create a machine-specific exodiff file for the problematic platforms (again, if it is determined that the results on the platforms where the tests fail are OK)
3.) Add a configure flag to turn off the problematic tests, and activate it on the problematic platforms.

3.) is not the best option but I will do this if there is no clear path towards getting the tests passing.

One can see the output from the failures here:

• http://cdash.sandia.gov/CDash-2-3-0/viewTest.php?onlyfailed&buildid=79283
• https://my.cdash.org/viewTest.php?onlyfailed&buildid=1579607

Parallel DynamicsTempus tests hang on Fedora 28 with gcc/8.1.1, it appears due to the Project IP to Nodal Field response

There is a problem with the DynamicsTempus tests, for instance Dynamic_ClampedSDBC_NewmarkImplicitDForm_Tempus . Specifically the problem seems to occur for parallel runs for the response computation. For this test case, what happens is the problem hangs when run on > 3 procs, or there is an MPI wait all error:

1: [camobap:7295] *** An error occurred in MPI_Waitall
1: [camobap:7295] *** reported by process [2300182529,2]
1: [camobap:7295] *** on communicator MPI_COMM_WORLD
1: [camobap:7295] *** MPI_ERR_TRUNCATE: message truncated
1: [camobap:7295] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

(see for example
http://cdash.sandia.gov/CDash-2-3-0/testDetails.php?test=3946016&build=76064 (hang)
http://cdash.sandia.gov/CDash-2-3-0/testDetails.php?test=3944285&build=76052 (MPI Waitall error)
)
Using print statements, I've identified that the hanging/error happens right after the following line:

https://github.com/gahansen/Albany/blob/master/src/Albany_Application.cpp#L2837

The following tests are affected:

Parallel_Dynamic_Cubes_NewmarkImplicitAForm_Tempus 
Dynamic_ClampedSDBC_NewmarkImplicitDForm_Tempus 
Dynamic_ClampedSDBC_NewmarkExplicitAForm_BLMesh_Tempus 
Dynamic_Torsion_NewmarkImplicitAForm_Tempus 
Dynamic_Torsion_NewmarkExplicitAForm_Tempus 

I'll post more info on when this started that might help ID the problem shortly.

More stringent comparisons in alternating Schwarz tests

Currently the alternating Schwarz tests just check if the Schwarz steps converged, which may not catch issues introduced into the code. It would be best to check comparisons. Is there a way to do this without using exodiff w.r.t. a gold file?

Row of Zeros in Jacobian for MechanicsWithTemperature Problem

I'm trying to add temperature as a DOF to the same panel-problem I've been working on. (Same as what I presented on back at the user's meeting.) I tried to follow this example but I'm getting an error that reads:

SuperLU's gsequ function returned with info = 4 > 0, and info <= A.nrow = 18904.  This means that row 4 of A is exactly zero.

I take this to mean there's a row of all zeros in my Jacobian; is that correct? My first thought was to check the BC's. But checking and double checking, both the mechanics and thermal problems have enough Dirichlet BC's to ground their respective problems. Any advice on what to look at next?

For completeness:
I'm using this version of Albany: 9f4290e
With this version of Trilinos: trilinos/Trilinos@a0ba716
My full output file is attached as well as the input and material yamls and the mesh (.msh) file.
mesh_and_lable_plate.msh.TXT
output.txt
input.yaml.TXT
material.yaml.TXT

LCM: Time evaluator gives negative deltaTime after STKDiscretization::writeSolution

The value of deltaTime(0) is negative after STKDiscretization::writeSolution(...) is called. I think this may be from workset.current_time being set to zero after the write here.

This is an issue for me as the creep model uses this deltaTime(0) value to compute its state and I happen to be using Gmsh meshes which are built into STK.

I made a commit on a separate branch that shows which other tests could be affected by this. I just added a simple Teuchos check for negative deltaTime(0) values.

Any ideas on a solution or temporary workaround?

Removal of the FieldNameMap class

The FieldNameMap class was added to support the use case where we have a model with multiple element blocks with differing integration rules. At the time, we could not register and output state variables with the same name in both blocks (a common use case), but with differing numbers of integration points.

As a developer, I want to scope out the removal and replacement of the FieldNameMap to reduce code complexity and confusion.

Negative Time Monitor times reported for long-time Tempus Newmark-Beta scheme called from Albany

This is related to issue sandialabs/Albany#11. Testing of Tempus Newmark-Beta scheme shows for long-time runs, some of the time-monitor times reported are negative, more so for explicit than implicit. Solution displacements appear correct. This issue may be due to the linear solve, from inspecting the specific timing numbers, but needs to be tracked down.

Numerous failing tests due to deprecation of status test parameter in NOX

I will fix. Here is the error.

p=0: *** Caught standard std::exception of type 'Teuchos::Exceptions::InvalidParameterName' :

 Error, the parameter {name="Scale Type",type="string",value="Unscaled"}
 in the parameter (sub)list "Albany Parameters->Piro->NOX->Status Tests->Test 0"
 was not found in the list of valid parameters!
 
 The valid parameters and types are:
   {
     "Test Type" : string = RelativeNormF
     "Tolerance" : double = 1e-08
     "Scale Norms by Length" : bool = 0
     "Norm Type" : string = Two Norm
     "Tag" : string = 
   }

Schwarz coupling involving explicit Newmark stepper is broken

I discovered yesterday that implicit-explicit Schwarz coupling is broken in LCM. It slipped through the cracks due to us not having a test of the capability. One of the issues was in Piro, which I have fixed in this PR: trilinos/Trilinos#9935 . There is still a problem, it looks like in StrongSchwarzBC_Def.hpp when calling getNumEquations on this_app. Below is the full backtrace of the error. @lxmota I might ask you to look at this since you put in the this_app/coupled_app logic and relevant routines.

(gdb) run clamped_impl-expl.yaml 
Starting program: /home/ikalash/ACE/Albany-LCM-master/build-dbg/tests/LCM/Schwarz/Alternating/Dynamics/Clamped/Albany clamped_impl-expl.yaml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Detaching after fork from child process 3062340]
[New Thread 0x7fffd7b7c640 (LWP 3062344)]
[New Thread 0x7fffd71b7640 (LWP 3062345)]
[New Thread 0x7fffd5908640 (LWP 3062350)]

   ###    ##       ########     ###    ##    ## ##    ##
  ## ##   ##       ##     ##   ## ##   ###   ##  ##  ## 
 ##   ##  ##       ##     ##  ##   ##  ####  ##   ####  
##     ## ##       ########  ##     ## ## ## ##    ##   
######### ##       ##     ## ######### ##  ####    ##   
##     ## ##       ##     ## ##     ## ##   ###    ##   
##     ## ######## ########  ##     ## ##    ##    ##   

** Trilinos git commit id - 37e9418d718
** Albany git branch ------ master
** Albany git commit id --- 37bc00f41
** Albany cxx compiler ---- GNU 11.2.1
** Simulation start time -- 2021-11-16 at 10:52:19
***************************************************************
Problem Name = Mechanics 3D
Solution Method = Transient Tempus
Mechanics problem:
	Spatial dimension             : 3
	Mechanics variables           : DOF
	Temperature variables         : None
	Dislocation Density variables : None
	Pore Pressure variables       : None
	Transport variables           : None
	HydroStress variables         : None
	Damage variables              : None
	Stabilized Pressure variables : None
Initializing material database from material_00.yaml
TmplSTKMeshStruct:: Creating 3D mesh of size 1x1x75 elements and scaled to 0.01x0.01x0.75
Num MeshSpecs: 1
Calling MechanicsProblem::buildEvaluators
Calling MechanicsProblem::constructDirichletEvaluators
Calling MechanicsProblem::constructNeumannEvaluators
[GenericSTKMeshStruct] Processing field requirements...
STKDisc: 75 elements on Proc 0 
STKDisc: nodeset NodeSet0 has size 152  on Proc 0.
STKDisc: nodeset NodeSet1 has size 152  on Proc 0.
STKDisc: nodeset NodeSet2 has size 152  on Proc 0.
STKDisc: nodeset NodeSet3 has size 152  on Proc 0.
STKDisc: nodeset NodeSet4 has size 4  on Proc 0.
STKDisc: nodeset NodeSet5 has size 4  on Proc 0.
STKDisc: nodeset NodeSet99 has size 1  on Proc 0.
STKDisc: sideset SideSet0 has size 75  on Proc 0.
STKDisc: sideset SideSet1 has size 75  on Proc 0.
STKDisc: sideset SideSet2 has size 75  on Proc 0.
STKDisc: sideset SideSet3 has size 75  on Proc 0.
STKDisc: sideset SideSet4 has size 1  on Proc 0.
STKDisc: sideset SideSet5 has size 1  on Proc 0.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Sacado ParameterLibrary has been initialized:
 Library of all registered parameters:
	SDBC on NS NodeSet0 for DOF X:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet1 for DOF X:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet2 for DOF Y:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet3 for DOF Y:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet4 for DOF Z:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet5 for DOF StrongSchwarz:  Supports AD = 1, Supports_Analytic = 0
	Time:  Supports AD = 1, Supports_Analytic = 0
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Number of parameter vectors  = 0
Number of response vectors  = 1
Number of distributed parameters vectors  = 0

A) Get the base parameter list ...

    Using Tempus 'Time Step Control'.

C) Create and initalize the forward model ...

D) Create the stepper and integrator for the forward problem ...
 Warning -- Did not find a Tempus Time Events to create!
 Should call add() with this (app-specific?) Time Event.
 
Problem Name = Mechanics 3D
Solution Method = Transient Tempus
Mechanics problem:
	Spatial dimension             : 3
	Mechanics variables           : DOF
	Temperature variables         : None
	Dislocation Density variables : None
	Pore Pressure variables       : None
	Transport variables           : None
	HydroStress variables         : None
	Damage variables              : None
	Stabilized Pressure variables : None
Initializing material database from material_01.yaml
TmplSTKMeshStruct:: Creating 3D mesh of size 1x1x75 elements and scaled to 0.01x0.01x0.75
Num MeshSpecs: 1
Calling MechanicsProblem::buildEvaluators
Calling MechanicsProblem::constructDirichletEvaluators
Calling MechanicsProblem::constructNeumannEvaluators
[GenericSTKMeshStruct] Processing field requirements...
STKDisc: 75 elements on Proc 0 
STKDisc: nodeset NodeSet0 has size 152  on Proc 0.
STKDisc: nodeset NodeSet1 has size 152  on Proc 0.
STKDisc: nodeset NodeSet2 has size 152  on Proc 0.
STKDisc: nodeset NodeSet3 has size 152  on Proc 0.
STKDisc: nodeset NodeSet4 has size 4  on Proc 0.
STKDisc: nodeset NodeSet5 has size 4  on Proc 0.
STKDisc: nodeset NodeSet99 has size 1  on Proc 0.
STKDisc: sideset SideSet0 has size 75  on Proc 0.
STKDisc: sideset SideSet1 has size 75  on Proc 0.
STKDisc: sideset SideSet2 has size 75  on Proc 0.
STKDisc: sideset SideSet3 has size 75  on Proc 0.
STKDisc: sideset SideSet4 has size 1  on Proc 0.
STKDisc: sideset SideSet5 has size 1  on Proc 0.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Sacado ParameterLibrary has been initialized:
 Library of all registered parameters:
	SDBC on NS NodeSet0 for DOF X:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet1 for DOF X:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet2 for DOF Y:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet3 for DOF Y:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet4 for DOF StrongSchwarz:  Supports AD = 1, Supports_Analytic = 0
	SDBC on NS NodeSet5 for DOF Z:  Supports AD = 1, Supports_Analytic = 0
	Time:  Supports AD = 1, Supports_Analytic = 0
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Number of parameter vectors  = 0
Number of response vectors  = 1
Number of distributed parameters vectors  = 0

A) Get the base parameter list ...

    Using Tempus 'Time Step Control'.

C) Create and initalize the forward model ...

D) Create the stepper and integrator for the forward problem ...
 Warning -- Did not find a Tempus Time Events to create!
 Should call add() with this (app-specific?) Time Event.
 

Thread 1 "Albany" received signal SIGSEGV, Segmentation fault.
0x00007ffff6f9e89c in Albany::Application::getNumEquations (this=0x0)
    at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.hpp:157
157	    return neq;
Missing separate debuginfos, use: dnf debuginfo-install blas-3.9.0-6.fc34.x86_64 cyrus-sasl-lib-2.1.27-8.fc34.x86_64 glibc-2.33-20.fc34.x86_64 hdf-4.2.15-4.fc34.x86_64 hdf-libs-4.2.15-4.fc34.x86_64 hdf5-1.10.6-5.fc34.x86_64 hwloc-libs-2.4.1-3.fc34.x86_64 infinipath-psm-3.3-26_g604758e_open.6.fc34.3.x86_64 keyutils-libs-1.6.1-2.fc34.x86_64 krb5-libs-1.19.2-2.fc34.x86_64 lapack-3.9.0-6.fc34.x86_64 libaec-1.0.5-1.fc34.x86_64 libbrotli-1.0.9-4.fc34.x86_64 libcurl-7.76.1-7.fc34.x86_64 libevent-2.1.12-3.fc34.x86_64 libfabric-1.12.1-1.fc34.x86_64 libgcc-11.2.1-1.fc34.x86_64 libgfortran-11.2.1-1.fc34.x86_64 libibverbs-36.0-1.fc34.x86_64 libnghttp2-1.43.0-2.fc34.x86_64 libpsl-0.21.1-3.fc34.x86_64 libquadmath-11.2.1-1.fc34.x86_64 librdmacm-36.0-1.fc34.x86_64 libssh-0.9.5-2.fc34.x86_64 libstdc++-11.2.1-1.fc34.x86_64 libtirpc-1.3.2-0.fc34.x86_64 libunistring-0.9.10-10.fc34.x86_64 libxcrypt-4.4.25-1.fc34.x86_64 netcdf-4.7.3-6.fc34.x86_64 numactl-libs-2.0.14-3.fc34.x86_64 openldap-2.4.57-5.fc34.x86_64 openmpi-4.1.0-5.fc34.x86_64 openssl-libs-1.1.1l-1.fc34.x86_64 pcre2-10.36-4.fc34.x86_64 pmix-3.2.3-1.fc34.x86_64 ucx-1.10.1-1.fc34.x86_64 zlib-1.2.11-26.fc34.x86_64
(gdb) bt
#0  0x00007ffff6f9e89c in Albany::Application::getNumEquations (this=0x0)
    at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.hpp:157
#1  0x00007ffff2ef0d22 in LCM::StrongSchwarzBC_Base<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>::computeBCsDTK (
    this=0xbd8c00) at /home/ikalash/ACE/Albany-LCM-master/src/LCM/evaluators/bc/StrongSchwarzBC_Def.hpp:280
#2  0x00007ffff2eef125 in LCM::fillSolution<LCM::StrongSchwarzBC<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>, PHAL::AlbanyTraits> (sbc=..., workset=...) at /home/ikalash/ACE/Albany-LCM-master/src/LCM/evaluators/bc/StrongSchwarzBC_Def.hpp:468
#3  0x00007ffff2eee069 in LCM::StrongSchwarzBC<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>::preEvaluate (
    this=0xbd8c00, workset=...) at /home/ikalash/ACE/Albany-LCM-master/src/LCM/evaluators/bc/StrongSchwarzBC_Def.hpp:612
#4  0x00007ffff6f36f72 in PHX::DagManager<PHAL::AlbanyTraits>::preEvaluate (this=0xba94e8, d=...)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Phalanx_DAG_Manager_Def.hpp:641
#5  0x00007ffff6f33a7f in PHX::EvaluationContainer<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>::preEvaluate (
    this=0xba94e0, d=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Phalanx_EvaluationContainer_Def.hpp:502
#6  0x00007ffff6ec3612 in PHX::FieldManager<PHAL::AlbanyTraits>::preEvaluate<PHAL::AlbanyTraits::Residual> (this=0xbe0e60, 
    d=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Phalanx_FieldManager_Def.hpp:349
#7  0x00007ffff6eaf3ec in Albany::Application::computeGlobalResidualImpl (this=0xb3bd20, current_time=0, x=..., x_dot=..., 
    x_dotdot=..., p=..., f=..., dt=0) at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.cpp:1142
#8  0x00007ffff6eb03c7 in Albany::Application::computeGlobalResidual (this=0xb3bd20, current_time=0, x=..., x_dot=..., 
    x_dotdot=..., p=..., f=..., dt=0) at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.cpp:1227
#9  0x00007ffff6f55a84 in Albany::ModelEvaluator::evalModelImpl (this=0xf91190, inArgs=..., outArgs=...)
    at /home/ikalash/ACE/Albany-LCM-master/src/Albany_ModelEvaluator.cpp:679
#10 0x00007ffff6b69809 in Thyra::ModelEvaluatorDefaultBase<double>::evalModel (this=0xf91190, inArgs=..., outArgs=...)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_ModelEvaluatorDefaultBase.hpp:709
#11 0x00007ffff6b80251 in Thyra::DefaultModelEvaluatorWithSolveFactory<double>::evalModelImpl (this=0xebbc90, inArgs=..., 
    outArgs=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_DefaultModelEvaluatorWithSolveFactory.hpp:325
#12 0x00007ffff6b69809 in Thyra::ModelEvaluatorDefaultBase<double>::evalModel (this=0xebbcd8, inArgs=..., outArgs=...)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_ModelEvaluatorDefaultBase.hpp:709
#13 0x00007ffff6b79e3d in Piro::InvertMassMatrixDecorator<double>::evalModelImpl (this=0xec2910, inArgs=..., outArgs=...)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_InvertMassMatrixDecorator_Def.hpp:348
#14 0x00007ffff6b69809 in Thyra::ModelEvaluatorDefaultBase<double>::evalModel (this=0xec2910, inArgs=..., outArgs=...)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_ModelEvaluatorDefaultBase.hpp:709
#15 0x00007fffefc01caf in Tempus::StepperExplicit<double>::evaluateExplicitODE (this=0xf2c850, xDotDot=..., x=..., xDot=..., 
    time=0, p=...) at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_StepperExplicit_impl.hpp:313
#16 0x00007fffefcb1d32 in Tempus::StepperNewmarkExplicitAForm<double>::setInitialConditions (this=0xf2c820, 
    solutionHistory=...) at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_StepperNewmarkExplicitAForm_impl.hpp:187
#17 0x00007fffefb3acf8 in Tempus::IntegratorBasicOld<double>::initializeSolutionHistory (this=0xe1d2c0, state=...)
    at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:149
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x00007fffefb3b563 in Tempus::IntegratorBasicOld<double>::setSolutionHistory (this=0xe1d2c0, sh=...)
    at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:208
#19 0x00007fffefb3c44f in Tempus::IntegratorBasicOld<double>::initialize (this=0xe1d2c0)
    at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:292
#20 0x00007fffefb36386 in Tempus::IntegratorBasicOld<double>::IntegratorBasicOld (this=0xe1d2c0, inputPL=..., model=..., 
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:28
#21 0x00007fffefb3eef3 in Tempus::integratorBasic<double> (pList=..., model=...)
    at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:667
#22 0x00007ffff6b38522 in Piro::TempusIntegrator<double>::TempusIntegrator (this=0xf2c7b0, pList=..., model=..., 
    sens_method=Piro::NONE) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusIntegrator_Def.hpp:63
#23 0x00007ffff6b29cc6 in Piro::TempusSolver<double>::initialize (this=0xe57d60, appParams=..., in_model=..., 
    in_adjointModel=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusSolver_Def.hpp:342
#24 0x00007ffff6b18bb3 in Piro::TempusSolver<double>::TempusSolver (this=0xe57d60, appParams=..., in_model=..., 
    in_adjointModel=..., piroObserver=..., __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusSolver_Def.hpp:120
#25 0x00007ffff6b0af29 in Piro::tempusSolver<double> (appParams=..., in_model=..., in_adjointModel=..., piroObserver=...)
    at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusSolver_Def.hpp:755
#26 0x00007ffff6afe5a1 in Piro::SolverFactory::createSolver<double> (this=0x7fffffff9d7f, piroParams=..., model=..., 
    adjointModel=..., observer=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_SolverFactory_Def.hpp:159
#27 0x00007ffff6ad41ae in Albany::SolverFactory::createAndGetAlbanyApp (this=0x7fffffffa520, albanyApp=..., appComm=..., 
    solverComm=..., initial_guess=..., createAlbanyApp=true)
    at /home/ikalash/ACE/Albany-LCM-master/src/Albany_SolverFactory.cpp:231
#28 0x00007ffff2bca198 in LCM::SchwarzAlternating::SchwarzAlternating (this=0x78bfb0, app_params=..., comm=..., 
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at /home/ikalash/ACE/Albany-LCM-master/src/LCM/solvers/Schwarz_Alternating.cpp:196
#29 0x00007ffff6ad3869 in Albany::SolverFactory::createAndGetAlbanyApp (this=0x7fffffffbbd0, albanyApp=..., appComm=..., 
    solverComm=..., initial_guess=..., createAlbanyApp=true)
    at /home/ikalash/ACE/Albany-LCM-master/src/Albany_SolverFactory.cpp:184
#30 0x0000000000466d48 in main (argc=2, argv=0x7fffffffd048) at /home/ikalash/ACE/Albany-LCM-master/src/Main_Solve.cpp:101

Get DTK to work with Tpetra deprecated code turned off

I discovered DTK does not work with

  -D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \
  -D Xpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \

due to the doPostsAndWaits routine in Tpetra having been rewritten to work with Kokkos:Views rather than Teuchos:Arrays (see trilinos/Trilinos#10202). Need to fix this this week so that LCM does not break when the Tpetra deprecated code is removed from Trilinos.

Explore enabling full support for PUMI in LCM to work around STK

For many years STK has caused a lot of pain and grief for Albany development and LCM in particular. See a small example here pointed out by @jtostie :
trilinos/Trilinos#451

Explore the possibility to enable full support for PUMI in LCM so that it can be used as an alternative to STK as a mesh database.

Besides the huge amount of work that this would involve, it is also necessary to make it possible to read exodus meshes from PUMI, otherwise this is DOA.

Project IP to Nodal Field fails with Composite Tets

@ikalash

Project IP to Nodal Field for computing an L2 projection of the stress to the nodes fails for composite tetrahedral elements.

This is a long-standing issue that we need to fix, as users keep running into it.

FPEs in some of the ACE nightlies

Looks like there are FPEs in some of the ACE nightlies starting today: https://sems-cdash-son.sandia.gov/cdash/index.php?project=Albany_LCM&date=2022-03-25&filtercount=1&showfilters=1&field1=groupname&compare1=61&value1=Nightly . @lxmota , could this be due to your cosmetic changes check-in yesterday? Hopefully it's that and not something in Trilinos.

Failing ACE MiniErosion Denudation test in parallel

The ACE_MiniErosion_Denudation_Parallel is failing with the following cryptic STK-related error:

3: STKDisc: sideset bottom has size 0  on Proc 0.
3: 
3: p=0: *** Caught standard std::exception of type 'std::logic_error' :
3: 
3:  Requirement( buf.remaining() ) FAILED
3:  Error occurred at: stk_mesh/stk_mesh/baseImpl/MeshCommImplUtils.cpp:215
3:   0# stk::output_stacktrace(std::ostream&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_util_util.so.13
3:   1# stk::mesh::impl::unpack_induced_parts_from_sharers(std::vector<unsigned int, std::allocator<unsigned int> >&, std::vector<stk::mesh::EntityCommInfo, std::allocator<stk::mesh::EntityCommInfo> > const&, stk::CommSparse&, stk::mesh::EntityKey) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   2# stk::mesh::BulkData::remove_unneeded_induced_parts(stk::mesh::Entity, std::vector<stk::mesh::EntityCommInfo, std::allocator<stk::mesh::EntityCommInfo> > const&, stk::mesh::PartStorage&, stk::CommSparse&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   3# stk::mesh::BulkData::internal_resolve_shared_part_membership_for_element_death() in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   4# stk::mesh::BulkData::internal_resolve_sharing_and_ghosting_for_sides(bool) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   5# stk::mesh::BulkData::make_mesh_parallel_consistent_after_skinning(std::vector<stk::mesh::sharing_info, std::allocator<stk::mesh::sharing_info> > const&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   6# stk::mesh::FaceCreator::create_side_entities_given_sideset(stk::mesh::SideSet const&, std::vector<stk::mesh::Part*, std::allocator<stk::mesh::Part*> > const&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   7# stk::mesh::create_all_sides(stk::mesh::BulkData&, stk::mesh::Selector const&, std::vector<stk::mesh::Part*, std::allocator<stk::mesh::Part*> > const&, bool) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   8# stk::mesh::create_adjacent_entities(stk::mesh::BulkData&, std::vector<stk::mesh::Part*, std::allocator<stk::mesh::Part*> >&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   9# LCM::Topology::createAllLevelsRelations() in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  10# LCM::Topology::graphInitialization() in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  11# LCM::Topology::Topology(Teuchos::RCP<Albany::AbstractDiscretization>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  12# AAdapt::Erosion::Erosion(Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Sacado::ScalarParameterLibrary<SPL_Traits> > const&, Albany::StateManager const&, Teuchos::RCP<Teuchos::Comm<int> const> const&) in /home/ikalash/Albany-LCM/build/src/adapt/libalbanyAdapt.so
3:  13# AAdapt::AdaptiveSolutionManager::buildAdapter(Teuchos::RCP<AAdapt::rc::Manager> const&) in /home/ikalash/Albany-LCM/build/src/adapt/libalbanyAdapt.so
3:  14# AAdapt::AdaptiveSolutionManager::AdaptiveSolutionManager(Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, Teuchos::RCP<Sacado::ScalarParameterLibrary<SPL_Traits> > const&, Albany::StateManager const&, Teuchos::RCP<AAdapt::rc::Manager> const&, Teuchos::RCP<Teuchos::Comm<int> const> const&) in /home/ikalash/Albany-LCM/build/src/adapt/libalbanyAdapt.so
3:  15# Albany::Application::finalSetUp(Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  16# Albany::Application::Application(Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, bool) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  17# Albany::SolverFactory::createAlbanyAppAndModel(Teuchos::RCP<Albany::Application>&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, bool) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  18# Albany::SolverFactory::createAndGetAlbanyApp(Teuchos::RCP<Albany::Application>&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, bool) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  19# LCM::ACEThermoMechanical::createThermalSolverAppDiscME(int, double) const in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  20# LCM::ACEThermoMechanical::ThermoMechanicalLoopDynamics() const in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  21# Thyra::ModelEvaluatorDefaultBase<double>::evalModel(Thyra::ModelEvaluatorBase::InArgs<double> const&, Thyra::ModelEvaluatorBase::OutArgs<double> const&) const in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  22# void Piro::Detail::PerformSolveImpl<double, Thyra::VectorBase<double> const, Thyra::MultiVectorBase<double> const>(Thyra::ModelEvaluator<double> const&, Teuchos::Array<bool> const&, bool, Teuchos::Array<Teuchos::RCP<Thyra::VectorBase<double> const> >&, Teuchos::Array<Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> > >&, Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> >&, Teuchos::Array<Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> > >&, Teuchos::RCP<Piro::SolutionObserverBase<double, Thyra::VectorBase<double> const> >) in /home/ikalash/Albany-LCM/build/src/Albany
3:  23# void Piro::Detail::PerformSolveImpl<double, Thyra::VectorBase<double> const, Thyra::MultiVectorBase<double> const>(Thyra::ModelEvaluator<double> const&, Teuchos::ParameterList&, Teuchos::Array<Teuchos::RCP<Thyra::VectorBase<double> const> >&, Teuchos::Array<Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> > >&) in /home/ikalash/Albany-LCM/build/src/Albany
3:  24# main in /home/ikalash/Albany-LCM/build/src/Albany
3:  25# __libc_start_main in /lib64/libc.so.6
3:  26# _start in /home/ikalash/Albany-LCM/build/src/Albany
3:  
3:  Error: P0 empty buf, expected to recv parts for: (FACE_RANK,966) from proc 1
3:  

The error happens immediately after the problem starts, and is therefore not an issue related to erosion. The problem does not appear to be related to recent changes in Albany, as the same error is encountered w/o the changes if the .jou file is modified to merge together the mesh, something that was not being done before (leading to incorrect behavior that fell through the cracks).

Some LCM tests sensitive to OMP settings

I was chasing down the issue with a failing test (MechanicsTensileCT), and I finally found the problem: the test passes with OMP_NUM_THREADS=1, but fails otherwise. I am not sure where the problem is, but some of the LCM folks might want to look into this.

Some extra details. With more than 1 thread, the tests fail due to a failing regression test. Something like

Response Test 0: 3.066335674548e-05 != 3.068000000000e-05 (rel 1.000000000000e-05 abs 1.000000000000e-08)

The computed value is deterministic, in the sense that different runs with the same number of threads lead to the same response value. However, different number of threads lead to different response values. This makes me think it is not necessarily a race condition, but I'm not really sure how to explain it otherwise.

Note: this happens on my machine also with master. Compiling with gcc 6.1.0, against Trilinos develop 8e9d4ba . It is not the most recent Trilinos, but I keep 'update trilinos' always as my last resort. Besides, the testsuite is otherwise passing, so I think it is not an issue with Trilinos...

Tagging some LCM people: @ikalash @lxmota

Implement Linear Algebra Modifiers

This issue actually encompasses many other things including sandialabs/Albany#68, sandialabs/Albany#77, and SNLComputation/Albany#13. It is a breakaway from trilinos/Trilinos#1204. We'd like to have the option to make several different kinds of modifications to the linear algebra structures:

1. For each Dirichlet row, zero out all but the diagonal, set the RHS entry to the diagonal entry times the DBC value. This is different from what we do now, which is a one on the diagonal. The difference was suggested by @bgranzow . We could also keep an option to do either.

2. For each Dirichlet column, subtract the column times the DBC value from the RHS, then zero out the non-diagonal column terms. This actually doesn't change the Residual.

3. For each Dirichlet global element, remove it from the linear system. This is basically a reconstruction of the Jacobian and Residual structures, as well as the Solution vector.

4. For each Dirichlet global element, set the value directly in the original Solution vector.

5. Scale all rows of the linear system by some statistic of the row

Each of these stages could be on or off, for example it some of the earlier modifications aren't necessary if the constrained elements will just be removed from the system altogether.

The first obstacle to this has been determining what state the non-linear solver is in from a Thyra::ModelEvaluator level (see sandialabs/Albany#77), but it seems from trilinos/Trilinos#1204 that the first thing we should try is using a NOX PrePostOperator to simply inform Albany of non-linear solver state.

Need to modify/rebaseline ACE_MiniErosion_WavePressNBC_Serial

With last week's bug fixes to the wave pressure NBC tests, the ACE_MiniErosion_WavePressNBC_Serial test is no longer showing erosion. The test needs to be modified to have erosion and rebaselined. @jennifer-frederick will do the former, I will do the latter.

Condensed representation for Dirichlet BCs

We would like to be able to construct the linear system without having to insert values of 1.0 in the system associated with the essential BCs.

Periodic boundary conditions

There is a need to model periodic microstructures in determining appropriate RVE sizes and effects. It would be useful to current projects for Albany/LCM to have this capability.