I discovered yesterday that implicit-explicit Schwarz coupling is broken in LCM. It slipped through the cracks due to us not having a test of the capability. One of the issues was in Piro, which I have fixed in this PR: trilinos/Trilinos#9935 . There is still a problem, it looks like in StrongSchwarzBC_Def.hpp when calling getNumEquations on this_app. Below is the full backtrace of the error. @lxmota I might ask you to look at this since you put in the this_app/coupled_app logic and relevant routines.
(gdb) run clamped_impl-expl.yaml
Starting program: /home/ikalash/ACE/Albany-LCM-master/build-dbg/tests/LCM/Schwarz/Alternating/Dynamics/Clamped/Albany clamped_impl-expl.yaml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Detaching after fork from child process 3062340]
[New Thread 0x7fffd7b7c640 (LWP 3062344)]
[New Thread 0x7fffd71b7640 (LWP 3062345)]
[New Thread 0x7fffd5908640 (LWP 3062350)]
### ## ######## ### ## ## ## ##
## ## ## ## ## ## ## ### ## ## ##
## ## ## ## ## ## ## #### ## ####
## ## ## ######## ## ## ## ## ## ##
######### ## ## ## ######### ## #### ##
## ## ## ## ## ## ## ## ### ##
## ## ######## ######## ## ## ## ## ##
** Trilinos git commit id - 37e9418d718
** Albany git branch ------ master
** Albany git commit id --- 37bc00f41
** Albany cxx compiler ---- GNU 11.2.1
** Simulation start time -- 2021-11-16 at 10:52:19
***************************************************************
Problem Name = Mechanics 3D
Solution Method = Transient Tempus
Mechanics problem:
Spatial dimension : 3
Mechanics variables : DOF
Temperature variables : None
Dislocation Density variables : None
Pore Pressure variables : None
Transport variables : None
HydroStress variables : None
Damage variables : None
Stabilized Pressure variables : None
Initializing material database from material_00.yaml
TmplSTKMeshStruct:: Creating 3D mesh of size 1x1x75 elements and scaled to 0.01x0.01x0.75
Num MeshSpecs: 1
Calling MechanicsProblem::buildEvaluators
Calling MechanicsProblem::constructDirichletEvaluators
Calling MechanicsProblem::constructNeumannEvaluators
[GenericSTKMeshStruct] Processing field requirements...
STKDisc: 75 elements on Proc 0
STKDisc: nodeset NodeSet0 has size 152 on Proc 0.
STKDisc: nodeset NodeSet1 has size 152 on Proc 0.
STKDisc: nodeset NodeSet2 has size 152 on Proc 0.
STKDisc: nodeset NodeSet3 has size 152 on Proc 0.
STKDisc: nodeset NodeSet4 has size 4 on Proc 0.
STKDisc: nodeset NodeSet5 has size 4 on Proc 0.
STKDisc: nodeset NodeSet99 has size 1 on Proc 0.
STKDisc: sideset SideSet0 has size 75 on Proc 0.
STKDisc: sideset SideSet1 has size 75 on Proc 0.
STKDisc: sideset SideSet2 has size 75 on Proc 0.
STKDisc: sideset SideSet3 has size 75 on Proc 0.
STKDisc: sideset SideSet4 has size 1 on Proc 0.
STKDisc: sideset SideSet5 has size 1 on Proc 0.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Sacado ParameterLibrary has been initialized:
Library of all registered parameters:
SDBC on NS NodeSet0 for DOF X: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet1 for DOF X: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet2 for DOF Y: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet3 for DOF Y: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet4 for DOF Z: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet5 for DOF StrongSchwarz: Supports AD = 1, Supports_Analytic = 0
Time: Supports AD = 1, Supports_Analytic = 0
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Number of parameter vectors = 0
Number of response vectors = 1
Number of distributed parameters vectors = 0
A) Get the base parameter list ...
Using Tempus 'Time Step Control'.
C) Create and initalize the forward model ...
D) Create the stepper and integrator for the forward problem ...
Warning -- Did not find a Tempus Time Events to create!
Should call add() with this (app-specific?) Time Event.
Problem Name = Mechanics 3D
Solution Method = Transient Tempus
Mechanics problem:
Spatial dimension : 3
Mechanics variables : DOF
Temperature variables : None
Dislocation Density variables : None
Pore Pressure variables : None
Transport variables : None
HydroStress variables : None
Damage variables : None
Stabilized Pressure variables : None
Initializing material database from material_01.yaml
TmplSTKMeshStruct:: Creating 3D mesh of size 1x1x75 elements and scaled to 0.01x0.01x0.75
Num MeshSpecs: 1
Calling MechanicsProblem::buildEvaluators
Calling MechanicsProblem::constructDirichletEvaluators
Calling MechanicsProblem::constructNeumannEvaluators
[GenericSTKMeshStruct] Processing field requirements...
STKDisc: 75 elements on Proc 0
STKDisc: nodeset NodeSet0 has size 152 on Proc 0.
STKDisc: nodeset NodeSet1 has size 152 on Proc 0.
STKDisc: nodeset NodeSet2 has size 152 on Proc 0.
STKDisc: nodeset NodeSet3 has size 152 on Proc 0.
STKDisc: nodeset NodeSet4 has size 4 on Proc 0.
STKDisc: nodeset NodeSet5 has size 4 on Proc 0.
STKDisc: nodeset NodeSet99 has size 1 on Proc 0.
STKDisc: sideset SideSet0 has size 75 on Proc 0.
STKDisc: sideset SideSet1 has size 75 on Proc 0.
STKDisc: sideset SideSet2 has size 75 on Proc 0.
STKDisc: sideset SideSet3 has size 75 on Proc 0.
STKDisc: sideset SideSet4 has size 1 on Proc 0.
STKDisc: sideset SideSet5 has size 1 on Proc 0.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Sacado ParameterLibrary has been initialized:
Library of all registered parameters:
SDBC on NS NodeSet0 for DOF X: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet1 for DOF X: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet2 for DOF Y: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet3 for DOF Y: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet4 for DOF StrongSchwarz: Supports AD = 1, Supports_Analytic = 0
SDBC on NS NodeSet5 for DOF Z: Supports AD = 1, Supports_Analytic = 0
Time: Supports AD = 1, Supports_Analytic = 0
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Number of parameter vectors = 0
Number of response vectors = 1
Number of distributed parameters vectors = 0
A) Get the base parameter list ...
Using Tempus 'Time Step Control'.
C) Create and initalize the forward model ...
D) Create the stepper and integrator for the forward problem ...
Warning -- Did not find a Tempus Time Events to create!
Should call add() with this (app-specific?) Time Event.
Thread 1 "Albany" received signal SIGSEGV, Segmentation fault.
0x00007ffff6f9e89c in Albany::Application::getNumEquations (this=0x0)
at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.hpp:157
157 return neq;
Missing separate debuginfos, use: dnf debuginfo-install blas-3.9.0-6.fc34.x86_64 cyrus-sasl-lib-2.1.27-8.fc34.x86_64 glibc-2.33-20.fc34.x86_64 hdf-4.2.15-4.fc34.x86_64 hdf-libs-4.2.15-4.fc34.x86_64 hdf5-1.10.6-5.fc34.x86_64 hwloc-libs-2.4.1-3.fc34.x86_64 infinipath-psm-3.3-26_g604758e_open.6.fc34.3.x86_64 keyutils-libs-1.6.1-2.fc34.x86_64 krb5-libs-1.19.2-2.fc34.x86_64 lapack-3.9.0-6.fc34.x86_64 libaec-1.0.5-1.fc34.x86_64 libbrotli-1.0.9-4.fc34.x86_64 libcurl-7.76.1-7.fc34.x86_64 libevent-2.1.12-3.fc34.x86_64 libfabric-1.12.1-1.fc34.x86_64 libgcc-11.2.1-1.fc34.x86_64 libgfortran-11.2.1-1.fc34.x86_64 libibverbs-36.0-1.fc34.x86_64 libnghttp2-1.43.0-2.fc34.x86_64 libpsl-0.21.1-3.fc34.x86_64 libquadmath-11.2.1-1.fc34.x86_64 librdmacm-36.0-1.fc34.x86_64 libssh-0.9.5-2.fc34.x86_64 libstdc++-11.2.1-1.fc34.x86_64 libtirpc-1.3.2-0.fc34.x86_64 libunistring-0.9.10-10.fc34.x86_64 libxcrypt-4.4.25-1.fc34.x86_64 netcdf-4.7.3-6.fc34.x86_64 numactl-libs-2.0.14-3.fc34.x86_64 openldap-2.4.57-5.fc34.x86_64 openmpi-4.1.0-5.fc34.x86_64 openssl-libs-1.1.1l-1.fc34.x86_64 pcre2-10.36-4.fc34.x86_64 pmix-3.2.3-1.fc34.x86_64 ucx-1.10.1-1.fc34.x86_64 zlib-1.2.11-26.fc34.x86_64
(gdb) bt
#0 0x00007ffff6f9e89c in Albany::Application::getNumEquations (this=0x0)
at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.hpp:157
#1 0x00007ffff2ef0d22 in LCM::StrongSchwarzBC_Base<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>::computeBCsDTK (
this=0xbd8c00) at /home/ikalash/ACE/Albany-LCM-master/src/LCM/evaluators/bc/StrongSchwarzBC_Def.hpp:280
#2 0x00007ffff2eef125 in LCM::fillSolution<LCM::StrongSchwarzBC<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>, PHAL::AlbanyTraits> (sbc=..., workset=...) at /home/ikalash/ACE/Albany-LCM-master/src/LCM/evaluators/bc/StrongSchwarzBC_Def.hpp:468
#3 0x00007ffff2eee069 in LCM::StrongSchwarzBC<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>::preEvaluate (
this=0xbd8c00, workset=...) at /home/ikalash/ACE/Albany-LCM-master/src/LCM/evaluators/bc/StrongSchwarzBC_Def.hpp:612
#4 0x00007ffff6f36f72 in PHX::DagManager<PHAL::AlbanyTraits>::preEvaluate (this=0xba94e8, d=...)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Phalanx_DAG_Manager_Def.hpp:641
#5 0x00007ffff6f33a7f in PHX::EvaluationContainer<PHAL::AlbanyTraits::Residual, PHAL::AlbanyTraits>::preEvaluate (
this=0xba94e0, d=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Phalanx_EvaluationContainer_Def.hpp:502
#6 0x00007ffff6ec3612 in PHX::FieldManager<PHAL::AlbanyTraits>::preEvaluate<PHAL::AlbanyTraits::Residual> (this=0xbe0e60,
d=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Phalanx_FieldManager_Def.hpp:349
#7 0x00007ffff6eaf3ec in Albany::Application::computeGlobalResidualImpl (this=0xb3bd20, current_time=0, x=..., x_dot=...,
x_dotdot=..., p=..., f=..., dt=0) at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.cpp:1142
#8 0x00007ffff6eb03c7 in Albany::Application::computeGlobalResidual (this=0xb3bd20, current_time=0, x=..., x_dot=...,
x_dotdot=..., p=..., f=..., dt=0) at /home/ikalash/ACE/Albany-LCM-master/src/Albany_Application.cpp:1227
#9 0x00007ffff6f55a84 in Albany::ModelEvaluator::evalModelImpl (this=0xf91190, inArgs=..., outArgs=...)
at /home/ikalash/ACE/Albany-LCM-master/src/Albany_ModelEvaluator.cpp:679
#10 0x00007ffff6b69809 in Thyra::ModelEvaluatorDefaultBase<double>::evalModel (this=0xf91190, inArgs=..., outArgs=...)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_ModelEvaluatorDefaultBase.hpp:709
#11 0x00007ffff6b80251 in Thyra::DefaultModelEvaluatorWithSolveFactory<double>::evalModelImpl (this=0xebbc90, inArgs=...,
outArgs=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_DefaultModelEvaluatorWithSolveFactory.hpp:325
#12 0x00007ffff6b69809 in Thyra::ModelEvaluatorDefaultBase<double>::evalModel (this=0xebbcd8, inArgs=..., outArgs=...)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_ModelEvaluatorDefaultBase.hpp:709
#13 0x00007ffff6b79e3d in Piro::InvertMassMatrixDecorator<double>::evalModelImpl (this=0xec2910, inArgs=..., outArgs=...)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_InvertMassMatrixDecorator_Def.hpp:348
#14 0x00007ffff6b69809 in Thyra::ModelEvaluatorDefaultBase<double>::evalModel (this=0xec2910, inArgs=..., outArgs=...)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Thyra_ModelEvaluatorDefaultBase.hpp:709
#15 0x00007fffefc01caf in Tempus::StepperExplicit<double>::evaluateExplicitODE (this=0xf2c850, xDotDot=..., x=..., xDot=...,
time=0, p=...) at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_StepperExplicit_impl.hpp:313
#16 0x00007fffefcb1d32 in Tempus::StepperNewmarkExplicitAForm<double>::setInitialConditions (this=0xf2c820,
solutionHistory=...) at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_StepperNewmarkExplicitAForm_impl.hpp:187
#17 0x00007fffefb3acf8 in Tempus::IntegratorBasicOld<double>::initializeSolutionHistory (this=0xe1d2c0, state=...)
at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:149
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x00007fffefb3b563 in Tempus::IntegratorBasicOld<double>::setSolutionHistory (this=0xe1d2c0, sh=...)
at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:208
#19 0x00007fffefb3c44f in Tempus::IntegratorBasicOld<double>::initialize (this=0xe1d2c0)
at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:292
#20 0x00007fffefb36386 in Tempus::IntegratorBasicOld<double>::IntegratorBasicOld (this=0xe1d2c0, inputPL=..., model=...,
__in_chrg=<optimized out>, __vtt_parm=<optimized out>)
at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:28
#21 0x00007fffefb3eef3 in Tempus::integratorBasic<double> (pList=..., model=...)
at /home/ikalash/ACE/Trilinos/packages/tempus/src/Tempus_IntegratorBasicOld_impl.hpp:667
#22 0x00007ffff6b38522 in Piro::TempusIntegrator<double>::TempusIntegrator (this=0xf2c7b0, pList=..., model=...,
sens_method=Piro::NONE) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusIntegrator_Def.hpp:63
#23 0x00007ffff6b29cc6 in Piro::TempusSolver<double>::initialize (this=0xe57d60, appParams=..., in_model=...,
in_adjointModel=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusSolver_Def.hpp:342
#24 0x00007ffff6b18bb3 in Piro::TempusSolver<double>::TempusSolver (this=0xe57d60, appParams=..., in_model=...,
in_adjointModel=..., piroObserver=..., __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusSolver_Def.hpp:120
#25 0x00007ffff6b0af29 in Piro::tempusSolver<double> (appParams=..., in_model=..., in_adjointModel=..., piroObserver=...)
at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_TempusSolver_Def.hpp:755
#26 0x00007ffff6afe5a1 in Piro::SolverFactory::createSolver<double> (this=0x7fffffff9d7f, piroParams=..., model=...,
adjointModel=..., observer=...) at /home/ikalash/ACE/Trilinos/build-dbg/install/include/Piro_SolverFactory_Def.hpp:159
#27 0x00007ffff6ad41ae in Albany::SolverFactory::createAndGetAlbanyApp (this=0x7fffffffa520, albanyApp=..., appComm=...,
solverComm=..., initial_guess=..., createAlbanyApp=true)
at /home/ikalash/ACE/Albany-LCM-master/src/Albany_SolverFactory.cpp:231
#28 0x00007ffff2bca198 in LCM::SchwarzAlternating::SchwarzAlternating (this=0x78bfb0, app_params=..., comm=...,
__in_chrg=<optimized out>, __vtt_parm=<optimized out>)
at /home/ikalash/ACE/Albany-LCM-master/src/LCM/solvers/Schwarz_Alternating.cpp:196
#29 0x00007ffff6ad3869 in Albany::SolverFactory::createAndGetAlbanyApp (this=0x7fffffffbbd0, albanyApp=..., appComm=...,
solverComm=..., initial_guess=..., createAlbanyApp=true)
at /home/ikalash/ACE/Albany-LCM-master/src/Albany_SolverFactory.cpp:184
#30 0x0000000000466d48 in main (argc=2, argv=0x7fffffffd048) at /home/ikalash/ACE/Albany-LCM-master/src/Main_Solve.cpp:101