precice / micro-manager

A manager tool to facilitate two-scale coupling in multi-physics simulations using preCICE.

Micro Manager

A tool to facilitate solving two-scale (macro-micro) coupled problems using the coupling library preCICE.

The main documentation is rendered on the preCICE website.

Please report any issues and give us feedback through one of our community channels.

micro-manager's Issues

Domain decomposition according to the number of nodes

Currently the micro_manager splits the domain on a mpi run according to the geographic macro_bounds, instead of the number of simulations contained in each section.
As the micro simulations are independent from each other, to allow an optimal resource usage the micro manager should allocate the micro simulations evenly across used cpu cores.

Allow micro simulations to pass data to be used only in the adaptivity calculation

Currently micro simulations are only allowed to pass data which is to be sent to the macro side. However, for the adaptivity calculation, the micro simulations may have data related to their state which is suitable for calculating the similarity distance. There should be an option for such data to be passed on to the Micro Manager.

Add functionality for spanshot computation from a full-order model

In the framework of the SimTech project PN5-9(II) the Micro Manager needs to be extended to be able to do an offline snapshot computation step for a given full-order model. The input parameter space for creating the snapshots will be defined by the user through a configuration file. The existing JSON file could be potentially used for this, or a new configuration file format can be explored. The data required from the simulations (output) is also defined by the user. The database developed from this computation should have an interface between its output and the reduced-order/surrogate models, presumably using hdf5. Rough tasks to add this functionality are:

• Design configuration options for parameter space and data output options to run a full-order model
• Finalize a file format for configuring the parameters of the snapshot computation
• Implement a new class in the Micro Manager to run the offline computation
• Find the most suitable file format to store the snapshots and create an interface to surrogate/reduced-order models
• For a simple dummy simulation, try to output the snapshots in the file format finalized in the previous step

Learning the adaptivity parameters on the fly

After numerical experiments with the implementations in #60 and #103, which requires tuning of the method parameters that might be case-dependent, we are motivated to find a way to learn the adaptivity parameters or the parameters in these methods on the fly with sequential optimization approaches.

Speaking of the adaptivity parameters, we would want to find the optimal coarsening_const $C_c$ - refining_const $r_c$-historical_parameter $h$ to minimize the runtime under simulation error constraints. According to Felix, it would benefit the prediction a lot if we have a priori knowledge(global model) about the relationship between error/runtime and the three parameters. And this knowledge also gives info on if we need to scale the parameters or the runtime/error, which might range across multiple orders, to a logarithmic space.

For the optimization we could start with classic Gaussian process. For the global model we could use paper data and do experiments with heat-conduction tutorial.

Duplicate micro-simulations on interior boundaries of macro-domain decomposition

Problem:
When using macro-domain decomposition each MPI rank is given a bounding box, which is passed to preCICE to obtain coordinates of coupled points contained therein. If such a coupled macro-point lies on the boundary between two bounding boxes it will be included in both and the Micro Manager currently creates a micro simulation for both duplicates of the macro-point and passes their output to preCICE.

• preCICE interprets this as two contributions to the participant's solution and accumulates them, producing incorrect results.
• duplicate micro simulations receive identical macro-data and with no/global adaptivity identical output, wasting computational effort
• with local adaptivity the two micro-simulations may have different associated active simulations and produce different outputs (differences should be minor if similarity measure is not overly generous)

Suggestion:
The Micro Manager should eliminate duplicate micro simulations to save computational time and avoid incorrect data being received by the macro simulation.

Overcounted active steps when using adaptivity in every implicit iteration

When using adaptivity in every implicit iteration, the count of active steps is still incremented at the beginning of every time window when writing a checkpoint in micro_manager.py#L202. The increment should be guarded by the check for adaptivity based on time windows (line 169).

Note: The setting of active_sim_ids in between could also be moved to the conditional block. It is set separately for the case of adaptivity in every implicit iteration without being used in between.

Adapt the similarity condition to the residual of the fixed-point problem

Generally, we postulate that the micro simulations takes up most of the computing time when we are using the Micro Manager for multiscale simulations with large number of micro simulations.

As Delaisse's paper indicated, we could

achieve a significant speed-up by not converging to the final sub-problem tolerance in each solver call.

Although we are not supposed to control the tolerance of sub-problems (like what Nicolas did in the paper) in preCICE, we could control the numbers of micro simulations through the Micro Manager to get more coarse or fine solution of the whole field. That means, we could use loose similarity conditions to solve less micro problems and thus get coarse result and use strict similarity condition when we are close to the convergence.

Based on this idea, we need to discuss following points:

• If we are going to adapt the similarity condition to the fixed-point tolerance from Quasi-Newton method and we only do QN computation at the end of each time window, we would also update the similarity condition in each time window between the implicit iterations, and restart from the most loose condition at each new time window.
• This method is only available for implicit coupling.
• How should we select the initial (relative) similarity condition and how do we rationally adapt the condition according to the fixed-point residual?
• Can we retain the accuracy of the original method under the assumption that the total Newton-iterations in sub-problems would reduce?

Micro manager should be able to control micro simulations written in C++

Currently the Micro Manager is constructed in a way that is can only control micro simulations written in Python. The user gives a path to a micro simulation script written in Python, in the manager configuration file. However it can be the case that a user has a micro simulation script written in some other language like C++, C of even Fortran. For the time being the manager can be updated in a way that scripts written in C and C++ can be used. Both C and C++ scripts cannot be used directly, and in most cases will be built into a shared library (.so) file. The Micro Manager needs to take the shared library file and run the appropriate functions.

For micro simulations written in C++, pybind11 was the tool used in #22.

Use Multiprocessing besides domain decomposition in solve function

Opening this as per the discussion in #68.

Maybe a user wants to have a certain number of micro simulations on a MPI process, so that the local adaptivity can still be used to avoid memory requirements instead of having one MPI process for every micro simulation. In this case one could benefit from multiprocessing in the solve function on the local domain.

Add unit tests for all user side functions of the Micro Manager

All the functions in the file micro_manager/micro_manager.py needs to have unit tests for testing their behavior in several situations.

Handle cases where some ranks do not have any micro simulations

Currently the manager requires that every rank must at least one micro simulation. However, this may not be the case for bigger simulations. The manager should be able properly handle empty ranks. Partial support for handling empty ranks was introduced before, but the new adaptivity functionality has removed the support again.

Output runtime for calculation of similarity

For really fast microsimulations (PDE-ODE), the calculation of similarity could be as costly as the runtime of the microsimulations itself (micro_sim_time). I think it makes sense to be able to compare the two variables as a user, since the calculation of similarity probably also depends on multiple factors.

I could also imagine a feature where the micro-manager decides by itself if adaptvitiy should be used or not by comparing runtime of microsimulations vs calculation of similarity.

Provide a micro simulation abstract class which users can inherit/create a subclass

In openjournals/joss-reviews#5842 (comment), @mbkumar suggested that the Micro Manager could provide an abstract class for the micro simulation code, which the users could subclass or inherit, and the give the name of their class to the Micro Manager via the JSON config file. This is a cleaner way to ensure that the users stick to the desired API while converting their micro simulations into callable libraries.

Add functionality to allow the Micro Manager to initialize micro simulations from different file/solver sources simultaneously

Currently the Micro Manager accepts one micro simulation file (with Python or C++ code) and initializes all micro simulation objects with it. In certain applications, a user may want to initialize more than one micro simulations simultaneously. For example, at some macro locations, high fidelity micro simulations may be necessary, while at other macro locations, low fidelity micro simulations would be sufficient.

As a first step, such a dynamic setup can be done at the time of initialization, and it will not change during the simulation.

Call micro simulation solve routine with multiple CPUs

In case of a large micro simulation (in the orders of million elements or degrees of freedom at each Gauss point of the macro problem) or even multiple scales for a single solve call, access to multiple CPUs will be required for undertaking the analysis. Can we give each solve() call access to multiple CPUs?

Compatability issue deepcopy and pybind?

A Dumux-Dumux coupled simulation (see https://github.com/HelenaKschidock/macro-micro fails with current develop branch. Pybind was built locally using pip (after micro-manager installation).

(0) 21:58:31 [cplscheme::BaseCouplingScheme]:235 in advance: Time window completed
(0) 21:58:31 [impl::SolverInterfaceImpl]:464 in advance: iteration: 1 of 30, time-window: 2, time: 0.005 of 0.25, time-window-size: 0.005, max-timestep-length: 0.005, ongoing: yes, time-window-complete: yes, write-iteration-checkpoint 
Saving state of micro problem (0)
Saving state of micro problem (1)
Saving state of micro problem (2)

[...]

Saving state of micro problem (126)
Saving state of micro problem (127)
Traceback (most recent call last):
  File "/home/kschidock/FM/dumux2/dumux/dumux-adapter/build-cmake/examples/macro-micro/micro-heat/run_micro_manager.py", line 11, in <module>
    manager.solve()
  File "/home/kschidock/.local/lib/python3.10/site-packages/micro_manager/micro_manager.py", line 505, in solve
    similarity_dists, micro_sim_states = self.compute_adaptivity(similarity_dists, micro_sim_states)
  File "/home/kschidock/.local/lib/python3.10/site-packages/micro_manager/micro_manager.py", line 391, in compute_adaptivity
    micro_sim_states_n = self._adaptivity_controller.update_inactive_micro_sims(
  File "/home/kschidock/.local/lib/python3.10/site-packages/micro_manager/adaptivity.py", line 168, in update_inactive_micro_sims
    micro_sims[i] = deepcopy(micro_sims[associated_active_id])
  File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'MicroProblem' object

Expose entry point for CLI script

The micro-manager could expose a main as a script entry point.
This would allow it to be installed and used directly using pipx.

We currently do the same for the fmirunner:
https://github.com/precice/fmi-runner/blob/93d6c218dbfa5554b8e809a1a76c991c6e5fd7ef/pyproject.toml#L36-L37

Investigate if quantities passed to adaptivity for similarity calculation can have weights

There needs to be an investigation if quantities passed to adaptivity for similarity calculation can have weights so that user can control which quantities are more important when it comes to adaptivity. If such weights are found to be effective, the functionality can be implemented in the manager.

Add pictorial description of macro-micro coupling to README

Currently the README has only text which describes the concept and working of the Micro Manager. In addition to the provided information, a pictorial description of the macro-micro coupling strategy would be beneficial for understanding the core concept of the Micro Manager.

The `data` in adaptivity setting didn't work as expected

With the develop branch, I want to run the two-scale-heat-conduction tutorial case with less data sets involved in the adaptivity study.
The original json file is here, I changed the simulation_params section to:

    "simulation_params": {
      "macro_domain_bounds": [0.0, 1.0, 0.0, 0.5],
      "adaptivity": "True",
      "adaptivity_settings": {
            "type": "global",
            "data": ["k_00", "concentration"],
            "history_param": 0.1,
            "coarsening_constant": 0.2,
            "refining_constant": 0.05,
            "every_implicit_iteration": "False",
            "similarity_measure": "L2rel"
        }
    },

to match the new format from commit e6641ba and use less data for the adaptivity.
It threw following error:

  File ".../two-scale-heat-conduction/micro-dumux/run_micro_manager.py", line 7, in <module>
    manager = MicroManager("micro-manager-config.json")
  File ".../python3.10/site-packages/micro_manager/micro_manager.py", line 128, in __init__
    self._initialize()
  File ".../python3.10/site-packages/micro_manager/micro_manager.py", line 483, in _initialize
    self._data_for_adaptivity[name][0] = initial_micro_output[name]
KeyError: 'k_11'

Add check if dependencies are installed correctly when Micro Manager is installed via pip

As suggested in openjournals/joss-reviews#5842 (comment) there needs to be a check to ensure that the dependencies are installed correctly when the Micro Manager is installed via pip.

Snapshot: Adaptive chunk sizes

Enabling faster reading and writing of subsets of an HDF5 file can be achieved by chunking. To make use of this, it is important to optimize the chunk size. As the size of snapshots is unknown before the computation of the first snapshot in parameter space, the snapshot computation needs to determine the chunk size automatically with the knowledge of the first output. In a parallel run of the snapshot computation, chunks mustn't allocate a lot of space that is not used to store data, and results from different processes can be merged. Currently, the snapshot computation uses a fixed chunk size

Check package deployed on PyPI via CI

Currently there is no check if the package deployed to PyPI actually works. Such a check should be done once a package is deployed after a release. Running just the solver dummy would be sufficient.

Allow user to provide comparison function for similarity calculation in adaptivity

Currently the adaptivity calculates similarity between two simulations based on an absolute difference:

micro-manager/micro_manager/adaptivity.py

Lines 59 to 62 in cec8910

	for d in range(dim):
	data_diff += abs(data[counter_1, d] - data[counter_2, d])
	else:
	data_diff = abs(data[counter_1] - data[counter_2])

Instead of doing an absolute difference, the user can provide a function of their choice to use in the similarity computation. Alternatively the manager can provide various options for the user to pick from.

Add documentation of the Micro Manager to precice.org

Handle crashing of micro simulations in a proper way

Currently when the Micro Manager is controlling and running micro simulations, if one simulation crashes or has an improper exit, the Micro Manager run just hangs. It is nearly impossible to know which micro simulation crashed, and why the overall execution has hanged.

The Micro Manager should be able to handle a simulation crash by ...

• ... continuing to run the rest of the micro simulations.
• ... logging the simulation crash in the log file.
• ... providing the macro location at which the simulation crashed.
• ... creating a new log file and parsing the error output from the crashed simulation to it.

It is not clear if all these things could be done, so initially some investigations are necessary.

Handling of BoundingBox for Receiving Mesh compared to preCICE API

In the couple your code section the Bounding Box is defined by two corners.

std::vector<double> boundingBox {
        0, 0, 0, // minimum corner
        1, 1, 1 // maximum corner
    };

While the macro domain bounds are defined that way

[xmin, xmax, ymin, ymax, zmin, zmax]

I was wondering why the two are not the same:D

Unable to write initial data to preCICE for macro-micro coupled scenarios

Describe your setup

Operating system (e.g. Linux distribution and version): Ubuntu 22.04
preCICE STATE: precice/precice@8393841

Add the versions/commit hashes of further adapter and/or bindings.

Micro Manager: 6dcbf68

Describe the problem

Although the Micro Manager uses the Python API, the problem is not related to it and is hence described below in terms of the original C++ API.

The Micro Manager does not need a coupling mesh of its own, but just needs the vertex information of the macro-scale participant. Using getMeshVertexIDsAndCoordinates(), it gets the vertex IDs and coordinates of the macro-scale participant mesh. It then defines one micro simulation per vertex of this mesh. Then it checks if the micro simulation objects have a function called initialize() and if they do, then it calls this function to get the initial state. Then it attempts to write the initial state to preCICE by checking requiresInitialData() and then initialize().

The problem is that the function getMeshVertexIDsAndCoordinates() cannot be run before initialize(). But without running it, the Micro Manager does not know how many micro simulations to create, and hence it cannot get their initial states.

This problem was not encountered before because the functions initialize() and initializeData() were separate. Back then the order of calling functions was:

1. initialize()
2. getMeshVertexIDsAndCoordinates()
3. isWriteDataRequired()
4. writeData...
5. initializeData()

Step To Reproduce

https://github.com/precice/micro-manager/actions/runs/5825484943/job/15797159769?pr=51

Expected behaviour

Currently I do not see a way that the Micro Manager can get the mesh information from the macro-scale participant and also give initial data of the micro simulations to preCICE. One solution is that we do not allow micro simulations to write initial data to preCICE, but I have to study how this would affect macro-micro simulations.

Opening the issue to discuss if I am looking over something rather obvious.

Invalid value in division warning in adaptivity functionality

For some cases, the following warning is observed from the file micro_manager/adaptivity/adaptivity.py:

RuntimeWarning: invalid value encountered in true_divide
  relative = np.nan_to_num((pointwise_diff / np.maximum(data[np.newaxis, :], data[:, np.newaxis])))

This needs to be investigated.

pyproject.toml instead of setup.py

The setup.py is deprecated and pyproject.toml is the recommended approach. Since this package does not have any complicated build setup, converting to pyproject.toml should be very easy to accomplish.

Add configuration options to dictate solving a full-order model for snapshot creation

This issue is mainly to discuss how the configuration functionality to dictate solving a full-order model for snapshot creation would look like.

Need CI to test package publishing workflow

The package publishing workflow is run every time a release is made. If changes to the workflow or package setup are done, the way to test the workflow is by doing a release candidate. This is a cumbersome way to test the publishing workflow. It would be better to have a workflow which publishes to TestPyPI every time a release PR is opened, and only when it passes, the release is allowed.

Compatibility of dummies with command line

The Micro Manager can be called from the command line with micro_manager. Currently the solverdummies are not compatible with this call as there are different solvers for python and c++.
Is this something we want to change or just clarify the usage in the documentation?

Interpolation of micro simulation output data

While working on #85 to handle crashing micro simulations, it came up that if a simulation crashes during the first iteration it is difficult to replace that simulation. It cannot be replaced with previous data and with no data available, it is also impossible to say, what other simulation is similar to this. In my opinion, it is necessary to use interpolation or extrapolation to continue a run if this kind of crash occurs. This could also be used if a simulation crash occurs in a later iteration.

Another potential use case for interpolation in the micro-manager is providing an alternative to adaptivity.

I've been looking at readily available interpolation tools in Python. For three-dimensional irregular grid points, I only found scipy's datagrid but it cannot handle extrapolation, which the micro would run into as soon as a corner simulation crashes.

Does it make sense to include interpolation into the micro-manager?

Is there an interpolation function available in Python that could be used? Or is it necessary to implement this tailored to the requirements of the micro-manager? How sophisticated should this be? One of the simplest implementations would be to take the mean value of neighboring micro simulations.

Does the micro problem API need a class structure?

Right now we ask the user to formulate the micro problem in a class structure, with a class name MicroSimulation which then has functions titled initialize, solve, etc. Is such a class structure even necessary? The Micro Manager can create a class and add functions to it.

Version incompatibility

The latest release version for libprecice is v2.5.0, so based on the documentation on precice website, I can only install pyprecice=v2.5.0 because of its dependency on libprecice. And micromanager depends on pyprecice>=v3.0dev2. How can this incompatibility be addressed?

Error in unit tests with v0.3.0

I am seeing the below errors when running the units.

======================================================================
ERROR: test_communicate_micro_output (test_adaptivity_parallel.TestGlobalAdaptivity)
Test functionality to communicate micro output from active sims to their associated inactive sims, for a global adaptivity setting.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bharatmedasani/Software/reviews/micro-manager-0.3.0/tests/unit/test_adaptivity_parallel.py", line 115, in test_communicate_micro_output
    adaptivity_controller.communicate_micro_output(is_sim_active, sim_is_associated_to, sim_output)
  File "/Users/bharatmedasani/miniconda3/envs/tmp38/lib/python3.8/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 153, in communicate_micro_output
    recv_reqs = self._p2p_comm(assoc_active_ids, micro_output)
  File "/Users/bharatmedasani/miniconda3/envs/tmp38/lib/python3.8/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 316, in _p2p_comm
    req = self._comm.irecv(bufsize, source=recv_rank, tag=tag)
  File "mpi4py/MPI/Comm.pyx", line 1502, in mpi4py.MPI.Comm.irecv
  File "mpi4py/MPI/msgpickle.pxi", line 417, in mpi4py.MPI.PyMPI_irecv
mpi4py.MPI.Exception: MPI_ERR_RANK: invalid rank

======================================================================
ERROR: test_update_inactive_sims_global_adaptivity (test_adaptivity_parallel.TestGlobalAdaptivity)
Test functionality to update inactive simulations in a particular setting, for a global adaptivity setting.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bharatmedasani/Software/reviews/micro-manager-0.3.0/tests/unit/test_adaptivity_parallel.py", line 70, in test_update_inactive_sims_global_adaptivity
    is_sim_active, sim_is_associated_to = adaptivity_controller._update_inactive_sims(
  File "/Users/bharatmedasani/miniconda3/envs/tmp38/lib/python3.8/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 230, in _update_inactive_sims
    recv_reqs = self._p2p_comm(list(to_be_activated_map.keys()), sim_states_and_global_ids)
  File "/Users/bharatmedasani/miniconda3/envs/tmp38/lib/python3.8/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 316, in _p2p_comm
    req = self._comm.irecv(bufsize, source=recv_rank, tag=tag)
  File "mpi4py/MPI/Comm.pyx", line 1502, in mpi4py.MPI.Comm.irecv
  File "mpi4py/MPI/msgpickle.pxi", line 417, in mpi4py.MPI.PyMPI_irecv
mpi4py.MPI.Exception: MPI_ERR_RANK: invalid rank

----------------------------------------------------------------------

Add CI to check test coverage

It would be good to check test coverage for every change via a pull request. As the Micro Manager is written in Python, any solution would involve using Coverage.py. There are ways to automate this in GitHub Actions via existing actions or Codecov, but it seems easier to implement a simple Python script which computes the test coverage and checks if it is above or below a predefined threshold.

Provide functionality to pass a quantities to the micro simulations during initialization

For certain cases, a user may want to set the initial state of a micro simulation based on macro information, like, for example, the macro location of the micro simulation.

@mathiskelm postulates such a scenario: Say I know (roughly) the heterogeneity of porosity in my macro domain, e.g. a simple space-dependent $ϕ = 0.5 + 0.01 y$ in a domain $[0,10]^2$. Then I would like the micro-sims that correspond to a macro $y = 0$ to have a different initial state to ones at macro $y = 10$.

The Micro Manager could give the user options between several quantities that it has, to pass to the micro simulation during initialization. Macro coordinates could be one of them. The set of quantities would be fixed and would be configured via the JSON config file.

Pickling support for C++ micro simulations does not work

Support for pickling micro simulations as shown in the dummy does not allow for passing the simulation id of the current instance of the simulation to the newly created instance of the simulation. If a micro simulation is pickled (for its transfer between ranks in a global adaptivity scenario), the simulation id that is given to it by the Micro Manager during initialization needs to be defined once more in the new instance. Either the Micro Manager needs to pass it in via the set state functionality, or somehow it needs to be passed internally from the old instance to the new instance. This problem was also observed by @mathiskelm while finalizing the DuMuX micro simulation for the two-scale-heat-conduction tutorial case.

Further investigations are necessary here.

Unbounded relative similarity measure for negative values

In the relative similarity measures the data differences are divided by the larger value (for each component) before applying the specified norm. If both values are negative, this selects the one with smaller absolute value, leading to potentially large "normalized" distances.

Exaggeration of this distance may lead not just to activation of micro simulations but by raising the maximal distance also to a relaxation of the refining/coarsening thresholds and solving too few micro simulations.

Addressing file duplication in `examples/*-dummy`

In the folders examples/python-dummy and examples/cpp-dummy, the file run_micro_manager.py exists. In principle this file can be put in examples/ and used for both applications. But this would mean that the Python and CPP dummy need to have their own JSON configuration files, because the name of micro code file is the same. If only one run_micro_manager.py is to exist, we would end up several configuration files:

• micro-manager-config-cpp.json
• micro-manager-config-python.json
• micro-manager-adaptivity-config-cpp.json
• micro-manager-adaptivity-config-python.json

Also, merging micro-manager-config-*.json and micro-manager-adaptivity-config-*.json is not possible because the adaptivity variant is tested in GitHub Actions.

@uekerman thoughts?

Parallelize the micro simulation to further speedup snapshot parallelization

Currently, the snapshot computation feature can only parallelize the computation of snapshots, i.e. the parameter space can be distributed and different processes compute different snapshots. For expensive microsimulations, it can be practical to parallelize the microsimulation rather than the snapshots.

Improve domain decomposition strategy

Currently the domain decomposition is done in way that the macro domain is partitioned in the X and Y directions as optimally as possible. No partition is done in the Z direction for 3D cases. This partitioning is not optimal, and the domain needs to be partitioned in the Z direction too.

In addition to the optimal partitioning strategy, the user can also be given a choice of partitions in X, Y and Z directions. Such a choice would be configured through the JSON configuration file.

Make mpi4py an optional dependency