sciml / scimlbenchmarks.jl

Scientific machine learning (SciML) benchmarks, AI for science, and (differential) equation solvers. Covers Julia, Python (PyTorch, Jax), MATLAB, R

SciMLBenchmarks.jl: Benchmarks for Scientific Machine Learning (SciML) and Equation Solvers

SciMLBenchmarks.jl holds webpages, pdfs, and notebooks showing the benchmarks for the SciML Scientific Machine Learning Software ecosystem, including:

• Benchmarks of equation solver implementations
• Speed and robustness comparisons of methods for parameter estimation / inverse problems
• Training universal differential equations (and subsets like neural ODEs)
• Training of physics-informed neural networks (PINNs)
• Surrogate comparisons, including radial basis functions, neural operators (DeepONets, Fourier Neural Operators), and more

The SciML Bench suite is made to be a comprehensive open source benchmark from the ground up, covering the methods of computational science and scientific computing all the way to AI for science.

Rules: Optimal, Fair, and Reproducible

These benchmarks are meant to represent good optimized coding style. Benchmarks are preferred to be run on the provided open benchmarking hardware for full reproducibility (though in some cases, such as with language barriers, this can be difficult). Each benchmark is documented with the compute devices used along with package versions for necessary reproduction. These benchmarks attempt to measure in terms of work-precision efficiency, either timing with an approximately matching the error or building work-precision diagrams for direct comparison of speed at given error tolerances.

If any of the code from any of the languages can be improved, please open a pull request.

Results

To view the results of the SciML Benchmarks, go to benchmarks.sciml.ai. By default, this will lead to the latest tagged version of the benchmarks. To see the in-development version of the benchmarks, go to https://benchmarks.sciml.ai/dev/.

Static outputs in pdf, markdown, and html reside in SciMLBenchmarksOutput.

Citing

To cite the SciML Benchmarks, please cite the following:

@article{rackauckas2019confederated,
  title={Confederated modular differential equation APIs for accelerated algorithm development and benchmarking},
  author={Rackauckas, Christopher and Nie, Qing},
  journal={Advances in Engineering Software},
  volume={132},
  pages={1--6},
  year={2019},
  publisher={Elsevier}
}

@article{DifferentialEquations.jl-2017,
 author = {Rackauckas, Christopher and Nie, Qing},
 doi = {10.5334/jors.151},
 journal = {The Journal of Open Research Software},
 keywords = {Applied Mathematics},
 note = {Exported from https://app.dimensions.ai on 2019/05/05},
 number = {1},
 pages = {},
 title = {DifferentialEquations.jl – A Performant and Feature-Rich Ecosystem for Solving Differential Equations in Julia},
 url = {https://app.dimensions.ai/details/publication/pub.1085583166 and http://openresearchsoftware.metajnl.com/articles/10.5334/jors.151/galley/245/download/},
 volume = {5},
 year = {2017}
}

Current Summary

The following is a quick summary of the benchmarks. These paint broad strokes over the set of tested equations and some specific examples may differ.

Non-Stiff ODEs

• OrdinaryDiffEq.jl's methods are the most efficient by a good amount
• The Vern methods tend to do the best in every benchmark of this category
• At lower tolerances, Tsit5 does well consistently.
• ARKODE and Hairer's dopri5/dop853 perform very similarly, but are both far less efficient than the Vern methods.
• The multistep methods, CVODE_Adams and lsoda, tend to not do very well.
• The ODEInterface multistep method ddeabm does not do as well as the other multistep methods.
• ODE.jl's methods are not able to consistently solve the problems.
• Fixed time step methods are less efficient than the adaptive methods.

Stiff ODEs

• In this category, the best methods are much more problem dependent.
• For smaller problems:
  • Rosenbrock23, lsoda, and TRBDF2 tend to be the most efficient at high tolerances.
  • Rodas4 and Rodas5 tend to be the most efficient at low tolerances.
• For larger problems (Filament PDE):
  • QNDF and FBDF does the best at all normal tolerances.
  • The ESDIRK methods like TRBDF2 and KenCarp4 can come close.
• radau is always the most efficient when tolerances go to the low extreme (1e-13)
• Fixed time step methods tend to diverge on every tested problem because the high stiffness results in divergence of the Newton solvers.
• ARKODE is very inconsistent and requires a lot of tweaking in order to not diverge on many of the tested problems. When it doesn't diverge, the similar algorithms in OrdinaryDiffEq.jl (KenCarp4) are much more efficient in most cases.
• ODE.jl and GeometricIntegrators.jl fail to converge on any of the tested problems.

Dynamical ODEs

• Higher order (generally order >=6) symplectic integrators are much more efficient than the lower order counterparts.
• For high accuracy, using a symplectic integrator is not preferred. Their extra cost is not necessary since the other integrators are able to not drift simply due to having low enough error.
• In this class, the DPRKN methods are by far the most efficient. The Vern methods do well for not being specific to the domain.

Non-Stiff SDEs

• For simple 1-dimensional SDEs at low accuracy, the EM and RKMil methods can do well. Beyond that, they are simply outclassed.
• The SRA and SRI methods both are very similar within-class on the simple SDEs.
• SRA3 is the most efficient when applicable and the tolerances are low.
• Generally, only low accuracy is necessary to get to sampling error of the mean.
• The adaptive method is very conservative with error estimates.

Stiff SDEs

• The high order adaptive methods (SRIW1) generally do well on stiff problems.
• The "standard" low-order implicit methods, ImplicitEM and ImplicitRK, do not do well on all stiff problems. Some exceptions apply to well-behaved problems like the Stochastic Heat Equation.

Non-Stiff DDEs

• The efficiency ranking tends to match the ODE Tests, but the cutoff from low to high tolerance is lower.
• Tsit5 does well in a large class of problems here.
• The Vern methods do well in low tolerance cases.

Stiff DDEs

• The Rosenbrock methods, specifically Rodas5, perform well.

Parameter Estimation

• Broadly two different approaches have been used, Bayesian Inference and Optimisation algorithms.
• In general it seems that the optimisation algorithms perform more accurately but that can be attributed to the larger number of data points being used in the optimisation cases, Bayesian approach tends to be slower of the two and hence lesser data points are used, accuracy can increase if proper data is used.
• Within the different available optimisation algorithms, BBO from the BlackBoxOptim package and GN_CRS2_LM for the global case while LD_SLSQP,LN_BOBYQA and LN_NELDERMEAD for the local case from the NLopt package perform the best.
• Another algorithm being used is the QuadDIRECT algorithm, it gives very good results in the shorter problem case but doesn't do very well in the case of the longer problems.
• The choice of global versus local optimization make a huge difference in the timings. BBO tends to find the correct solution for a global optimization setup. For local optimization, most methods in NLopt, like :LN_BOBYQA, solve the problem very fast but require a good initial condition.
• The different backends options available for Bayesian method offer some tradeoffs between time, accuracy and control. It is observed that sufficiently high accuracy can be observed with any of the backends with the fine tuning of stepsize, constraints on the parameters, tightness of the priors and number of iterations being passed.

Interactive Notebooks

To generate the interactive notebooks, first install the SciMLBenchmarks, instantiate the environment, and then run SciMLBenchmarks.open_notebooks(). This looks as follows:

]add SciMLBenchmarks#master
]activate SciMLBenchmarks
]instantiate
using SciMLBenchmarks
SciMLBenchmarks.open_notebooks()

The benchmarks will be generated at your pwd() in a folder called generated_notebooks.

Note that when running the benchmarks, the packages are not automatically added. Thus you will need to add the packages manually or use the internal Project/Manifest tomls to instantiate the correct packages. This can be done by activating the folder of the benchmarks. For example,

using Pkg
Pkg.activate(joinpath(pkgdir(SciMLBenchmarks),"benchmarks","NonStiffODE"))
Pkg.instantiate()

will add all of the packages required to run any benchmark in the NonStiffODE folder.

Contributing

All of the files are generated from the Weave.jl files in the benchmarks folder of the SciMLBenchmarks.jl repository. The generation process runs automatically, and thus one does not necessarily need to test the Weave process locally. Instead, simply open a PR that adds/updates a file in the benchmarks folder and the PR will generate the benchmark on demand. Its artifacts can then be inspected in the Buildkite as described below before merging. Note that it will use the Project.toml and Manifest.toml of the subfolder, so any changes to dependencies requires that those are updated.

Reporting Bugs and Issues

Report any bugs or issues at the SciMLBenchmarks repository.

Inspecting Benchmark Results

To see benchmark results before merging, click into the BuildKite, click onto Artifacts, and then investigate the trained results.

Manually Generating Files

All of the files are generated from the Weave.jl files in the benchmarks folder. To run the generation process, do for example:

]activate SciMLBenchmarks # Get all of the packages
using SciMLBenchmarks
SciMLBenchmarks.weave_file(joinpath(pkgdir(SciMLBenchmarks),"benchmarks","NonStiffODE"),"linear_wpd.jmd")

To generate all of the files in a folder, for example, run:

SciMLBenchmarks.weave_folder(joinpath(pkgdir(SciMLBenchmarks),"benchmarks","NonStiffODE"))

To generate all of the notebooks, do:

SciMLBenchmarks.weave_all()

Each of the benchmarks displays the computer characteristics at the bottom of the benchmark. Since performance-necessary computations are normally performed on compute clusters, the official benchmarks use a workstation with an AMD EPYC 7502 32-Core Processor @ 2.50GHz to match the performance characteristics of a standard node in a high performance computing (HPC) cluster or cloud computing setup.

scimlbenchmarks.jl's Issues

Re-run DynamicalODE benchmarks

@SebastianM-C it looks like Weave.jl has issues with the modules. Could you update these to note use modules in the scripts?

Re-run parameter estimation benchmarks

This one needs to get an installation of Stan. DiffEqBayes does it like:

https://github.com/SciML/DiffEqBayes.jl/blob/master/.travis.yml#L23-L28

so we probably need to do that on the Docker.

BCR import location

@isaacsas we need to find out how to make the import command computer-independent.

http://benchmarks.juliadiffeq.org/html/StiffODE/BCR.html

prnbng = loadrxnetwork(BNGNetwork(), "BNGRepressilator",
                       joinpath(pathof(ReactionNetworkImporters),"..","..",
                       "data","bcr","bcr.net"))

That works on my laptop, but some reason not on the benchmarking server.

Documentation for getting started

I ran into a few issues trying to follow the instructions for getting the benchmarks up and running locally. I'm sure I'm missing some steps. I'll update the documentation, but I want to make sure that I understand the workflow before I do so.

Here are the issues I've run into:

1. In Interactive Notebooks, SciMLBenchmarks.open_notebooks() fails because the notebook folder does not exist. I'm assuming you have to run weave on one of the example to populate this folder before running open_notebooks()
2. In Contributing, running SciMLBenchmarks.weave_folder("NonStiffODE") fails with IOError: readdir("NonStiffODE"): no such file or directory (ENOENT). Looks like it needs the full path: SciMLBenchmarks.weave_folder("benchmarks/NonStiffODE") works.
3. Now that the notebook folder is populated, when running a notebook, it appears that the dependencies aren't being resolved properly. It seems that the notebooks are using the global environment, instead of the Project file for the folder. I'm not familiar enough with Julia's Pkg system to know how this is supposed to work. For instance, in notebooks/Jumps/Mendes_multistate_example.ipynb, the first cell fails with ArgumentError: Package DiffEqBase not found in current path for me.

So, how is the workflow supposed to go? Something like the following?

1. Load SciMLBenchmarks
2. Run weave
3. Run open_notebooks()
4. Open a notebook and run it in Jupyter (How do we resolve the package issue?)

Benchmarking with Gillespy2

Opening here a thread to keep track of progress for benchmarking with GillesPy2 and Gillepy2julia. (Have some health issues so will be little slow for now)

Bdw, has anybody tried using julia wrapper of gillespy2?? I had following issue
StochSS/GillesPy2lia#1

Add CMA packages to parameter estimation benchmarks

https://github.com/AStupidBear/CMAES.jl
https://github.com/AStupidBear/CMA.jl

@Vaibhavdixit02

No recompilation of Stan models in Bayesian benchmarks

https://github.com/SciML/DiffEqBayes.jl/pull/164/files#diff-ff2792dafb55c80dc353dbd4ff08d701R25-R30

Benchmarks for DDEs

We should start getting some benchmarks together for DDEs. There probably aren't many good ones with analytical solutions, so the appxsol stuff will have to be used on the simplest examples. Show the difference between constrained and unconstrained, the the difference that order makes in DDEs. Also see the difference between the OwrenZen optimized interpolant methods and the classic optimized RK methods to see if it really makes a difference.

@devmotion do you have any model equations to consider?

Add MKM stiff ODE Benchmark

using ModelingToolkit, Sundials, OrdinaryDiffEq, DiffEqDevTools, Plots

f = begin
        f = ((var"##MTIIPVar#15413", var"##MTKArg#15409", var"##MTKArg#15410", var"##MTKArg#15411")->begin
                    @inbounds begin
                            begin
                                (ModelingToolkit.fill_array_with_zero!)(var"##MTIIPVar#15413")
                                let (x₁, x₂, x₃, x₄, x₅, x₆, x₇, x₈, x₉, x₁₀, x₁₁, x₁₂, x₁₃, α₁, α₂, α₃, α₄, α₅, α₆, α₇, α₈, α₉, α₁₀, α₁₁, α₁₂, α₁₃, α₁₄, α₁₅, α₁₆, α₁₇, α₁₈, α₁₉, α₂₀, α₂₁, α₂₂, α₂₃, α₂₄, α₂₅, α₂₆, α₂₇, α₂₈, α₂₉, α₃₀, α₃₁, α₃₂, t) = (var"##MTKArg#15409"[1], var"##MTKArg#15409"[2], var"##MTKArg#15409"[3], var"##MTKArg#15409"[4], var"##MTKArg#15409"[5], var"##MTKArg#15409"[6], var"##MTKArg#15409"[7], var"##MTKArg#15409"[8], var"##MTKArg#15409"[9], var"##MTKArg#15409"[10], var"##MTKArg#15409"[11], var"##MTKArg#15409"[12], var"##MTKArg#15409"[13], var"##MTKArg#15410"[1], var"##MTKArg#15410"[2], var"##MTKArg#15410"[3], var"##MTKArg#15410"[4], var"##MTKArg#15410"[5], var"##MTKArg#15410"[6], var"##MTKArg#15410"[7], var"##MTKArg#15410"[8], var"##MTKArg#15410"[9], var"##MTKArg#15410"[10], var"##MTKArg#15410"[11], var"##MTKArg#15410"[12], var"##MTKArg#15410"[13], var"##MTKArg#15410"[14], var"##MTKArg#15410"[15], var"##MTKArg#15410"[16], var"##MTKArg#15410"[17], var"##MTKArg#15410"[18], var"##MTKArg#15410"[19], var"##MTKArg#15410"[20], var"##MTKArg#15410"[21], var"##MTKArg#15410"[22], var"##MTKArg#15410"[23], var"##MTKArg#15410"[24], var"##MTKArg#15410"[25], var"##MTKArg#15410"[26], var"##MTKArg#15410"[27], var"##MTKArg#15410"[28], var"##MTKArg#15410"[29], var"##MTKArg#15410"[30], var"##MTKArg#15410"[31], var"##MTKArg#15410"[32], var"##MTKArg#15411")
                                    var"##MTIIPVar#15413"[2] = ((((identity(0.0) + ((α₁₄ * (x₁ / 0.5)) * (x₁₀ / 0.5) - ((α₁₄ / exp((((identity(0) + 1.0 * ((α₁ / 2477.572) * true)) + -1.0 * ((α₂ / 2477.572) * true)) + 1.0 * ((α₁₀ / 2477.572) * true)) + -3.697891137634378)) * true) * (x₂ / 0.5))) - ((α₁₅ * (x₂ / 0.5)) * (x₁₁ / 0.5) - (((α₁₅ / exp(((((identity(0) + 1.0 * ((α₂ / 2477.572) * true)) + -1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₄ / 0.5)) * (x₇ / 0.5))) - ((α₁₉ * (x₂ / 0.5)) * (x₁₂ / 0.5) - ((α₁₉ / exp((((identity(0) + 1.0 * ((α₂ / 2477.572) * true)) + -1.0 * ((α₃ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₃ / 0.5))) + (α₂₀ * (x₃ / 0.5) - (((α₂₀ / exp((((identity(0) + -1.0 * ((α₂ / 2477.572) * true)) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₂ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[3] = (((((((identity(0.0) - ((α₁₆ * (x₃ / 0.5)) * (x₁₁ / 0.5) - (((α₁₆ / exp(((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₇ / 0.5))) + ((α₁₉ * (x₂ / 0.5)) * (x₁₂ / 0.5) - ((α₁₉ / exp((((identity(0) + 1.0 * ((α₂ / 2477.572) * true)) + -1.0 * ((α₃ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₃ / 0.5))) - (α₂₀ * (x₃ / 0.5) - (((α₂₀ / exp((((identity(0) + -1.0 * ((α₂ / 2477.572) * true)) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₂ / 0.5)) * (x₁₂ / 0.5))) - ((α₂₁ * (x₃ / 0.5)) * (x₁₂ / 0.5) - (((α₂₁ / exp(((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₄ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₄ / 0.5)) * (x₁₃ / 0.5))) + ((α₂₂ * (x₄ / 0.5)) * (x₁₃ / 0.5) - (((α₂₂ / exp(((((identity(0) + -1.0 * ((α₃ / 2477.572) * true)) + 1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₃ / 0.5)) * (x₁₂ / 0.5))) - ((α₃₁ * (x₃ / 0.5)) * (x₁₂ / 0.5) - ((α₃₁ / exp((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₆ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₆ / 0.5))) + (α₃₂ * (x₆ / 0.5) - (((α₃₂ / exp((((identity(0) + -1.0 * ((α₃ / 2477.572) * true)) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₃ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[4] = (((((identity(0.0) + ((α₁₅ * (x₂ / 0.5)) * (x₁₁ / 0.5) - (((α₁₅ / exp(((((identity(0) + 1.0 * ((α₂ / 2477.572) * true)) + -1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₄ / 0.5)) * (x₇ / 0.5))) + ((α₂₁ * (x₃ / 0.5)) * (x₁₂ / 0.5) - (((α₂₁ / exp(((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₄ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₄ / 0.5)) * (x₁₃ / 0.5))) - ((α₂₂ * (x₄ / 0.5)) * (x₁₃ / 0.5) - (((α₂₂ / exp(((((identity(0) + -1.0 * ((α₃ / 2477.572) * true)) + 1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₃ / 0.5)) * (x₁₂ / 0.5))) - ((α₂₃ * (x₄ / 0.5)) * (x₁₂ / 0.5) - ((α₂₃ / exp((((identity(0) + 1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₅ / 0.5))) + (α₂₄ * (x₅ / 0.5) - (((α₂₄ / exp((((identity(0) + -1.0 * ((α₄ / 2477.572) * true)) + 1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₄ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[5] = ((((((identity(0.0) + ((α₁₆ * (x₃ / 0.5)) * (x₁₁ / 0.5) - (((α₁₆ / exp(((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₇ / 0.5))) + ((α₁₇ * (x₆ / 0.5)) * (x₁₁ / 0.5) - (((α₁₇ / exp(((((identity(0) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₈ / 0.5))) + ((α₂₃ * (x₄ / 0.5)) * (x₁₂ / 0.5) - ((α₂₃ / exp((((identity(0) + 1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₅ / 0.5))) - (α₂₄ * (x₅ / 0.5) - (((α₂₄ / exp((((identity(0) + -1.0 * ((α₄ / 2477.572) * true)) + 1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₄ / 0.5)) * (x₁₂ / 0.5))) - ((α₂₅ * (x₅ / 0.5)) * (x₁₂ / 0.5) - (((α₂₅ / exp(((((identity(0) + 1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₁₀ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₁₀ / 0.5)) * (x₁₃ / 0.5))) + ((α₂₆ * (x₁₀ / 0.5)) * (x₁₃ / 0.5) - (((α₂₆ / exp(((((identity(0) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₁₀ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₅ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[6] = ((((identity(0.0) - ((α₁₇ * (x₆ / 0.5)) * (x₁₁ / 0.5) - (((α₁₇ / exp(((((identity(0) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₈ / 0.5))) - (α₁₈ * (x₆ / 0.5) - (((α₁₈ / exp((((identity(0) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₉ / 2477.572) * true)) + -1.0 * ((α₁₀ / 2477.572) * true)) + 3.697891137634378)) * true) * (x₁₀ / 0.5)) * (x₉ / 0.5))) + ((α₃₁ * (x₃ / 0.5)) * (x₁₂ / 0.5) - ((α₃₁ / exp((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₆ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₆ / 0.5))) - (α₃₂ * (x₆ / 0.5) - (((α₃₂ / exp((((identity(0) + -1.0 * ((α₃ / 2477.572) * true)) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₃ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[7] = ((((identity(0.0) + ((α₁₅ * (x₂ / 0.5)) * (x₁₁ / 0.5) - (((α₁₅ / exp(((((identity(0) + 1.0 * ((α₂ / 2477.572) * true)) + -1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₄ / 0.5)) * (x₇ / 0.5))) + ((α₁₆ * (x₃ / 0.5)) * (x₁₁ / 0.5) - (((α₁₆ / exp(((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₇ / 0.5))) - ((α₂₇ * (x₇ / 0.5)) * (x₁₂ / 0.5) - ((α₂₇ / exp((((identity(0) + 1.0 * ((α₇ / 2477.572) * true)) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₈ / 0.5))) + (α₂₈ * (x₈ / 0.5) - (((α₂₈ / exp((((identity(0) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₈ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₇ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[8] = (((((identity(0.0) + ((α₁₇ * (x₆ / 0.5)) * (x₁₁ / 0.5) - (((α₁₇ / exp(((((identity(0) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₈ / 0.5))) + ((α₂₇ * (x₇ / 0.5)) * (x₁₂ / 0.5) - ((α₂₇ / exp((((identity(0) + 1.0 * ((α₇ / 2477.572) * true)) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -3.697891137634378)) * false) * (x₈ / 0.5))) - (α₂₈ * (x₈ / 0.5) - (((α₂₈ / exp((((identity(0) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₈ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 3.697891137634378)) * false) * (x₇ / 0.5)) * (x₁₂ / 0.5))) - ((α₂₉ * (x₈ / 0.5)) * (x₁₂ / 0.5) - (((α₂₉ / exp(((((identity(0) + 1.0 * ((α₈ / 2477.572) * true)) + -1.0 * ((α₁₁ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₁₃ / 0.5)) * (x₁₁ / 0.5))) + ((α₃₀ * (x₁₃ / 0.5)) * (x₁₁ / 0.5) - (((α₃₀ / exp(((((identity(0) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₈ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[10] = ((((identity(0.0) - ((α₁₄ * (x₁ / 0.5)) * (x₁₀ / 0.5) - ((α₁₄ / exp((((identity(0) + 1.0 * ((α₁ / 2477.572) * true)) + -1.0 * ((α₂ / 2477.572) * true)) + 1.0 * ((α₁₀ / 2477.572) * true)) + -3.697891137634378)) * true) * (x₂ / 0.5))) + (α₁₈ * (x₆ / 0.5) - (((α₁₈ / exp((((identity(0) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₉ / 2477.572) * true)) + -1.0 * ((α₁₀ / 2477.572) * true)) + 3.697891137634378)) * true) * (x₁₀ / 0.5)) * (x₉ / 0.5))) + ((α₂₅ * (x₅ / 0.5)) * (x₁₂ / 0.5) - (((α₂₅ / exp(((((identity(0) + 1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₁₀ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₁₀ / 0.5)) * (x₁₃ / 0.5))) - ((α₂₆ * (x₁₀ / 0.5)) * (x₁₃ / 0.5) - (((α₂₆ / exp(((((identity(0) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₁₀ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₅ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                    var"##MTIIPVar#15413"[11] = (((((identity(0.0) - ((α₁₅ * (x₂ / 0.5)) * (x₁₁ / 0.5) - (((α₁₅ / exp(((((identity(0) + 1.0 * ((α₂ / 2477.572) * true)) + -1.0 * ((α₄ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₄ / 0.5)) * (x₇ / 0.5))) - ((α₁₆ * (x₃ / 0.5)) * (x₁₁ / 0.5) - (((α₁₆ / exp(((((identity(0) + 1.0 * ((α₃ / 2477.572) * true)) + -1.0 * ((α₅ / 2477.572) * true)) + -1.0 * ((α₇ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₇ / 0.5))) - ((α₁₇ * (x₆ / 0.5)) * (x₁₁ / 0.5) - (((α₁₇ / exp(((((identity(0) + -1.0 * ((α₅ / 2477.572) * true)) + 1.0 * ((α₆ / 2477.572) * true)) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + 0.0)) * true) * (x₅ / 0.5)) * (x₈ / 0.5))) + ((α₂₉ * (x₈ / 0.5)) * (x₁₂ / 0.5) - (((α₂₉ / exp(((((identity(0) + 1.0 * ((α₈ / 2477.572) * true)) + -1.0 * ((α₁₁ / 2477.572) * true)) + 1.0 * ((α₁₂ / 2477.572) * true)) + -1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₁₃ / 0.5)) * (x₁₁ / 0.5))) - ((α₃₀ * (x₁₃ / 0.5)) * (x₁₁ / 0.5) - (((α₃₀ / exp(((((identity(0) + -1.0 * ((α₈ / 2477.572) * true)) + 1.0 * ((α₁₁ / 2477.572) * true)) + -1.0 * ((α₁₂ / 2477.572) * true)) + 1.0 * ((α₁₃ / 2477.572) * true)) + 0.0)) * false) * (x₈ / 0.5)) * (x₁₂ / 0.5))) * 0.5
                                end
                            end
                        end
                    nothing
                end)
        tgrad = nothing
        jac = ((var"##MTIIPVar#15420", var"##MTKArg#15416", var"##MTKArg#15417", var"##MTKArg#15418")->begin
                    @inbounds begin
                            begin
                                (ModelingToolkit.fill_array_with_zero!)(var"##MTIIPVar#15420")
                                let (x₁, x₂, x₃, x₄, x₅, x₆, x₇, x₈, x₉, x₁₀, x₁₁, x₁₂, x₁₃, α₁, α₂, α₃, α₄, α₅, α₆, α₇, α₈, α₉, α₁₀, α₁₁, α₁₂, α₁₃, α₁₄, α₁₅, α₁₆, α₁₇, α₁₈, α₁₉, α₂₀, α₂₁, α₂₂, α₂₃, α₂₄, α₂₅, α₂₆, α₂₇, α₂₈, α₂₉, α₃₀, α₃₁, α₃₂, t) = (var"##MTKArg#15416"[1], var"##MTKArg#15416"[2], var"##MTKArg#15416"[3], var"##MTKArg#15416"[4], var"##MTKArg#15416"[5], var"##MTKArg#15416"[6], var"##MTKArg#15416"[7], var"##MTKArg#15416"[8], var"##MTKArg#15416"[9], var"##MTKArg#15416"[10], var"##MTKArg#15416"[11], var"##MTKArg#15416"[12], var"##MTKArg#15416"[13], var"##MTKArg#15417"[1], var"##MTKArg#15417"[2], var"##MTKArg#15417"[3], var"##MTKArg#15417"[4], var"##MTKArg#15417"[5], var"##MTKArg#15417"[6], var"##MTKArg#15417"[7], var"##MTKArg#15417"[8], var"##MTKArg#15417"[9], var"##MTKArg#15417"[10], var"##MTKArg#15417"[11], var"##MTKArg#15417"[12], var"##MTKArg#15417"[13], var"##MTKArg#15417"[14], var"##MTKArg#15417"[15], var"##MTKArg#15417"[16], var"##MTKArg#15417"[17], var"##MTKArg#15417"[18], var"##MTKArg#15417"[19], var"##MTKArg#15417"[20], var"##MTKArg#15417"[21], var"##MTKArg#15417"[22], var"##MTKArg#15417"[23], var"##MTKArg#15417"[24], var"##MTKArg#15417"[25], var"##MTKArg#15417"[26], var"##MTKArg#15417"[27], var"##MTKArg#15417"[28], var"##MTKArg#15417"[29], var"##MTKArg#15417"[30], var"##MTKArg#15417"[31], var"##MTKArg#15417"[32], var"##MTKArg#15418")
                                    var"##MTIIPVar#15420"[2] = 2.0 * x₁₀ * α₁₄
                                    var"##MTIIPVar#15420"[10] = -2.0 * x₁₀ * α₁₄
                                    var"##MTIIPVar#15420"[15] = 0.5 * (-4.0 * x₁₁ * α₁₅ + -4.0 * x₁₂ * α₁₉ + -2.0 * exp(-3.697891137634378 + -0.00040362096439578745α₂ + 0.00040362096439578745 * (α₁ + α₁₀)) ^ -1 * α₁₄)
                                    var"##MTIIPVar#15420"[16] = 2.0 * x₁₂ * α₁₉
                                    var"##MTIIPVar#15420"[17] = 2.0 * x₁₁ * α₁₅
                                    var"##MTIIPVar#15420"[20] = 2.0 * x₁₁ * α₁₅
                                    var"##MTIIPVar#15420"[23] = exp(-3.697891137634378 + -0.00040362096439578745α₂ + 0.00040362096439578745 * (α₁ + α₁₀)) ^ -1 * α₁₄
                                    var"##MTIIPVar#15420"[24] = -2.0 * x₁₁ * α₁₅
                                    var"##MTIIPVar#15420"[28] = α₂₀
                                    var"##MTIIPVar#15420"[29] = 0.5 * (-2.0α₂₀ + -4.0 * x₁₁ * α₁₆ + -4.0 * x₁₂ * (α₂₁ + α₃₁))
                                    var"##MTIIPVar#15420"[30] = 2.0 * x₁₂ * α₂₁
                                    var"##MTIIPVar#15420"[31] = 2.0 * x₁₁ * α₁₆
                                    var"##MTIIPVar#15420"[32] = 2.0 * x₁₂ * α₃₁
                                    var"##MTIIPVar#15420"[33] = 2.0 * x₁₁ * α₁₆
                                    var"##MTIIPVar#15420"[37] = -2.0 * x₁₁ * α₁₆
                                    var"##MTIIPVar#15420"[41] = 2.0 * x₇ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[42] = 2.0 * x₁₃ * α₂₂
                                    var"##MTIIPVar#15420"[43] = 0.5 * (-4.0 * x₁₂ * α₂₃ + -4.0 * x₁₃ * α₂₂ + -4.0 * x₇ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1)
                                    var"##MTIIPVar#15420"[44] = 2.0 * x₁₂ * α₂₃
                                    var"##MTIIPVar#15420"[46] = -2.0 * x₇ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[50] = 2.0 * x₇ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[55] = 2.0 * x₇ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[56] = α₂₄
                                    var"##MTIIPVar#15420"[57] = 0.5 * (-2.0α₂₄ + -4.0 * x₁₂ * α₂₅ + -4.0 * x₇ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1 + -4.0 * x₈ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1)
                                    var"##MTIIPVar#15420"[58] = 2.0 * x₈ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1
                                    var"##MTIIPVar#15420"[59] = -2.0 * x₇ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[60] = -2.0 * x₈ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1
                                    var"##MTIIPVar#15420"[62] = 2.0 * x₁₂ * α₂₅
                                    var"##MTIIPVar#15420"[63] = 0.5 * (4.0 * x₇ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1 + 4.0 * x₈ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1)
                                    var"##MTIIPVar#15420"[68] = α₃₂
                                    var"##MTIIPVar#15420"[70] = 2.0 * x₁₁ * α₁₇
                                    var"##MTIIPVar#15420"[71] = 0.5 * (-2.0 * (α₁₈ + α₃₂) + -4.0 * x₁₁ * α₁₇)
                                    var"##MTIIPVar#15420"[73] = 2.0 * x₁₁ * α₁₇
                                    var"##MTIIPVar#15420"[75] = α₁₈
                                    var"##MTIIPVar#15420"[76] = -2.0 * x₁₁ * α₁₇
                                    var"##MTIIPVar#15420"[80] = 2.0 * x₄ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[81] = 2.0 * x₅ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[82] = -2.0 * x₄ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[83] = -2.0 * x₅ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1
                                    var"##MTIIPVar#15420"[85] = 0.5 * (-4.0 * x₁₂ * α₂₇ + -4.0 * x₄ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1 + -4.0 * x₅ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1)
                                    var"##MTIIPVar#15420"[86] = 2.0 * x₁₂ * α₂₇
                                    var"##MTIIPVar#15420"[89] = 0.5 * (4.0 * x₄ * α₁₅ * exp(0.00040362096439578745 * (α₁₁ + α₂) + -0.00040362096439578745 * (α₄ + α₇)) ^ -1 + 4.0 * x₅ * α₁₆ * exp(0.00040362096439578745 * (α₁₁ + α₃) + -0.00040362096439578745 * (α₅ + α₇)) ^ -1)
                                    var"##MTIIPVar#15420"[96] = -2.0 * x₅ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1
                                    var"##MTIIPVar#15420"[97] = 2.0 * x₅ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1
                                    var"##MTIIPVar#15420"[98] = α₂₈
                                    var"##MTIIPVar#15420"[99] = 0.5 * (-2.0α₂₈ + -4.0 * x₁₂ * α₂₉ + -4.0 * x₅ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1)
                                    var"##MTIIPVar#15420"[102] = 0.5 * (4.0 * x₁₂ * α₂₉ + 4.0 * x₅ * α₁₇ * exp(0.00040362096439578745 * (α₁₁ + α₆) + -0.00040362096439578745 * (α₅ + α₈)) ^ -1)
                                    var"##MTIIPVar#15420"[110] = 2.0 * x₁₀ * exp(3.697891137634378 + 0.00040362096439578745α₆ + -0.00040362096439578745 * (α₁₀ + α₉)) ^ -1 * α₁₈
                                    var"##MTIIPVar#15420"[114] = -2.0 * x₁₀ * exp(3.697891137634378 + 0.00040362096439578745α₆ + -0.00040362096439578745 * (α₁₀ + α₉)) ^ -1 * α₁₈
                                    var"##MTIIPVar#15420"[119] = 2.0 * x₁ * α₁₄
                                    var"##MTIIPVar#15420"[122] = 2.0 * x₁₃ * α₂₆
                                    var"##MTIIPVar#15420"[123] = 2.0 * x₉ * exp(3.697891137634378 + 0.00040362096439578745α₆ + -0.00040362096439578745 * (α₁₀ + α₉)) ^ -1 * α₁₈
                                    var"##MTIIPVar#15420"[127] = 0.5 * (-4.0 * x₁ * α₁₄ + -4.0 * x₁₃ * α₂₆ + -4.0 * x₉ * exp(3.697891137634378 + 0.00040362096439578745α₆ + -0.00040362096439578745 * (α₁₀ + α₉)) ^ -1 * α₁₈)
                                    var"##MTIIPVar#15420"[132] = -2.0 * x₂ * α₁₅
                                    var"##MTIIPVar#15420"[133] = -2.0 * x₃ * α₁₆
                                    var"##MTIIPVar#15420"[134] = 2.0 * x₂ * α₁₅
                                    var"##MTIIPVar#15420"[135] = 0.5 * (4.0 * x₃ * α₁₆ + 4.0 * x₆ * α₁₇)
                                    var"##MTIIPVar#15420"[136] = -2.0 * x₆ * α₁₇
                                    var"##MTIIPVar#15420"[137] = 0.5 * (4.0 * x₂ * α₁₅ + 4.0 * x₃ * α₁₆)
                                    var"##MTIIPVar#15420"[138] = 0.5 * (4.0 * x₁₃ * α₃₀ + 4.0 * x₆ * α₁₇)
                                    var"##MTIIPVar#15420"[141] = 0.5 * (-4.0 * x₁₃ * α₃₀ + -4.0 * x₂ * α₁₅ + -4.0 * x₃ * α₁₆ + -4.0 * x₆ * α₁₇)
                                    var"##MTIIPVar#15420"[145] = -2.0 * x₂ * α₁₉
                                    var"##MTIIPVar#15420"[146] = 0.5 * (4.0 * x₂ * α₁₉ + -4.0 * x₃ * (α₂₁ + α₃₁))
                                    var"##MTIIPVar#15420"[147] = 0.5 * (4.0 * x₃ * α₂₁ + -4.0 * x₄ * α₂₃)
                                    var"##MTIIPVar#15420"[148] = 0.5 * (4.0 * x₄ * α₂₃ + -4.0 * x₅ * α₂₅)
                                    var"##MTIIPVar#15420"[149] = 2.0 * x₃ * α₃₁
                                    var"##MTIIPVar#15420"[150] = -2.0 * x₇ * α₂₇
                                    var"##MTIIPVar#15420"[151] = 0.5 * (4.0 * x₇ * α₂₇ + -4.0 * x₈ * α₂₉)
                                    var"##MTIIPVar#15420"[153] = 2.0 * x₅ * α₂₅
                                    var"##MTIIPVar#15420"[154] = 2.0 * x₈ * α₂₉
                                    var"##MTIIPVar#15420"[159] = 2.0 * x₄ * α₂₂
                                    var"##MTIIPVar#15420"[160] = -2.0 * x₄ * α₂₂
                                    var"##MTIIPVar#15420"[161] = 2.0 * x₁₀ * α₂₆
                                    var"##MTIIPVar#15420"[164] = 2.0 * x₁₁ * α₃₀
                                    var"##MTIIPVar#15420"[166] = -2.0 * x₁₀ * α₂₆
                                    var"##MTIIPVar#15420"[167] = -2.0 * x₁₁ * α₃₀
                                end
                            end
                        end
                    nothing
                end)
        M = UniformScaling{Bool}(true)
        ODEFunction{true}(f, jac = jac, tgrad = tgrad, mass_matrix = M, jac_prototype = nothing, syms = [:x₁, :x₂, :x₃, :x₄, :x₅, :x₆, :x₇, :x₈, :x₉, :x₁₀, :x₁₁, :x₁₂, :x₁₃])
    end
u0 = [0.25, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 1.81e-6, 55500.0]
tspan = (0.0, 1.0e-5)
p = [153400.0, 134110.0, 116740.0, -9648.3, -14472.0, 147620.0, 51136.0, 19296.6, 179458.0, 0.0, 0.0, 0.0, 0.0, 1.0e8, 47320.473243810055, 3.4311035379976365e6, 3.4311035379976365e6, 1.6603867185365406, 5.52486e14, 2.3763009937807666e12, 5.52486e14, 8.308386964362935e-9, 5.52486e14, 7.882700332950447e13, 5.52486e14, 1.900083786848033e17, 5.52486e14, 2.0709841124207065e8, 5.52486e14, 2.2895136995798813e11, 5.52486e14, 3.010694854288576e19]
ODEProblem(f, u0, tspan, p)

sol = solve(prob,CVODE_BDF(),abstol=1e-20,reltol=1e-15);
test_sol = TestSolution(sol)
abstols = 1.0 ./ 10.0 .^ (4:0.5:8);
reltols = 1.0 ./ 10.0 .^ (5:13);
length(abstols), length(reltols)
setups = [Dict(:alg=>Rosenbrock23(),:dtmin=>0),
          Dict(:alg=>Rodas5(),
               :abstols=>1.0 ./ 10.0 .^ (6:0.5:8),
               :reltols=>1.0 ./ 10.0 .^ (7:11),
               :dtmin=>0),
          Dict(:alg=>Rodas4(),
                 :dtmin=>0),
          Dict(:alg=>KenCarp4()),
          Dict(:alg=>KenCarp47()),
          #Dict(:alg=>lsoda(),
            #   :abstols=>1.0 ./ 10.0 .^ (3:0.5:7),
            #  :reltols=>1.0 ./ 10.0 .^ (3:11),
            #   :verbose => false),
          Dict(:alg=>RadauIIA5(),:dtmin=>0),
          Dict(:alg=>CVODE_BDF(),
               :abstols=>1.0 ./ 10.0 .^ (3:0.5:7),
               :reltols=>1.0 ./ 10.0 .^ (3:11)),
          Dict(:alg=>TRBDF2()),
          #Dict(:alg=>ODEInterfaceDiffEq.ddebdf()),
          #Dict(:alg=>ODEInterfaceDiffEq.rodas()),
          #Dict(:alg=>ODEInterfaceDiffEq.radau()),
          ];
@time wp = WorkPrecisionSet(prob,abstols,reltols,setups;verbose=true,
                      save_everystep=false,appxsol=test_sol,maxiters=Int(1e5),numruns=10)
plot(wp)

From the original ReactionMechanismSimulator code:

using OrdinaryDiffEq, DiffEqDevTools, Sundials, ParameterizedFunctions, Plots, ODE, ODEInterfaceDiffEq, LSODA
gr() # gr(fmt=:png)
using LinearAlgebra
LinearAlgebra.BLAS.set_num_threads(1)

using ReactionMechanismSimulator
phaseDict = readinput("ORR.rms")
spcs = phaseDict["phase"]["Species"]; #mechanism dictionaries index: 
rxns = phaseDict["phase"]["Reactions"];
AdivV = 1.0
is = IdealSurface(spcs,rxns;name="surf");
initialconds = Dict(["T"=>298.0,"A"=>0.5,"Phi"=>0.0,"OHA"=>0.0,"B*"=>0.5,"O2"=>0.25*AdivV,"H+"=>1.81e-6*AdivV,"H2O2aq"=>0.0*AdivV,"H2O"=>5.55e4*AdivV])
domain,y0,p =ConstantTAPhiDomain(phase=is,initialconds=initialconds;sensitivity=false,
    constantspecies=["H+","O2","H2O2aq","H2O"]);
react = Reactor(domain,y0,(0.0,1e-5),p=p,forwarddiff=true);
sol = solve(react.ode,CVODE_BDF(),abstol=1e-20,reltol=1e-15);
test_sol = TestSolution(sol)
abstols = 1.0 ./ 10.0 .^ (4:0.5:8);
reltols = 1.0 ./ 10.0 .^ (5:13);
length(abstols), length(reltols)
setups = [Dict(:alg=>Rosenbrock23(),:dtmin=>0),
          Dict(:alg=>Rodas5(),
               :abstols=>1.0 ./ 10.0 .^ (6:0.5:8),
               :reltols=>1.0 ./ 10.0 .^ (7:11),
               :dtmin=>0),
          Dict(:alg=>Rodas4(),
                 :dtmin=>0),
          Dict(:alg=>KenCarp4()),
          Dict(:alg=>KenCarp47()),
          Dict(:alg=>lsoda(),
               :abstols=>1.0 ./ 10.0 .^ (3:0.5:7),
               :reltols=>1.0 ./ 10.0 .^ (3:11),
               :verbose => false),
          Dict(:alg=>RadauIIA5(),:dtmin=>0),
          Dict(:alg=>CVODE_BDF(),
               :abstols=>1.0 ./ 10.0 .^ (3:0.5:7),
               :reltols=>1.0 ./ 10.0 .^ (3:11)),
          Dict(:alg=>TRBDF2()),
          Dict(:alg=>ODEInterfaceDiffEq.ddebdf()),
          Dict(:alg=>ODEInterfaceDiffEq.rodas()),
          Dict(:alg=>ODEInterfaceDiffEq.radau()),
          ];
@time wp = WorkPrecisionSet(react.ode,abstols,reltols,setups;verbose=true,
                      save_everystep=false,appxsol=test_sol,maxiters=Int(1e5),numruns=10)
plot(wp)

using ModelingToolkit
sys = modelingtoolkitize(react.ode)

ODEProblemExpr(sys,react.ode.u0,react.ode.tspan, react.ode.p, jac=true)

savefig("plots.png")

State-dependent delay differential equation benchmark

@slwu89 donated the following benchmark problem

https://github.com/davewi13/Temperate-Mosquito-DDE
https://gist.github.com/ChrisRackauckas/26b97f963c5f8ca46da19959a9bbbca4

The CR version of the Julia code is a version I updated a little bit. This will be a good benchmark problem for eventual RADAR5-like implementations and such since the Fortran code is right there.

Missing BCR (1122) benchmark page

The page describing the BCR (1122) stiff ODE benchmark at https://benchmarks.sciml.ai/html/Bio/BCR.html is missing and returns a 404 error.

Fix AdaptiveSDE benchmarks

https://github.com/SciML/DiffEqBenchmarks.jl/blob/5acdeba99e5f43e2748c0beadf90a48233f2f373/markdown/AdaptiveSDE/AdaptiveEfficiencyTests.md

https://github.com/SciML/DiffEqBenchmarks.jl/blob/5acdeba99e5f43e2748c0beadf90a48233f2f373/markdown/AdaptiveSDE/qmaxDetermination.md

Filament needs to be fixed

@dextorious if you could help :)

Oval2 Timings need an update

Benchmarks for SSA/jump systems

Before starting any benchmarking, it's probably a good idea to think about what we want to look at. Some ideas:

1. Varying number of species, number of reactions, and network topologies (i.e. sparsity). Many SSAs are designed to scale well in one of number of species or number of reactions, but may not scale well as both are increased. Many SSAs are designed to exploit sparsity in network structure, but may be slower than the basic direct method for dense networks.

2. Increasing simulation stopping time and/or species population sizes. Both lead to an increase in the number of jumps (i.e. events) within one simulation.

3. Separation vs. clustering of jump rates. Some SSAs work very well for systems with large timescale separations (i.e. they are optimized to quickly handle frequent jumps). Systems with many identical jump rates (like diffusion systems) may cause them issues.

Testing "accuracy" seems less relevant for exact SSAs; this would really just be looking at sampling error, which should converge the same for all exact methods...

Fix automated notebooks

Was done in SciMLTutorials in SciML/SciMLTutorials.jl#445, SciML/SciMLTutorials.jl#446, + SciML/SciMLTutorials.jl#448 . But it's not clear it's worth the effort for this repo.

Benchmarks for Dynamical ODEs

We should get some dynamical ODEs up, and benchmark 1st order methods, RKN methods, and symplectic methods against each other. Then we should show the energy preservation property and how well different symplectic methods achieve it on given problems.

@YingboMa

Fix NBodySimulator Benchmarks

Continuouing #245 here.

@SebastianM-C these time out. How long do they take?

Benchmarks for stiff DDEs

@devmotion does the Enright DDE test problem set have some good stiff problems?

Fix Jumps Benchmarks

Continuation of #230

@isaacsas

Finishing up the new benchmarking suite

https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/44#e548f23b-fe9b-423f-a56e-6d119291c3bd

• This only ran linear_wpd.jmd, so I'm a bit confused why there's so many artifacts?
• Where is it writing the artifacts to?
• The bottom of the benchmarks has `ERROR: ArgumentError: Package SciMLBenchmarks not found in current path:
• Run import Pkg; Pkg.add("SciMLBenchmarks") to install the SciMLBenchmarks package.`: do we need to add that to every Project.toml or can we add it globally?
• It seems like it only triggers on changes to the .jmd, and not the Project.toml. Does that mean it won't trigger on updates to the Manifest.toml either yet?

Benchmarks for ExpRK methods

We should add one or two typical ExpRK methods to the stiff ODE benchmarks, and then do a separate one that benchmarks the impact of parameters specific to Krylov methods (e.g. Krylov dimension size and IOP).

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Make a Brusselator benchmark

How have we never made one?

using OrdinaryDiffEq

const N = 128
const xyd_brusselator = range(0,stop=1,length=N)
brusselator_f(x, y, t) = (((x-0.3)^2 + (y-0.6)^2) <= 0.1^2) * (t >= 1.1) * 5.
limit(a, N) = a == N+1 ? 1 : a == 0 ? N : a
function brusselator_2d_loop(du, u, p, t)
  A, B, alpha, dx = p
  alpha = alpha/dx^2
  @inbounds for I in CartesianIndices((N, N))
    i, j = Tuple(I)
    x, y = xyd_brusselator[I[1]], xyd_brusselator[I[2]]
    ip1, im1, jp1, jm1 = limit(i+1, N), limit(i-1, N), limit(j+1, N), limit(j-1, N)
    du[i,j,1] = alpha*(u[im1,j,1] + u[ip1,j,1] + u[i,jp1,1] + u[i,jm1,1] - 4u[i,j,1]) +
                B + u[i,j,1]^2*u[i,j,2] - (A + 1)*u[i,j,1] + brusselator_f(x, y, t)
    du[i,j,2] = alpha*(u[im1,j,2] + u[ip1,j,2] + u[i,jp1,2] + u[i,jm1,2] - 4u[i,j,2]) +
                A*u[i,j,1] - u[i,j,1]^2*u[i,j,2]
    end
end
p = (3.4, 1., 10., step(xyd_brusselator))

function init_brusselator_2d(xyd)
  N = length(xyd)
  u = zeros(N, N, 2)
  for I in CartesianIndices((N, N))
    x = xyd[I[1]]
    y = xyd[I[2]]
    u[I,1] = 22*(y*(1-y))^(3/2)
    u[I,2] = 27*(x*(1-x))^(3/2)
  end
  u
end
u0 = init_brusselator_2d(xyd_brusselator)

prob_ode_brusselator_2d_dense = ODEProblem(brusselator_2d_loop,
                                     init_brusselator_2d(xyd_brusselator),
                                     (0.,11.5),p)

using BenchmarkTools
@btime solve(prob_ode_brusselator_2d_dense,TRBDF2(),save_everystep=false,progress=true)


using SparsityDetection, SparseArrays
input = rand(N,N,2)
output = similar(input)
sparsity_pattern = sparsity!(brusselator_2d_loop,output,input,p,0.0)
jac_sparsity = Float64.(sparse(sparsity_pattern))

using SparseDiffTools
colorvec = matrix_colors(jac_sparsity)
@show maximum(colorvec) # maximum(colorvec) = 12

f_nocolor = ODEFunction(brusselator_2d_loop;jac_prototype=jac_sparsity)
prob_ode_brusselator_2d_sparse_nocolor = ODEProblem(f_nocolor,
                                     init_brusselator_2d(xyd_brusselator),
                                     (0.,11.5),p)
@btime solve(prob_ode_brusselator_2d_sparse_nocolor,TRBDF2(),save_everystep=false,progress=true)

f = ODEFunction(brusselator_2d_loop;jac_prototype=jac_sparsity,
                                    colorvec=colorvec)
prob_ode_brusselator_2d_sparse = ODEProblem(f,
                                     init_brusselator_2d(xyd_brusselator),
                                     (0.,11.5),p)

println("Rodas5")
@btime solve(prob_ode_brusselator_2d_sparse,Rodas5(),save_everystep=false)

import Pardiso
println("Rodas5 MKLPardiso")
@btime solve(prob_ode_brusselator_2d_sparse,Rodas5(linsolve = DiffEqBase.MKLPardisoFactorize()),save_everystep=false)

println("TRBDF2")
@btime solve(prob_ode_brusselator_2d_sparse,TRBDF2(),save_everystep=false)
println("TRBDF2 MKLPardiso")
@btime solve(prob_ode_brusselator_2d_sparse,TRBDF2(linsolve = DiffEqBase.MKLPardisoFactorize()),save_everystep=false)

println("KenCarp4")
@btime solve(prob_ode_brusselator_2d_sparse,KenCarp4(),save_everystep=false)
println("KenCarp4 MKLPardiso")
@btime solve(prob_ode_brusselator_2d_sparse,KenCarp4(linsolve = DiffEqBase.MKLPardisoFactorize()),save_everystep=false)

using Sundials
println("CVODE with GMRES")
@btime solve(prob_ode_brusselator_2d_sparse,CVODE_BDF(linear_solver=:GMRES),save_everystep=false)

Convert reaction diffusion problem into a benchmark

using OrdinaryDiffEq, RecursiveArrayTools, LinearAlgebra, Test, SparseArrays, SparseDiffTools, Sundials

# Define the constants for the PDE
const α₂ = 1.0
const α₃ = 1.0
const β₁ = 1.0
const β₂ = 1.0
const β₃ = 1.0
const r₁ = 1.0
const r₂ = 1.0
const D = 100.0
const γ₁ = 0.1
const γ₂ = 0.1
const γ₃ = 0.1
const N = 256
const X = reshape([i for i in 1:N for j in 1:N],N,N)
const Y = reshape([j for i in 1:N for j in 1:N],N,N)
const α₁ = 1.0.*(X.>=4*N/5)

const Mx = Tridiagonal([1.0 for i in 1:N-1],[-2.0 for i in 1:N],[1.0 for i in 1:N-1])
const My = copy(Mx)
Mx[2,1] = 2.0
Mx[end-1,end] = 2.0
My[1,2] = 2.0
My[end,end-1] = 2.0

# Define the initial condition as normal arrays
u0 = zeros(N,N,3)

const MyA = zeros(N,N);
const AMx = zeros(N,N);
const DA = zeros(N,N);
# Define the discretized PDE as an ODE function
function f(du,u,p,t)
   A = @view  u[:,:,1]
   B = @view  u[:,:,2]
   C = @view  u[:,:,3]
  dA = @view du[:,:,1]
  dB = @view du[:,:,2]
  dC = @view du[:,:,3]
  mul!(MyA,My,A)
  mul!(AMx,A,Mx)
  @. DA = D*(MyA + AMx)
  @. dA = DA + α₁ - β₁*A - r₁*A*B + r₂*C
  @. dB = α₂ - β₂*B - r₁*A*B + r₂*C
  @. dC = α₃ - β₃*C + r₁*A*B - r₂*C
end

Iy = SparseMatrixCSC(I,N,N)
Ix = SparseMatrixCSC(I,N,N)
fJ = ones(3,3)
Dz = [1 0 0
      0 0 0
      0 0 0]
jacsparsity = kron(Dz,Iy,sparse(Mx)) + kron(Dz,sparse(My),Ix) + kron(fJ,Iy,Ix)
@time colorvec = matrix_colors(jacsparsity)

# Solve the ODE
ff = ODEFunction(f,colorvec=colorvec,jac_prototype=jacsparsity)
prob = ODEProblem(ff,u0,(0.0,100.0))
sol = solve(prob,BS3(),progress=true,save_everystep=false,save_start=false)
sol = solve(prob,ROCK2(),progress=true,save_everystep=false,save_start=false)
sol = solve(prob,TRBDF2(autodiff=false),progress=true,save_everystep=false,save_start=false)
@time sol = solve(prob,CVODE_BDF(linear_solver=:GMRES),progress=true,save_everystep=false)

println("CPU Times")
println("BS3")
@time sol = solve(prob,BS3(),progress=true,save_everystep=false)
println("ROCK2")
@time sol = solve(prob,ROCK2(),progress=true,save_everystep=false)
println("ROCK4")
@time sol = solve(prob,ROCK4(),progress=true,save_everystep=false)
println("TRBDF2")
@time sol = solve(prob,TRBDF2(autodiff=false),progress=true,save_everystep=false)
println("Rodas5")
@time sol = solve(prob,Rodas5(autodiff=false),progress=true,save_everystep=false)
println("CVODE_BDF")
@time sol = solve(prob,CVODE_BDF(linear_solver=:GMRES),progress=true,save_everystep=false)

using CuArrays
gu0 = CuArray(Float32.(u0))
const gMx = CuArray(Float32.(Mx))
const gMy = CuArray(Float32.(My))
const gα₁ = CuArray(Float32.(α₁))
const gMyA = CuArray(zeros(Float32,N,N))
const gAMx = CuArray(zeros(Float32,N,N))
const gDA = CuArray(zeros(Float32,N,N))

function gf(du,u,p,t)
   A = @view  u[:,:,1]
   B = @view  u[:,:,2]
   C = @view  u[:,:,3]
  dA = @view du[:,:,1]
  dB = @view du[:,:,2]
  dC = @view du[:,:,3]
  mul!(gMyA,gMy,A)
  mul!(gAMx,A,gMx)
  @. gDA = D*(gMyA + gAMx)
  @. dA = gDA + gα₁ - β₁*A - r₁*A*B + r₂*C
  @. dB = α₂ - β₂*B - r₁*A*B + r₂*C
  @. dC = α₃ - β₃*C + r₁*A*B - r₂*C
end

prob2 = ODEProblem(gf,gu0,(0.0,100.0))
CuArrays.allowscalar(false)
sol = solve(prob2,BS3(),save_everystep=false,save_start=false)
sol = solve(prob2,ROCK2(),save_everystep=false,save_start=false)
@test sol.t[end] == 100.0

println("GPU Times")
println("BS3")
@time sol = solve(prob2,BS3(),progress=true,save_everystep=false,save_start=false)
println("ROCK2")
@time sol = solve(prob2,ROCK2(),progress=true,save_everystep=false,save_start=false)

Interesting benchmark problems

Here are two we can add. Not everything needs to be a wp plot.

Not small chemical reaction network:
SciML/OrdinaryDiffEq.jl#664

Mixed neural ODE adjoint:
SciML/SciMLSensitivity.jl#46

Jump equations need to run

I'm not sure if they are fully updated or not. @isaacsas help? :)

Add more Gillespie libraries

See the discussion in #26

Non-diagonal and commutative SDE benchmarks

@onoderat has moved us forward finally in the Non-diagonal SDE department, and so we should think about creating some benchmark notebooks in this area.

Add heating system benchmark

https://gist.github.com/ranjanan/48cf5db52fc340eb504af8cee3972771#file-heatingsystemnew-jl

@ranjanan

Pleiades

Ode function has an error:

This code is wrong,
for i in 14:28
du[i] = zero(u[1])
end

It would be
for i in 15:28
du[i] = zero(u[1])
end

Regards,

PDE notebooks need to be rerun or re-linked

What I've noticed is that most (I didn't check all of them) PDE notebooks need to be rerun: all cells just display error messages.

E.g.:

http://benchmarks.juliadiffeq.org/html/MOLPDE/allen_cahn_spectral_wpd.html

http://benchmarks.juliadiffeq.org/html/MOLPDE/kdv_spectral_wpd.html

Pictures

Benchmark Tim Holy's global algorithms on parameter estimation problems

@timholy made

https://github.com/timholy/QuadDIRECT.jl

and

https://github.com/timholy/MultilevelCoordinateSearch.jl

We should give these a try in the Lorenz parameter estimation notebook.

@Vaibhavdixit02

Build log

!rebuild NonStiffODE

Add first difference, multiple shooting and MAP benchmarks.

Issue with DiffEqBase in LorenzParameter Estimation

Thanks for all the recent tremendous improvments in all directions.
Trying to run the LorenzParameterEstimation notebook, after Pkg.update(), I get:

At
g1 = @ode_def LorenzExample begin
dx = σ*(y-x)
dy = x*(ρ-z) - y
dz = xy - βz
end σ ρ β
p = [10.0,28.0,2.66]
r0 = [1.0; 0.0; 0.0]
tspan = (0.0, 30.0)
prob = ODEProblem(g1, r0, tspan,p)
tspan2 = (0.0, 3.0)
prob_short = ODEProblem(g1, r0, tspan2,p)

I get the following error:

MethodError: no method matching DiffEqBase.ODEProblem(::LorenzExample, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Array{Float64,1})
Closest candidates are:
DiffEqBase.ODEProblem(::Any, ::Any, ::Any; kwargs...) at C:\Users\Denis.julia\v0.6\DiffEqBase\src\problems/ode_problems.jl:26

Stacktrace:
[1] include_string(::String, ::String) at .\loading.jl:522

Weak SDE estimation is currently broken

Wheldon, Kirk, and Finlay

HI all,

some bugs from benchmark test (Wheldon, Kirk, and Finlay)

UndefVarError: prob_dde_wheldon not defined

syntax: invalid syntax "1./"; add space(s) to clarify

best

BioPreDyn-bench Benchmarks

This seems to be a very good benchmark set:

http://download.springer.com/static/pdf/264/art%253A10.1186%252Fs12918-015-0144-4.pdf?originUrl=http%3A%2F%2Fbmcsystbiol.biomedcentral.com%2Farticle%2F10.1186%2Fs12918-015-0144-4&token2=exp=1496951319~acl=%2Fstatic%2Fpdf%2F264%2Fart%25253A10.1186%25252Fs12918-015-0144-4.pdf*~hmac=2832248480a14616e75641200f8d2d60fbc3de9057f565c77ca7d4ecf873a664

https://sites.google.com/site/biopredynbenchmarks/home

We should benchmark our performance on these.

Fix DynamicalODE Benchmarks

Continuation of #229

@SebastianM-C

Benchmarks for stiff SDEs

Regressions from v1.0 upgrade

List of regressions found in #32:

• RKMil in Lotka-Volterra SDE has a divergence
• single_pendulums needs to precompile
• Jump ones need to be ran
• [ ]

Precompilation broken on julia 1.5 beta1

julia> using DiffEqBenchmarks
[ Info: Precompiling DiffEqBenchmarks [31c91b34-3c75-11e9-0341-95557aab0344]
ERROR: LoadError: InitError: Evaluation into the closed module `Markdown` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `Markdown` with `eval` during precompilation - don't do this.
Stacktrace:
 [1] eval at .\boot.jl:331 [inlined]
 [2] eval at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Markdown\src\Markdown.jl:6 [inlined]
 [3] __init__() at C:\Users\sebastian\.julia\packages\Weave\zT0iu\src\WeaveMarkdown\markdown.jl:10
 [4] _include_from_serialized(::String, ::Array{Any,1}) at .\loading.jl:697
 [5] _require_from_serialized(::String) at .\loading.jl:749
 [6] _require(::Base.PkgId) at .\loading.jl:1040
 [7] require(::Base.PkgId) at .\loading.jl:928
 [8] require(::Module, ::Symbol) at .\loading.jl:923
 [9] include(::Function, ::Module, ::String) at .\Base.jl:380
 [10] include(::Module, ::String) at .\Base.jl:368
 [11] top-level scope at none:2
 [12] eval at .\boot.jl:331 [inlined]
 [13] eval(::Expr) at .\client.jl:467
 [14] top-level scope at .\none:3
during initialization of module WeaveMarkdown
in expression starting at C:\Users\sebastian\.julia\dev\DiffEqBenchmarks\src\DiffEqBenchmarks.jl:3
ERROR: Failed to precompile DiffEqBenchmarks [31c91b34-3c75-11e9-0341-95557aab0344] to C:\Users\sebastian\.julia\compiled\v1.5\DiffEqBenchmarks\3YclJ_OeiSs.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1290
 [3] _require(::Base.PkgId) at .\loading.jl:1030
 [4] require(::Base.PkgId) at .\loading.jl:928
 [5] require(::Module, ::Symbol) at .\loading.jl:923

This is due to Weave evaling in Markdown during __init__. This is tracked here.

Register

@JuliaRegistrator register()

It turns out that registering would make handling the manifests in the subpackages much easier.

update BCR benchmark for Catalyst

Though this should wait till it has been verified we've fixed the MT issues that were giving enormous ODE generation times.

Re-run stiff ODE benchmarks

It's having troubles with the ODEInterface.jl binaries. @christopher-dG might need to help with getting those weird Fortran binaries to work.

DiffEqBayesLorenz and DiffEqBayesLotkaVolterra need to be re-ran

@Vaibhavdixit02 is it setup to run?

Convert benchmarks to use Weave.jl

A bunch of tumor models

https://ascpt.onlinelibrary.wiley.com/doi/epdf/10.1002/psp4.12450