Moving the memory space detection outside the Engine classes for the GPU-aware logic about adios2

[eisenhauer]

There are several things that are currently done in at least partially duplicative code across all engines (MemorySelection comes to mind in addition to memory space detection). Each of these is kind of funky and it may be difficult to rationalize this everywhere. However, we might do a better job on testing. Part of the problem is that the tests in engines/bp aren’t structured so that they can be used with streaming engines. However, PR #3823 demonstrated an approach whereby an engines/bp tests could be restructured slightly in order to run it in testing/adios/engines/staging-common so that it can be run against staging engines too. Whatever we do with unifying memory space detection, we could also move towards running the tests we have against all the engines we have.

[anagainaru]

I like your solution to use MPI_Comm_split to move from BP to SST testing. I will try to update the device testing to include something similar.

I would still like to trigger the detection mechanism for memory space as early as possible. There are so many paths that engines take to move the data, I am starting to regret the decision to allow Puts/Gets on different memory space for the same variable. I am thinking of forcing just one (once you send device data to a variable, all data needs to be on the device) since I am having trouble transposing on the other side data if you don't read it on the same space.