ornladios / adios2 Goto Github PK

BPWriteReadJson test would fail from time to time with a Json parser error. For now it's disabled to make dashboard green.
CDash reference link.

m_Name is changed over time

From https://github.com/LLNL/spack/pull/4944/files#r130646113:

It would be awesome for packaging if the currently shipped embedded dependencies can optionally be ignored and taken from external versions.

At rank 10 and 11, when running mpirun -n 12 ./heatTransfer_write_adios2 ....

@chuckatkins I got this error from the pull request checking. It compiles perfectly on Mac and Linux but apparently on Windows it cannot find the JSON header file.

Affected: PR #270

C:\projects\adios2\source\adios2/toolkit/transportman/dataman/DataMan.h(15): fatal error C1083: Cannot open include file: 'json.hpp': No such file or directory [C:\projects\adios2build\build_visual-studio\source\adios2\adios2.vcxproj]

Minimum version required is 0.5.1, but FindZFP doesn't currently handle version checking

Three of the test in testing/adios2/engine/hdf5/TestHDF5WriteRead.cpp write data using the HDF5Writer engine and validate it using HDF5 natively:

• ADIOS2HDF5WriteHDF5Read1D8
• ADIOS2HDF5WriteHDF5Read2D2x4
• ADIOS2HDF5WriteHDF5Read2D4x2

The native HDF5 read validation still needs to be implemented. See testing/adios2/engine/adios1/TestADIOS1WriteRead.cpp for an example of how the ADIOS1Writer engine is used with ADIOS1 natively reading back the data to validate. Once implemented, you should be able to run the tests via ./bin/TestHDF5WriteRead and have it pass.

There seems to be some problem finding all the pieces needed to enable python when configuring on windows:

C:\Users\kitware\projects\adios2\build>"C:\Program Files\CMake\bin\cmake.exe" ../source -DADIOS2_USE_Python:Bool=ON
-- Building for: Visual Studio 14 2015
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find BZip2 (missing:  BZIP2_LIBRARIES BZIP2_INCLUDE_DIR)
-- Could NOT find ZFP (missing:  ZFP_LIBRARY ZFP_INCLUDE_DIR)
-- Could NOT find MPI_C (missing:  MPI_C_LIBRARIES MPI_C_INCLUDE_PATH)
-- Could NOT find MPI_CXX (missing:  MPI_CXX_LIBRARIES MPI_CXX_INCLUDE_PATH)
-- Could NOT find ZeroMQ (missing:  ZeroMQ_LIBRARY ZeroMQ_INCLUDE_DIR)
-- Could NOT find HDF5 (missing:  HDF5_LIBRARIES HDF5_INCLUDE_DIRS C)
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Could NOT find ADIOS1 (missing:  ADIOS1_LIBRARY ADIOS1_INCLUDE_DIR) (Required is at least version "1.12.0")
-- Found PythonInterp: C:/Python27/python.exe (found version "2.7.13")
-- Found PythonModule_numpy: C:/Python27/lib/site-packages/numpy
CMake Error at C:/Program Files/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonFull (missing: PYTHON_LIBRARIES Libs)
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:377 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindPythonFull.cmake:53 (find_package_handle_standard_args)
  cmake/DetectOptions.cmake:117 (find_package)
  CMakeLists.txt:95 (include)


-- Configuring incomplete, errors occurred!
See also "C:/Users/kitware/projects/adios2/build/CMakeFiles/CMakeOutput.log".
See also "C:/Users/kitware/projects/adios2/build/CMakeFiles/CMakeError.log".

GNU is permissive. Solution: eliminate the copy constructor explicitly.

Pybind11 only supports Python 2.7 and above, should we handle this nicely on CMake (message e.g. Python 2.6 is not supported, required 2.7 and above).

The pybind11::dtype object allows access to low-level type information via the kind() method, which maps to the numpy.dtype.kind python property. This is only a coarse level of type information and is missing sizing. For instance, pybind11::dtype::of<float>().kind() is equal to pybind11::dtype::of<double>::kind() because they are both floating point types. The numpy.dtype class exposes more detailed information via the type property but that's not exposed in pybind11, thus leaving only the coarser level type-information.

Without this limit it fails on Titan for POSIX write

../../../lib64/libadios2sys.a(SystemTools.cxx.o): In function adios2sys::SystemTools::ConvertToUnixSlashes(std::string&)': SystemTools.cxx:(.text+0x1cbe): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking ../../../lib64/libgtest.a(gtest-all.cc.o): In function testing::internal::StreamingListener::SocketWriter::MakeConnection()':
gtest-all.cc:(.text+0xd718): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Remove cdash warnings for unused variables, type cast (including HDF5 char vs. int8) and virtual functions overloads.

Numpy is a mandatory dependency as it provides sizeof(T) and contiguous memory data (1D numpy arrays) needed by adios2 operations.

Due to going back to bp1 format: name.bp.dir/name.bp.rank

-- Found PythonModule_mpi4py: /usr/lib/python2.7/dist-packages/mpi4py
CMake Error at bindings/python/CMakeLists.txt:37 (install):
install TARGETS given no LIBRARY DESTINATION for module target "adios2py".

Not sure if related to issue #193

Found PythonInterp: /usr/bin/python (found version "2.7.11")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so
-- Found PythonModule_mpi4py: /usr/lib/python2.7/dist-packages/mpi4py

Installation prefix: /usr/local
Features:
Library Type: shared
Build Type: Debug
Testing: ON
Build Options:
BZip2 : ON
ZFP : ON
MPI : ON
DataMan : ON
ZeroMQ : ON
HDF5 : ON
ADIOS1 : ON
Python : ON
SysVShMem: ON

$ ls ./bin
globalArray_write heatTransfer_write_a2h5 heatTransfer_write_hdf5 hello_bpFlushWriter hello_datamanReader joinedArray_write reader_stepping TestXMLConfig
heatTransfer_read_adios1 heatTransfer_write_adios1 heatTransfer_write_ph5 hello_bpTimeWriter hello_datamanWriter localArray_write TestADIOSInterfaceWrite values_write
heatTransfer_read_adios2 heatTransfer_write_adios2 hello_adios1Writer hello_bpWriter hello_hdf5Writer reader_allsteps TestHDF5WriteRead writer_multiste

Python adios2.so not created under build/lib/python/... , but under build/Library/python/... on Mac. This was reproduced on @JasonRuonanWang Mac as well. Make install stops when reaching the Python bindings section.

Final BP file is ill-formed.

Possibly by generating a pkg-config file?

Using Windows 10 and Visual Studio 17. Errors related to google-test and libcpmtd.lib. Compilation works with -DADIOS2_BUILD_TESTING=OFF

See https://open.cdash.org/viewBuildError.php?buildid=4922792

.../Code/adios2/source/Builds/Intel-17.0.3.191_MPICH/ADIOS2/source/adios2/engine/hdf5/HDF5Common.cpp(73): error: namespace "std" has no member "runtime_error"
                  throw std::runtime_error("HDF5: Unable to create group " + ts0);
                             ^

Can't be overloaded with std:vectorstd::string and std::map<std::string, std::string> when using an initializer list

build_gnu> cmake -DCMAKE_BUILD_TYPE=Release -DADIOS2_USE_BZip2=OFF -DADIOS1_DIR=/sw/xk6/adios/1.12.0/cle5.2_gnu4.9.3 ../ADIOS2
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Cray Programming Environment 2.5.9 C
-- Check for working C compiler: /opt/cray/craype/2.5.9/bin/cc
-- Check for working C compiler: /opt/cray/craype/2.5.9/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.5.9 CXX
-- Check for working CXX compiler: /opt/cray/craype/2.5.9/bin/CC
-- Check for working CXX compiler: /opt/cray/craype/2.5.9/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find ZFP (missing: ZFP_LIBRARY ZFP_INCLUDE_DIR)
-- Found MPI_C: /opt/cray/craype/2.5.9/bin/cc
-- Found MPI_CXX: /opt/cray/craype/2.5.9/bin/CC
-- Could NOT find ZeroMQ (missing: ZeroMQ_LIBRARY ZeroMQ_INCLUDE_DIR)
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C)
[Sun Jun 25 07:48:40 2017] [unknown] Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(525):
MPID_Init(228).......: channel initialization failed
MPID_Init(611).......: PMI2 init failed: 1
/sw/xk6/adios/1.12.0/cle5.2_gnu4.9.3/bin/adios_config: line 242: 29050 Aborted ${MYDIR}/adios_list_methods
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Could NOT find ADIOS1 (missing: ADIOS1_LIBRARY ADIOS1_INCLUDE_DIR) (found suitable version "1.12.0", minimum required is "1.12.0")

Found or not found?

A generated ADIOS2Config.cmake needs to be created for easy import by other cmake based projects

Moving XML examples to examples/experimental

Add continuous Integration for Windows

Need to implement with CMake FortranCInterface module

Auto mode tries to compile with BZip2 even though it's not found.
/usr/bin/ld: cannot find -lBZip2::BZip2
../../../lib64/libadios2sys.a(SystemTools.cxx.o): In function `adios2sys::SystemTools::ConvertToUnixSlashes(std::string&)':
SystemTools.cxx:(.text+0x1cbe): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/sha1sum: ../../../bin/heatTransfer_write_adios2: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [bin/heatTransfer_write_adios2] Error 1
make[1]: *** [examples/heatTransfer/write/CMakeFiles/heatTransfer_write_adios2.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

The current master tries to compile tests that depend on ADIOS1 even though the dependency was not found by cmake

The ADIOS1Writer tests all output the following warning message:

WARN : Unable to write /__adios__/timer_labels_1, continuing

long int not compatible with int64_t , it's long long int on Mac

It currently throws an exception in debugMode if file is not found. Just let it run and take settings from compiled source code in the heat transfer example.

The size, shape, and count parameters are being used incorrectly. See 1d14e4d, which fixes the 1D array for the ADIOS1 test. This needs to be applied similarly to the 2D tests and the other engines.

source/adios2/CMakeFiles/adios2.dir/toolkit/profiling/iochrono/Timer.cpp.o
/mnt/c/Users/ywy/Software/ADIOS2/source/adios2/toolkit/format/bp1/BP1Aggregator.cpp: In member function ‘std::string adios2::format::BP1Aggregator::GetGlobalProfilingLog(const string&)’:
/mnt/c/Users/ywy/Software/ADIOS2/source/adios2/toolkit/format/bp1/BP1Aggregator.cpp:120:74: error: invalid conversion from ‘const void*’ to ‘void*’ [-fpermissive]
MPI_Isend(&rankLogSize, 1, MPI_INT, 0, 0, m_MPIComm, &requestSize);

Start 24: PythonBPWrite
24/25 Test #24: PythonBPWrite ....................................***Failed 0.12 sec

/path/to/file doesn't produce /path/to/file.bp.dir/file.bp.0