(written by Alfredo Correa)
|
Feb 15, 2018 The latest version of HDFView is 11.5 on Mac Informer. It is a perfect match for Viewers & Editors in the Design & Photo category. The app is developed by The HDF Group and its user rating is 1 out of 5. HDF5 1.8.10, and contains information on the platforms tested and known problems in HDF5-1.8.10. For more details, see the files HISTORY-10-180rc3.txt and HISTORY-18.txt in the releasedocs/ directory of the HDF5 source. Multi-threaded HDF5 filter implementing the LPC-Rice codec. This lossless codec is intended to compress correlated signal data, such as image data, stored in HDF5 chunks. The codec/filter is optimized for both compression and decompression speed, but compression ratios are still comparable to JPEG-LS and CCSDS-123 codecs. Version 7.3 of MAT-files uses HDF5 format, this format has a significant storage overhead to describe the contents of the file, especially so for complex nested cellarrays and structures. Its main advantage over previous versions of MAT-files is that it allows storing data larger than 2GB on 64-bit systems.
HDF5 is a library for storing large numerical data sets. HDF stands for Hierarchical Data Format. It was designed for saving and retrieving data to/from structured large files. It also supports parallel access to files in HDF5 format, in particular within the MPI environment. HDF5 library can be used from C, C++ (with some limitations) and Fortran. Files can be saved optionally in text format, binary and compressed format (if zlib is available).
HDF5 is recommended by the developers of FFTW as a means to output data from different MPI processes. This format can be read by Mathematica and Matlab.
Using HDF5 (and HDF) in Matlab and Mathematica
Some popular numerical packages, like MATLAB, Mathematica, Octave and ROOT, already have native support for the HDF5 format. If you want to quickly experiment with HDF5 files, you can use those programs to see how it works.
For example, in Mathematica:
will create a binary file called 'matrix.h5' with the matrix data.
Matlab can read and write HDF5 file with the hdf5read and hdf5write command (see note about Matlab 7.0 below). Octave can do the same with 'load' and 'save' commands.
Warning: There is a consistent 'bug' in Matlab and Mathematica, when you try to export (with hdf5write or Export) only the real part of the array entries are saved to the file and that happens without any warning.
Partial workaround: Mathematica does not seem to support exporting HDF5 with complex (i.e. not real) data. This seems to be more a limitation of the data types supported by HDF5 combined with the fact that Matlab and Mathematica ignore this limitation. You can still export complex data by taking real and imaginary parts as a workaround:
Since the data contained in the result is indeed numerical and homogeneous you can take advantage of Mathamatica optimizations such as converting the result ToPackedArray.
Something similar can be done in Matlab.
Matlab 7.0 vs. HDF5 1.8
There are two major versions of the library, namely HDF (also known as HDF4) and HDF5, which are totally incompatible, and minor versions of each which are compatible among them in theory.
In practice Matlab 7's HDF5 interface seems to be unable to read HDF5 files created with (at least) version 1.8 of the HDF5 library. This problem does not exists with HDF5 1.6 (also marked as stable). In general Matlab 7 will produce the following message if trying to read a file that is writen by the newer versions of the HDF5 library:
Hdf5 For Mac Download
A possible workaround is to convert the HDF5 file to HDF4 file and then read it with Matlab 7 with the HDF interface. The command line conversion utility can be downloaded (binary or source) from this link. Once installed (or just copied to the PATH) it can used to convert the file,
Later, from Matlab 7, the file can be read without problems:
Note the difference in syntax ('hdf5read' vs 'hdfread', and '/array' vs 'array' for the name of the dataset).'h5toh4' can convert most files with simple structure.
Matlab 7.6 and Mathematica 7 (at least) do not seem to have these issues. So, an alternative is to upgrade Matlab 7 or use the more stable HDF5 1.6.
The following are instructions to install HDF5 in different systems. Note that, as of HDF5 version 1.9 there is no way to use the MPI version and the C++ interfaces together. The C interface can be used from C++ anyway.
Ubuntu
HDF5 1.6.6 can be installed directly in Ubuntu 8.10 by doing:
This will install the C, C++ and Fortran versions of the library and development (header) files, but it will not include the MPI version.
The MPI version with C and Fortran interfaces, (which will remove the previous serial version) can be installed by:
where 'mpich' can be replaced by 'openmpi' or 'lam'. Unfortunately the C++ interface is not provided for this MPI version -- the two features can not coexist, either you have the C++ interface or you have the MPI support, but not both.
Build and Installation from Sources
We will try to install the parallel version of HDF5 1.9 in our user space. (HDF5 1.8 --official release-- does not play well when compiling with gcc4.)
from a download location
Other versions (useful for compatibility):
- see here for latest version
- download ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/hdf5/snapshots/v18/hdf5-1.8.4-snap9.tar.gz for the last stable version, or use
- download ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/hdf5/snapshots/v16/hdf5-1.6.10-snap6.tar.gz for the legacy version (compatible with Matlab 7, but it has a different interface [function calls are different]). Or use
SVN
Alternatively the SVN version can be downloaded:
Build
then we can configure:
On su-ahpcrc we use the following configure command:
Other options are described in ./configure --help. The option --enable-cxx can be specified but not together with --enable-parallel. (For non-parallel version adding 'CC=mpicc' (or equivalent) is not necessary). Check that your MPI compiler is present by doing:
Then we can make and install
The compilation takes ~5 minutes and several warning messages will appear.Many header files will be installed in ~/usr/include and ~/usr/lib
The most important for us are hdf5.h and libhdf5.a.There are also some command line utilities to manage HDF5 files installed:
Among them, there is 'h5dump', which will be used in the next section.
To remove the installed library:
The source files contain examples on the usage of HDF5, including C++ examples. (See directories ./examples, ./hl/examples, ./c++/examples. and ./hl/c++/examples)
Simple introductory examples are also provided online, but they are outdated and are incompatible with this version of HDF5 1.8 (this issue can be very confusing). It is better to use the examples contained in the distribution file and use the online documentation to read the details of the examples. In any case here I provide the sources and makefile I used for testing the installation h5_test.tar.gz. Use the example as follows:
Internally, in the Makefile, the compilation is performed by the command line:
Depending on the code, in some cases we have to add '-lz -lrt'.
A write and read program will be compiled. The write program will create a binary file named SDS.h5 with the data of a certain array, then this array will be loaded from the file by the read program and printed.
The SDS.h5 is in a compressed binary format, which means that it can not be read directly. However there exists a bunch of HDF5 utilities (external programs) that allows humans to see what is contained in the files:
For the moment this document is not a tutorial on HDF5 itself but it is only to document on its installation. However, we can already mention something about the structure of the file: For example, in the previous h5dump you can read 'GROUP '/', this indicates that the dataset is at root level ('/') of the file. The HDF5 file can look pretty much like a filesystem, with directories, subdirectories and files/datasets.
Hdf5 For Mac Installer
MPI tests
I collected this three examples that can test the parallel capabilities of the library. These are based in the parallel HDF5 tutorial and official example file, but since those are outdated and do not compile out of the box, I corrected modified them and posted them here.
Note that last quoted code is not the real content of the file but just a human readable translation, that was accessed by means of one of the tools installed in ~/usr/bin. There are even graphical programs to visualize the contents of HDF5 file. These programs that read HDF5 have the advantage that they do not load the data into memory. You can inspect the contents of a 40GB file without hanging the system.
The official viewer is HDFView. Besides showing the table it has a limited capability to plot (parts) of the data. It is installed as follow
An alternative program is ViTables.
Note: the installation doesn't work in Ubuntu 9.04 (shipped with python 2.6) at the moment [1], as a temporary workaround install python2.5 and use the following line instead:
After the installation the program runs normally (command line is vitables). The program can show information of multidimensional arrays (more than 2 dimensions) by opening sub-windows.
CMake links HDF5 into your C, C++, or Fortran program with just two lines in CMakeLists.txt.If experiencing trouble finding HDF5 with CMake, try ourFindHDF5.cmake,which is more up to date than the FindHDF5.cmake included with CMake.An example CMake for writing network data to HDF5 in C:CMakeLists.txt.
A simple HDF5 read/write example is given below.
HDF5 prereqs
- Linux:
apt install libhdf5-dev - Mac:
brew install hdf5 - Windows MSYS2:
pacman -S mingw64/mingw-w64-x86_64-hdf5
Windows Subsystem for Linux
and be sure /etc/wsl.conf includes:
CMakeLists.txt HDF5
Here’s two examples, one for C, one for Fortran.“HL” refers to the high-level HDF5 interface that is more convenient and thus commonly used.
Note: if terminal has the Conda environment loaded and you keep getting the Conda HDF5 library, do first:
before running the CMake configure commadn.
C HDF5 CMakeLists.txt
Fortran HDF5 CMakeLists.txt
HDF5 C example
The HDF5 syntax is quite similar (and simpler) for Fortran.See the hdf5 directory ofFortran 2018 examplesfor more.
HDF5 compiler macros
Hdf5 For Mac Installer
As an alternative (or companion to) CMake, one can use compiler macros for HDF5.HDF group provides compiler macroh5cclinking the needed HDF5 libraries upon installing libhdf5-dev:
Hdf5 For Mac Pro
h5cc: Ch5c++: C++h5fc: Fortran