Concatenate Two Big Pandas.HDFStore HDF5 Files

This question is somehow related to 'Concatenate a large number of HDF5 files'. I have several huge HDF5 files (~20GB compressed), which could not fit the RAM. Each of them stores

Solution 1:

see docs here for the odo project (formerly into). Note if you use the into library, then the argument order has been switched (that was the motivation for changing the name, to avoid confusion!)

You can basically do:

from odo import odo
odo('hdfstore://path_store_1::table_name',
    'hdfstore://path_store_new_name::table_name')

doing multiple operations like this will append to the rhs store.

This will automatically do the chunk operations for you.