1,339 questions
0
votes
1
answer
67
views
Error when trying to access data in h5 file using h5py
I am trying to read an h5 file using python
with h5py.File(filename, 'r') as file:
print("file.keys() = ", file.keys())
a_group_key = list(file.keys())[0]
data = list(file[...
- 4,478
0
votes
1
answer
175
views
OSError: Can't synchronously read data (can't open directory) when trying to index retrieve values from a hdf5 file
I generated a hdf5 file with the code below:
# Create forecast training data file
import h5py
import hdf5plugin
#mmap_array = np.memmap('video_prediction_224.dat', dtype='uint8', mode='w+', shape=(...
- 53
0
votes
0
answers
142
views
How to speedup cupy code with frequent data transfer & writing
I have a computational code largely written based on cupy. The computation I'm running now requires high-frequency data transfer (from GPU to CPU) and writing to a h5 file.
Here is a sketch of what I'...
- 37
0
votes
1
answer
53
views
Dynamically update h5py VDS from data written to in parallel -- multiple writer multiple reader case
I am trying to simulate the creation of multiple h5 files in parallel while trying to read out the results live via a dynamically updated H5 VDS in swmr mode. All h5 files are processing different ...
- 113
0
votes
0
answers
71
views
Speed up read access of large (~300mb) samples with H5py
I have a large .h5 file of high resolution images (~300MB each, 200 images per .h5 file) and need to load samples in python. The current setup uses a separate dataset for each sample.
data_group....
- 179
-4
votes
1
answer
108
views
How to save simulation parameters in HDF5 file with h5py?
I'm using H5PY to store a high number of simulations' outputs. Now of course these simulations are parametrized, hence I need to also store what parameters were used for which simulation output.
At ...
- 27
0
votes
1
answer
89
views
How to extend type-hinting for a class in a stub file
I have this code, and it annoys me that I have to cast f twice:
with h5py.File(computed_properties_path, "r") as f:
# get the set of computed metrics
computed_metrics = ...
- 5,220
0
votes
1
answer
121
views
Writing complex Pandas DataFrame to HDF5 using h5py
I have a Pandas DataFrame with mixed scalar and array-like data of different raw types (int, float, str). The DataFrame's types look like this:
'col1', dtype('float64')
'col2', dtype('O') <-- array,...
- 1,193
0
votes
2
answers
147
views
Replace table in HDF5 file with a modified table
I have an existing HDF5 file with multiple tables. I want to modify this HDF5 file: in one of the tables I want to drop some rows entirely, and modify values in the remaining rows.
I tried the ...
- 8,674
1
vote
0
answers
36
views
Python multiprocessing: Exponential slowdown when processing batches of HDF5 files
I'm trying to load and process large amounts of HDF5 files using Python and convert them into dataframes. The HDF5 files are scattered in equal-sized batches. I've tried two approaches using ...
- 11
0
votes
0
answers
89
views
Does h5py+MPI ("parallel HDF5") offer simultaneous writes, or just parallelized sequential writes?
I implemented a data ingestion tool in Python using h5py+MPI to ingest data from a large number of plaintest files into an equal number of HDF5 datasets. However, I'm finding that using multiple ...
- 403
0
votes
0
answers
179
views
Unable to import h5py in anaconda jupyter on windows 10
I want to save tensorflow model using Jupyter on Windows 10. I get import error that h5py must be installed to save the model. I installed the latest version of h5py. I have tried saving the model as ...
- 1
-1
votes
1
answer
825
views
What steps should I take to address the issue of installing h5py using pip?
I am facing an issue while trying to install packages with pip on Debian 12 and I'm unsure how to address it. As an illustration, here is what occurs when I attempt to install h5py and other packages:
...
- 3
0
votes
1
answer
241
views
how to set a proper chunk size in hdf5
according to this answer, a proper chunk size is important for optimizing I/O perfromance.
I am 3000 jpg images, whose size vary from 180kB to 220kB. I am going to save them as bytes.
I know 2 methods ...
- 1,462
0
votes
1
answer
111
views
Error building Docker image for AWS Lambda function: Unable to install h5py due to missing HDF5 library
I'm trying to build a Docker image for an AWS Lambda function that requires the h5py library. However, I'm encountering an error related to missing HDF5 dependencies during the build process.
=> ...
