Reading Binary Data Using Python

Question

I am trying to read files containing gridded data that are formatted as flat binary 16-bit signed integer big-endian. I am using struct.unpack(), which I believe is correct as it allows me to specify the data is both signed and big-endian, but I am not sure of it recognizes the data as 16-bit. If possible, can someone confirm this is the correct method for reading the type of data I have described.

>>>file_name = 'some_file.dat'
>>>file = open(file_name, 'rb')
>>>data = struct.unpact('>h', file.read())
>>>print(data)
(-9999,)

I would like to read the entire file at once, and insert the data into a numpy array. I know the dimensions of the array and the direction used to populate the array from these files.

Thank you for any and all assistance.

while and with do completely different things. with isn't even a looping construct. Trying to compare the "performance differences" between these things is like asking if carrots taste like blue. — larsks
– larsks, Commented Mar 12, 2015 at 1:38
How many values are in a "real" file? Does the file contain only 16 bit integers (i.e. no header, no other data)? — Warren Weckesser
– Warren Weckesser, Commented Mar 12, 2015 at 3:14
@WarrenWeckesser The file contains only 16bit integers, the header is provided in a separate file. — jrwebster
– jrwebster, Commented Mar 12, 2015 at 12:20

Warren Weckesser · Accepted Answer · 2015-03-12 13:42:46Z

1

You have the numpy tag on the question, so I assume a numpy solution is acceptable. You can read the data using numpy's fromfile function. fromfile allows you to specify the data type, including endianness. For example,

In [1]: !hexdump x16.dat
0000000 00 01 01 01 ff ff 04 00 04 01 ff e8 00 00 00 f0
0000010

In [2]: x = np.fromfile('x16.dat', dtype='>i2')

In [3]: x
Out[3]: array([   1,  257,   -1, 1024, 1025,  -24,    0,  240], dtype=int16)

answered Mar 12, 2015 at 13:42

Warren Weckesser

116k20 gold badges207 silver badges224 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Matt Cooper · Accepted Answer · 2015-03-12 02:36:57Z

1

16 bit signed integers are h; b is 8-bit. So you want struct.unpack('>h', file.read()).

answered Mar 12, 2015 at 2:36

Matt Cooper

9379 silver badges16 bronze badges

Collectives™ on Stack Overflow

Reading Binary Data Using Python

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related