Python fromfile returning incorrect values when reading binary file generated with tofile

Question

I found several similar questions but none of them described exactly the problem I'm facing. I've an OBJ file which contains lines describing vertex positions:

v 0.01214 0.4242 0.82874

I want to transform this data into a matlab-friendly format so that I can plot some data with these and other values obtained from other files. I'm using python (and numpy) to transform this data to a binary file so I use the following function:

def extract_verts(in_file, out_file):
    v = np.zeros(3, np.float32)
    for line in in_file:
        words = line.split()
        if words[0] == 'v':
           v[:] = [np.float32(s) for s in words[1:4]]
           print v[:]
           v.tofile(out_file)

The problem is that when I read it using matlab (fread does the job) the first values are correct but then incorrect values are read. after some incorrect values correct values are read again, but they seem to be shifted (when using a vector-3 structure the x-component appears as the y-component, for example). Later, it happens again and incorrect and then correct but shifted values are read, and so on. I've checked that the file is read correctly since I can see the read values from the print line.

I've tried to read the file with python, just in case it was a matlab issue:

data = np.fromfile('file.dat', np.float32)
i = 0
while i < 100:
     print data[i]
     i = i+1

and it happens exactly the same (even the incorrect values are the same).

I think it may have something to do with byte ordering or OS dependent issues, because this set of scripts I'm using works on MacOS (the scripts were created by a colleague), but I'm using Windows 7. Has anyone faced a similar problem in the past?

Thanks.

do you read/write using binary mode? for correct packing check the struct module — Fredrik Pihl
– Fredrik Pihl, Commented Jun 1, 2011 at 10:49

Sven Marnach · Accepted Answer · 2011-06-01 11:31:03Z

1

Most probably, the error is in the code you din't show. The out_file parameter to extract_verts() seems to be an open file object, and it is important that you open this file in binary mode:

out_file = open("out.dat", "wb")

Files are opened in text mode by default. On Windows, this means automatic newline mangling takes place: All occurences of the byte '\n' are replaced by '\r\n'. This is why all values up to the first occurrence of '\n' are correct.

answered Jun 1, 2011 at 11:31

Sven Marnach

607k123 gold badges966 silver badges865 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

emiguel Over a year ago

Thanks! It solved the problem. Anyway, I can't understand why this code worked on MacOS... Does it manage newlines in a different way?

Sven Marnach Over a year ago

@Eder: As stated in my post, this is only an issue on Windows with its daft '\r\n' line endings. Every other operating system on the planet is more sane in this regard.

Collectives™ on Stack Overflow

Python fromfile returning incorrect values when reading binary file generated with tofile

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related