Convert a string of ndarray to ndarray

Question

I have a string of ndarray. I want to convert it back to ndarray. I tried newval = np.fromstring(val, dtype=float). But it gives ValueError: string size must be a multiple of element size

Also I tried newval = ast.literal_eval(val). This gives

File "<unknown>", line 1
[-1.45181984e-01  1.51671678e-01  1.59053639e-01 -1.02861412e-01
                               ^
SyntaxError: invalid syntax

String of ndarray

 '[-1.45181984e-01  1.51671678e-01  1.59053639e-01 -1.02861412e-01
 -9.70948339e-02 -1.75551832e-01 -7.24434480e-02  1.19182713e-01
 -4.54084426e-02 -9.23779532e-02  8.87222588e-02  1.05331177e-02
 -1.31792471e-01  3.50326337e-02 -6.58577830e-02  1.02670217e+00
 -5.29987812e-02  2.09167395e-02 -1.19845152e-01  2.30511073e-02
  2.89404951e-02  4.17387672e-02 -2.08203331e-01  2.34342851e-02]'

How can I convert this back to ndarray?

Honestly, by not having something like that in the first place. (You're already losing information since numbers are being rounded there.) Please serialize your arrays into a machine-readable format instead of trying to go through a string meant for human consumption. — AKX
– AKX, Commented Mar 18, 2019 at 11:42
If you're saving to disk to solely use the data with NumPy, use docs.scipy.org/doc/numpy/reference/generated/numpy.save.html and numpy.load. — AKX
– AKX, Commented Mar 18, 2019 at 11:45
I am storing these arrays in LevelDB as key value pairs. The arrays are fasttext vectors. In levelDB vector (value) for each ngram (key) are stored. Is what you mentioned above applicable here? — Kabilesh
– Kabilesh, Commented Mar 18, 2019 at 11:50
Yes. You'd use something like bio = io.BytesIO(); np.save(bio, my_array); ldb.put('my-key', bio.getvalue()) to save and bio = io.BytesIO(ldb.get('my-key')); arr = np.load(bio); to load. — AKX
– AKX, Commented Mar 18, 2019 at 11:59

AKX · Accepted Answer · 2019-03-18 12:03:03Z

To expand upon my comment:

If you're trying to parse a human-readable string representation of a NumPy array you've acquired from somewhere, you're already doing something you shouldn't.

Instead use numpy.save() and numpy.load() to persist NumPy arrays in an efficient binary format.

Maybe use .savetxt() if you need human readability at the expense of precision and processing speed... but never consider str(arr) to be something you can ever parse again.

However, to answer your question, if you're absolutely desperate and don't have a way to get the array into a better format...

>>> data = '''
... [-1.45181984e-01  1.51671678e-01  1.59053639e-01 -1.02861412e-01
...  -9.70948339e-02 -1.75551832e-01 -7.24434480e-02  1.19182713e-01
...  -4.54084426e-02 -9.23779532e-02  8.87222588e-02  1.05331177e-02
...  -1.31792471e-01  3.50326337e-02 -6.58577830e-02  1.02670217e+00
...  -5.29987812e-02  2.09167395e-02 -1.19845152e-01  2.30511073e-02
...   2.89404951e-02  4.17387672e-02 -2.08203331e-01  2.34342851e-02]
... '''.strip()
>>> list_of_floats = [float(x) for x in data.strip('[]').split(None)]
[-0.145181984, 0.151671678, 0.159053639, -0.102861412, -0.0970948339, -0.175551832, -0.072443448, 0.119182713, -0.0454084426, -0.0923779532, 0.0887222588, 0.0105331177, -0.131792471, 0.0350326337, -0.065857783, 1.02670217, -0.0529987812, 0.0209167395, -0.119845152, 0.0230511073, 0.0289404951, 0.0417387672, -0.208203331, 0.0234342851]

EDIT: For the case OP mentioned in the comments,

I am storing these arrays in LevelDB as key value pairs. The arrays are fasttext vectors. In levelDB vector (value) for each ngram (key) are stored. Is what you mentioned above applicable here?

Yes – you'd use BytesIO from the io module to emulate an in-memory "file" NumPy can write into, then put that buffer into LevelDB, and reverse the process (read from LevelDB into an empty BytesIO and pass it to NumPy) to read:

bio = io.BytesIO()
np.save(bio, my_array)
ldb.put('my-key', bio.getvalue())
# ...
bio = io.BytesIO(ldb.get('my-key'))
my_array = np.load(bio)

Collectives™ on Stack Overflow

Convert a string of ndarray to ndarray

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related