DEPR: GH10623 remove items from msgpack.encode for blocks

[kawochen]

closes #10623

[kawochen]

hmm legacy file for 0.17.1 not passing. I think it's tz-related. Will check later.

[kawochen]

passing now. had to make changes for DatetimeTZBlock (it wasn't tested?) and skip files packed in P2 with 0.17 :( due to #12142

[jreback]

this is way hacky. you shouldn't need to specifically do this (test for DatetimeIndex. It should just be converted normally. The entire difference here is the dispatch on the dtype, which in this case would be a string.

[kawochen]

if it gets converted to a numpy array now, wouldn't the whole encode/decode for DatetimeIndex be skipped? Or is there a way to construct DatetimeTZBlock from numpy arrays?

[jreback]

actually I think you could trivially change core/internals.py/make_block to take a DatetimeTZDtype and correctly create it. Then you would have to make to serialize that properly (as a string is prob ok) I think (you would need to fix the reverse lookup as it only deals with numpy atm).

I think the same problem exists with categorical, prob not testing it.

[kawochen]

updated

[jreback]

you should do this first, then you can coerce if you have a dtype

[jreback]

fyi some of these fixes will prob address: #11455, maybe #11755

[kawochen]

updated... but I forgot to squash :( will do that later

[jreback]

I get that you have to do this, we prob need something like (in core/common.py)

def pandas_dtype(dtype):
     if isinstance(dtype, compat.string_types):
            try:
                return DatetimeTZDtype.construct_from_string(dtype)
            except TypeError:
                pass

            try:
               return CategoricalDtype.construct_from_string(dtype)
            except TypeError:
               pass

           dtype = np.dtype(dtype)

    return dtype

[jreback]

pls add a whatsnew as well (put in API changes).
make a note in io.rst that the writing format is changing in 0.18.0 so we won't have forward compat (IOW older version of pandas cannot read newer version)

[jreback]

this would close #12142 as well? (should add a warning in the docs about this) (e.g. show your matrix)

[kawochen]

I can change all the keys to unicode, and then it will. will update tonight

[kawochen]

Ah half way through but have to go. I will get it done tonight but is that too late? :(

[jreback]

sure no prob

[kawochen]

updated. manually tested with python 2.7 and 3.5 on linux (zlib and no compress).

[jreback]

just use u() from compat/__init__.py (and amend with this).

[kawochen]

updated. So this is how I tested P2 & P3 compatibility: use the script to generate legacy data for 0.18.0 in P2 and P3, put them in 0.18.0 folder, and let the tests pick them up.

[jreback]

thanks!

I added some more test msgpacks for 0.17.1 to cover the bases

[jreback]

https://travis-ci.org/pydata/pandas/jobs/109620001

I think we need to skip if blosc is not installed on these tests.

can you do a PR for this?