Python 2.7 CSV graph time format

Question

I have a CSV file, one of colons value is timestamps but when I use numpy.getfromtxt it change it to string. My goal is to create a graph but with normal time format I prefer seconds only.

this is my array that I get from bellow code:

array([('0:00:00',), ('0:00:00.001000',), ('0:00:00.002000',),
 ('0:00:00.081000',), ('0:00:00.095000',), ('0:00:00.195000',),
 ('0:00:00.294000',), ...

this is my code:

col1 = numpy.genfromtxt("mycsv.csv",usecols=(1),delimiter=',',dtype=None, names=True)

The problem that I am having that format is in string but I need it in seconds (us can be ignored or not). How can I achive that?

Answer 1

If you can, the best way for working with csv files in python is to use pandas. It takes care of this for you. I will assume the name of the time column is time, change it to whatever you use:

>>> import numpy as np
>>> import pandas as pd
>>> 
>>> df = pd.read_csv('test.csv', parse_dates=[1])  # read time as date
>>> print(df)
   test1                    time  test2  test3
0      5 2015-08-20 00:00:00.000     10   11.7
1      5 2015-08-20 00:00:00.001     11   11.6
2      5 2015-08-20 00:00:00.002     12   11.5
3      5 2015-08-20 00:00:00.081     13   11.4
4      5 2015-08-20 00:00:00.095     14   11.3
5      5 2015-08-20 00:00:00.195     15   11.2
6      5 2015-08-20 00:00:00.294     16   11.1
>>> df['time'] -= pd.datetime.now().date()  # convert to timedelta
>>> print(df)
   test1            time  test2  test3
0      5        00:00:00     10   11.7
1      5 00:00:00.001000     11   11.6
2      5 00:00:00.002000     12   11.5
3      5 00:00:00.081000     13   11.4
4      5 00:00:00.095000     14   11.3
5      5 00:00:00.195000     15   11.2
6      5 00:00:00.294000     16   11.1
>>> df['time'] /= np.timedelta64(1,'s')  # convert to seconds
>>> print(df)    
   test1   time  test2  test3
0      5  0.000     10   11.7
1      5  0.001     11   11.6
2      5  0.002     12   11.5
3      5  0.081     13   11.4
4      5  0.095     14   11.3
5      5  0.195     15   11.2
6      5  0.294     16   11.1

You can work with pandas dataframes (what you have here) and series (what you would from getting a single column, such as df['time']) in most of the same ways as numpy arrays, including plotting. However, if you really, really need to convert it to a numpy array, it is as easy as arr = df['time'].values.

Answer 2

use the datetime library

 import datetime

 for x in array:
     for y .... # it's not realy obvious what the nesting is here...
        timestamp = datetime.strptime(y, '%H:%M:%S.%f')

Answer 3

You can use a converter for the timestamp field.

For example, suppose times.dat contains:

time
0:00:00
0:00:00.001000
0:00:00.002000
0:00:00.081000
0:00:00.095000
0:00:00.195000
0:00:00.294000

Define a converter that converts a timestamp string into the number of seconds as a floating point value:

In [5]: def convert_timestamp(s):
   ...:     h, m, s = [float(w) for w in s.split(':')]
   ...:     return h*3600 + m*60 + s
   ...: 

Then use the converter in genfromtxt:

In [21]: genfromtxt('times.dat', skiprows=1, converters={0: convert_timestamp})
Out[21]: array([ 0.   ,  0.001,  0.002,  0.081,  0.095,  0.195,  0.294])