Python "with" statement syntax

Question

I have some python code that parse the csv file. Now our vendor decide to change the data file to gzip csv file. I was wondering what's the minimal/cleanest code change I have to make. Current function:

def load_data(fname, cols=()):
    ... ...
    with open(fname) as f:
        reader = csv.DictReader(f)
        ... ...

I don't want to duplicate the code to load_data2(), and change the with statement to, thought it works perfectly.

with gzip.open(fname) as f:

How can I factor out the with statement?

def load_data(fname, cols=()):
    ... ...
    if fname.endswith('.csv.gz'):
        with gzip.open(fname) as f:
    else:
        with open(fname) as f:

        reader = csv.DictReader(f)
        ... ... # code to parse

mdml · Accepted Answer · 2013-10-28 15:35:15Z

8

You can do this by assigning the function you want to use to open the file to a different variable, depending on the properties of the file name:

opener = gzip.open if fname.endswith('.csv.gz') else open
with opener(fname) as f:
    ... # code to parse

answered Oct 28, 2013 at 15:35

mdml

23k8 gold badges61 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

kindall Over a year ago

You can even do this directly in the with statement: with gzip.open(fname) if fname.endswith('.csv.gz') else open(fname) as f

fivelements Over a year ago

Thanks, python is beautiful!

Alexander Zhukov · Accepted Answer · 2013-10-28 16:30:41Z

6

The universal approach is to dynamically choose an opener:

openers = {
    'http': urllib2.urlopen,
    '.csv.gz': gzip.open
    '.csv': open
}

resource_type = get_resource_type(resource) # determine the type of the resource

with openers[resource_type](resource) as f:
    # do stuff ...

That way, you can painlessly add more openers, when needed. Here is another example of the factory method design pattern.

edited Oct 28, 2013 at 16:30

answered Oct 28, 2013 at 15:43

Alexander Zhukov

4,5871 gold badge22 silver badges32 bronze badges

Collectives™ on Stack Overflow

Python "with" statement syntax

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related