importing csv data in python

Question

im am trying to import data into database from a csv downloaded from URL. The csv is 100MB large.

def get_csv_data():
   url = 'http://dati.ur.gov.lv/register/register.csv'
   response = urllib2.urlopen(url)

   cr = csv.reader(response, delimiter=';', quotechar='"')

   for row in cr:
       if row[0] != 'regcode':
          print row[2]
          ur = Ur()
          ur.regcode=row[0]
          ur.sepa=row[1]
          ur.name=row[2]
          ur.name_before_quotes=row[3]
          ur.name_in_quotes=row[4]
          ur.name_after_quotes=row[5]
          ur.without_quotes=row[6]
          ur.regtype=row[7]
          ur.regtype_text=row[8]
          ur.type=row[9]
          ur.type_text=row[10]
          ur.registered=row[11]
          ur.terminated=row[12]
          ur.closed=row[13]
          ur.address=row[14]
          ur.adressid=row[15]
          ur.region=row[16]
          ur.city=row[17]
          ur.atvk=row[18]
          ur.reregistration_term=row[19]
          ur.uri=row[20]
          ur.save()

I go through only two rows and then get a 500 error code. whats wrong and how should i do this?

Answer 1

A 500 HTTP response code is a problem with the server not on your end:

Response status codes beginning with the digit "5" indicate cases in which the server is aware that it has erred or is incapable of performing the request. Except when responding to a HEAD request, the server SHOULD include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. User agents SHOULD display any included entity to the user. These response codes are applicable to any request method.

FWIW, it may work out better for you to pull down the entires page with a page = response.read() and then loop over the lines after retrieval using cr = csv.reader(page.splitlines(), delimiter=';', quotechar='"').

Answer 2

500 is Internal Server Error, which means either the security settings are blocking the access or the database is corrupt and you are not able to read. Check the security settings and see if you are able to query the DB through website.