How do I handle different errors when working with an API?

Question

I am working a project in Python were I am gathering rain data and in this case temperature data from the Netatmo weather stations (Its basically just a private weather station you can set up in you garden and it will collect rain data, temperature, wind etc.)

When using the patatmo API you need a user with credential, a client. This client then has 500 requests pr. Hour which can be used on different requests, among these are the client.GetPublicdata request and the client.Getmeassure request. The Getmeassure request requires a station id and a module id, which I get from the the Getpuclicdat request. If I run out of requests I will catch that error using the except ApiResponseError: and I will then change the client credentials since I am not alone in this project and have credentials from two other people. My issues are:

If a station or module ID is not found using the Getmeassure request it will return a different type of ApiResponseError, which in my current code also is caught in the previously mentioned except, and this results in an endless loop where the code just changes credential all the time.

The code looks like this:

from patatmo.api.errors import ApiResponseError
...
...
...
a_test1 = 0
        while a_test1 == 0:
            try:
                Outdoor_data = client.Getmeasure(
                        device_id = stations_id , 
                        module_id = modul_ID , 
                        type = typ ,
                        real_time = True ,
                        date_begin = Start  , 
                        date_end = End
                        )
                time.sleep(p)
                a_test1 = 1
            except ApiResponseError:
                credentials = cred_dict[next(ns)]
                client = patatmo.api.client.NetatmoClient()
                client.authentication.credentials = credentials
                client.authentication.tmpfile = 'temp_auth.json'
                print('Changeing credentials')
                credError = credError + 1
                if credError > 4:
                    time.sleep(600)
                    credError = 0
                pass
            except:
                print('Request Error')
                time.sleep(p)

The documentation for the error.py script, that was made by someone else, looks like this:

class ApiResponseError(BaseException):
    pass

class InvalidCredentialsError(ApiResponseError):
    pass

class InvalidApiInputError(BaseException):
    pass

class InvalidRegionError(InvalidApiInputError):
    def __init__(self):
        message = \
            ("'region' required keys: "
             "lat_ne [-85;85], lat_sw [-85;85], "
             "lon_ne [-180;180] and lon_sw [-180;180] "
             "with lat_ne > lat_sw and lon_ne > lon_sw")
        super().__init__(message)


class InvalidRequiredDataError(InvalidApiInputError):
    def __init__(self):
        message = "'required_data' must be None or in {}".format(
            GETPUBLICDATA_ALLOWED_REQUIRED_DATA)
        super().__init__(message)


class InvalidApiRequestInputError(BaseException):
    pass


class InvalidPayloadError(InvalidApiRequestInputError):
    pass


API_ERRORS = {
    "invalid_client": InvalidCredentialsError("wrong credentials"),
    "invalid_request": ApiResponseError("invalid request"),
    "invalid_grant": ApiResponseError("invalid grant"),
    "Device not found": ApiResponseError("Device not found - Check Device ID "
                                         "and permissions")
}

What I want to do is catch the error depending what type of error I get, and I just doesn't seem to have any luck doing so

Answer 1

Which of those exceptions subclasses do you get on request overrun? Use only that one to drive the auth swap. If it’s not a particular one but is shared with other bad events, you will need to examine the exception’s variables. Also you will need to figure out which exception to work around - bad station id might mean someone is offline so ignore and try later. Vs logic flaws in your program, abend on those, fix, retry.

Exception handling is on a first-catch, first-handled basis. Your most specific classes have to do the “except” first to match first, else a generic one would grab it and handle it. Watch your APIs exception hierarchy carefully!

maxexc = 1000
countexc = 1
while ...
  # slow your loop
  time.sleep(p)
  try:
    ... what you normally do


  # dont use the too generic ApiResponseError here yet
  except requestoverrunexception as e:
    ... swap credendentials
    countexc+=1 

  except regionexception as e:
    ... ignore this region for a while
    countexc += 1

  # whoops abend and fix
  except ApiResponseError as e:
    print vars(e)
    raise
  except Exception as e:
    print(e)
    raise

I would bail after too many exceptions, thats what countexc is about.

Also 500 requests an hour seems generous. Don’t try to fudge that unless you have a real need. Some providers may even have watchdogs and get rid of you if you abuse them.