Calling HTTP API from script generates 400, calling from browser generates 200

Question

When I attempt to exercise this HTTP API endpoint programmatically from a python script, the server usually (but not always) sends a 400.
Such responses have empty bodies -- so I don't know why the server doesn't like my requests.
When I take a given url that generated a 400, and:
- Paste it into Chrome, I get a 200!
- Fire up python REPL and feed it to requests.get, I get a 200! I can do this repeatedly from "for loop" and always get 200's!
- Open up Chrome debug tools and request it using fetch method, I get a 200!

Check out this gist for details: https://gist.github.com/stockninja/7b9bcbfc8f338da414ae9678ec98016d

The offending script is called main.py, and I included output of some attempts I made to understand what's failing.

I'm completely at a loss as to what's failing!

Does your script use the same proxy as Chrome or the browser? Proxies tend to cause issues. Especially if the proxy needs one to login first before using it (Common in corporate networks) — Ganesh R.
– Ganesh R., Commented Jun 25, 2016 at 20:58
@GaneshR.: I'm not intentionally using a proxy for either the script or the browser. I'm using a mac -- and everything is unchecked in "System Preferences > Network > Advanced > Proxies". Don't know where else I should look for unintended proxying. The weird thing is that I get 200s calling the API from a Python REPL ... — justinmoon
– justinmoon, Commented Jun 25, 2016 at 21:05

Padraic Cunningham · Accepted Answer · 2016-06-25 21:46:10Z

Very simple solution,nothing to do with proxies, you need to add a user-agent:

def sync():
    head = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"}
    all_urls = urls()
    for url in all_urls:
        res = requests.get(url, headers=head)
        print(url)
        print(res.json())

Once you do:

In [2]: sync()
http://stats.nba.com/stats/teamgamelog?TeamID=1610612737&Season=2016-15&SeasonType=Regular+Season
{'resource': 'teamgamelog', 'parameters': {'TeamID': 1610612737, 'Season': '2016-15', 'LeagueID': None, 'SeasonType': 'Regular Season'}, 'resultSets': [{'name': 'TeamGameLog', 'rowSet': [], 'headers': ['Team_ID', 'Game_ID', 'GAME_DATE', 'MATCHUP', 'WL', 'MIN', 'FGM', 'FGA', 'FG_PCT', 'FG3M', 'FG3A', 'FG3_PCT', 'FTM', 'FTA', 'FT_PCT', 'OREB', 'DREB', 'REB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS']}]}
http://stats.nba.com/stats/teamgamelog?TeamID=1610612738&Season=2016-15&SeasonType=Regular+Season
{'resource': 'teamgamelog', 'parameters': {'TeamID': 1610612738, 'Season': '2016-15', 'LeagueID': None, 'SeasonType': 'Regular Season'}, 'resultSets': [{'name': 'TeamGameLog', 'rowSet': [], 'headers': ['Team_ID', 'Game_ID', 'GAME_DATE', 'MATCHUP', 'WL', 'MIN', 'FGM', 'FGA', 'FG_PCT', 'FG3M', 'FG3A', 'FG3_PCT', 'FTM', 'FTA', 'FT_PCT', 'OREB', 'DREB', 'REB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS']}]}
http://stats.nba.com/stats/teamgamelog?TeamID=1610612751&Season=2016-15&SeasonType=Regular+Season
{'resource': 'teamgamelog', 'parameters': {'TeamID': 1610612751, 'Season': '2016-15', 'LeagueID': None, 'SeasonType': 'Regular Season'}, 'resultSets': [{'name': 'TeamGameLog', 'rowSet': [], 'headers': ['Team_ID', 'Game_ID', 'GAME_DATE', 'MATCHUP', 'WL', 'MIN', 'FGM', 'FGA', 'FG_PCT', 'FG3M', 'FG3A', 'FG3_PCT', 'FTM', 'FTA', 'FT_PCT', 'OREB', 'DREB', 'REB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS']}]}
http://stats.nba.com/stats/teamgamelog?TeamID=1610612766&Season=2016-15&SeasonType=Regular+Season
{'resource': 'teamgamelog', 'parameters': {'TeamID': 1610612766, 'Season': '2016-15', 'LeagueID': None, 'SeasonType': 'Regular Season'}, 'resultSets': [{'name': 'TeamGameLog', 'rowSet': [], 'headers': ['Team_ID', 'Game_ID', 'GAME_DATE', 'MATCHUP', 'WL', 'MIN', 'FGM', 'FGA', 'FG_PCT', 'FG3M', 'FG3A', 'FG3_PCT', 'FTM', 'FTA', 'FT_PCT', 'OREB', 'DREB', 'REB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS']}]}
http://stats.nba.com/stats/teamgamelog?TeamID=1610612741&Season=2016-15&SeasonType=Regular+Season
{'resource': 'teamgamelog', 'parameters': {'TeamID': 1610612741, 'Season': '2016-15', 'LeagueID': None, 'SeasonType': 'Regular Season'}, 'resultSets': [{'name': 'TeamGameLog', 'rowSet': [], 'headers': ['Team_ID', 'Game_ID', 'GAME_DATE', 'MATCHUP', 'WL', 'MIN', 'FGM', 'FGA', 'FG_PCT', 'FG3M', 'FG3A', 'FG3_PCT', 'FTM', 'FTA', 'FT_PCT', 'OREB', 'DREB', 'REB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS']}]}
http://stats.nba.com/stats/teamgamelog?TeamID=1610612739&Season=2016-15&SeasonType=Regular+Season
{'resource': 'teamgamelog', 'parameters': {'TeamID': 1610612739, 'Season': '2016-15', 'LeagueID': None, 'SeasonType': 'Regular Season'}, 'resultSets': [{'name': 'TeamGameLog', 'rowSet': [], 'headers': ['Team_ID', 'Game_ID', 'GAME_DATE', 'MATCHUP', 'WL', 'MIN', 'FGM', 'FGA', 'FG_PCT', 'FG3M', 'FG3A', 'FG3_PCT', 'FTM', 'FTA', 'FT_PCT', 'OREB', 'DREB', 'REB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS']}]}

And so on ....................

Without the you get a 400 every time, you might also want to consider a sleep between requests and to investigate the rate limit/requests per day.

Collectives™ on Stack Overflow

Calling HTTP API from script generates 400, calling from browser generates 200

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related