1

I'm doing a project where my Arduino uses an external ESP8266 wifi module (with AT firmware) to send http requests and receive server response.

I'm trying to get an NBA scoreboard, and here's what my Arduino sends to ESP:

> AT+CIPSTART=0,"TCP","data.nba.net",80
0,CONNECT

OK

> AT+CIPSEND=0,103
SEND OK

> GET /data/10s/prod/v1/20190214/scoreboard.json HTTP/1.1
Host: data.nba.net
Connection: keep-alive

+IPD,0,1460:HTTP/1.1 200 OK..
.............................
...here comes the response...
.............................

Now the code works fine, and I'm able to get the json file needed. The problem is that the response is sometimes too long and the file gets truncated, thus the response is incomplete. The file itself is sometimes pretty large, but I only need a few numbers from it (current teams + current score). So my questions are:

  1. I use readString() method (of SoftwareSerial object instance) to read the response. Is there a way to parse the response chunk by chunk without loading everything in my ESP memory?
  2. Is there a way to limit the response?
  3. Are there any alternatives to the method I'm using? Maybe parsing the website (nba.com or espn.com/nba) directly? If so, how can I do that?
  4. Maybe there's a way to write a separate (free) online API to parse json, constraint the response and send the small version upon request?

PS. I tried doing the same without AT firmware, flashing the module with a code that used HTTPClient directly (with ESP8266WiFi.h and ESP8266HTTPClient.h libraries):

HTTPClient http;
http.begin(request);
if (http.GET() > 0) {
    response = http.getString();
}
http.end();

That didn't help much. I'm still running out of memory since I'm directly loading the response to memory with getString().

PPS. Just in case it matters, I'm using ATmega2560 and a small ESP-01 module (similar to this).

Improve this question
6
  • large ... you'll need to be more specific than large Commented Feb 21, 2019 at 5:15
  • @JaromandaX .... go here .... data.nba.net/data/10s/prod/v1/20190214/scoreboard.json Commented Feb 21, 2019 at 7:46
  • what data are you retrieving? ..... you probably already know this, but here is the API for anyone else reading this github.com/kshvmdn/nba.js/blob/master/docs/api/DATA.md .......... there may be another way to pull the data Commented Feb 21, 2019 at 8:14
  • No. You go there. How hard is it to quantify large Commented Feb 21, 2019 at 9:46
  • @JaromandaX, > 10 KB is what I would call large. I think above that it just ignores the rest of the response. Commented Feb 21, 2019 at 17:54

2 Answers 2

Reset to default
1

Ok, so I was able to do this. Here's the workaround.

Let's say the GET response is a flat string containing a large .json file with all the commas and curly brackets. The idea is to read it byte-by-byte, and instead of storing it, just find the key you need and extract its value.

We want to find the field containing, say, "97.5 FM" from http://data.nba.net/data/10s/prod/v1/20190223/scoreboard.json (that's at the very end), skip 3 bytes (which is just " | ") and then read and Serial.print 4 bytes (which is "1280")*. We would do something like this (of course make sure to connect to wifi and include <ESP8266HTTPClient.h>:

    HTTPClient http;
    http.begin("http://data.nba.net/data/10s/prod/v1/20190223/scoreboard.json");
    int httpCode = http.GET();
    if (httpCode == HTTP_CODE_OK)
      Serial.println(getValue(http, "97.5 FM", 3, 4));

Where the getValue() function is defined as follows:

    String getValue(HTTPClient &http, String key, int skip, int get) {
      bool found = false, look = false;
      int ind = 0;
      String ret_str = "";
    
      int len = http.getSize();
      char char_buff[1];
      WiFiClient * stream = http.getStreamPtr();
      while (http.connected() && (len > 0 || len == -1)) {
        size_t size = stream->available();
        if (size) {
          int c = stream->readBytes(char_buff, ((size > sizeof(char_buff)) ? sizeof(char_buff) : size));
          if (len > 0)
            len -= c;
          if (found) {
            if (skip == 0) {
              ret_str += char_buff[0];
              get --;
            } else
              skip --;
            if (get <= 0)
              break;
          }
          else if ((!look) && (char_buff[0] == key[0])) {
            look = true;
            ind = 1;
          } else if (look && (char_buff[0] == key[ind])) {
            ind ++;
            if (ind == key.length()) found = true;
          } else if (look && (char_buff[0] != key[ind])) {
            ind = 0;
            look = false;
          }
        }
      }
      return ret_str;
    }

You can call this getValue() whatever many times you like, but the strings you're looking for need to be only after each other, because the response to GET is read byte-by-byte just once.

There is probably (i.e., most likely) a better way of doing this, but this approach works and it's pretty fast.

* This, of course, is just an example, and you can overload this getValue() function definition to, say, read until it gets a curly bracket or whatever.

Improve this answer
0

Check out the Streaming JSON Parser. It's designed to parse large JSON objects on processors with little memory. You'll just read one character at a time from the HTTP connection rather than the whole response; pass the character to the library and it'll call code you provide every time it sees an array, object, key or value. Then you can extract just the pieces you want without the overhead of loading it all in memory at once.

The library has good examples that'll show you how to use it.

Improve this answer
2
  • Thanks, I've seen this before, but I'm not completely sure how this works. (a) In the example, in ESP.getFreeHeap(), is ESP the software serial instance? (b) What should I actually feed to the parse.parse(...)? Is it the http.getString() return? Commented Feb 21, 2019 at 20:02
  • ESP.getFreeHeap() is a diagnostic function. It returns how much memory is left to allocate. The example calls it to prove that the parser isn't leakimg or running out of memory. In the example it should be the same before and after running the parser. ESP is a special class with functions for dealing with the ESP8266 processor. You don't need to deal with any of the ESP functions. You need to read a single character at a time from your HTTP client and then feed each character to parse.parse(). Commented Feb 22, 2019 at 4:53

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.