Reading JSON into GO Strings

Question

So I have a JSON File in the format...

[
  {
    "Key":"Value",
    "Key2":"Value2",
    "Key3":"Value3"
  },
  {
    "Foo":"Bar",
    "Blah":2
  }
]

I want to just read in the hash parts of it and pass them to an HTTP request like in goRequest, because goRequest is fine with just the JSON being in a String.

package main
request := gorequest.New()
resp, body, errs := request.Post("http://example.com").
Set("Notes","gorequst is coming!").
Send(`{"Foo":"Bar","Blah":2}`).
End()

I don't care what the JSON is and I don't need to unmarshal it to any go Structs or anything of the sort, it's fine just remaining as a string and being totally untouched, just passed along to the request.

I've seen a lot online about it, but it always seems to wanna un-marshal the JSON to Go Structs and the sort, which is fine if you want to care about what actually is in the JSON, but in my case this seems like unnecessary overhead.

How would I accomplish something like this? It seems pretty simple, but none of the existing JSON libraries for Go seem to be able to accomplish this.

Thanks.

Perhaps I don't understand your question, but you don't need to use a JSON library if all you want to do is pass around JSON text. — user4122236
– user4122236, Commented Dec 11, 2014 at 19:39
I need each individual hash to be it's own request, so I need to get just each individual hash. The JSON file is huge and I want to stream it because loading it in memory is not possible. — Stephen Carman
– Stephen Carman, Commented Dec 11, 2014 at 20:52

dyoo · Accepted Answer · 2014-12-11 22:15:27Z

You are probably looking for json.RawMessage.

For example:

package main

import (
    "encoding/json"
    "fmt"
    "log"
)

func main() {
    txt := []byte(`
    [
      {"key1"  : "value1" },
      {"key2"  : "value2" }
    ]`)
    msg := []json.RawMessage{}
    err := json.Unmarshal(txt, &msg)
    if err != nil {
        log.Fatal(err)
    }
    for _, c := range msg {
        fmt.Printf("%s\n", string(c))
    }
}

Note that the redundant white space in the example separating the key/value pairs is intentional: you will see that these are preserved in the output.

Alternatively, even if you don't care about the exact structure, you can still dynamically poke at it by using an interface{} variable. See the JSON and Go document for a running example of this, under the Generic JSON with interface{} section.

If we are trying to do something like a streaming approach, we may attempt to do something custom with the io.Reader. The JSON parser assumes you can represent everything in memory at once. That assumption may not hold in your situation, so we have to break a few things.

Perhaps we might manually consume bytes in the io.Reader till we eat the leading [ character, and then repeatedly call json.Decode on the rest of the io.Reader. Something like this:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "log"
)

func main() {
    var txt io.Reader = bytes.NewBufferString(`
    [
      {"key1"  : "value1" },
      {"key2"  : "value2" }
    ]`)
    buf := make([]byte, 1)
    for {
        _, err := txt.Read(buf)
        if err != nil {
            log.Fatal(err)
        }
        if buf[0] == '[' {
            break
        }
    }
    for {
        decoder := json.NewDecoder(txt)
        msg := json.RawMessage{}
        err := decoder.Decode(&msg)
        if err != nil {
            break
        }
        fmt.Printf("I see: %s\n", string(msg))

        txt = decoder.Buffered()
        for {
            _, err := txt.Read(buf)
            if err != nil {
                log.Fatal(err)
            }
            if buf[0] == ',' || buf[0] == ']' {
                break
            }
        }

    }
}

This code is severely kludgy and non-obvious. I also don't think it's a good idea. If you have to deal with this in a streaming fashion, then JSON is likely not a good serialization format for this scenario. If you have control over the input, then you should consider changing it so it's more amendable to a streaming approach: hacks like what we're doing here are a bad smell that the input is in the wrong shape.

Thanks Dyoo, this is fantastic! I can infact change the input format to CSV, which probably would be easier to stream, but I'd have to take the headers and map them to the values of each line, so I guess afterall I am going to have to encode some JSON from the CSV in order to make this work in a streaming fashion. Is the standard CSV package capable of that?
One possibility: you might consider changing input so it consists of a bunch of individual JSON objects in sequence, but not within an array, but rather just arranged serially in the byte stream. You'll note that all the hackery in the original approach is there just to deal with the artificial syntactic structure placed there due to being in an explicit array.
Also read similar approaches in: developers.google.com/protocol-buffers/docs/…. The problem you're running into is common, and so are the solutions: rather than build a single large structure and send that over the wire, send each individual piece, delineated in some way.
I've suggested a kind of solution below, exactly what you suggest. Basically rather than trying to convert all the CSV to JSON, just taking each CSV instance and converting that to JSON then firing off a request. The code is below, does that seem like what you were talking about?
@stephen I don't believe so. I added the comment because it seemed like the questioner unfortunately misinterpreted my meaning. CSV is probably not the right envelope, since the content may contain arbitrary structure. When I said: "JSON isn't the right serialization format", I should have been more clear: the individual message chunks can be encoded any way you want, and JSON should be fine there. It's a matter of encoding the stream, and trying to encode the stream as one JSON object is probably not a good idea, because most JSON parsers will try to read the whole object at once.

Stephen Carman · Accepted Answer · 2014-12-12 20:01:05Z

Here is what I was thinking as a solution, does this look sane?

package main
import (
        "encoding/csv"
        "fmt"
        "os"
    "bytes"
    "flag"
    "github.com/parnurzeal/gorequest"
)
func process_line(headers []string, line []string) {
    var comma string = ""
    var buffer bytes.Buffer
    buffer.WriteString("[{")
        for i := range headers {
            buffer.WriteString(fmt.Sprintf("%s\"%s\":\"%s\"", comma, headers[i], line[i]))
                comma = ","
        }
        fmt.Fprintf(&buffer,"}]\n")
    request := gorequest.New()
    resp, body, errs := request.Post("www.something.com").
                Set("Content-Type", "application/json").
                Set("Accept", "application/json").
                Send(buffer.String()).End()
    if errs == nil {
        return resp
    }else{
        fmt.Println(errs)
    }
}
func main() {
    file := flag.String("file", "", "Filename?")
    flag.Parse()

    if *file == "" {
        fmt.Println("No file specified. :-(")
        os.Exit(1)
    }
        csvFile, err := os.Open(*file)
        if err != nil {
        fmt.Println(err)
        }
        defer csvFile.Close()
        reader := csv.NewReader(csvFile)
    var i int = 0
    var headers []string
    for {
        line, err := reader.Read()
        if err != nil {
            break
        }
        if i == 0 {
            headers = line
        }else{
            go process_line(headers, line)
        }
        if i%100 == 0 {
            fmt.Printf("%v records processed.\n", i)
        }
        i += 1
    }
}

Collectives™ on Stack Overflow

Reading JSON into GO Strings

2 Answers 2

5 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Related