Parsing multiple JSON objects in Go

Question

Objects like the below can be parsed quite easily using the encoding/json package.

[ 
    {"something":"foo"},
    {"something-else":"bar"}
]

The trouble I am facing is when there are multiple dicts returned from the server like this :

{"something":"foo"}
{"something-else":"bar"}

This can't be parsed using the code below.

correct_format := strings.Replace(string(resp_body), "}{", "},{", -1)
json_output := "[" + correct_format + "]"

I am trying to parse Common Crawl data (see example).

How can I do this?

Note that the document at the URL in the question contains a text file with line-delimited JSON (i.e. one JSON document per line). — maerics
– maerics, Commented May 15, 2018 at 16:24

RobEarl · Accepted Answer · 2024-09-25 10:57:12Z

17

Use a json.Decoder to decode them:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "strings"
)

var input = `
{"foo": "bar"}
{"foo": "baz"}
`

type Doc struct {
    Foo string
}

func main() {
    dec := json.NewDecoder(strings.NewReader(input))
    for {
        var doc Doc

        err := dec.Decode(&doc)
        if err == io.EOF {
            // all done
            break
        }
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("%+v\n", doc)
    }
}

Playground: https://play.golang.org/p/ANx8MoMC0yq

edited Sep 25, 2024 at 10:57

RobEarl

7,9226 gold badges38 silver badges52 bronze badges

answered May 15, 2018 at 14:28

Peter

32k6 gold badges57 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

LucaB Over a year ago

This is a great solution because it works both for one object per line, and for formatted streams. that the check on io.EOF can be eliminated by using json.Decoder.More(). go.dev/play/p/Ouo7xlDV6iy

Peter Over a year ago

More is intended to be used with the Token interface of the decoder. When used with newline delimited JSON it silently accepts certain types of malformed inputs: go.dev/play/p/ync4g9OuTrq

Nebril · Accepted Answer · 2018-05-15 12:26:27Z

2

Seems like each line is its own json object.

You may get away with the following code which will structure this output into correct json:

package main

import (
    "fmt"
    "strings"
)

func main() {
    base := `{"trolo":"lolo"}
{"trolo2":"lolo2"}`

    delimited := strings.Replace(base, "\n", ",", -1)

    final := "[" + delimited + "]"
    fmt.Println(final)
}

You should be able to use encoding/json library on final now.

answered May 15, 2018 at 12:26

Nebril

3,3211 gold badge35 silver badges51 bronze badges

Comments

dm03514 · Accepted Answer · 2018-05-15 12:28:45Z

Another option would be to parse each incoming line, line by line, and then add each one to a collection in code (ie a slice) Go provides a line scanner for this.

yourCollection := []yourObject{}
scanner := bufio.NewScanner(YOUR_SOURCE)
for scanner.Scan() {
    obj, err := PARSE_JSON_INTO_yourObject(scanner.Text())
    if err != nil {
       // something
    }
    yourCollection = append(yourCollection, obj)
}
if err := scanner.Err(); err != nil {
    fmt.Fprintln(os.Stderr, "reading standard input:", err)
}

Manoj · Accepted Answer · 2020-01-14 07:48:47Z

You can read the ndjson from the file row by row and parse it then apply the logical operations on it. In the below sample instead of reading from the file, I have used an Array of JSON string.

import (
    "encoding/json"
    "fmt"
)

type NestedObject struct {
    D string
    E string
}

type OuterObject struct {
    A string
    B string
    C []NestedObject
}

func main() {

    myJsonString := []string{`{"A":"1","B":"2","C":[{"D":"100","E":"10"}]}`, `{"A":"11","B":"21","C":[{"D":"1001","E":"101"}]}`}
    for index, each := range myJsonString {
        fmt.Printf("Index value [%d] is [%v]\n", index, each)
        var obj OuterObject
        json.Unmarshal([]byte(each), &obj)
        fmt.Printf("a: %v, b: %v, c: %v", obj.A, obj.B, obj.C)
        fmt.Println()
    }
}

Output:

Index value [0] is [{"A":"1","B":"2","C":[{"D":"100","E":"10"}]}]
a: 1, b: 2, c: [{100 10}]
Index value [1] is [{"A":"11","B":"21","C":[{"D":"1001","E":"101"}]}]
a: 11, b: 21, c: [{1001 101}]

Try it on golang play

Collectives™ on Stack Overflow

Parsing multiple JSON objects in Go

4 Answers 4

2 Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

Comments

Comments

Linked

Related