Efficient way of storing and retrieving Large Json ( 100 mb +) from a file using this article and Json.Net

Question

I have been trying to figure out the best way to store and retrieve data from a file containing Json using JSon.net. During my research i found couple of ways.

http://www.drdobbs.com/windows/parsing-big-records-with-jsonnet/240165316
http://www.ngdata.com/parsing-a-large-json-file-efficiently-and-easily/ Jackson Api ( only available for Java )

Can any of this method be used with Json.Net? Is there an actual implementation available for the article at DrDoobs? I could not figure out the HandleToken(reader.TokenType, reader.Value) method.

Current Stats:

Around 5 min to write the Json of size ~ 60Mb
Around 3 min to read the Json.

Current Code:

public static T DeserializeJsonFromStream<T>(Stream s)
    {
        using (StreamReader reader = new StreamReader(s))
        {
            using (JsonTextReader jsonReader = new JsonTextReader(reader))
            {
                JsonSerializer ser = new JsonSerializer();
                ser.Formatting = Newtonsoft.Json.Formatting.None;
                ser.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
                ser.TypeNameHandling = TypeNameHandling.All;
                ser.NullValueHandling = NullValueHandling.Ignore;
                ser.Error += ReportJsonErrors;
                ser.DateFormatHandling = DateFormatHandling.IsoDateFormat;
                return ser.Deserialize<T>(jsonReader);
            }
        }
    }
    public static void SerializeJsonIntoStream(object value, Stream s)
    {

        using (StreamWriter writer = new StreamWriter(s))
        {
            using (JsonTextWriter jsonWriter = new JsonTextWriter(writer))
            {
                JsonSerializer ser = new JsonSerializer();

                ser.Formatting = Newtonsoft.Json.Formatting.None;
                ser.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
                ser.TypeNameHandling = TypeNameHandling.All;
                ser.NullValueHandling = NullValueHandling.Ignore;
                ser.Error += ReportJsonErrors;

                ser.Serialize(jsonWriter, value);
                jsonWriter.Flush();
            }
        }
    }

A file might be a bad place for storing and retrieving large amounts of data. Perhaps you should consider storing data in a database. — NotARobot
– NotARobot, Commented Oct 14, 2014 at 19:18
I did try using SqlLite for windows store apps, but ended up frustrated with all the exception coming out of SqlLite3.dll. — AGCodes
– AGCodes, Commented Oct 14, 2014 at 19:21
It seems to me like the second one could be implemented with a custom JsonConverter, which Json.NET provides. — Nate Diamond
– Nate Diamond, Commented Oct 16, 2014 at 17:16
The second one makes a call to jp.readValueAsTree(); which is not available under JsonReader. Currently i do have — AGCodes
– AGCodes, Commented Oct 16, 2014 at 18:57

AGCodes · Accepted Answer · 2014-10-17 06:29:35Z

As i did not need to read the serialized file manually, Using Protocol Buffer solved the problem, It takes 6 seconds to serialize /deserialize the same amount of data.

It's available on Nuget at http://www.nuget.org/packages/protobuf-net

    public static async Task<T> DeserializeFromBinary<T>(string filename, StorageFolder storageFolder)
    {
        try
        {

            var file = await storageFolder.GetFileAsync(filename);
            if (file != null)
            {
                var stream = await file.OpenStreamForReadAsync();
                T content = Serializer.Deserialize<T>(stream);
                return content;
            }
        }

        catch (NullReferenceException nullException)
        {
            logger.LogError("Exception happened while de-serializing input object, Error: " + nullException.Message);
        }
        catch (FileNotFoundException fileNotFound)
        {
            logger.LogError("Exception happened while de-serializing input object, Error: " + fileNotFound.Message);
        }
        catch (Exception e)
        {
            logger.LogError("Exception happened while de-serializing input object, Error: " + e.Message, e.ToString());
        }
        return default(T);
    }


    public static async Task<bool> SerializeIntoBinary<T>(string fileName, StorageFolder destinationFolder, Content content)
    {
        bool shouldRetry = true;
        while (shouldRetry)
        {
            try
            {

                StorageFile file = await destinationFolder.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);

                using (var stream = await file.OpenStreamForWriteAsync())
                {
                    Serializer.Serialize(stream, content);
                    shouldRetry = false;
                    return true;
                }
            }
            catch (UnauthorizedAccessException unAuthorizedAccess)
            {
                logger.LogError("UnauthorizedAccessException happened while serializing input object, will wait for 5 seconds, Error: " + unAuthorizedAccess.Message);
                shouldRetry = true;
            }
            catch (NullReferenceException nullException)
            {
                shouldRetry = false;
                logger.LogError("Exception happened while serializing input object, Error: " + nullException.Message);
            }
            catch (Exception e)
            {
                shouldRetry = false;
                logger.LogError("Exception happened while serializing input object, Error: " + e.Message, e.ToString());
            }

            if (shouldRetry)
                await Task.Delay(5000);
        }
        return false;
    }

Collectives™ on Stack Overflow

Efficient way of storing and retrieving Large Json ( 100 mb +) from a file using this article and Json.Net

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related