How do I stream a large file from api to api without using disk and running out of memory?

Question

I have a API HTTPGET that retrieves a file from another API, and spits it out. This works for smaller files, but the problem I'm having is that the files retrieved can be rather large in size (up to 2gb), and the MemoryStream is a limitation. Any ideas how to stream the file content without using disk and avoiding the 'out of memory' exception?

Controller:

[Route("{id}/file", Name = "GetContentFile")]
[HttpGet]
public IHttpActionResult GetContentFile(string id)
{
    if (String.IsNullOrEmpty(id))
        return BadRequest();
    ContentFile cfl = new ContentFile();
    var ret = new HttpResponseMessage(HttpStatusCode.OK);
    try
    {
        cfl = otcrepo.GetContentFile(id);
        var mstream = new MemoryStream(cfl.Data);
        ret.Content = new StreamContent(mstream);
        ret.Content.Headers.ContentType = new MediaTypeHeaderValue(cfl.ContentType);
        ret.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
    }
    catch
    {
        return InternalServerError();
    }
    if (cfl != null)
    {
        ResponseMessageResult responseMessageResult = ResponseMessage(ret);
        return responseMessageResult;
    }
    else
    {
        return NotFound();
    }
}

Model:

public class ContentFile
{
    public string Filename { get; set; }
    public byte[] Data { get; set; }
    public StreamContent DataStream { get; set; }
    public string ContentType { get; set; }
}

Repository call:

public ContentFile GetContentFile(string id)
{
    ContentFile fl = new ContentFile();
    using (var htc = new HttpClient())
    {
        var response = htc.GetAsync(ConfigurationManager.AppSettings["BaseUrl"] + "/api/v2/nodes/" + id + "/content/").Result;

        fl.Data = response.Content.ReadAsByteArrayAsync().Result;
        fl.ContentType = response.Content.Headers.GetValues("Content-Type").FirstOrDefault();
    }
    return fl;
}

Thanks.

If you aren't willing to use memory or disk, I can't see how this is possible. What is the other alternative (than memory or disk)? — mjwills
– mjwills, Commented Oct 18, 2018 at 20:25
I'll rephrase my question, since as you point out, some memory must be used. How to do it while avoiding the outofmemoryexception? — Not_Sure
– Not_Sure, Commented Oct 18, 2018 at 20:31
Whoa, hold on a minute here, you have other problems to worry about. You are using .Result on an asynchronous task. That is incredibly dangerous! This can deadlock your server. It's asynchronous for a reason; you have to await that. Get your code right before you attempt to get it fast. — Eric Lippert
– Eric Lippert, Commented Oct 18, 2018 at 20:39
That aside, I do not understand why you are doing what you are doing. You say you don't want to put the file in memory, but you are reading it into memory as a byte array. If you want to stream it then why are you moving it into a byte array, and then putting a stream on top of the byte array? That is a very strange thing to do. Why are you not just calling GetResponseStream ? You want a stream, well, get a stream. Don't get a byte array and build a stream on top of it. — Eric Lippert
– Eric Lippert, Commented Oct 18, 2018 at 20:41

Magnus · Accepted Answer · 2018-10-23 14:06:31Z

So instead of GetContentFile reading the complete stream into an array and returning it we return the stream. Than there is no need for the MemoryStream.

[Route("{id}/file", Name = "GetContentFile")]
[HttpGet]
public async Task<IHttpActionResult> GetContentFile(string id)
{
    ...
    var result = await otcrepo.GetContentFile(id);
    ret.Content = new StreamContent(result.stream);
    ret.Content.Headers.ContentType = new MediaTypeHeaderValue(result.contentType);
    ...

}

public async Task<(Stream stream, string contentType)> GetContentFile(string id)
{
    var htc = new HttpClient()
    var response = await htc.GetAsync(ConfigurationManager.AppSettings["BaseUrl"] + "/api/v2/nodes/" + id + "/content/");

    var stream = await response.Content.ReadAsStreamAsync();
    var contentType = response.Content.Headers.GetValues("Content-Type").FirstOrDefault();
    return (stream, contentType);
}

Collectives™ on Stack Overflow

How do I stream a large file from api to api without using disk and running out of memory?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related