An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll in LINQ Search

Question

Using this article from MSDN, I'm trying to search through files in a directory. The problem is, every time I execute the program, I get:

"An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll".

I have tried to some other options like StreamReader, but I can't get it to work. These files are HUGE. Some of them range in upwards to 1.5-2GB each and there could be 5 or more files per day.

This code fails:

private static string GetFileText(string name)
{
    var fileContents = string.Empty;
    // If the file has been deleted since we took  
    // the snapshot, ignore it and return the empty string. 
    if (File.Exists(name))
    {
        fileContents = File.ReadAllText(name);
    }
    return fileContents;
}

Any ideas what could be happening or how to make it read without memory errors?

Entire code (in case you don't want to open the MSDN article)

class QueryContents {
public static void Main()
{
    // Modify this path as necessary. 
    string startFolder = @"c:\program files\Microsoft Visual Studio 9.0\";

    // Take a snapshot of the file system.
    System.IO.DirectoryInfo dir = new System.IO.DirectoryInfo(startFolder);

    // This method assumes that the application has discovery permissions 
    // for all folders under the specified path.
    IEnumerable<System.IO.FileInfo> fileList = dir.GetFiles("*.*", System.IO.SearchOption.AllDirectories);

    string searchTerm = @"Visual Studio";

    // Search the contents of each file. 
    // A regular expression created with the RegEx class 
    // could be used instead of the Contains method. 
    // queryMatchingFiles is an IEnumerable<string>. 
    var queryMatchingFiles =
        from file in fileList
        where file.Extension == ".htm" 
        let fileText = GetFileText(file.FullName)
        where fileText.Contains(searchTerm)
        select file.FullName;

    // Execute the query.
    Console.WriteLine("The term \"{0}\" was found in:", searchTerm);
    foreach (string filename in queryMatchingFiles)
    {
        Console.WriteLine(filename);
    }

    // Keep the console window open in debug mode.
    Console.WriteLine("Press any key to exit");
    Console.ReadKey();
}

// Read the contents of the file. 
static string GetFileText(string name)
{
    string fileContents = String.Empty;

    // If the file has been deleted since we took  
    // the snapshot, ignore it and return the empty string. 
    if (System.IO.File.Exists(name))
    {
        fileContents = System.IO.File.ReadAllText(name);
    }
    return fileContents;
}

}

recursive · Accepted Answer · 2015-08-19 23:40:49Z

3

The problem you're having is based on trying to load multiple gigabytes of text at the same time. If they're text files, you can stream them and just compare one line at a time.

var queryMatchingFiles =
    from file in fileList
    where file.Extension == ".htm" 
    let fileLines = File.ReadLines(file.FullName) // lazy IEnumerable<string>
    where fileLines.Any(line => line.Contains(searchTerm))
    select file.FullName;

answered Aug 19, 2015 at 23:40

recursive

86.4k36 gold badges155 silver badges246 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

RazorSharp Over a year ago

Exactly what I needed.. Thank you very much!

Rob Over a year ago

Just make sure your search term doesn't include a line break ;)

Community · Accepted Answer · 2017-05-23 12:06:17Z

I would suggest that you are getting an out of memory error because the way the query is written I believe that you will need to load the entire text of every file into memory and none of the objects can be released until the entire file set has been loaded. Could you not check for the search term in the GetFileText function and then just return a true or false?

If you did that the file text at least falls out of scope at the end of the function and the GC can recover the memory. It would actually be better to rewrite as a streaming function if you are dealing with large files/amounts then you could exit your reading early if you come across the search term and you wouldn't need the entire file in memory all the time.

Previous question on finding a term in an HTML file using a stream

Objects used from previous iterations of linq queries are eligible for GC. But the stream approach is certainly reasonable.

Collectives™ on Stack Overflow

An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll in LINQ Search

2 Answers 2

2 Comments

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Linked

Related