2

I have a huge collection, over which i have to perform a specific task(which involves calling a wcf service). I want to control the number of threads instead of using Parallel.ForEach directly. Here i have 2 options: I am using below to partition the data:

List<MyCollectionObject> MyCollection = new List<MyCollectionObject>();
 public static IEnumerable<List<T>> PartitionMyData<T>(this IList<T> source, Int32 size)
        {
            for (int i = 0; i < Math.Ceiling(source.Count / (Double)size); i++)
            {
                yield return new List<T>(source.Skip(size * i).Take(size));
            }
        }

Option 1:

MyCollection.PartitionMyData(AutoEnrollRequests.Count()/threadValue).AsParallel().AsOrdered()
                                        .Select(no => InvokeTask(no)).ToArray();

 private void InvokeTask(List<MyCollectionObject> requests)
{
   foreach(MyCollectionObject obj in requests)
  {
    //Do Something
  }
}

Option2:

MyCollection.PartitionMyData(threadValue).AsOrdered()
                                        .Select(no => InvokeTask(no)).ToArray();

private void InvokeTask(List<MyCollectionObject> requests)
{
    Action<MyCollectionObject> dosomething = 
    {
    }
    Parallel.ForEach(requests,dosomething)
}

If i have 16 objects in my collection, as per my knowledge Option1 will launch 4 threads, each thread having 4 objects will be processed synchronously. Option 2 will launch 4 threads with 1 object each, process them and again will launch 4 threads. Can anyone please suggest which option is better?

P.S. I understand .Net framework does thread pooling and we need not control the number of threads but due to some design decision we want to use it.

Thanks In Advance, Rohit

2 Answers 2

2

I want to control the number of threads instead of using Parallel.ForEach directly

You can control de number of threads in Parallel.ForEach if you use this call with a ParallelOptions object:

Parallel.ForEach(requests,
                 new ParallelOptions(){MaxDegreeOfParallelism = 4}, //change here
                 dosomething)
Sign up to request clarification or add additional context in comments.

1 Comment

Hi Julian,its an old post but need ur help here. I used the above option but now getting OutOfMemory Exception sometimes. We monitored Memory Consumption on Server but it was below 60%. The stack trace shows:Exception Info: System.OutOfMemoryException Stack: at System.Threading.ThreadPoolWorkQueue.EnsureCurrentThreadHasQueue() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
1

It's impossible to give an A or B answer here. It depends on too many unknowns.

I will assume you want the fastest approach. To see which is better, run both on the target environment (or closest approximation you can get) and see which one completes fastest.

5 Comments

Hi James,its an old post but need ur help here. I used the option mentioned below by Julian but now getting OutOfMemory Exception sometimes. We monitored Memory Consumption on Server but it was below 60%. The stack trace shows:Exception Info: System.OutOfMemoryException Stack: at System.Threading.ThreadPoolWorkQueue.EnsureCurrentThreadHasQueue() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
OutOfMemory Exception means the process was unable to allocate memory. If the server has plenty of memory then I would posit it's most likely you are running .NET code compiled for x86 and have exceeded the 2Gb memory available. Check memory consumption of the process rather than the server. If you are not sure about x86, run corflags.exe (installed to windows sdk folder in program files when installing visual studio) from the windows SDK against your binaries to check if it is 32bit.
@user2159471 also, you'll probably find more help if you open a new question about that mem exception
Hi James, Thanks for your insights, all have been helpful till now. The process is 32 bit only and we have been able to trace down 2 memory leaks in one downstream process. But the frustrating part we are facing is sometimes application aborts after just 100 records and sometimes it processes 2500 records. If ParallleOption is modified to use a single thread , will it be of any help. [The memory leaks should not amount to 2 GB memory and it will take some time for us to get those fix in prod environment.]Also can you please have a look at code i posted @:
Also the memory leaks should not amount to 2 GB memory and it will take some time for us to get those fix in prod environment. So for time being we need urgently some temporary fix. Converting it to 64 bit process is one option, but need to be 100% sure that it will work. I am doubtful because sometimes application aborts just after 100 records.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.