Implement Parallel.Foreach inside a while loop

Question

I have scenario where I need to run a Parallel.Foreach within a while loop. I need to understand the impact of this implementation in terms of how the processing will take place. I will have an implementation something like this

 ConcurrentQueue<MyTable> queue = new ConcurrentQueue<MyTable>();

Here, I have initially added lot of items in queue but while execution also, more items can be added in the queue.

while(true)
{
    Parallel.Foreach(queue, (myTable) => {some processing});
    Sleep(sometime);
}

Each time one item will be de-queued and new thread will be spawned to work with it, in the meanwhile new items will be added for that I need to keep an infinite while loop.

Now, I need to understand that as concurrent queue is thread safe, I think each item will be processed one time only in spite of while above foreach but I am not sure about is that there will be multiple threads of foreach itself that will be spawning child threads or single copy of foreach will be running within while loop. I do not know how foreach itself is implemented.

I'm not really sure what you're asking, but Parallel.Foreach() will wait until it's tasks are complete before continuing on, as if it were any simple statement. — Jonesopolis
– Jonesopolis, Commented Aug 16, 2014 at 1:49
You don't need an infinite loop to poll for new items if you know when new items are added. — king jah
– king jah, Commented Aug 16, 2014 at 2:40
@kingjah I need infinite loop, because parallel.foreach will exit, if no items are added and all running tasks are completed. — Varun Gupta
– Varun Gupta, Commented Aug 18, 2014 at 17:51

svick · Accepted Answer · 2014-08-19 12:23:44Z

I have scenario where I need to run a Parallel.Foreach within a while loop.

I don't think you do. You want to process new items as they come in in parallel, but I think this is not the best way to do that.

I think the best way is to use ActionBlock from TPL Dataflow. It won't waste CPU or threads when there are no items to process and if you set its MaxDegreeOfParallelism, it will process items in parallel:

ActionBlock<MyTable> actionBlock = new ActionBlock<MyTable>(
    myTable => /* some processing */,
    new ExecutionDataflowBlockOptions
    {
        MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded
    });


...

actionBlock.Post(someTable);

If you don't want to or can't (it's .Net 4.5 only) use TPL Dataflow, another option would be use a single Parallel.Foreach() (no while) together with BlockingCollection and GetConsumingPartitioner() (not GetConsumingEnumerable()!).

Using this, the Parallel.Foreach() threads will be blocked when there are no items to process, but there also won't be any delays in processing (like the ones caused by your Sleep()):

BlockingCollection<MyTable> queue = new BlockingCollection<MyTable>();

...

Parallel.ForEach(
    queue.GetConsumingPartitioner(), myTable => /* some processing */);

...

queue.Add(someTable);

I think each item will be processed one time only in spite of while above foreach but I am not sure

That's one reason why you should use one of the options above, since they mean you don't need to know much about the details of how they work, they just work.

I have >100 biometric n/w devices, I need to read/write data from/to them. what is your advise regarding the above case. which methods is more efficient for my case. I was using P.For() inside while() BUt after I read your post, I came to know Im wrong. so please advise.

Collectives™ on Stack Overflow

Implement Parallel.Foreach inside a while loop

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related