How can I optmize my ExecuteInParallel function

Question

I have been reading quite a lot about Parallel .net 4 and I have to say that I am a bit confused when to use it.

This is my common scenario I have been given a task to migrate lots of xml files to a database.

I typically I have to

1. Read Xml Files (100.000) and more and order them numerically (each file is named 1.xml, 2.xml etc.).
2. Save to a database.

I thought the above was a perfect candidate for parallel programming.

Conceptually I would like to process many files at a times.

I am currently doing this:

private ResultEventArgs  progressResults=new ResultEventArgs();

public void ExecuteInParallelTest()
{
    var sw=new Stopwatch();
    sw.Start();
    int index = 0;
    cancelToken = new CancellationTokenSource();
    var parOpts = new ParallelOptions();
    parOpts.CancellationToken = cancelToken.Token;
    parOpts.MaxDegreeOfParallelism = Environment.ProcessorCount;  //It this correct?

    FileInfo[] files = myDirectory.EnumerateFiles("*.xml").ToArray();//Is this faster?
    TotalFiles = files.Count();
    try
    {
        Task t1 = Task.Factory.StartNew(() =>
        {
            try
            {
                Parallel.ForEach(files, parOpts, (file, loopState) =>
                {

                    if (cancelToken.Token.IsCancellationRequested)
                    {
                        cancelToken.Token.ThrowIfCancellationRequested();
                    }

                    index = Interlocked.Increment(ref index);

                    ProcessFile(file,index);

                                progressResults.Status=InProgress                                   

                    OnItemProcessed(TotalFiles,index,etc..);
                });
            }
            catch (OperationCanceledException ex)
            {
                OnOperationCancelled(new progressResults
                    {
                        progressResults.Status=InProgress                               
                        progressResults.TotalCount = TotalFiles;
                        progressResults.FileProcessed= index;
                        //etc..                                  
                    });

            }

            //ContinueWith is used to sync the UI when task completed.
        }, cancelToken.Token).ContinueWith((result) => OnOperationCompleted(new ProcessResultEventArgs
            {
                        progressResults.Status=InProgress
                        progressResults.TotalCount = TotalFiles;
                        progressResults.FileProcessed= index;
                        //etc..
            }), new CancellationTokenSource().Token, TaskContinuationOptions.None, TaskScheduler.FromCurrentSynchronizationContext());
    }
    catch (AggregateException ae)
    {
        //TODO:
    }
   }

My Questions: I am using .net 4.0 Is using Parallel the best/simpler way to speed up the processing of these files. Is the above psudo code good enough or Am I missing vital stuff,locking etc...

The most important question is: Forgetting the "ProcessFile" as I cannot optmize that as I have no control Is there room for optmisation

Should I partition the files in chunks eg 1-1000 - 1001-2000-2001-3000 would that improve performance (how do you do that)

Many thanks for any replies or link/code snippet that can help me understand better how I can improve the above code.

Answer 1

The reason you are not receiving responses is because your code is so horribly wrong. AsParallel() does not do anything for GetFiles(), files.Count() actually iterates the enumerable, so not only you read the files (or just the directory) twice, but doing Count() first, and then iterating through them later will read the files twice and could produce inconsistent counts if the directory is modified. It does not look like it's necessary to do Task.Factory.StartNew since it's your only task (which spawns parallel processing inside it). Parallel.ForEach will encapsulate all OperationCancelledException's into single AggregateException, but it will only do that after all parallel threads finish their work.

Answer 2

I left the code as it is as nobody provided me with a suitable answer