Throws TargetInvocationException when downloading with WebClient DownloadStringAsync

Question

I am trying to download multiple webpages using the WebClient class. When I try to download a website's html, a TargetInvocationException is thrown, and I do not know why it happens. Here is my code:

    public HashSet<string> DownloadWebpages(HashSet<string> urls)
    {
        HashSet<string> HTML = new HashSet<string>();

        for (int i = 0; i < urls.Count; i++)
        {
            WebClient client = new WebClient();
            client.DownloadStringCompleted += (s, e) =>
            {
                try
                {
                    lock (HTML)
                    {
                        HTML.Add(e.Result); //The exception happens on this line  
                    }
                }
                catch { }
            };
            client.DownloadStringAsync(new Uri(urls.ElementAt(i)));
        }
        return HTML;
    }

Is there any way to fix this. All I'm trying to do is download multiple webpages using async, trying to make it has fast as possible.

Answer 1

The TargetInvocationException is thrown, because the webclient is not able to download the website. Here is a test,

string html = new WebClient().DownloadString("https://www.siteth@tw!llcause an error!/randompage/");

This will cause an exception. So if you tried to download the same webpage with your code, it will cause an TargetInvocationException

Answer 2

WebClient is an obsolete class replaced since 2012 by HttpClient. It was never built with HTTP APIs or thread safety in mind. It's easier to do what you want with HttpClient's GetStringAsync:

public async Task<HashSet<string>> DownloadWebpages(IEnumerable<string> urls)
{
    HashSet<string> HTML = new HashSet<string>();

    var client=new HttpClient();
    foreach (var url in urls)
    {
        var source=await client.GetStringAsync(url);
        HTML.Add(source);
    }
    return HTML;
}

Since .NET 6 you can even retrieve the URLs concurrently with Parallel.ForEachAsync. In this case you'd need a thread-safe collection to store them, eg a ConcurrentDictionary :

HttpClient _client=new HttpClient();

public async Task<ConcurrentDictionary<string,string>> DownloadWebpages(IEnumerable<string> urls)
{
    var HTML = new ConcurrentDictionary<string,string>();

    await Parallel.ForeachAsync(urls,async url=>{
    {
        var source=await client.GetStringAsync(url);
        HTML.Add(url,source);
    });
    return HTML;
}

HttpClient is thread-safe and meant to be reused.

If you absolutely have to use WebClient (why???) you can use DownloadStringTaskAsync. You won't be able to make concurrent calls though because WebClient isn't thread-safe.

public async Task<HashSet<string>> DownloadWebpages(IEnumerable<string> urls)
{
    HashSet<string> HTML = new HashSet<string>();

    var client=new WebClient();
    foreach (var url in urls)
    {
        var source=await client.DownloadStringTaskAsync(url);
        HTML.Add(source);
    }
    return HTML;
}