1
\$\begingroup\$

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this webapp on azure, I have tested this for 100 it works fine but for 10k it got timeout issue.

This is working code , I need a improvement here :)

private List<Task> taskEventList = new List<Task>();
public async Task ProcessStart()
{
    string messageData = "{\"name\":\"DemoData\",\"no\":\"111\"}";
    RegistryManager registryManager;

    Parallel.ForEach(deviceList, async (device) =>
    {
        // get details for each device and use key to send message
        device = await registryManager.GetDeviceAsync(device.DeviceId);
        SendMessages(device.DeviceId, device.Key, messageData);
    });

    if (taskEventList.Count > 0)
    {
        await Task.WhenAll(taskEventList);
    }
}

private void SendMessages(string deviceId, string Key, string messageData)
{
    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);
    //created separate Task
    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));
    taskEventList.Add(taskEvents);
}


private async Task ProcessMessages(string deviceId, string messageData)
{
    var startTime = DateTime.UtcNow;
    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))
    {
        await deviceClient.SendEventAsync(messageData);
    }
}
\$\endgroup\$
12
  • 2
    \$\begingroup\$ You are probably getting into a thread starvation as you just scheduling tasks and also borrowing threads from the thread pool. You need some kind of batching or throttling, as the thread pool is limited ( a thread has 2 MB, imagine 10k requests for threads) \$\endgroup\$ Commented Mar 15, 2019 at 17:34
  • 1
    \$\begingroup\$ Also if DeviceClient.Create creates and disposes a new HttpClient everytime, then you'll get out of available sockets \$\endgroup\$ Commented Mar 15, 2019 at 17:40
  • 3
    \$\begingroup\$ Hey @AdrianIftode that seems to be the beginnings of an answer. Are you sure you don't want to write one, now that the post has been reopened? \$\endgroup\$ Commented Mar 15, 2019 at 18:47
  • \$\begingroup\$ Thanks, I would like also to know about DeviceClient.Create, possible @Neo ? \$\endgroup\$ Commented Mar 15, 2019 at 20:40
  • \$\begingroup\$ @Neo, are you sure everything is ok with this check: while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15)) ? \$\endgroup\$ Commented Mar 15, 2019 at 21:07

0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.