Skip to content

ObjectDisposedException fail-fast in DownloadAndRegisterEpsAsync progress callback (Progress<T> on ThreadPool touches disposed CTS) — WinML 1.2.3 #867

Description

@andrewtheart

Summary

An intermittent, unhandled System.ObjectDisposedException: The CancellationTokenSource has been disposed. is thrown on a ThreadPool thread from inside the SDK's own progress-callback marshaling during FoundryLocalManager.DownloadAndRegisterEpsAsync(...). Because it is unhandled on a ThreadPool thread, it fail-fasts the entire host process (0xc0000409). The calling application has no way to catch it.

Environment

  • Package: Microsoft.AI.Foundry.Local.WinML 1.2.3 — faulting module Microsoft.AI.Foundry.Local.Core.dll, assembly/file version 1.2.0.0
  • Host: WinUI 3 / Windows App SDK desktop app targeting net10.0-windows (the .NET Runtime crash event reported CoreCLR Version: 9.0.17)
  • OS: Windows 11 Pro, build 26200 (x64)
  • GPU / EP: NVIDIA GeForce RTX 5090 (CUDA execution provider)

What the app was doing

One-time initialization. After FoundryLocalManager.CreateAsync(...) we register the hardware execution providers with a progress callback, then fetch the catalog:

await manager.DownloadAndRegisterEpsAsync(
    (epName, pct) => { /* report progress to the UI */ },
    cancellationToken);

The crash occurs during this call. It is intermittent — in our logs, EP registration completed successfully 23 times and fail-fasted once. It appears most likely when the EPs are already cached, i.e. the download_and_register_eps command returns very quickly.

Crash details

Windows Error Reporting — Application Error (Event ID 1000):

Faulting application name: <host>.exe
Faulting module name:      Microsoft.AI.Foundry.Local.Core.dll, version: 1.2.0.0
Exception code:            0xc0000409

.NET Runtime (Event ID 1026):

Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<ExecuteCommandWithCallbackManaged>b__9(String, Double)
   at System.Progress`1.InvokeHandlers(Object)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext`1.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.WindowsThreadPool.DispatchCallback(IntPtr, IntPtr, IntPtr)

Root cause (analysis)

ExecuteCommandWithCallbackManaged marshals native progress updates through a System.Progress<T>, whose handler invocations are posted to the ThreadPool (the Progress<T> is constructed on a background thread with no SynchronizationContext). The command also uses a per-command CancellationTokenSource.

When the download_and_register_eps command finishes — quickly, when EPs are already cached — the SDK disposes that CTS, but a progress callback (b__9) is still queued on the ThreadPool. When that queued callback runs, it touches the now-disposed CTS, throwing ObjectDisposedException. Since it runs as a ThreadPool work item with no handler, the runtime terminates the process.

The throw happens inside the SDK's own wrapper (b__9), before control reaches the app-supplied (epName, pct) callback, so the calling application cannot observe or handle it (it is not surfaced through AppDomain.UnhandledException in a way that can prevent termination).

Note: appears already addressed on main

This managed-interop code path does not appear to exist in the current open-source tree — ExecuteCommandWithCallbackManaged, ExecuteWithTracker, ExecuteCommandManaged, and a Progress<T>-based NativeInterop all return no hits — and the current EP-download paths use disposal-safe patterns:

  • sdk/cs routes EP progress through a plain string ICoreInterop.CallbackFn (no Progress<T>).
  • sdk_v2/cs checks ct?.IsCancellationRequested (safe after dispose) inside the native callback.

So this looks specific to the compiled 1.2.x WinML assembly and likely resolved by the SDK v2 / unified-package refactor.

Ask

  1. Can you confirm this Progress<T> EP-progress / disposed-CTS race is gone in the unified/v2 package?
  2. Would you consider a 1.2.x servicing backport — e.g. guard the progress handler so it cannot touch the CTS after disposal, or defer disposing the per-command CTS until all queued Progress<T> callbacks have drained?

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions