Castalia’s Clipboard history + TRichEdit = IDE deadlock

By | August 15, 2016

Eugene Kotlyarov found a problem with Castalia’s clipboard history feature combined with a TRichEdit in a debugged application. If you copy text from the RichEdit to the clipboard the first time, the IDE and the debugged application stop responding and start reacting after a ~30 seconds timeout. After that, copy to clipboard works until you restart the debugged application. Win32 and Win64 have the same problem.
https://plus.google.com/+EugeneKotlyarovPlus/posts/R783AB3DfYL

What happens:

  • The debugged application invokes WM_COPY by Ctrl+C/Ctrl+Insert/Context-menu “Copy”.
  • WM_COPY set the RichEdit as clipboard renderer without putting the actual data into the clipboard.
  • Castalia’s TClipboardHistoryForm receives the WM_CLIPBOARDUPDATE message in the main thread and calls GetClipboardData (via Clipboard.AsText).
  • The debugged application’s RichEdit loads a DLL to provide the actual clipboard data for the GetClipboardData call.
  • The debugger receives a LOAD_DLL_DEBUG_EVENT for the DLL.
  • The debugger posts a message to the debugger window in the main thread and goes to sleep with WaitForSingleObject to wait for the debugger window to process the DLL load event.

Now everything waits. GetClipboardData is blocked because the clipboard owner, the debugged application, is trapped in a debugger event. The debugger is blocked because the debugger window doesn’t get the posted message because the blocked GetClipboardData prevents the main thread from processing messages.

Fortunately a Microsoft developer knew about this possible deadlock and put a timeout into GetClipboardData.

 

A solution that I’ve put into the IDE Fix Pack development version (not available) moves the WM_CLIPBOARDUPDATE message handling into a thread window, so that the call to GetClipboardData doesn’t block the main thread and the debugger can process the DLL load event.