Using Fibers for tab-modal forms

Maybe some don’t know that I’m not only writing IDE plugins but also write code for real world problems. One of those real world problems was that I needed a way to have multiple windows in a “tab-modal” state. If the user switches to another tabsheet, he can open another tab-modal form without closing the one on the other tabsheet. The main issue you have to face is how you get the “ShowTabModal” method to block the execution until the tab-modal form is closed, especially because there can be more than one blocking “ShowTabModal” call.

The naive way would be to call ShowTabModal which has its own message loop like ShowModal. But what if the user closes the parallel opened tab-modal forms in a different order than he opened them? He can’t close the first tab-modal form until he closed the second tab-modal form because the second message loop doesn’t return until the second form is closed.
The first idea to solve this was to use threads for each tabsheet. But that doesn’t work because the VCL isn’t thread-safe and windows have a thread affinity. OK, I got this working by changing the VCL. But with all the global variables in 3rd party controls it is too risky for production usage. The next idea was to follow Google Chrome’s way by using one process per tab. But with the time constraint that I have this can’t be done because I would have to make major changes to our data layer and set up a communication channel between the processes. Some tests also showed me that this is a debugging-nightmare in Delphi (Debug into spawned processes), and that Avira AntiVir would mark my EXE file as a false-positive unless I remove the MainIcon resource.

With all the thinking about how I could solve this issue, I remembered that Windows has something called Fibers. The idea is to execute the message loop in a Fiber making the ShowTabModal call blocking until the SwitchToFiber call returns. This message loop looks for the ModalResult value of all active tab-modal forms and if one is closed it returns to the Fiber that called ShowTabModal for this specific form and deletes the message loop Fiber that was active before returning because it isn’t needed anymore. That way there are n+1 message loops for n ShowTabModal calls (one for the main message loop and n for n ShowTabModal calls) but only one Fiber is active.
This works really well. But then I came to the problem of closing the top level window that hosts all tab-modal forms. If the users presses the window’s X button all tab-modal forms must be closed after consulting their OnCloseQuery event handler. But when closing a form the execution pointer must return to the ShowTabModal call in order to keep the execution flow correct. This is a problem because the code has to wait in the top level window’s OnCloseQuery handler until all tab-modal forms are closed, otherwise the CanClose flag can’t be set correctly. The solution to this was easier than I thought because I just had to do the same thing that I did in ShowTabModal. I had to create a new message loop Fiber that waited until all tab-modal forms are closed or one of them aborted the close. And because all message loop Fibers had to do the job for all forms I put that code into the message loop Fiber procedure that I already had written for ShowTabModal.
The theory and paper drafts of how this all works looked right. But when I closed the top level window I got an access violation in the code the followed the SwitchToFiber call in the top level window’s CloseQuery handler. After some investigation I found out that the stack of the Fiber that ran the CloseQuery handler was taken over by another message loop Fiber. And when the program switched back to the CloseQuery handler it got a trashed stack. But I didn’t see what I was doing wrong, especially if I only opened one level of tab-modal forms, it worked but if I opened two levels (tab-modal forms opens another tab-modal form) it crashed. And if I closed the top level window by using  File/Quit, it worked. So only the system menu, Alt+F4 and the X button crashed the application.
What do those three ways of closing a form have in common? They all come through the WM_SYSCOMMAND->SC_CLOSE message into your program. I overrode the WM_SYSCOMMAND handler and when it is a SC_CLOSE command I called PostMessage(WM_CLOSE) instead of inherited. This trick solved the crash. So there must be something really weird going on in the WM_SYSCOMMAND message handler that makes Fibers suddenly use the stack of another Fiber.