CLANG, template debuginfo and linker out of memory

The CLANG compiler that is used for BCC64 in XE3, XE4 and XE5 has a “bloated object file” problem if it comes to debug information for deeply nested templates. The generated object file can easily grow to 100 MB with a single line of code.

int main(int argc, char *argv[])
{
    std::map<TEnumA, std::map<TEnumB, std::map<TEnumC, std::map<TEnumD, std::map<TEnumE, std::map<TEnumF, std::map<TEnumG,
      std::map<TEnumH, std::map<TEnumI, std::map<TEnumJ, std::map<TEnumK, std::map<TEnumL, std::map<TEnumM, char* > > > > > > > > > > > > > Dummy;

    Dummy[TEnumA::TEnumAVal1][TEnumB::TEnumBVal1][TEnumC::TEnumCVal1][TEnumD::TEnumDVal1][TEnumE::TEnumEVal1][TEnumF::TEnumFVal1]
      [TEnumG::TEnumGVal1][TEnumH::TEnumHVal1][TEnumI::TEnumIVal1][TEnumJ::TEnumJVal1][TEnumK::TEnumKVal1][TEnumL::TEnumLVal1]
      [TEnumM::TEnumMVal1] = "Booooom!";

    return 0;
}

The problem comes from creating a debug information string for template arguments in the method TemplateSpecializationType::PrintTemplateArgumentList in clang/lib/AST/TypePrinter.cpp. If you have deeply nested templates like the one above, all the code and declarations that std::map brings with it, creates really long debug information strings (600,000 chars and more). And with CLANG’s ability to be helpful in finding template errors, it also records the original template name of a typedef’ed template parameters and creates a debug information string that containing the original template name everywhere where the typedef is used.

The code above creates so many huge debug information strings that the object file becomes over 100 MB large. Depending on the number of such template instantiations in your code, the file can easily become larger than the available RAM. But that isn’t necessary to get the out of memory. The linker (ilink64) will run out of memory even with this “little” 100 MB object file.

The easiest but unpractical solution is to disable debug information.

With the next IDE Fix Pack there is another solution. IDE Fix Pack hooks into the compclang.dll and shortens the strings. If the number of chars exceeds 2048, every further template parameter will be replaced by “…”. That makes those strings not only much smaller but the compiler will also generate the debug information much much faster because there is less recursion and less string concatenation.

I can only guess why the linker runs out of memory with a 100 MB object file. I assume that it does some unfortunate memory allocations for the debug information. But after finding a solution to shrink the debug information in the object file, I didn’t need to look into the linker.