Why do we need Clang and LLVM?

[lerno]

LLVM and Clang are huge codebases that takes forever to compile whenever you want to play around with C2. That is bad.
LLVM offers excellent optimizations. That is good.

Here is an idea:

- For debug builds, initially use TinyCC (Tiny C Compiler)
- For optimized builds, use LLVM / Clang (and possibly later LLVM IR directly)

TinyCC is... tiny compared to LLVM. It's also very, very fast. TinyCC is also built to be fed C code directly.

[lerno]

On my older laptop LLVM compilation takes about 12 hours. When trying the c2-docker version, it died halfway through due to lack of virtual memory. After running a few hours. LLVM is a resource hog.

[bas]

(see other post about LLVM/Clang).

In the beginning I was hoping on being able to reuse Clang's AST. That was not the case. Still
just using the Tokenizer and Diagnostics engine already saved a lot of time. Additionally most of
C2's parsing is heavily based on/inspired by Clang's parser.

[lerno]

My experience is that lexing is actually the most straightforward part of writing a parser?

[lerno]

One thing would be to localize the DiagnosticsEngine to part of the code and call it through a wrapper. That way it's straightforward to replace it.

Can you explain what you use Clang's preprocessing for? Are we parsing any of standard C so that we need it as input?

[lerno]

For fun I wrapped all the calls of DiagnosticsEngine aside from the config in C2Builder::build.

The only calls outside of C2Builder would be starting a new file, Report, hasErrorOccurred + DiagnosticErrorTrap.

However, it's also passed into the Preprocessor and the SourceManager. From what I understand they are actually both not required in the sense that C2 doesn't have a complex library search, nor any preprocessing (or at least trivial such processing)

If we consider dog fooding C2 eventually, then this is a task that has to be done sooner or later. A self-hosting c2c might rely on LLVM IR and optimization, but not Clang for frontend (unless it's actually used for C parsing – which it isn't)

[lerno]

BTW, at some point a document outlining the coding standard for the C++ part is needed. It's occasionally inconsistent, and because the indentation style differs from what I'm used to I need some sort of reference to work with in order to use auto formatting.

[lerno]

Now that I have a better overview of the code I feel it's even more urgent that the Lexer is swapped for a custom one:

• The lexer parses a lot of stuff that C2 shouldn't, this makes the code rather unclear at places.
• The lexer has a firm idea about how to lex certain tokens, like "@" which might complicate parsing.
• We have full C pre-processing. I don't think this is a good thing.
• Keywords need to be defined in the Clang source
• Code requires reading up on a lot of Clang code to see how things work, rather than just the c2compiler code.

[lerno]

Status update:

1. Relevant parts of Clang lifted out, verified to work stand alone.
2. Now working on cutting away the huge amounts of unnecessary code needed for various versions of C, C++ and different extensions that don't make any sense for C2.

[lerno]

Interestingly, about 90% of the complexity of Preprocessor work comes from resolving modules and macros from modules correctly. So this is why macro resolution should come AFTER parsing and not as a pre-processor

[lerno]

Code in clang's lexer:

Code: [Select]

    Char = getCharAndSize(CurPtr, SizeTmp);
    if (Char == '=') {
      Kind = tok::percentequal;
      CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
    } else if (LangOpts.Digraphs && Char == '>') {
      Kind = tok::r_brace;                             // '%>' -> '}'
      CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
    } else if (LangOpts.Digraphs && Char == ':') {
      CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
      Char = getCharAndSize(CurPtr, SizeTmp);
      if (Char == '%' && getCharAndSize(CurPtr+SizeTmp, SizeTmp2) == ':') {
        Kind = tok::hashhash;                          // '%:%:' -> '##'
        CurPtr = ConsumeChar(ConsumeChar(CurPtr, SizeTmp, Result),
                             SizeTmp2, Result);
      } else {                                         // '%:' -> '#'
        // We parsed a # character.  If this occurs at the start of the line,
        // it's actually the start of a preprocessing directive.  Callback to
        // the preprocessor to handle it.
        // TODO: -fpreprocessed mode??
        if (TokAtPhysicalStartOfLine && !LexingRawMode && !Is_PragmaLexer)
          goto HandleDirective;

        Kind = tok::hash;
      }
    } else {
      Kind = tok::percent;
    }

Indentation is not a sufficient way to make C's if-statements readable.

Makes you want to do something like this:

if-stmt
 : "if" expr "->" stmt
 | "if" expr "{" stmt_list "}" else-clause
 ;

else-clause
 : <empty>
 : "else" "{" smt_list "}"
 : "elseif" expr "{" smt_list "}" else-clause 
 ;

[bas]

Yes, lexical macros (like C) allow everything, but cause so much issues in tooling etc. Once C2 has a
replacement macro system, we can simplify the Lexer a lot.