State of progress?

[lerno]

Is C2 really alive or is it put in suspended animation?

[bas]

C2 is definitely not going as fast as I would like, but it's definitely not dead either. Work and family life does
take a big toll on available time. Luckily it's already quite usable, so people (me included) can play around
with it to get a feel and evaluate it for parts that could be improved.

Parts where C2 still needs developers:
- LLVM IR  generation
- inline assembly support
- some tooling (syntax highlighting, style formatting)

[lerno]

How is compilation speed at this point in time?

[bas]

Since rebasing on Clang 7.0 it has gone down a bit. parsing + analysing is now at roughly 1.2 million lines of code per second
on my laptop. The backend that does code-generation is also quite fast (roughly takes twice the time of parse+analyse). But the
C-generation backend will be replaced by an LLVM IR code generation backend at some point.

[lerno]

Can you explain the current pipeline? How does it depend on clang and llvm?

[bas]

The documentation describes this a bit in http://c2lang.org/site/development/internals/. It comes down to this:

Clang:
- Tokenizer (includes preprocessor)
- Diagnostics Engine
- Filemanagement (needed for Tokenizer + Diagnostics)

LLVM:
- Utils used by Clang components above.

It would have been fantastic to use the AST from Clang, but the AST is really tuned to single pass parsing+analysis, so really didn't fit..

[lerno]

Why is the lexer and preprocessor + diagnostics engine needed from Clang? Can't C2 do that on its own?

[bas]

Of course we could write that ourselves and maybe we will in the end. But as a start, this was a faster path.
Especially the DiagnosticsEngine does quite a lot..

[lerno]

I've looked at the lexer. I would be interested in writing a C-style lexer that more easily could be turned into C2 later on.

Do we need multiple active lexers?

[bas]

Dropping the Clang lexer would also effectively remove the more complex #ifdef feature support (unless the
new lexer is also quite complex). But it would make the lexer itself somewhat faster I think. Clang's lexer needs
to support multiple languages and tokensets. The C2 parser would not. Additionally I've noticed that Clang's
Lexer seems to have slowed down in Clang 7. I haven't looked into it in more detail, but measurements are
definitely slower. I've written plenty of lexers, so I estimate it will take around 2 days to write a C2 replacement..

[lerno]

I was thinking that maybe we could write it in C2, then convert that C2 code to C and then move the generated C into the normal source dir.

I worked on a (C style) lexer last night that could be turned into C2 code.

[lerno]

I imagine something along these lines: https://github.com/lerno/c2compiler/blob/lexer_example/c2c/Parser/Lexer.cpp

I realize that this one was very much like your TOML lexer.

[bas]

Lexers are always fun to write (I think). The Clang Lexer can do lookahead() a few tokens and the C2Parser does rely
on that feature. Otherwise I think it's important to put all TokenKinds in an enum, so the compiler can warn if you miss
some in a switch statement.

For preprocessing, I think we (for now) need at least '#if 0/1 .. #endif' and maybe a basic '#ifdef FEATURE1 .. #endif'.
Otherwise we would need to think about another way to offer those features..

[lerno]

Right now I'm working on actually pulling the Clang lexer into a flat directory (c2c/Clang) to see how many things are necessary. Needless to say, trying to pull it out brings with it CRAZY AMOUNTS OF CODE. I'm then going to try to cut away what we don't use. If we get some subset of less than 50 files, we could actually move those into C2 and then refactor it down to the bare essentials, swapping things out as we go along.

I'm going to give it a few more hours and see where it leads.