Today is the second birthday of the preprocessor project. I'm still a bit conflicted, but the best choice for now seems to be to proceed with it on a closed source basis.
A closed-source standalone preprocessor is fairly ludicrous, but for most of the project lifetime the focus has been on the underlying framework. (This is the part which would be of any interest to anyone.) The problem is that it's very promising, and a lot of effort. If I'm working very hard on something which should become very valuable, it's not really useful to work even harder to properly document and support it so others can realize that value for free.
Hopefully this can be turned back around at some point. The framework now consists of several independent modules:
stage.h — Links templates into a modular data-processing pipeline.
construct.h — Provides a uniform interface for mapping between source and target code.
parameter.h — Represents metadata regarding format and desired semantics, i.e. compiler options.
The last one has taken almost a month to get fully working, because those options have to propagate to the stages that need them, and stages can be combined and instantiated in various ways. Applied uniformly, it should decouple the implementation of the stages from the output types the user desires. The whole compiler may be fully templated, like a generic algorithm, without needing explicit template arguments to be passed in or global typedefs.
A month spent on purely abstract templates that don't "do" anything is crazy-making. Not starting with a comprehensive theory, I tried to shoot for just getting the right effects. It fell short semantically, resulting in infinite template recursion when applied to a (non-infinitely) recursive stage. So I ended up rewriting almost all of it.
Still to do are more modules: for memory allocation (previously drafted as a string class), small-scale decisions, and serialization. These should be more instrumental in the creation of a faster, more flexible compiler.
The stage and construct libraries are just a couple hundred lines each, and should be instructive and useful to others by themselves. As for the parameter library, I'm not sure if it's too advanced and special-purpose to serve the open-source community. The future libraries look like they will sit squarely in trade-secret territory.
Hopefully there will be an update here before long, with some kind of online demo where the preprocessor colors your code interactively. I need to focus on functionality rather than build a community. But I do support open source, and open-source updates will return.