Some questions about cpp parser and preprocess about astminer

[egor-bogomolov]

Hello @jumormt and sorry for the late reply!

1. Parsing C++ with an ANTLR parser is a very complex task (see here), even when the file was transformed by the preprocessor. We initially tried to use a parser built from an ANTLR grammar in our other projects, but faced problems. To give you an example, when ANTLR C++ grammar parses a declaration of vector<vector<int>> and vector< vector<int> >, it gives different results, because in one case (the latter one if I remember correctly) it treats > as a comparison operator (which is weird I would say). After trying to fix the grammar manually, we found more and more new cases and finally switched to another parser. While the parser we use is indeed built on top of the ANTLR4 grammar, it does a lot of post-processing, and from what we see, provides way more robust results.
2. Regarding your second question, we run preprocessing only for individual files and do not consider complex includes and dependencies. Thus, we use g++ only as a preprocessor, without running linking/compilation. The goal of it is to substitute #define directives that are present in the same file to increase our chances of successful parsing. We omit includes during preprocessing on purpose, because if we actually substitute them, files might become too large (e.g., if the developer included a huge part of the STL) and we will analyze some included libraries instead of the actual developer-written code. As a downside of omitting includes, we might miss some defines from the header files, but I don't see an easy way to fix it without unfolding #include statements.

Hope that it answers your questions.

[jumormt]

Hi,
Thanks a lot for your patient answer! The parser you recommended can indeed yield more robust results. However, it is relatively hard to use because its documents are limited. As per the second question, I think it is an interesting topic to do more transformation for the source code before parsing or feeding into our neural network.