missing antlr4-4.2.1-SNAPSHOT-complete.jar for the genParser.sh about joern

[gengyifeng]

It can be got from https://github.com/sharwell/antlr4/

[fabsx00]

Just a comment: http://joern.readthedocs.org/en/latest/development.html could be interesting for you.

[gengyifeng]

Hi, Fabian,

I have built the required jar successfully. Thank you very much.

I notice that the grammar of joern for the C/C++ prepossessing is quite
limited, especially for the nesting #ifdef #elif and #endif mingling with
common statement blocks. Current generated ast is not correct for some of
my testcases compiled with gcc. Do you think it is an important problem to
fix?

Best,
Yifeng

On Wed, Jul 30, 2014 at 5:06 PM, Fabian Yamaguchi [email protected]
wrote:

Just a comment: http://joern.readthedocs.org/en/latest/development.html
could be interesting for you.

—
Reply to this email directly or view it on GitHub
#31 (comment).

耿益锋
Yifeng Geng

ASSOCIATE RESEARCHER
MICROSOFT CHINA CO LTD

[fabsx00]

Hi Yifeng,

yes, we have actually just recently discussed this problem in our group: currently, parsing of pre-processor directives mostly comes down to discarding the directives. I think the best solution would actually be to create a "fuzzy pre-processor" that runs before calling the actual parser. This would allow us to do more interesting things as well, such as creating several functions from a single function by taking into account possible combinations of #ifdefs. If you are considering to approach this problem, let me know and I can give you some more details on what we discussed.

Best,
Fabian

[gengyifeng]

Hi Fabian,

I also agree that it is better to separate the pre-processor and the
parser. For the pre-processor, a simple solution is to use a parameter list
with assigned values to get a code path. But this loses some portion of the
code especially when the number of the parameter is large. Also it becomes
impossible to enumerate all the paths. Another way is to rewrite the #if to
if clause to keep most of the code in the AST. But the rewrite rule is
complicated and it requires code copy in some cases. What's your suggestion?

BTW, try-catch-finally clause is not treated the same as if-else clause in
joern, is it intended?

Thanks,
Yifeng

On Thu, Jul 31, 2014 at 6:08 PM, Fabian Yamaguchi [email protected]
wrote:

Hi Yifeng,

yes, we have actually just recently discussed this problem in our group:
currently, parsing of pre-processor directives mostly comes down to
discarding the directives. I think the best solution would actually be to
create a "fuzzy pre-processor" that runs before calling the actual
parser. This would allow us to do more interesting things as well, such as
creating several functions from a single function by taking into account
possible combinations of #ifdefs. If you are considering to approach this
problem, let me know and I can give you some more details on what we
discussed.

Best,
Fabian

—
Reply to this email directly or view it on GitHub
#31 (comment).

耿益锋
Yifeng Geng

ASSOCIATE RESEARCHER
MICROSOFT CHINA CO LTD

[fabsx00]

Hi Yifeng,

so, my suggestion would be to generate two functions for each #ifdef, one for the case where the condition evaluates to true, another for the case where it evaluates to false. That way, no information is lost and the code can be analyzed for different build settings. The line containing the pre-processor instruction could then be replaced by a comment indicating whether the 'true' or 'false' case was assumed.

Concerning try-catch-finally: can you provide a test-case? Might be a bug.

Best,
Fabian

[gengyifeng]

Hi Fabian,

Your suggestion for the pre-processor is quite reasonable.

For the issue of try-catch issue, in the AST of joern, "else" is treated as
a child of "if", but "catch" is not the child of "try". In another words,
"try" and "catch" are at the same level and treated as two independent
statements. In the source code, "if" and "else" block are connected with
NestingReconstructor. But nothing is done for "try" and "catch". I think
"try" and "catch" block should be connected as well. But this may not be
called a bug.

Thanks,
Yifeng

On Fri, Aug 1, 2014 at 1:52 AM, Fabian Yamaguchi [email protected]
wrote:

Hi Yifeng,

so, my suggestion would be to generate two functions for each #ifdef, one
for the case where the condition evaluates to true, another for the case
where it evaluates to false. That way, no information is lost and the code
can be analyzed for different build settings. The line containing the
pre-processor instruction could then be replaced by a comment indicating
whether the 'true' or 'false' case was assumed.

Concerning try-catch-finally: can you provide a test-case? Might be a bug.

Best,
Fabian

—
Reply to this email directly or view it on GitHub
#31 (comment).

耿益锋
Yifeng Geng

ASSOCIATE RESEARCHER
MICROSOFT CHINA CO LTD

[pgh15]

I'd like to make a small change to the parser and have tried to follow the above instructions without any luck. The current version of antlr4 doesn't build with maven but has it's own python build script which didn't work. So I downloaded the precompiled jars and tried all of them in place of antlr4-4.2.1-SNAPSHOT-complete.jar hoping that at least antlr-4.2.2-complete.jar would be compatible. Without making any changes to any g4 file, genparsers.sh produces something that no longer builds successfully with ant. Is there any chance I can get a pre-built antlr4-4.2.1-SNAPSHOT-complete.jar from somewhere that definitely works?

Thanks,
Paul

[fabsx00]

Hi Paul,
here you go: http://mlsec.org/joern/lib/antlr4-4.2.1-SNAPSHOT-complete.jar
Hope this helps.
Fabian

[pgh15]

Hi Fabian,
Perfect. Thanks.

Paul