Warnings and crash when using the new C frontend with joern-scan on VLC about codepropertygraph

[johannescoetzee]

I reran the test using the codepropertygraph changes from #1360, and that seems to have addressed the third warning I mentioned in the issue, i.e.

2021-08-18 09:00:54.328 WARN Linker: Invalid AST_PARENT_TYPE=Optional[<empty>]; astChild LABEL=METHOD; astChild FULL_NAME=Optional[VLCVideo.VLCVideo.anonymous_lambda_0]
2021-08-18 09:00:54.328 WARN Linker: Could not create edge. Source lookup failed. edgeType=AST, srcNodeType=<empty>, srcFullName=<empty>, dstNodeType=METHOD, dstNodeId=1005246

The rest of the warnings are still there, as well as the eventual crash. The number of findings also did not change.

[ursachec]

@max-leuthaeuser looks like this one's for you

[max-leuthaeuser]

I am already on it.

[johannescoetzee]

Running ./joern-scan --language newc ../../test_repos/vlc --tags <TAG> for any specific tag doesn't result in the same crash, but ./joern-scan --language newc ../../test_repos/vlc --tags all does. The crash is probably due to a specific query being broken

[johannescoetzee]

./joern-scan --language newc ../../test_repos/vlc --tags default,badfn,metrics,integers,uaf,setxid,badimpl,posix,race-condition,sql-injection,strings,xss | wc
  19669  160657 2582127

Only running queries with tags gives ~20k findings, so much closer to fuzzyc.

[max-leuthaeuser]

The issues within the MemberAccessLinker and AccessPathUsage are also known to me. They are due to missing of a proper CallExpression handling (this argument, call receiver, etc.). Its already on my TODO list.

[johannescoetzee]

The crash seems to be triggered by the simple-constant-detection query:

({cpg.assignment
  .groupBy(_.argument.order(1).code.l)
  .flatMap {
    case (_: List[String], as: Traversal[Assignment]) => Option(as.l)
    case _                                            => Option.empty
  }
  .filter(_.size == 1)
  .flatMap {
    case as: List[Assignment] =>
      Option(as.head.argument.head, as.head.argument.l.head.typ.l)
    case _ => Option.empty
  }
  .filter {
    case (_: Identifier, ts: List[Type]) =>
      ts.nonEmpty &&
        ts.head.namespace.l.exists { x =>
          x.name.contains("<global>")
        } &&
        !ts.head.fullName.contains("[]")
    case _ => false
  }
  .flatMap {
    case (i: Identifier, _: List[Type]) => Option(i)
    case _                              => Option.empty
  }}).l

[johannescoetzee]

Details about an issue that explains some of the discrepancies between findings using the new and old frontends can be found here: joernio/joern#577. It looks like, when using the new frontend, joern-scan incorrectly reports twice for functions defined in header files. I opened that issue in the joern repo, since I'm not sure where the fix for it will happen.

[max-leuthaeuser]

Running c2cpg on VLC does not crash anymore and the query in question also runs fine.
Closing this.
(other warnings, e.g., header files missing and unsupported AST nodes are tackled separately)