kythe / kythe

Kythe is a pluggable, (mostly) language-agnostic ecosystem for building tools that work with code.

Kythe

Features

• Extensive documentation of the Kythe schema

• Indexer implementations for C++, Go, and Java

• Compilation extractors for javac, Maven, cmake, Go, and Bazel

• Generic verifier for indexers

• Sample cross-reference service

• Many useful utility commands to work with Kythe artifacts

Getting Started

Download the latest Kythe release from https://github.com/kythe/kythe/releases and then unpack it for a snapshot of Kythe’s toolset.

tar xzf kythe-v*.tar.gz
rm -rf /opt/kythe
mv kythe-v*/ /opt/kythe

See /opt/kythe/README for a complete description of the packaged tools and their usages.

Contributing and Building Kythe

https://kythe.io/contributing

Issue Tracker

https://github.com/kythe/kythe/issues

Documentation

https://kythe.io/docs/

Examples

https://kythe.io/examples

Contact

• https://groups.google.com/forum/#!forum/kythe - General Kythe Community

• https://groups.google.com/forum/#!forum/kythe-dev - Kythe Development

kythe's Issues

Windows support [phab:T4]

Created by schroederc at 2015-01-22 21:56:40:

campfire: ensure rules work with POSIX shells [phab:D5]

Created by schroederc at 2015-01-28 20:16:29:

Ninja hardcodes '/bin/sh' as the shell it uses for all rule commands. On many systems this may be a symlink to /bin/bash, but it can't be guaranteed. This change ensures that all ninja rules work with the dash, a minimalistic POSIX shell, and fixes #189.

Update contribution documentation [phab:D1]

Created by schroederc at 2015-01-27 02:34:39:

This change updates http://kythe.io/contributing to include a passage
about landing changes and syncs it with README.adoc.

Releases do not document what OS the release was for

I'm looking to get kythe to run on my macOS machine, but it looks like the releases are not compatible with that and I need to build from source. It would be helpful if the release specified which OS it was built for.

Fix ui start command [phab:D10]

Created by tsl0922 at 2015-01-30 03:30:19:

Mac OS X support [phab:T3]

Created by schroederc at 2015-01-22 21:56:13:

Remove main repo dependencies on LLVM

Right now the main Kythe repo depends on LLVM in ways other than just the C++ indexer itself. It's fine for the C++ indexer to depend on LLVM, but the rest of the main repository shouldn't.

As far as I know the only reason we depend on it is for some library components, and we may be able to get those from Abseil instead.

Document how to add Kythe support for a new language

There are various parts of this, some of which have existing documentation. Link these together in a survey document that describes, among other things:

1. How to obtain compilation records for the language,
2. How to write an indexer for the language,
3. How to propose schema extensions (and when they may be appropriate).

This may wind up being more than one document, but ideally there should be one fairly clean and informal summary that points to the others.

Release explore.proto APIs

Document the supported formats for cross-language linkage metadata

http://www.kythe.io/docs/schema/indexing-protobuf.html talks about embedding metadata in separate files, but we also allow a couple of new formats too (cf. #70 for example). We should document these in a more public fashion so that users will have a way to decide which to use.

Inner classes should be children of enclosing classes [phab:T8]

Created by zarko at 2015-01-28 19:06:07:

We should add the childof edge required by the last goal in:

template <typename T>
//- C defines AbsC
//- ClassC childof AbsC
class C {
  //- R defines EnumR
  //- R childof ClassC
  enum R : T {
  };
};

When encoding entries to triples, use the correct escaping rules. [phab:D8]

Created by fromberger at 2015-01-29 18:37:20:

The string encoding used for RDF triples isn't quite the same as the Go
quoted-string rules, in particular it doesn't allow the \xHH syntax for literal
bytes. This change uses the correct encoding, and allows a workaround for
files that have Unicode characters that aren't in UTF-8 format.

Fixes #195.

Presubmit hook should also build affected binaries

Our presubmit should stop you from checking in a change that breaks a binary but not a test.

Remove the link to kythe-early-interest, as it is now unnecessary. [phab:D3]

Created by fromberger at 2015-01-27 20:46:47:

Release Kythe [phab:T1]

Created by schroederc at 2015-01-22 21:54:04:

Improve instructions for how to contribute to Kythe

We need a good summary of how to get started contributing to Kythe. This is much simpler now that we are using pull requests instead of Phabricator, but there is probably still some old documentation to update.

extractor.go should be able to work without copying

Right now the behavior of extractor.go necessitates copying the source repo before doing extraction. It does this primarily because we muck with stuff, e.g. pom.xml to expose our extractor code in the javac run.

Ideally we would be able to not do this - either figure out how to invoke the builds done inside extractor.go with an in-memory modified copy, or possibly by restoring any modifications made upon exit (though that's a bit dangerous).

Likely doing away with copies could yield not-insignificant performance benefit.

staticanalysis smoke test harness should have hard timeout

Right now there's a few test repos we're using that hang indefinitely. We should have a timeout before we automatically fail and move on, since the primary purpose here is breadth of repos.

kythe.io: add analytics for launch [phab:D2]

Created by schroederc at 2015-01-27 16:51:52:

Add RDF encoding package. [phab:D12]

Created by fromberger at 2015-01-30 18:13:03:

This change separates the encoding of RDF triples into its own package so we
can have proper tests. The triples tool now uses this package.

Supersedes #199.

Export the schema overview examples

The full schema is primarily a reference document, and people new to the project find it difficult to get oriented. Clean up and export the "overview" document we prototyped, that illustrates the various components of the schema in common use.

Fix a mistaken installation instruction. [phab:D9]

Created by fromberger at 2015-01-29 19:05:51:

Python indexer [phab:T6]

Created by schroederc at 2015-01-22 21:57:24:

Implement Beam Post-Processor

Move Go generated source to a more go-tool friendly output path. [phab:D4]

Created by fromberger at 2015-01-27 22:52:17:

The go tool likes to find source and built packages in a particular directory
structure relative to its import paths. This change modifies Campfire to
respect that for generated code (specifically, protobuf outputs).

Various unstructured grumblings about undocumented flag syntax will be left as
an exercise for the reader.

triples.go does not correctly encode UTF-8 for RDF [phab:T10]

Created by fromberger at 2015-01-29 17:07:23:

Per http://www.w3.org/TR/n-quads/#sec-grammar, RDF triples should use the escaping rules from http://www.w3.org/TR/2014/REC-turtle-20140225/#sec-escapes. Right now, the tool is using the standard Go escaping rules, which use \x notation, and (justifiably) Cayley doesn't like that.

Serving Representation [phab:T9]

Created by schroederc at 2015-01-29 15:55:09:

Surfacing an xrefs service based directly on a GraphStore is very low-performance. Kythe needs a separate, faster representation of its data for serving.

aliases/root stops at nested template alias rather than real root

In the following example, the aliases/root edges should point to CharBuiltin, rather than AliaserType:

template <typename Named>
struct Aliaser {
  //- @Type defines/binding AliaserType
  //- @Type defines/binding InnerAliaserType
  //- @Named ref NamedVar
  using Type = Named;
};

//- @Root defines/binding RootAlias
//- RootAlias aliases/root CharBuiltin
//- CharBuiltin.node/kind tbuiltin
using Root = char;

//- @Type ref AliaserType
//- @First defines/binding FirstAlias
//- FirstAlias aliases/root AliaserType
//- !{ FirstAlias aliases/root CharBuiltin }
using First = typename Aliaser<Root>::Type;
//- @Type ref AliaserType
//- @Second defines/binding SecondAlias
//- SecondAlias aliases/root AliaserType
//- !{ SecondAlias aliases FirstAlias }
//- !{ SecondAlias aliases CharBuiltin }
using Second = typename Aliaser<First>::Type;
//- @Type ref AliaserType
//- @Third defines/binding ThirdAlias
//- ThirdAlias aliases/root AliaserType
//- !{ ThirdAlias aliases SecondAlias }
//- !{ ThirdAlias aliases FirstAlias }
//- !{ ThirdAlias aliases CharBuiltin }
using Third = typename Aliaser<Second>::Type;

specifically, the negated groups should succeed, but they currently do not.

@zrlk it seems like you may have a fix for this?

Docker in Docker - probably harmful?

I'll preface this by saying I have precious little experience with Docker, so I might be missing things / misstating things / etc. Mostly just writing this down because it's taking me longer than I was expecting to resolve.

We're currently looking for a clean way to package up extractrepo for deployment. An obvious choice there is docker. All well and good, except, oh wait, extractrepo itself builds and runs docker images:

$ bazel build kythe/go/platform/tools/extraction/extractrepo:extractrepo
$ cp ^ that thing -> some directory with a Dockerfile that just copies in the binary and runs it, along with any deps it needs (git commandline, kythe extraction scripts, etc)
$ docker build --tag test-extract aforementioned-directory

^ The above 3 steps can be replaced with a docker file that handles pulling in kythe, running tools/modules/update, building the extractrepo binary, etc. But that (a) takes a long time, and (b) results in a much larger binary.

Anyways that's not the difficulty, the problem is then:

$ docker run -i -t test-extract -local ~/some/repo ~/some/output/dir
... exec: "docker": executable file not found in $PATH

There's various posts on how to do Docker in Docker, indicating this is technically possible to do:

https://github.com/jpetazzo/dind

But apparently it's not a great idea for all situations:
https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
https://stackoverflow.com/questions/27879713/is-it-ok-to-run-docker-from-inside-docker

Particularly damning is that link from jpetazzo, since they're the one who came up with docker-in-docker in the first place.

I suppose one other alternative might be docker API?

https://godoc.org/github.com/docker/docker/client

Remove unused xrefs appengine module. [phab:D15]

Created by schroederc at 2015-01-30 19:09:54:

This module isn't currently used and is holding up some changes to use
the new xrefs serving representation. I plan to bring this module back
in a simpler form (without the schema) once the new server is working.

Fix a couple more misquoted values in RDF conversion. [phab:D11]

Created by fromberger at 2015-01-30 17:07:07:

Proto Indexer [phab:T2]

Created by schroederc at 2015-01-22 21:55:43:

Javascript Indexer [phab:T5]

Created by schroederc at 2015-01-22 21:57:03:

'./campfire build' gets sh errors [phab:T7]

Created by lahosken at 2015-01-28 03:26:31:

Naively pasting the ./campfire build from the README like the trusting fellow I am, I saw some errors.

$ ./campfire build
[6/302] protoc_cpp //kythe/proto:storage_proto
FAILED: mkdir -p campfire-out/gen/kythe/proto/storage_proto/cxx && third_party/protobuf/bin/protoc --cpp_out=campfire-out/gen/kythe/proto/storage_proto/cxx kythe/proto/storage.proto && for i in $(find campfire-out/gen/kythe/proto/storage_proto/cxx -name '*.cc'); do clang++ -c -I campfire-out/gen/kythe/proto/storage_proto/cxx $(sed -r 's/( |^)([^$])/ -I\2/g' <<<'third_party/protobuf/include') $i -o $i.o; done && ar cr campfire-out/bin/kythe/proto/storage_proto.a $(find campfire-out/gen/kythe/proto/storage_proto/cxx -name '*.o')
/bin/sh: 1: Syntax error: redirection unexpected
[6/302] protoc_go //kythe/proto:storage_proto
FAILED: mkdir -p campfire-out/gen/kythe/proto/storage_proto/go && third_party/protobuf/bin/protoc --plugin=third_party/protobuf/bin/protoc-gen-go --go_out=import_path=kythe/proto/storage_proto,Mkythe/proto/storage.proto=kythe/proto/storage_proto:campfire-out/gen/kythe/proto/storage_proto/go kythe/proto/storage.proto && go tool 6g -p kythe/proto/storage_proto -complete -pack -o campfire-out/go/pkg/linux_amd64/kythe/proto/storage_proto.a -I $(sed 's/ / -I /g' <<<'campfire-out/gen/kythe/proto/storage_proto/go campfire-out/go/pkg/linux_amd64/') $(find campfire-out/gen/kythe/proto/storage_proto/go -name '*.go')
/bin/sh: 1: Syntax error: redirection unexpected
[6/302] protoc_go //third_party/protobuf:descriptor_proto
FAILED: mkdir -p campfire-out/gen/third_party/protobuf/descriptor_proto/go && third_party/protobuf/bin/protoc --plugin=third_party/protobuf/bin/protoc-gen-go --go_out=import_path=third_party/protobuf/descriptor_proto,Mthird_party/protobuf/include/google/protobuf/descriptor.proto=third_party/protobuf/descriptor_proto:campfire-out/gen/third_party/protobuf/descriptor_proto/go third_party/protobuf/include/google/protobuf/descriptor.proto && go tool 6g -p third_party/protobuf/descriptor_proto -complete -pack -o campfire-out/go/pkg/linux_amd64/third_party/protobuf/descriptor_proto.a -I $(sed 's/ / -I /g' <<<'campfire-out/gen/third_party/protobuf/descriptor_proto/go campfire-out/go/pkg/linux_amd64/') $(find campfire-out/gen/third_party/protobuf/descriptor_proto/go -name '*.go')
/bin/sh: 1: Syntax error: redirection unexpected
[6/302] protoc_java //kythe/proto:storage_proto
FAILED: set -o pipefail; mkdir -p campfire-out/bin/kythe/proto/storage_proto.jar.{srcs,classes} && 'third_party/protobuf/bin/protoc' --java_out=campfire-out/bin/kythe/proto/storage_proto.jar.srcs kythe/proto/storage.proto && find campfire-out/bin/kythe/proto/storage_proto.jar.srcs -name "*.java" | xargs 'javac' -cp 'third_party/protobuf/protobuf-java-2.5.0.jar' -d campfire-out/bin/kythe/proto/storage_proto.jar.classes && 'jar' cf campfire-out/bin/kythe/proto/storage_proto.jar -C campfire-out/bin/kythe/proto/storage_proto.jar.classes .
/bin/sh: 1: set: Illegal option -o pipefail
[6/302] cpp_compile third_party/leveldb/db/db_iter.cc
ninja: build stopped: subcommand failed.
desktop:kythe (master)$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

I'm no shell-programming guru, but when I google teh internets, I think set -o pipefail is a bash thing, so it might make sense that /bin/sh would be confused by it. Mmmmaybe that means that something should be running bash instead of running sh. Or mmmaybe that means that it should be running sh, but shouldn't be using bash-isms. Anyhow, I point it out since other README-followers might bump into it. (Maybe the README shouldn't encourage me to try to build everything, but instead nudge me towards targets? anyhow...)

Avoid making assumptions about protobuf's string. [phab:D6]

Created by zarko at 2015-01-28 21:17:10:

Generally, we should not assume that *_mutable()
return pointers to ::std::string.

Replace delimited stream with more robust format

Riegeli is the leading contender, but we would need to port it.

Maven extraction should consider removing dependency on git binary

Right now we just shell out to git commandline in a few spots (depending on the operation). It'd be cleaner if we just use go-git library. There's proof of concept for this in e.g. b2677ed

Short term this just means any packaging we do needs to remember to copy in git commandline as well.

Ensure the examples on the Kythe web site work correctly

The examples for how to build and run various Kythe tools should be verified to work. Those that do not should either be fixed & updated, or removed.

staticanalysis test harness should support indexing

In addition to testing extraction on repos, we should test some actual analysis. Indexing is a natural fit here, since the indexer lives in the same repo. Obviously for future system it'd include arbitrary other analyzers, but this can be a good starting point.

Test failure: //third_party/beam:gbk_test

https://buildbot-dot-kythe-repo.appspot.com/#/builders/1/builds/3

--- FAIL: TestCombinePerKey (0.00s)
	gbk_test.go:92: panic: value 9 expected, but not present goroutine 8 [running]:

Fix some documentation errors and omissions [phab:D13]

Created by schroederc at 2015-01-30 18:21:39:

Release Typescript indexer

Implement kzip reader and writer support in Java

As part of migrating away from .kindex and index pack formats to .kzip, we will need a way for the Java extractor to generate .kzip output, and for the Java indexer to read it (to keep the verifier tests working).

JsonParseException when indexing an indexpack with JavaIndexer

Hi,

I build kythe from ebce651 and created some .kindex files from a project consisting of multiple java compilation units. Then I created an indexpack from these files and tried to index them with:

java -Xbootclasspath/p:kythe/indexers/java_indexer.jar com.google.devtools.kythe.analyzers.java.JavaIndexer --index_pack=./pack <unit-hash> > entries.proto

This fails with a JsonParseException:

Exception in thread "main" com.google.gson.JsonParseException: com.google.protobuf.InvalidProtocolBufferException: Missing type url when parsing: {"type_url":"kythe.io/proto/kythe.proto.JavaDetails","value":"<encoded>"}
	at com.google.devtools.kythe.util.JsonUtil$GeneratedMessageV3TypeAdapter.deserialize(JsonUtil.java:95)
	at com.google.devtools.kythe.util.JsonUtil$GeneratedMessageV3TypeAdapter.deserialize(JsonUtil.java:64)
	at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
	at com.google.gson.Gson.fromJson(Gson.java:888)
	at com.google.gson.Gson.fromJson(Gson.java:953)
	at com.google.gson.Gson.fromJson(Gson.java:926)
	at com.google.devtools.kythe.platform.indexpack.Archive.readUnit(Archive.java:166)
	at com.google.devtools.kythe.platform.indexpack.Archive.readUnit(Archive.java:149)
	at com.google.devtools.kythe.platform.indexpack.Archive.readDescription(Archive.java:144)
	at com.google.devtools.kythe.analyzers.java.JavaIndexer.main(JavaIndexer.java:90)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Missing type url when parsing: {"type_url":"kythe.io/proto/kythe.proto.JavaDetails","value":"<encoded>"}
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeAny(JsonFormat.java:1342)
	at com.google.protobuf.util.JsonFormat$ParserImpl.access$2000(JsonFormat.java:1116)
	at com.google.protobuf.util.JsonFormat$ParserImpl$1.merge(JsonFormat.java:1181)
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1270)
	at com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:1784)
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeRepeatedField(JsonFormat.java:1523)
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1481)
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1315)
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1273)
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1155)
	at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:338)
	at com.google.devtools.kythe.util.JsonUtil$GeneratedMessageV3TypeAdapter.deserialize(JsonUtil.java:89)
	... 9 more

After some research in gunzipped the unit file, replaced type_url with @type and gzipped it again.

This leaves me with:

Exception in thread "main" com.google.gson.JsonParseException: com.google.protobuf.InvalidProtocolBufferException: Cannot resolve type: kythe.io/proto/com.google.devtools.kythe.proto.JavaDetails
	at com.google.devtools.kythe.util.JsonUtil$GeneratedMessageV3TypeAdapter.deserialize(JsonUtil.java:95)
	at com.google.devtools.kythe.util.JsonUtil$GeneratedMessageV3TypeAdapter.deserialize(JsonUtil.java:64)
	at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
	at com.google.gson.Gson.fromJson(Gson.java:888)
	at com.google.gson.Gson.fromJson(Gson.java:953)
	at com.google.gson.Gson.fromJson(Gson.java:926)
	at com.google.devtools.kythe.platform.indexpack.Archive.readUnit(Archive.java:166)
	at com.google.devtools.kythe.platform.indexpack.Archive.readUnit(Archive.java:149)
	at com.google.devtools.kythe.platform.indexpack.Archive.readDescription(Archive.java:144)
	at com.google.devtools.kythe.analyzers.java.JavaIndexer.main(JavaIndexer.java:90)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Cannot resolve type: kythe.io/proto/com.google.devtools.kythe.proto.JavaDetails
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeAny(JsonFormat.java:1347)
	at com.google.protobuf.util.JsonFormat$ParserImpl.access$2000(JsonFormat.java:1116)
	at com.google.protobuf.util.JsonFormat$ParserImpl$1.merge(JsonFormat.java:1181)
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1270)
	at com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:1784)
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeRepeatedField(JsonFormat.java:1523)
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1481)
	at com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1315)
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1273)
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1155)
	at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:338)
	at com.google.devtools.kythe.util.JsonUtil$GeneratedMessageV3TypeAdapter.deserialize(JsonUtil.java:89)
	... 9 more

I am not sure what to do now. Could you point me towards a solution?

Thanks and have a nice day,
Carlo

Implement kzip reader and writer support in C++

As part of migrating away from .kindex and index pack formats to .kzip, we will need a way for the C++ extractor to generate .kzip output, and for the C++ indexer to read it (to keep the verifier tests working).

Automate the generation of release notes

In previous releases, we have generally assembled the release notes by manually trawling the commit logs. While a certain amount of manual vetting is probably OK, most of the process of adding release notes should be automated.

Good commit-log hygiene might help with this, and/or we may want to consider using an explicit tagging mechanism for what gets counted as "interesting" for release purposes.

Support Bazel v0.14.0

Build failure when using Bazel v0.14.0:

Traceback (most recent call last):
        File "/usr/local/google/home/schroederc/kythe/kythe/cxx/extractor/testdata/BUILD", line 238
                cc_extract_kindex(name = 'extract_verify_std_string_test_kindex')
        File "/usr/local/google/home/schroederc/kythe/tools/build_rules/verifier_test/cc_indexer_test.bzl", line 102, in _cc_extract_kindex_impl
                depset([extract(ctx = ctx, kindex = get...])
        File "/usr/local/google/home/schroederc/kythe/tools/build_rules/verifier_test/cc_indexer_test.bzl", line 103, in depset
                extract(ctx = ctx, kindex = getattr(ctx.ou...), <5 more arguments>)
        File "/usr/local/google/home/schroederc/kythe/tools/build_rules/verifier_test/cc_indexer_test.bzl", line 109, in extract
                _compiler_options(cpp)
        File "/usr/local/google/home/schroederc/kythe/tools/build_rules/verifier_test/cc_indexer_test.bzl", line 57, in _compiler_options
                cpp.compiler_options([])
struct has no method 'compiler_options'

Add serving protocol buffer definitions [phab:D14]

Created by schroederc at 2015-01-30 18:42:27:

Protocol buffer definitions for efficient server lookup tables for the xrefs service and the filetree service.

Unable to use "go get" on kythe.io/kythe/go/util/schema packages

Since we added the text-format proto genrule, this package is no longer fetchable in the conventional way with go get. We should either generate a version to check in (as we do for the proto packages), or find a different way to represent that value.

Downgrade Java sources/tools.jar to be JDK7 compatible. [phab:D7]

Created by schroederc at 2015-01-28 22:43:55:

This change allows Kythe to be built with the JDK7 while still supporting Java8.