src-d / gitbase-web

gitbase web client; source{d} CE comes with a new UI, check it at https://docs.sourced.tech/community-edition/

Gitbase Web

Application to query Git repositories using SQL. Powered by gitbase, it allows to perform SQL queries on the Git history and the Universal AST of the code itself.

Usage

With Docker Compose

The easiest way to run Gitbase Web and its dependencies is using Docker Compose.

The first step is to populate a directory with some Git repositories to be served by gitbase before running it. For example:

$ mkdir $HOME/repos
$ cd $HOME/repos
$ git clone [email protected]:src-d/gitbase.git
$ git clone [email protected]:bblfsh/bblfshd.git
$ git clone [email protected]:src-d/gitbase-web.git

Next you will need to download the docker-compose.yml file from this repository and run docker-compose up. This tool will run three different containers: the gitbase-web frontend itself, gitbase, and bblfshd. To kill the running containers use Ctrl+C.

$ wget https://raw.githubusercontent.com/src-d/gitbase-web/master/docker-compose.yml
$ docker-compose pull
$ GITBASEPG_REPOS_FOLDER=$HOME/repos docker-compose up --force-recreate

The server should be now available at http://localhost:8080.

In case there are any containers left, you can use

docker-compose down

for cleanup.

Without Docker Compose

The application will run the queries against a gitbase server, and will request UASTs to a bblfsh server. Make sure both are properly configured.

Then you can choose to run the web client either as a docker container, or as a binary.

As a Docker

$ docker pull srcd/gitbase-web:latest
$ docker run -d \
    --publish 8080:8080 \
    --env GITBASEPG_DB_CONNECTION="root@tcp(<gitbase-ip>:3306)/none" \
    --env GITBASEPG_BBLFSH_SERVER_URL="<bblfshd-ip>:9432" \
    srcd/gitbase-web:latest

As a Binary

Download the binary from our releases section, and run it:

$ export GITBASEPG_DB_CONNECTION="root@tcp(<gitbase-ip>:3306)/none"
$ export GITBASEPG_BBLFSH_SERVER_URL="<bblfshd-ip>:9432"
$ ./gitbase-web serve

Configuration

Any of the previous execution methods accept configuration through the following environment variables or CLI arguments.

Variable	Argument	Default value	Meaning
GITBASEPG_HOST	--host	0.0.0.0	IP address to bind the HTTP server
GITBASEPG_PORT	--port	8080	Port to bind the HTTP server
GITBASEPG_SERVER_URL	--server		URL used to access the application in the form HOSTNAME[:PORT]. Leave it unset to allow connections from any proxy or public address
GITBASEPG_DB_CONNECTION	--db	root@tcp(localhost:3306)/none?maxAllowedPacket=4194304	gitbase connection string. Use the DSN (Data Source Name) format described in the Go MySQL Driver docs.
GITBASEPG_CONN_MAX_LIFETIME	--conn-max-lifetime	30	Maximum amount of time a SQL connection may be reused, in seconds. Make sure this value is lower than the timeout configured in the gitbase server, set with GITBASE_CONNECTION_TIMEOUT
GITBASEPG_BBLFSH_SERVER_URL	--bblfsh	127.0.0.1:9432	Address where bblfsh server is listening
GITBASEPG_SELECT_LIMIT	--select-limit	100	Default LIMIT forced on all the SQL queries done from the UI. Set it to 0 to remove any limit
GITBASEPG_FOOTER_HTML	--footer		Allows to add any custom html to the page footer. It must be a string encoded in base64. Use it, for example, to add your analytics tracking code snippet
LOG_LEVEL	--log-level=	info	Logging level (info, debug, warning or error)
LOG_FORMAT	--log-format=		log format (text or json), defaults to text on a terminal and json otherwise
LOG_FIELDS	--log-fields=		default fields for the logger, specified in json
LOG_FORCE_FORMAT	--log-force-format		ignore if it is running on a terminal or not

Contribute

Contributions are more than welcome, if you are interested please take a look to our Contributing Guidelines. There you will also find information on how to build and run the project from sources.

Code of Conduct

All activities under source{d} projects are governed by the source{d} code of conduct.

License

GPL v3.0, see LICENSE

gitbase-web's Issues

Inform the user of the elapsed time

The frontend should count the time needed for each query, and reflect that in a success message

Frontend: Perform 'describe table' queries from the sidebar

Clicking on a table name in the sidebar will automatically execute a 'describe table' query.

Show UAST from code

The details need to worked out, but the code viewer (#47) should be able to provide the UAST from the code, using the /parse endpoint (#46).

Integrate uast-viewer

Integrate the uast-viewer lib to open when the uast button (#23) is clicked.

Update docker-compose

Update https://github.com/src-d/gitbase-playground/blob/master/docker-compose.yml

This will help avoiding breaking users i.e on transition of gitbase v0.11.0 -> v0.12.0

• fix a particular version of bblfsh image
• make playground image depends on gitbase. Latest version should be fine for both

  services:
    playground:
      depends_on:
        - gitbase
  

• update quickstart instructions
  should include $ docker-compose pull playground --include-deps

Create a drone.yaml file

Handle queries with 'limit' gracefully

Right now if a query contains a 'limit' the server fails with invalid syntax.
Ideally the limit provided by the user should be allowed if it's below the forced limit, or replaced otherwise.

Enable gitbase squash feature in docker compose and staging

As requested by data retrieval team, add GITBASE_UNSTABLE_SQUASH_ENABLE: "true" to docker compose and drone.

Before updating drone.yml we need to have support in the helm chart, PR src-d/charts#51.

Wrong behaviour tab title edits

• When the pencil icon is clicked the new textarea does not get focused automatically. This is confusing because you start writing and nothing happens
• If you click on more than one pencil icons, you can have a few tab titles as textarea. I think this could be related to the previous point, if the textarea does not get focused the event on focus lost does not get fired.
• There space key does not work
• The arrow keys (or ctrl+arrow) do not move the cursor, they change between tabs

New parse endpoint to interact with babelfish

New POST /parse endpoint will receive code and optional filter, forward that to babelfish server, and return a UAST.

Apply product team design

Apply the design details, as provided by the product team

Staging deployment

Needs: an env that dev team can control (including gitbase version, etc) and deploy up on often.

It includes: gitbase, some .siva files from PGA, CD for releases.

Talk to @rporres to see, what would be the best way.

Invalidate cache between releases

Otherwise when a new release is deployed users have to hard-refresh the browser to see the changes.

Improve code detection and guess a language

A language should be guessed using enry.

I propose to add information which column is a code and language in meta of the query endpoint response.

Code viewer

Integrate the uast-viewer lib to open code viewer when the code button (#23) is clicked.

Frontend: SQL syntax highlight

Add SQL syntax highlight using the same mechanism as babelfish dashboard.

Hide history tab when the history is empty.

As detailed by the product team.

Add export button

Instead of RUN the user will be able to click on EXPORT to download a file.
In this case the query will not automatically have the imposed limit.

The file format was not specified, will probably be csv.

Update query examples to support latest gitbase

Latest gitbase has been released and it includes some breaking changes i.e in table names src-d/gitbase#291

Playground should be updated, so all query examples including a default one support it.

Define the help contents

The current design spec includes a 'help' button, but we still need to define:

• Behavior. Will this be a modal window? a tooltip? an external link?
• Contents.

Related to #44.

New endpoint to get the complete DB schema

Context: #35 (comment)

quoting @bzz:

Instead for

• API call /tables
• then, for each table call DESCRIBE TABLE from the frontend

We should have

• API call to /tables, so that the backend returns all this information in a single call

on top of per improvements (less HTTP calls), one of the purposes of having separate API endpoint on the backend was to abstract the details of how meta-information (schema) is collected.

Current approach, for a case if it is not a DESCRIBE TABLE - would require us to change both, backend and frontend. The one suggested above would requite only the backend change.

Suspend inactive tabs

The basic requirement is to have tabs drop the loaded information and go into a suspended state after a specific period of time being hidden.
The suspended tab will have a "reload" button to run the query again.

Create project skeleton

Update project structure with the setup used for applicastion

• FE: a package.json \w standard scripts that do echo 1
• BE: Empty test. Almost the same code structure as in CAT (use chi, but a separate dir for FE)
• Build structure, single binary
• TravisCI

Frontend: Add code and UAST buttons in results table

Insert buttons in code and uast columns. For now the buttons will open a modal window with the raw contents.

Frontend: sidebar with reference

Add a left sidebar with the gitbase tables and columns reference in a collapsable structure.

This sidebar should also include a header with a small logo, and a footer with relevant links:

• https://sourced.tech
• https://github.com/src-d/gitbase
• https://doc.bblf.sh
• https://github.com/src-d/go-git

Define the example queries

The left bar contains placeholder examples that need to be defined

Frontend: Add tabs

Each query request opens a new tab containing:

• the SQL query string
• a link or button "edit query" that fills the main query box with this tab's query
• the results table

Ideally the top and bottom areas (query and tabs) can be resized with a drag-able separator.

Backend: http proxy to gitbase

Implement 2 endpoints

1. Query: POST /query

req: { query:””, limit: N }
res: { meta:{..table headers..}, data:[row,...,], 
        status:{ elapsed, error/warning } 
}

No streaming yet. Include UAST -> JSON conversion.

2. Tables: GET /tables

res:  same

And two more endpoints as just a mocks

1. UAST: POST /parse

 req: { code: “”, filter: “” }
 res: { uast: “”, status: {} }

2. Filter: POST /filter

  req: {} //need to include a binary PB
  res: { uast: “”, status {} }

Frontend: display errors in the results section

Display any error messages where the results table would appear.

This may change to follow the wireframes later on.

make compose-serve requires a project in $GOPATH

Building from sources for docker-compose through make compose-serve requires that the project is in the $GOPATH; otherwise the make target will fail.

Move "api.js" into its own place

from #16 (comment)

We could move frontend/src/api.js into its own place as done in CAT for example.

I see two reasons:

• first one is kindof opinionated: imo the root dir should be populated with as fewer things as possible because it helps to understand better the structure of the code.
• second one is generic: now we have in this file the logic (copy-pasted from cat) to connect with the API (business agnostic), and also the "repository-like" code (business related). Splitting both parts properly, and moving business agnostic to a service, and the business logic to a repository would help us to reuse and test better.

CI: test go tip

We should configure the matrix for tip and allow_failures for that.

Execute the default query on page load

As detailed by the product team.

Inform the user of the hardcoded LIMIT

We add a hardcoded limit to every query, but it's not obvious to the user.
When you realize there is missing data it looks like something is broken.

My proposal is to return the edited query and display that inside the results tab. So for the query
SELECT * FROM files

You would get on top of the results table
SELECT * FROM files LIMIT 100

cc @src-d/product team.

UAST filtering from the frontend

To allow UAST XPATH queries in the UAST viewer we would need a new endpoint, POST /filter, to ask babelfish to apply an XPATH query to a UAST.

This endpoint will need the protobuf UAST data, which we currently do not send to the frontend.
We will need to add the protobuf data to the POST /query response; and decide if we want to decode the protobuf data in the frontend, or send both json and protobuf.

Modify uast nodes the same way as babelfish dashboard

From this comment by @smacker:

dashboard replaces Role ID by Role name. Most probably playground should be compatible with it:
https://github.com/bblfsh/dashboard/blob/master/server/server.go#L217

commits.commit_parents is detected as UAST

The column commits.commit_parents is an array of strings, and we should not try to render this as UAST.

Update build, for a single binary to include all fronted

Use the same approach as in CAT:

• release is a single binary
• every PR does not include re-generated JS but go-bindata proxies instead
• is not go getable

SQL input box is broken in FF

1. If the height of the box isn't too big I can't scroll down to see the last line of the default query
2. Selection works weird:

Add queries examples to the sidebar

The user will have a few examples in the sidebar that, when clicked, will fill the input area.

Make the sidebar collapsable

Test handlers

Code coverage is really bad...

Forbid "console" funcs

from #16 (comment)

It could be a good idea

• to forbid console commands (.eslintrc.js::rules['no-console']), and
• to provide a logger wrapper as in CAT?

Frontend: implement simple UI

Implement main functional components:

• query
• run button
• result table (no any pagination is ok at this stage)

take care of passing env var to fronted.
At this point, query should always add LIMIT N with N=100 defined though env var.

Querybox scroll is in the wrong place in Firefox

Chrome, everything correct (scroll inside codemirror):

Firefox (scroll is in the pane):

Format files according to .editorconfig

Some files do not follow the .editorconfig style. Context: #27 (comment).

Edit tab titles

Each tab should have an edit button that will allow to edit the displayed tab title.

Add a history tab

The frontend will store (locally in the browser) a history of the queries.

More details about the design and functionality in the Visual Design & Pattern Library.

• query string
• elapsed time
• status (success/failure)
• failure reason
• start time
• We might want to store the tab title (editable)

Shared frontend library \w common dependencies

Similar experience to using react-create-app, but instead we will have most of the things FE apps depend on, inside a single library.

It defines:

• build/lint tasks
• React
• Bootstrap with Less

Backend: Rethink how to work with assets, bindata...

from #15 (comment) @smacker suggested that it would be great to:

[...] re-think how we work with assets. Maybe we can find better solution.