elscrux / taskevaluator

Task Evaluator

This tool is built to evaluate the code generated by generative AI tools for coding such as GitHub Copilot. In total, there are 4 major components to achieve this:

• Code Generator
• Code Evaluator
• Database & Data Visualization
• Interface (App, Web Api or Cli)

Setup

The tool is using multiple external services that need to be set up to gain full functionality. There are external tools for code generation, code evaluation and data visualization.

In any case, you'll first have to

• Install Docker
• Start Docker-Engine
• Copy .env.example to .env and fill the commented out variables (see the individual sections below for more information)
• Start the Docker Containers with

  docker-compose up -d

Manual Setup

Detailed explanations for the setup of the individual Docker containers.

GitHub Copilot

• Add the following .NET User Secrets for TaskEvaluator

{
    "GitHubCopilot": {
        "CompletionsUrl": "https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions",
        "TokenUrl": "https://api.github.com/copilot_internal/v2/token",
        "UserAgent": "GithubCopilot",
        "UserAgentVersion": "1.138.0",
        "EditorVersion": "vscode/1.84.1",
        "EditorPluginVersion": "copilot/1.138.0",
        "BearerToken": "YOUR_BEARER_TOKEN",
        "Openai-Organization": "github-copilot",
        "Openai-Intent": "copilot-ghost"
    }
}

• Replace the Bearer Token with your GitHub Copilot Bearer Token
  • Run this python script
  • Connect with GitHub Account
  • Copy the Bearer Token from the console output

Tabby

• Pull Tabby Image

  docker pull tabbyml/tabby

• Add the following .NET User Secrets for TaskEvaluator

{
    "Tabby": {
        "CompletionsUrl": "http://localhost:8080/v1/completions"
    }
}

• Start Tabby Container (when running on GPU with CUDA support)

  docker run -it --gpus all -p 8080:8080 -v $HOME/.tabby:/data tabbyml/tabby serve --model TabbyML/StarCoder-1B --device cuda

• Start Tabby Container (when running on CPU)

  docker run --entrypoint /opt/tabby/bin/tabby-cpu -it -p 8080:8080 -v $HOME/.tabby:/data tabbyml/tabby serve --model TabbyML/StarCoder-1B

SonarQube

• Pull SonarQube Image

  docker pull sonarqube

• Start SonarQube Container

  docker network create taskevaluator_sonarqube_net
  docker run -d --name sonarqube -p 9000:9000 --net taskevaluator_sonarqube_net sonarqube

  • Open localhost:9000
    • Optionally set custom Environment Variable SONARQUBE_URL
  • Add the following .NET User Secrets (use custom credentials if applicable)

{
    "SonarQube": {
        "Url": "http://sonarqube:9000",
        "User": "admin",
        "Password": "admin"
    }
}

Data Sink

PostgreSQL Sink

• Pull Postgres Image

  docker pull postgres

• Start Postgres Container

  docker network create taskevaluator_postgres_net
  docker run -d --name postgres -u postgres -e POSTGRES_PASSWORD=YOUR_PASSWORD -p 5432:5432 --net taskevaluator_postgres_net postgres

  • Add the following .NET User Secrets

{
    "Database": {
        "ConnectionString": "User ID=postgres;Host=localhost;Port=5432;Password=YOUR_PASSWORD;"
    }
}

Visualization

Grafana

• Pull Grafana Image

  docker pull grafana/grafana

• Start Grafana Container

  docker run -d --name grafana -u grafana -p 3000:3000 --net taskevaluator_postgres_net grafana/grafana

• Login (default credentials: admin/admin)
• Add a new Data Source
  • Type: PostgreSQL
  • Host: taskevaluator-db-1:5432
  • Database: postgres
  • User: postgres
  • Password: YOUR_PASSWORD
  • SSL Mode: disable

Language Support

Currently, the tool only supports C# as a programming language.

Running a Task Set

• Create a directory which contains your task set
• Add the path to the directory to your .NET User Secrets

{
    "TaskSet": {
        "DirectoryPath": "YOUR_TASK_SET_DIRECTORY_PATH"
    }
}

• The directory should have the following structure:
  • [Language]
    • [TestName]
      • File including the name "Program" for the source code
      • File including the name "UnitTest" for the unit tests
      • metadata.json with additional information
        • Example:

      {
          "id": "112c5a6e-0e7c-4e49-b699-8e2be2e24e4a",
          "isHumanEval": true
      }

• Here an example
  • CSharp
    • Test1
      • Program.cs
      • UnitTests.cs
      • metadata.json
    • Test2
      • Program.cs
      • UnitTest
      • metadata.json
    • Test3
      • MyProgram
      • UnitTests // No metadata.json - default values will be used
    • ...
  • ...