zkoppert / innersource-crawler

This project creates a repos.json that can be utilized by the SAP InnerSource Portal.

InnerSource Crawler

This project creates a repos.json that can be utilized by the SAP InnerSource Portal. The current approach assumes that the repos that you want to show in the portal are available in a GitHub organization, and that they all are tagged with a certain topic.

Support

If you need support using this project or have questions about it, please open up an issue in this repository. Requests made directly to GitHub staff or support team will be redirected here to open an issue. GitHub SLA's and support/services contracts do not apply to this repository.

Use as a GitHub Action

1. Create a repository to host this GitHub Action or select an existing repository
2. Create the env values from the sample workflow below (GH_TOKEN, ORGANIZATION) with your information as repository secrets. More info on creating secrets can be found here. Note: Your GitHub token will need to have read/write access to all the repositories in the organization
3. Copy the below example workflow to your repository and put it in the .github/workflows/ directory with the file extension .yml (ie. .github/workflows/crawler.yml)
4. Don't forget to do something with the resulting repos.json file. You can move it to another repository if needed or save it as a build artifact. This will all depend on what you are doing with it and what repository you are running this action out of.

Example workflow

name: InnerSource repo crawler

on:
  workflow_dispatch:
  schedule:
    - cron: '00 5 * * *'

jobs:
  build:
    name: InnerSource repo crawler
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2
    
    - name: Run crawler tool
      uses: docker://ghcr.io/zkoppert/innersource-crawler:v1
      env:
        GH_TOKEN: ${{ secrets.GH_TOKEN }}
        ORGANIZATION: ${{ secrets.ORGANIZATION }}
        # for multiple topics, add them after a comma eg:
        # TOPIC: inner-source,actions,security,python
        TOPIC: inner-source

Local usage without Docker

1. Copy .env-example to .env
2. Fill out the .env file with a token from a user that has access to the organization to scan (listed below). Tokens should have admin:org or read:org access.
3. Fill out the .env file with the exact topic name you are searching for
4. Fill out the .env file with the exact organization that you want to search in
5. (Optional) Fill out the .env file with the exact URL of the GitHub Enterprise that you want to search in. Keep empty if you want to search in the public github.com.
6. pip install -r requirements.txt
7. Run python3 ./crawler.py, which will create a repos.json file containing the relevant metadata for the GitHub repos for the given topic
8. Copy repos.json to your instance of the SAP-InnerSource-Portal and launch the portal as outlined in their installation instructions

License

MIT

innersource-crawler's Issues

For each resulting project add a key _InnerSourceMetadata to the result from the GitHub API call and fill it with additional metadata about the project:

innersource-crawler/crawler.py

Line 39 in 3730f4b

# TODO: For each resulting project add a key _InnerSourceMetadata to the result from the GitHub API call and fill it with additional metadata about the project:

handle rate limiting

innersource-crawler/crawler.py

Line 32 in 3730f4b

# TODO: handle rate limiting

Add support for gitHub Enterprise

Hi, it would be nie to be able to crawl internal GitHub Enterprise.

What kind of permission is needed ?

When we use this script, which permission should we define when creating the GH_TOKEN token ?

Do we only need the admin:org / read:org right ?

Convert this repository into a formal GitHub Action

Fix linting issues

This application has the following linting issues that should be resolve:

• crawler.py:1:0: C0114: Missing module docstring (missing-module-docstring)
• crawler.py:40:24: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
• crawler.py:47:22: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
• crawler.py:95:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)

Write pytests

See https://www.tutorialspoint.com/pytest/pytest_starting_with_basic_test.htm for help

repositories with no description cause an error

Repositories that don't have a description cause the following error:

Traceback (most recent call last):
  File "/action/workspace/crawler.py", line 84, in <module>
    innersource_repo["score"] = repo_activity.score.calculate(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.[11](https://github.com/innersource-portal-demo/project-portal-for-innersource/actions/runs/6279610170/job/17055587875#step:4:12)/site-packages/repo_activity/score.py", line 54, in calculate
    len(repo["description"]) > 30
    ^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()

A workaround is to add a description but it would be better if the crawler could handle repositories without descriptions gracefully and continue running.

Any plan to integrate the Repository Activity Score pattern?

Hi. Do you have plans to integrate the Repository Activity Score pattern? Or do you think that is out of the scope of the project?

I could help you if you want to move forward with that.