packit / ogr

One Git library to Rule -- one API for many git forges

One Git library to Rule

Library for one API for many git forges. (e.g. GitHub, GitLab, Pagure).

Currently supported git forges:

• GitHub
• GitLab
• Pagure

To start using ogr:

See Documentation of Index of Sub-modules, Functions & Classes

For examples of how to use ogr see Jupyter examples.

GitHub

This snippet shows how to obtain all releases for certain GitHub project using ogr.

from ogr.services.github import GithubService

service = GithubService(token="your_token")

ogr_project = service.get_project(
        repo="ogr",
        namespace="packit-service"
)

ogr_releases = ogr_project.get_releases()


for release in ogr_releases:
    print(release.tag_name)

This will output:

0.7.0
0.6.0
0.5.0
0.4.0
0.3.1
0.3.0
0.2.0
0.1.0
0.0.3
0.0.2
0.0.1

You can use the same API for other forges, you just need to replace GithubService with PagureService.

Supported functionality

For more info on functionality that is not supported in all services the same way see compatibility tables.

Installation

On Fedora:

$ dnf install python3-ogr

You can also use our packit-releases Copr repository (contains also released versions of OGR):

$ dnf copr enable packit/packit-releases
$ dnf install python3-ogr

Or from PyPI:

$ pip3 install --user ogr

You can also install OGR from the main branch, if you are brave enough:

You can use our packit-dev Copr repository (contains the latest development version of ogr):

$ dnf copr enable packit/packit-dev
$ dnf install python3-ogr

Or

$ pip3 install --user git+https://github.com/packit/ogr.git

Requirements

Makefile

• podman
• ansible-bender

Tests

Make sure to install prerequisite packages before first make check, make build, or make check-in-container. See CONTRIBUTING.md for details.

Contribution notes

• Property should not connect to network.

Contribution guidelines

For more info about contributing to our project see our contribution guide.

Deprecation policy

For more info about deprecation policy see Deprecation policy

ogr's Issues

get_releases for Pagure

services.github.py already can list all releases. We need this for Pagure too.

better name for @readonly decorator

The @readonly decorator should be renamed as it can confuse newcomers that the method itself is "readonly". The fact is that it has the opposite meaning of the read-only.

Feel free to add your ideas as a single comment so we can easily vote.

---

Follow-up from #48

0.1.0 release

we need this b/c 0.0.2 doesn't have specfile

and I want to use packit to bring the release to fedora

GitLab support

It would be nice to finally start implementing the GitLab support.

We can probably combine the code from GitHub and the already present code.

The documentation can be found here: https://python-gitlab.readthedocs.io/en/stable/

Support releases in Pagure

Figure out how to handle releases in Pagure.

new minor release

0.0.2 release

Thank you in advance!

Page `/fork/phracek/rpms/rebase-helper/pull-request/new` not found when calling Pagure API

I would like to solve an issue packit/packit#328
but it failed with traceback:

$ packit -d propose-update
[..snip..]
10:40:20.990 upstream.py       DEBUG  Version in spec file: 0.16.3
Picking version found in spec file.
Checking out upstream version 0.16.3
10:40:21.013 base_git.py       DEBUG  Allowed GPG keys are not set, skipping the verification.
10:40:21.014 base_git.py       DEBUG  Running ActionName.pre_sync hook.
10:40:21.014 base_git.py       DEBUG  Running ActionName.pre_sync.
10:40:21.014 base_git.py       DEBUG  Running default implementation for ActionName.pre_sync.
Using 'master' dist-git branch
10:40:21.017 base_git.py       DEBUG  It seems that branch master already exists, checking it out.
10:40:21.020 distgit.py        DEBUG  About to update branch 'master'
10:40:21.727 api.py            DEBUG  Path of README /tmp/tmpzla3qy3y/README.packit
10:40:21.728 base_git.py       DEBUG  Running ActionName.prepare_files.
10:40:21.729 base_git.py       DEBUG  Running default implementation for ActionName.prepare_files.
10:40:21.729 sync.py           DEBUG  Copy synced files [RawSyncFilesItem(src=/home/vagrant/rebase-helper/.packit.yml, dest=/tmp/tmpzla3qy3y/.packit.yml, dist_is_dir=False), RawSyncFilesItem(src=/home/vagrant/rebase-helper/rebase-helper.spec, dest=/tmp/tmpzla3qy3y/rebase-helper.spec, dist_is_dir=False)]
10:40:21.730 sync.py           DEBUG  src = /home/vagrant/rebase-helper/.packit.yml, dest = /tmp/tmpzla3qy3y/.packit.yml
Copying /home/vagrant/rebase-helper/.packit.yml to /tmp/tmpzla3qy3y/.packit.yml.
10:40:21.731 sync.py           DEBUG  src = /home/vagrant/rebase-helper/rebase-helper.spec, dest = /tmp/tmpzla3qy3y/rebase-helper.spec
Copying /home/vagrant/rebase-helper/rebase-helper.spec to /tmp/tmpzla3qy3y/rebase-helper.spec.
10:40:21.814 distgit.py        DEBUG  Upstream archive name is 'rebase-helper-0.16.3.tar.gz'
Archive rebase-helper-0.16.3.tar.gz found in lookaside cache (skipping upload).
10:40:22.347 distgit.py        DEBUG  Upstream archive name is 'rebase-helper-0.16.3.tar.gz'
10:40:22.348 base_git.py       DEBUG  About to add all & commit
10:40:22.366 distgit.py        DEBUG  About to force push changes to branch 0.16.3-master-update of a fork fork of the dist-git repo
10:40:32.251 distgit.py        DEBUG  About to create dist-git pull request from 0.16.3-master-update to master
ERROR    There was an error while creating the PR: PagureAPIException('Page `https://src.fedoraproject.org//api/0/fork/phracek/rpms/rebase-helper/pull-request/new` not found when calling Pagure API.')
ERROR    Page `https://src.fedoraproject.org//api/0/fork/phracek/rpms/rebase-helper/pull-request/new` not found when calling Pagure API.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/packit/cli/utils.py", line 58, in covered_func
    func(config=config, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/packit/cli/update.py", line 113, in update
    upstream_ref=upstream_ref,
  File "/usr/local/lib/python3.7/site-packages/packit/api.py", line 229, in sync_release
    dist_git_branch=dist_git_branch,
  File "/usr/local/lib/python3.7/site-packages/packit/api.py", line 316, in push_and_create_pr
    target_branch=dist_git_branch,
  File "/usr/local/lib/python3.7/site-packages/packit/distgit.py", line 211, in create_pull
    target_branch=target_branch,
  File "/usr/local/lib/python3.7/site-packages/ogr/mock_core.py", line 49, in readonly_func
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ogr/services/pagure.py", line 377, in pr_create
    "initial_comment": body,
  File "/usr/local/lib/python3.7/site-packages/ogr/services/pagure.py", line 232, in _call_project_api
    url=request_url, method=method, params=params, data=data
  File "/usr/local/lib/python3.7/site-packages/ogr/services/pagure.py", line 71, in call_api
    raise PagureAPIException(f"Page `{url}` not found when calling Pagure API.")
ogr.exceptions.PagureAPIException: Page `https://src.fedoraproject.org//api/0/fork/phracek/rpms/rebase-helper/pull-request/new` not found when calling Pagure API.
10:40:34.958 local_project.py  DEBUG  Cleaning: /tmp/tmpzla3qy3y

Referenced issues:
https://pagure.io/pagure/issue/4512
packit/packit#328

Update pull-request information

Add methods to GithubProject/PagureProject for editing title and description (first-commit in Pagure) of the pull-request.

For Pagure see https://src.fedoraproject.org/api/0/ --> Update pull-request information

Change "Status" to "Flag" in the class names

What about changing CommitStatus/PullRequestStatus to CommitFlag/PullRequestFlag?

Original discussion: #71 (review)

libpagure has changed

libpagure==0.20 changed headers handling, so that
code like:
https://github.com/packit-service/ogr/blob/master/ogr/services/our_pagure.py#L172
has to be adapted, currenly I've added dependency on lower version of libpagure to
https://github.com/packit-service/ogr/pull/63/files#diff-380c6a8ebbbce17d55d50ef17d3cf906R44
to avoid this issue.

ogr doesn't work with python3.6

138 Bytecompiling .py files below /builddir/build/BUILDROOT/python-ogr-0.0.2-1.fc28.x86_64/usr/lib/python3.6 using /usr/bin/python3.6
139 *** Error compiling '/builddir/build/BUILDROOT/python-ogr-0.0.2-1.fc28.x86_64/usr/lib/python3.6/site-packages/ogr/abstract.py'...
140   File "/usr/lib/python3.6/abstract.py", line 1
141 SyntaxError: future feature annotations is not defined
142 *** Error compiling '/builddir/build/BUILDROOT/python-ogr-0.0.2-1.fc28.x86_64/usr/lib/python3.6/site-packages/ogr/services/github.py'...
143   File "/usr/lib/python3.6/github.py", line 1
144 SyntaxError: future feature annotations is not defined
145 *** Error compiling '/builddir/build/BUILDROOT/python-ogr-0.0.2-1.fc28.x86_64/usr/lib/python3.6/site-packages/ogr/services/pagure.py'...
146   File "/usr/lib/python3.6/pagure.py", line 1
147 SyntaxError: future feature annotations is not defined

get installation ID: use code from pygithub once a new release is available

PyGithub/PyGithub#1135 is now merged

Use the code above to initiate GitHubService, related code (which we are using in the meantime):

pretty much drop this:
https://github.com/packit-service/ogr/blob/253e93bc87c079d0cd8f3dc87fc5e11f5078e53b/ogr/services/github_tweak.py#L11

Use the newly added functionality in pygithub here:
https://github.com/packit-service/ogr/blob/5729e29cb4e5bea7eb6ed68e1c573ca7f2d86051/ogr/factory.py#L142

Also please write a test case for this.

testing: split yaml files with data for each test to avoid regeneration big file and big commits

now each integration test class contains right one file with stored API data.
We could reconsider it, and maybe have file for each test, to avoid to regenerate big file and have big commits.
We have to solve several issues:

• How to handle it per test, now there is right one service pytest fixture function what contains it
• How to regenerate just relevant part
  • maybe remove some timestamps, UIDs and similar objects what can change on every request
  • or manually(semiautomatically) call just affected tests - ie: run tests, find failures (key errors) and then try to just regenerate files for failed tests

user's permissions on Issues/PRs

Updated by @lachmanfrantisek :

We have a lot of methods related to the permissions (e.g. who can merge/close,..)

It's not consistent, let's clean that!

Feel free to solve this in smaller parts. Just write on what you are going to work...

AC:

• Make sure, that we have all of the provided methods in the abstract.py
  (raising the NotImplementedError)
• Make sure, that we have all of these in all the implementations:
  • GitLab
  • GitHub
  • Pagure
• Deprecate the project methods related to specific issue/pull-request.
• Move helping methods to the base classes.
  • The only exception is a more efficient solution in the code of the specific implementation.
  • Example:
    https://github.com/packit-service/ogr/blob/89435655150b8dd79ef93209c3463585d0715e0d/ogr/services/github/project.py#L219-L220

The progress is tracked in the following tables. (Any update in the comments is appreciated.)

Project	who_can_close_issue() -> Set[str]	who_can_merge_pr() -> Set[str]	can_close_issue(username: str) -> bool	can_merge_pr(username: str) -> bool
abstract	👍	👍	❗ deprecate the Issue argument
base	nothing	nothing	implement here	implement here
Github	?	?	?	?
GitLab	?	?	?	?
Pagure	?	?	?	?

Issue	who_can_close() -> Set[str]	can_close(username: str) -> bool
abstract	?	?
base	nothing	implement here
Github	?	?
GitLab	?	?
Pagure	?	?

PullRequest	who_can_close() -> Set[str]	who_can_merge() -> Set[str]	can_close(username: str) -> bool	can_merge(username: str) -> bool
abstract	?	?	?	?
base	nothing	nothing	implement here	implement here
Github	?	?	?	?
GitLab	?	?	?	?
Pagure	?	?	?	?

Missing functionality for getting content of a file.

In the case of Github, it is possible to get a repository's file content via API or PyGithub lib. In the case of Pagure, I am not sure if it is possible.

(I need this for release-bot, I can hack it for now, but would be wonderful to have such feature in ogr.)

Improve testing

• don't fail when PAGURE_{TOKEN,USER} are not specified
• mock remote API and write complete unit tests using those data (or even have a single test suite and enable it to run against live instance and mocked data)
• tox-ify it (#27)

get_email for GitUser

We have get_username method in GitUser class. At the moment we are missing method get_email for both Github and Pagure services.

---

• GitHub
• GitLab
• Pagure

new minor release

Add compatibility with Github's GraphQL API v4 and rest API v3

I'm trying to implement ogr into release-bot and it's a little bit painful without documentation from ogr.

I was playing with ogr on my Github projects and everything is working fine. When I tried to reimplement release-bot's functionalities which communicate with Github via GraphQL or rest API I got stuck.

One of the examples could be comments. I want to implement commenting on Github issues and PRs via ogr.

• In release-bot it works via graphQL and sending some node_id inside query which recognizes a PR or an Issue.
• In ogr there is pr_comment function in ogr/services/github.py. If am I right this is an alternative I'm looking for. However, I don't fully understand the function's parameters since it needs something else then API query.

I understand that this could be a very complex problem since ogr need to preserve compatibility with all git forges. I'm just looking for some starting point on how to change GraphQL/rest API with ogr. Maybe I just misunderstand some ogr's fundamentals.

0.2.0 release

new minor release

ReleaseBot, we would like to release a new version.

Thanks in advance!

missing annotations library for testing

after installing ogr via pip3 install .
I wanted to run make check PYTHONPATH=`pwd` pytest-3 -v ./tests/

it leads to errors like:

E     File "/home/jscotka/git/userspace-containerization/ogr/ogr/abstract.py", line 1
E       from __future__ import annotations
E                                        ^
E   SyntaxError: future feature annotations is not defined

seems that annotations module is not accessible in python3-3.6.8-1.fc28.x86_64 by default

Deprecation policy

Figure out a deprecated strategy for ogr, get inspired by packit: https://packit.dev/ [or django, ansible].

• documentation
• solution for method/classes

Allow creating releases

For Github we can get the releases:
https://github.com/packit-service/ogr/blob/master/ogr/services/github.py#L571

We need to also create a new release from API.

---

Relevant documentation:
https://pygithub.readthedocs.io/en/latest/github_objects/Repository.html#github.Repository.Repository.create_git_release

new minor release

Add get_owner/get_owners method to project classes

It would be useful to have a method for getting the owner(s) of the project.

If it is possible to have multiple owners, create get_owners and return list of strings.

0.0.3 release

Release bot, please!

String representations for classes

It would be nice to implement __str__ methods for all classes (e.g. GithubProject, Issue,...).

create_fork should not fail if the fork already exists

        if req.status_code != 200:
            LOG.error(output)
            if 'error_code' in output:
>               raise APIError(output['error'])
E               libpagure.exceptions.APIError: Repo "forks/ttomecek/sen" already exists     

/home/tt/.local/lib/python3.7/site-packages/ogr/services/pagure.py:151: in fork_create
    self._pagure.create_fork()
/home/tt/.local/lib/python3.7/site-packages/ogr/services/our_pagure.py:254: in create_fork                                                                                  
    data={"repo": self.repo_name, "namespace": self.namespace, "wait": True},

WDYT?

libpagure.Pagure expects '/' not in repo attribute

While debugging packit/packit#305 I realized that it fails because Pagure.create_basic_url() creates:
https://src.fedoraproject.org//api/0/rpms/rpms/packit/pull-requests which does not exist, correct is:
https://src.fedoraproject.org//api/0/rpms/packit/pull-requests.

The problem is that OurPagure.repo attribute has a value rpm/packit, while libpagure probably expects it to be packit only.

Anybody has any idea whether this is a ogr or packit or libpagure issue?
I tend to think it's ogr issue, because OurPagure.repo_name expects / in self.repo, which is probably not what libpagure wants.

And it gets even more puzzled when I tell you that Pagure.create_basic_url() has changed even more recently.

run our tests in Fedora CI

this means to use standard-test-roles to wrap our tests so they can run when we create PRs on dist-git

see python-docker's test wrapper for more details

add a way to set check results for a commit

we need this in packit this sprint when we start building in copr

this should be implemented soonish and land in next release

relevant code for github:
https://github.com/packit-service/packit/blob/04793f16d5ced27b5e7fec7d963d650b065a4276/packit/watcher.py#L94

Run tests on the OGR project itself

• Try to have only one project for testing (ideally OGR itself in case of GitHub).
• Then it is easy for everyone to regenerate the test responses.

Document how to save HTTP sessions for sake of testing

Hi, see lines in: https://github.com/packit-service/ogr/blob/master/tests/integration/test_github.py#L20
So for regenereation these files, you have to use syntax eg:
``FORCE_WRITE=yes GITHUB_TOKEN=yourtoken GITHUB_USER=yourname pytest3 tests/integration/test_github.py``
or this will also work ``FORCE_WRITE=yes GITHUB_TOKEN=yourtoken GITHUB_USER=yourname make check`` with make check, you have to have set PAGURE_TOKEN and USER env vars

Originally posted by @jscotka in #80 (comment)

Commenting on Issues

Based on our discussion on Issue #79, there is no functionality in ogr for commenting on Github/Pagure issues.

Modules with poor code coverage

Name                     Stmts   Miss  Cover   Missing
----------------------------------------------------------
services/github.py         114     63    45%   17-19, 24-26, 30, 33-34, 41-42, 47, 51, 58-61, 64, 67, 70, 73-79, 82-83, 86-87, 95-98, 108, 111, 114-115, 118, 121, 124, 127-132, 135, 149, 170, 184, 193-203, 207-209, 212-213, 216-217, 227, 230
services/gitlab.py          66     66     0%   1-144
services/our_pagure.py      79     46    42%   33, 36-39, 134-148, 160-176, 180-186, 249-256, 259-264, 267-270, 273, 276, 279-282, 285-288, 291-294, 297-298
utils.py                   150    110    27%   35-77, 82-83, 88-89, 94, 98-116, 120-130, 145-148, 163-165, 169-176, 180-214, 219-248, 252, 260, 273

I know the gitlab module is WIP, but the others deserve more tests IMHO

get_file_content: catch only 404, raise everything else

Just spent some time debugging an issue where packit.yaml was not found while being 100 % it's in there.

It turned out that my token was incorrect and github kept throwing 401: that should bubble up, we should catch 404 only.

object-specific methods bound to ogr classes

• Forward some object to the issue/pr/... instances to allow connection to the service. (Classes are not only data-holders.)
• All ogr classes to have properties with setters allowing to update its attributes
  • I can do my_pr.title = "something"
• Objects are linked together (Service <-> Project <-> PR <-> PRComment)
• Keep old methods, mark them as deprecated.

needs #121

Add a link to `GitTag` from `Release`.

Add a link to GitTag from Release.

Originally posted by @lachmanfrantisek in https://github.com/packit-service/ogr/pull/71/files

Better name

The ogr is a nice, short name, but the 'meaning' should be changed.

Feel free to edit this comment.

O	G	R
our	git	rule
one	git-forge	revolution
open		rebellion
object		realm
only		route
omnipotent		rocket
		recipe
		rainbow
		roadway
		reactor
		reunion
		rebel
		ruler

Possible combinations (and votes):

• our-git-rebel @phracek
• open-git-rainbow @phracek @TomasTomecek
• object-git-reactor @phracek
• the [O]ne [G]it [R]uler @TomasTomecek
• [O]ne [G]it library to [R]ule @TomasTomecek @jpopelka
• [O]mniponent [G]it-forge [R]uler @jpopelka @lachmanfrantisek @TomasTomecek

Add a documentation for API

There is no documentation on how to use the library, currently. A documentation will help people on how to use ogr, and showcase it's features and functions.

This can be done by adding the wiki pages (on Github), since the documentation might be quite long and the Wiki will help in segregating several topics (for instance, how to use ogr for GitHub, GitLab and Pagure).

Clean the ogr/utils.py

In #102, I've realized that most of the functions in utils.py are never used.

We need to go through the functions and check if they are not used anywhere (please, control the packit codebase as well) and remove the unused ones.

---

In the other hand, we can move here the LocalProject or other git-related code from the Packit. Do we want OGR to be only remote API or git-helper?

edit Github release

Can we create a method for editing a release? We can use this for it.

create target for makefile for regeneration testing yaml files

as integration tests needs regeneration stored yaml API communication.
Would be nice to add there target to be able to regenerate these files

update contribution guide

Can we please update the contribution guide. Ogr now uses new zuul CI (or softwarefactory-project-zuul).

I'm a little bit stuck on what should I do to make my tests passing when PR is opened. On my localhost, everything is working in the "old way" and tests are passing.

Remove PullRequests from list of Github Issues.

Based on Github API doc:

Every pull request is an issue, but not every issue is a pull request.

Therefore, in the case of Github, functions as get_issue_list, get_issue_info etc. are working also with Pull Requests and behavior isn't tested.

We should get rid of Pull Requests in functions working with Issues. Pagure doesn't have this problem.

---

@lachmanfrantisek added:

To be able to be compatible with other git forges, let's differentiate between issues and pull-requests. (And do not allow working with GitHub issues, that are actually pull-requests. Issues and pull-requests are distinct sets.)

AC:

• Go through the issue related code in GithubProject and check if it does not return any PullRequest.
• Create test cases for that.

github: fork repository name may not match the upstream repo name

FML!

So my fork of packit is named source-git, not packit.

We should use an API call to obtain github forks, not a guess work.

Unify external command invocation

utils.py uses several different approaches to invoke an external command:

• subprocess.Popen
• subprocess.check_call
• subprocess.check_output

The recommended approach (since Python 3.5) is to use subprocess.run.

Also, there's lots of external libraries for this use case:

• http://amoffat.github.io/sh/
• https://www.pyinvoke.org/
• https://plumbum.readthedocs.io
• https://pexpect.readthedocs.io

missing ansible-bender dep for building

Hi,
it is unbale to get or install ansible-bender for make build target.

It is not part of Fedora-29.

I've found that @TomasTomecek done this: https://github.com/TomasTomecek/ansible-bender
but it still failed:

12:43 $ sudo pip3 install ansible-bender
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting ansible-bender
  Downloading https://files.pythonhosted.org/packages/fa/5c/04c822b3ee6e2ff41eed372f15d954cccf164d0340341fbd0270d500bb83/ansible_bender-0.4.0-py2.py3-none-any.whl
Requirement already satisfied: PyYAML in /usr/lib64/python3.7/site-packages (from ansible-bender) (4.2b4)
Collecting tabulate (from ansible-bender)
  Downloading https://files.pythonhosted.org/packages/c2/fd/202954b3f0eb896c53b7b6f07390851b1fd2ca84aa95880d7ae4f434c4ac/tabulate-0.8.3.tar.gz (46kB)
    100% |████████████████████████████████| 51kB 2.0MB/s 
Installing collected packages: tabulate, ansible-bender
  Running setup.py install for tabulate ... done
Successfully installed ansible-bender-0.4.0 tabulate-0.8.3
✔ ~/git/userspace-containerization/ogr [tests1 L|…38] 
12:44 $ sudo make build 
sudo ansible-bender build --build-volumes /home/jscotka/git/userspace-containerization/ogr:/src:Z -- ./recipe.yaml registry.fedoraproject.org/fedora:29 ogr
There was an error during execution: buildah command doesn't seem to be available on your system. Please follow the upstream instructions available at https://github.com/projectatomic/buildah/blob/master/install.md
make: *** [Makefile:7: build] Error 2

after installing: sudo dnf install buildah it finally works.