iqandreas / github-issues-import Goto Github PK

There is no way to specify an issue ID when creating a new issue, so unless precautions are taken, new issues will have a different id than the ones they were created in the image of (which means that relative links, example: #3, won't work in the target repo).

I want to notify the user if:

• Only some issues are imported, meaning there will be "gaps" causing issue ids to be off by that amount
• The target repository already has issues in it

I also want to detect if the id of the new issue's id will exactly match up even if imported (say, if new issues are added to the source repository, and those new issues are added after the original "import").

There have been several requests for supporting closed issues, I'm wondering how people's projects are set up, why they would want already resolved issues in the target repository?

Also, for the final product, should issues that are closed in the source repository also be closed in the target repository, or should there just be a little "notice" saying "This issue was closed 2013/10/26"?

This enhancement would make imports even easier.

Instead of:
>>> sys.argv = ["","-i","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94"]
Allow a range:

>>> import runpy
>>> import sys
>>> sys.argv = ["","-i","29-94"]

End result would be the same.
Of course this assumes that > 30 issues can be handled in one call...

Now it fails with some ugly stacktrace:

$ ./gh-issues-import.py -s lkde/git -t learn-it/scripting --issues 5
Traceback (most recent call last):
  File "./gh-issues-import.py", line 54, in init_config
    config_file = open(config_file_name)
IOError: [Errno 2] No such file or directory: '/home/user/src/github/github-issues-import/config.ini'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./gh-issues-import.py", line 369, in <module>
    issue_ids = init_config()
  File "./gh-issues-import.py", line 56, in init_config
    except FileNotFoundError:
NameError: global name 'FileNotFoundError' is not defined

Expected behaviour:

Ask username and password from terminal and proceed further.

When I confirm I want to continue.

ERROR: There was a problem importing the issues.
422 Unprocessable Entity
DETAILS: Validation Failed

Looking at the GitHub API, I understand that there are also some (a bit more) detailed errors but there are not displayed.

Any hint to be able to display those errors?

Thanks!

Took me a few to figure out that's why authentication was failing.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed posuere neque eros, eu varius nisl volutpat ac. In hac habitasse platea dictumst. Maecenas in leo eget massa placerat porttitor nec sed justo. Quisque laoreet egestas dui tristique dignissim. Integer porttitor venenatis porttitor. Vestibulum urna ipsum, aliquam accumsan ante vitae, lacinia laoreet lectus. Proin suscipit nisl aliquam mollis elementum. Nulla ac feugiat nunc. Cras a dictum dui, at pellentesque massa. Sed accumsan accumsan tortor, sed auctor massa rutrum eu. Nunc et mollis nisi. Vivamus ac nisl iaculis, commodo mi non, sagittis orci. Nullam consequat nec nisi quis volutpat. Etiam at purus sit amet nibh consequat dapibus id et arcu. Duis ut leo id dui pretium ullamcorper. Nullam eget suscipit lorem.

Vestibulum tristique tortor a erat sodales, sed molestie sapien accumsan. Ut convallis diam vitae nunc fringilla convallis. Curabitur malesuada risus a consequat pharetra. Curabitur commodo, nibh vitae ullamcorper dapibus, turpis risus suscipit risus, ut porta odio risus et mauris. Praesent hendrerit felis at eros cursus, sed pharetra ligula suscipit. Vestibulum faucibus velit id consectetur adipiscing. Morbi cursus pellentesque fermentum. Cras porta purus sed placerat tincidunt. Nunc imperdiet, leo et dictum vulputate, nisl risus molestie lacus, in tristique diam ipsum id sem. Phasellus sit amet tristique erat, id ornare urna. Sed sollicitudin, diam eget cursus mattis, purus turpis aliquet velit, a eleifend magna turpis vitae dui. Cras nec lectus odio. Donec non felis pharetra urna posuere scelerisque.

file:///storage/emulated/0/Android/data/com.teejay.trebedit/files/TrebEdit%20user%20files/Sample%20project%20-%20Acme/index.html

Found this very useful, but got this email when I used it.

On April 8th, 2020 at 03:56 (UTC) you used a password to access an endpoint through the GitHub API using IQAndreas/github-issues-import:

https://api.github.com/repositories/214032931/issues?state=open&direction=asc&page=1

Basic authentication using a password to the API is deprecated and will soon no longer work. Visit https://developer.github.com/changes/2020-02-14-deprecating-password-auth/ for more information around suggested workarounds and removal dates.

Thanks,
The GitHub Team

Today GitHub shipped issue transfers as a public beta on GitHub.com https://blog.github.com/changelog/2018-10-30-issue-transfer/. This allows a repository administrator to transfer an issue to another repository where the user also has repository administration privileges. This particular project does a couple things that GitHub does not at this time such as moving labels and milestones.

Check out the Help Docs for more details: https://help.github.com/articles/transferring-an-issue-to-another-repository/.

When you transfer an issue, comments, assignees, and issue timeline events are retained. The issue's labels, projects, and milestones are not retained, although you can see past activity in the issue's timeline. People or teams who are mentioned in the issue will receive a notification letting them know that the issue has been transferred to a new repository. The original URL redirects to the new issue's URL. People who don't have read permissions in the new repository will see a banner letting them know that the issue has been transferred to a new repository that they can't access.

Even though the script doesn't do anything "funny" with the user's credentials (which can be clearly seen since the code is open source) it's still not a good idea requiring a password in the script (and it's especially not a good idea to store that password in a plaintext file on your computer!)

I added the option of prompting for a password when the script, but I still would like the capability to use API keys instead.

I haven't tested to see if the script is able to import issues if the account running the script requires two-factor authorization. See:

• GitHub Help: About Two-Factor Authentication

Also, all the more reason to use an API key instead of storing using passwords.

• Add option to pass config file location as a command line argument. (f0fdcc1)
• Prevent any actual Python code from being able to be included in the config file (cd9f6fd)
• Allow the username and password to be passed in as arguments (f0fdcc1)
• Optionally, prompt the user for a username and password if none is provided in the config file. (06b6910)
• Allow for the option of using API keys rather than entering your password and username (as was done in https://github.com/IQAndreas/ghup/tree/dev-generate-token).
• Allow the option to import closed issues. I would recommend the flags --all, --open, --closed, or the default for the type of import.
• Add --verbose options
• Add --dry-run option (this would have made all this debugging easier)
• Add --ignore-labels and --ignore-milestones (48917c8)

When I move an issue, I typically want to close the issue in the old repository.

It does:

def import_label(source):
    data = {
        "name": source['name'],
        "color": source['color']
    }

    result_label = send_request('target', "labels", source)

but it should do:

    result_label = send_request('target', "labels", data)

Similar to #55, but with multiple pages of comments.

Please, indicate clearly in documentation what timestamps will be of imported messsages: either orignal ones or current time?

Alas, issues aren't as simple as having an open flag. Instead, there is an Events list for an issue which determines all this: http://developer.github.com/v3/issues/events/

Those notices should be added as comments inside of the issue (I would also like little fancy "CLOSED", "MERGED", etc images like GitHub does now).

As far as I know, there is no way to actually close an issue from the GitHub API, so hopefully these notices should suffice.

Surprisingly difficult, as there are so many to choose from, and it's difficult find non-legalese explanations for them.

I've checked #28 and remove all the existed milestones and labels, then execute the script to import, but it still failed again. The error is occurred at once after creating the first milestone successfully.

Any suggestion?? Or it's just a real bug??
Thank all.

Could you support Python 2? Even the latest yet-unreleased version of OS X doesn't come with python3.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus euismod dolor vitae ante posuere, et luctus tellus pretium. Sed tempus nisi sed velit ultrices fringilla. Mauris semper sapien ac mi lacinia, consequat vehicula urna rhoncus. Nullam eu porta massa. Maecenas placerat congue odio nec euismod. Etiam et euismod magna. Duis vitae mollis ante. Fusce tempor gravida ligula, quis viverra lectus suscipit a. Mauris ut ligula arcu. Etiam pulvinar placerat libero, non suscipit diam imperdiet pharetra. Maecenas mollis in purus at faucibus. Vestibulum convallis id justo eu pulvinar. Nunc ac est lectus.

Mauris et lectus non neque luctus egestas. Nulla id varius elit, ac aliquet elit. Donec risus velit, volutpat vel arcu in, aliquam dictum sapien. Quisque faucibus porttitor tincidunt. Aliquam congue purus at nisi laoreet aliquam. Proin vulputate, dui et viverra condimentum, sem quam ultrices felis, sit amet tempor turpis mauris nec libero. Vivamus sodales libero sed mi dapibus, vel dictum tellus tristique. Sed sollicitudin augue dictum, mollis risus vel, eleifend lacus. Nulla elit justo, porttitor non massa et, malesuada dignissim ante. Sed dapibus sodales massa sit amet eleifend. Nam tristique aliquam tortor in euismod. Donec facilisis, dui sit amet adipiscing placerat, erat neque consequat dolor, id semper eros mauris eu velit. Fusce a enim fermentum metus hendrerit fringilla a ac turpis. Nulla ligula ipsum, tincidunt quis fringilla vitae, vehicula vel dolor. Nam eu laoreet dolor. Nullam id quam ipsum.

Sed iaculis pellentesque sem sed euismod. Nunc ac euismod nulla. Sed sem orci, aliquam quis quam eget, sollicitudin sodales felis. Quisque nisi felis, iaculis vitae erat a, luctus molestie massa. Duis nibh lectus, dapibus et egestas nec, tristique eget purus. Sed in lorem euismod, hendrerit elit et, sagittis neque. Phasellus laoreet lectus justo, sed hendrerit augue tincidunt non.

Etiam euismod diam ligula, sed fermentum lorem sollicitudin in. Sed eu nisi velit. Proin in augue urna. Curabitur pulvinar nisl convallis ante ultrices egestas. Quisque eget auctor orci, ac tempus mi. Quisque a laoreet mi, ut suscipit purus. Maecenas lorem elit, placerat non congue faucibus, ultrices ut enim. Integer posuere suscipit ullamcorper. Mauris aliquam arcu eu tristique rutrum. Quisque adipiscing orci sit amet pretium elementum. Donec viverra diam eget lorem euismod vestibulum. In mattis elit mi, in auctor mauris tincidunt gravida. Etiam scelerisque pellentesque tellus, sed suscipit leo condimentum vitae. Proin eu mauris odio.

In et quam eu dolor sollicitudin mattis. Praesent eu mi eget dui iaculis posuere. Proin at gravida elit, eu placerat velit. Proin eleifend id risus sit amet ultricies. Sed ut neque ut nunc sodales aliquet. Nulla in leo et eros eleifend scelerisque. Nulla interdum justo at ornare feugiat. Donec viverra vehicula enim, eget ultricies lacus rhoncus id. Aenean posuere lectus at luctus laoreet. Sed volutpat commodo adipiscing. Cras laoreet sed velit et egestas. Ut imperdiet, elit ut convallis adipiscing, mauris ante suscipit nulla, id pellentesque ante eros nec elit. Maecenas nec tortor sodales, vehicula augue eget, luctus urna.

It would be great to have a feature to add assignee for issue when porting to target repo if assignees have been already added as collaborators in target repo.

I'm trying to import all open tickets from semantic-ui repository, but it seems that repo has too many tickets. I can't understand, is the program working and trying to import all that stuff or its just idle. I think it need lazy importing or something like that with better outputs during the process.

I received the following error: any ideas?

Loaded options from default config file in '[redacted]/config.ini'
y
You are about to add to 'glauth/glauth':
 * 11 new issues
 * 4 new comments
 * 0 new milestones
 * 4 new labels
Are you sure you wish to continue? [Y/n] ERROR: There was a problem importing the issues.
422 Unprocessable Entity
DETAILS: Validation Failed

If you have more than one page of labels, the import will fail by trying to create labels that already exist.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque feugiat faucibus nulla, non suscipit tortor. Nullam volutpat tempor arcu, vel varius orci sollicitudin vel. Aliquam erat volutpat. Nulla sagittis arcu quis varius condimentum. Donec enim justo, suscipit eget sollicitudin sed, condimentum gravida quam. Duis lobortis vitae quam at feugiat. Vestibulum porta iaculis eros, nec gravida felis dictum a. Ut nec nunc sed nisl varius bibendum. Sed in lacus molestie, dictum ipsum sit amet, ultricies erat. Quisque ac venenatis purus. Cras ullamcorper orci ac elementum congue. Mauris nisi sapien, posuere quis luctus iaculis, ornare a metus. Integer nec condimentum nisi. Maecenas imperdiet, odio vel gravida pulvinar, leo lorem lobortis ipsum, vitae posuere dui mauris ac leo.

Morbi molestie sodales dui eu gravida. In posuere pellentesque nibh. Nam gravida venenatis libero a aliquet. Suspendisse potenti. Morbi interdum est id tempus ultricies. Vestibulum posuere molestie lacus, id eleifend mi blandit eget. Integer et augue id erat auctor imperdiet.

To support Github Enterprise installs with self-signed certificates. It's a simple monkey-patch:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

I had an issue with a typographic quote in the title, got:

Traceback (most recent call last):
  File "gh-issues-import.py", line 387, in <module>
    import_issues(issues)
  File "gh-issues-import.py", line 356, in import_issues
    print("Successfully created issue '%s'" % result_issue['title'])
  File "c:\python33\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u201d' in position 44: character maps to <undefined>

github/issue-metrics@88570fc

I have encountered an strange behavior in github-issues-import when transferring an issue with many comments. If the issue in the source repository has >30 comments, only the first 30 comments in the issue will be transferred to the destination repository. github-issues-import does however detect the full number of comments in the source issue. I initially found this in a private repository, but I was able to reproduce the problem using the following invocation:

$ python3 gh-issues-import.py --no-config -u cbergman -p XXXXXX -s cbergman/source -t cbergman/destination -i 1
Ignoring default config file. You may be prompted for some missing settings.
You are about to add to 'cbergman/destination':
 * 1 new issues
 * 40 new comments
 * 0 new milestones
 * 0 new labels
Are you sure you wish to continue? [Y/n] Y
Successfully created issue 'source'
 > Successfully added 30 comments.

The relevant source and desitnation issues are here:
https://github.com/cbergman/source/issues/1
https://github.com/cbergman/destination/issues/1

Thanks in advance for looking into what might be causing this behavior.

Does this script handle two factor authentication? I think that might be why I have authentication issues.

Knock. Knock. The 18 pull requests are still waiting for long years. Who are responsible for review and merge procedure?

Everywöne löves ŭmláüts!

I ran this script and it failed because of the error described in #17 (I think), but because the script doesn't output anything indicating which issue caused the failure, I can't:

• diagnose the issue
• figure out how to get the remaining issues

A nice little touch which makes things clearer and easier to read.

The only problem will be making the image wrap around the text nicely, which I'm not sure if it's even possible.

Import issues is not working when source and target repo name is similar on forked repo. Throws an error.

For example:
Source: xxxxxx/SampleRepo
Target: yyyyyy/SampleRepo

Error Log:

Traceback (most recent call last):
  File "gh-issues-import.py", line 307, in 
    import_all_open_issues()
  File "gh-issues-import.py", line 297, in import_all_open_issues
    return import_issues(issues)
  File "gh-issues-import.py", line 184, in import_issues
    known_milestones = get_milestones(target_url)
  File "gh-issues-import.py", line 116, in get_milestones
    return send_get_request("%s/milestones?state=open" % url)
  File "gh-issues-import.py", line 111, in send_get_request
    response = urllib.request.urlopen(url)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 156, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 475, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 587, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 513, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 447, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 595, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 410: Gone

Currently dates and timestamps look like 2013-09-12T22:42:02Z (not a pretty sight). This obviously should be fixed.

I'm not sure if the dates also need to be adjusted in some way based on the user's current time zone, but I'm going to assume that's beyond the scope of what you can do in a GitHub comment.

Milestones have different issue IDs across repos (like imported issues do) which will cause problems in the target repo. Therefore, importing Milestones has been disabled until this has been fixed.

It seems like someone already has a fix for this, but I have to double check the code: https://github.com/tacker/tools/commit/7ccb67adac67ffe0d1e97cd1a1c2137910bba9fc#L0R93

While trying to migrate issues from a GitHub Enterprise to github.com, I run into

ERROR: There was a problem during authentication.
Double check that your username and password are correct, and that you have permission to read from or write to the specified repositories.

Weird enough, the milestones, labels, and the first 11 issues (with comments) are succesfully migrated, so I don't understand where the sudden 401 is coming from.

I'm working on simultaneously building Android and iOS versions of an app.
We're using Github issues to manage all our todo tasks (new features, bugs, etc)
We frequently have identical requirements on both the Android and iOS app.

It would be very convenient if there was a way to just copy issues from one repo to another.

After tweaking the config.ini like so:

[login]
username = brainstorm
password = somepwd

[source]
repository = INCF/biostar-central

[target]
repository = ialbert/biostar-central

I am getting the following traceback:

Loaded options from default config file in '/Users/roman/tmp/github-issues-import/config.ini'
Traceback (most recent call last):
  File "gh-issues-import.py", line 431, in <module>
    issues += get_issues_by_id('source', issue_ids)
  File "gh-issues-import.py", line 237, in get_issues_by_id
    issues.append(get_issue_by_id(which, int(issue_id)))
  File "gh-issues-import.py", line 231, in get_issue_by_id
    return send_request(which, "issues/%d" % issue_id)
  File "gh-issues-import.py", line 207, in send_request
    response = urllib.request.urlopen(req)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 153, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 455, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 478, in _open
    'unknown_open', req)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1257, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>

Related to #48. When a failure occurs, there doesn't seem to be a good way to resume an import. My guess is that I would wind up with all the successfully imported issues duplicated if I tried to run it again, and I also don't have a straightforward way to figure out which issues were imported successfully.

Maybe, at a minimum, the script could output a progress log so that it would be easier to determine which IDs are left that need to be imported? Even better would be support for some kind of --resume arg.

Likely related to #35 , however while transferred issues are identified as [CLOSED] they are not "closed" in GitHub and appear when filtering "issues:open" and vice versa for open issues. This may be a coincidence but the # of open issues imported to destination = # of closed issues in source (& vice versa).

Milestones completed in source were incomplete in destination.

Milestone issues are tagged as for a specific milestone, but looking via the milestone shows no issues.

Started out with 66 issues with milestones & comments in various states (closed/open) and ran

>>> import runpy
>>> import sys
>>> sys.argv = ["","-i","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94"]

to start.
First issues (30?) were copied over with the following traceback to request.py [file specific tracebacks removed]:

   raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    runpy.run_path('gh-issues-import.py', run_name='__main__')
  File "X:\DevApps\Python33\lib\runpy.py", line 231, in run_path
    mod_loader, pkg_name)
  File "X:\DevApps\Python33\lib\runpy.py", line 83, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "X:\DevApps\Python33\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "gh-issues-import.py", line 445, in <module>
    import_issues(issues)
  File "gh-issues-import.py", line 410, in import_issues
    result_comments = import_comments(issue['comments'], result_issue['number'])
  File "gh-issues-import.py", line 295, in import_comments
    result_comment = send_request('target', "issues/%s/comments" % issue_number, comment)
  File "gh-issues-import.py", line 215, in send_request
    sys.exit(http_error_messages[error.code])
SystemExit: ERROR: There was a problem during authentication.
Double check that your username and password are correct, and that you have permission to read from or write to the specified repositories.

So I attempted to chunk the remaining issues, yielding portions of success (copied issues/comments) but unfortunately not everything migrating as expected.

Great script with some tweaks it will be even better. Thanks!

This allows the source to be a private repository.

Code already exists for these changes:

• asolkar/mkorenkov-tools@de4eff4
• daniel-liuzzi/tools@8220ff2

Also, why am I not using urllib2? Was there ever a good reason for that decision?

Allow for better error handling for the following cases:

• There is a problem with the config options
• The GitHub API returns an error
• The target issue already exists in the repository (should at least warn)

I also want to prompt the user "You are about to import all 42 issues into hello-world. Are you sure you want to continue?"