tonythomas01 / gdrive-to-commons

Tool to upload pictures from Google Drive to Wikimedia Commons ⚠️ Moved to Wikimedia Gerrit, so this repo is read only.

Google drive to Wikimedia Commons (Read only)

Please see: https://github.com/wikimedia/labs-tools-gdrive-to-commons

This tool is developed to Upload images directly from Google drive to Wikimedia Commons and is currently deployed at Toollabs.

⚠️ Contributing ⚠️

Development and codebase moved to Wikimedia Gerrit. This repo is read only now. Read how to contribute on the Gerrit Tutorial. Thank you!

Dependencies

We use python 3.5.3 on production. Make sure you have this installed on your machine, or use pyenv as described later in this documentation under steps for local development.

Pre-requisites for local development

The tool authenticates to the Wikimedia cluster using Wikimedia OAuth and to Google using Google OAuth. Hence, we need a couple of secret keys in your local_settings.py to start development.

1. SOCIAL_AUTH_MEDIAWIKI_KEY & SOCIAL_AUTH_MEDIAWIKI_SECRET: You can request for an OAuth client using Wikimedia:OAuth. Once created, you will get both keys. Remember that you set the right Mediawiki OAuth callback URL.
2. GOOGLE_APP_ID: We need an application running on Google cloud to authorize and access Google drive. You can create your application at Google cloud console. GOOGLE_APP_ID is your project id.
3. GOOGLE_CLIENT_ID: You can create a new OAuth 2.0 Client at Google cloud credentials.
4. GOOGLE_API_DEV_KEY: You need to enable the Google Picker API on Google cloud console. Once enabled, a dev key is generated, and you can find it on your Google cloud credentials.

For Wikimedia developers, you can find more information on development keys here

And also, the Google Drive to Wikimedia Commons Developer credentials are provided at the following link: https://phabricator.wikimedia.org/P10014 .
Developers need not create new secret keys and can use the credentials given in the above link as instructed later in this documentation under steps for local development.

Steps for local development

1. Create a local fork of https://github.com/tonythomas01/gdrive_to_commons using your GitHub account by running the following command in terminal:

   $  git clone https://github.com/username/gdrive-to-commons.git
   

2. If homebrew is not installed before, install homebrew package manager by following the instructions given in the below link:
   https://docs.brew.sh/Homebrew-on-Linux

3. Install pyenv and virtual manager by running the following commands in terminal:

   $ brew install pyenv
   $ brew install pyenv-virtualenv
   $ export PATH="$HOME/.pyenv/bin:$PATH"   
   $ eval "$(pyenv init -)"
   $ eval "$(pyenv virtualenv-init -)"
   $ pyenv install 3.5.3
   $ eval "$(pyenv init -)"
   

4. navigate into the repository by running the following command in terminal:

   $ cd gdrive_to_commons   
   

   Now you are in the gdrive_to_commons directory and all the following commands are run in this directory.

5. Create pyenv-virtualenv by running the following command in terminal:

   gdrive_to_commons/$ pyenv virtualenv 3.5.3 gdrive-env-3.5.3   
   

   This will create a pyenv-virtualenv for you and probably place it on your ~/home/<username>/.pyenv/versions/.

6. Activate the virtual environment manually by running the following command:

   gdrive_to_commons/$ source ~/.pyenv/versions/gdrive-env-3.5.3/bin/activate
   

   or even better:

   gdrive_to_commons/$ pyenv activate gdrive-env-3.5.3
   

   or,
   there are better ways to do this if you follow https://github.com/pyenv/pyenv-virtualenv.

   Now you are in the right environment if your terminal shows:

   (gdrive-env-3.5.3) gdrive_to_commons/$
   

   All the following commands are run in this environment and in this directory only.

7. Install dependencies by running the following command:

   (gdrive-env-3.5.3) gdrive_to_commons/$ pip install -r requirements.txt
   

8. We use pre-commit hooks to format code. install precommit framework by running the following command: (gdrive-env-3.5.3) gdrive_to_commons/$ pip install pre-commit
   Now, install our pre-commit hooks using: (gdrive-env-3.5.3) gdrive_to_commons/$ pre-commit install

9. There are some localsettings you need to have as part of running the server. You can copy a template using: (gdrive-env-3.5.3) gdrive_to_commons/$ cp gdrive_to_commons/local_settings_sample.py gdrive_to_commons/local_settings.py
   Few credentials have to be modified in local_setting.py
   The credentials that are to be pasted in the local_settings.py are given in the following link:
   https://phabricator.wikimedia.org/P10014.
   Paste the Google Drive to Wikimedia Commons Dev credentials in local_settings.py

10. Run the Django standard runserver steps:

(gdrive-env-3.5.3) gdrive_to_commons/$ python manage.py makemigrations
(gdrive-env-3.5.3) gdrive_to_commons/$ python manage.py migrate
(gdrive-env-3.5.3) gdrive_to_commons/$ python manage.py collectstatic
(gdrive-env-3.5.3) gdrive_to_commons/$ python manage.py runserver localhost:8000

or even better, run it from pyCharm using your debugger.

Note: Only paste the command after $ in terminal while setting up and do not paste the whole line which might cause error.

gdrive-to-commons's Issues

Arrows not showing up on latest Ubuntu/Firefox (F0DA)

Ubuntu Linux 2019.04 Firefox 66.0.5 64-bit

Thanks!

Compress images before upload

Description

Caring about file size on internet pages it might be a good idea to compress images lossless when uploading them to Wikimedia Commons. As described on this Google site: https://web.dev/use-imagemin-to-compress-images it is fine to compress images losslessly.

Current State

Images are not uploaded compressed.

Expected State

Images should be uploaded compressed. This can be achieved with imagemin (https://github.com/imagemin/) for example. I do not know if there is a Python solution/wrapper which could be applied here.

The process would be:

1. Selection of files
2. Retrieve files from Google Drive
3. Compressing of images
4. Upload to Wikimedia Commons

Update setup documentations

Missing:

1. Development environment setup information.
2. Deployment setup information.

Fix file thumbnails on the review screen

Currently, for all file thumbnails, we are using iconUrl returned in the Google API callback for all files. But, this is not the actual file thumbnail and URL to a default icon which is the same for all files.

Post upload, show a better screen listing uploaded files

Right now, we have this one:

can we change it a bit like this:

1. Make the second column bigger, and maybe just have 2 columns there and make the name a link which opens in a new page ? Like that, we can totally avoid the 3rd column.
2. Lets see if we can take out a link to the file instead ? for eg, https://test.wikipedia.org/wiki/File:Tail_latency_comparison_between_DS,_OS_and_FS_at_varying_latency_sleep_bxp-1515415386.png and not the upload file iteslf ?

After login, files are listed in a random order

The google drive to commons uploader tool has been tested thoroughly on several operating systems (Windows 10, Ubuntu 18.04 and mac OS and Android 10 (Mobile web browser))
After we sign in with our google account and when we need to select the images, files are being shown randomly and the original google drive user interface (I mean the UI which we see when we originally use google drive) is not being shown. This increases the difficulty of choosing a particular image which the user might have stored in a particular folder according to himself.

Here is the screenshot where we can only see random images(files only) only and no folders are shown.

Support editing of media files in Step 3

After the files have been selected from Google Drive in Step 2, we currently show them for review in Step 3. We display a thumbnail, file name, description, and edit buttons next to each file information. The edit buttons are currently not functional. It would be a nice addition to support editing of file titles and description before uploading them to Wikimedia Commons.

Get Google OAuth app verified

Missing

• Application Privacy Policy link on our app. Create from https://app-privacy-policy-generator.firebaseapp.com/ and lets put it on a footer link ?
• Application Terms of Service link.

Token error: Missing unauthorized token

While on my dev setup, I am getting "AuthTokenError at /oauth/complete/mediawiki/", precisely a missing unauthorized token error.
@tonythomas01 Is it because we are using a different MediaWiki OAuth consumer and I have some old credentials?

POST via Ajax to uploads API fileLists

I expect something like

{
  "fileList": [
    {
      "id": "fasdfasdfa",
      "name": "file_name"
    },
    {
      "id": "fasdfasdfa",
      "name": "file_name"
    }
  ]
  "token": "1231fasdfasdf"
}

on http://localhost:8000/uploads/

You can get the fielList from data.docs (index.html:74) right now.
token is oauthToken. Just use HTTP POST and we should be good

Make the tool more responsive and mobile friendly

I tried using this tool on a mobile device and I noticed that the fonts are way too small that I basically had to pinch and zoom to be able to read the words.

I think it is important to make it mobile friendly

Build a functional stats counter for uploads

We are storing information about the number of files uploaded by the app. it would be a nice thingy to show a counter or something like that on the home page ?

1. Track the counts at the backend in a table.
2. Show them up on the homepage.

Button 'I've read and accept the privacy policy' turns completely blue when clicked

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://tools.wmflabs.org/google-drive-photos-to-commons/ '
2. Click on 'I've read and accept the privacy policy'
3. The button turns completely blue

Expected behavior
The button should show a tick rather than turning completely blue

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu 18.04
• Browser [firefox]
• Version [73.0.1 (64 bit)]

Additional context
The button works fine on Chrome

Potentially leaking keys, secrets and tokens

Hi Tony, I'm not 100% sure what this project is about, but I just quickly browsing through it, quite somes keys, secrets and tokens of Google Accounts and Mediawiki are exposed.

You might want to revoke those before something bad happens https://github.com/tonythomas01/gdrive_to_commons/blob/master/gdrive_to_commons/local_settings_sample.py

Change tool name on Toolforge

@tonythomas01 google-drive-photos-to-commons is a really long name for a tool. We should consider changing it to something like gdrive2commons.

Show a loading spinner when uploading images

Problem
I'm always confused on what is happening when I click on the button "Upload to Wikimedia Common" because button is disabled and I don't see any indication that a request is actually processing

Solution
While the request to upload is being processed, display a loading spinner on the disabled button to show the user that the server is actually processing a request

Ask for more information for uploaded files (comment)

There is a bit more information that https://commons.wikimedia.org/wiki/Special:UploadWizard asks for at the time of uploading a file. We have not considered this in the tool as of now.

It might be useful to allow users to choose, for example, a license, category, date work was created or published, etc.

Create documentation to configure app to upload to a local test wiki

We should've done this earlier, but now that we are uploading a lot of things outside (say to test and commons), I would suggest we have this setup. One would argue:

1. Setup a locally running medaiwiki instance with file upload capabilities. Lets use vagrant as it is easy to setup.
2. Enable roles OAuth maybe ? Register a consumer in this dev wiki to point to your localhost:8000 so your local gdrive app can authenticate.
3. Try uploading, and see that the file got uploaded to dev-wiki.

Would be easy to test, and quite good if we can have this documented. Thank you!

Tool is not working in internet explorer

The bug
The tool is not working in internet explorer as expected.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://tools.wmflabs.org/google-drive-photos-to-commons/
2. Accept privacy policy and click on the button "Start Uploading"

Expected behavior
The user should be redirected to the "Login to Wikimedia Commons" button page

Actual Behavior
The user is being redirected to an abnormal user interface where three buttons are being displayed at same time as shown below.

Desktop:

• OS: Windows 10
• Internet Explorer
• Version 11.657.18362.0

style: Lets make "Date of Creation (first published)" column header a bit nicer ?

This one looks a bit weird now. How will we make it fit with the date input column width ?

Inform the users that they can edit the details of the image and addition of details

After the user selects the pictures, they can view the list of images they have selected along with their title, description and option to unselect.
But if a normal user uses this tool, it is difficult for them to find out that they can edit the title and add the description and this might be a disadvantage for the user and the feature is not being utilised by the user.

Also as mentioned already,
More metadata fields can be added to the image. The important metadata for a image might be:
Date image is created
Categories
Captions
License
Image type
Width and height of the image
Hashtags for the image etc.

The expected solution for the above mentioned issue is:
The user should be informed that they can edit the details of the image by giving user interface instructions and the important metadata fields can also be added.

Phab link: https://phabricator.wikimedia.org/T247369

Move the repo to labs/tools in Gerrit

@tonythomas01 If we are considering to do some more work on this app, and make it available for users and contributors, it would be nice to host it in Gerrit under labs/tools, etc.

Separate javascript code from HTML file

Well, it was a Hackathon, but this file is really messed up https://github.com/tonythomas01/gdrive_to_commons/blob/master/uploader/templates/base/base.html ;-)

All the javascript code needs to be moved out in a separate file.

Frontend code cleanup

In addition to #9, do some more frontend code cleanup. Run HTML, JS, CSS files through relevant linters and fix corresponding issues.

Button "choose photos from google drive" not working

To Reproduce
Steps to reproduce the behavior:

1. Login to Wikimedia commons with meta account
2. Select the "CHOOSE PHOTOS FROM GOOGLE DRIVE" action button

Actual behavior
After clicking on the "CHOOSE PHOTOS FROM GOOGLE DRIVE" button, there is no action taken, that is, it is stuck. After click on this multiple times also it is not showing any response.

Expected behavior
After clicking on the "CHOOSE PHOTOS FROM GOOGLE DRIVE" button, it should show us the photos from our google drive and should let us select the photos.

Screenshots

Desktop:

• OS: Ubuntu 18.04.3 LTS
• Browser: Firefox
• Version: 70.0.1 (64-bit)

Document the use of Prettier in the project

git commit generated the following output for me without indicating which files are problematic and any hints for fixing the errors in them. Took some time to realize, that I've been happily ignoring the keyword "Failed" in the log and the fact that the git commit is failing.

Srish:gdrive_to_commons srish$ git commit -m "improved the main screen design"
black................................................(no files to check)Skipped
prettier.................................................................Failed
hookid: prettier

Files were modified by this hook. Additional output:

assets/css/style.css 106ms
uploader/templates/home.html 302ms

It would be useful to document the steps for using Prettier (https://prettier.io/) before pushing changes, perhaps in the README for now, but in the contribution guidelines in the future. I imagine, the steps would be:

• npm install prettier # Install Prettier
• prettier --check "*.html" #Run Prettier through all HTML files
• Fix errors...

Verify that images used in the application are not copyright infringed

We use Google logos couple of times in the application. An expert in this field should verify that if this is legal and we would not end up in some trouble for that.

Create setup guide for windows users

Is your feature request related to a problem? Please describe.
The project setup guide was created without consideration for non-linux users. while it might be easy for experienced contributors to pick up, newbies will find it harder to understand what is going on as the commands won't be working on their system and this can potentially scare them away

Describe the solution you'd like

• create setup guide for windows users
• rename the existing setup guide to indicate that it is only for Linux based systems

Describe alternatives you've considered
An alternative would be to combine both windows and Linux setup guides into a single file. It all depends on the most desired approach, the most important thing is windows users seeing what they could be looking at while setting up their local environment for the project

Finish deploy on Wikimedia Toolforge

Finish deployment of the project at https://tools.wmflabs.org/google-drive-photos-to-commons

Improve the accessibility of this tool

Is your feature request related to a problem? Please describe.
Running lighthouse audit pointed out that this tool has an accessibility rating of 74%.
that should not be so given uncomplicated this tool's dom is.

Describe the solution you'd like
Increase the accessibility audit rating to 100% or something close.
This will make things easy for people who temporarily or permanently finds it hard to access this tool because of some disabilities

Upload from Google drive to wikimedia stuck at 75%

I tried using this tool on my mobile phone as well as on my laptop and in both cases, it doesn't move beyond 75%. Maybe this has something to do with my account but I would love it if someone can confirm.

To Reproduce
Steps to reproduce the behavior:

1. Open up the tool on either your mobile phone or laptop and try uploading from Google drive to wikimedia

Expected behavior
The upload should complete successfully but it isn't

Image

Desktop (please complete the following information):

• OS: Windows 7 (×86)
• Browser Chrome
• Version 80

Smartphone (please complete the following information):

• Device: Gionee M7 power
• OS: Android 7.1.1
• Browser Chrome
• Version 79

Images are not being uploaded to creative commons from google drive

To Reproduce
Steps to reproduce the behavior:
Login to Wikimedia commons with meta account
Login to google drive with google account
Select images to be uploaded from google drive
Select the "Upload to Wikimedia commons" action button

Actual Results:
After clicking on the "Upload to Wikimedia commons" button, the button gets a bit transparent but there is no action taken, that is, the images are not being uploaded to commons and also if we again click on that button it gets back to it's original state and there is no uploading happening. Hence when we click the upload button, the graphic of button is only hanged but there is nothing else happening.

Expected Results:
After clicking in the "Upload to Wikimedia commons" button, the upload should start and images should be uploaded to Wikimedia commons from google drive.

Testing done on Windows 10, Ubuntu 18.04, mac OS and Android 10(Google chrome Mobile browser)

Improve the design of the main screen that has agreement info

The current design of the main screen recently added is inconsistent with the other three screens and the information in Step 1-3 boxes are not relevant for the user.

Create a logo for the app

Required to get Google OAuth consent. @srish can you take care of this with your UX skills ?

Handle errors during upload process and show user friendly messages with modal

Description
When the user tries uploading pictures to wikimedia commons and an error occurs, nothing is shown to the user. An example of an error I experience was mwclient.errors.APIError: ('fileexists-no-change', 'The upload is an exact duplicate of the current version of [[:File:Registrar.jpg]].', 'See https://test.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.')

To Reproduce
Steps to reproduce the behavior:

1. Go to https://tools.wmflabs.org/google-drive-photos-to-commons/
2. Accept privacy policy and click on the button "Start uploading"
3. Login to wikimedia commons
4. Click on button "Choose photos from Google Drive"
5. Select a particular photo from your PC and upload
6. Click on button "Upload More" and select the same photo as selected in step 5, then upload

Expected behavior
The user receives a well structured message telling him/her what the error is (in this case for example "You already uploaded file to wikimedia commons").

Actual behavior
No feedback. The user has no idea about the error

Desktop

• OS: Windows 10
• Browser: Mozilla firefox
• Version: 74.0

Not able to select multiple images

On Windows 10, Ubuntu 18.04, mac OS (Desktop Web Browsers)
The user is not able to select multiple images from google drive according to their wish. The only way to select multiple files is using the selection tool with cursor but in this way the user can select only consecutive images and they can't select random images.

On Android 10 (Mobile browser)
The user is not able to select multiple images from google drive according to their wish. They can't even select consecutive multiple files because there is no selection tool available in mobiles. Hence, the user can upload only one photo at a time.

Users who want to upload 2 or more than 2 pictures may not like the way of uploading only one picture at a time and most probably won't like to repeat the whole process for uploading all the pictures.

See: https://phabricator.wikimedia.org/T247367