![]() |
|
|
|
|
|
|
The program it allows you to clone repositories from github in bulk and store them in specific directories from a csv file. Aditionally it saves the data of every student & course into a json with the name of the course.
All this is possible by the use of Pandas library and dataframes to manipulate the data, sorting and filtering the courses, students and repositories to get a list of dataframes, one for each course with all the students data sorted by course, surname and name and then, it will create a Pie Chart, using Matplotlib Pyplot, with the percentages of students categorized according to their course whose repositories have been cloned. Then will compare and generates a CSV file indicating which files are similar to each other and finally will write the info in a Database using SQLite3.
At the end of the clonning, the program will download the files of each student and save them in the directory of the course that they belong to.
like this:
Courses & students directories |
---|
![]() |
And the JSON generates with the data of the students and courses will be like this:
[
{
"Temporal Sign":"2022\/02\/13 10:26:52 p.\u00a0m. GMT-3",
"Name":"Heracles",
"Surname":"Grecian DemiGod",
"Division":"1A - Professor 1 - Helper 3",
"Student ID":"999999",
"E-Mail":"[email protected]",
"Link to repository":"https:\/\/github.com\/caidevOficial\/CaidevOficial"
},
{
"Temporal Sign":"2022\/02\/13 10:26:52 p.\u00a0m. GMT-3",
"Name":"Hades",
"Surname":"Grecian God",
"Division":"1A - Professor 1 - Helper 3",
"Student ID":"111111",
"E-Mail":"[email protected]",
"Link to repository":"https:\/\/github.com\/caidevOficial\/CaidevOficial.git"
},
{
"Temporal Sign":"2022\/02\/13 10:26:52 p.\u00a0m. GMT-3",
"Name":"Odin",
"Surname":"Nordic God",
"Division":"1A - Professor 1 - Helper 3",
"Student ID":"777777",
"E-Mail":"[email protected]",
"Link to repository":"https:\/\/github.com\/caidevOficial\/Python_RepositoryCloner.git"
},
{
"Temporal Sign":"2022\/02\/13 10:26:52 p.\u00a0m. GMT-3",
"Name":"Valhalla",
"Surname":"Nordic Reign",
"Division":"1A - Professor 1 - Helper 3",
"Student ID":"999999",
"E-Mail":"[email protected]",
"Link to repository":"https:\/\/github.com\/caidevOficial\/CaidevOficial"
}
]
Meanwhile the program is cloning the repositories, the console will show messages like showns below:
Console Messages |
---|
![]() |
When finish, you look a final message (with the elapsed time of the execution) like this:
Console Final Message |
---|
![]() |
At the end of the execution, the program will download the files of every student and save them in the directory of the course that they belong to. Additionally, the program will generate a JSON with the data of the students and courses and it will generate a Pie Chart with the percentage of students that have downloaded the repositories...
Like the image below:
Example Pie Chart |
---|
![]() |
- 1st Column: Date time. [it isn't used yet.]
- 2nd Column: Student Name
- 3rd Column: Student Surname
- 4th Column: Student Division
- 5th Column: Student ID. [it isn't used yet.]
- 6th Column: Student E-mail. [it isn't used yet.]
- 7th Column: Repository Name to download (It could skip the '.git' part)
Like this:
"Marca temporal","Nombre/s","Apellido/s","División","DNI / Legajo","E-Mail","Link al repositorio"
"2022/02/13 10:26:52 p. m. GMT-3","Neptune","Romane God","1G - Professor 1 - Helper 1","222222","[email protected]","https://github.com/caidevOficial/SPD2022_TPS.git"
"2022/02/13 10:26:52 p. m. GMT-3","Poseidon","Grecian God","1F - Professor 2 - Helper 2","333333","[email protected]","https://github.com/caidevOficial/Python_ITBA_IEEE.git"
"2022/02/13 10:26:52 p. m. GMT-3","Hades","Grecian God","1F - Professor 2 - Helper 2","111111","[email protected]","https://github.com/caidevOficial/CaidevOficial.git"
In order to use this Cloner, you should configure the file configs.json with your Github API's information as shown below.
{
"repo_cloner": {
"Github": {
"URL": "https://api.github.com/repos",
"USER": "CaidevOficial",
"REPO": "Python_copydetector",
"BRANCH": "master"
},
"DataFrame": {
"Fields": {
"Date": "Temporal Sign",
"Name": "Name",
"Surname": "Surname",
"Course": "Division",
"ID": "Student ID",
"Email": "E-Mail",
"GitLink": "Link to repository"
}
},
"Files": {
"Dir_Plots_img": "./Plot_Images",
"Dir_Cloned_Repos": "./Repositories",
"Dir_Statistics": "./Statistics"
}
}
}
for example:
{
{
"URL": "https://api.github.com/repos",
"USER": "CaidevOficial",
"REPO": "Python_RepositoryCloner",
"BRANCH": "main"
},
"DataFrame": {
"Fields": {
"Date": "Marca temporal",
"Name": "Nombre/s",
"Surname": "Apellido/s",
"Course": "División",
"ID": "DNI / Legajo",
"Email": "E-Mail",
"GitLink": "Link al repositorio"
}
},
"Files": {
"Dir_Plots_img": "./Plot_Images",
"Dir_Cloned_Repos": "./Repositories"
}
}
Then the code will make the link like:
https://api.github.com/repos/CaidevOficial/Python_RepositoryCloner/commits/main
This way the program will take the 'Date' of the last commit of the branch 'main' and will use it to create the folder with the name of the repository. Obviously, the repository MUST BE PUBLIC, otherwise the program won't be able to access its API.
Regarding the 'DataFrame' Key, al the keys inside are configured to use them with a 'csv' file with at least theses columns. [Could have more columns, but it's not necessary for us.]
Finally, respect the 'Files' Key, where you can configure the directory where the plots will be saved and the directory where the cloned repositories will be saved.
For our example, the columns of the csv file are:
Marca Temporal | Nombre/s | Apellido/s | División | DNI / Legajo | Link al repositorio | |
---|---|---|---|---|---|---|
Compares files with configurable extensions (for example: '.ino', '.cpp', '.c', '.h') in different directories and generates a CSV file indicating which files are similar to each other.
- Ignores language reserved words like break, include, {,},\n,\t.
The file config.json has the configuration of the script, edit it in case you want to calibrate it according to need.
{
"copy_detector": {
"configs": {
"script": {
"percentage": 60,
"filename_output": "./possible_copies.csv",
"sort_by_percent_desc": false,
"files_sufix": [".cpp", ".ino"],
"excluded_files": ["spect.c", "spects.c"]
},
"database": {
"name": "copies_db",
"table_name": "students_copies",
"delete_before_insert": true,
"paths": {
"db_file": "./modules/database/copies_db.db",
"DDL": {
"create": "./modules/database/queries/DDL/create.sql",
"drop": "./modules/database/queries/DDL/drop.sql"
},
"DML": {
"delete": "./modules/database/queries/DML/delete.sql",
"insert": "./modules/database/queries/DML/insert.sql",
"select": "./modules/database/queries/DDL/select.sql",
"update": "./modules/database/queries/DML/update.sql"
}
}
}
}
}
}
Regarding to the scripts key section:
"script": {
"percentage": 60,
"filename_output": "./possible_copies.csv",
"sort_by_percent_desc": false,
"files_sufix": [".cpp", ".ino"],
"excluded_files": ["spect.c", "spects.c"]
}
- the key percentage indicates that from 60% similarity, it will be considered a copy
- The key filename_output will indicate the name and output directory of the file which will contain the data of the copies
- The key sort_by_percent_desc will indicate, if set to True, that the data in the output file will be ordered in descending order according to copy percentage. Otherwise, it will be ordered as the copies are found.
- The key files_sufix will indicate a list of the suffixes of the files you want to compare.
- The key excluded_files will indicate the files u want to exclude from the comparation (can be empty)
Regarding to the database key section:
"database": {
"name": "copies_db",
"table_name": "students_copies",
"delete_before_insert": true,
"paths": {
"db_file": "./modules/database/copies_db.db",
"DDL": {
"create": "./modules/database/queries/DDL/create.sql",
"drop": "./modules/database/queries/DDL/drop.sql"
},
"DML":{
"insert": "./modules/database/queries/DML/insert.sql",
"delete": "./modules/database/queries/DML/delete.sql",
"update": "./modules/database/queries/DML/update.sql",
"select": "./modules/database/queries/DDL/select.sql"
}
}
}
- The key name: contains the name of the database (file) where the table will be stored.
- The key table_name: contains the name of the table in which the records will be written.
- The key delete_before_insert: will be true if before adding new records you want to delete the previous ones, otherwise it will be false.
Regarding the Paths
- The key db_file: will be the path where the file generated with the script database will be saved.
- The key DDL: will contain within the paths of the DDL queries that the script will use.
- The key DML: will contain within the paths of the DML queries that the script will use.
- Install requirements contained in the file requirements.txt
pip install -r requirements.txt
- Clone this repository and execute the script.
$ cd this_repository_dir
$ python3 main.py /path/to/directory
![]() |
Python |
|
Pandas |
|
Numpy |
|
MatPlotLib |
|
SQLite |
|
VSCode |
LICENSE | |
---|---|
Git Repository Cloner 2022 | |
License | Author |
[GNU General Public License V3] | [Facundo Falcone - CaidevOficial] |
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see GNU Licenses. |
|
🤴 Facu Falcone - Data Engineer |
|
|
|
|