kasyapwiki / bacteria

Bacteria

This repository contains all the work that was done as a part of enriching Hindi wikipedia. The aim is to deploy a large number of articles for different Bacteria in Hindi.

Stages of the project

The following ordered list will give an idea as to what stage the project currently is in:

• Scrape Data from Web sources
• Clean and Format the data
• Create a sample article
• Review of the sample article
• Work on feedback from review of sample article
• Review of the dataset
• Create template for article generation
• Review of the template
• Work on feedback from review of template
• Create the XML dump for all the Bacteria to be published

Folders

• Database: Contains all the dataset (csv) files that have been used for this project.
• Code: Contains all the code that has been used in the project.

Database

This folder contains the final dataset as well as some other dataset that have been used in the project:

• Scraped_db.csv: This is the dataset that has been directly scraped from BacDive.
• Scraped_db_added_attrib.csv: This file contains the English dataset that is used for article generation after merging multivalued attributes and adding some addtional data. The file contains data on 17245 bacteria and has 24 attributes.
• reflist.csv: This csv file contains a list of all the references from which we took out the data.
• Fina_eng_db : This csv file is the final English database with the refernces added for each bacteria.
• Fina_hindi_db : This csv file is the final Hindi database after translating and transliterating the final English database.

Code

This folder contains various files:

• manual_ref: Contains code which combines the multiple refernces of each article alongwith the url and text.
• merge: Contains code which merges all the multi-valued attributes with a proper delimiter such that it can be separated out when required. Also does some conversions and addition of attributes.
• ref_scrape: Contains the code used for extraction of references.
• transaltion and transliteration: Includes the code for translating and transliterating the main dataset.

Generating articles

• hindi_template: It contains the template for the structure of the article.
• xmlgen: It contains the backbone structure for conversion to XML.
• generation: it contains the code for generating XML dump of articles using the template and the dataset
• xmldump50: the XML dump file of intial 50 articles bacteria.

Contributors

Darshana Baruah [email protected]
Vineet Agarwal [email protected]