shabados / database

A digital representation of Sikh Bani and other Panthic texts with a public logbook of sangat-sourced corrections.

Database

A digital representation of Sikh Bani and other Panthic texts with a public logbook of sangat-sourced corrections

Table of Contents

• Introduction
  • Features
  • Screenshots
    • Proofreading Flow [GIF]
    • Database Schema Layout
    • NPM JS package API [GIF]
• Community
• Contributing
• People
• Feedback
• Related Projects
• Code of Conduct
• License
• Gurbani and Panthic Compositions
  • Sri Guru Granth Sahib Ji
  • Sri Dasam Granth
  • Bhai Gurdaas Ji
  • Bhai Nand Laal Ji
• Acknowledgments

Introduction

Shabad OS Database is an open source project with ongoing development, project management, and marketing made possible entirely by the support and dedication of unpaid volunteers.

Features

• Open-source
  • Content, history, and modifications are publicly accessible
  • Transparency through open-source is more accountable (i.e. anyone can verify integrity)
  • Duplication of effort among different parties is reduced
  • Sustainability and longevity of project is improved
• Quality / Accuracy
  • Backed by real, physical sources (no controversy)
  • Digitally represents what was written/printed
  • Photographic evidence continuously reviewed for accuracy
  • Multiple sources used to digitize gurbani, panthic texts, language translations, and exegesis
• Modern database schema
  • Designed to capture data effectively
  • Ideal for developing apps, analyzing data, or undertaking research
• API
  • The @shabados/database npm JS package can query the database without SQL

Screenshots

Proofreading Flow [GIF]

Shabad OS Database follows a high-quality, publicly accessible, verifiable, and accountable proofreading process. The following GIF contains:

1. PR for Ganj Nama
2. Git diff of all changes
3. Commit message with citation)

Database Schema Layout

This image may be out of date. View the schema on SQLDBM for the most up-to-date file.

NPM JS package API [GIF]

Example of querying database without SQL:

Community

Get updates on Shabad OS and chat with the project maintainers and community members.

• Follow Shabad OS on Instagram
• Follow Shabad OS on Twitter.
• Join the official Slack channel.

Contributing

There are multiple ways to contribute whether you are a user or developer. For example:

• Review physical sources and compare them to what has been digitized. Learn more about proofreading >
• Submit proofreading tickets. Help us verify whether they are legit. Explore our proofreading issue tracker >
• Submit bugs and feature requests.
• Review documentation and make pull requests for anything from typos to new content.
• Give feedback on the onboarding process to make it easier for others to join the project.

If you're interested in contributing to the source code of Database, then please see Contributing Guidelines.

People

The original code was written by the current lead maintainer, Harjot Singh (@harjot1singh).

"Thank you!" to all the volunteers who've contributed to Database.

Feedback

• Ask questions and get help in our community chat via Slack
• Upvote popular feature requests using the thumbs-up/+1 reaction on the first post of a feature request
• Follow @shabad_os on Instagram and @shabad_os on Twitter and let us know what you think!

Related Projects

Projects in the Shabad OS ecosystem of free and open source software include:

• Viewer
• Presenter
• Mobile
• Gurmukhi Utils

Code of Conduct

Please note that this project is released under the Contributor Covenant. By participating in this project you agree to abide by its terms. Our intention is to signal a safe open-source community by welcoming all people to contribute, and pledging in return to value them as whole human beings and to foster an atmosphere of kindness, cooperation, and understanding.

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

The Contributor Covenant

License

NOTE: Applies to code and content resting outside of the data folder.

This project is under v3 of the GPL. It is similar to the Golden Rule: do unto others as you would have them do unto you. In exchange for benefitting from the work completed in this repo, others must share their derivative work under v3 of the GPL.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

Gurbani and Panthic Compositions

NOTE: Applies to different texts inside the data folder, generated inside the build folder, and as releases (e.g. GitHub, npm).

As typical of many old, historical works, most gurbani and panthic texts are free of known copyright restrictions. We identify it as being in the public domain as a work of factual compilation with originality. The honor and reputation of the original works are to be maintained. Derogatory treatments (including adding to, deleting from, altering of, or adapting) the words in a way that distorts or mutilates the original work is forbidden. That is why, whenever possible, physical sources are used to determine the digital representation of these works as to avoid any controversy.

Citations for accuracy and review are based on MLA formatting. More often than not, a condensed version of the following is used when citing changes to any of the sources in the database:

Author. Title of source. Title of container, Other contributors, Version, Number, Publisher, City of Publication, Publication Date, Location. Date of access.

Sri Guru Granth Sahib Ji

Shabadaarth. Vol. 1-4, SGPC, Sri Amritsar, 2009-2012, www.archive.org/details/ShabdaarthSriGuruGranthSahibJi-Part1. Accessed 22 Jan 2019.

Vishraam / Pauses:

Dauder, Jaswant S. Sri Guru Granth Sahib Ji Shudh Ucharan, Arth Paath Vishraam Pothi, Sri Damdami Taksaal. Oct 2015, www.gursevak.com/drupal7/sites/default/files/Vishram%20Sri%20Guru%20Granth%20Sahib%20Jee.pdf. Accessed 18 May 2020.

Arth / Translations:

Singh, Sahib. Sri Guru Granth Darpan. Vol. 1-10, Raaj Publishers, Hoshiarpur/Jalandhar, 1972, www.archive.org/details/SriGuruGranthSahibDarpan-Volume1. Accessed 18 May 2020.

Singh, Badan. Fridkot Wala Teeka. Bh. Baljinder Singh Rara Sahib, www.ik13.com/PDFS/Fridkot_Wala_Teeka.pdf. Accessed 24 May 2020.

Singh, Manmohan. Sri Guru Granth Sahib (English & Punjabi Translation). Vol 1-8, 6th & 8th ed., SGPC, Amritsar, 2006-2009. www.archive.org/details/SriGuruGranthSahibEnglishAndPunjabiTranslation-Vol.1. Accessed 24 May 2020.

Sri Dasam Granth

ਜਾਪੁ ਸਾਹਿਬ, ਤ੍ਵ ਪ੍ਰਸਾਦਿ ਸਵਯੇ (ਸ੍ਰਾਵਗ ਸੁਧ), ਕਬਿਯੋਬਾਚ ਬੇਨਤੀ ਚੌਪਈ, ਤ੍ਵ ਪ੍ਰਸਾਦਿ ਸ੍ਵਯੇ (ਦੀਨਨ ਕੀ), ਚਉਬੀਸ ਅਵਤਾਰ (Ending Savaiya and Dohira), ਵਾਰ ਸ੍ਰੀ ਭਗਉਤੀ ਜੀ ਕੀ (First Pauri):

Nitnem Te Hor Baniaa(n). SGPC, Sri Amritsar, May 2017, www.drive.google.com/open?id=1AyQLCQgrNznDxI9Vby3knE4Piyrdsnxs. Accessed 22 Jan 2019.

ਸ਼ਬਦ ਹਜ਼ਾਰੇ ਪਾਤਿਸ਼ਾਹੀ ੧੦, ਅਕਾਲ ਉਸਤਤ, ਬਚਿਤ੍ਰ ਨਾਟਕ, ਚੰਡੀ ਚਰਿਤ੍ਰ ਉਕਤਿ ਬਿਲਾਸ, ਚੰਡੀ ਚਰਿਤ੍ਰ, ਵਾਰ ਸ੍ਰੀ ਭਗਉਤੀ ਜੀ ਕੀ, ਗਿਆਨ ਪ੍ਰਬੋਧ:

Das Granthi. SGPC, Sri Amritsar, Jul 2014.

Das Granthi. SGPC, Sri Amritsar, Mar 2006, www.sikhbookclub.com/Book/Das-Granthi. Accessed 22 Jan 2019.

Zafarnama:

Padam, Piara S. Zafarnama. 6th ed., Singh Brothers, Amritsar, Sep 2013, www.archive.org/details/ZafarnamaTePanjHorNamae. Accessed 22 Jan 2019.

Rest:

Dasam Sri Guru Granth Sahib Ji. Vol. 1-2, Baba Teja Singh Ji Nihung Singh Mehron Wale, Moga Punjab, 2011.

Ramgarhia, Eshar S. Dasam Sri Guru Granth Sahib Ji Shiromani Steek. 1st ed., Vol. 1-6, Baba Teja Singh Ji Nihung Singh Mehron Wale, Moga Punjab, Vaisakhi 1999, www.archive.org/details/DasamSriGuruGranthSahibJiShiromaniSteek1. Accessed 22 Jan 2019.

Arth / Translation:

Jaggi, Ratan S. Sri Dasam Granth. Gobind Sadan, New Delhi, www.gobindsadan.org/download/148/siri-dasam-granth/1363/dasam-granth-complete.pdf. Created 15 Sep 2007. Uploaded 02 Nov 2017. Accessed 22 Jan 2019.

Bhai Gurdaas Ji

Ashok, Shamsher S, and Chakar, Amar S. Varan Giaan Ratnaavli Bhai Gurdaas Ji. SGPC, Sri Amritsar, Nov 2011, www.vidhia.com/Bhai%20Gurdaas%20Ji/Vaaran%20Bhai%20Gurdaas%20Ji%20-%20SGPC.pdf. Accessed 22 Jan 2019.

Amrit Keertan. 40th ed., Khalsa Brothers, Amritsar, Jul 2011, www.archive.org/details/AmritKirtan_226. Accessed 22 Jan 2019.

Singh, Sewa. Kabit Sawaiye Bhai Gurdas Ji Steek. 7th ed., Singh Brothers, Amritsar, Dec 2011, www.archive.org/details/KabitSavaiyeBhaiGurdasJiSteek. Accessed 22 Jan 2019.

Teeka / Translation:

Singh, Vir. Varan Bhai Gurdas Steek. 22nd ed., New Delhi, Jul 2012, www.archive.org/details/VaraanBhaiGurdasSteek-BhaiVirSingh. Accessed 18 May 2020.

Singh, Jodh. Vārān Bhāī Gurdās (Text, Transliteration, and Translation). Vol. 1-2, 2nd ed., B. Chattar Singh Jiwan Singh, Amritsar, 2013, www.archive.org/details/VaraanBhaiGurdasTextTransliterationAndTranslation-Vol1. Accessed 18 May 2020.

Bhai Nand Laal Ji

Ghazals, Zindagi Nama, Ganj Nama, Jot Bigas (Punjabi & Farsi), Rehit Nama, Tankah Nama, Dasturul-Insa, Arjul-Alfaz, Tausifo-Sana, Khatima and their Punjabi Translations:

Singh, Ganda. Bhai Nand Lal Granthavali. 4th ed., Punjabi University, Patiala, 2009, www.archive.org/details/BhaiNandLalGranthavali. Accessed 22 Jan 2019.

English Translations:

Kalaam-e-Goya. Translated by Pritpal S Bindra, Institute of Sikh Studies, Chandigarh, 2003

Acknowledgments

NOTE: If we have missed any attribution, credits, or copyrights, please let us know or fork this repo and submit a pull request.

Though unmentioned in the bibliography of sources above, the following are acknowledged for their work in pioneering the digitization process of gurbani, translations, and pronunciations:

• Dr. Kulbir S Thind
• SHARE Charity UK

database's Issues

Implement fuzzy search

Add as an option (Default: on) to core. If the entire issue can be fixed on the frontend then move this issue to core.

Examples:

• If someone searches Sat Naam it should show results for Satnaam
• If someone searches Vismaad it should show results for Bismaad
• If someone searches K vs ^ (khakka pair bindi) for Khalsa, show both

There are plenty of examples, Oorha vs Open Oorha. But it should not affect every single search. Searching Satnaam with S would show Sat Naam results, but not every result with S followed by N.

Gunvanti Wrong Author ang 763 should be guru arjan dev ji

jo deesai gursikharra tis niv niv laavo paae jeeo

written by guru arjan dev ji

in shabad os shows guru nanak dev ji

Correct Shabad writer_ids

If I remember correctly, many shabads are marked M1 when they shouldn't be, and some shabads have writer_id 9 instead of M9 (id 6) or vice versa. Additionally, some shabads/saloks in vaars are incorrect. Will have to go through all shabads carefully to correct these.

Track changes made to db

It would be great to see the diffs between db changes. This should also allow users, via UI, suggest changes.

Chandi Charittar

Missing text and subsections

Consistent storage of Punjabi

The database contains the unicode representation of punjabi in the SHABAD:PUNJABI column, yet the SHABAD:GURMUKHI column and others contain the ASCII representation used by the Akhar font.

I think this should be all converted into the unicode format; it also makes troubleshooting a bit easier.

Implement Line Types

Line Types

• mangal or manglacharan
• sirlekh
• rahao
• pankti or line

Checks:

mangal: Check for <> in Gurmukhi

rahao: check for ] rhwau ] and ] rhwau dUjw ]†

sirlekh: This is the most complex out of the bunch, will need to figure out how to differ from pankti

line: Everything else

† Rahao majority of the time contains more than 1 line, check could be implemented to check the previous line before the rahao line if it has a number in the end (ex. ]1]) or not.

Write tests

Write some default tests using the JS API so that we don’t miss it when content/IDs change

Switch to Travis

More OSS friendly, plus will use for Mac OS X builds in electron wrapper

Different versions of Baanis from School of Thought

Different groups of people read differing compilations of rehraas. We can:

• Include popular versions with labels (Such as Rehat Maryada, Extended Banis, Extended Chaupai & Anand Sahib)
• Include ability in desktop to create custom banis. Possibly extend to a "banis store" online where people can download popular compilations
• Include a setting to set all banis according to a specific School of Thought. (Personally against this approach as there are more customizable options above without hindering efficiency).

Missing Lines

Extended Chaupai Sahib, After Arril, 7555.

Include `॥` in `first_letters`

Searches leak over to next line when using the LIKE operator, so should probably include ॥ when generating first_letters.

Running SELECT * FROM lines WHERE first_letters LIKE 'ਸਕਨਨਅ' will return ਅਰਜਨ ਸਿੰਘ ਅਰੁ ਅਮਿਟ ਸਿੰਘ ਕ੍ਰਿਸ਼ਨ ਨਿਹਾਰਓ ਨੈਨ ॥ ਅਾਠ ਭੂਪ ਮਿਲਿ ਪਰਸਪਰ ਬੋਲਤ ਅੈਸੇ ਬੈਨ ॥੧੧੮੩॥.

Unnecessary FIRST_LETTER column

It seems the values for this column can be generated, maybe as a view.

Doing this would save a lot of space, and make the database less prone to human error.

Using a generated view would also allow for the FIRST_LETTER generations to be corrected and updated with ease, without affecting how any applications use the database currently.

Normalise database

Tables like bani_adv, bani_anand all have the same schemas. These can be changed into two tables, one which has the bani-name mappings, and another with the baniNameID->repeated schema mappings.

This will result in a cleaner and more extensible database.

Additionally, I noticed that the GURMUKHI and SHABAD tables have the same schemas. On an initial glance, both have the same number of rows, and so I presume, a lot of repeated data. This repetition in particular doubles the size of your database, by almost 2, and will impact user download time. Can this be normalised?

Is there also description of the schema fields available anywhere?

Add banis from old DB to new

The banis from the very first release need to be added into the Banis and BaniLines table.

Multiple lines on same id

Example: Chaupai Sahib

id >= 81121

] character as line break

Fix ?'s in Gurmukhi

There are a lot of lines (primarily from Sri Dasam Granth) with ?'s in the text. In some places, it should be an adhak, in others a vavva below the letter.

RWCX - Saral Steek

Key	Value
ID	RWCX
Shabad OS	sRI piq sRI Bgvwn Bjy ibnu; AMq kau, AMq ky Dwm isDwry ]3]23]
Saral Steek	AMq ky ≠ AMqky

Photo of Saral Steek page 204:

Complete documentation

Still missing many sections - see docs/

Typo in writers for some shabads

Some shabads have multiple writers accidentally. See SHABAD_ID 54 as an example. This should be corrected to the writer that the rest of the shabad uses.

Shabad 4WQ - Gurbani Dian Laga Matra Di Vilakhanta

Key	Value
ShabadID	4WQ
Shabad OS	iq nr duK nh BuK; iq nr inDn, nhu khIAih ] iq nr soku, nhu huAY; iq nr sy AMqu, n lhIAih ] iq nr syv. nhu krih; iq nr. sX shs smpih ] iq nr dulIcY bhih; iq nr auQip ibQpih ] suK lhih. iq nr sMswr mih; ABY ptu, irp miD iqh ] skXQ iq nr. jwlpu BxY; gur Amrdwsu supRsMnu ijh ]2]11]
Gurbani Dian Laga Matra Di Vilakhanta	iq nr ≠ iqnr

Gurbani Dian Laga Matra Di Vilakhanta (3rd Edition, November 2003)
Page 343

Page 344

Page 345

Reduce file size

Currently there is at least a duplicate amount of data in the db. The further we reduce it the faster the installer can be downloaded as well as faster downloads to db updates in the future.

Switch to unicode gurmukhi

This has native support in enough devices today. It also allows for copy pasting in more cases. Handy for those that want to share a shabad through text.

• Fix search from US keyboard to unicode hotkeys in core
• Find suitable font in core

Inconsistent Unicode Fonts

Not sure if this should be an "issue," but we should be careful with the unicode font we decide to use as some accent marks are displayed differently in different fonts. For example, on GuruGranthDarpan.net, the font they use requires them to write "ਕਲ੍ਯ੍ਯ" where other fonts would simply write it "ਕਲ੍ਯ" (http://www.gurugranthdarpan.net/1392.html).

Add dictionary and topics

Gyan Prabodh

Missing text and subsections

Kabiyobach Benti Choupai

Missing subsections

Dynamic Transliterations

Generate the transliterations via the API, and drop the column from the DB?

Vaar Sri Bhagauti Ji Ki

Missing subsections

Add withTranslations method

Add a withTranslations() chainable method to the API that can take an object specifying which translations & languages are desired.

Need something like this with the advent of #53.

Allow for subsections with the same name

Some subsections have the same name in a section.

DTH6 - Saral Steek

Key	Value
ID	DTH6
Shabad OS	koit iesnwn gjwidk dwn; Anyk suAMbr swij, brYNgy ]
Saral Steek	iesnwn ≠ snwn

Photo of Saral Steek:

Bachittar Natak

Missing text and subsections

Add banis

No banis are in the new DB. Add them by tagging the lines in the JSON with the start and end positions.

Wrong author of shabad (jo nar dukh mai dukh nehee manai)

jo nru duK mY duKu nhI mwnY ]
jo nar dhukh mai dhukh nehee maanai ||

This shabad on ang 633 is showing that the author is Bhagat Bhikhan Ji. This shabad is written by M9.

Source - the title of the shabad says sorath mehalaa 9

Akaal Ustat

Missing subsections

Add test for build-json

pad seeds files with 0s so they are ordered correctly

1, 10, 100, then back to 11 currently. Better would be naming them 001, 010, 011, 100 etc.

Correct typo in line

Current:
sÍsiq ibvsQw hir kI syvw; mDM´q pRB jwpx ]

Correct:
sÍsiq ibvsQw hir kI syvw; mD´Mq pRB jwpx ]

'?' in between shabad

Replace transliteration of "Ma" with Mehalaa

Some of the transliterations have Ma 3, Salok Ma 3 etc, instead of Mehalaa

Remove SnakeCase conversion?

Currently all snake_case fields in DB are outputted as camelCase when using the API.

Unsure if this is confusing. @bhajneet @sarabveer, any input?

Either remove, or improve consistency by using knex mapper instead.

Rename raags to sections

Add withTransliterations method

Similar to #55, but for transliterations.

Generate ASCII and Gurmukhi variants of DB

Currently only one of these are produced.

Release both in production.

Chandi Charittar Ukat Bilas

Missing text and subsections

Publish to npm

The database should be published to npm on CircleCI build. This means that the dependency management and updating can be outsourced to npm, rather than relying on downloading from github. We should also do this for the frontend and backend.

WYU8 - Saral Steek

Key	Value
ID	WYU8
Shabad OS	qw ky, kwl inkit nih AYhY ]
Saral Steek	AYhY ≠ AY hY

Photo of Saral Steek page 228:

Rename writers, sources, raags, line types

The corresponding json files can be changed, and everything should be regenerated with the new file names. The old folders in shabads and sources may need to be removed, since they're generated based on the names of writers and sources

Question mark and possible vishraam mistakes

AknqAsr

Has : which might have been meant to be ; and ? symbol in gurmukhi. Not sure about first word either, can't remember those two vowels used together.

@sarabveer