jadeocr / jadeocr

Elegant spaced-repetition flashcards with handwriting recognition for learning languages

jadeocr

Elegant spaced-repetition flashcards with handwriting recognition for learning languages — jadeocr.com

Jadeocr has won the 3rd place award in the CA18 Congressional App Challenge!

Features

• Handwriting recognition
• Spaced repetition flashcards
• Multilingual support
• Elegant UI

Built With

• Keras (model not currently in use)
• Vue.js
• Firebase
• Node.js
• Electron (desktop app only)

Quickstart

In the jadeocr-webapp directory, run the following to quickly spin up a development instance.

$ yarn install    # Installs dependencies
$ yarn twbuild    # Builds Tailwind CSS files
$ yarn serve      # Compiles/hot-reloads dev server
$ yarn electron:serve # Compiles/hot-reloads desktop app

Adding Firebase

To add the backend, create a project in Firebase. Copy the JS config snippets from the Firebase console into the firebaseConfig in jadeocr-webapp/src/firebase/credentials.js to add the SDK credentials.

export default {
  firebaseConfig: {
    apiKey: "API_KEY",
    authDomain: "AUTH_DOMAIN",
    ...
  }
}

Development

$ yarn serve      # Compiles/hot-reloads dev server
$ yarn electron:serve # Compiles/hot-reloads desktop app
$ yarn build      # Compiles/minifies -> dist for production
$ yarn electron:build # Builds and packages desktop app
$ yarn lint       # Lints/fixes files

Deployment

$ yarn deploy   # Builds, deploys to Firebase Hosting, and removes dist
$ yarn electron:build # Builds and packages desktop app

Desktop App

There is an experimental (cross-platform) electron desktop application should you choose to use it. There are a few caveats:

• It acts basically the same as the website
  • This means no optimizations for desktop have yet been made (removing landing screen, tweaking UI, etc.)
• Only supports sign in with email and password
  • This is likely a Firebase issue that we might work on in the future to add Google sign-in

OCR CNN

Currently, jadeocr uses the Google Cloud Vision API to handle handwriting recognition. In the future, we would like to implement custom neural network for better detection of languages not using the Latin script.

The OCR neural network is trained on a 100-class subset of the CASIA Chinese Handwriting Dataset. To train on the full dataset effectively, it is necessary to have more training examples per class.

Dependencies

Create a virtual environment and install dependencies with:

$ cd convnet && virtualenv env     # Creates virtual env
$ source env/bin/activate          # Activates virtual env
$ pip3 install -r requirements.txt # Installs dependencies

Obtaining the Dataset

To obtain the full dataset, download HWDB1.1train_gnt (2741MB) and HWDB1.1test_gnt (681MB) and extract the zip files. Store them in the directory convnet/data and check that the extracted folders are named HWDB1.1trn_gnt and HWDB1.1tst_gnt, respectively.

Preprocessing

Run convnet/src/preprocess.py to convert from GNT to png.

Training

Run convnet/src/train.py to train the model. You can tweak the hyperparameters at the top of the file.

Prediction

Save an image convnet/data/tests/test.png that you would like to have the network predict to and run convnet/src/predict.py.

Contribute

Thank you for reading this far and considering contributing! Contributions are very much appreciated, and they are what makes this project what it is! We are looking for contributions of features/feature requests, bug fixes, documentation fixes, and anything else you can think of! The best contributions are those that will add value to the user experience. If you need to get in touch, please see the contact/help section.

Please take a look at the information below before contributing. Also, please take a look at the code of conduct. We expect that you follow these standards, and we will enforce them. Please report any violations by sending me an email.

Issues

Browse through the issues or submit one. Here are a couple guidelines to follow:

• Make sure all of your dependencies are up to date
• Include steps to reproduce the issue
• Expected behavior and what went wrong
• Screenshots/terminal output if necessary

Pull Requests

Pull requests are also always welcome. Here are a couple simple guidelines:

• Make sure your code is readable and commented when necessary
• Document your changes adequately when opening a pull request

Donations

If jadeocr has been of some value to you, and if you can afford it, please consider buying us some 奶茶. Your donations will also support the development of new features and code maintenance. Here's how you can donate:

• BTC: 15Y9NZjxTLWHDU8kVsqN7FKey3c1RPNiFi
• Venmo
• Paypal

Thank you so much for taking the time to contribute!

Credits

• integeruser
• 想飞的石头
• 蹦跶的小羊羔
• Suragch

Contact/Help

If you would like to get in touch with me for any (legitimate) reason, please do not hesitate to contact me. If you need more help, please open an issue or send me an email (if an issue isn't appropriate).

License

This repository is licensed under the MIT License.

Tanay Biradar

jadeocr's Issues

Renaming deck creates a new deck

When going into the edit menu for a given deck and changing the name, the original deck remains and a copy with the new name. Make it so that renaming the deck keeps only the updated copy of the deck.

Add statistics to decks JS object

Add more metadata to the decks object so that the SRS for the flashcards can work effectively.
(Tanay)

Change Chinese font to match material aesthetic

Use a CJK-friendly font like Source Sans for Chinese text

Make sign-in button conform to branding guidelines

Issue

Make sure that the sign in button on the landing page (Title.vue component) conforms to Google's guidelines. Use the white button with opacity 0.87.

Handwrite 囧 on 404 page

Make the 404 page a bit prettier by digitally handwriting 囧 using a 毛笔 or something similar.

• Pick an ink color that fits well with the material aesthetic.
• Save as PNG and superimpose it on the page (it depends on what looks best.)

Design about section

Design a minimalist about section below the main title on the landing page using Figma.

Add compressed dataset

Add a compressed dataset for the ML model so that contributors can try their hand at training it and tweaking the code.

Register stylus touches on tablets

On tablets/touchscreens, drawing on the canvas in Review.vue does not produce drawing. This is an issue with drawing code

Set store persistence length

Set a time for the store to persist so that the dashboard always shows the correct days until the next review.

Add email login validation

Use an API or a Firebase Cloud Function to send a verification email to each user upon registration

Implement the SRS system

(Riley/Tanay)

Sizing of display to match monitor size

Get rid of the scrollbar to fit the monitor's size completely.

Extracting CSS Components From Tailwind

Issue

Right now, applying Tailwind CSS classes is horrendous, and it'll likely worsen as the project continues. Here's an example of what how the classes are applied from Title.vue:

<router-link :to='{ name: "login" }'
      class='btn bg-purple-500 hover:bg-purple-600 mt-8 px-3 py-1 
      text-base sm:text-xl md:text-2xl lg:text-3xl xl:text-4xl rounded-lg'
      >
        Get Started | 开始
      </router-link>

Clearly, the classes are not very readable, and it would be a nightmare to switch later on. The goal is to fix this so we can have three or four classes max applied to each HTML element.

Ideas

This section of the Tailwind CSS Docs describes how to extract components. It looks like PostCSS is needed to use the @apply directive, but that could be false.

Try using the v-bind:class = ' ' attribute and take in an array of classes as a prop from the data function in App.vue (this can just be static binding).

Update Google sign in button styling

Issue

The Google sign in button is not responsive, nor is it pretty. Here's what we need to fix this up:

1. Responsive sizing
2. Rounded corners (equivalent to the Tailwind rounded-lg attribute)
3. Change opacity/background color on hover
4. Hide after signing in and replace with a "to profile" routing

Ideas

General

Use the guidelines for building a custom Google sign-in button. This is probably the best solution

Item 1

Use the v-bind attribute to data-width and data-height to conditionally bind the width/height of the button based on the viewport width. It appears we can't use rem or Tailwind media queries because Google only accepts pixels.

Look at the guidelines in the general section.

Item 4

Use vue-router and route to /profile, which will be a dummy page for the time being.

Block routes if not authenticated

Block any of the dashbaord routes if the user is not authenticated

Landing page UI issue on small screen landscape

Go to the Brave (or Chromium/Chrome) devtools and select a small screen size for the device emulator (such as an iPhone 8) and put it in landscape. The <div id='intro'> does not show fully due to the overflow-y-hidden (it's cut off at the moment). However, if that is not present, there will be significant overlap that makes the UI look ugly.

Fix this so that the UI on Landing.vue looks good. This should ideally entail making the <div id='intro'> contents like that on a lg screen size (Tailwind query). Any other way to fix this also works.

OCR only recognizes first word

When writing multiple words with the OCR, the model sometimes only returns a prediction for the first word. This has to do with only displaying the 0th element of the returned predictions array. To fix this, show more elements of that array.

Add Drawing Canvas

We need to add a drawing canvas so the user can draw 汉字 with the mouse.

Preferably find a node-js module, otherwise build it from scratch. It should be on the frontend and send the canvas as an image to the server. Make sure this works with Express

Implement binary search through decks

Check if decks are sorted then add search bar + binary search implementation

Store User Login Info

When a user signs in, save their login state, profile URL, UID, and relevant data in a cookie so that when they refresh the page their data is not lost.

Migrate to Vue 3.0

Vue 3.0 is in the Release Candidate stage, and we should switch to it. Here's why:

1. Faster compiler: Better development experience for new features. It checks for capitalization instead of going down the tree to check if a tag is component.
2. Better tree-shaking: Gets rid of unused bloat code
3. Better TypeScript support: TS is good, we should switch to it. The compiler itself is written in TS.
4. Better native support: We can bring jadeocr to mobile!

@kongriley what are your thoughts?

:wq

Add forgot password to login form

Add a forgot password button to the login form and then create a page/modal to sent a password reset email.

Change mobile hamburger menu background color

The mobile hamburger menu icon would look better with no background and with a size of 1.5

Deck trim not working

Edit the trimDeck() function so that it actually trims the deck.

Writing not working on mobile

1. When trying to write, the line would follow very shortly then stop. It is not one continuous flow, so the user would have to keep pressing down each time in order to make the lines connect. It looks somewhat like this: ----

I used a mobile phone, a Motorola G6. I tried writing with both my finger and a stylus.

2. When going under "Profile" option, it shows the words "Profile Image", where I'd assume an actual image is supposed to show.

Add sidebar on desktop to a hamburger menu

This could save space and look nicer as more information is on the screen

Google sign-in not working on desktop app

This is likely a Firebase issue with permissions.

Deck sometimes deletes edit progress

Sometimes, when editing decks for a long time, the deck will not be saved, and will delete the edits after a certain amount of time.

Add login with Firebase auth

Use Firebase authentication to securely log in with Google (add other services in the future)

Google sign in button not adding user

Issue

After going through the redirect menu following the Google sign in button, the user does not show up on the Firebase console under the list of users. Correct this so that the user is added to the list of users.

Fix Image Fed Into Model Prediction

Make sure the drawing is a black-on-white image, not black-on-alpha or white-on-black. This should fix the incorrect predictions.

Style buttons to match material aesthetic

Replace the purple buttons with gray buttons on the following routes:

• 404
• Landing

Review UI for the Review.vue template

Currently Review.vue contains a basic template for showing flash cards. Before adding JS functionality, fix the UI for this page and make it look pretty. Then, move on to adding intermediate JS functionality (will be a separate issue).

(Riley)

DeckList not updating properly consistently

Decklist.vue doesn't always show all the decks immediately after adding, deleting, or reviewing a deck. Make it so that it updates and displays the most up-to-date information.

NaN after multiple reviews

After the first review, everything works great. However, all the easiness values in the database show up as NaN on the second review onward.

Web Handwriting Recognition API

Hello,

We (Chrome and Chrome OS) are working on a standardized Web API for handwriting recognition. The proposal is here: explainer.

The API would allow websites to access handwriting recognizers that's locally available on device (instead of sending the requests to cloud servers). It's currently implemented on Chrome OS (backed by the same technology in Google Cloud Vision API).

Does the API interest you?

We'd also like to hear your feedback and make it easier to integrate with your app.

Make store state update immediately on sign in

Currently, the store does not update right away after the sign in methods are called (verified issue for Google sign-in, not for email and password––although it’s probably the same)

Make sure that after the user signs in (this is probably asynchronous, so use actions), the state updates with the values of the user object and also the UID, email, name, etc.

Send email when decks are due

Chinese font not displaying properly

Create new design scheme

While the Material Design is nice, I think it leaves a bit to be desired. We should devise a color pallette and design scheme for the redesign

Add the ability to pin decks and sort decks on the Learn page

Hidden and expandable sidebar

On mobile devices (screen size "sm"), hide the sidebar but make it expandable using Vue/Tailwind (and ideally no other frameworks/plugins).

Some boilerplate code is already written on Profile.vue.

Integrate deck data with the Review.vue template

Use the deck data from the Vuex store and make parameters such as the days until review, cards, etc. work with the UI elements. Also add the ability to flip cards or check OCR.
(Tanay)

Overdue by NaN days

On the learn and stats page, the interval indicator occasionally indicates "Overdue by NaN days"

Mobile sidebar button weird on scrolling

While scrolling on the mobile version, there is a button to open/collapse the sidebar. While scrolling without the sidebar open, however, the button seems to get cut off by the div content. Fix this.

Design dashboard page(s)

Use figma.com to come up with a design for the dashboard. It should have the following:

• User information page
• Settings
• Sign out
• Space for a list of decks
• Space for analytics

Spaced repetition testing

It seems that you can't review past the scheduled cards. The SRS algorithm also needs more testing.

Update Firestore Security Rules

Issue

The current Firestore security rules are extremely insecure, allowing anyone to read/write to the database. After implementing user authentication, ensure users are signed in to be able to modify the database.

Ideas

Firebase Security Rules Guide

Check for email already in use

Check if the email input for the sign in is already in use. Firebase already returns an error for this.