mspaintide / newocr

A custom OCR library in pure Java made as a replacement for MS Paint IDE's OCR

Website | Wiki | Discord | Installation/Demo | Custom OCR

People joke about what IDE they use often, things like Word, MS Notepad, sometimes even Eclipse, and then often times MS Paint. People joke about MS Paint because it's not even a text editor, people joke about it because it doesn't have one feature in common with IDEs. Well, this application gives MS Paint a boost, and lets MS Paint highlight, compile, and execute code, with just a few clicks of a button, and only text coming from MS Paint. It is now much more practical than things like Word, Notepad, and obviously Eclipse.

Features

MS Paint IDE has grown an insane amount over the past year, and I plan for it to grow even more in the following year. Currently, these are some of the features present in MS Paint IDE:

• Can read, parse, and highlight code from purely image files
• Finding and replacing of text from image files
• Supports Java, Python, JavaScript, and Go, with many more planned
  • Includes LSP support alongside a very simple language API to add custom languages
  • Expandable build system support, with Gradle included
• Git tools, to create and manage your project in Git from within the IDE
• Google Assistant support
• Right-click context menu to open and edit ANY text file in an MS Paint instance
• Integrated buttons within MS Paint (No program modifications) to do basic IDE tasks
• Native installer for easy usage
• Full Discord RPC Support
• IDE theming via CSS files
• Program and compiler outputs as image logs

The following is a video of the IDE being installed and used (For just usage, skip to 2:18)

How It Works

MS Paint IDE is an application that runs alongside MS Paint that reads its images you make from MS Paint. It then uses a custom OCR to read, parse, highlight, and execute the text. All output files and logs are created as images, to show you the program output, and compilation/interpretation output.

Wiki

Every feature, button, usage instructions, and whatever you want is on the wiki, which is the best place to start if you're interested in trying out the IDE. https://wiki.ms-paint-i.de/

Contributing

Contributors are always welcome, and for setting up the repository locally, there's a wiki page walking through the whole process: https://wiki.ms-paint-i.de/developing. If you simply have a bug or feature request, please make a issue on GitHub using the corresponding template.

A video of getting the codebase locally is available below.

Screenshots

Many screenshots are available through the wiki, but some showing off the IDE are below so you don't need to dig through it.

Donations

I've been doing mainly this project for over a year now, and even though I'm not making money off of it, I would be extremely appreciative for any contributions to help pay for the domain and the website's VPS. If you donate and want to be mentioned here, just put it in the donation message. Feel free to donate via PayPal or Open Collective.

Thank you very much to the current donors:

@Mr. Midnight

@Stef

@iCodeHaven

@AL1L

Special Thanks To:

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.

Contributors

Code Contributors

This project exists thanks to all the people who contribute. [Contribute].

Financial Contributors

Become a financial contributor and help us sustain our community. [Contribute]

Individuals

Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]

newocr's Issues

Slight messups with font heights < 50

In font heights < 50, there are some minor problems with seeing periods on different lines (Can be solved by a more accurate threshold/whatever it's called), underscores not being detected properly, etc. Here's an example of the output of detecting a training image: https://rubbaboy.me/code/dzawcvs

Abstract character classes

Currently, there are 4 classes all meant to store letter data all sharing similar methods (#getX(), #getY(), #getWidth(), #getHeight(), etc.). These 4 classes should be abstracted to a single character interface/superclass, with any necessary implementations made which should simplify a lot of code.

The classes in question are:

• ImageLetter - Meant to store finalized data which is used by programs using the library
• SearchCharacter - Stores the coordinate data and some calculated data created directly when an image is read
• DatabaseCharacter - Averaged character sections and other trained data derived only from the database
• TrainedCharacterData - Stores data during training to then average stored data and insert them into the database

I'm guessing these can probably be merged into 2, maybe 3 classes with better abstraction. This isn't currently a super high priority, but I'd like to see it done before the v2 release.

Fix character detection

Fix the actual recognition of characters by using more standard/common practices, such as full-line separation versus just character pixel grouping. Currently, it's not accurate and very inefficient.

This has been in development for a few weeks now on a private test repo, and is starting to be brought into NewOCR now.

Add GUI display

Add a GUI that shows the input image, and then bounding boxes around found characters, with the ability to hover over them/click them to see things like potential other matches, thresholds, and relations to other characters, etc. Currently, there is no way to easily do this which makes debugging a pain.

This could make higher-level debugging easier in the future when more fonts are supported and developers are taking advantage of internal APIs. This isn't high-priority, but potentially could be added for v3 of the OCR.

Note: This would not be in the core API of the OCR, but either in the current demo module or another new gui module meant just for GUI handling.

Exampels

Hello. Have you tried using recognition on more complex examples, for example, on scanned passports?

Add custom alphabets

The addition for custom alphabets would result in the ability to support more languages, and potentially things like emoji detection. This is not a priority in any way, but may be nice to have in the future.

Fix line detection during training

When underscores are fully below all other characters in certain fonts (e.g. Monospaced.plain) the underscores are detected as fully separate lines. This completely throws off training, though can probably be fixed by adding a separate threshold for training alone for characters being attached to larger and close lines.

How to run the program to get the result from a image?

Hello! I would like to know where the database is for storing the characters or you need to form it yourself, it is not clear from the code, you could a little
explain how to use this database?

Keep up the good work

👍