academia-edu / dejunk

Detect keyboard mashing and other junk in your data

Dejunk

Detect keyboard mashing and other junk in your data.

For example, if you allow user-entered tags, but want to hide bad ones. Or if you want to detect user frustration filling out a particular field, and do something about it!

Uses a variety of heuristics, the most sophisticated being a comparison of bigrams in the input to the frequencies in a "known-good" corpus vs. their proximity on a keyboard. Achieves pretty good precision on Academia.edu's data, but might need adjustment for yours.

Installation

Add this line to your application's Gemfile:

gem 'dejunk'

And then execute:

$ bundle

Or install it yourself as:

$ gem install dejunk

Usage

The main interface is Dejunk.is_junk?. Pass a string, and get a truthy value if it looks junky, and false otherwise.

$ Dejunk.is_junk?('Hello World')
=> false
$ Dejunk.is_junk?('qwefqwef')
=> :mashing_bigrams

$ Dejunk.is_junk?('asdf')
=> :asdf_row
$ Dejunk.is_junk?('fads')
=> false

$ Dejunk.is_junk?('Hi')
=> :too_short
$ Dejunk.is_junk?('Hi', whitelist_regexes: [/\Ahi\z/i])
=> false

Returns a reason when junk is detected for aid in debugging. Optional parameters are min_alnum_chars (defaults to 3), and whitelist_strings and whitelist_regexes (both default to none, but you'll likely want some domain-specific strings here, which you might discover by checking against a sample from your existing corpus).

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/academia-edu/dejunk

License

Apache 2.0

dejunk's Issues

Bug: `mashing_probability` tests wrong variable

Hey there,

Thanks for this library. I've been going through source code and have a question about the logic in the mashing_probability function here.

I don't know ruby too well so sorry if I'm misunderstanding this but in the elsif test, should that not be testing against bigram? If f is the result of mashing_bigram_frequencies[bigram] it should either be a float or nil no?

  def mashing_probability(bigram)
    if (f = mashing_bigram_frequencies[bigram])
      f
    elsif f =~ /[a-z]{2}/i # <- should this be testing bigram?
      0.0005
    else
      1e-6
    end
  end

Anyways, sorry if I've misunderstood & thanks again. I'm learning heaps looking at the approach you've taken here 👍.