nakilon / dhash-vips

vips-powered ruby gem to measure images similarity, implementing dHash and IDHash algorithms

I code since 1994.
I'm skilled in Ruby and A.I.
I can solve unsolvable.

---

Here are some of my

• most successfull and handy projects:
  • IDHash -- the most accurate and fast perceptual image fingerprinting algorithm and Ruby gem
  • nakischema -- object validation with the simplest syntax
  • lineprofiler -- Ruby statistical flat profiler
  • PCBR -- ranking algorithm, usable for non-heuristic optimization, agnostic to data distribution shape
  • p5rb -- Ruby DSL for p5.js
• most fun projects:
  • RASEL -- esoteric programming language
  • utility chat bots (Slack, Telegram, IRC, Twitch, Trovo, etc.)
    one of them could learn to talk on the fly, another one is currently used by multiple Twitch streamers
  • converted S.T.A.L.K.E.R. game assets to NPC/artifacts/etc. maps
  • procedurally generated CS:GO map
  • visual web page structure recognition proof-of-concept
• other projects:
  • Shelenium -- a tool to automate or test terminal UIs, such as made with curses
  • directlink -- CLI to convert arbitrary webpage URL to direct images links

A GitHub stars list of tools and libraries I use.

---

Sometimes I contribute to popular gems and SDKs with bugfixes or documentation improvements.
In spare time I translated ruby-lang.org to Russian.
I can make slides and give a talk.

If you are a recruiter, here is my CV in English (updated Mar 2023).
And recently I started a little cringe blog with notes about coding.

dhash-vips's Issues

How ... Interlace handling should be turned on

/usr/local/lib/ruby/gems/3.0.0/gems/ruby-vips-2.0.17/lib/vips/operation.rb:190:in `build': vipspng: Interlace handling should be turned on when using png_read_image (Vips::Error)
vipspng: Interlace handling should be turned on when using png_read_image
extract_area: bad extract area
	from /usr/local/lib/ruby/gems/3.0.0/gems/ruby-vips-2.0.17/lib/vips/operation.rb:402:in `call'
	from /usr/local/lib/ruby/gems/3.0.0/gems/ruby-vips-2.0.17/lib/vips/image.rb:211:in `method_missing'
	from /usr/local/lib/ruby/gems/3.0.0/gems/dhash-vips-0.1.1.3/lib/dhash-vips.rb:24:in `calculate'

My code:

Dir["/home/xxx/tmp/**/*"].each do |fichero|
      begin
	      puts fichero
	      hash1 = DHashVips::DHash.calculate fichero
	      hash2 = DHashVips::DHash.calculate fichero
	      #puts "#{hash1} #{hash2} #{fichero}"
      #rescue
      #	puts "ERROR------------: #{fichero}"
      end
end

How... Interlace handling should be turned on ?

thank you in advance

Rails server keeps quitting after calling functions from the library

Hi guys, first of all thanks for this amazing gem! IDHash works beautifully.

I'm experiencing a very weird issue where my rails server keeps crashing after I call an action that uses the gem
I confirmed this by calling other actions that aren't using the gem and the server stays alive, so I know it's something to do with the gem.

Has anyone experience this? Help greatly appreciated.

ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x64-mingw-ucrt]
Rails 7.0.2.4

Gemfile:
gem "dhash-vips"

Running windows 10

Thanks!

TODO: fix Github Actions

• MiniMagick::Invalid: You must have ImageMagick or GraphicsMagick installed
• LoadError: cannot load such file -- phashion

Issue installing v0.1.1.4

Hello :)
We started having some issue on our CI after the new version was pushed but I can't track it down the issue, it seems to be happening on ruby 2.7.x and 3.0.x

I tried locally on my machine and I have the same error:

Fetching dhash-vips-0.1.1.4.gem
Building native extensions. This could take a while...
ERROR:  Error installing dhash-vips:
        ERROR: Failed to build gem native extension.

    current directory: /home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/lib/ruby/gems/2.7.0/gems/dhash-vips-0.1.1.4
/home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/bin/ruby -I /home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/lib/ruby/2.7.0 -r ./siteconf20220221-38934-8qm4nf.rb extconf.rb
checking for whether -I/ruby/ is accepted as CPPFLAGS... yes
checking for whether -DRUBY_EXPORT is accepted as CPPFLAGS... yes
creating Makefile

current directory: /home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/lib/ruby/gems/2.7.0/gems/dhash-vips-0.1.1.4
make "DESTDIR=" clean

current directory: /home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/lib/ruby/gems/2.7.0/gems/dhash-vips-0.1.1.4
make "DESTDIR="
compiling idhash.c
idhash.c:1:10: fatal error: bignum.c: No such file or directory
    1 | #include <bignum.c>
      |          ^~~~~~~~~~
compilation terminated.
make: *** [Makefile:258: idhash.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/lib/ruby/gems/2.7.0/gems/dhash-vips-0.1.1.4 for inspection.
Results logged to /home/renatolond/.rvm/rubies/ruby-2.7.2-jemalloc/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/dhash-vips-0.1.1.4/gem_make.out

I checked the diff and I can't understand what is causing it, since there seems no change was made in that file since 0.1.1.3 🤔

For more info: I'm running on Linux (the CI is too)

NoMethod Error During Post-Install Check

When trying to install the dhash-vips gem on MacOS 12.0.1 on Apple Silicon (ARM) using Ruby 3.0.2, I'm getting the following error:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/christopher/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/dhash-vips-0.1.1.3
/Users/christopher/.rbenv/versions/3.0.2/bin/ruby -I /Users/christopher/.rbenv/versions/3.0.2/lib/ruby/3.0.0 -r
./siteconf20211104-38007-8h8fj7.rb extconf.rb
checking for whether -I/Users/christopher/.rbenv/sources/3.0.2/ruby-3.0.2/ is accepted as CPPFLAGS... yes
checking for whether -DRUBY_EXPORT is accepted as CPPFLAGS... yes
creating Makefile

current directory: /Users/christopher/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/dhash-vips-0.1.1.3
make DESTDIR\= clean

current directory: /Users/christopher/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/dhash-vips-0.1.1.3
make DESTDIR\=
compiling idhash.c
linking shared-object idhash.bundle
/Users/christopher/.rbenv/versions/3.0.2/bin/ruby -r./lib/dhash-vips.rb ./lib/dhash-vips-post-install-test.rb
Ignoring eventmachine-1.2.7 because its extensions are not built. Try: gem pristine eventmachine --version 1.2.7
Ignoring http_parser.rb-0.6.0 because its extensions are not built. Try: gem pristine http_parser.rb --version 0.6.0
Testing native extension...
[1014914048, 1715898367, 528465920, 2122267904, 4622, 2156271871, 3229616096, 4286545664]
[2117872660, 876379772, 1216315170, 2139029504, 4325892, 1074036350, 33023, 2147483392]
./lib/dhash-vips-post-install-test.rb:10:in `<main>': undefined method `distance3_c' for DHashVips::IDHash:Module
(NoMethodError)
Did you mean?  distance3
               distance
01111110011111100111110000010100
00000000010000100001000000001010
01110110011111101111111111111111
11000000100000100110011010000001
01011111011111111111111100100010
11000000100000001000011100011111
01111111011111110011110100000000
10000000100000001000000000000000
make: *** [test] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/christopher/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/dhash-vips-0.1.1.3
for inspection.
Results logged to
/Users/christopher/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/extensions/arm64-darwin-21/3.0.0/dhash-vips-0.1.1.3/gem_make.out

An error occurred while installing dhash-vips (0.1.1.3), and Bundler cannot continue.
Make sure that `gem install dhash-vips -v '0.1.1.3' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  dhash-vips

This happens when using either gem "dhash-vips" or `gem "dhash-vips", git: "https://github.com/nakilon/dhash-vips/"

Thank you for any help you can provide in figuring this out!

idhash.c:1:10: fatal error: 'bignum.c' file not found when using v 0.1.1.1 and 0.1.1.2

Came across this bug today while trying to bundle install our project. Posting this issue in case it's of some help to someone else. I tried to track it down but couldn't find anything that helped (including brew reinstall vips, brew upgrade && brew update.

Eventually I fixed the bundling error for dhash-vips by rolling the version specified in the Gemfile back to '0.1.1'.

I'm running:
OS X 10.15.5
Homebrew 2.5.1
Ruby 2.7.1 (via rbenv v1.1.2)

(I suspect the problem to have arisen from the changes to extconf.rb in this commit: http://github.com/Nakilon/dhash-vips/commit/56fba57aac82552f26086474cd57b63a4a99dc36%20 but I can't be sure as I wasn't able to verify any fix.)

Comparison quality loss, probably due to libvips update

Hello @jcupitt. I believe that with some of the libvips updates my gem has lost a comparison quality a bit but it was tedious to install different versions of the library (and I don't really have a machine right now to make it compile things a lot) so I didn't figure out when exactly it happened or if the quality would return if I switch back. But I decided to inform you anyway. (I'll switch this Issue to Discussions tab later)

It may seem to be difficult to reproduce (and partially that's why I'm noting this only after a year or so) but I suppose that somewhere between vips-8.9.1 and vips-8.11.3 there was a change/tweak in the interpolation algorithm IIRC and it made my perceptual fingerprints worse. Specifically for the DHash in this line b6e0945#diff-a7bce69e0d9f02a55d36f1b84429134d052557c4aaaeaf5caa6a0ca73d72c437R8 it means that two images had the difference 2 between them but became as close as 0, and another pair is just some b/w filter that was making the distance of 4 but now it's 6. For the IDHash "the most distant among the least distant" image pairs became of the same distance as "the least distant among the most distant" image pairs b6e0945#diff-a7bce69e0d9f02a55d36f1b84429134d052557c4aaaeaf5caa6a0ca73d72c437R46 and so the test reveals that it became harder to find a good threshold and so IDHash has almost reached the Dhash gem quality: b6e0945#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R152

Memory usage with new_from_file

Hi!
We've been testing dhash-vips to compare images and the fingerprint method seems to use a lot of memory for our use case.
I checked ruby-vips and it seems using access: :sequential improves the memory usage for our use case, while still giving apparently the same results: libvips/ruby-vips#197 (comment)
Do you think access: :sequential would make sense for the fingerprint method?

Add Rails Caveats Around Spring

I think it would be a good idea to add instructions for Rails users (just to be nice), which involves removing Spring as referenced here by @jcupitt.

I can do submit a PR for this, but just wanted to make sure everyone is cool with it?

FFI::NullPointerError: invalid memory read at address=0x0000000000000000

[40] pry(main)> DHashVips::DHash.calculate("00001.jpg")
vips warning: tilecache: error in tile 0 x 0
FFI::NullPointerError: invalid memory read at address=0x0000000000000000
from /var/lib/gems/2.5.0/gems/ruby-vips-2.0.12/lib/vips/image.rb:478:in `get_bytes'
[41] pry(main)> DHashVips::IDHash.fingerprint("00001.jpg")
vips warning: tilecache: error in tile 0 x 0
FFI::NullPointerError: invalid memory read at address=0x0000000000000000
from /var/lib/gems/2.5.0/gems/ruby-vips-2.0.12/lib/vips/image.rb:478:in `get_bytes'

dpkg -l | grep libvips-dev
ii  libvips-dev                                     8.4.5-2+b1                              amd64        image processing system good for very large ones (dev)

cat /etc/issue.net 
Debian GNU/Linux buster/sid

Use Dhash for comparing a set of 5 images to frames from video to match the image?

Is it possible to use Dhash to compare for similarity(it will be exactly same anyway because the output is from a handheld atm swipe machine screen) to video frames (even 1fps will do the job)
so i can match the image to template(template matching seems a fit but not really working well for me)
and do a task like send a message?

I will be using openCV (c++ or python doesnt realy matter) and planning to implement on raspberry pi and 5Mp camera attached

Hanging when used in Rails app

First: thanks for the great tool, I was going to implement this myself but instead I'll definitely help out if possible to improve via a PR or two.

Second: When attempting to calculate within a Rails console the system just hangs indefinitely. I'm using Rails 6.1.4 and Ruby 3.0.1. Running just a basic Ruby console (i.e. irb) in the same version works perfectly, so it's something within the Rails stack that's causing this.

Any chance there's something off the top of anyone's head that could be causing this?