Not working on M1 mac about hermes

[sidharthramesh]

Hey @wardle had been busy. Finally got the latest version working!
The performance improvement while indexing is very noticeable, even when running on Rosetta mode.

Great work!

[sidharthramesh]

Upon running it on Google cloud (AMD64) I get the below error. The exact same build worked with hermes v0.8.4--alpha
Currently updated to the latest version: v0.12.654.

This is the Dockerfile that is being built: https://github.com/medblocks/mlds/blob/master/Dockerfile. It sources some of the SNOEMD packages from a Google Cloud Storage Bucket.

Step 14/21 : RUN java -jar hermes.jar -d ./snomed.db import ./extracts
 ---> Running in 481266795a68
2022-06-02 18:39:24,598 [main] INFO  com.eldrix.hermes.core - importing 1 distributions from "./extracts"
2022-06-02 18:39:24,600 [main] INFO  com.eldrix.hermes.core - distribution:  "SnomedCT_InternationalRF2_PRODUCTION_20220531T120000Z"
2022-06-02 18:39:24,601 [main] INFO  com.eldrix.hermes.core - license:  "© 2022 International Health Terminology Standards Development Organisation 2002-2022.  All rights reserved.  SNOMED CT® was originally created by the College of American Pathologists.  'SNOMED' and 'SNOMED CT' are registered trademarks of International Health Terminology Standards Development Organisation, trading as SNOMED International.  SNOMED CT has been created by combining SNOMED RT and a computer based nomenclature and classification known as Clinical Terms Version 3, formerly known as Read Codes Version 3, which was created on behalf of the UK Department of Health and is Crown copyright.  This document forms part of the International Edition release of SNOMED CT® distributed by SNOMED International, which is subject to the SNOMED CT® Affiliate License, details of which may be found at  https://www.snomed.org/snomed-ct/get-snomed."
2022-06-02 18:39:24,602 [main] INFO  com.eldrix.hermes.core - importing 0 modules
2022-06-02 18:39:24,906 [async-thread-macro-1] INFO  com.eldrix.hermes.importer - Processing:  "der2_cciRefset_RefsetDescriptorSnapshot_IN1000189_20210806.txt"  type:  "RefsetDescriptorRefset"
2022-06-02 18:39:24,931 [async-thread-macro-1] INFO  com.eldrix.hermes.importer - Processing:  "sct2_Concept_Snapshot_IN1000189_20210806.txt"  type:  "Concept"
2022-06-02 18:39:24,981 [async-thread-macro-34] ERROR com.eldrix.hermes.impl.store - import error: failed to import data:  {:type :info.snomed/RefsetDescriptorRefset, :parser #object[com.eldrix.hermes.snomed$parse_snomed_filename$fn__1048 0x70360744 "com.eldrix.hermes.snomed$parse_snomed_filename$fn__1048@70360744"], :headings ["id" "effectiveTime" "active" "moduleId" "refsetId" "referencedComponentId" "attributeDescription" "attributeType" "attributeOrder"], :data [#com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem{:id #uuid "3dddb01f-6e1d-473e-8e70-b1b3f2887439", :effectiveTime #object[java.time.LocalDate 0x1e065e3f "2019-11-22"], :active true, :moduleId 683851000189105, :refsetId 672411000189107, :referencedComponentId 672411000189107, :attributeDescriptionId 900000000000461009, :attributeTypeId 900000000000461009, :attributeOrder 0}]}
Exception in thread "main" clojure.lang.ExceptionInfo: Import error {:batch {:type :info.snomed/RefsetDescriptorRefset, :parser #object[com.eldrix.hermes.snomed$parse_snomed_filename$fn__1048 0x70360744 "com.eldrix.hermes.snomed$parse_snomed_filename$fn__1048@70360744"], :headings ["id" "effectiveTime" "active" "moduleId" "refsetId" "referencedComponentId" "attributeDescription" "attributeType" "attributeOrder"]}, :data {:type :info.snomed/RefsetDescriptorRefset, :parser #object[com.eldrix.hermes.snomed$parse_snomed_filename$fn__1048 0x70360744 "com.eldrix.hermes.snomed$parse_snomed_filename$fn__1048@70360744"], :headings ["id" "effectiveTime" "active" "moduleId" "refsetId" "referencedComponentId" "attributeDescription" "attributeType" "attributeOrder"], :data [#com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem{:id #uuid "3dddb01f-6e1d-473e-8e70-b1b3f2887439", :effectiveTime #object[java.time.LocalDate 0x1e065e3f "2019-11-22"], :active true, :moduleId 683851000189105, :refsetId 672411000189107, :referencedComponentId 672411000189107, :attributeDescriptionId 900000000000461009, :attributeTypeId 900000000000461009, :attributeOrder 0}]}, :exception {:via [{:type java.lang.ClassCastException, :message "class com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem cannot be cast to class com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem (com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem is in unnamed module of loader clojure.lang.DynamicClassLoader @7a606260; com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem is in unnamed module of loader 'app')", :at [com.eldrix.hermes.impl.ser$write_refset_descriptor_refset_item invokeStatic "ser.clj" 357]}], :trace [[com.eldrix.hermes.impl.ser$write_refset_descriptor_refset_item invokeStatic "ser.clj" 357] [com.eldrix.hermes.impl.ser$write_refset_descriptor_refset_item invoke "ser.clj" 356] [com.eldrix.hermes.impl.ser$fn__11117 invokeStatic "ser.clj" 411] [com.eldrix.hermes.impl.ser$fn__11117 invoke "ser.clj" 409] [clojure.lang.MultiFn invoke "MultiFn.java" 234] [com.eldrix.hermes.impl.lmdb$write_refset_items invokeStatic "lmdb.clj" 257] [com.eldrix.hermes.impl.lmdb$write_refset_items invoke "lmdb.clj" 226] [com.eldrix.hermes.impl.store$fn__11381 invokeStatic "store.clj" 434] [com.eldrix.hermes.impl.store$fn__11381 invoke "store.clj" 432] [clojure.lang.MultiFn invoke "MultiFn.java" 234] [com.eldrix.hermes.impl.store$write_batch_one_by_one$fn__11398 invoke "store.clj" 441] [com.eldrix.hermes.impl.store$write_batch_one_by_one invokeStatic "store.clj" 440] [com.eldrix.hermes.impl.store$write_batch_one_by_one invoke "store.clj" 436] [com.eldrix.hermes.impl.store$write_batch_with_fallback invokeStatic "store.clj" 451] [com.eldrix.hermes.impl.store$write_batch_with_fallback invoke "store.clj" 447] [com.eldrix.hermes.core$do_import_snomed$fn__14256 invoke "core.clj" 687] [clojure.core$map$fn__5931$fn__5932 invoke "core.clj" 2759] [clojure.core.async.impl.channels$chan$fn__1328 invoke "channels.clj" 304] [clojure.core.async.impl.channels.ManyToManyChannel put_BANG_ "channels.clj" 147] [clojure.core.async$fn__6536 invokeStatic "async.clj" 172] [clojure.core.async$fn__6536 invoke "async.clj" 164] [clojure.core.async$pipeline_STAR_$process__6720 invoke "async.clj" 534] [clojure.core.async$pipeline_STAR_$fn__6732 invoke "async.clj" 549] [clojure.core.async$thread_call$fn__6643 invoke "async.clj" 484] [clojure.lang.AFn run "AFn.java" 22] [java.util.concurrent.ThreadPoolExecutor runWorker nil -1] [java.util.concurrent.ThreadPoolExecutor$Worker run nil -1] [java.lang.Thread run nil -1]], :cause "class com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem cannot be cast to class com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem (com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem is in unnamed module of loader clojure.lang.DynamicClassLoader @7a606260; com.eldrix.hermes.snomed.RefsetDescriptorRefsetItem is in unnamed module of loader 'app')"}}
	at com.eldrix.hermes.impl.store$write_batch_one_by_one$fn__11398.invoke(store.clj:444)
	at com.eldrix.hermes.impl.store$write_batch_one_by_one.invokeStatic(store.clj:440)
	at com.eldrix.hermes.impl.store$write_batch_one_by_one.invoke(store.clj:436)
	at com.eldrix.hermes.impl.store$write_batch_with_fallback.invokeStatic(store.clj:451)
	at com.eldrix.hermes.impl.store$write_batch_with_fallback.invoke(store.clj:447)
	at com.eldrix.hermes.core$do_import_snomed$fn__14256.invoke(core.clj:687)
	at clojure.core$map$fn__5931$fn__5932.invoke(core.clj:2759)
	at clojure.core.async.impl.channels$chan$fn__1328.invoke(channels.clj:304)
	at clojure.core.async.impl.channels.ManyToManyChannel.put_BANG_(channels.clj:147)
	at clojure.core.async$fn__6536.invokeStatic(async.clj:172)
	at clojure.core.async$fn__6536.invoke(async.clj:164)
	at clojure.core.async$pipeline_STAR_$process__6720.invoke(async.clj:534)
	at clojure.core.async$pipeline_STAR_$fn__6732.invoke(async.clj:549)
	at clojure.core.async$thread_call$fn__6643.invoke(async.clj:484)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
The command '/bin/sh -c java -jar hermes.jar -d ./snomed.db import ./extracts' returned a non-zero code: 1

[wardle]

Yes. To run on my m1, I install an x86 jvm and run using that. Currently lmdb doesn't come with arm64 binaries. The performance remains excellent despite using Rosetta.

[wardle]

See lmdbjava/native#10

[wardle]

Also, @sidharthramesh - I looked at your docker file.

You no longer need -Xmx. You should compact before indexing. It will make indexing faster I think.

[wardle]

Hi @sidharthramesh , did you get all of this working with the 0.12 series hermes, which uses an lmdb backend instead of the mapdb backend? If so, how has it gone in production? We have not had any issues whatsoever, but it will be useful to get independent verification!

[wardle]

Thanks. Glad it is working.

[wardle]

Fixed in baca791
See https://github.com/wardle/hermes#can-i-use-hermes-on-apple-silicon

In essence, there are now two ways to run Hermes on Apple Silicon.

First, run using an x86-64 JVM. This works because Mac OS X can run x86 binaries using Rosetta.
Second, since v1.0.822, you can build your own native lmdb binary, and use that.