Comments (3)
Check whether CustomRunAutomaton can create a transition table!
Update: "@param tableize if true, a transition table is created which makes the run
method faster in return of a higher memory usage" -> turn tableize off
from chemspot.
Does that change make any significant difference? I think the way LINNAEUS is encoding each dictionary entry may be intrinsically quite memory inefficient; I believe it is essentially creating a state machine that matches n * (entry + delimiter +id)
As the ids are all unique it is not possible for the states that recognise the different entries to be shared so the state machine scales linearly with the number of entries in the input rather than sublinearly.
If you can do without the IDs, have you considered using the underlying libraries function:
http://www.brics.dk/automaton/doc/dk/brics/automaton/BasicAutomata.html#makeStringUnion%28java.lang.CharSequence...%29
(once created you probably would use the Automaton to create a RunAutomaton which is faster/more memory efficient and serialise that)
Due to not capturing the ids the resultant automaton will be substantially smaller. By taking advantage of the fact that the input is only strings (not regexes) the time to create the automaton is also vastly reduced. I assume building the automaton with LINNAEUS took hours?
from chemspot.
Many thanks Daniel! You were absolutely right. I am now using brics directly and it reduced the main-memory consumption to 25% compared to LINNAEUS. Also, building this automaton now takes 2 days maximum (compared to 2 weeks). I agree... normalization should be a subsequent step and not included in the automaton.
from chemspot.
Related Issues (20)
- Evaluate OSCAR on CRAFT HOT 1
- Solve problem with abstract entities in CRAFT corpus
- Improve match expansion HOT 1
- Improve recognition of short terms
- Make ChemSpot configurable HOT 1
- Maven deployment HOT 1
- Tagging text from command-line does not work
- Brics error when initializing ChemSpot HOT 1
- Loading resources from within the jar fails
- Tagging text is slow HOT 1
- Generic drug tagger constructor HOT 1
- Maven import is missing eumed.jar dependency HOT 3
- Suppress "Couldn't open edu.umass.cs.mallet.base.util.MalletLogger resources/logging.properties file" error HOT 1
- eumed-light.jar with newer version of scala
- change scala version to 2.11.8
- Unable to access jarfile chemspot.jar HOT 1
- Couldn't download SCAI corpus
- Seeking description of the columns in the output
- Tagging 1000 documents HOT 1
- Chemspot REST interface
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chemspot.