Welcome to the Encyclopedia of Life project. The bulk of the code needed to run www.eol.org is written in Ruby on Rails and is made available to anyone for re-use, repurposing or for improvement. This is both an ambitious project and an ambitious codebase and we are excited to share it with the open source community. The code has been under development since approximately September 2007, but has undergone many revisions and updates from June 2008-December 2008. There is much work to be done, both in adding new features, and in the ongoing process of code refactoring and performance improvements. If you see something you like, share it with your colleagues and friends and reuse it in your own projects. If you see something you don’t like, help us fix it or join the discussion on GitHub or Google Code.
The full code base is released under the MIT License. Details are available in the “MIT-LICENSE.txt” file at the root of the code folder.
This is a big Rails project. Clearly, some of the installation steps below are big and require multiple steps, but if you are currently a Rails developer, some are already done (like installing Ruby). If you are not a Rails developer, we suggest you first visit www.rubyonrails.org for more information on getting started with Rails and then return to us when you have your feet wet. The www.eol.org codebase probably shouldn’t be the first Rails project you’ve ever seen.
For seasoned Rails developers, you’ll also notice the codebase does some mix and matching–both restful controllers and regular controllers, for example. We like the restful way of doing things and plan to move in that direction. Some methods are monolithic, others are quite small. …and so on. This codebase has been developed by quite a mix of developers and skill levels and we apologize for the lack of homogeneity. Please bear with us as we improve the code, slowly, over time.
Hint: start with the TaxonConcept model. Everything stems from there.
To get things up and running, these are the steps you need to take. If you actually run through this process, please update this list with any changes you find necessary!
Note that many of these steps require root access on your machine. You have been warned and may need to run them as “sudo” on a Mac/Linux or as an administrator on Windows (there, I acknowledged the existence of Windows).
The following set of commands is pretty much all I needed to do to get up and running on Ubuntu 10.10. Your mileage may, of course, vary:
1. sudo apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 rubygems 2. sudo apt-get install libreadline-ruby1.8 libruby1.8 libopenssl-ruby 3. sudo apt-get install libxml2 libxslt-dev libxml2-dev 4. sudo apt-get install git 5. ssh-keygen -t rsa -c "[email protected]" 6. cat .ssh/id_rsa.pub # ...and put that in your github account. 7. git clone [email protected]:EncyclopediaOfLife/eol.git eol 8. sudo gem install rails -v 2.3.8 9. sudo gem install rspec rspec-rails ruby-debug 10. sudo apt-get install rake 11. sudo apt-get install mysql-server libmysqlclient-dev # It asked for the pw three times, here... 12. sudo gem install will_paginate sanitize mysql daemons optiflag # Last two are required by Solr 13. sudo gem install capybara -v=0.3.9 14. rake gems:refresh_specs 15. rake db:migrate ; rake db:migrate RAILS_ENV=test ; rake db:migrate RAILS_ENV=test_master 16. sudo apt-get install memcached 17. sudo vim /etc/apt/sources.list # ... and uncomment the lines for the canonical source. 18. sudo apt-get update 19. sudo apt-get install sun-java6-jre sun-java6-plugin 20. ...set up your configuration. This is, of course, a long story. Read on.
TODO - Could we specific all of those gems (at least) in the test.rb environment and use gems:install RAILS_ENV=test ?
Things you need:
1. Ruby 1.8 ...EOL won't work with 1.9 and we have no immediate plans to port. 2. Rails 2.3.8 ...No other version will do (again, no plans to port)! 2. MySQL (server) 3. libxml 4. rake 5. The following gems: rspec rspec-rails ruby-debug will_paginate sanitize mysql daemons optiflag capybara 0.3.9 6. memcached 7. Java ...to run Solr.
EOL requires Solr (a fast indexing engine) to run properly. Solr is run using Java, and the required JAR file is included with the EOL project’s source code. (TODO - we need to check the legal situtaion with this–we may need to include a LICENCSE for it.) There is a rake task for starting (solr:start) and stopping (solr:stop) the Solr server. Searches will not function without Solr, and at some point, the entire codebase (rendering a taxa page) may require Solr as well.
To re-build your indexes for Solr searching (up to 100 TaxonConcepts), run the command:
rake solr:build
…Note that this command first deletes all existing entries, then adds entires (max: 100) for each TaxonConcept in the development. If you want to build indexes based on the data in your test (or integration, or…) environments, specify the RAILS_ENV:
rake solr:build RAILS_ENV=test
-
Setup the config/database.yml
-
Copy “config/database.sample.yml” to “config/database.yml”
-
Create/update the appropriate entries for your development, test, and test_master environments.
-
For development purposes, the
demo,integration, andproductionenvironments may be removed. -
For production purposes, set the “master_database” to the master database for the core rails database and the “master_data_database” to the master database for the data database (see more info below in the “READ/WRITE SPLITTING” topic)
-
-
Installing and running memcached
This project doesn’t run without memcached. The reason for this is that no other caching mechanism seems to properly persist Ruby objects. If you really need to run some other caching system, modify your config/environments/development.rb to use what you prefer, but you’re on your own.
To install memcached on OSX with MacPorts already on the system is quite easy:
a. Run "sudo port install memcached". This will install the packages required. b. At the end of the output from this, it should recommend you run another command, probably "sudo launchctl load \ -w /Library/LaunchDaemons/org.macports.memcached.plist", but you should just copy/paste and run whatever it suggests, as the path may be different.
That’s it! Memcached is now running and will work for development. Be mindful of this–it may require periodic restarting or flushing (from a Rails console, you can just type “$CACHE.clear”).
Note that you do NOT need to install the “memcache” Ruby gem.
-
Create the tables. Here’s a handy list of commands:
mysql -u root drop database eol_data_development ; drop database eol_development; drop database eol_logging_development; drop database eol_data_test; drop database eol_test; drop database eol_logging_test; drop database eol_test_master; drop database eol_data_test_master; drop database eol_logging_test_master; create database eol_data_development character set = utf8 collate = utf8_general_ci; create database eol_development character set = utf8 collate = utf8_general_ci; create database eol_logging_development character set = utf8 collate = utf8_general_ci; create database eol_data_test character set = utf8 collate = utf8_general_ci; create database eol_test character set = utf8 collate = utf8_general_ci; create database eol_logging_test character set = utf8 collate = utf8_general_ci; create database eol_test_master character set = utf8 collate = utf8_general_ci; create database eol_data_test_master character set = utf8 collate = utf8_general_ci; create database eol_logging_test_master character set = utf8 collate = utf8_general_ci; quit; rake db:migrate ; rake db:migrate RAILS_ENV=test ; rake db:migrate RAILS_ENV=test_master
Your “URL” (or an alternative) will be provided to you privately if needed. Most developers who will not run code in production can skip this one.
rake eol:site_specific repo=URL
Run the following two commands:
rake truncate rake scenarios:load NAME=bootstrap,demo rake solr:start script/server
Go to localhost:3000 and (hopefully) see stuff.
If you can’t get running the search feature hopefully it’s only because of Solr indexes missing.
Run the following commands:
rake db:drop:all rake db:create:all rake db:migrate rake db:migrate RAILS_ENV=test rake db:migrate RAILS_ENV=test_master rake solr:start rake truncate rake scenarios:load NAME=bootstrap,demo ruby spec/integration/search_spec.rb rake solr:build rake solr:build RAILS_ENV=test rake solr:rebuild_site_search rake solr:rebuild_collection_items
We’re using RSpec for our testing (see the spec/ directory). Run ‘rake’ or ‘rake spec’ (or ‘rake specdoc’ for specdoc output)
Possible error: “You haven’t loaded the foundation scenario, and tried to build a TaxonConcept with no vetted id.”
If you get this error, you may need to point your code to the correct version of mysqldump.
Create config/environments/local.rb (this will be ignored by revision control), and put this in it:
$MYSQLDUMP_COMPLETE_PATH = '/usr/bin/mysqldump'
(But make sure that the path you use here is really a working version of mysqldump.)
The following accounts are set up by the “bootstrap” scenario, so you can usually get into them:
Admin user username: admin password: admin Content partner username: test_cp password: test password User username: test_user2 password: password Curator of Animalia username: test_curator password: password
You can log in as any of the users or content partners with the password “test password”. There are no other consistent users, so either create your own, log in with OpenID, or use the console to find a username.
There are several consistent content partners, and the password is also “test password”. There will be IUCN, catalogueoflife, and test_cp.
Once logged in, every taxon concept should have various images that will only show up based on Vetted/Visibility values: each combination is represented.
The EOL code base uses metric_fu to evaluate the overall code quality.
To use Metric Fu, you must install the gem YOURSELF. It is the team’s opinion that metric-fu, and it’s various dependencies, are system level tools (like ruby-debug) and should not be checked into the code base.
NOTE: When I tried this recently, there was a problem in metric_fu-2.1.1/lib/metric_fu.rb …On line 4, I needed to change the “rescue LoadError” to simply “rescue”, because the error being thrown was actually “undefined method ‘map’ for nil:NilClass”. This change fixes the problem.
You can run the metrics on EOL with this command:
rake metrics:all RAILS_ENV=test
The results end up in HTML. Start with tmp/metric_fu/output/index.html
The site is built to allow for master/slave database read/write splitting for the core rails database and the core “data” database. There are two plugins involved in the use of multiple databases and read/write splitting (and please note that EOL has made changes to both plugins to accomodate our own systems):
*use_db*:: used to direct some models to a different database
(http://rails.elctech.com/blog/using-and-testing-rails-with-multiple-databases)
*masochism*:: used to split read/writes when using ActiveRecord
(http://www.planetrubyonrails.org/tags/view/masochism)
New abstract class models are created which make connections to the other databases required, and then any models which need to connect to the other databases are subclassed from the new abstract class. In our case, we have two abstract classes representing connections to the data database and the logging database:
- SpeciesSchemaModel - LoggingModel
These extra two databases are referenced in the database.yml in the following way:
- environment_data (e.g. development_data) - environment_logging (e.g. development_logging)
Read/write splitting is accomplished with the masochism plugin by adding two new database connections to the config/database.yml file:
- master_database (the master database connection for the core rails database) - master_data_database (the master database connection for the "data" database)
In addition there are new abstract classes representing a connection to each master database that can be used to run direct SQL queries against the masters:
- MasterDatabase (for the core rails database) - SpeciesSchemaWriter (for the species data database)
The logging database does not require read/write splitting since there is only a single server for this purpose.
To enable read/write splitting via ActiveRecord, include the following in the approriate environment.rb file (e.g. config/environments/production.rb):
config.after_initialize do ActiveReload::ConnectionProxy.setup! ActiveReload::ConnectionProxy.setup_for SpeciesSchemaWriter, SpeciesSchemaModel end
Note that you must also enable class caching for this to work (this is the default in production, but not in development, which is important to note if you wish to test this functionality in development mode):
config.cache_classes = true
Manually crafted SQL queries with SELECT statement will be redirected to slave while all other queries like in the following example will be redirected to master:
SpeciesSchemaModel.connection.execute("DELETE FROM data_objects WHERE id in (#{data_objects})")
You don’t have to worry about master/slave databases in development mode unless you want to test your code against splitting queries. When in development, you could make the master_database and master_data_database must point to the same place as development and development_data respectively. Things should work even if these entries are left out (since the master databases are only connected in a configuration entry in the production environment) but it doesn’t hurt if they are there.
Spots in the code requiring some attention for refactoring, cleanup or further work are marked with a “TODO” comment and sometimes with a level of priority. You can quickly locate all these comments with your IDE, an app like TextMate, or with a rake command:
rake notes:todo
Here are our rake stats as of May 7th:
+----------------------+-------+-------+---------+---------+-----+-------+ | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+-------+-------+---------+---------+-----+-------+ | Controllers | 3848 | 2665 | 34 | 262 | 7 | 8 | | Helpers | 713 | 503 | 2 | 49 | 24 | 8 | | Models | 7120 | 3981 | 104 | 473 | 4 | 6 | | Libraries | 980 | 676 | 9 | 69 | 7 | 7 | | Model specs | 1644 | 637 | 1 | 5 | 5 | 125 | | View specs | 0 | 0 | 0 | 0 | 0 | 0 | | Controller specs | 0 | 0 | 0 | 0 | 0 | 0 | | Helper specs | 10 | 7 | 0 | 0 | 0 | 0 | | Library specs | 0 | 0 | 0 | 0 | 0 | 0 | +----------------------+-------+-------+---------+---------+-----+-------+ | Total | 14315 | 8469 | 150 | 858 | 5 | 7 | +----------------------+-------+-------+---------+---------+-----+-------+ Code LOC: 7825 Test LOC: 644 Code to Test Ratio: 1:0.1
We really need to start improving our tests. To that end, I would like to start using:
rails stats reek flay flog roodi reek saikuro rcov churn
There are lots of configuration settings, and they load in the following order:
1) config/environment.rb 2) config/environments/[RAILS_ENV].rb 3) config/environments/[RAILS_ENV]_eol_org.rb 4) config/environment_eol_org.rb
The logging model is intended to be thought of as a data mining system. A separate database is used to store all log data, which must be defined in your config/database.yml file. (See sample file for naming.) Models and operations tend to fall into two categories: dimensions and facts. In short, dimensions represent collected (primary) data. Facts are derived (secondary) caches of information which is more meaningful to the user. Fact table design is highly dependent on the user interface design, because we only need to generate facts if the information will actually be shown. For performance reasons regarding the expected database size, fact tables are also intended to be highly denormalized, non-authoritive sources of information.
Location-based facts require the primary data to go through a geocoding process which requires an external web service. This process is thus performed asynchronously from the main site. Results of IP location lookups are cached and reused whenever possible. While IP location lookups are non-authoritative “best guesses”, they nevertheless provide meaningful information.
In production mode it is CRITICALLY important to understand the automated logging tasks before invoking them to avoid deletion of precious data. To develop logging features, run the following tasks in the given order to populate your logging database with mock data…
rake logging:clear # Deletes all logging-related records. (WARNING: NEVER run in production.) rake logging:dimension:mock THOUSANDS=2 # Creates 2,000 psedo-random mock log entries (a.k.a. primary data). rake logging:geocode:all # Performs geocoding on the primary data, using caches where possible. rake logging:fact:all # Derives secondary data from primary data.
…at this point you should see data in the graph pages of the web application. Alternatively, run the following which does all of the above in one step.…
script/runner script/logging_mock
For cron jobs, you’ll likely want to log all facts for a particular date range:
rake logging:fact:today rake logging:fact:yesterday rake logging:fact:range FROM='01/15/2007' TO='12/19/2008'
Any links to external sites that need to be tracked should use the following two helpers:
external_link_to(text, url) external_link_to(image_tag(image), url)
Both will generate a link (with either the supplied text or the supplied image url) to the supplied URL. The link will be logged in the database, and if the $USE_EXTERNAL_LINK_POPUPS parameter is set to TRUE in the environment.rb file, a javascript pop-up warning window is shown prior to following the link. The following additional parameters can be passed after the URL for both methods:
+:new_window => true or false+:: determines if link appears in new browser window (defaults to true) +:show_only_if_link => true or false+:: determines if image or text is shown if no URL was supplied (defaults to false) +:show_link_icon => true or false+:: determines if the external icon image is shown after the link (defaults to true for text links and false for image links)
For images, the following parameters can also be passed:
+:alt => 'value'+:: alt tag is set with the value passed +:title => 'value'+:: title tag is set with the value passed
Currently no reports are provided for external link tracking, all links are stored in the “external_link_logs” in the logging database for later reporting.
Fragment caching is enabled in the specific environment file (e.g. config/production.rb) and the storage mechanism (i.e. memcached) must be set as well.
For memcached:
config.cache_store = :mem_cache_store, '10.0.0.1:11211', '10.0.0.2:11211'
To enable caching:
config.action_controller.perform_caching = true
All “static” pages coming out of the CMS are fragment cached and the home page cache is cleared each hour (or as set in the $CACHE_CLEAR_IN_HOURS value set in the config/environment.rb file), using language as key to enable multiple fragments. The header and footer navigation of each page is also fragment cached on cleared at the same time interval. When changes are made in the admin interface, these caches are automatically cleared.
Names searches are cached by query type, language and vetted/non-vetted status.
Species pages are cached using the following attributes as keys (since each will cause a different species page to be created). Note that when logged in as an administrator or content partner, the pages are not cached and are generated dynamically each time.
Variables for naming species page fragment caches:
- taxon_id - language - expertise level - vetted or all information - default taxonomic browser - curator for page
Species page caches can be cleared by taxon ID by a CMS Site Administrator by logging into the admin console, and going to “General Site Admin”. Clearing a species page cache automatically clears all of its ancestors as well.
The following URLs can be used to trigger page expiration either manually in the browser or via a web service call. They only work if called from “allowed” IPs (i.e. as specified in configuration) as defined in the application level method “allowed_request” (which returns TRUE or FALSE).
localhost:3000/expire_all # expire all non-species pages localhost:3000/expire_taxon/TAXON_ID # expire specified taxon ID (and it's ancestors) localhost:3000/expire_taxa/?taxa_ids=ID1,ID2,ID3 # will expire a list of taxon IDs (and their unique ancestors) specified in the querystring (or post) parameter "taxa_ids" (separate by commas) localhost:3000/clear_caches # expire all fragment caches (if supported by data store mechanism)
From within the Rails applications, use the following application level methods:
expire_all # expire all non-species pages expire_taxon(taxon_ID) # expire specified taxon id and ancestors (unless :expire_ancestors=>false is set) expire_taxa(taxon_ID_array)# expire specified array of taxon ID and unique ancestors (unless :expire_ancestors=>false is set) clear_all_caches # expire all fragment caches (everything!)
For testing purposes, to install memcached on a Mac see readystate4.com/2008/08/19/installing-memcached-on-os-x-1054-leopard/ If you have a local memcached server installed (Mac/Linux), start it with: “memcached -d -m 24 -p 11211” and stop it with “killall memcached”
This is now using the asset_packager plugin, see details at synthesis.sbecker.net/pages/asset_packager
If you add a javascript include files and you want them included in the page, you must edit the “config/asset_packager.yml” file and place them in the order you wish them to be loaded. When running in development mode, the JS and CSS are included separately each time. When running in production mode, assets are included from packaged entities. A rake task is used to combine and minify CSS and JS referenced. Note that the order the JS files are listed in the config file is the order they are merged together and this order can matter.
You must run this rake task each time the JS/CSS is updated to ensure the latest version is present when running in production mode. The minification process is very sensitive to missing semicolons, extra commas and what not that are dealt with by modern browsers (not IE though…). You have been warned - minification can and will break your JS if you are not careful (watch those semicolons)!
To update/create the combined versions:
rake asset:packager:build_all
In production, this rake command is run as part of the capistrano deploy script.
For testing purposes, you can force the minified/combined version to be referenced in your pages for a particular environment by adding the following line to your “config/environments/development.rb” or “config/environment.rb” file:
Synthesis::AssetPackage.merge_environments = ["development", "production"]
To get attribution for a given taxon concept ID:
-
Get TaxonConcept e.g.
t=TaxonConcept.find(101)
-
Look at hierarchy_entry for that taxon (could be many) e.g.
he_all=t.hierarchy_entries OR he=t.entry (for the default)
-
Look at the associated hierarchy (could be one of many if you get them all) e.g.
h=he_all[0].hierarchy # OR h=he.hierarcy h.label h.agent.full_name h.agent.hompage h.agent.logo_cache_url
-
Look at the associated agents for the hierarchy_entry e.g.
agents=he[0].agents # OR agents=he.agents agents.each {|agent| puts agent.full_name + " " + agent.homepage + " " + agent.logo_cache_url}
To create Google SiteMap files in the correct format, run the following rake task for your requested environment:
rake sitemap:create RAILS_ENV=production[,BASEURL="http://www.eol.org/pages/",BASEURL_SITEMAP="http://www.eol.org/sitemaps/",MAXPERFILE="50000",OUTPUTPREFIX="eol_sitemap",PRIORITY="1",CHANGEFREQ="monthly",LASTMOD="2009-03-01"]
All of the parameters in brackets are optional and have the default shown (except for ‘lastmod’ which defaults to today).
The URLS placed into the site map file are based on ‘BASEURL/XXX’ where XXX is a valid published, trusted taxon concept ID pulled from the specified environment.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent