Coder Social home page Coder Social logo

usda-nutrient-database's Introduction

Stories in Ready Code Climate Build Status Coverage Status Dependency Status Gem Version Gitter chat

USDA Nutrient Database

The USDA nutrition database is a great source of nutrition information. However, the data formatting options (plain text ASCII files or MS-Access) leave a little to be desired.

This is a simple gem to import the database into your ruby application.

Using the USDA Nutrient Database REST API instead

Even after importing this data, keeping it up to date and adding/maintaining additional logic such as full text search, weight conversion, etc is a pain. As an alternative I have put together a small RESTful web service to do this for you. You can find the docs at http://docs.usdanutrientservice.apiary.io/

Installation

Add this line to your application's Gemfile:

gem 'usda-nutrient-database'

And then execute:

$ bundle

Or install it yourself as:

$ gem install usda-nutrient-database

If you're using rails then copy the migrations across:

rake usda_nutrient_database_engine:install:migrations

Configuration

UsdaNutrientDatabase.configure do |config|
  config.batch_size = 20000 # import batch size, if using activerecord-import
  config.perform_logging = true # default false
  config.logger = Rails.logger # default Logger.new(STDOUT)
  config.usda_version = 'sr25' # default sr28
end

Usage

Importing with UPSERT (takes around 2 minutes)

If you are running MySQL => 5.6 or PostgreSQL => 9.5 then you're in luck, you can use UPSERT (insert or update) to speed up imports x30. To do this you're going to need to install activerecord-import

require 'activerecord-import/base'
ActiveRecord::Import.require_adapter('postgresql')

# You may want to disable logging during this process to avoid dumping huge SQL
# strings in to your logs
ActiveRecord::Base.logger = Logger.new('/dev/null')

Now run the rake task which will import everything in around 2 minutes.

rake usda:import

Importing without UPSERT (takes 60+ minutes)

Import the latest data with the import task:

rake usda:import

This is going to take a while. 60+ minutes on my 2.66 GHz i7 macbook pro. You ca also import individual tables using the other import tasks. To see the available tasks:

rake -T usda

Models

Use the models to query and profit:

UsdaNutrientDatabase::FoodGroup
UsdaNutrientDatabase::Food
UsdaNutrientDatabase::Nutrient
UsdaNutrientDatabase::FoodsNutrient
UsdaNutrientDatabase::Weight
UsdaNutrientDatabase::SourceCode

Issues

If you find a bug then add it to the issues here on github. Ideally with a pull request to fix it, or at least with a failing test. When you report a bug, number 1 rule is: don't be a dick.

Any feature requests/ideas can also go in the issues list.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Contributors

Many thanks to the following contributers:

Versioning

This gem follows Semantic Versioning

usda-nutrient-database's People

Contributors

benfalk avatar mattbeedle avatar waffle-with-pears avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

usda-nutrient-database's Issues

Clarify activerecord-import instructions

The README states that the following is needed after installing activerecord-import:

require 'activerecord-import/base'
ActiveRecord::Import.require_adapter('postgresql')

# You may want to disable logging during this process to avoid dumping huge SQL
# strings in to your logs
ActiveRecord::Base.logger = Logger.new('/dev/null')

Where is this added?

MySQL error,

Error received after successful migration file imports on rake db:migrate in a clean (new) rails application. All remove_index functions are getting some sort of error, be it this, or other.

== 20160609043411 AddReferentialIntegrity: migrating ========================== -- remove_index(:usda_nutrients, :nutrient_number) rake aborted! StandardError: An error has occurred, all later migrations canceled: Mysql2::Error:' Error on rename of './fooder/#sql-400_23' to './fooder/usda_nutrients' (errno: 150 - Foreign key constraint is incorrectly formed): DROP INDEX 'index_usda_nutrients_on_nutrient_number' ON 'usda_nutrients' /home/xxx/Sites/fooder/db/migrate/20160609043411_add_referential_integrity.usda_nutrient_database_engine.rb:4:in 'change' -e:1:in '' ActiveRecord::StatementInvalid: Mysql2::Error: Error on rename of './fooder/#sql-400_23' to './fooder/usda_nutrients' (errno: 150 - Foreign key constraint is incorrectly formed): DROP INDEX 'index_usda_nutrients_on_nutrient_number' ON 'usda_nutrients' /home/xxx/Sites/fooder/db/migrate/20160609043411_add_referential_integrity.usda_nutrient_database_engine.rb:4:in 'change' -e:1:in '' Mysql2::Error: Error on rename of './fooder/#sql-400_23' to './fooder/usda_nutrients' (errno: 150 - Foreign key constraint is incorrectly formed) /home/xxx/Sites/fooder/db/migrate/20160609043411_add_referential_integrity.usda_nutrient_database_engine.rb:4:in 'change' -e:1:in '' Tasks: TOP => db:migrate

Error on rake db:migrate after commenting the full referential integrity migration:

== 20160609043412 FixFootnotesIndexes: migrating ============================== -- change_table(:usda_footnotes) rake aborted! StandardError: An error has occurred, all later migrations canceled: Mysql2::Error: Cannot drop index 'index_usda_footnotes_on_nutrient_databank_number': needed in a foreign key constraint: DROP INex_usda_footnotes_on_nutrient_databank_number' ON 'usda_footnotes' /home/xxx/Sites/fooder/db/migrate/20160609043412_fix_footnotes_indexes.usda_nutrient_database_engine.rb:5:in 'block in chang /home/xxx/Sites/fooder/db/migrate/20160609043412_fix_footnotes_indexes.usda_nutrient_database_engine.rb:4:in 'change' -e:1:in '' ActiveRecord::StatementInvalid: Mysql2::Error: Cannot drop index 'index_usda_footnotes_on_nutrient_databank_number': needed in a key constraint: DROP INDEX 'index_usda_footnotes_on_nutrient_databank_number' ON 'usda_footnotes' /home/xxx/Sites/fooder/db/migrate/20160609043412_fix_footnotes_indexes.usda_nutrient_database_engine.rb:5:in 'block in chang /home/xxx/Sites/fooder/db/migrate/20160609043412_fix_footnotes_indexes.usda_nutrient_database_engine.rb:4:in 'change' -e:1:in '' Mysql2::Error: Cannot drop index 'index_usda_footnotes_on_nutrient_databank_number': needed in a foreign key constraint /home/xxx/Sites/fooder/db/migrate/20160609043412_fix_footnotes_indexes.usda_nutrient_database_engine.rb:5:in 'block in chang /home/xxx/Sites/fooder/db/migrate/20160609043412_fix_footnotes_indexes.usda_nutrient_database_engine.rb:4:in 'change' -e:1:in '' Tasks: TOP => db:migrate

Import download fails on Heroku

I'm using:

  • Ruby 2.2.4
  • Rails 4.2.6
  • usda-nutrient-database 2.1.1 branch no-https

I switched to the no-https branch and am still seeing a problem with the import downloading on Heroku:

$ heroku run rake usda:import
Running rake usda:import on ⬢ tranquil-citadel-34922... up, run.5471
Downloading USDA data version: sr28
Cleaning up data
rake aborted!
Faraday::ConnectionFailed: Connection refused - connect(2) for "www.ars.usda.gov" port 80
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/request/url_encoded.rb:15:in `call'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:140:in `head'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:62:in `final_path'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:42:in `block in download'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:41:in `open'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:41:in `download'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:20:in `download_and_unzip'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/importer.rb:11:in `import'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/tasks/usda_nutrient_database.rake:7:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.2.0/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
Errno::ECONNREFUSED: Connection refused - connect(2) for "www.ars.usda.gov" port 80
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/request/url_encoded.rb:15:in `call'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
/app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:140:in `head'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:62:in `final_path'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:42:in `block in download'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:41:in `open'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:41:in `download'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/import/downloader.rb:20:in `download_and_unzip'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/importer.rb:11:in `import'
/app/vendor/bundle/ruby/2.2.0/bundler/gems/usda-nutrient-database-42689b756341/lib/usda_nutrient_database/tasks/usda_nutrient_database.rake:7:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.2.0/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
Tasks: TOP => usda:import
(See full trace by running task with --trace)

From the app console on Heroku I tried:

$ heroku run console
Running console on ⬢ tranquil-citadel-34922... up, run.2796
Loading production environment (Rails 4.2.6)
irb(main):001:0> connection = Faraday.new(url: 'http://www.ars.usda.gov')
=> #<Faraday::Connection:0x007f49321c5250 @parallel_manager=nil, @headers={"User-Agent"=>"Faraday v0.9.2"}, @params={}, @options=#<Faraday::RequestOptions (empty)>, @ssl=#<Faraday::SSLOptions (empty)>, @default_parallel_manager=nil, @builder=#<Faraday::RackBuilder:0x007f49321c4fd0 @handlers=[Faraday::Request::UrlEncoded, Faraday::Adapter::NetHttp]>, @url_prefix=#<URI::HTTP http://www.ars.usda.gov/>, @proxy=nil>
irb(main):002:0> connection.head('/SP2UserFiles/Place/12354500/Data/SR/SR28/dnload/sr28asc.zip')
Faraday::ConnectionFailed: Connection refused - connect(2) for "www.ars.usda.gov" port 80
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:879:in `initialize'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:879:in `open'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:879:in `block in connect'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:73:in `timeout'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:878:in `connect'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:852:in `start'
    from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:1375:in `request'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/request/url_encoded.rb:15:in `call'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
    from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:140:in `head'
    from (irb):2
    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'irb(main):003:0> connection.head('/SP2UserFiles/Place/12354500/Data/SR/SR28/dnload/

Any ideas? Thanks!

Failure while database import

ActiveRecord::StatementInvalid: PG::InvalidColumnReference: ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.