gsa / i14y

Search engine for agencies' published content

i14y

Search engine for agencies' published content

Dependencies/Prerequisites

• Ruby

Use rvm to install the version of Ruby specified in .ruby-version.

Docker

Docker can be used to: 1) run just the required services (MySQL, Elasticsearch, etc.) while running the i14y application in your local machine, and/or 2) run the entire i14y application in a Docker container. Please refer to searchgov-services for detailed instructions on centralized configuration for the services.

When running in a Docker container (option 2 above), the i14y application is configured to run on port 3200. Required dependencies - (Ruby, and Gems) - are installed using Docker. However, other data or configuration may need to be setup manually, which can be done in the running container using bash.

Using bash to perform any operations on i14y application running in Docker container, below command needs to be run in search-services.

$ docker compose run i14y bash

For example, to setup DB in Docker:

$ docker compose run i14y bash
$ bin/rails i14y:setup

The Elasticsearch services provided by searchgov-services is configured to run on the default port, 9200. To use a different host (with or without port) or set of hosts, set the ES_HOSTS environment variable. For example, use following command to run the specs using Elasticsearch running on localhost:9207:

ES_HOSTS=localhost:9207 bundle exec rspec spec

Verify that Elasticsearch 7.17.x is running on the expected port (port 9200 by default):

$ curl localhost:9200
{
  "name" : "002410188f61",
  "cluster_name" : "es7-docker-cluster",
  "cluster_uuid" : "l3cAhBd4Sqa3B4SkpUilPQ",
  "version" : {
    "number" : "7.17.7",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78dcaaa8cee33438b91eca7f5c7f56a70fec9e80",
    "build_date" : "2022-10-17T15:29:54.167373105Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Development

• bundle install.
• Copy config/secrets_example.yml to config/secrets.yml and fill in your own secrets. To generate a random long secret, use rake secret.
• Run bundle exec rake i14y:setup to create the neccessary indexes, index templates, and dynamic field templates.

If you ever want to start from scratch with your indexes/templates, you can clear everything out: bundle exec rake i14y:clear_all

• Run the Rails server on port 8081 for compatibility with the search-gov app:

$ rails s -p 8081

You should see the default Rails index page on http://localhost:8081/.

Code Quality

We use Rubocop for static code analysis. Settings specific to I14Y are configured via .rubocop.yml. Settings that can be shared among all Search.gov repos should be configured via the searchgov_style gem.

Basic Usage

Create a collection for storing documents

$ curl -u dev:devpwd -XPOST http://localhost:8081/api/v1/collections \
 -H "Content-Type:application/json" -d \
 '{"handle":"test_collection","description":"my test collection","token":"test_collection_token"}'

Create a document within that collection

Use the collection handle and token for authorization:

curl http://localhost:8081/api/v1/documents \
  -XPOST \
  -H "Content-Type:application/json" \
  -u test_collection:test_collection_token \
  -d '{"document_id":"1",
      "title":"a doc about rutabagas",
      "path": "http://www.foo.gov/rutabagas.html",
      "created": "2020-05-12T22:35:09Z",
      "description":"Lots of very important info on rutabagas",
      "content":"rutabagas",
      "promote": false,
      "language" : "en",
      "tags" : "tag1, another tag"
      }'

Search for a document within a collection

$ curl -u dev:devpwd http://localhost:8081/api/v1/collections/search?handles=test_collection&query=rutabaga

Tests

# Fire up Elasticsearch in search-services
$ docker-compose up elasticsearch7

$ bundle exec rake i14y:setup
$ rake

i14y's Issues

Implement Swagger API docs

https://github.com/tim-vandecasteele/grape-swagger

I kept running into problems integrating this. Perhaps someone else will have better luck.

fix order-dependent spec

Steps:
Run the spec suite with the 35490 seed:
rspec spec/ --seed 35490

Expected Results:

• all specs pass

Actual Results:

• One spec fails:

Failures:

  1) API::V1::Documents POST /api/v1/documents success case stores the appropriate fields in the Elasticsearch document
     Failure/Error: expect(document.path).to eq("http://www.gov.gov/goo.html")

       expected: "http://www.gov.gov/goo.html"
            got: "http://www.gov.gov/url4.html"

       (compared using ==)
     # ./spec/requests/api/v1/documents_spec.rb:62:in `block (4 levels) in <top (required)>'

Support Attachments

Problem:
I need to index documents (PDF, .DOCX, etc)

(Generic) Solution
Create mapping for attachments. Allow users to insert attachment metadata (base64 encoding?) to i14y.

Add this repository to the GSA code inventory

(I work in GSA IT, Office of the CTO. I am submitting this as part of our work to ensure GSA complies with the new Federal Source Code Policy.)

GSA needs to create an inventory of all agency source code, whether open source or closed source. The inventory we create will appear on Code.gov. The inventory will contain basic information about each source code repository, but will not include the source code itself. Please read the implementation guide and use it to submit this repository to the inventory by December 5.

Basically, please do one of the following, the details of which are described in the implementation guide:

• Add a metadata file (.codeinventory.yml or .codeinventory.json) to this repository (optionally, use this tool to generate a metadata file)
• Submit your metadata via a form (only do this if you cannot add a metadata file to this repository)

Let me know if you would like me to open a PR with an example .codeinventory.yml file.

Please let me know if you have any questions.

Thanks!

---

References:

• GSA Open Source Implementation Guide
• GSA Open Source Policy
• Federal Source Code Policy

Add collection stats API endpoint

Initially provide document count & last upload datetime.

Add CircleCI testing support

We have enabled CircleCI testing for this repo but we still need to add a properly-configured .circleci/config.yml file to enable testing of this app against CircleCI 2.0.

Since there are no secrets in this repo, we have enabled CircleCI testing of forked branches. We'd love it if someone would step in and implement a .circleci/config.yml with these behaviors:

• Trigger on pull requests only (already configured in CircleCI)
• Execute the specs in spec/
• Fail on any spec failures, or on any drop below 100% coverage
• Verify that the CircleCI status badge in the README.md works correctly.

Allow GET/HEAD requests for documents

Problem:
Applications using i14y might need to know the existence of a particular resource. For example, a document that was created, then deleted would have to guess whether or not a document exists for future requests. This leads to a lot of unhelpful errors.

Solution
Allow GET requests on a document, to test existence.