sastafford / cooking-show

Tutorial to help learn MarkLogic

MarkLogic Cooking Show

The MarkLogic Cooking Show is a tutorial designed to help learn developers learn MarkLogic. The tutorial starts by ingesting several data sets from the New York City Open Data Project via Apache NiFi. Once data is ingested, several queries will be executed in MarkLogic's Query Console. Setup for MarkLogic and Apache NiFi are facilitated using Docker.

Prerequisites

• Docker version 18.06.1-ce
• docker-compose version 1.22.0
• Docker Hub account

How do I set up?

1. Clone this repository

2. Create a MarkLogic Docker Image. See README under the Docker directory

3. Edit the docker-compose file under the project root directory. Change the image for the "marklogic" service to the docker image created in the previous step.

4. Start the set of containers. Make sure to wait about a minute for all these containers to initialize.

docker-compose up -d

If the docker containers fail to initialize, make sure to allocate more memory and CPU's to the docker daemon.

5. Copy the following data sets into the data directory.

• New York City Borough Boundaries
• New York City Community Districts
• New York City Neighborhood Names
• New York City Places of Interest

What docker containers are running?

MarkLogic

• MarkLogic Admin Console: http://localhost:8001
• Query Console: http://localhost:8000/qconsole

Apache NiFi

Apache NiFi is used to batch ingest data into MarkLogic.

• Apache NiFi: http://localhost:8080/nifi

Gradle

Gradle is a build tool that is used to initialize MarkLogic. There are three steps of initialization that are executed. The first step is to initialize MarkLogic including installing the security database and the admin user. The final step is to install a database called cooking-show-content and an app server called cooking-show.

Ingest NYC Data

1. The Points of Interest data file is not valid JSON due to an extra carriage return on line 278826. Recommend running this JSON file through a JSON Lint program.

2. Open up Apache NiFi.

3. Import the template

./nifi/templates/Ingest_GeoJSON.xml

4. Instantiate the template that was imported on the previous step.

5. Open up the configuration of the PutMarkLogic processor (there should be the WARNING sign on this processor).

6. Select the properties tab. Under the DatabaseClient Service, click on the arrow pointing to the right.

7. Click on the settings for the first row (the gear icon)

8. Click on the properties tab, set the password to "admin" and hit apply.

9. Enable the connection by clicking on the lightning bolt icon.

10. Select all the processors and press the Play button from the Operate panel. This starts up the data flow.

11. Copy the four geojson data files from ./data to ./nifi/staging

Query the Data

1. Open up Query Console.
2. Select the cooking-show-content database from the drop down in the top left hand panel.
3. Select the "Explore" button. There should be 20196 documents in the database.
4. Import the workspace found under ./MarkLogic/Workspace.xml. The Workspace panel is in the top right panel.
5. Review each of the queries starting from left to right.

How do I uninstall?

To remove these docker images execute the following command.

docker-compose down

The MarkLogic forest information is stored in a docker volume and will persist beyond the lifetime of your MarkLogic docker container.

cooking-show's Issues

Add example of geospatial feature conflation