ibm / watson-second-opinion

Get a second opinion on Amazon products by analyzing product reviews with Watson Natural Language Understanding

WARNING: This repository is no longer maintained ⚠️ USE AT YOUR OWN RISK

Create a Review Analyzer with Watson Natural Language Understanding

https://watson-second-opinion.mybluemix.net/

In this Code Pattern, we will create a Node.js app that takes the reviews from an online shopping website, Amazon, and feeds them into the Watson Natural Language Understanding service. The reviews will be stored in a Cloudant database. The Watson Natural Language Understanding service will show the overall sentiments of the reviews. The sample application will do all the reading of reviews for you and will give an overall insight about them. The Code Pattern can be useful to developers that are looking into processing multiple documents with Watson Natural Language Understanding.

When the reader has completed this Code Pattern, they will understand how to:

• Interact with Watson Natural Language Understanding using Watson's Node SDK
• Build a user interface around the result of Watson Natural Language Understanding
• Create and use Cloudant NoSQL Database
• Deploy a Nodejs application to analyze product reviews

Flow

1. The user deploys the app in IBM Cloud. The user interacts with the user interface of the app.
2. The user enters the product URL and the app would start getting the reviews of the Product.
3. The app then stores the reviews in a database for later use.
4. The app starts to upload the reviews in Watson Natural Language Understanding.
5. After Watson Natural Language Understanding finishes processing the reviews, the app then stores the result (General Sentiment and Top Entities) in Cloudant. The user will see the result in the UI.

Included components

• Watson Natural Language Understanding: Analyze text to extract meta-data from content such as overall sentiment, emotion, concepts, entities, keywords, categories, relations and semantic roles.
• Cloudant NoSQL DB: A fully managed data layer designed for modern web and mobile applications that leverages a flexible JSON schema.

Featured technologies

• Node.js: An open-source JavaScript run-time environment for executing server-side JavaScript code.
• Databases: Repository for storing and managing collections of data.
• Cloud: Accessing computer and information technology resources through the Internet.

Watch the Video

Steps

Use the Deploy to IBM Cloud button OR create the services and run locally.

Deploy to IBM Cloud

If you do not have an IBM Cloud account yet, you will need to create one here.

1. Press the above Deploy to IBM Cloud

   The toolchain uses GitHub for its source control. You may be asked to authenticate the toolchain to use your account. The toolchain will clone this repo and will be used for its deployment.

2. In Toolchains, click on Delivery Pipeline to watch while the app is deployed. Once deployed, the app can be viewed by clicking View app.

3. To see the app and services created and configured for this Code Pattern, use the IBM Cloud dashboard. The app will be named according to what you inputted in the toolchain. The following services are created and easily identified by the wso- prefix:
   • wso-nlu
   • wso-cloudant

Update the Environment of your deployed app

1. Navigate to https://cloud.ibm.com/dashboard/apps/
2. Located and click on your newly created application
3. Select Runtime in the left menu
4. Select the Environment Variables tab in the middle of the page
5. Scroll down to the User defined variables section
6. Click on add.
7. THIS IS EXTREMELY IMPORTANT. Make sure to write the name of the env variable EXACTLY as shown, otherwise, the app wont work. Scroll up until you see VCAP_SERVICES. You will then see cloudantNoSQLDB and under that url. Under Name, type in CLOUDANT_URL, and under Value, paste the url value from the cloudantNoSQLDB section of VCAP_SERVICES.

8. 🚧🚧🚧PLEASE NOTE THE GIF ABOVE IS OUTDATED. INSTEAD OF NLU_USERNAME you will have to write iam_apikey🚧🚧🚧 Repeat step 6 but now under name, type in iam_apikey and under value go to the credential section of natural-language-understanding section of the VCAP_SERVICES and get the value for apikey.
9. Click save.
10. Nice job! You are done. Click on visit App URL at the top of the page to interact with the app. Simply copy and paste an Amazon URL from a product page into the app, and click on the magnifying glass and voilà! You will get some valuable insights from Watson.

Deploy Locally

1. Clone the repo

$ git clone https://github.com/IBM/watson-second-opinion
$ cd watson-second-opinion/

2. Install Dependencies

$ npm install

3. Create IBM Cloud services

Create the following service:

• Watson Natural Language Understanding
• Cloudant NoSQL DB

4. Get Service Credentials

Once the service is created, click on it, and then click on Service credentials in the top left corner of the screen. Then click New credential and then add. For Cloudant NoSQL DB, save the value for url. For Natural Language Understanding save the value for iam_apikey.

5. Set Environment Variables

After we run npm install from step 2 above, we need to set 3 environment variables that we got from step 4. Run the following commands to set these variables (your usernames / passwords / url will be different):

$ export CLOUDANT_URL=https://f9d9e9d9-8b3f-4625-a425-7a8bbd57aeaf-bluemix:[e-mail]

$ export IAM_APIKEY=xJLkOtKCXISqQHwjfdRBb99u2r-5sJ7n_sEsfsTJXZiJ

6. Run the App

Run the app by starting Node.

$ node app.js

Go to http://localhost:4000/ in your browser of choice, and you should be greeted with the Watson Second Opinion UI.

Find a product in Amazon that you want to learn more about, copy the URL of the product page, paste it into the app, and click 🔍. After Watson Natural Language Understanding finishes processing all the reviews, the app will show you its General Sentiment and Top entities found:

Great job! You've successfully run Watson Second Opinion on your local machine! Thanks for taking the time to walk through the app with me. I am welcoming all pull requests, and any updates to the project. Have fun with it!

Links

• Watson Node.js SDK

Learn more

• Node.js Code Patterns: Enjoyed this Code Pattern? Check out our other Node.js Code Patterns
• Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns.
• AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
• With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.

Apache Software License (ASL) FAQ

watson-second-opinion's Issues

Missing file

I can't seem to find the config.json.sample file. I'm also having issues using the term "export", I have tried to use "set" instead. Either way, I can't get the app to work, because I can't put in my credentials. Please help.

rework code to only use natural language understanding

After speaking with our Discovery OM (Deepika Devarajan) we realized that the application was made unnecessarily complicated by using Discovery. This application could entirely be done with using Watson's Natural Language Understanding, a critical part of Discovery. Discovery buys us searchability and indexing of documents. But in this instance, we're simply showing results one at a time. We can make API calls to NLU, which provides the sentiment, tone, keywords, all the stuff we're showing in our UI. See https://natural-language-understanding-demo.ng.bluemix.net/ as an example.

The proposal here is to swap out the discovery bits for NLU. This will a) greatly speed up load time of initial products, b) simplify the code base since we'll remove a lot of logic for handling collections.

@horeaporutiu do you think you could do this, otherwise I will ask @rhagarty to take a whack at it.

Building and Deploying Failed

Readme is not clear on building and deploying. there are few missing steps. Below are some suggested changes:

1. In the install docker section we can add “If you get docker login error, try sudo docker login use sudo docker login”
2. its good to mention 2 steps to deploy: a. IBM cloud and b. kebernetes
3. github configuration steps in deploying to ibm cloud is missing.
4. Step 4 in IBM cloud deploy needs to be elaborated. Even after doing that the deployment steps fails. choosing cloudfoundry or kubernetes as deployment type also fails.

5. There are common/redundant steps in both deployment ways which can be abstracted.
6. Steps section is only for "deploying using kubernetes" . Need to have steps for IBM cloud deploy as well.

Make the code repository pattern compliant

Looking good so far team! We need to do a few other things to make this a "code pattern":

1. Create a thorough README.md
2. Create a wiki to draft IBMCode content
3. Add an architectural diagram
4. Add Deploy to IBM Cloud support
5. Create a video
6. Create a blog

See https://github.com/IBM/watson-discovery-ui as an example.

showing low star rated reviews with high positive sentiment

Along with keywords, entities and concepts, the app currently shows reviews in a "Customer Reviews" section. I don't know how these are sorted but they seem to just be generally high star reviews.

This issue is to propose adding a new section to show "interesting" results, where the review is a low star review but the sentiment is high.

provide a few sample product links

A minor UX/UI improvement would be to have a few sample URLs/products that users could click on and view results quickly. Maybe whatever is the last few entries of the database.

Job Processing in Background

https://github.com/Automattic/kue

This can be useful. We could run our functions in the backround (from scraping, uploading to cloudant to discovery) since it takes a while for them to complete when doing a request on /reviews/:reviewId

Make a frontend

User will input a link to product

Make a backend for getting reviews

Backend will use nodejs with express. Should output JSON

Explore Watson Discovery

Explore the Watson Discovery service with the data from reviews