officialpatterson / level4project

Repository for storing scripts, documents and live code for the level4 project GoodTweetBadTweet

UK-based Software engineer with domain experience in automated control systems, medical informatics, and financial services.

level4project's Issues

Create notification script

Notification script will run as a stand alone application. A notification will occur if more than 30 tweets occur in a single hour about an entity(dimension).

pseudocode:

1. for each entity in notifications
   request time distribution per hour for past 24 hours
2. if last hour is 10% than mean of the previous 23 hours notify(entity, dimension)

notify() is just a simple lookup and post.

Add/remove track button on entity page

If a user is not tracking an entity then the page should display a 'track' button.

if they are tracking that entity, the page should display an 'untracked' button

Implement pie chart showing distribution of tweets over all dimensions

use the d3 library piecharts to implement this feature

Stop unnecessary printing to terminal in services component

Implement geographical chart using location distributions

Create map showing the distributions over countries of entity tweets using google geochart.

Have entity list in appropriate datastore

Currently the live classification component retrieves its entities from a static file called entities.tsv.

This should be in a database.
Use MongoDB - as its the one used throughout GTBT - to create a collection and document attributes name and active.

Create Notification class

static methods:
notifications(tweet)
notify(entity)
send_notifications(user, entity, msg):

notifications(tweet)- callback executed from POST requests to data-service.

notifications(tweet):
1. if entity is in track List && favourites > 15:
2. notify(entity)

notify(entity):
1. get list of users that track entity
2. for user in list
3. send_notification(user, entity, “high favourite tweet”)

send_notifications(user, entity, msg):
1. HTTP post to /app/addnotification/

Extract JS files from templates.

Currently there is javascript code in the templates files. For reusability/maintainability reasons this should be extracted and placed under a js folder in the static directory.

Add Classification module to system

Large part of system.
The module must use a statistical/probabilistic classifier to classify tweets into a specified reputation dimension of the model.

Implement exact match searching

The UI currently shows a search box that has zero-functionality. change this so that when the user submits a query the client interface uses exact match searching to find the entity and redirects to the correct entity page.

stop adding undefined tweets to the classified tweets collection

Display tweets related to entity on page

Show a list of 10 tweets at the bottom of the entity page.

Create request for retrieving dimension time distributions

request for retrieving the times of each dimension.
url request:
/entity_name/time?dimension=d

Map:
for each tweet in entity(dimension)
emit (yymmdd, 1)

reduce:
sum all individual dates.

Alther activity chart to show a line for each dimension

Currently the activity chart for each entity only shows the rate of tweets coming through. change this so that it shows the rate of tweets coming through for each dimension.

Create request for retrieving location distributions

To reduce request times/response sizes, create a request type that returns the country-level location distributions of an entity.

the url for requesting should be:
/entity_name/locations

The data returned should be a son array of objects in the following format:
{Dimension: "dimension_name", locations:["US": 10, "UK": 5]}

Notifications system

The user must be notified of any events that happen regarding entities that they watch.
the events are:

• rate of tweets increase/decrease by x percent
• large change in sentiment value of tweets

The numbers for this is abritrary. Probably set low for testing purposes.

Create request for retrieving dimension distributions

To reduce request times/response sizes, create request that will return the size of each reputation dimension in a json array.

• url:
  /entity_name/dimensions/

*object format
{dimension: "dimension_name" size: 56}

Use Templating for UI

The GTBTApplication does not use templating for its views therefore there is duplicated HTML/JS in the views.
Use Django's templating system - or any other framework - to make good use of templating.

Stop the feed-enrichment component from printing to the console unnecessarily

it prints "iterating" repeatedly. stop that.

POST Request on data service for adding new tweets

Currently, the classifier posts new tweets directly to the database.

This must be altered so that the classifier inserts new tweets into the database using a post request to the data-service component

Create view for displaying tweets associated with an entity

Currently, the client application displays all the tweets belonging to an entity below the graphs.
Change this so that the tweets are instead displayed within their own view.

Implement Add Track feature

This featue requires a user to be logged in.

The first step the user must take is to see if the entity they wish to track is already in the system. If it is present in the database and is active, add the entity name to the list of the users tracks.

If the entity is not present in the system add it to the database as active.

Change feed-enrichment so that retweeted tweets are removed

many of the tweets being classified and stored are retweets, change this so that only original tweets are classified and stored.

Sentiment Analysis for Feed-Enrichment

create a new classifier that would be able to classify a tweet as either positive/negative/neutral in sentiment towards the entity.

• This may require more crowdsourcing to create the gold standard collection

Have live classification component use entity datastore

Have the program retrieve the list of active entities from the collection created in #15

At this stage it does not need to be live

Display Personalised Track List on front page.

for a logged in user, the list of entities that the user is tracking should be displayed. if they are not tracking any entities the area should say "You are not tracking entities yet"

Show barchart representing the 5 biggest countries

each bar should represent a signle country. the size of the bar is defined by how many defined tweets came from that country.

Create Architecture Diagram

The system composes of 3 distinct components:

• data-service
• feed-enrichment
• client-application

with one external component: twitter.

Create a diagram to show the relationship between each.

Incorrect login details redirects to blank page

If a user enters incorrect login details and then submits the form the user is redirected to a blank page that says "incorrect login details" . This should be redirected to the original login page with an appropriate error message given this way.

Modify feed-enrichment component to insert tweets into database using data-service

Currently, the component connects directly to the database. Modify this so the component uses the data service.

Change Entity page title to reflect entity name

Currently made up value. change it

Build profile view

This view should show the name of the user along with the email, username, and list of watched entities.

implement multi series line chart

use this chart to show the rate of tweets per hour over a 24 hour period.

Implement User system

The project should allows users to register with GTBT.
Each user should have an email address and password that would be able to identify them.
The main purpose of the user system is to give users a choice of tracking specific entities in the system, i.e. instead of being shown every entity in the system only show the ones they are interested in.
Therefore the requirements are:

• Registration/login form for users
• user attribute called tracked entities

This will obviously require a SQL database and the tables would be users(id, email, password), watched_entities(user_id, enity_name)

Extract Superclass of RequestHandler in Tornado REST Service

Currently Each handler is specified separately in each file as totally independent classes.

However this is not the case and the common features of which should be pulled out. For example, the database connection instantiation should be pulled up.

Replace Chart.js graph rendering with D3

Chart.js fails to provide means of logarithmic scaling where chart data can be wide ranging means the graph is rendered improperly.

Replace all graphs/charts with d3

Add callback to data-service for notification system

Rethink Entity List overview.

The current way of retrieving a list of entities in the system is to query for all entities in the classification collection.

This may return no entities if the entity is new to the system and/or is a rarely occurring entity in the Twittersphere

Build Edit profile view

Very similar to issue #6 - except this time the data hsould be displayed in a form apart from the watched entities. then any changes made in the form should be submitted and the database updated.