roundware / roundware-server

Location-based contributory audio AR platform for art, education and documentary

ROUNDWARE SERVER

Overview

Roundware is a client-server system. The server runs using Apache HTTP Server and mod_wsgi on Ubuntu Linux 20.04 LTS Focal Fossa and clients are available for iOS and HTML5 browsers (mobile and desktop). There is an old Android client as well, but it is not supported currently. This document outlines the steps required to setup a basic Roundware server that can be accessed through any of these clients.

For more information about Roundware functionalities and projects that use the platform, please check out: roundware.org

Installation

Roundware includes an install.sh to handle installation of the software and its dependencies. The majority of the process is automated. Further configuration is required for a production system, application specific details are below.

user@server:~ $ git clone https://github.com/roundware/roundware-server.git
user@server:~ $ cd roundware-server
user@server:~/roundware-server $ sudo ./install.sh

The installation process creates a roundware user as project owner. su to that user to load the required virtual environment:

sudo su - roundware

Production Settings

The production Roundware/Django settings file used by WSGI Apache2 is stored outside of the source code directory in the file /var/www/roundware/settings/roundware_production.py. All settings in /var/www/roundware/source/roundware/settings/common.py can be overridden there. Do not modify any file within the /var/www/roundware/source directory unless you intend to maintain your own fork of Roundware Server.

Vagrant

A VagrantFile is included for local development and testing with Vagrant and VirtualBox. Usage:

user@local-machine:~ $ git clone https://github.com/roundware/roundware-server.git
user@local-machine:~ $ cd roundware-server
user@local-machine:~/roundware-server $ vagrant up
user@local-machine:~/roundware-server $ vagrant ssh
(roundware)vagrant@roundware-server:~$ cd roundware-server/scripts
(roundware)vagrant@roundware-server:~$ ./runserver.sh

Notes:

• The installation process uses the default vagrant user as project owner.
• The install script relies on the Vagrant default file share of host:~/roundware-server to vm:/vagrant for installation and development.
• There are multiple port forwards from the host to the VM:
  • VM:80->host:8080 for Apache hosting the demo "live" environment available at http://127.0.0.1:8080/
  • VM:8888->host:8888 for the manage.py runserver development webserver available at http://127.0.0.1:8888/
• Edit the development environment code on your local machine, then refresh to see the changes reflected in the virtual machine.

Code Upgrades

deploy.sh exists to update the Apache WSGI production code. Note: It fully replaces any code in /var/www/roundware/static/. So, for example, customizations to roundware/rw/settings/common.py must be done to the code base the deploy.sh script is run from. To perform a code upgrade:

user@server:~/roundware-server $ git pull
user@server:~/roundware-server $ sudo ./deploy.sh

Additional Details

Postgres

Roundware uses Postgresql and requires a dedicated database with a dedicated user.

You may change the database name and account info to fit your needs, but if you do, be sure to change the Roundware config (/etc/roundware/rw) and the Django settings (roundware-server/roundware/settings.py) to reflect your changes.

Apache

Apache must be configured to use mod_wsgi to host Roundware. A default config is included at roundware-server/files/etc-apache2-sites-available-roundware. If manually installing on a clean 14.04 machine, this file can simply be copied to the Apache configuration directory, though there are several changes that should be made to reflect your environment.

Configure Django, the Python framework Roundware uses.

(roundware)user@machine:~/roundware-server/roundware$ ./manage.py syncdb

Note - this script may prompt for the username and password for your database. If you changed these values from the defaults when creating the Roundware DB, the changes must be reflected here. You'll be asked if you want to create a superuser like so:

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):

Answer yes, and provide the default values for username ('round') and password ('round'). Note that any subsequent changes must be reflected in settings.py. The email address can be anything, and is not currently used.

If you have a fixture data file, you can populate your database with this data by running the Django command:

(roundware)user@machine:~/roundware-server/roundware$ ./manage.py loaddata <path_to_fixture>

The base Roundware install package includes a standard DB fixture file to populate a default database with the basic data you will need to test your installation. That can be installed similarly, if you so choose:

(roundware)user@machine:~/roundware-server/roundware$ ./manage.py loaddata rw/fixtures/base_rw.json

Make some edits to the Django settings file, roundware/settings.py:

ANONYMOUS_USER_ID = 0 // change this to the proper id for AnonymousUser in database for Guardian
# settings for notifications module - email account from which notifications will be sent
EMAIL_HOST = 'smtp.example.com'
EMAIL_HOST_USER = '[email protected]'
EMAIL_HOST_PASSWORD = 'password'
EMAIL_PORT = 587
EMAIL_USE_TLS = True

Open a browser and browse to http://example.com/admin, and verify you see the Django admin page.

You can check the Apache log files for debugging information:

• /var/log/apache2/access.log
• /var/log/apache2/error.log

as well as the Roundware log:

• /var/log/roundware

Development

Roundware uses separate pip requirements files and Django settings files for development. If you are running Roundware as a development server, you should run:

user@machine:~/roundware-server $ pip install -r requirements/dev.txt

to get the additional requirements for development and testing. You should also edit your ~/.bashrc file (or other method of setting persistent environment variables) to add:

export DJANGO_SETTINGS_MODULE=roundware.settings.dev

Note: You can use a local_settings.py (not in version control) inside of the roundware/settings/ directory.

Testing

To run unit and functional tests and see test coverage, you will need the development requirements (see above). To run tests and get a report of test coverage:

(roundware)user@machine:~/roundware-server$ ./test.sh

Upgrading notes

During significant server upgrades the Python VirtualEnv may need to be rebuilt. Enter the following as root to recreate/rebuild the VirtualEnv. $USER is roundware or vagrant depending on your installation type (production or vagrant.)

rm -rf /var/www/roundware/include/ /var/www/roundware/lib/ \
/var/www/roundware/local/ /var/www/roundware/bin/
su - $USER -c "virtualenv --system-site-packages /var/www/roundware"
./deploy.sh

roundware-server's Issues

investigate data analysis tools

Ideally, would like to implement flexible user-friendly way of slicing and dicing the data in the RW database to glean useful information. This will likely be some combination of flexible filtering along with graphing of the resulting data.

I will use this issue to store research notes on what's available and potentially useful:

• django-filter https://github.com/alex/django-filter
• django-tables2 https://github.com/bradleyayers/django-tables2
• chartit http://chartit.shutupandship.com/
• tastybackbonepie https://github.com/sspross/tastybackbonepie

fix repeating asset issue in playback

The intended behavior of Roundware geo-listen playback is as follows:

• a user enters the range of a group of assets while listening
• RW calculates which assets are available and selects randomly which one plays back first according to the audiotrack settings
• all other available assets are played back in random order regardless of any modify_stream events that occur within the range of said assets
• once all available assets are played, either no more assets play or the process is repeated (depending on project.repeat_mode setting)
• if the listener moves outside of the range of voices and then moves back within range, the assets are re-triggered and the process starts again.
• if modify_stream events happen within the range of the assets, without one occurring outside the range, no re-triggering occurs
• this results in the only way for an asset to be played more than once is by the listener entering, leaving and entering again its range OR if project.repeat_mode is set to 'continuous'

This is the way it has worked in the past, but it is no longer behaving this way. Currently, if a modify_stream call is received when within the active range of an asset that has already been played, it will be triggered to play again despite the listener not having left the active range and returned.

As mentioned, this is related to the project.repeat_mode setting which determines if when all available assets are played back whether or not the whole "bucket" of available recordings should be reset and played again (continuous) or whether no more assets should play (stop).

Gconf and dbus errors in logs

Make this stop.

(gst-plugin-scanner:17560): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Unable to autolaunch a dbus-daemon without a $DISPLAY for X11
GConf Error: No D-BUS daemon running

use zones for polygon speaker ranges rather than circle

Assign speakers a "zone" rather than a lat/lon/radius such that either circles or polygons could be used.

• increases flexibility tremendously
• build web interface with google maps to draw polygon for creating these zones

This obviously hinges on the ZONE idea:

• https://github.com/hburgund/roundware-server/issues/98

"AttributeError: 'module' object has no attribute" when tags are not valid

When calling: operation=modify_stream&session_id=2891&tags=3,5,8 A D-Bus Exception can occur if client tag id numbers are out of date.

2014-06-17 18:40:49,942 connection.py:234 ERROR Exception in handler for D-Bus signal:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/dbus/connection.py", line 230, in maybe_handle_message
    self._handler(*args, **kwargs)
  File "/home/vagrant/roundware-server/roundwared/rounddbus.py", line 41, in handler
    stream.modify_stream(request)
  File "/home/vagrant/roundware-server/roundwared/stream.py", line 128, in modify_stream
    self.refresh_recordings()
  File "/home/vagrant/roundware-server/roundwared/stream.py", line 151, in refresh_recordings
self.recordingCollection.nearby_unplayed_recordings = getattr(asset_sorters, tag.filter)(assets=self.recordingCollection.all_recordings, request=self.request)
AttributeError: 'module' object has no attribute '--'

Issue may be low priority if difficult for client devices to cause.

in continuous mode, asset filters are not applied upon 'refresh'

When I put a project into 'continuous' repeat mode and then make certain filter selections to narrow down what I hear, I am getting incorrect recordings in the stream. It seems like when the repeat is triggered, the current filters are not applied. In other words, the filters seem to be applied initially, but when all available recordings are exhausted and the 'refresh' occurs, the current filters are not applied.

That's my best guess, but it's hard to tell. I looked at the logs and I see statements like this:

2012-05-21 21:39:03,223 recording_collection.py:71 DEBUG GET_RECORDING UPDATE:
all_recordings count: 1, far_recordings count: 0, nearby_played_recordings count: 0, nearby_unplayed_recordings count: 1

after I apply filters that should return 1 recording, and then when I look at the next similar statement to happen, based on the continuous mode refresh, it says:

2012-05-21 21:39:03,223 recording_collection.py:71 DEBUG GET_RECORDING UPDATE:
all_recordings count: 3, far_recordings count: 0, nearby_played_recordings count: 0, nearby_unplayed_recordings count: 3

ie. returns all available recordings.

thankfully, this is a bit of a fringe case currently since we are not actively using continuous mode, but I'm hoping it's fairly easy to remedy.

add project lat/lon to get_config response

add to 'project' section for use in things like centering Voice Map, Listening Map

new API call: get_events

NEW IDEA: maybe make this get_events and then it can take session_id as parameter to behave as below or no parameters to return the whole event table or eventtype as parameter to return only certain types of events for graphing etc. Or date range parameters? Would need to be able to segment by project though, and that is gotten via the session_id, so ideally project_id is another parameter option.

+++++++++++
takes session_id as parameter and returns all event data (ordered by event.server_time). This will be very useful for creating maps of the paths users walked on as well as other session-based data display and analysis.

Response will look something like this:

{
"events": [
    {
        "server_time": 2012-01-29 19:10:00,
        "client_time": 2012-01-29 19:10:00,
        "session_id": 1234,
        "event_type": "modify_stream",
        "latitude": 51.975113,
        "longitude": 5.663555,
        "data": "some data in a string",
        "tags": [
            {
                "localized_value": "young",
                "tag_category_name": "age",
                "tag_id": 1
            },
            {
                "localized_value": "What did you eat today?",
                "tag_category_name": "question",
                "tag_id": 6
            }
        ]
    },
    {
        "server_time": 2012-01-29 19:10:00,
        "client_time": 2012-01-29 19:10:00,
        "session_id": 1234,
        "event_type": "request_stream",
        "latitude": 51.975113,
        "longitude": 5.663555,
        "data": "some data in a string",
        "tags": []
    }
],
"number_of_events": {
    "modify_stream": 36,
    "heartbeat": 52,
    "start_session": 1,
    "start_upload": 3
}
}

replace dbus with more modern inter-process messaging system

RabbitMQ: http://www.rabbitmq.com/
ActiveMQ: http://activemq.apache.org/
ZeroMQ: http://zeromq.org/

We need a solution that is a bit more robust, modern and less touchy than dbus has been. There are likely other options, but these three seem to all be well-used and supported, so they are a good place to start.

Make sample data installation optional

The current installation process includes sample/testing data (database and audio.) It should be optional for production systems in the future.

don't require tags for asset to be submitted

The behavior I am seeing now is that if an add_asset_to_envelope request is made that does not contain any tags, the request fails and the asset is not added to the database. I believe the file is uploaded, but the asset record is not created.

assets should not trigger twice unless listener moves outside of range and back into range

This used to work properly on RW V1, but for some reason, it is no longer working as intended. Currently, if a modify_stream event happens within an asset range after that asset has played, it will trigger it to play again.

Instead, the "bucket" of available assets should only reset if the listener moves outside of the range and then back into range.

This bug is causing major issues for all geo-listen projects.

add asset via admin erroring out before completion

When adding an asset through the admin, the file is uploaded fine and the new record in the db is created fine. But then the further processing of audio length happens and it seems as though the database is cached or something by this process so that it doesn’t see the more recent asset and therefore errors out.

2014-05-11 09:24:19,019 __init__.py:13 INFO caught add or edit asset, created: True
2014-05-11 09:24:19,020 __init__.py:20 INFO asset 51
2014-05-11 09:24:19,021 __init__.py:31 INFO []
2014-05-11 09:24:19,075 views.py:94 ERROR u'Invalid Asset ID Provided. No Asset exists with ID 51'Traceback (most recent call last):
  File "/home/ubuntu/roundware-server/roundware/rw/views.py", line 92, in catch_errors
    return function(request)
  File "/usr/local/lib/python2.7/dist-packages/roundwared/server.py", line 507, in add_asset_to_envelope
    raise roundexception.RoundException("Invalid Asset ID Provided. No Asset exists with ID %s" % asset_id)
RoundException: u'Invalid Asset ID Provided. No Asset exists with ID 51'

2014-05-11 09:24:19,078 signals.py:70 INFO error message is post_save: u'Invalid Asset ID Provided. No Asset exists with ID 51'
2014-05-11 09:24:19,090 __init__.py:13 INFO caught add or edit logentry, created: True

RoundException: u'Invalid Asset ID Provided. No Asset exists with ID 51'

Our hypothesis is that something in the django upgrade from 1.3 to 1.6 has caused it to behave differently with regards to caching and/or managing db connections or something like that. There are a number of pieces of evidence online that suggest it is a reasonable guess.

I found this stack overflow post that seems to be a very similar issue:

http://stackoverflow.com/a/4739960/2073262

We tried to implement the suggested solution, but then get a different error:

TransactionManagementError: This is forbidden when an 'atomic' block is active.

Some more research suggests that indeed there are some significant changes in django 1.6 regarding transactions, so I’m pretty convinced that this is causing the problem. For some reason, since adding assets from the admin uses both django transactions and pure python/roundwared transactions, there is some inconsistency that causes the problem, whereas when an asset is added via the clients, it is done without django somehow (or at least without an admin connection), so it is able to complete successfully.

Here’s a good post as to the differences in 1.6 wrt transaction management:

http://www.realpython.com/blog/python/transaction-management-with-django-1-6

And another potentially useful link to someone experiencing a similar issue in a different project:

linuxlewis/djorm-ext-pgfulltext#10

create 'asset-edit-map' custom view

Basically this is a visual, mapped version of the Asset list view editor of the Admin.

• voice map that lays out all voices for a particular project (would be awesome to be able to select which project and have the map refresh)
• each pin can be listened to and moved, rearranged
• eventually listening radius is adjustable as well once we have that feature, but for now, standard radius is displayed for each
• basic asset info in popup with audio player
• Save button saves all at once (or saving done with AJAX calls)

We already have a listen-map created which creates the display of assets on a map with all the radius and info window stuff. We just need to add to it the ability to move each asset and save the lat/lon for them. listen-map.html and associated files are in the roundware-web-base repo on bitbucket.

This will be amazingly useful for sculpting the audio experience holistically rather than one asset at a time.

add waveform display to asset admin with segment selection capability

• display audio assets using Peaks.js: http://waveform.prototyping.bbc.co.uk/
• on Github: https://github.com/bbcrd/peaks.js
• create segment within audio file bounded by asset.start_time and asset.end_time
• allow the moving of the start and end of the segment selectors to change asset.start_time and asset.end_time

request_stream execution duration issues

I've been getting more and more frustrated by the issues I've been seeing on my server (rwdev.dyndns.org) with it taking forever to generate a stream; by 'forever' I mean around 25 seconds instead of what it used to be which was ~6s. Not only that, but as soon as one stream is going on my server, a subsequent request_stream almost always yields an ISE. The server is definitely capable of running more than 1 stream at a time and has done so in the past as a general rule.

So I did more testing and tried things on a laptop I have at home rather than my long-standing dev instance on AWS. The results were encouraging which is that streams took ~5-6s to generate consistently and I was able to get ~20 streams going simultaneously without any issues or increase in generation time. I believe the code is the same on both machines, so my gut is telling me there isn't a RW code issue.

Of course there are significant hardware differences between the AWS instance and my laptop, but given the change in performance of the AWS one, I'm inclined to believe there is something else going on.

So I have some questions:

• could the size of the database be causing this issue? my dev server has a large database and numerous different projects as it has been around for a long time.
• could django be getting bogged down by a larger database or some other accumulation that might be present on my older server and not on the fresher install?
• can you think of anything else that would cause this discrepancy? logfile length, number of audio files stored, etc?

Let's keep this ticket going as a place to discuss theories and possible solutions.

global listening not working as intended

UPDATE

Required fixes

• new stream should always be live stream and never be out of range
• all speakers should be mixed together at speaker.maxvolume regardless of listener location
• filtering by tags should reset playlist so that assets can be re-played if desired (but still obey BANNED_TIMEOUT_LIMIT)

---

Seems that you have to sort of trick the system into global listening right now. Speaker volume is still calculated from distance to speaker center point somehow even when lat/lon values for listener are not sent to the server in request_stream. this makes no sense...must investigate.

Why is speaker volume ever attenuated in global_listening?

Ideally, with global listening turned on (ie. project.geolisten=false), the project only needs one speaker and the distance values make no difference, but this is not what I am seeing.

new API call: get_sessions

takes optional project_id as parameter and returns session data for all sessions related to that project (or for all projects if no project_id is specified).

Response may look like:

{
"sessions": [
    {
        "session_id": 17,
        "device_id": "712200b0-e956-41a7-9515-0980fd19239e",
        "start_time": "2011-12-11 09:14:05",
        "stop_time": "2011-12-11 09:18:05",
        "project_id": 4,
        "language_id": 1,
        "client_type": "iPad",
        "client_system": "iPhone OS-7.0.4",
        "demo_stream_enabled": false
    },
    {
        "session_id": 18,
        "device_id": "712200b0-e956-41a7-9515-0980fd19239e",
        "start_time": "2011-12-11 09:14:05",
        "stop_time": "2011-12-11 09:18:05",
        "project_id": 4,
        "language_id": 1,
        "client_type": "iPad",
        "client_system": "iPhone OS-7.0.4",
        "demo_stream_enabled": false
    }
],
"number_of_sessions": {
    "Test Project": 36,
    "Exploratorium": 52,
    "Ghost Crowd": 17,
    "SFMS": 3
}
}

fix 'stop' playback mode with repeated asset playback

current behavior is that if a listener is within range of a specific asset and the project is set to 'stop' mode, the asset will get played once and then stop, but if a new GPS fix is obtained that is still within the range of the asset, that will trigger the asset to be played back again. This results in assets that are isolated being played numerous times in a row when they should only be heard once.

The behavior in RWV1 is what should still happen which is that any number of adjacent GPS fixes within the range of an asset will NOT re-trigger that asset to play. But if one GPS fix is outside the range of the asset, then things are reset and the next one within range of said asset will again trigger it to play.

build code channel functionality

allow tags of the tag category "channel" to contain a field tag.filter which causes a block of filtering code to run on the asset list. This allows for channels of audio to be created algorithmically rather than manually. Things like:

• Recent Recordings channel
• Nearby Recordings channel

and many others can be created this way.

build unit testing mechanism and basic unit tests

Here are some ideas for what specific functions need to be tested. These are not all required initially by any means, just a braindump of thoughts.

• check project repeat mode - stop vs. continuous
• elapsed time from request_stream request sent to stream url returned
• modify_stream tag filters functioning properly
• out of range sensing and stream serving
• demo-stream sensing and stream serving
• location asset filtering
• code-channels (not ready yet, but when in place, we'll want to check)
• speaker ranges
• asset ordering - random, by_like, by_weight
• full set of get_available_assets request tests with different parameters (mediatype, envelope_id, asset_id, lat/lon/radius, tagids)

add asset.loc_description to get_available_assets response

among other things, this would allow us to include any caption information available for a photo and stored in the description so that it could be displayed as an overlay.

this field clearly needs localization as the contained strings will often be displayed to the users.

add asset.start_time and asset.end_time values for each audio asset

• initially start_time default set to 0 and end_time set to audiolength
• need to modify various audiotrack calculations based on these values; essentially make sure the segment is used as the full audio rather than the entire audio file
• values are blank for non-audio assets, so not required fields
• should asset.audiolength be active segment length or full length of audio file as is currently? (gut says leave as is since active segment length is a trivial calculation)
• include data migration to populate start_time and end_time for all audio assets with 0 and audiolength

Server Steps

• add new optional fields asset.start_time and asset.end_time
• update admin to handle new fields
• update all api/2/assets/ requests accordingly
• update documentation
• create data migration/defaults for audio assets
• ensure when adding new asset, fields are calculated and populated properly

filter tags based on selected project in admin

when there get to be a lot of tags in a RW install, it would be very nice to have the available tags filtered by the selected project in the asset admin.

tags are connected to a project as such:
tag -> tag category -> master ui -> project

not sure if it is even possible to filter the tag multi-select widgets, but if it is, this would be useful in numerous places. the search box on the multi-select widget is very useful, but unfortunately, it does not search on related project, but only the text in the tags themselves

uptime tests for production servers

Implement a regular test on a running RW server to ensure that it is still operational. Traditionally, we have run a script every 10 minutes testing to see if request_stream returns a stream. This is a good test of full server functionality as I've never had a situation where that works, but something else is wacky.

We need the test to send a request_stream call, but to not make event entries that would pollute the real data. Also, as soon as the stream is created, it should be killed to not cause undue server strain. If it fails, send an email to some set of users.

If this could be integrated into the Notifications django module, that would be ideal.

flexible asset ranges

Currently, each asset is forced to be within a circular range defined by a center lat/lon and a radius. Furthermore, each asset in a particular project has to have the same radius value. So it is not very flexible at all.

Here are two proposed methods for increasing flexibility:

Asset-based recording radius

• new field asset.radius added which defines the radius for the specific asset
• if no radius provided, project global radius used instead

Zone-based asset ranges

• Once the Zone functionality is implemented, assets can be assigned to a zone or zones
• Zones can be circles or polygons, vastly increasing flexibility

Clearly the latter is the better solution, but is also clearly more complicated and depends on implementing various other requirements like converting to more GIS-friendly systems.

Either method will require updates to the Asset admin screen to facilitate setting the radius or zone on a map.

Here are some related Issues:

• #70
• #98

---

At some point it would be interesting to allow each participant the ability to adjust their own radius (sensitivity) by sending the value in request_stream url (userradius=10 for 10meter radius) so that for some projects we give the user that ability to hear things further away.

fix guardian privileges for project model itself

when I assign a user to have access only to projectA, it works great for assets related to projectA, but the user is able to edit project model info about projects other than projectA, which is clearly a problem.
I'm hoping this is just a simple guardian setup issue.

project export/import functionality

I need to be able to export RW projects from one server/instance and import them into another easily via the Django Admin. This can likely be done with a 3rd party app, through admin actions or some other method and I am open to recommendations on the best approach. It would be nice to be able to export various pieces of data to csv as well for external analysis, but that's not critical at this point. I assume for import/export functionality, using sql, json or a fixture would make the most sense.

There are two critical aspects to be considered:

• there are tons of relationships to the 'project' model and all related data needs to be included, some of which isn't picked up by the standard Django admin related items query (for deletion purposes) for some reason.
• all relationships need to be preserved when data is imported into another Roundware instance, so ids will need to be adjusted on import to fit the new instance without repeating.

Some preliminary research suggests that this app might do exactly what I want, so this may simply require integration and a bit of configuration of this app:

https://github.com/bmihelac/django-import-export

Data structure that needs to be included for complete project export

All data from project model and foreign keys connecting to the following:

• asset
  • asset_loc_description
    • localizedstring
  • asset_tags
• audiotrack
• session
  • event
  • envelope
    • envelope_assets
  • listeninghistoryitem
  • vote
• speaker
• masterui
  • ui_mapping
    • tag
      • tag_loc_description
        • localizedstring
      • tag_loc_message
        • localizedstring
      • tag_relationships
  • tagcategory
  • masterui_header_text_loc
    • localizedstring
• project_demo_stream_message_loc
  • localizedstring
• project_legal_agreement_loc
  • localizedstring
• project_out_of_range_loc
  • localizedstring
• project_sharing_message_loc
  • localizedstring

Some useful resources:

• https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.ForeignKey.on_delete
• https://docs.djangoproject.com/en/dev/topics/serialization/
• https://docs.djangoproject.com/en/dev/topics/serialization/#deserializing-data

build tag_data object to store additional tag-related data

Right now, I am manually creating a JSON object in the tag.data field that is returned as part of get_tags and contains a number of project-specific useful pieces of data like:

• html_class
• latitude/longitude (if site)
• tile_image
• keywords
• etc etc

There should be a separate table to store this stuff and the get_tags should read from that table. We could add it as inlines at the bottom of the tag edit screen? Each tag will have a different combination of these additional data fields; some will be used more frequently than others. It would be best to have a drop down to select the commonly used ones, but we need to have the ability to add custom ones easily as well. Not sure how best to handle that flexibility.

Remove old fixture JSON files from repo

There are a number of fixture JSON files in repository comprising the majority of repo data. The files are stored in /roundware/rw/fixtures and require ~50MB of data. It makes sense to have some example data, but not this much.

I'd go as far as to purge the files from the repository's history using the procedure outlined here: https://help.github.com/articles/remove-sensitive-data

build charting mechanism and dashboard

Build flexible, extensible data analysis framework for charts, maps etc. We have a good start using Chartit and a flexible bootstrap dashboard interface. Chartit is good, but is a bit stale (last update 2 years ago); Graphos is much more recent and handles more charting backends and data sources, so ultimately, it is likely the better choice.

• django-chartit: https://github.com/pgollakota/django-chartit
• django-graphos: http://agiliq.com/demo/graphos/

It would be amazing to add the listening map, voice map, session map and other useful tools to this same dashboard as new 'tabs'. Seems to me this should be possible without too much effort, but I don't know exactly how to bring it all together.

install Sphinx, create documentation and link to Read the Docs

Sphinx - http://sphinx-doc.org/
Read The Docs - https://read-the-docs.readthedocs.org/en/latest/getting_started.html

ZONE concept design and implementation

Introduce ability to create 'zones' that are geographic areas defined as circles or polygons and then assign certain other objects to these zones. Things like recordings, speakers, questions, and likely others as well.

Some objects could be simply assigned a zone_id rather than a lat/lon/radius etc

Use GeoJSON to define zones?

• http://www.geojson.org/geojson-spec.html
• http://pypi.python.org/pypi/geojson/1.0.1/

also Shapely

• http://pypi.python.org/pypi/Shapely

Google Maps conversion

• http://geojason.info/2010/geojson-to-google-maps-utility/

Could the old branches be deleted?

This project contains many older out-of-date branches. Could they be removed?

allow for different zip files per platform

ie. one zip file for iPhone, one for Android phone, one for iPad etc etc. I could simply hand-code a JSON object as the response in the files_url field like so:

{
"iphone" : "http://halseyburgund.com/iphone.zip",
"ipad" : "http://halseyburgund.com/ipad.zip",
"android" : "http://halseyburgund.com/android.zip"
}

The app would need to parse the JSON and download the correct version as well.

add button in Admin for the Master UI Tag organization form

ie this page:

http://localhost/rw/masteruis/setup_tag_ui/

And I think the button to access that page should be either on the UI Mappings page next to Add UI Mapping or potentially on the main RW admin screen next to the Add and Change icons for the UI Mappings row?

What would be best is a quick hint on how I can do this myself so I can learn. I'm guessing it's pretty easy and I know html/css/js fairly well. I am planning to just look at how you did it for the Batch Tags form, but any hints are welcome.

Implement Travis CI for tests

After the unit tests are fixed they should be run automatically by Travis CI: http://docs.travis-ci.com/user/languages/python/

Potential implementation details: http://datadesk.latimes.com/posts/2012/06/test-your-django-app-with-travisci/

incorrect recordings played in stream

streams are often getting incorrect recordings played, typically right after a request_stream or a modify_stream. It seems that the incorrect recording that is played is often the same recording for some reason.

The logs indicate that everything is behaving properly. In other words, the incorrect recording is not "officially" playing per the logs, but nonetheless, it is consistently getting into the stream for some reason. This is extremely disruptive and confusing for geo-listen projects that have many modify_stream calls that cause this to happen.

Add to Trac?

Could you add me to your trac? Maybe I can fix some bugs.

add parameter to get_config to preclude it from creating a new session

This is necessary so that calls directly from webviews and various web maps etc. don't skew statistics by creating additional session_ids for each session.

Default behavior should be what is is now, but if the parameter:

new_session=false

is included, then just return the data, but don't create a new session in the db or include a session_id in the response.

Another approach would be to include a session_id as an optional parameter and if it is sent, no new session is created. In some ways this seems more sensible, but in other ways, it just seems unnecessarily complicated.

make localized strings editable directly from other related objects

In other words, when I am editing a tag, for example, I should be able to change the text of the localized_message without going to the localized string table and editing it there.

It works great to add a new localized string from a different object, but if you just want to edit the text, I don't think this is possible.

Perhaps double clicking the selected entry in the selection box could open up the same popup window for adding a localized string, but for editing instead?

Fix unit tests, 20 failures, 3 errors.

The existing unit tests are not passing.
Ran 83 tests in 2.843s
FAILED (failures=20, errors=3)

From test.sh script:

roundware/manage.py test --settings=roundware.settings.testing roundware.rw roundwared --pattern="test_*.py"

Master UI organization page won't save without tag selected as default

Seems to be consistent that I cannot save a master ui using the new view without at least one of the tags selected as a default. It just hangs and eventually gives an internal error.
I can get around this by selecting one tag as default when saving and then un-defaulting this in the UI Mappings admin view.
For most Speak situations, I don't want any tags in a master UI to be the default, so this is a common situation that needs to work properly without the workaround.
Possibly related to issue #71

implement location-based tags

Essentially, tags should have an optional geographic range associated with them which defines the range within which they should be made available to users (for now, defined as center lat/lon plus radius). This allows for handy things like asking questions specific to certain locations only when in said locations. If a tag has no center/radius, the it is assumed the tag is "global".

Ideally, this could be done in tandem with the introduction of the Zone concept. Tags will be able to be assigned to one or more Zones (or none for global availability) so every time a client asks for the tags, they will be filtered not only by the relationships, but also by the Zones.

This was done and working in RWV1 in a rudimentary way per this ticket in the old Trac system which has some useful background info:

http://www.roundware.org/trac/ticket/12

But I don't think any of this code was ported to RWV2 for some reason, though I'm not 100% sure. Even if it was ported, only some of it will be useful as I would like to expand this functionality to not only questions, but to any tag.

roundware/rw/templates/rw/chart_template.html contains hardcoded URLs

The chart_template.html file contains hardcoded links to halseyburgund.com.

https://github.com/hburgund/roundware-server/blob/develop/roundware/rw/templates/rw/chart_template.html#L339

Line 339

<iframe id="listen-map-frame" class="tab-frame" src="http://halseyburgund.com/dev/rw/dashboard/listen-map-dash.html?iwloc" frameborder="0" scrolling="no"></iframe>

Line 351

<iframe id="session-map-frame" class="tab-frame" src="http://halseyburgund.com/dev/rw/share/session-map.html?session_id=13754" frameborder="0" scrolling="no"></iframe>

Master UI organization page doesn't save ordering of UI tags

I feel like this must have worked at one point since we both did a bunch of testing, but it is not working currently so it needs to be revisited regardless.
Possibly related to #67

force certain streams to be persistent

setting for stream so that it plays forever without heartbeats or listeners. useful for demo_stream.
perhaps just add a parameter to request_stream persistent=true? that causes the server to not require any continuous checks to keep it alive.
need to figure out method of telling it to stop eventually. maybe there is a kill_stream API call that takes a session_id and forces a stream kill and clean-up?
Streams can, of course, be manually killed in the icecast admin.

develop get_session API call

takes session_id as parameter and returns all info about that particular session_id.

• events
• assets/envelopes
• votes
• listening history

Each of the above should be in a top-level node of the JSON return object.

This will be useful for generating session maps and other data displays for particular sessions.

updates to tastypie API

Add additional fields for display (foreign keys mainly)

• asset.project_id
• asset.language_id
• asset.session_id

integrate geodjango and upgrade db accordingly

• convert database to handle geo related fields better ie Point, Region etc
• update calculations to take advantage of built-in geo operations such as "in-range" etc
• this relates to the "zones" idea discussed a while back and can add major flexibility to speakers (ie different shapes, polygons) as well as assets