josiah-wolf-oberholtzer / discograph Goto Github PK

Because of SVG's implicit z-ordering, text elements would need to be moved out of all node groups and then translated separately (and therefore kept in sync with the nodes), based on the node motion.

Better matches Discogs nomenclature.

Either by color or by stroke thickness.

Will require storing which edges are connected to which nodes, as well as untangling primary / secondary spline edges.

Eliminate the constant existence-checking for aliases, as they are currently only stored as names.

Should initially support:

• Released on
• Collaborated with
• Remixed
• Produced
• Performed on / Featuring
• Mixed by / Compiled by

Spline-based edges, using a hidden node as their control point, should allow for multiple edges between pairs of nodes.

• The original link should be the one that is animated in D3.
  • It should receive a reference to the intermediate node.
  • It should receive isSpline=false
• The generated links should be renamed for clarity:
  • entity-key-A-role-[entity-key-B] (going to the intermediate)
  • [entity-key-A]-role-entity-key-B (coming from the intermediate)
• The isPrimary key-value is not necessary then

• On each entity encountered during a round of collection, collect that entities relations
• Instead of adding the entities referenced in the relations to the pending list, add the relations themselves.
• On the next pass, sort the pending relations by role, prioritizing membership and aliases.
• Collect the entities from the sorted relations, iterate over those to collect the next batch of entities and relations.

This conflicts with "dblclick" events, which have to be treated as a special case of "click" events.

Supporting "click" events will let us add highlighting to selected nodes and pull up meta-data panels (YT videos, release histories, bios, etc.).

Where to place Mongoengine connection?

Basically, for all non-structural relations (Member Of, Alias, Sublabel Of), the left-hand side should be the agent, and the right-hand the receiver.

L "Released on" R
L "DJ Mix (of)" R
L "Remix (of)" R
L "Performer (for)" R
L "Curated By" R
L "Written-By (Writing for)" R

Providing the initial JSON data inline will obviate the need for a second request when first accessing the site.

The JS JSON loader will need to know to look for this data on page load.

Extraction from classes:

• Artist
• Label
• Release

Other improvements necessary:

• Store genre
• Store country
• Store style (?)

Find solutions for automated testing of the front-end Javascript components.

Study the D3 source, as that project has a test suite.

It's hard to click on nodes when on cellphones.

• What is the state of Bootstap-TypeAhead / Twitter TypeAhead / other tools?
• Typeahead (or similar) needs to be integrated into the site's Javascript.
• Any dynamically-created elements need to be styled correctly.
• Caching (maybe pre-fetching?) to speed-up the search process.
• Hitting enter on the search bar should navigate to the relevant page.

On insert, set "random" to a random value between 0 and 1.

This provides the most performant technique for selecting random values from a collection.

Simply use cls.objects(random__gte=random.rand()).first().

All tests pass when testing in models/. And all tests pass when testing in graphs/. But if testing from discograph/, multiple tests fail.

Probably due to a stray mongoengine connection.

Name the files by their intended class and Discogs ID.

The incredibly massive files provided are cumbersome. If any part of an import fails, the file must be scanned up to that point. This process can takes hours and hours. By providing individual files per record, it should be easier to simply sort and search the file system, compare against the current contents of the database, and continue from there.

Our test machine is simply too slow and under-powered.

API queries should return all nodes and edges up to just beyond node/link limits. Each node and link should be associated with one or more "pages". For example, the center node should appear in all pages.

On the client-side, the entire network will be received, but only one " page" shown at a time. This will ensure that a) the visualization is always responsive and b) all 1-degree relations can be seen from the same center.

This will make visualizing non-alias, non-band relations much more palatable.

Discograph's graph search is always going to be potentially complex. This will become even more crucial before role-based searching goes live, as the number of relations increases by one to two orders of magnitude.

We should migrate to an asynchronous framework to make this as pleasant as possible.

See:

• https://python-aiopeewee.readthedocs.org/en/latest/index.html
• http://tavendo.com/blog/post/going-asynchronous-from-flask-to-twisted-klein/
• http://www.tornadoweb.org/en/stable/

Given degree=12 and max_nodes=100:

• 1012125 is missing 1094097 (bad ID in 1161804.members)
• 728867 is missing 4329435 (bad ID in 1247565)
• 1523700 is missing 1205066
• 2898681 is missing 1728462
• 733145 is missing 734498, 734496, 734494

Providing a small (~6 artists, ~3 labels, ~12 releases) XML collection will allow for testing all system functionality without relying on a) the complete Discogs data dump or b) hand-extracted XML excerpts from the same.

Example:

"Hisham Mayet

Note above the unmatched double-quote. This causes Peewee to barf.

Should be possible to navigate forward and backward through the states and have the graph update dynamically.

This will require moving some of the logic out of the node double-click callback into a standalone function which can be called to update the entire graph based on a node id.

A single link can continue to use a line.

Multi-links need to use arcs. They therefore also need to be grouped by source and target. Grouping is going to be tricky, as we can't rely on simply pre-computing their group and giving them an index. Why? Links may appear and disappear as auto-pruning happens. Unless, of course, we make it an invariant that so long as two nodes appear on screen, all of their links also appear.

These currently render as a link segment normal to the node's circumference. Render instead as some kind of arc moving away and back, with a slight separation within.

This will simplify insertion as we will no longer need to query for the existence of a competing relation. They can simply be upserted in bulk, thereby vastly speeding-up the relation extraction process.

Will improve clarity as the JS increases in size and complexity.

Some questions:

• What is the directory structure of the exploded code versus the single-file built version.
• What tools to use for concatenating (mbostock's smash?)
• Does this require that the entire project (including Python source) have a src/dist or src/build structure, with the built code deployed from source into build directory?
• What tools to use for that?
• Can JS rebuilds be automated like Flask's server-restarting?

1. Click on a node (A).
2. Click on another node (B).
3. Before A's halo has faded out, click on A.
4. Both halos fade out.

Implement /api/random/, and use that to load in a random artist, rather than using a redirect.

Caching each encountered artists' aliases, members, and groups should optimize the search process considerably.

Clicking on a node should emit a "discograph:node-selected" event. Something - the window itself? - should listen for and respond to this event.

The intent is to migrate much of the logic out of the event handlers currently attached to various SVG elements.

PostgreSQL supports full-text search and BSON fields, making it able to mimic MongoDB nested structures.

If the memory requirements are not prohibitive, PostgreSQL could replace the current MongoDB/Sqlite frankenstein.

Place it in a small box, with a single letter, centered on the midpoint of the spline.

For example, "Compiled On" would show a tiny "C".

The new version should:

• Rely on Relation, not simply Artist.
• Support all relation types available.
• Allow filtering by relation type.
• Report the type of each node returned (Artist or Label).

Highly-interconnected alias networks have O(n^2) connections.

Investigate rendering these as as either single nodes or as bounded subgraphs without internal links.

ReferenceFields are too slow, and related documents can simply be accessed via read-only properties.

This should be done on the back-end, but could also (additionally) be implemented on the front-end to speed up history pushing/popping.

Merge SQLArtist and SQLLabel into a single table. This can be extended to include Master releases.

• id, primary key
• entity_id
• entity_type (1=artist, 2=label, 3=master)
• name
• description (?)

Investigate best practices.

• http://eloquentjavascript.net/10_modules.html
• http://www.adequatelygood.com/JavaScript-Module-Pattern-In-Depth.html

These should work:

/123/
/224242/

How to inform the javascript?