josiah-wolf-oberholtzer / discograph Goto Github PK
View Code? Open in Web Editor NEWSocial Graphing for the Discogs Database
License: MIT License
Social Graphing for the Discogs Database
License: MIT License
Because of SVG's implicit z-ordering, text elements would need to be moved out of all node groups and then translated separately (and therefore kept in sync with the nodes), based on the node motion.
Better matches Discogs nomenclature.
Either by color or by stroke thickness.
Will require storing which edges are connected to which nodes, as well as untangling primary / secondary spline edges.
Eliminate the constant existence-checking for aliases, as they are currently only stored as names.
Should initially support:
Spline-based edges, using a hidden node as their control point, should allow for multiple edges between pairs of nodes.
isSpline=false
entity-key-A-role-[entity-key-B]
(going to the intermediate)[entity-key-A]-role-entity-key-B
(coming from the intermediate)isPrimary
key-value is not necessary thenThis conflicts with "dblclick" events, which have to be treated as a special case of "click" events.
Supporting "click" events will let us add highlighting to selected nodes and pull up meta-data panels (YT videos, release histories, bios, etc.).
Where to place Mongoengine connection?
Basically, for all non-structural relations (Member Of, Alias, Sublabel Of), the left-hand side should be the agent, and the right-hand the receiver.
L "Released on" R
L "DJ Mix (of)" R
L "Remix (of)" R
L "Performer (for)" R
L "Curated By" R
L "Written-By (Writing for)" R
Providing the initial JSON data inline will obviate the need for a second request when first accessing the site.
The JS JSON loader will need to know to look for this data on page load.
Extraction from classes:
Other improvements necessary:
Find solutions for automated testing of the front-end Javascript components.
Study the D3 source, as that project has a test suite.
It's hard to click on nodes when on cellphones.
On insert, set "random" to a random value between 0 and 1.
This provides the most performant technique for selecting random values from a collection.
Simply use cls.objects(random__gte=random.rand()).first()
.
All tests pass when testing in models/. And all tests pass when testing in graphs/. But if testing from discograph/, multiple tests fail.
Probably due to a stray mongoengine connection.
Name the files by their intended class and Discogs ID.
The incredibly massive files provided are cumbersome. If any part of an import fails, the file must be scanned up to that point. This process can takes hours and hours. By providing individual files per record, it should be easier to simply sort and search the file system, compare against the current contents of the database, and continue from there.
Our test machine is simply too slow and under-powered.
API queries should return all nodes and edges up to just beyond node/link limits. Each node and link should be associated with one or more "pages". For example, the center node should appear in all pages.
On the client-side, the entire network will be received, but only one " page" shown at a time. This will ensure that a) the visualization is always responsive and b) all 1-degree relations can be seen from the same center.
This will make visualizing non-alias, non-band relations much more palatable.
Discograph's graph search is always going to be potentially complex. This will become even more crucial before role-based searching goes live, as the number of relations increases by one to two orders of magnitude.
We should migrate to an asynchronous framework to make this as pleasant as possible.
See:
Given degree=12
and max_nodes=100
:
Providing a small (~6 artists, ~3 labels, ~12 releases) XML collection will allow for testing all system functionality without relying on a) the complete Discogs data dump or b) hand-extracted XML excerpts from the same.
Example:
"Hisham Mayet
Note above the unmatched double-quote. This causes Peewee to barf.
Should be possible to navigate forward and backward through the states and have the graph update dynamically.
This will require moving some of the logic out of the node double-click callback into a standalone function which can be called to update the entire graph based on a node id.
A single link can continue to use a line.
Multi-links need to use arcs. They therefore also need to be grouped by source and target. Grouping is going to be tricky, as we can't rely on simply pre-computing their group and giving them an index. Why? Links may appear and disappear as auto-pruning happens. Unless, of course, we make it an invariant that so long as two nodes appear on screen, all of their links also appear.
These currently render as a link segment normal to the node's circumference. Render instead as some kind of arc moving away and back, with a slight separation within.
This will simplify insertion as we will no longer need to query for the existence of a competing relation. They can simply be upserted in bulk, thereby vastly speeding-up the relation extraction process.
Will improve clarity as the JS increases in size and complexity.
Some questions:
Implement /api/random/, and use that to load in a random artist, rather than using a redirect.
Caching each encountered artists' aliases, members, and groups should optimize the search process considerably.
Clicking on a node should emit a "discograph:node-selected" event. Something - the window itself? - should listen for and respond to this event.
The intent is to migrate much of the logic out of the event handlers currently attached to various SVG elements.
PostgreSQL supports full-text search and BSON fields, making it able to mimic MongoDB nested structures.
If the memory requirements are not prohibitive, PostgreSQL could replace the current MongoDB/Sqlite frankenstein.
Place it in a small box, with a single letter, centered on the midpoint of the spline.
For example, "Compiled On" would show a tiny "C".
The new version should:
Highly-interconnected alias networks have O(n^2) connections.
Investigate rendering these as as either single nodes or as bounded subgraphs without internal links.
ReferenceFields are too slow, and related documents can simply be accessed via read-only properties.
This should be done on the back-end, but could also (additionally) be implemented on the front-end to speed up history pushing/popping.
Merge SQLArtist and SQLLabel into a single table. This can be extended to include Master releases.
Investigate best practices.
These should work:
/123/
/224242/
How to inform the javascript?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.