emlama / sitemapper Goto Github PK

Should start planning how these will be used so that deployment to production will go smoother.

I think much of the crawler code can be eliminated by keeping all of the queue information in the Meteor database and writing cache information to said location. This will require a lot of rewiring of simplecrawler but will also improve the app.

Should delegate to a PhantomJS script that takes care of this.

As this is becoming more advanced its becoming time to make this multiuser. Should be as simple as adding authentication hooks and filtering queries based on the current user.

Default page should be a homepage.

Eventually, this will be used to create controls for mapping a websites data back to a content model.

This hasn't been implemented yet but certainly will be helpful for making sure everything is kosher.

This can also be incorporated into the testing framework. See Simplecrawler's tests for how that was done.

Use Mocha and Chai

Right now we are only saving information for links that respond. All of this information is available but not being sent back. Tons of use cases for using this data.

See Laika