clojusc / dragon Goto Github PK

This should mean we can also drop the lein library dependency.

Dependency: finish updating this fork in clojusc: https://github.com/clojusc/cfg-edn-prop-env

This component responds to the events published by the watcher component. It will dispatch specific functions upon published event type/s, but with controls/limits appropriate for computationally expensive or time-consuming handlers. This is delay/batching mechanism is intended to prevent handler execution floods, keep rapid-fire changes from bogging down the system.

Development tasks:

• Add component placeholder to the components ns
• Add atom data structure to component record
  • Keys will represent type of action that will take place
  • Types of actions will have a one-to-one correspondence with actual watcher-induced calls (e.g., regenerate content, regenerate CSS, restart system a component, restart the system, etc.)
  • Values will be incremented numbers representing how many times a corresponding trigger event for they key's action has been requested since the last handler call
  • A key's value (an action's request count) in the data structure will be reset out once the action's handler executes
• Add a channel for each type of supported action (i.e., each key)
• Define timeouts (will be used as a countdown) for each channel (appropriate for the corresponding type of action)

The FRMX blog has created several nifty functions that would be generally useful; @bobbermac, would you be willing to contribute these?

Note that clozhang/blog#8 is blocked on this.

Follow-up work from issue #60:

• add more supporting functions:
  • add support for getting ranges of posts (e.g., top 20 for the front page)
  • add support for storing and retrieving textual stats
  • add support for storing and retrieving category and tags stats
  • update the file processing steps to finish with all-post functions that store these stats
• iterate through all posts in db and generate pages for each post
  • create new function for efficiently iterating through all the posts
  • create new general redis pipeline function for submitting multiple queries at once
  • update the generator call to efficiently iterate through db records
  • create actual post page
• generate archives, authors, category, and tags pages
• iterate through top N most recent posts to generate front page
• generate static content pages

Once these are done, we'll be able to close #31

Tasks:

• devise an appropriate mechanism for a global registry - going to use an in-memory db (see #13)
  • possibly a new :registry key on the event component with an associated data structure?
  • define an appropriate schema for the subscriber registry
  • needs to take into account topic(s), tag(s), and subscriber functions
  • identify the operations (functions) necessary to be called upon the registry
  • define the API and create the functions in the appropriate place
• migrate the current, draft subscriber functionality to the new subscriber registry + API
• utilize the newly-created API (above) for adding subscribers to the registry (this should be an action that is take upon event system initialization)
• testing: ensure that upon addition, subscription functions are ready to use/be applied

Depends upon #13 - "Add db component"

The biggest offender here is dragon.config.core, where many functions in that namespace rightly belong in the config component namespace where it makes sense to have functions with a system argument.

In order to best support a flexible, extensible text processing framework, Dragon should support a user's or other application's ability to respond to published Dragon events. Event processing should also be a first-class citizen, with Dragon using it natively for all of its own internal processing.

Tasks:

• Examine the existing general workflow for processing posts in Dragon
• Identify all the stages at which points event publication might be useful. E.g., read file, write file; before-read, after-read, before-write, after-write
• Convert Dragon parsing process to async
  • use a core.async pub-sub
  • devise a message structure that Dragon can use externally but which could be used by other apps (that use Dragon) too
  • keep in mind that plugins would likely use this as well
• Identify all areas of Dragon that can and should be converted to using this new infrastructure
• Create and add an event-system component
• Publish an event when the event-system is up
• Declare the full set of events and functions to run
• Set up a subscriber registry in the event system component
  • create a simple, first draft for mass-subscription

Perform misc fixes.

For iterators and transducers (as implementations).

With the addition of 15 years worth of old material to a blog, operations can no longer be done efficiently entirely in memory. A new approach is required:

• generate list of files
• sort on date
• read one file at a time:
  • parse it
  • store content in db
  • store metadata in db
  • process and store per-post stats in db
• after last file is parsed/processed, write reducers for:
  • all categories on stored data
  • all keywords on stored data
  • all authors on stored data
  • stats on stored data
    • total line count on stored data
    • total word count on stored data
    • total char count on stored data
• once this is prototyped in a series of isolated functions, refactor current workflow to use it.

It's gotten a bit large now ...

• Devise schema for posts
  • Include a checksum
• Ingest posts into Redis (if the checksums haven't changed)
• Devise schema for blog stats (all the current posts stats)
• Crunch numbers on posts
• Insert these into Redis using the stats schema
• Write queries to extract posts and stats from db

I'd like to write a dragon plugin that adds Flickr images to blog posts (of a given size, arrangement, and link back to Flickr). There's a ticket for this here:

• forgotten-roads/blog#82

In order to do this, though, I need to inject a task into the normal content-parsing workflow, before markdown is converted to HTML.

In order to that, I need Dragon to be event-driven (see #1). With that in place, adding plugin support should be pretty straight-forward.

Phase I Tasks (Milestone 0.5.0):

• See #2

Phase II Tasks:

• Add a plugin component
  • should include a plugin registry at the very least
• Identify existing ad-hoc content parsing/munging/hacking rules
  • separate these from existing, critical, general workflow
  • add them as new "plugins" (i.e., use the new data workflow API)
• Implement an initial plugin (e.g., forgotten-roads/blog#82)

Phase III Tasks:

• See #44

• Add logging level/nss for dev/REPL
• Add logging level/nss for app/web
• Update configuration API to pull these values from configuration
• Update the dev REPL to use this configuration (see issue #3)
• Update web.run to use this configuration (see issue #5)

Dragon currently requires that one run Docker in order to manage the content cache (Redis). We need to support those who can't (or won't) run Docker by allowing Redis to be run natively (not in a guest os/container).

This should essentially amount to creating a new backend for native Redis. It might also be a good idea to rename the current Redis backend to docker-redis or somesuch.

Dev tasks:

• Rename redis data source impl to redis-docker
• Rename redis configuration to redis-docker
• Create common configuration for both docker and native redis
• Refactor redis-docker config to use common config
• Test redis docker refactoring on devices that support docker
• Create native config
• Create native data source
• Test native data source on older iMac
• Move dragon config into profile
• Document overriding the default on an older iMac without making config change (README)

• Add httpd component
• Only bring up HTTP if run on web/app mode (or if explicitly called)
  • add a new init to system that brings up HTTP
  • update main or web to start the system
  • ensure the new init is run, not the default init
• Clean up old web ns

The docker image currently being used is 4.something; Redis 5.x has been released.

This is more stylistically motivated, more than anything. Had to do this recently for a different project which used the extend approach to make it easier for Lifecycle implementations to add more methods.

See:

• cmr-exchange/dev-env-manager#27

In particular, the change sets:

• cmr-exchange/dev-env-manager@2dde3e1
• cmr-exchange/dev-env-manager@d42a542
• cmr-exchange/dev-env-manager@d42a542

For an example of use from the controlling namespace, see:

• https://github.com/cmr-exchange/dev-env-manager/blob/master/src/cmr/dev/env/manager/components/process.clj

Tasks:

• For each component, group function in the component implementation namespace according to:
  • The application-specific API that the component offers
  • The component lifecycle functions
• Simplify the lifecycle records to be just the records (no method implementations)
• Convert the methods to functions that take this and update references to component
• Add behaviour definition
• Add extend call
• Add/update create-* constructors
• Update component system ns (i.e., the "master" or "controller" ns) to use the new constructors

Looks like we're not developing Datomic support side-by-side with Redis support; let's just dump Datomic and the extra dep, then. We can always add it back later, if that's ever a need.

This will help bugs in metatags that are including HTML in their strings.

In particular, dragon.blog.post.impl.default/get-excerpts needs to be updated to generate more k/v pairs that have been stripped of HTML.

• Add carmine dep
• Update dev ns
• Add Redis config settings
• Add Redis support functions for Dragon DB API
• Add Redis support to DB component

When running the watcher, editing (looking at, really) a file with vi creates a .swp file and this gets processed by the watcher. We shouldn't do that ;-)

I'd like to write a dragon plugin that adds Flickr images to blog posts (of a given size, arrangement, and link back to Flickr). There's a ticket for this here:

• forgotten-roads/blog#82

In order to do this, though, I need to inject a task into the normal content-parsing workflow, before markdown is converted to HTML.

In order to that, I need Dragon to be event-driven (see #1). With that in place, adding plugin support should be pretty straight-forward.

Phase I Tasks (Milestone 0.5.0):

• See #2

Phase II Tasks:

• See #43

Phase III Tasks:

• Identify other areas of Dragon that can be converted to using this new infrastructure
• Document how plugins need to be configured in order to
  • define a specific function that gives all its subscribers
  • define and publish pre- and post- events for every function that will operate on dragon data (so that other plugins may determine if they need to go before or after another plugin
  • listen for the right events
  • take actions at the right times
  • publish their own events
  • suggest plugin code layout:
    • name.plugin.main?
    • name.plugin.events?
    • name.plugin.subscribers?

Use https://github.com/wkf/hawk.

See:

• cmr-exchange/dev-env-manager#34
• cmr-exchange/dev-env-manager@5ea8ae2
• cmr-exchange/dev-env-manager@0d55566
• cmr-exchange/dev-env-manager@b98a858

0.6.0 - Tasks:

• Add configuration for paths to watch
• Add configuration for watcher implementation to use
• Add abstract watcher protocol
• Add new dragon events for file changes that include ways of indicating file type/category
• Add Hawk watcher implementation that publishes dragon events
• Add watcher component

0.7.0 - Tasks:

• Add responder component - see #49
• Add subscribers to dragon file events the fire off the file change event handlers (part of responder)

This component responds to the events published by the watcher component. It will dispatch specific functions upon published event type/s, but with controls/limits appropriate for computationally expensive or time-consuming handlers. This is delay/batching mechanism is intended to prevent handler execution floods, keep rapid-fire changes from bogging down the system.

The first half of this work was done in 0.6.0 (see #49)

The following tasks remain:

• Add timeout-tracking capability
  • Defining a countdown protocol
  • Define countdown behaviour functions
  • Extend the component to use the protocol and behaviour
  • Implement state tracking with async go loop
• Add channel logic to fire after the countdown (timeout) has completed
• If an additional request is received before during the countdown and be fore the firing of the handler, re-do the countdown -- only fire the handler if no new events have made a request during the countdown (timeout) window

Instead of the awkward docker integration, use this instead:

• https://github.com/clojusc/process-manager

May need to file some support tickets, since this hasn't been used after the NASA fork ,,,

This should be an in-memory db. Will be used for such things as:

• storing and updating subscriber functions for the event system (subscriber registry)
• storing plugin data (plugin registry)
• refactoring blog processing into ingest ("posts" table) and stats ("stats" table)

Tasks:

• Create initial/placeholder db component, ensuring the system still works with it
• Investigate the in-memory db Clojure ecosystem
  • Datascript - https://github.com/tonsky/datascript
  • Datomic - has an in-memory mode and free starter license ...
• Select an in-memory db
• Add any missing / necessary db configuration
• Update the component to properly initialize the database "connection"
• Have the component start up datomic itself (instead of requiring it to be started on the system already)
• Also have the component kill the database process on shutdown

I suspect there is an event that needs to be published, but only when not in web or help cli mode, just regular cli mode.

I.e., instead of working off of the old, large data structure that hold everything.

Now that dragon uses component, lets get the dev system using it.

Sub-tasks:

• Refactor base/common functions
• Bring in static page data fns
• Bring in page listing data fns
• Bring in dynamic page data fns

Part of #28

We've established a bad habit of passing around the whole system data structure.

Sub-tasks:

• Bring in default page listings (archives, categories, tags, authors)
• Bring in default static pages (about, powered-by)
• Bring in default dynamic pages (front-page, post)

Part of #28

The Flickr embedded JS sucks ass (slow page loads, image distortion during load, requires additional JS in the page, etc.). Can we get a Flickr tag that does just what we want?

• Save a Flickr API key to a private dir/file and have Dragon read it
  • Add configuration for key
    • placement in defaults
    • function for access
• And support for multiple content types in the content metadata
• Add content parse cases for selmer and [md selmer]
• Add support for a custom Selmer tag
  • Create a new namespace for custom Selmer tags
  • Use a Selmer tag (e.g., {% flickr-img ... %})
  • Provide the Flickr Photo ID as a tag parameter
  • Provide the preferred size as a tag parameter
  • Provide an album name (optional) as a tag parameter
  • Choose the closest static size
  • Resize that (with CSS or HTML) to the actual, desired size
  • Keep the image proportional
  • Link to the photos page, using the album name/id, if provided
  • Handle the Selmer code in Markdown
• For all posts wishing to use the new Flickr tag, update content type from md to md,selmer

I'd like to write a dragon plugin that adds Flickr images to blog posts (of a given size, arrangement, and link back to Flickr). There's a ticket for this here:

• forgotten-roads/blog#82

In order to do this, though, I need to inject a task into the normal content-parsing workflow, before markdown is converted to HTML.

In order to that, I need Dragon to be event-driven (see #1). With that in place, adding plugin support should be pretty straight-forward.

Phase I Tasks (Milestone 0.5.0):

• Examine the markdown-specific workflow
  • identify any inconsistencies or redundancies therein
  • generalize these for all types of content
  • once #1 is complete, add any missing content-specific events
• Identity a general way of handling content data - this will essentially become the "plugin framework"
  • from data source through to final output form
  • should be amenable to use with threading macros
  • possibly take inspiration from Ring handlers
• Identify all new default/general content events that should be published
  • identify ways in which "plugins" could publish events that other plugins could consume
  • maybe specify that each plugin should provide a before and after event for each action event?

Phase II Tasks:

• See #43

Phase III Tasks:

• See #44

A la https://github.com/hexagram30/societatis/blob/master/resources/scripts/switch-to-https-submodules.sh

Note that without this, CI/CD will be broken in the release/0.6.x branch ...

(or multimethods ...)

Replace the current system setup in dev.clj, the CLI, anywhere else with a system created using the system-manager project (from the clojusc org).

The component/system.clj file should also be refactored appropriately as part of this.

There's a show subdir for other show namespaces; let's keep it all together, like the rest of the codebase does.

Tasks:

• Add Component dependency
• Reorganzie code for use as components
• Add a configuration component
• Add a logging component

This will allow for the down-stream overriding of the blog post processor implementation, addressing much of what the plugin ticket (see #2) aims to do.