dracidoupe / graveyard

Open Source reimplementation of https://www.dracidoupe.cz/

Graveyard: Place for Dead (and Undead)

Graveyard is an attempt at open-source reimplementation of DraciDoupe.cz (referred to as DDCZ in this text).

Developer's documentation is at Read the Docs.

Production is running at http://nove.dracidoupe.cz/ . But be warned, we are heading for the root domain soon!

Contributions

Contributions are welcome provided you agree your work will be shared under the same license as Graveyard (MIT). Please use ruff for code formatting and verification (VS Code integration here).

If you don't know where to start, take a look at the roadmap or ask Almad on development Slack or community Discord or in Pošta on DraciDoupe.cz.

Please install EditorConfig support for your editor (plugin for VS Code, plugin for PyCharm/WebStorm/IDEAJ)

Installation

You can run Graveyard either directly on your machine or inside Docker. Arm64 systems (like M1 Mac) are supported.

Installing and running Graveyard directly is faster (on some systems) and removes one lever of indirection, but it makes the setup more complicated.

Running in Docker requires familiarity with it, but it makes setup easier and guarantees consistency with the testing environment (and hopefully in the future, production environment as well).

In both cases, first clone this repository and run all commands in its directory.

Graveyard is not yet compatible with Python 3.10+.

Installing in Docker

Requirements:

• You have Docker CE installed

• You have installed docker-compose

• Createyour own copy of docker-compose configuration

  • cp docker-compose.example.yml docker-compose.yml
  • You can locally change your port if you do not want to run it at 8000

• Create your own copy of local configuration

  • cp graveyard/settings/local.example.py graveyard/settings/local.py

Verify you have everything ready by running the test suite:

• docker-compose run web python3 manage.py test

If you see output like this:

(graveyard-venv) almad@zeruel:~/projects/graveyard$ docker-compose run web python3 manage.py test
Starting graveyard_db_1 ... done
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK
Destroying test database for alias 'default'...
(graveyard-venv) almad@zeruel:~/projects/graveyard$

You are all set. Afterwards, install database schema by running

• docker-compose run web python3 manage.py migrate

and load data about pages

• docker-compose run web python3 manage.py loaddata pages

You are done! Now you can just run the project and develop using

• docker-compose start

Verify your application works and open http://localhost:8000 (localhost may be a different host if you are not working on linux). If so, create yourself a superuser.

For all commands in the manual that ask for python manage.py command, run docker-compose run web python3 manage.py command instead

Installing on your machine

Graveyard is currently written in Django. Requirements to develop it:

• You have working Python 3 installation on your machine
• You have working MySQL installation on your machine

To use the project, clone this repository, enter its directory with cd graveyard and:

• Create a virtual environment: python3 -m venv gvenv
  • If this fails and you are on Ubuntu, you may need to sudo apt-get update && sudo apt-get install python3-pip && sudo pip3 install virtualenv
• Enter it (on Mac OS X or Linux): source gvenv/bin/activate
• Install dependencies within the pip install -r requirements.txt
• Copy settings template: cp graveyard/settings/local.example.py graveyard/settings/local.py
• Edit the settings above, especially enter credentials to your local MySQL (see Stack Overflow on how to do that)
  • When creating database, for compatibility, use latin2 as default charset: CREATE DATABASE dracidoupe_cz DEFAULT CHARACTER SET latin2;
• Verify you have correct installation and run tests with python manage.py test. You should see output like this:

(graveyard-venv) almad@zeruel:~/projects/graveyard$ python manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...........
----------------------------------------------------------------------
Ran 11 tests in 0.031s

OK
Destroying test database for alias 'default'...
(graveyard-venv) almad@zeruel:~/projects/graveyard$

• If you see error like selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home, you have to go to the linked page and download the chromedriver to your computer. Place it somewhere in $PATH like /usr/local/bin
• Create the database schema: python manage.py migrate
• Load data about pages to see what's on production: python manage.py loaddata pages
• Run the thing! python manage.py runserver
• Observe if you have contact at http://localhost:8000

Installing on Windows

Follow the list above, except:

• It is strongly recommended to use Python 3.7 and MySQL 8, otherwise you'll have to compile mysqlclient manually
• Activate virtualenv using gvenv\Scripts\activate instaed of source gvenv/bin/activate

After Installation

• Load initial version of editorial articles: python manage.py loaddata editorarticles
• Create a superuser for yourself: python manage.py createsuperuser
• Look around the administration interface at http://localhost:8000/admin/

Installation issues

• Installation failes with "mysql_config not found"

If you get something like this:

    File "/tmp/pip-install-wfhe9zue/mysqlclient/setup_posix.py", line 29, in mysql_config
        raise EnvironmentError("%s not found" % (_mysql_config_path,))
    OSError: mysql_config not found

you may be using MariaDB fork of MySQL that the Python client is not equipped to talk to, installation-wise. You need to manually symlink the appropriate command:

ln -s /usr/bin/mariadb_config /usr/bin/mysql_config

If even mysql_config is not there, you have to install development headers for the database. That's apt-get install libmariadbclient-dev on Debian.

• error: invalid command 'bdist_wheel'

Old setuptools: pip install setuptools -U

Setup

Create user account

Use python manage.py registeruser command, see manage.py registeruser --help for parameters.

Example:

python manage.py registeruser mytestuser [email protected] logintograveyard

Debugging with Django Debug Toolbar

• pip install django-debug-toolbar==2.2.1
• Add to local.py:

from .base import INSTALLED_APPS, MIDDLEWARE

INSTALLED_APPS.append("debug_toolbar")
MIDDLEWARE.insert(
    0,
    "debug_toolbar.middleware.DebugToolbarMiddleware",
)

INTERNAL_IPS = ["127.0.0.1", "::1"]

graveyard's Issues

Keep track of loosely associated media

In the current state, some Creations have associated media (usually images). In the current implementation, those are uploaded separately into a directory and referred to from Creation text via HTML tag.

No association is stored; than only happens in cases where the media is the article (i.e. Gallery).

Find a way to look those up and keep track of the relation in order to be migrate the media to other store reliably.

Properly link Creation authors to their profiles

Housekeeping:

• Have separate Creation Author profile
• Redirect profiles to normative URL in case of user nick slug mismatch
• Creation models should know their canonical link & add link to profile page
  • Migrate Downloads and Quests to use the unified Creation views (and URLs and links) and rename the URL patterns appropriately (commonarticle -> creation)
  • Consider using get_creation_canonical_url in list templates as well
• User icon on user profile
• Consider template tag for displaying authors
• Fix a particular author going to unknown http://zeruel:8000/rubriky/galerie/966-truhlice-ii/
• Handle News authors properly
• Handle Authors without corresponding User properly
• Sanity integration check for all creativepages on example writings
• Link to Author's profile from User's profile

Proper semantic web annotations and json-ld

Translate model attributes in English

This can be done without db migration since db column can be overruled. It will make codebase more consistent and readable.

• Tavern @Almad
• News @Almad
• Creations @Almad
• Discussion @Almad
• Social @Almad
• Users @Almad

Fuckup recovery

• Save original branch for easy final version copypasting
• Squash original translation blob
• Copypaste back all original column names
• Create db_column AlterColumn migrations
• Copy all models from the original final branch
• yes | rename migrations
• Verify tests 😬

Proper ETags for items and lists

Be able to reset password

This needs to work around the Django vulnerability in >2.0<2.2.x version.

Print style sheets for creations printing

Although reliably, I'd probably still add JS counter for the amount of people who printed using the button :)

Proper and full User Profile site

• Displaying basic info
• Handle permissions
• Reputation summary and reputation detail page (Part of task #126 )
  - [ ] Newbies&Mentats
• Runes

Figure out mailing infrastructure

• We should sent emails as we did
• Email settings should be respected (and make extra sure unsubscribes still work)

Actually two issues:

• Transactional emails #90
• Mailing lists

Fix semantics of article detail

Article name should be h1, figure out good element / class for sections.

Verify accessibility

Turn off styles, verify responsiveness and accessibility guidelines.

Do not hardcode port for attaching test liveserver

• Delete the hardcoded port from Selenium class
• Figure out the assigned port range and allow it in django-compose

Figure out proper pagination

Properly paginate lists.

Implement proper paging for creation pages lists

Refactor CommonArticles and Creations

Unify them:

• Common ancestor Creation
• Unified Templates
• Common View correctly named
• Create fixtures for current hardcoded model
• Prevent runtime from messing it up
• Fix article detail

What needs to be configurable?

• Template for particular Creation Page (probably more readable than dynamic magic around fields)
• Template could probably be hardcoded / set on CreativePage model attribute directly

Considerations:

• Solve creations with custom media being uploaded

Problem with common articles when spidering

NoReverseMatch: Reverse for 'common-article-detail' with keyword arguments '{'creative_page_slug': 'clanky', 'article_id': 2677, 'article_slug': ''}' not found. 1 pattern(s) tried: ['rubriky\/(?P<creative_page_slug>[^/]+)\/(?P<article_id>[0-9]+)\-(?P<article_slug>[^/]+)\/$']

Properly redo packaging to use setup.cfg

See how Django does it and look for official documentation.

Figure out upgrade to Django 2.2

Seems like newer schema migration framework or some other component is incompatible with mysql 5.5, which is unfortunately used on old production: https://circleci.com/gh/dracidoupe/graveyard/113

Make all CreativePage sections available

• Hřbitov
• Dovednosti
• Linky
• Alchymistické předměty
• Hraničářská kouzla
• Kouzelnická kouzla
• Předměty
• Downloady
  • Display the current state
  • Reconcile migrations with the fix-items branch/master
  • Correct download as in the original version
  • Make new version-specific download counter
  • Migrate to django-storage and handling uploads
  • Figure out local development and display
  • [ ] Prepare the final migration script for merging download counts (-> #100 )
• Dobrodružství

---

• Check in database/codebase what Dovednosti.hlasoval is used for
• Update infrastructure documentation with S3 and env var setup
• Verify on production current state of predmety.precteno attribute/db column

--

Design for Dobrodruzstvi (Quests) and Downloady (Downloads):

• s3sync cron on the original server to synchronize two way with a bucket from which we'll serve the data
• handled on the graveyard side by using django-storages

Verify Misencoded Queries

What is the best (and future-proof way) to query a database in a way where misencoded fields are managed transparently?

See http://ddcz.readthedocs.io/en/latest/migration.html#database-encoding and https://github.com/dracidoupe/graveyard/blob/master/ddcz/models/magic.py for context.

Switch to CircleCI pipelines

Hi there,

You are receiving this email because you are an active member of one or more projects building on CircleCI that do not have pipelines enabled. Below are the settings URLs for these projects:

[https://circleci.com/gh/dracidoupe/graveyard/edit#advanced-settings]

We will be automatically turning on pipelines for these projects starting March 9th, 2020.

If you are an admin for any of these projects, please visit Project Settings > Advanced Settings (or use the URLs above) and enable the “pipelines” setting for the project. A pipeline contains all of the workflows (and the jobs inside those workflows) that CircleCI runs after a trigger on your project. A pipeline can be triggered by a pull request, a commit, or even an API call.

Enabling pipelines will offer the following improvements to your CircleCI experience:

Access to our new user interface
Auto-cancel redundant builds on non-default branches
Ability to reuse config and share config using orbs
New API endpoints (v2 API)
Our goal is to make this transition seamless, but we encourage you to enable pipelines manually at your earliest convenience to mitigate any unforeseen errors.

If you run into any issues, check out this Discuss post for more resources. You can also register for this webinar on March 5th where a member of our team will walk you through enabling pipelines and the advantages of pipelines.

Thank you!
The team at CircleCI

Fix gallery grid

Consider using Solid / storing data in Pod

Custom media manager

As a follow-up for #3 , provide a way for user to upload their own media (to a certain size limit) and link to them from within articles).

Production consistency check

Given the amount of handwaving, do a consistency check command that would check underlying assumptions.

Examples:

[ ] Items for downloady are where it's assumed

Upgrade to CircleCI 2.0

1.0 is getting sunset: https://circleci.com/sunset1-0/#migration-steps

`ddcz` application breakdown

By breaking down the model field, more apps with unidirectional dependency graph shall be considered:

• Creations
• Social
• Chat (rewritten)

Figure out filters

G'old article filters as the old DDCZ had it.

Properly set conditional views / ETag rules

Fix gallery overflow into right menu

Exception reporting infrastructure

OK, number of integrity problems are coming up and all the 500s/logging.fatals need to be reported to developer.

Figure out how to do that. Sentry is a good contestant, but try to be GDPR-conscious and try to use least amount of 3rd party infrastructure possible.

Link editors to Creative Pages

Allow editors to be set for CreativePages and drive the permissions from there. Fix the problem with migrations.

(or should it be directly derived by permissions on CreativePage?)

Verify proper logging of errors and slugs

Send a warning / error on UserProfile.slug being empty

Redo stars as SVGs

Rating stars have the g'old feeling...that doesn't scale to new monitors. Also, GIFs.

Redo those as scalable SVGs.

Creations as first-class citizens

• Have an official enumerator for them
• Allow people to use database for things that are now implemented as ad-hoc templates (i.e. rules for adding articles)

Provide reasonable fixtures for other developers and testers

Example articles and overall content should be provided for other devs.

Changing skin takes you away from the page you are at

• Add returnto argument to http://nove.dracidoupe.cz/nastaveni/zmena-skinu/?skin=light
• Do not forget to validate it's on the same server to prevent redirect hijacking

Add Charters for Creative Pages

Currently, they are randomly linked and not editable by editors, that's relatively easy to fix and helps migration.

Offload regular tasks using Celery

Celery now has direct Django support: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

Level system computations etc. should be handled using that (with Admin interface as a bonus). If Graveyard works properly™, that should actually be done while the old version is still running.

Refactor APPROVAL_CHOICES

Move them out of model and reuse them in views instead of relying on "a" and "n" as magical constants.

Defer creations descriptions in list

Once views are stable, use defer('text') everywhere.

Creation Groups should be defined explicitly

• Extend CreativePageSection:

class CreativePageSection(models.Model):
    """ Section within a Creative Page """
    name = models.CharField(max_length=30)
    slug = models.SlugField(max_length=30)
    creative_page = models.ForeignKey(CreativePage)

• Put all existing sections into pages.yaml / new sections.yaml fixture
• Link all existing creation pages to it
• Write migration for existing data

Proper separation of static media

Because of modern technology, static media can now be deployed on a different domain and moved to CDN.

Implement sending new creations to approval queue

• CommonArticle
• Graveyard
• (Photo)Gallery
• Skills
• Monster Manual
• Campaigns
• Items
• Downloads
• Links
• Alchemist tools
• Ranger's magic
• Sorcerer's magic
• Dating
• Advertising

Handle creations without authors properly

(originally, there are creations with sources (zdroje) without authors)

Support full HTML range from original page

Only a subset is supported now. Support the full range of HTML weirdness.

Current requirements:

• Proper th/tr/td colspan/rowspan handling

Follow-up todo:

• Check for need for addslashes
• Make sure we are handling check on tag pair on input correctly

Reflect user settings in pagination and list styles

Write down old router

Old routes must resolve to new once.

Hence, one big fat index.php dispatcher is needed to route to new URLs.

Figure out interop with "uploaded" media

This means especially:

• Custom uploaded "article" content in obr_pris
• Downloady

Set CachedStaticFilesStorage / properly version static files