Data store/cache service improvements needed about slackemon

[tdmalone]

@Naramsim I'm glad you've suggested that! I had a bit of a read into Compose today but it looked too complicated for me to understand at first, so to start development on this I just installed PG and manually linked the containers together. It was a challenge to get the PHP PG extension installed but I got there in the end!

If you could help with the docker-compose.yml that would be fantastic. I would prefer Apache for now but maybe we can move to nginx later - I know it's faster but just trying to manage how much I take on that I haven't used before ;)

[Naramsim]

I would not stick with alpine-linux, and installing php/apache/postgre on it.
I would prefer use three containers that are based on Alpine. That are

• php:alpine
• postgres:alpine
• httpd:alpine

So, in the end, we would have a system composed by three small (and secure) containers, which are started automatically with the command docker-compose up -d

What do you think?

[tdmalone]

I'm currently working on implementing Postgres as an option for the data store, using docker postgres for dev and intending to use the free hobby plan of Postgres on Heroku for prod (allows up to 10k rows).

Redis is another potential option but is only free on Heroku without data persistence, so that's on the backburner for now.

10k row limit for free Heroku prod should be fine providing we can handle large JSON strings - in the current structure, that's 1 per player, and 1 each per spawn instance and battle instance both of which can be pruned.

[Naramsim]

Hey, @tdmalone, I see the project is expanding 😄

Maybe it's time to create a docker-compose.yml. Basically, it is a file processed by Docker Compose (on Linux it is not shipped with Docker) which creates several containers at the same time and orchestrate them. So we could split the project into the following images/containers:

• Apache2 / Nginx (you choose)
• Php7
• PostgreSQL

[tdmalone]

(We could even look at Alpine Linux maybe to try to reduce the size later, but I'm not sure if that would just increase complexity with getting things working)

[tdmalone]

I think that sounds perfect!

[tdmalone]

Postgres should now be working in https://github.com/tdmalone/slackemon/tree/data-store as an option for SLACKEMON_DATA_STORE_METHOD. I just need to test this on Heroku and then will probably push it to master and bump to v0.0.39.

This partially implements point 2 (Postgres data store ability) mentioned above at #21 (comment)

The rest of point 2 (using Postgres on Docker) will be covered by using Docker Compose.

Point 3 (file locking) is probably the next priority, then point 1 (augment AWS cache with local cache).

Of course, Postgres isn't implemented properly at this stage - we're basically treating it like a filesystem with each row being a 'file' and containing a big JSON string. In future we should completely rewrite this, drop support for a local data store, and use the database properly.

[tdmalone]

• Automatically create database if it doesn't exist yet, for example when run in a new Docker setup
• Data caching appears it might be broken at the moment, may have been a regression caused by the work in this ticket so far, need to look into
• Fix regression with $action not sending through full data anymore
• Verify that slackemon_get_files_by_prefix() is returning in alphanumeric order when the database storage is being used

---

• Docker Compose (see #35)
• Initial file locking work
  • Basic file locking
  • Make locking an optional variable for now, until it is improved
  • User feedback if waiting for a lock to become available
  • Make locking more intelligent to only lock when writing is just about to happen
• Augment AWS cache with local cache

[tdmalone]

Most of the work this ticket asks for is now in master!

A few more things to look into to make file locking more robust:

• Timeout if lock takes too long to acquire
• Is it worth implementing a queueing system rather than just the first try after lock is available?
• Is it worth trying to make file_exists() then file_put_contents() atomic? (Probably possible with DB, but what about local filesystem?) Potentially look into locks with flock() and fopen()'s modes.

[tdmalone]

In addition to the above, there probably needs to be a regular cronned check to expire locks that did not get removed properly - probably after 2 minutes? Or even 1 minute.