opendataservices / developer-docs

Documentation about how we get our work done. For internal use, but happy to share

Open Data Services Developer Documentation

A repository of documentation about how we get our work done.

To read on GitHub (or locally), choose the files directly, or see the Table of Contents.

It's also available to read and download on GitBook at https://opendataservices.gitbook.io/developer-docs/

This is broken up into sections:

• How to guides - step by step guides on how to do certain tasks
• Reference - direct information on how we do our work

We try not to list basic instructions for common tools. Different developers will be at different levels for different tools, and in any case it's possible to look up these instructions online. Instead we list step by step guides for specific tasks we perform in a certain way at Open Data Services, and we make sure each how-to guide is focused on a specified user need. How to guides should be linked to relevant places in the reference, and vice versa.

developer-docs's Issues

Where is our client specific code?

https://github.com/OpenDataServices/developer-docs/blob/master/code-checklist.md#our-codeprojects-should-be-in-version-control-and-present-links-to-issue-trackers-and-source-code

I think this section is useful as it is, but I think maybe "our code" (code we're responsible for building, and possibly maintaining) is in our client's organisations. Maybe we want a policy of always maintaining forks in our organisation, so that people can easily find "our code".

Problems with maintaining our own copy of client repos for tests/deployment

Currently for NRGI we're maintaining a fork of the NRGI code in our OpenDataServices organisation - and it's this fork we're using to do tests and deployments from. To keep both of these up to date, me and @caprenter are both set up to push to both repos at once - #38

However, we're still using the NRGI repo for issues and pull requests. This is fine for the most part, put I've been tripped up a couple of times because merging a pull request in the UI only merges it in the NRGI repo. To get travis to see it, I need to remember to pull and then push again locally.

The solution to this might just be to remember to pull and push, or we may want a more automatic way of keeping the two repos in sync. I'm wary of automating this too much though, because one of the advantages of having our fork is it gives us more control over what then gets pushed onto our servers...

We should use travis' container based infrastructure wherever possible?

Prompted by open-contracting/sample-data@f5f6b9a

http://docs.travis-ci.com/user/migrating-from-legacy/

Security update reverse dependencies

When there are security updates for a piece of our software, any software depending on that will also need updating. E.g. flattentool -> Cove, Cove -> Resource Projects ETL

It's probably useful to log these reverse dependencies somehow (currently the list if small and lives in my head).

We now have deploys on Ubuntu 16.04 / Python 3.5

The docs should reflet this.

I had to update cove travis as well: OpenDataServices/cove@5477ea6

We try to employ agile approaches where appropriate

We don't currently do this fully, because of our small team size amongst other reasons. Might be good to have a section in here about our aspirations though.

I think ultimately we want a separate doc about how we do agile, which hopefully these docs could link to.

Default Huboard columns changes

I've just set this up for developer-docs and it looks like the initial defaults have changed - https://huboard.com/OpenDataServices/developer-docs/

Looks like there's now a Review column, and sort of limit on how many issues can be "Working". We may want to evaluate this, and decide whether to set it up for other projects.

We might also want to think about what other column changes we want to make, as I think it's all very configurable.

Approaches to profiling the slowness of our page loads

http://yslow.org/
https://developers.google.com/speed/pagespeed/?hl=en
https://gtmetrix.com/reports/cove.opendataservices.coop/7kxyORI3
http://modpagespeed.com/

Should we use integration branches?

We discussed on the train to Berlin how to deal with the fact that merge conflicts may happen in future. My main suggestion was rebase a lot, but, looks like having an integration branch, and using git rerere may also be a useful approach.

https://github.com/affinitybridge/git-bpf/wiki/Branch-per-feature-process#the-integration-branch
http://git-scm.com/2010/03/08/rerere.html

Add some information about our deployment approach?

This should heavily link over to https://github.com/OpenDataServices/opendataservices-deploy

We should explain our middle aground approach for semi automated deployments for our small setup. I think we have "Pets with strong configuration management", as described in some CERN slides:

Future application architectures should use Cattle, but pets with strong configuration management are viable and still needed.

First revision

@Bjwebb I'd appreciate it if you could take some time to check this first set of documents over.

There is some rst to md tidying up to do, which I think you could do quickly, but the content (text) also needs a bit of a check, but only a little needs re-writes I hope.

All code should be seen/understood by more than one person

Someone suggested this as a general approach to avoiding the loss of knowledge when a member of the team leaves.

This would involve someone extra checking all pull requests / newly commited code, even if the code if written by the "owner" of that repo.

I don't think we do this currently, but I think it's a good aspirational goal, and one that should hopefully become more achievable as the team grows.

We might want some sort of process in place for deciding who the second person should be, if it isn't the "owner".

We should lint our HTML (and templates)

Add information about setting up a flake8 git hook

https://github.com/Bjwebb/bookish#setting-up-your-local-system-for-development

(split out of #4)

Should we be telling people to use temporary branches and git amend?

https://github.com/OpenDataServices/developer-docs/blob/master/tools/git.md#temporary-branches

In general having temporary commits on temporary branches is better than using git stash. ... they can also be pushed to the server. .... If you have a temporary commit you can update it using git commit --amend -a --date="date``

If you push to a remote, git amend and then try and push again that's not great? Maybe take advice out about commit --amend?

(Also maybe tell people to delete temporary branches from GitHub when done?)

Checklist before merging a pull request

I don't think this is necessarily a strict list of things that must be done, but would be good to have a list of things to check for, e.g.

• Code review...
• Tests?
• Documentation

Add page/section on using the terminal effecitely

Particularly, I've found the readline shortcuts at https://wiki.archlinux.org/index.php/Keyboard_shortcuts#Readline very useful.

Useful git commands

Might be useful for me to create a section of these.

For my OCDS work I think I'm going to want to git blame a diff, which lead to me coming across https://github.com/dmnd/git-diff-blame and and http://stackoverflow.com/questions/5098256/git-blame-prior-commits

Converse of adapting with external changes

https://github.com/OpenDataServices/developer-docs/blob/master/code-checklist.md#10-our-code-will-need-to-adapt-with-schema-changes-and-changes-to-external-systems-upon-which-it-relies

Reading this, it occured to me we might want to consider the opposite of this - that we might be providing APIs on which others rely, and we want to be careful to give people sufficient warning of changes etc.

Testing our code under load

We should know that our code can handle expected load / file sizes - and fails gracefully e.g. for large files.

Maybe this belongs in
https://github.com/OpenDataServices/developer-docs/blob/master/code-checklist.md ?

Also add link to our public page on a external service (where it exists)

e.g. in https://github.com/OpenDataServices/developer-docs/blob/master/code.md#external-testing-services we should link to https://travis-ci.org/OpenDataServices etc.

Frontend docs are out of date

https://github.com/OpenDataServices/developer-docs/blob/master/front-end-development.md

We have more frontend build steps for stuff now, e.g. sass, but this varies a lot between projects.

@kindly Could you take a stab at updating this page?

Notes about Python code out of date

https://github.com/OpenDataServices/developer-docs/blob/master/code-python.md

We now use Python 3 by default (although we have inherited a bunch of Python 2 code) - and Python2/3 support for libraries.

lxml, flask and Jinja are not such common dependencies for us - instead we might list Django.

We now have some python packages (although not on PyPI).

Rewrite domains section of code.md

Wget only fetches the first page of issues

Using the command at https://github.com/OpenDataServices/developer-docs/blob/master/working-off-line.md#grab-github-issues-to-work-offline-eg

Wherever possible, we support browsers without javascript

http://kryogenix.org/code/browser/everyonehasjs.html might be nice to link to.

Our approach to front-end development

This probably won't say much, but should set out how we don't do very sophisticated frontend stuff atm - lots of bootstrap etc. The fact that we're using plain CSS + JS everywhere, and don't have any sort of transpilation (or for the most sort any sort of build) setup.

Our use of JS is generally fairly oldschool progressive enhancement (JQuery table sorters etc.).

We tend to use external CDNs for our dependencies, as it's easier, and people might have these cached. Might be worth setting out the advantages/disadvantages of this - I think mostly our setup is not the greatest, but it is simple - which works well as we're not heavily frontend.

Roadmap for our approaches to code

What things we're currently trailing - e.g. Docker https://github.com/OpenDataServices/developer-docs/blob/master/tools/docker.md, Agile approaches #11
What's on our wishlist e.g. Email signing

#21 - general approaches to software development may be relevant to this.

Should we use GitHub's protected branches?

I currently have this set up for Cove - for the branches master and live.

It's not possible to push directly to the branch - tests must pass on Travis before code can go in. If you try you get:

! [remote rejected] master -> master (protected branch hook declined)

Fastforward merges into master of commits that have already passed - it's not necessary to use GitHub's merge interfaces.

I think it would be good to set this up for all our live branches, and most of our master branches on our code repositories.

We should link to huboard from READMEs, including this one

Where we use huboard to manage a repositories issues, we should add a link from the README so that others can find it.

Tools that may be of interest

(Moved out of planio wiki)

Open Calais
Swagger - API Documentation (via http://discuss.iatistandard.org/t/api-standard-resources/65/4)

Useful resource for switching between programming languages

http://hyperpolyglot.org/

Add notes about pushing to two remotes at once

http://stackoverflow.com/questions/14290113/git-pushing-code-to-two-remotes#answer-14290145

git remote set-url --add

Plan for moving Python 2 code to Python 3

I was going to add this to the Python 3 doc:

The plan is to transition to Python 3 next time we have a major development sprint on any of these projects.

But, I think this needs to be agreed first.

The lead for a project, should be the one to merge pull requests?

We should have a lead person identified - https://github.com/OpenDataServices/developer-docs/blob/master/code-checklist.md#2-all-code-should-have-a-lead-person-identified

Should they responsible for merges etc.? This responsibility may be delegated, but it might be good to have a default that they are the only one to push to master/live?

Tips for working offline

E.g. wget -m --no-parent -e robots=off "https://github.com/OpenDataServices/developer-docs/issues/"

Notes about restarting services

On modern Ubuntu, don't use /etc/init.d, use service or systemctl depending on version. This will avoid issues with multiple processes running e.g. with logstash and upstart.

General approaches to software development

We may want to add some information about this, e.g.:

• https://en.wikipedia.org/wiki/Technical_debt
• 12 factor web apps
• DRY, and similarly having a SSOT

Other possibilities / aspirations:

• Agile

Our code should be public

... and our development processes should be public.

This is implied throughout https://github.com/OpenDataServices/developer-docs/blob/master/code.md and there's some mention of public workplans https://github.com/OpenDataServices/developer-docs/blob/master/code.md#workplans , but I think maybe we could be more explicit. (e.g. saying the code should be in the GitHub organisation implies public, but private repos in that organisation are possible).

Deployed dev copy of feature branches

Not sure how best to write this up in a generic way, so we should document the Cove specific case first of all.

Concrete example:
OpenDataServices/cove#76
branch: minor-live-improvements
Added to bottom of pillar https://github.com/OpenDataServices/opendataservices-deploy/blob/master/pillar/dev_pillar.sls
Deployed using highstate at http://minor-live-improvements.dev.cove.opendataservices.coop/

(Internally we create a cove macro, and apply this for every time a branch happens.)

Debugging Python code

More info about python debugging - ipdb in general and in particular py.test --ipdb

What does our overall workflow for "doing" software look like?

Some notes about this (may be out of date):

Our internal wiki has some specific information about deployment, but nothing about our general approach to "doing" software, through development to deployment and hosting.

There are issues for me to write up about possible agile approaches, and some software development guidlines [ie. this repository].

However, none of this actually describes our overall "pipeline" for doing software....

I drew some diagrams to help explain this ofr our June OGM. It would be good to write these up, and get them in a slightly cleaner format.

This ties into two of three? core principles of our software approach:
Testability
Reproducibility
Modulatrity ?

This also ties into our use of third party services e.g. Travis, Sentry

Reproducibility ties into dependency pinning https://github.com/OpenDataServices/developer-docs/blob/9771d75392b3cd23dec2a9dcd00804365a6df107/code-python.md#pinned-dependencies. We may want to also list other similar such practices...

See also this issue about adding more info about our deployment approach #28

Some extra questions:

• What are the risks with the services we're using? (particularly third party ones)

Version control checklist item is not numbered

https://github.com/OpenDataServices/developer-docs/blob/master/code-checklist.md#our-codeprojects-should-be-in-version-control-and-present-links-to-issue-trackers-and-source-code

I'm not sure if this should be at the bottom, or whether we want to bump all the numbers below this?

Review code-python.md

@Bjwebb - I think code-python.md could do with your eyes running over it.

Start of sprint checklist

This would suggest tasks we may want to possibly add at the start of any of our sprints.

This may include:

• Checking that dependencies are up to date, and that we're using a recent version of Python (see #23 )

Useful docker commands

https://imagelayers.io/?images=opendataservices%2Fresourceprojects.org-frontend:master,opendataservices%2Fresourceprojects.org-frontend:248-missing-markers

docker exec
So, for a container named frontend: docker exec -it frontend bash

How we use git

• Our workflow has some similarities to, but is not gitflow
• Working on feature branches, and rebasing/squashing these (rebase -i) to give a useful clean, usefully broken down view of history. (including rebasing on master rather than meging master in, if possible)
• Using [#issueno] rather than fixes #issueno in order to leave closing issues until it's been checked they're "Done"

Also maybe:

• Committing tmp commits to tmp branches as an alternative to git stash

Add a table of contents to the README

This willl probably be manually curated, in order to allow us to pick an ordering, and also so we can continue to serve this on GitHub directly for the immediate future.

Odd wording

https://github.com/OpenDataServices/developer-docs/blob/master/tools-generic.md#wget "Open Data Services data" doesn't really make sense, as it's not our data.

Check whether there's anything else useful in the IATI developer docs

These docs are based on the IATI developer docs https://github.com/IATI/IATI-Developer-Documentation . Task is to double check whether there's any information there that we haven't got here that would be useful.