Create a data flow diagram related to the m-lab deployment of ooni about probe

[stephen-soltesz]

Hi, @hellais,

I'm happy to create a DFD.

The basic flow follows these steps:

1. Every machine running the oonib slice on M-lab also has an rsync daemon listening on port 7999.
2. OONIb writes files to the RSYNCDIR inside the slice filesystem (i.e. /var/spool/mlab_ooni).
3. Periodically, an M-Lab data harvester tries to read new data from the rsync daemon.
4. Anything that is downloaded for a given day is stored in cloud storage: https://storage.cloud.google.com/?arg=m-lab#m-lab
5. If there are scripts available to parse the raw data and translate it into an SQL record, then this runs next on each raw data file.
6. The "last successful collection time" for the data is recorded. And, a garbage collection script on the nodes removes collected files from the slice's RSYNCDIR.

Best,
Stephen

[hellais]

@stephen-soltesz Right on! Do you want to create the graph and push a .svg and .pdf version of it and send a pull request here: https://github.com/TheTorProject/ooni-spec/tree/master/diagrams?

[stephen-soltesz]

No problem. Before I do the pull request, does this sample look clear to you?

[hellais]

This looks good to me, just send a pull request and I will merge it.

[stephen-soltesz]

Ok. I've sent the above pull request that includes a pdf and svg for the above diagram.

[hellais]

Thanks I'm going to consider this closed.