sandorjuhasz / mobility

Supporting research using mobility data

ANET Mobility project

Reports so far

To play around with the data

1. fork and clone the repo

git clone https://github.com/sscu-budapest/mobility

(preferably fork it first)

2. install deps

pip install -r requirements.txt

3. get the sample data

if you are using the anet server:

dvc pull

otherwise, set up the anet server to be anetcloud in ssh config and then

dvc pull --remote anetcloud-ssh

4. load some samples and look around

from src.data_dumps import ParsedCols
from src.load_samples import covid_tuesday

def total_range(s):
    return s.max() - s.min()

samp_df = covid_tuesday.get_full_df()

samp_df.groupby(ParsedCols.user).agg(
    {
        ParsedCols.lon: ["std", total_range],
        ParsedCols.lat: ["std", total_range],
        ParsedCols.dtime: ["min", "max", "count"],
    }
).agg(["mean", "median"]).T

		mean	median
lon	std	0.033919	0.001588
	total_range	0.080638	0.000471
lat	std	0.01889	0.001067
	total_range	0.045475	0.000336
dtime	min	2020-11-03 07:38:23.900066048	2020-11-03 06:33:42
	max	2020-11-03 18:43:00.657093888	2020-11-03 21:08:10
	count	87.307432	21.0

+ if you want to run something that can run on the full data set, I suggest using dask

from src.data_dumps import ParsedCols
from src.load_samples import covid_tuesday
import matplotlib.pyplot as plt


samp_ddf = covid_tuesday.get_full_ddf()

ddf_aggs = (
    samp_ddf.assign(hour=lambda df: df[ParsedCols.dtime].dt.hour)
    .groupby("hour")
    .agg({ParsedCols.lon: ["std"], ParsedCols.lat: "std", "dtime": "count"})
    .compute()
)

fig, ax1 = plt.subplots()

ddf_aggs.iloc[:, :2].plot(figsize=(14, 7), ax=ax1, xlabel="hour in the day").legend(
    loc="center left"
)
ddf_aggs.loc[:, "dtime"].plot(figsize=(14, 7), ax=ax1.twinx(), color="green").legend(
    loc="center right"
)

load a full week of data

from src.create_samples import covid_sample, non_covid_sample

# this is about 3GB of memory, use get_full_ddf for lazy dask dataframe
cov_df = covid_sample.get_full_df()

TODO

• "reliable user" counts

  • number of pings
  • "do we know where they live"
  • every month at least once a week
  • 30 / day (?)
  • 3 in teh morning, 3 in teh evening

• dump by month

• dump by user