The parquet files not found in output directory about cosmos

[RichardScottOZ]

So on this one

Writes things to temp directory - at what stage do you expect to see output - e.g. working/not working - per file/worker job or all at the end?

Haven't had time to digest all the code as yet, sorry.

[RichardScottOZ]

Testing on a not that impressive machine but does have a 24GB gpu, I just left it at the default PROC/WOKER numbers

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
30737 root      20   0 6017288 3.656g 152480 S 102.7 11.8 113:14.47 python3.8
30041 root      20   0 1154276 355576  39648 S   9.0  1.1  16:31.49 dask-scheduler
30226 root      20   0 2855796 167980  58136 R   3.7  0.5   4:37.07 dask-worker
30561 root      20   0 3464460 628032 172920 S   2.3  1.9  12:40.62 python3.8
30550 root      20   0 3329896 490296 172880 S   2.0  1.5  12:10.79 python3.8
30553 root      20   0 3460444 627172 173124 S   2.0  1.9  13:31.27 python3.8
30557 root      20   0 3461712 630872 172760 S   2.0  1.9  12:21.35 python3.8
30565 root      20   0 3603740 760952 172064 S   2.0  2.3  12:13.14 python3.8
30574 root      20   0 3468772 631712 172168 S   2.0  1.9  12:19.38 python3.8
30569 root      20   0 3461620 623408 172252 S   1.7  1.9  12:01.31 python3.8
30576 root      20   0 3426160 601428 172904 S   1.7  1.8  16:51.17 python3.8
30383 root      20   0 2213932 274448 139332 S   1.0  0.8   1:27.01 dask-worker
29813 ubuntu    20   0  554556  70940   9528 S   0.7  0.2   0:22.43 docker-compose
26729 root      20   0 1499400  50144  28256 S   0.3  0.2   0:02.62 containerd
28547 root      20   0 1802856 132844  55960 S   0.3  0.4   3:16.77 dockerd
    1 root      20   0  225536   9440   6788 S   0.0  0.0   0:05.21 systemd

[iross]

I suspect the issue here is that you're trying to use the images built against master while using the docker-compose file from the v0.5.0 release. Taking a quick look, I think your current env file and the docker-compose-ingest.yml from the master branch should work, or updating the env file to use the docker images with v0.5.0 tag (in place of latest).