Validate results for phasing pipeline about pipelines

[gbggrant]

I've copied the two VCFs to /lustre/scratch118/malaria/team112/personal/gg18/phasing-validation - let you get started while we figure out the access issues with gs://malariagen/

[hardingnj]

Hi @gbggrant , just wondering who owns the gs://malariagen bucket? I can see files, but I can't download them.

Copying gs://malariagen/genetic_maps/ag1000g/phase2/AR1/X.gmap...               
AccessDeniedException: 403 HttpError accessing <https://storage.googleapis.com/download/storage/v1/b/malariagen/o/genetic_maps%2Fag1000g%2Fphase2%2FAR1%2F2R.gmap?generation=1607974239579471&alt=media>: response: <{'x-guploader-uploadid': 'ABg5-Uy2UzVpHMg0YylQMsF7pOCnc8EsXngQ6KQ2lfKA06AVXnb0CBXASi0b0kznLyBwe2wB6U9DuHSLvCuWWDb1zko', 'content-type': 'text/html; charset=UTF-8', 'date': 'Mon, 11 Jan 2021 14:17:04 GMT', 'vary': 'Origin, X-Origin', 'expires': 'Mon, 11 Jan 2021 14:17:04 GMT', 'cache-control': 'private, max-age=0', 'content-length': '106', 'server': 'UploadServer', 'status': '403'}>, content <[email protected] does not have storage.objects.get access to the Google Cloud Storage object.>

[gbggrant]

@hardingnj access is managed by a google group. 'malariagen' which you are in (under [email protected]). Can you confirm that's the user you are trying to access the files as?

[hardingnj]

I'm pretty sure (unless I am misunderstanding something). That's the email in the error message above at least

If I switch to my personal account, I can't even ls:

(base) njh@debian:~$ gcloud auth list
        Credentialed Accounts
ACTIVE  ACCOUNT
        [email protected]
*       [email protected]

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

(base) njh@debian:~$ gsutil ls gs://malariagen

AccessDeniedException: 403 [email protected] does not have storage.objects.list access to the Google Cloud Storage bucket.

[gbggrant]

okay - it must be something about the way we have it configured.

[gbggrant]

@hardingnj can you give it another try - we've just updated permissions here. No worries if you don't get to this until tomorrow.

[hardingnj]

Still a 403 unfortunately...

[gbggrant]

Dang. Using the [email protected] I assume? (the gmail hasn't been added to the malariagen group).

[hardingnj]

Dang. Using the [email protected] I assume? (the gmail hasn't been added to the malariagen group).

Unfortunately so!

[gbggrant]

Hi again @hardingnj - we've updated read permissions on that bucket (gs://malariagen) again. Can you try to download data again and see if it works?

[hardingnj]

That's done it. Thanks!

[gbggrant]

Great! Sorry for the problems.

[hardingnj]

Results look good. A more detailed PR is referenced below in vector-ops

Chromosome plots of the h12 summary statistic.

The phase 2 plots of the GSTE locus (top) compared to the new pipeline (bottom)

[alimanfoo]

Hi all, just to confirm that the results from @hardingnj are excellent. Previous analyses replicate extremely well with the haplotypes from the new pipeline. I think this gives us all the confidence we need to go ahead with the new pipeline. 🍾

[hardingnj]

Great- excited to see this going so well- looking forward to analysing the whole cohort!

[alimanfoo]

Reopening this issue to record results from a second round of validation against the pipeline with genome region scatter and ligation.

From @jessicaway on 1 April:

The phasing outputs for the 167 sample validation set are available at gs://malariagen/Phasing/validation/Ag1000Phase2_BurkinaFaso/ (these were run with the genome region scatter and ligation).

I've rerun the validation analysis against these new results and everything looks great. Here's the results of H12 selection scans, comparing the new pipeline ("dev-release-2") against the Ag1000G phase 2 haplotypes:

Happy to sign off on the new pipeline implementation, sorry it took so long!

[alimanfoo]

xref https://github.com/malariagen/vector-ops/pull/1402