sentineldatahub / scripts Goto Github PK

Since
AUTH="--user=${USERNAME} --password=${PASSWORD}"
will expose the user and password information in the system by ps and in the command history, strongly suggested to rewrite related code by using wgetrc or other ways for downloading the files in the data hubs.
Especially for those large files, the user and password is visible through easy ps ax | grep wget which is very dangerous.

Is it possible to use this tool and only download products that aren't already downloaded?

For instance, query for the a whole week but if I already have every day apart from Saturday, it would only download data for Saturday.

Thanks,

I'm trying to downloading with dhusget data from scihub.copernicus.eu. The result is 2 correctly downloaded file and a lot of zero byte zip file.
To understand the problem I get the uiid from the dhusget output and I tried to download directly with
wget --content-disposition --continue --user=[myuser] --password=[mypassword] "https://scihub.copernicus.eu/dhus/odata/v1/Products('d11d4457-2ec2-4914-88fd-9ca9d1e37ef4')/\$value"
I get the following output:

--2020-05-01 15:34:39--  https://scihub.copernicus.eu/dhus/odata/v1/Products('d11d4457-2ec2-4914-88fd-9ca9d1e37ef4')/$value
Risoluzione di scihub.copernicus.eu (scihub.copernicus.eu)... 131.176.236.28
Connessione a scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 401 Unauthorized
Autenticazione selezionata: Basic realm="OData service"
Connessione a scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 202 Accepted
Lunghezza: 0
--2020-05-01 15:34:39--  https://scihub.copernicus.eu/dhus/odata/v1/Products('d11d4457-2ec2-4914-88fd-9ca9d1e37ef4')/$value
Riutilizzo della connessione esistente a scihub.copernicus.eu:443.
Richiesta HTTP inviata, in attesa di risposta... 202 Accepted
Lunghezza: 0
Salvataggio in: "$value"

$value                                                [ <=>                                                                                                       ]       0  --.-KB/s    in 0s

2020-05-01 15:34:40 (0,00 B/s) - "$value" salvato [0/0]

Trying in different virtual and physical machine with a script with multiple sequential wget invocation I get the same result: The first 2/5 correctly downloaded, then I get "$value" zero byte files.

I don't understand if the problem is a daily quota limit on server, a bug in wget, a bug in the OS. or other...

Hi SentinelDataHub-Team,

I tried to download Sentinel-2 data using the dhusget.sh script.
I get the data onto the virtual machine but with the following error:

Product S2B_MSIL1C_20210812T102559_N0301_R108_T32UNF_20210812T124236 successfully downloaded at 4.92 MB/s
Checksum for product S2B_MSIL1C_20210812T102559_N0301_R108_T32UNF_20210812T124236 failed

I am not able to figure out why the MD5 check fails.

Can you help?
Regards
Christian

Hi All,
Today, I used the dhusget_0.3.4.sh to download OLCI data with a AOI in Ubuntu 14.04. However, a problem always occured, I have no idea how to deal with it.
The exception was :
"outdated: unexpected operator
checksum for product failed
the end"
Here are my inputs:
url="https://scihub.copernicus.eu/s3/#/home"
usr_name="s3guest"
usr_pwd="s3guest"
mission="Sentinel-3"
instrument_name="OLCI"
sensing_date_from="2016-10-01T00:00:00.000Z"
sensing_date_to="2017-08-29T23:59:59.999Z"
limit="115.8,28.3:116.8,29.8"
product_type="OL_1_EFR_"
xmlfile='/media/sda3/RawData/OSquery-result.xml'
csvfile='/media/sda3/RawData/products-list.csv'
outfolder='/media/sda3/RawData/'
/media/sda3/Codes/dhusget_0.3.4.sh -d $url -u $usr_name -p $usr_pwd -m $mission -i $instrument_name -s $sensing_date_from -e $sensing_date_to -S $sensing_date_from -E $sensing_date_to -c $limit -T $product_type -q $xmlfile -C $csvfile -o 'all' -O $outfolder -D_

The http request done:HTTP request done: https://scihub.copernicus.eu/s3/#/home/search?q= platformname:Sentinel-3 AND instrumentshortname:OLCI AND producttype:OL_1_EFR__ AND ingestiondate:[2016-10-01T00:00:00.000Z TO 2017-08-29T23:59:59.999Z] AND beginPosition:[2016-10-01T00:00:00.000Z TO 2017-08-29T23:59:59.999Z] AND ( footprint:"Intersects(POLYGON((115.8000000000000 28.3000000000000,116.8000000000000 28.3000000000000,116.8000000000000 29.8000000000000,115.8000000000000 29.8000000000000,115.8000000000000 28.3000000000000 )))")&rows=25&start=0, was similar with the results of GUI in Scihub.

Could someone give me some suggestions?

Cheers,
zhigang

Hello

I am trying to use dhusget.sh for downloading Sentinel 5 P product. Specifying the URL as: 'https://s5phub.copernicus.eu/dhus/#/home' allows me to authenticate with guest user details but gives me "Checksum for product failed".

Looking at previous issues answers, when I remove '#/home' from the URL the authentication fails.

I would really appreciate some help.

Cheers!
Harshil

Hi there,

Thank you for the great script!
I have been using it for sometime and always had the same issues:

"OSquery-result.xml 100%[================================================================================================================>] 6.58K --.-KB/s in 0s

2018-06-26 11:05:55 (1.58 GB/s) - ‘OSquery-result.xml’ saved [6734/6734]

 1	Displaying 1 results. Request done in 0 seconds.

 1	S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354
 1	https://scihub.copernicus.eu/dhus/odata/v1/Products('a3590915-979d-4083-a103-1aa077c9066b')

Downloading manifest S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354 from link https://scihub.copernicus.eu/dhus/odata/v1/Products('a3590915-979d-4083-a103-1aa077c9066b')/Nodes('S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354.SAFE')/Nodes('manifest.safe')/$value
Manifest S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354 successfully downloaded at
1: 8: 1: [[: not found
Downloading product S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354 from link https://scihub.copernicus.eu/dhus/odata/v1/Products('a3590915-979d-4083-a103-1aa077c9066b')/$value
Product S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354 successfully downloaded at
1: 11: [: 250C1E5AD68BA2FF86B49EC6B8979B06: unexpected operator
Checksum for product S2A_MSIL1C_20180402T065621_N0206_R063_T41UNA_20180402T092354 failed"

1. 1: 8: 1: [[: not found
2. 1: 11: [: _________: unexpected operator
3. Checksum fails.

My request is
./dhusget_0.3.5.sh -d https://scihub.copernicus.eu/dhus -u _____ -p ______-m Sentinel-2 -i MSI -c 59.8161,55.0061:73.072,50.764 -T S2MSI1C -F 'cloudcoverpercentage:[0 TO 2]' -s 2018-4-02T06:00:00.000Z -e 2018-4-02T12:00:00.000Z -o all

Please, help me fix these.