[BUG]: AzureFileCopyV5: Incorrectly returns "Storage account not found" error for recently created Storage account. about azure-pipelines-tasks

[ybadragon]

I've sent an email with the issue number 19566 in the header. I has a screenshot of the task itself and a debug log output from the pipeline with our subscription, tenant information redacted. It's a very simple thing to set up and replicate.

[v-mohithgc]

Hi @mm2709, is there any fix do you like to purpose here?

[mm2709]

Hi @mm2709, is there any fix do you like to purpose here?

Yes. I have the following options to fix the issue.

Option 1: I have already mentioned it in the description that using Get-AzStorageAccount cmdlet instead of Get-AzResource should fix the issue. All we have to do is replace

$azureStorageAccountResourceDetails = (Get-AzResource -ErrorAction Stop) | Where-Object { ($_.ResourceType -eq $ARMStorageAccountResourceType) -and ($_.Name -eq $storageAccountName)}
with

$azureStorageAccountResourceDetails = (Get-AzStorageAccount -ErrorAction Stop) | Where-Object {($_.StorageAccountName -eq $storageAccountName)}

Option 2: We can supply the ResourceGroupName (keep it as optional for backward compatibility) as input parameter from the pipeline task. Then we can skip the call Get-AzureStorageAccountResourceGroupName (in below screenshot) if the ResourceGroupName is not null or empty.

[pdtit]

having the exact same issue since a few days, but not constant. 1 out of 5 pipeline runs still get through. Had the same with azurefilecopy@4 lately, and started integrating a 5min wait task before creating the storage account and running azcopy. To no avail. The storage account and container is getting created fine, but the task doesn't see it...

[ybadragon]

We recently started having this issue very frequently now. It started for us a couple days ago but would fix itself after about 5 minutes, today it is taking much longer to fix itself. This is occuring even on storage accounts that are brand new and have never been deleted/re-created

[wjdavis5]

I'm also running into this on the windows-2022 agent in ADO. I create a storage account and a few steps later in the pipeline try to copy a file to it and it fails.

[rkistinger-carvana]

I'm also running into this on the windows-2022 agent in ADO. I create a storage account and a few steps later in the pipeline try to copy a file to it and it fails.

I am experiencing the same issue with this agent.

[robdlee]

Also happening to me too

[mm2709]

Hi @mm2709, is there any fix do you like to purpose here?

Yes. I have the following options to fix the issue.

Option 1: I have already mentioned it in the description that using Get-AzStorageAccount cmdlet instead of Get-AzResource should fix the issue. All we have to do is replace

$azureStorageAccountResourceDetails = (Get-AzResource -ErrorAction Stop) | Where-Object { ($_.ResourceType -eq $ARMStorageAccountResourceType) -and ($_.Name -eq $storageAccountName)} with

$azureStorageAccountResourceDetails = (Get-AzStorageAccount -ErrorAction Stop) | Where-Object {($_.StorageAccountName -eq $storageAccountName)}

Option 2: We can supply the ResourceGroupName (keep it as optional for backward compatibility) as input parameter from the pipeline task. Then we can skip the call Get-AzureStorageAccountResourceGroupName (in below screenshot) if the ResourceGroupName is not null or empty.

@v-mohithgc Option 1 is a straight-forward fix. Can you please prioritize and assign this bug?

[v-mohithgc]

Hi all, I have created the PR for the proposed changes #19588
I will notify and try to get it approved from relevant owners.
Just to be sure that the changes are serving the purpose, if possible, can anyone follow the below simple steps to validate real pipeline scenario?

Local task test:
Have code changes ready (checkout to the branch in that PR), navigate to task root path ie C:...\azure-pipelines-tasks
Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run "npm i"

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\AzureFileCopyV5

New task version will be uploaded to your org, please validate the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

[v-mohithgc]

looks to be a breaking change found while validating.

[mm2709]

looks to be a breaking change found while validating.

Looking at your error logs, it seems the code is failing even before hitting the updated line of code. May be you are not supplying the mandatory input parameters.

[ybadragon]

This is still occurring very frequently on new storage account deployments, is there any update on the fix suggested @v-mohithgc ?

[v-mohithgc]

This is still occurring very frequently on new storage account deployments, is there any update on the fix suggested @v-mohithgc ?

I have notified the relevant task owners; team is working on it.

[v-mohithgc]

looks to be a breaking change found while validating.

Looking at your error logs, it seems the code is failing even before hitting the updated line of code. May be you are not supplying the mandatory input parameters.

Hi, the pre-configured PR checks are failing with the same error, and I don't have access to modify/control any input parameter related to PR checks. In order to merge the PR, all those checks should succeed. PR: #19588

Let me see what I can do here.
Thanks

[v-mohithgc]

Hi, can anyone try to enable the debug logs (system.debug = true) and share us the recent failure logs related to this issue at "[email protected]". and also let us know the type of service connection been used.
Thanks

[v-mohithgc]

Hi all, can anyone please confirm if there is any recent occurrence on this issue? if so please share us the complete debug logs, details of the task configuration and type of service connection used, if possible, send all these info to [email protected], for team to proceed with further analysis.
Thanks

[ybadragon]

yes we had a pipeline fail this week due to this issue. Let me see if I can get the logs for that.

[garun-kumar]

looks to be a breaking change found while validating.

Looking at your error logs, it seems the code is failing even before hitting the updated line of code. May be you are not supplying the mandatory input parameters.

@mm2709, The error in the Logs point to the line (line number 13) where we replace Get-AzResource with Get-AzStorageAccount.

[ybadragon]

@v-mohithgc any update on this? We are still getting this error today, and it happens in pipelines where we are deploying an arbitrary number of storage accounts based on configurations in the pipeline, so run by run this fails large portion of the time.

[garun-kumar]

@v-mohithgc any update on this? We are still getting this error today, and it happens in pipelines where we are deploying an arbitrary number of storage accounts based on configurations in the pipeline, so run by run this fails large portion of the time.

@ybadragon We tried to reproduce this issue by deploying a storage account and created release by adding Azure file copy Task to see if fails. But the release was successful without any error. So, we suppose this to be a replication error due to large number of storage accounts.
So, could you please try to create release after a gap of few hours of deploying new storage accounts and confirm whether it still gives the same error?
Thanks

[ybadragon]

Yes we've already said this previously, after an arbitrary amount of time the task eventually works, however waiting for hours is not an acceptable resolution to this bug. Were you running the task in AzureDevOps? What agent were you using? As I said this is very easily replicate able. In addition looking at the previous comments @mm2709 seems to have found where the issue is, has there been an attempt to revert that change, or implement either of the fixes mentioned?

[garun-kumar]

Yes we've already said this previously, after an arbitrary amount of time the task eventually works, however waiting for hours is not an acceptable resolution to this bug. Were you running the task in AzureDevOps? What agent were you using? As I said this is very easily replicate able. In addition looking at the previous comments @mm2709 seems to have found where the issue is, has there been an attempt to revert that change, or implement either of the fixes mentioned?

I am using Microsoft hosted agent to run the task in AzureDevops.
Yes we tried to implement the changes suggested by @mm2709. But it was breaking, leading to another error.

[garun-kumar]

Yes we've already said this previously, after an arbitrary amount of time the task eventually works, however waiting for hours is not an acceptable resolution to this bug. Were you running the task in AzureDevOps? What agent were you using? As I said this is very easily replicate able. In addition looking at the previous comments @mm2709 seems to have found where the issue is, has there been an attempt to revert that change, or implement either of the fixes mentioned?

Is it feasible to introduce a delay task after you create storage account and then try to run AzureFileCopy task?

[ybadragon]

@garun-kumar not for hours, but even if we could it is an arbitrary amount of time so it still would not succeed 100% of the time and is not a fix for the bug.

[ybadragon]

FYI even after the merge from 2 weeks ago we are still seeing this error occur.

[eriktack]

Seeing this with AzureFileCopyV6 as well. The request is unable to see the container and tries to create it (even though it already exists), but our service principal do not have create container access (which it doesn't even need as the container is there and working just fine):

##[debug][Azure Call]Retrieved storage account type successfully for the storage account: XXXXXXX in resource group: YYYYYYY
##[debug]Obtained Storage Account type: Standard
##[debug][Azure Call]Getting container: CONTAINERNAME in storage account: XXXXXXX
##[debug]Container: CONTAINERNAME does not exist in storage account: XXXXXXX
##[debug]Creating container if the containerName provided does not exist
##[debug][Azure Call]Creating container: CONTAINERNAME in storage account: XXXXXXX
##[debug]Azure.RequestFailedException: This request is not authorized to perform this operation.
##[debug]RequestId:81753e3c-201e-0015-15ca-8fad6a000000
##[debug]Time:2024-04-16T06:50:04.2396633Z
##[debug]Status: 403 (This request is not authorized to perform this operation.)
##[debug]ErrorCode: AuthorizationFailure

Noticing that "sometimes" the operation works, but extremely inconsistent.