ramblingcookiemonster / invoke-parallel

Speed up PowerShell with simplified multithreading

Invoke-Parallel

This function will take in a script or scriptblock, and run it against specified objects(s) in parallel. It uses runspaces, as there are many situations where jobs or PSRemoting are not appropriate.

Instructions

# Download and unblock the file(s).
# Dot source the file.
    . "\\Path\To\Invoke-Parallel.ps1"


# Get help for the function
    Get-Help Invoke-Parallel -Full


# Use Invoke-Parallel with variables in your session

    $Number = 2
    1..10 | Invoke-Parallel -ImportVariables -ScriptBlock { $Number * $_ }


# Use the $Using Syntax, currently restricted to PowerShell v3 and later

    $Path = 'C:\temp\'

    'Server1', 'Server2' | Invoke-Parallel {

        #Create a log file for this server, use the root $Path
        $ThisPath = Join-Path $Using:Path "$_.log"
        "Doing something with $_" | Out-File -FilePath $ThisPath -Force

    }


# Import modules found in the current session

    #From https://psremoteregistry.codeplex.com/releases/view/65928
    Import-Module PSRemoteRegistry

    $ServerList | Invoke-Parallel -ImportModules -ScriptBlock {

        $Key = 'Software\Microsoft\Windows\CurrentVersion\Policies\System'
        Get-RegValue -ComputerName $_ -Hive LocalMachine -Key $Key |
            Select ComputerName, Value, Data

    }


# Want to time out items that take too long?

    1..5 | Invoke-Parallel -RunspaceTimeout 2 -ScriptBlock {

        "Starting $_"
        Start-Sleep -Seconds $_
        "If you see this, we didn't timeout $_"
    }


# Is one thread freezing up when you time it out, and preventing your scripting from moving on?

    $ServerList | Invoke-Parallel -RunspaceTimeout 10 -NoCloseOnTimeout -ScriptBlock {

            Get-WmiObject -Class Win32_OperatingSystem -ComputerName $_ | select -Property PSComputerName, Caption, Version

    }

Some outdated notes and details are available on the TechNet Galleries submission.

Help!

Would love contributors, suggestions, feedback, and other help! Split this out at the suggestion of @vors to help enable collaboration.

Notes

• Credit to Boe Prox for the base code this uses. Boe has a number of references on runspaces, including the presentation and materials here and the excellent PoshRSJob module; check them out!

invoke-parallel's Issues

This repo no longer works in AppVeyor - "Import-Module Pester" fails on PS Version 2

The title says it all pretty much - you haven't changed your code, but something about AppVeyor must have changed. I just went through your blog post following the instructions and this was failing. I implemented the fix for it in Pull Request #41 so you can use that...

or else at least try to re-build your current solution and see if it still works... because it isn't working for those going through your blog post on how to set this up. (if you know of a work-around, posting in your blog would be sweeeet!)

Multiple $Using statements for the same variable fails

Example code:

$x = 1; 0 | Invoke-Parallel {$using:x; "$using:x"}

Example outcome:

Cause: Poor copy and paste abilities : ) Fix incoming.

index out of range, but I can not find which line throw exception

Get-RunspaceData : Index was out of range. Must be non-negative and less than 
the size of the collection.
Parameter name: index
At D:\UtilityTools\Invoke-Parallel.ps1:590 char:29
+                             Get-RunspaceData
+                             ~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], ArgumentOutOfRa 
   ngeException
    + FullyQualifiedErrorId : System.ArgumentOutOfRangeException,Get-RunspaceD 
   ata
 

Physical Memory usage

This code hangs after processing about 150 objects, increases my thread count to over a 2000 and caps my physical memory out when running. The physical memory usage maintains at 31.8 GB after I have exited my powershell process and persists through the restart of another powershell process. How would you reconfigure the code to make it faster and stop hanging?

# contains a list of 10000 files. 
$files = Get-Content C:\Users\Administrator\Desktop\files.csv

$files | Invoke-Parallel -ImportModules -runspacetimeout 20 -nocloseontimeout -ScriptBlock{ 
Get-WinEvent -FilterHashtable @{Path=$_;id=4624;data="user.name1","user.name2","user.name3","user.name4","user.name5","user.name6" } | 
    Select-Object  -Property MachineName,RecordId, TimeCreated,Id,
    @{Name="SubjectUserSid"; Expression={$_.Properties[0].Value}},
    @{Name="SubjectUserName";Expression={$_.Properties[1].Value}},
    @{Name="SubjectDomainName";Expression={$_.Properties[2].Value}},
    @{Name="SubjectLogonId";Expression={$_.Properties[3].Value}},
    @{Name="TargetUserSid";Expression={$_.Properties[4].Value}},
    @{Name="TargetUserName"; Expression={$_.Properties[5].Value}},
    @{Name="TargetDomainName";Expression={$_.Properties[6].Value}},
    @{Name="TargetLogonId";Expression={$_.Properties[7].Value}},
    @{Name="LogonType";Expression={$_.Properties[8].Value}},
    @{Name="LogonProcessName";Expression={$_.Properties[9].Value}},
    @{Name="AuthenticationPackageName";Expression={$_.Properties[10].Value}},
    @{Name="WorkstationName";Expression={$_.Properties[11].Value}},
    @{Name="LogonGuid";Expression={$_.Properties[12].Value}},
    @{Name="TransmittedServices";Expression={$_.Properties[13].Value}},
    @{Name="LmPackageName";Expression={$_.Properties[14].Value}},
    @{Name="KeyLength";Expression={$_.Properties[15].Value}},
    @{Name="ProcessId";Expression={$_.Properties[16].Value}},
    @{Name="ProcessName";Expression={$_.Properties[17].Value}},
    @{Name="IP"; Expression={$_.Properties[18].Value}},
    @{Name="IpPort";Expression={$_.Properties[19].Value}} | Export-Csv -path "C:\users\Administrator\Desktop\folder1\full_$name.csv"  } -throttle 100 

$PWD inside the script block doesn't update after changing the current directory

This causes some issues when running scripts that interact with the filesystem using a relative path, e.g. downloading a file

Invoke-Parallel skips processing some objects when throttle > 1

I am processing large objects and I would like Invoke-Parallel to push 1000 threads at a time. The job is done for only 400+ objects. But when I change the throttle to 1, all the objects are processed. Is there any MAX for throttle?

$all = Get-Content $TextFilePath
$a = Invoke-Parallel -InputObject $all -throttle 1000 -ImportVariables -EA SilentlyContinue -runspaceTimeout 30 -ScriptBlock

How do run without needing input objects?

I'm trying to hit a website untill I want to stop with 50 threads with various URLs. Right now I have to do 1..1000000 | Invoke-Parallel {} which I assume creates some enormous hashtable, enumerates it etc with none of those things needed. I understand it's needed obviously for progress bar etc but I'd rather have something like below or what is the best way to accomlish it?

while ($true) {Invoke-Parallel}

Invoke-Parallel Scriptblock [Enviornment]::GetEnvironmentVariable

It seems when using environment variables [Enviornment]::GetEnvironmentVariable("ComputerName") inside a scriptblock returns the local PC computername not the remote environment variable.

I would have expected the local environment variable not the local variable.

How does throttle work

I am a bit confused how "throttle" is supposed to work.

I have a code where I have to do web requests to various endpoints. I was hoping that using Invoke-Parallel would make it faster but no luck

Invoke-Parallel -InputObject $inputObject -ImportVariables -ImportModules -Throttle 10 { // do some web requests to endpoints contained in inputObject }
Where $inputObject is object array size >= 1000

But when I run in ISE I can see "Starting threads" going > 10. At some point ISE just freezes. I understood "Throttle" is the number of threads running parallel at one time and it will never exceed the given limit. Probably I am doing something wrong or not using some other parameters?

Out-File and multi-threading

Me again :)

Sometimes CI build agents give the following. Cause we run up to 16-20 items in parallel from different builds. Just FYI.

Out-file : The process cannot access the file 'C:\temp\log.log' because it is being used by another process.
The process cannot access the file 'C:\temp\log.log' because it is being used by another process.

it is not working to get AD group member using Get-ADGroupMember

Hello,

I am not able to run my scriptblock to fetch the ad group members after providing group name in Invoke-parallel.

getting this error
`Get-RunspaceData : Cannot convert 'System.String[]' to the type 'Microsoft.ActiveDirectory.Management.ADUser' required by parameter 'Identity'. Specified method is not supported.
At line:504 char:13

•         Get-RunspaceData -wait
  

•         ~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidArgument: (:) [Write-Error], ParameterBindingException
  • FullyQualifiedErrorId : CannotConvertArgument,Get-RunspaceData
    `

Here is the block
`
$groupmemberBlock = {

[CmdletBinding()]
param (
[Parameter(Mandatory=$false)]
[string[]]$groupName #
)

<#

if (Get-Module -ListAvailable -Name ActiveDirectory) {
Write-Host "Module exists"
} else {
Write-Host "Module does not exist"
Import-Module ActiveDirectory
}

#>
$ResultOut = @()

Write-Host "Checking Group - $groupName"
#$members = Get-ADGroupMember $Group -recursive | Select-Object *
$members = Get-ADGroupMember -Identity $groupName #| Select-Object *

$ADgrp = Get-ADGroup $groupName -Properties *

$MemNames1 = $($members.Name)-join (",`n")
$memCount = $($members.Name).count

    If($members.objectClass -eq "Group"){

    #Write-Host "Checking Inside Groups - $member"
    $memgrpnm = $members | ?{$_.ObjectClass -eq "Group"} 
    $membersGroup =  Get-ADGroupMember $memgrpnm.name 
    $membersGroup1 =$($membersGroup.Name) -join (",`n")
    $MemNames = ($MemNames1)+ -join (",`n") +$membersGroup1
    }

    Else {
    $MemNames = ($MemNames1)

    }

$MemSAM = $($members.SamAccountName) -split "," -join (",`n")

$Results = New-Object Object
$Results | Add-Member -Type NoteProperty -Name 'Group' -Value $groupName
$Results | Add-Member -Type NoteProperty -Name 'MembersCount' -Value $memCount
$Results | Add-Member -Type NoteProperty -Name 'MembersName' -Value $MemNames
$Results | Add-Member -Type NoteProperty -Name 'info' -Value $ADgrp.info
$Results | Add-Member -Type NoteProperty -Name 'ManagedBy' -Value $ADgrp.ManagedBy
$Results | Add-Member -Type NoteProperty -Name 'DistinguishedName' -Value $ADgrp.DistinguishedName

$ResultOut += $Results

$ResultOut

}

#$groups = $($grouplist.name)

$groups = "automation-apps"

$GroupMemberOut = Invoke-Parallel -InputObject $groups -ScriptBlock $groupmemberBlock -RunspaceTimeout 300 -NoCloseOnTimeout
$GroupMemberOut`

$ErrorActionPreference = "Stop" may have unintended consequences

This line:

$runspace.powershell.EndInvoke($runspace.Runspace)

Means that if your session is set to stop on errors, and any of your jobs write an error, Invoke-Parallel will error out at the point it is receiving the data.

You might want to evaluate:

try { 
$runspace.powershell.EndInvoke($runspace.Runspace) 
} catch { 
Write-Error $_ -ErrorAction Continue
}

This still writes the error but doesn't abort the pipeline; it can also be stored for later when called using something like Invoke-Parallel -ErrorVariable MyErrorList

NullReferenceException

Get-RunspaceData : Object reference not set to an instance of an object.
At D:\prism\CDaaS\UtilityTools\Invoke-Parallel.ps1:570 char:25
+                         Get-RunspaceData
+                         ~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], NullReferenceEx 
   ception
    + FullyQualifiedErrorId : System.NullReferenceException,Get-RunspaceData
 

Is it possible to add support for $using:variable inside script block?

Consider:

$variable = "foo"; 1..2 | Invoke-Parallel {"$_ : $using:variable"}

Currently it outputs errors

Crashing IDE / Powerhell console all the time

Hey guys, started using this script and hitting crashes most of the time.

Here is an example of Event log with EventID 1001. Same with PS ISE.
Any clue?

Fault bucket , type 0
Event Name: PowerShell
Response: Not available
Cab Id: 0

Problem signature:
P1: powershell.exe
P2: 10.0.10586.122
P3: System.IO.FileLoadException
P4: System.IO.FileLoadException
P5: unknown
P6: Microsoft.WSMan.Management.IGroupPolicyObject.Save
P7: unknown
P8: 
P9: 
P10: 

How can we share variable state?

Had a question from another medium, boiled down to the lack of shared state (variables).

Will look into implementing something along the lines of Boe's synchronized hash table idea, or Dave Wyatt's Lock-Object.

Get-RunspaceData error: Collection was modified; enumeration operation may not execute

I am leveraging Invoke-Parallel to process VM actions in parallel in an Azure Automation runbook:

        $AzureRMVMs | Invoke-Parallel -ScriptBlock {

            $ARMVM = $PSItem

            $VMStatus = $ARMVM | Get-AzureRmVM -Status | Select-Object Name, @{n = 'PowerState'; e = {($PSItem.Statuses | Where-Object Code -Like "*PowerState*").DisplayStatus}}

            switch ($using:VMAction) {

                'Start' {                        

                    if ($VMStatus.PowerState -ne 'VM running') {

                        Write-Output "VM $($ARMVM.Name) is not running - starting"

                        $ARMVM | Start-AzureRmVM

                    } else {

                        Write-Output "VM $($ARMVM.Name) is already running - no action triggered"

                    }

                }
                'Stop' {

                    if ($VMStatus.PowerState -eq 'VM running') {
                        
                        Write-Output "VM $($ARMVM.Name) is running - stopping"
                        
                        $ARMVM | Stop-AzureRmVM -Force
                        
                    } else {
                        
                        Write-Output "VM $($ARMVM.Name) is already stopped - no action triggered"
                        
                    }

                }
                'Default' {Write-Output "VM Action $($using:VMAction) not defined in runbook, no action taken"}

            }

        } -RunspaceTimeout 300 -NoCloseOnTimeout

This works as expected, however, the following error is generated in the error stream:
Get-RunspaceData : Collection was modified; enumeration operation may not execute.
At Invoke-Parallell 542 char:13

•         Get-RunspaceData -wait
  

•         ~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : CloseError: (:) [Write-Error], InvalidOperationException
  • FullyQualifiedErrorId : Get-RunspaceData

Any idea what might be causing this?

$launchDate never initialized

(0..1) | Invoke-Parallel -ScriptBlock { 
    $instance = $_
    (1..10) | % {
        Write-Host "Hello from $instance : $_"    
        sleep 1
    }
}

Get error:

The variable '$launchDate' cannot be retrieved because it has not been set.
At D:\dev\Invoke-Parallel\Invoke-Parallel.ps1:402 char:29
+                 $log.Date = $launchDate
+                             ~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (launchDate:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

Get-RunspaceData : ERROR: Deployment failed

Hello,
First, I love this thing! wish I found it 2 weeks ago!

Anyway, I am trying to automate building out an azure env using the azure v2 cli and it seems to blow out on commands like this:

$cmd = "az vm extension set --verbose --resource-group $rg --vm-name $name --name customScript --publisher Microsoft.Azure.Extensions --settings '{\`"fileUris\`": [ \`"$fileUris\`" ], \`"commandToExecute\`": \`"$commandToExecute\`"}'"
$cmd = Invoke-Expression $cmd
Write-Host "[$name] Executed $commandToExecute"

With this error:
Get-RunspaceData : ERROR: Deployment failed. Correlation ID: 3406f079-12d5-4eb3-a423-cef1d2a29696. Operation PutSecurityRuleOperation (1dd51017-88ce-421a-971d-ae07ef29f83b) was canceled and superseded by operation PutSecurityRuleOperation (b100457e-8062-4bcb-bcff-0778cfa80780).
At /Users/foo/Documents/Projects/myproject/bin/Invoke-Parallel.ps1:543 char:13

•         Get-RunspaceData -wait
  

•         ~~~~~~~~~~~~~~~~~~~~~~
  

• CategoryInfo : NotSpecified: (ERROR: Deployme...-0778cfa80780).:String) [Write-Error], RemoteException
• FullyQualifiedErrorId : NativeCommandError,Get-RunspaceData

Are there any clues to look for to fix this?

Import-Modules in parallel when session profile loaded?

Hello

I've noticed that as I've added modules to my $PROFILE script, the startup time to open a PS terminal has increased. PsFZF, in particular is slow and the author knows this.

I import three modules in my $PROFILE:

Import-Module PSFzf -ArgumentList 'Ctrl+t','Ctrl+r'
Import-Module posh-git
Import-Module oh-my-posh

Is there a way to use Invoke-Parallel (or something else) to load these in parallel to speed up terminal start time?

Thanks

Issue with Get-WMIObject

When I try to use Invoke-Parallel with Get-WMIObject, I get an access denied on every object.
I trimmed down the scriptblock to be just:
$scriptblock={
Get-WMIObject -Class Win32_OperatingSystem -ComputerName $_ -Credential $credential
}

I've tried with with and without $credential (simply my credentials for accessing servers via Get-Credential).

The command line used is: Invoke-Parallel -Scriptblock $scriptblock -InputObject $servers -Verbose

Might I be missing something? FWIW, a scriptblock that calls Test-Connection works fine.

Invoke-Parallel: When launched within a function, does not carry param()-variables forward

I added Invoke-Parallel.ps1 to my module and dot source it as Invoke-Parallel

However, I am struggling with some scoping issues it seems. If I make functions/cmdlets in another module that uses Invoke-Parallel and want to use bound parameters, they will not be available in the scriptblock.

Function foo {
[cmdletbinding()]
param($bar)

$foo = "test"

0 | Invoke-Parallel -ScriptBlock {
write-host $bar
} -ImportVariables -Verbose
}

$a = "test"
Foo -bar $a

Will only work if the function foo is a cmdlet/function in the same module as Invoke-Parallel. Also, $Using:bar will not yield anything.

Is there any chance that

1. it can be made to work cross-modules?
2. $using:variable can work inside functions?

I know this works in PoshRSJob, but the job i'm running seems to struggle after more than 10 elements with it for some reason, throwing all kinds of errors.

Invoke-Parallel should work with PS strict mode

Import-Module .\Invoke-Parallel.ps1 -Force
Set-StrictMode -Version latest

@(1,2,3) | Invoke-Parallel { $_ }

Parallel execution of psake

Could you please add an example to add parallelism to Psake tasks.

Convert .ps1 to .psm1

Modules is the preferred way to distribute reusable components.

Calling custom functions from inside Invoke-Parallel

I'm trying to call some custom functions inside of Invoke-Parrallel as such:

if ($xmlfile.configuration.DBAndFilesBackup.item("MSSQL").haschildnodes) {
    foreach ($backup in $xmlfile.configuration.DBAndFilesBackup.MSSQL.backup) {     

        Invoke-Parallel -ImportVariables -Throttle 2 {

            # Create a unique log file for this particular backup job
            $log="$ScriptDir\logs\corebackup\$ScriptName $($backup.Name) $DateStamp.log"

            # Begin the Backup
            BeginBackup $backup.Name $log

            # Backup the mssql database
            MSSQLBackup $backup.Name $backup.DBPath 

            # Backup the files
            FilesBackup $backup.Name $backup.FolderName $backup.FilesPath 

            # End the Backup
            EndBackup $backup.FolderName $backup.Name $log $emailTo

                }
    }
}

However when I do this I get the following error:

Get-RunspaceData : The term 'BeginBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+             Get-RunspaceData -wait
+             ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (BeginBackup:String) [Write-Error], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData

Get-RunspaceData : The term 'MSSQLBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+             Get-RunspaceData -wait
+             ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (MSSQLBackup:String) [Write-Error], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData

Get-RunspaceData : The term 'FilesBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+             Get-RunspaceData -wait
+             ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (FilesBackup:String) [Write-Error], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData

Get-RunspaceData : The term 'EndBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+             Get-RunspaceData -wait
+             ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (EndBackup:String) [Write-Error], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData

What am I doing wrong and how can I avoid this error?

As a side note - how to i avoid entering an input variable? I don't think I necessarily have anything to input if that makes sense?

Thanks
Brad

Log file locks if multiple scripts invoke-parallel

We have a script box that is running many scripts on a schedule and we run into situations where scripts that run on the same schedule using invoke-parallel will fail due to "The process cannot access the file 'C:\temp\log.log' because it is being used by another process." Can we make the log file optional with a switch? I've made the following edit to the copy we run.

[Parameter(Mandatory=$false)]
[validatescript({Test-Path (Split-Path $_ -parent)})]
[string]$LogFile,

Fast-Ping Sweep (recommendations or tricks)

Goal:
I am trying to only use PowerShell to conduct a ping sweep and I want to perform the fastest ping sweep possible.

Research:
I found a quick single line script from “https://learn-powershell.net/2011/01/31/quick-hits-ping-sweep-one-liner/” and leveraged invoke-parallel in place of foreach (%). Also, I added Where-Object {$_ -like "True"} to filter out the false and/or non-pingable IPs.

Request:
So far, I can ping sweep a /24 in about 20 to 25 seconds. I want to know if anyone else has any recommendations or tricks to cut the time down (if possible) only using PowerShell.

Reason:
I am looking into building a branch with a PowerShell based GUI that leverages invoke-parallel to ping sweep subnets and port scan live IPs to validate findings of other commercial and/or opensource networking products.

Script block:

#Very short Script block used with invoke-parallel function
#Requires change of the IP ranges to meet pinged area subnet.
#There are wildcards on each side of the word True. See bottom of the attached Fast-Ping.txt

1..254 |

Invoke-Parallel {

$IPsping = '192.168.3.'

"$IPsping$($): $(Test-Connection -count 1 -comp $IPsping$($) -quiet)" |

Where-Object {$_ -like "True"}}

fast-ping.txt

Cannot make scriptblocks as a parameters executes in parallel

If you run this line, it will take more then 3 seconds.

@({sleep 1; 1}, {sleep 1; 2}, {sleep 1; 3}) | Invoke-Parallel { return $_.Invoke() }

One of the things that can be really cool: make Invoke-Parallel work with pester tests. So I can pass describe blocks as an -InputObject

Get-RunspaceData exceptions

Hey guys,
Having sometimes the following exception, not too sure how to trace and fix it. Any ideas?

Get-RunspaceData : A transfer operation with the same source and destination already exists.
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Write-Error], TransferException
+ FullyQualifiedErrorId : TransferException,Get-RunspaceData

Exception handling question

Hello team,

Havign the folllowing code which raises and exception. The issue is that it is extremely hard to trace the error code, the line or the funtion once it was run with Invok-Parallel.

function RunAs-Parallel ($array, $scriptBlock) {
     $array | Invoke-Parallel -ImportVariables -ImportModules -Throttle $array.Count $scriptBlock  
}

$array = @(1,2,3,4,5,6,7,8,9,10)

RunAs-Parallel $array {

    $item = $_
    Write-Host $_

    if($item -eq 4) {

        haha $tmp

        throw "sdfds"
    }

}

The output would be like:

System.Management.Automation.CommandNotFoundException: The term 'haha' is not recognized as the name of a cmdl
et, function, script file, or operable program. Check the spelling of the name, or if a path was included, ver
ify that the path is correct and try again.

Another silly take:

try {
            #$array | Invoke-Parallel -ImportVariables -ImportModules -Throttle $array.Count $scriptBlock -Verbose -Quiet
            $array | Invoke-Parallel -ImportVariables -ImportModules -Throttle $array.Count $scriptBlock  

        }
        catch {

            Write-Host "Catching issue..."

            $allErrors = $_

            foreach($err in $allErrors) {

                Write-Host "..."

                $ex = $err.Exception;

                while($ex -ne $null)
                {
                    Write-Host "Error message:" -ForegroundColor green
                    Write-Host $ex 

                    Write-Host "Error InnerExceptiuon:" -ForegroundColor green
                    Write-Host $ex.

                    Write-Host "Error stacktrace:" -ForegroundColor green
                    Write-Host $ex.InnerException

                    Write-Host "Position message:" -ForegroundColor green
                    Write-Host $err.InvocationInfo
                    Write-Host $err.InvocationInfo.PositionMessage

                    Write-Host "Script name:" -ForegroundColor green
                    Write-Host $err.InvocationInfo.ScriptName

                    Write-Host "Line:" -ForegroundColor green
                    Write-Host $err.InvocationInfo.LIne

                    Write-Host "..."

                    $ex  = $ex.InnerException
                }
            }

Pretty same outcome, no clue on where the error can be.

With hundreds of functions and scripts, it is a nightmare to figure out what caused the error.

Would you suggest how to handle that with Invoke-Parallel? So that it can be possible to trace the origin of the exception to the actual function or something?

skips

Invoke-parallel Ignores wait commands

Invoke-parallel seems to ignore wait commands which are built-into the PS modules I am using in my script.

Here’s what I am doing

1. The script works against an input CSV which has multi rows and columns.

2. The script’s goal is to restore a VM from Azure Recovery Services– this restore can take 30 minutes and more per VM. I want to run the restore operations in parallel.

3. The Restore operations involves 2 main actions
   a. Restore VM from Azure Recovery Vault. This can take 30 minutes and more.
   b. The Azure Recovery Vault PS commands have the logic to wait for the restore to complete before proceeding to the next line. When I run the script without invoke-parallel, the sequence of steps is as expected - first finish 3b and then proceed to 3c.
   c. Create a new VM based on the output from (a) above.

4. Currently, the script can successfully do these operations for multiple VMs, but only serially.

Questions: With invoke-parallel, the Restore VM Operations gets kicked off in parallel but for whatever reason it doesn’t wait for the restore to finish. It immediately moves ahead to Create new VM (3c) which obviously fails since the VM hasn’t yet be restored. The logic to wait is ignored when using invoke-parallel. (3c)

Anyone know how to fix this?

Using a shared & synchronized variable to hold a counter

I need to display the number of records being processed by Invoke-Parallel in a WinForms variable. For that I need to keep a Counter variable. I tried Boe's synchronized hash table however it did not work, numbers were erratic and counter was wrong. I am able although to display an online array list of the records being processed, however the count is needed for displaying a progress bar using WinForms.

I assume this is an old request, so I wonder if you can throw a sample script to show how to achieve it.

Running Out Of Memory

When I use the module to process ~70K memory utilization increases steadily and does not go down.

Invoke-Parallel -ScriptFile "C:\Scripts\Sync-AssetData.ps1" -InputObject $AssetList -Parameter $ConnectionInfo -Throttle 40

Adding item to external collection

How we can use Invoke-Parallel to fetch ADUser data?

I want to get all the users from AD and foreach user I want to fetch multiple properties parallel.
Because ForEach process is taking time to get the details.

sample dummy code.
$users = Get-ADUser -Filter *

ForEach($user in $users){

$SamName = $user.samaccountname

$memberof = $user.memberof

}

Works also within azure automation?

Hello,

really great work! 2 questions:

1. Has anybody ever tested whether or how this function works within azure automation? Some guys used this function with azure powershell. But azure powershell is a local environment with rest connection to azure. So it's very different from running a runbook within azure automation, which means executing a runbook in a remote environment.
2. And, would it be possible to package this function as azure automation module? Modules are able to be imported into azure automation one time and can be used afterwards in many runbooks.

Cheers Greg

Error A null key is not allowed in a hash literal

Description

Not sure if this error is due to incorrect usage. But I get this error for every object Invoke-Parallel is run.

Get-RunspaceData : A null key is not allowed in a hash literal.
At line:523 char:21
'+ Get-RunspaceData
'+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Collecti...deredDictionary:OrderedDictionary) [Write-Error], RuntimeException
+ FullyQualifiedErrorId : InvalidNullKey

Here is how I am calling function:

$all = Get-Content $TextFilePath
$a = Invoke-Parallel -InputObject $all -throttle 300 -ImportVariables -runspaceTimeout 30 -ScriptBlock {

Invoke-Parallel and PowerCLI

Have you had any luck running this with PowerCLI? I'm hitting a brick wall doing so and was hoping for some suggestions on what to try. In my script block I am adding the PowerCLI snapin so that it's functions can be used but I keep getting the following:

Get-RunspaceData : An item with the same key has already been added.
At C:\capacity-planning\windows\Invoke-Parallel\Invoke-Parallel\Invoke-Parallel.ps1:568 char:21
+                     Get-RunspaceData
+                     ~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], ArgumentException
    + FullyQualifiedErrorId : System.ArgumentException,Get-RunspaceData

If I try running with the ImportModules parameter I get this:

Get-RunspaceData : You have  modified the global:DefaultVIServer and global:DefaultVIServers system variables. This is
not allowed. Please reset them to $null and reconnect to the vSphere server.
At C:\capacity-planning\windows\Invoke-Parallel\Invoke-Parallel\Invoke-Parallel.ps1:568 char:21
+                     Get-RunspaceData
+                     ~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], InvalidState
    + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidState,Get-RunspaceData

It is all very confusing to me. This post seems to make it sound easy if I were to not use this module but I'd prefer to stick with this plumbing if at all possible:

http://velemental.com/2012/03/11/multithreading-powercli/

Weird error running Invoke-Parallel

Please, observe the following script:

param([switch]$Cleanup, [Switch]$MockMutex, $N = 1)

function Cleanup()
{
    1..$N | ForEach-Object {
        Unregister-PSRepository "r$_" -ErrorAction SilentlyContinue
    }
    Remove-Item $WorkDir -ErrorAction SilentlyContinue -Recurse
}

$WorkDir = "$env:TEMP\fb22c60e-a0c5-48b3-953a-0b580c6a2f5e"
if ($Cleanup)
{
    Cleanup
}
else
{
    if ($MockMutex)
    {
        class MutexMock
        {
            WaitOne() { }
            ReleaseMutex() { }
        }
        $mtx = [MutexMock]::New()
    }
    else
    {
        $mtx = [Threading.Mutex]::New($false)
    }

    1..$N | Invoke-Parallel {
        $RepositoryName = "r$_"
        $WorkDir = $Using:WorkDir

        $null = mkdir "$WorkDir\$RepositoryName" -Force -ErrorAction SilentlyContinue

        $null = $parameter.WaitOne()
        try
        {
            if (!(Get-PSRepository -Name $RepositoryName -ErrorAction SilentlyContinue))
            {
                Write-Host "[$([AppDomain]::GetCurrentThreadId())] BEGIN[$RepositoryName]"
                Register-PSRepository $RepositoryName `
                    -SourceLocation "$WorkDir\$RepositoryName" `
                    -PublishLocation "$WorkDir\$RepositoryName" `
                    -InstallationPolicy Trusted
                Write-Host "[$([AppDomain]::GetCurrentThreadId())]      [$RepositoryName]END"
            }
        }
        finally
        {
            $parameter.ReleaseMutex()
        }
    } -Parameter $mtx
    Get-PSRepository | Where-Object { $_.Name.StartsWith('r') }
    Cleanup
}

It just tries to run Register-PSRepository in parallel the given number of times. The default is to use a mutex to actually serialize the calls to the functions Get-PSRepository and Register-PSRepository (thus killing the parallelism, of course, but this is a different story). Passing -MockMutex switch replaces a real mutex object with a mock.

Strange things happen when calling with the mock and N greater than 1:

PS C:\> .\1.ps1 -MockMutex -N 2
Get-RunspaceData : A parameter cannot be found that matches parameter name 'Provider'.
At C:\misc\Invoke-Parallel.ps1:591 char:13
+             Get-RunspaceData -wait
+             ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Write-Error], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Get-RunspaceData
 
[37616] BEGIN[r1]
[37616]      [r1]END

Name                      InstallationPolicy   SourceLocation
----                      ------------------   --------------
r1                        Trusted              C:\Users\MKHARI~1\AppData\Local\Temp\fb22c60e-a0c5-48b3-953a-0b580c6a2f5e\r1


PS C:\>

All works fine when N = 1 or when called with a real mutex object.

The only explanation that I see is that the Get/Register-PSRepository functions are not multi-thread safe and calling them in parallel screws things up in such a way that it affects Invoke-Parallel somehow.

Running Invoke-Parallel under VSO build agent stuck

Hello team,

Invoke-Parallel looks really good working well under ISE / powershell console.
However using Invoke-Parallel under VSO build agent just stuck. It might be due to progress report (trying to figure out) or other things related to non-interactive session.

Anything you might suggest to look at?

Psexec through invoke-parallel?

So I have a script that does some remote things using psexec. It looks like invoke-parallel doesn't like the psexec state. I'm assuming because it runs in a psexec process and not a powershell service and it throws a bunch og the get-runspacedata errors. It still runs really well. I would just like some way of getting some log information back to make sure everything ran smoothly. I've tried throwing in some start transcripts, but maybe I'm doing it wrong of the runspaces aren't allowing it to log correctly. Is there a way to get around the get-runspace errors when using psexec or a simpler way to get a return log for the various computers piped to the function?

Thanks for any help!
-Hunter

The variables in the LICENSE.md are not expanded

Hi,
I know that is not very valuable remarks but in the LICENSE.md variables are not expanded so the license is not correct.

The current state
Copyright (c) {{{year}}} {{{fullname}}}

$MaxQueue is 0 in End Block, PowerShell 2

MaxQueue is set to 0 in End block. Reproducible in PowerShell 2 (Native and -Version switch):

1..2 | Invoke-Parallel {"What?"} -Verbose

Result:

VERBOSE: Throttle: '20' SleepTimer '200' runSpaceTimeout '0' maxQueue '0' logFile 'C:\temp\log.log'
VERBOSE: Creating runspace pool and session states
VERBOSE: Creating empty collection to hold runspace jobs
VERBOSE: Adding 1 to collection at 1/20/2015 7:52:26 PM
VERBOSE: "1/20/2015 7:52:26 PM";"Removing:'1'";"0 minutes";"Completed";
what
VERBOSE: 0 items running - exceeded 0 limit.

First item runs, then we get stuck thinking we've reached the MaxQueue ceiling.

Variable Creep

This change fixed variable creep for me:

Line 215 on the ps1

Before change:
}).invoke()[0]

After change:
},$true).invoke()[0]

source