ramblingcookiemonster / invoke-parallel Goto Github PK
View Code? Open in Web Editor NEWSpeed up PowerShell with simplified multithreading
License: MIT License
Speed up PowerShell with simplified multithreading
License: MIT License
Hey guys, started using this script and hitting crashes most of the time.
Here is an example of Event log with EventID 1001. Same with PS ISE.
Any clue?
Fault bucket , type 0
Event Name: PowerShell
Response: Not available
Cab Id: 0
Problem signature:
P1: powershell.exe
P2: 10.0.10586.122
P3: System.IO.FileLoadException
P4: System.IO.FileLoadException
P5: unknown
P6: Microsoft.WSMan.Management.IGroupPolicyObject.Save
P7: unknown
P8:
P9:
P10:
Hello team,
Invoke-Parallel looks really good working well under ISE / powershell console.
However using Invoke-Parallel under VSO build agent just stuck. It might be due to progress report (trying to figure out) or other things related to non-interactive session.
Anything you might suggest to look at?
Modules is the preferred way to distribute reusable components.
This line:
$runspace.powershell.EndInvoke($runspace.Runspace)
Means that if your session is set to stop on errors, and any of your jobs write an error, Invoke-Parallel will error out at the point it is receiving the data.
You might want to evaluate:
try {
$runspace.powershell.EndInvoke($runspace.Runspace)
} catch {
Write-Error $_ -ErrorAction Continue
}
This still writes the error but doesn't abort the pipeline; it can also be stored for later when called using something like Invoke-Parallel -ErrorVariable MyErrorList
Get-RunspaceData : Object reference not set to an instance of an object.
At D:\prism\CDaaS\UtilityTools\Invoke-Parallel.ps1:570 char:25
+ Get-RunspaceData
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], NullReferenceEx
ception
+ FullyQualifiedErrorId : System.NullReferenceException,Get-RunspaceData
I need to display the number of records being processed by Invoke-Parallel in a WinForms variable. For that I need to keep a Counter variable. I tried Boe's synchronized hash table however it did not work, numbers were erratic and counter was wrong. I am able although to display an online array list of the records being processed, however the count is needed for displaying a progress bar using WinForms.
I assume this is an old request, so I wonder if you can throw a sample script to show how to achieve it.
MaxQueue is set to 0 in End block. Reproducible in PowerShell 2 (Native and -Version switch):
1..2 | Invoke-Parallel {"What?"} -Verbose
Result:
VERBOSE: Throttle: '20' SleepTimer '200' runSpaceTimeout '0' maxQueue '0' logFile 'C:\temp\log.log'
VERBOSE: Creating runspace pool and session states
VERBOSE: Creating empty collection to hold runspace jobs
VERBOSE: Adding 1 to collection at 1/20/2015 7:52:26 PM
VERBOSE: "1/20/2015 7:52:26 PM";"Removing:'1'";"0 minutes";"Completed";
what
VERBOSE: 0 items running - exceeded 0 limit.
First item runs, then we get stuck thinking we've reached the MaxQueue ceiling.
Not sure if this error is due to incorrect usage. But I get this error for every object Invoke-Parallel is run.
Get-RunspaceData : A null key is not allowed in a hash literal.
At line:523 char:21
'+ Get-RunspaceData
'+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Collecti...deredDictionary:OrderedDictionary) [Write-Error], RuntimeException
+ FullyQualifiedErrorId : InvalidNullKey
$all = Get-Content $TextFilePath
$a = Invoke-Parallel -InputObject $all -throttle 300 -ImportVariables -runspaceTimeout 30 -ScriptBlock {
I want to get all the users from AD and foreach user I want to fetch multiple properties parallel.
Because ForEach process is taking time to get the details.
sample dummy code.
$users = Get-ADUser -Filter *
ForEach($user in $users){
$SamName = $user.samaccountname
$memberof = $user.memberof
}
Hello,
First, I love this thing! wish I found it 2 weeks ago!
Anyway, I am trying to automate building out an azure env using the azure v2 cli and it seems to blow out on commands like this:
$cmd = "az vm extension set --verbose --resource-group $rg --vm-name $name --name customScript --publisher Microsoft.Azure.Extensions --settings '{\`"fileUris\`": [ \`"$fileUris\`" ], \`"commandToExecute\`": \`"$commandToExecute\`"}'"
$cmd = Invoke-Expression $cmd
Write-Host "[$name] Executed $commandToExecute"
With this error:
Get-RunspaceData : ERROR: Deployment failed. Correlation ID: 3406f079-12d5-4eb3-a423-cef1d2a29696. Operation PutSecurityRuleOperation (1dd51017-88ce-421a-971d-ae07ef29f83b) was canceled and superseded by operation PutSecurityRuleOperation (b100457e-8062-4bcb-bcff-0778cfa80780).
At /Users/foo/Documents/Projects/myproject/bin/Invoke-Parallel.ps1:543 char:13
Get-RunspaceData -wait
~~~~~~~~~~~~~~~~~~~~~~
Are there any clues to look for to fix this?
(0..1) | Invoke-Parallel -ScriptBlock {
$instance = $_
(1..10) | % {
Write-Host "Hello from $instance : $_"
sleep 1
}
}
Get error:
The variable '$launchDate' cannot be retrieved because it has not been set.
At D:\dev\Invoke-Parallel\Invoke-Parallel.ps1:402 char:29
+ $log.Date = $launchDate
+ ~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (launchDate:String) [], RuntimeException
+ FullyQualifiedErrorId : VariableIsUndefined
Hello
I've noticed that as I've added modules to my $PROFILE script, the startup time to open a PS terminal has increased. PsFZF, in particular is slow and the author knows this.
I import three modules in my $PROFILE:
Import-Module PSFzf -ArgumentList 'Ctrl+t','Ctrl+r'
Import-Module posh-git
Import-Module oh-my-posh
Is there a way to use Invoke-Parallel (or something else) to load these in parallel to speed up terminal start time?
Thanks
Consider:
$variable = "foo"; 1..2 | Invoke-Parallel {"$_ : $using:variable"}
Currently it outputs errors
Hi,
I know that is not very valuable remarks but in the LICENSE.md variables are not expanded so the license is not correct.
The current state
Copyright (c) {{{year}}} {{{fullname}}}
Please, observe the following script:
param([switch]$Cleanup, [Switch]$MockMutex, $N = 1)
function Cleanup()
{
1..$N | ForEach-Object {
Unregister-PSRepository "r$_" -ErrorAction SilentlyContinue
}
Remove-Item $WorkDir -ErrorAction SilentlyContinue -Recurse
}
$WorkDir = "$env:TEMP\fb22c60e-a0c5-48b3-953a-0b580c6a2f5e"
if ($Cleanup)
{
Cleanup
}
else
{
if ($MockMutex)
{
class MutexMock
{
WaitOne() { }
ReleaseMutex() { }
}
$mtx = [MutexMock]::New()
}
else
{
$mtx = [Threading.Mutex]::New($false)
}
1..$N | Invoke-Parallel {
$RepositoryName = "r$_"
$WorkDir = $Using:WorkDir
$null = mkdir "$WorkDir\$RepositoryName" -Force -ErrorAction SilentlyContinue
$null = $parameter.WaitOne()
try
{
if (!(Get-PSRepository -Name $RepositoryName -ErrorAction SilentlyContinue))
{
Write-Host "[$([AppDomain]::GetCurrentThreadId())] BEGIN[$RepositoryName]"
Register-PSRepository $RepositoryName `
-SourceLocation "$WorkDir\$RepositoryName" `
-PublishLocation "$WorkDir\$RepositoryName" `
-InstallationPolicy Trusted
Write-Host "[$([AppDomain]::GetCurrentThreadId())] [$RepositoryName]END"
}
}
finally
{
$parameter.ReleaseMutex()
}
} -Parameter $mtx
Get-PSRepository | Where-Object { $_.Name.StartsWith('r') }
Cleanup
}
It just tries to run Register-PSRepository
in parallel the given number of times. The default is to use a mutex to actually serialize the calls to the functions Get-PSRepository
and Register-PSRepository
(thus killing the parallelism, of course, but this is a different story). Passing -MockMutex
switch replaces a real mutex object with a mock.
Strange things happen when calling with the mock and N greater than 1:
PS C:\> .\1.ps1 -MockMutex -N 2
Get-RunspaceData : A parameter cannot be found that matches parameter name 'Provider'.
At C:\misc\Invoke-Parallel.ps1:591 char:13
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Write-Error], ParameterBindingException
+ FullyQualifiedErrorId : NamedParameterNotFound,Get-RunspaceData
[37616] BEGIN[r1]
[37616] [r1]END
Name InstallationPolicy SourceLocation
---- ------------------ --------------
r1 Trusted C:\Users\MKHARI~1\AppData\Local\Temp\fb22c60e-a0c5-48b3-953a-0b580c6a2f5e\r1
PS C:\>
All works fine when N = 1
or when called with a real mutex object.
The only explanation that I see is that the Get/Register-PSRepository
functions are not multi-thread safe and calling them in parallel screws things up in such a way that it affects Invoke-Parallel
somehow.
Hello,
I am not able to run my scriptblock to fetch the ad group members after providing group name in Invoke-parallel.
getting this error
`Get-RunspaceData : Cannot convert 'System.String[]' to the type 'Microsoft.ActiveDirectory.Management.ADUser' required by parameter 'Identity'. Specified method is not supported.
At line:504 char:13
Get-RunspaceData -wait
~~~~~~~~~~~~~~~~~~~~~~
Here is the block
`
$groupmemberBlock = {
[CmdletBinding()]
param (
[Parameter(Mandatory=$false)]
[string[]]$groupName #
)
<#
if (Get-Module -ListAvailable -Name ActiveDirectory) {
Write-Host "Module exists"
} else {
Write-Host "Module does not exist"
Import-Module ActiveDirectory
}
#>
$ResultOut = @()
Write-Host "Checking Group - $groupName"
#$members = Get-ADGroupMember $Group -recursive | Select-Object *
$members = Get-ADGroupMember -Identity $groupName #| Select-Object *
$ADgrp = Get-ADGroup $groupName -Properties *
If($members.objectClass -eq "Group"){
#Write-Host "Checking Inside Groups - $member"
$memgrpnm = $members | ?{$_.ObjectClass -eq "Group"}
$membersGroup = Get-ADGroupMember $memgrpnm.name
$membersGroup1 =$($membersGroup.Name) -join (",`n")
$MemNames = ($MemNames1)+ -join (",`n") +$membersGroup1
}
Else {
$MemNames = ($MemNames1)
}
$Results = New-Object Object
$Results | Add-Member -Type NoteProperty -Name 'Group' -Value $groupName
$Results | Add-Member -Type NoteProperty -Name 'MembersCount' -Value $memCount
$Results | Add-Member -Type NoteProperty -Name 'MembersName' -Value $MemNames
$Results | Add-Member -Type NoteProperty -Name 'info' -Value $ADgrp.info
$Results | Add-Member -Type NoteProperty -Name 'ManagedBy' -Value $ADgrp.ManagedBy
$Results | Add-Member -Type NoteProperty -Name 'DistinguishedName' -Value $ADgrp.DistinguishedName
$ResultOut += $Results
$ResultOut
}
#$groups = $($grouplist.name)
$groups = "automation-apps"
$GroupMemberOut = Invoke-Parallel -InputObject $groups -ScriptBlock $groupmemberBlock -RunspaceTimeout 300 -NoCloseOnTimeout
$GroupMemberOut`
I'm trying to call some custom functions inside of Invoke-Parrallel as such:
if ($xmlfile.configuration.DBAndFilesBackup.item("MSSQL").haschildnodes) {
foreach ($backup in $xmlfile.configuration.DBAndFilesBackup.MSSQL.backup) {
Invoke-Parallel -ImportVariables -Throttle 2 {
# Create a unique log file for this particular backup job
$log="$ScriptDir\logs\corebackup\$ScriptName $($backup.Name) $DateStamp.log"
# Begin the Backup
BeginBackup $backup.Name $log
# Backup the mssql database
MSSQLBackup $backup.Name $backup.DBPath
# Backup the files
FilesBackup $backup.Name $backup.FolderName $backup.FilesPath
# End the Backup
EndBackup $backup.FolderName $backup.Name $log $emailTo
}
}
}
However when I do this I get the following error:
Get-RunspaceData : The term 'BeginBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (BeginBackup:String) [Write-Error], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData
Get-RunspaceData : The term 'MSSQLBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (MSSQLBackup:String) [Write-Error], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData
Get-RunspaceData : The term 'FilesBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (FilesBackup:String) [Write-Error], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData
Get-RunspaceData : The term 'EndBackup' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\scripts\includes\invoke-parallel.ps1:591 char:13
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (EndBackup:String) [Write-Error], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException,Get-RunspaceData
What am I doing wrong and how can I avoid this error?
As a side note - how to i avoid entering an input variable? I don't think I necessarily have anything to input if that makes sense?
Thanks
Brad
It seems when using environment variables [Enviornment]::GetEnvironmentVariable("ComputerName") inside a scriptblock returns the local PC computername not the remote environment variable.
I would have expected the local environment variable not the local variable.
Me again :)
Sometimes CI build agents give the following. Cause we run up to 16-20 items in parallel from different builds. Just FYI.
Out-file : The process cannot access the file 'C:\temp\log.log' because it is being used by another process.
The process cannot access the file 'C:\temp\log.log' because it is being used by another process.
I'm trying to hit a website untill I want to stop with 50 threads with various URLs. Right now I have to do 1..1000000 | Invoke-Parallel {} which I assume creates some enormous hashtable, enumerates it etc with none of those things needed. I understand it's needed obviously for progress bar etc but I'd rather have something like below or what is the best way to accomlish it?
while ($true) {Invoke-Parallel}
Have you had any luck running this with PowerCLI? I'm hitting a brick wall doing so and was hoping for some suggestions on what to try. In my script block I am adding the PowerCLI snapin so that it's functions can be used but I keep getting the following:
Get-RunspaceData : An item with the same key has already been added.
At C:\capacity-planning\windows\Invoke-Parallel\Invoke-Parallel\Invoke-Parallel.ps1:568 char:21
+ Get-RunspaceData
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Get-RunspaceData
If I try running with the ImportModules
parameter I get this:
Get-RunspaceData : You have modified the global:DefaultVIServer and global:DefaultVIServers system variables. This is
not allowed. Please reset them to $null and reconnect to the vSphere server.
At C:\capacity-planning\windows\Invoke-Parallel\Invoke-Parallel\Invoke-Parallel.ps1:568 char:21
+ Get-RunspaceData
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], InvalidState
+ FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidState,Get-RunspaceData
It is all very confusing to me. This post seems to make it sound easy if I were to not use this module but I'd prefer to stick with this plumbing if at all possible:
I am a bit confused how "throttle" is supposed to work.
I have a code where I have to do web requests to various endpoints. I was hoping that using Invoke-Parallel would make it faster but no luck
Invoke-Parallel -InputObject $inputObject -ImportVariables -ImportModules -Throttle 10 { // do some web requests to endpoints contained in inputObject }
Where $inputObject is object array size >= 1000
But when I run in ISE I can see "Starting threads" going > 10. At some point ISE just freezes. I understood "Throttle" is the number of threads running parallel at one time and it will never exceed the given limit. Probably I am doing something wrong or not using some other parameters?
When I try to use Invoke-Parallel with Get-WMIObject, I get an access denied on every object.
I trimmed down the scriptblock to be just:
$scriptblock={
Get-WMIObject -Class Win32_OperatingSystem -ComputerName $_ -Credential $credential
}
I've tried with with and without $credential (simply my credentials for accessing servers via Get-Credential).
The command line used is: Invoke-Parallel -Scriptblock $scriptblock -InputObject $servers -Verbose
Might I be missing something? FWIW, a scriptblock that calls Test-Connection works fine.
The title says it all pretty much - you haven't changed your code, but something about AppVeyor must have changed. I just went through your blog post following the instructions and this was failing. I implemented the fix for it in Pull Request #41 so you can use that...
or else at least try to re-build your current solution and see if it still works... because it isn't working for those going through your blog post on how to set this up. (if you know of a work-around, posting in your blog would be sweeeet!)
I am processing large objects and I would like Invoke-Parallel to push 1000 threads at a time. The job is done for only 400+ objects. But when I change the throttle to 1, all the objects are processed. Is there any MAX for throttle?
$all = Get-Content $TextFilePath
$a = Invoke-Parallel -InputObject $all -throttle 1000 -ImportVariables -EA SilentlyContinue -runspaceTimeout 30 -ScriptBlock
This code hangs after processing about 150 objects, increases my thread count to over a 2000 and caps my physical memory out when running. The physical memory usage maintains at 31.8 GB after I have exited my powershell process and persists through the restart of another powershell process. How would you reconfigure the code to make it faster and stop hanging?
# contains a list of 10000 files.
$files = Get-Content C:\Users\Administrator\Desktop\files.csv
$files | Invoke-Parallel -ImportModules -runspacetimeout 20 -nocloseontimeout -ScriptBlock{
Get-WinEvent -FilterHashtable @{Path=$_;id=4624;data="user.name1","user.name2","user.name3","user.name4","user.name5","user.name6" } |
Select-Object -Property MachineName,RecordId, TimeCreated,Id,
@{Name="SubjectUserSid"; Expression={$_.Properties[0].Value}},
@{Name="SubjectUserName";Expression={$_.Properties[1].Value}},
@{Name="SubjectDomainName";Expression={$_.Properties[2].Value}},
@{Name="SubjectLogonId";Expression={$_.Properties[3].Value}},
@{Name="TargetUserSid";Expression={$_.Properties[4].Value}},
@{Name="TargetUserName"; Expression={$_.Properties[5].Value}},
@{Name="TargetDomainName";Expression={$_.Properties[6].Value}},
@{Name="TargetLogonId";Expression={$_.Properties[7].Value}},
@{Name="LogonType";Expression={$_.Properties[8].Value}},
@{Name="LogonProcessName";Expression={$_.Properties[9].Value}},
@{Name="AuthenticationPackageName";Expression={$_.Properties[10].Value}},
@{Name="WorkstationName";Expression={$_.Properties[11].Value}},
@{Name="LogonGuid";Expression={$_.Properties[12].Value}},
@{Name="TransmittedServices";Expression={$_.Properties[13].Value}},
@{Name="LmPackageName";Expression={$_.Properties[14].Value}},
@{Name="KeyLength";Expression={$_.Properties[15].Value}},
@{Name="ProcessId";Expression={$_.Properties[16].Value}},
@{Name="ProcessName";Expression={$_.Properties[17].Value}},
@{Name="IP"; Expression={$_.Properties[18].Value}},
@{Name="IpPort";Expression={$_.Properties[19].Value}} | Export-Csv -path "C:\users\Administrator\Desktop\folder1\full_$name.csv" } -throttle 100
I added Invoke-Parallel.ps1 to my module and dot source it as Invoke-Parallel
However, I am struggling with some scoping issues it seems. If I make functions/cmdlets in another module that uses Invoke-Parallel and want to use bound parameters, they will not be available in the scriptblock.
Function foo {
[cmdletbinding()]
param($bar)
$foo = "test"
0 | Invoke-Parallel -ScriptBlock {
write-host $bar
} -ImportVariables -Verbose
}
$a = "test"
Foo -bar $a
Will only work if the function foo is a cmdlet/function in the same module as Invoke-Parallel. Also, $Using:bar will not yield anything.
Is there any chance that
I know this works in PoshRSJob, but the job i'm running seems to struggle after more than 10 elements with it for some reason, throwing all kinds of errors.
When I use the module to process ~70K memory utilization increases steadily and does not go down.
Invoke-Parallel -ScriptFile "C:\Scripts\Sync-AssetData.ps1" -InputObject $AssetList -Parameter $ConnectionInfo -Throttle 40
Could you please add an example to add parallelism to Psake tasks.
Hello,
really great work! 2 questions:
Cheers Greg
Get-RunspaceData : Index was out of range. Must be non-negative and less than
the size of the collection.
Parameter name: index
At D:\UtilityTools\Invoke-Parallel.ps1:590 char:29
+ Get-RunspaceData
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], ArgumentOutOfRa
ngeException
+ FullyQualifiedErrorId : System.ArgumentOutOfRangeException,Get-RunspaceD
ata
If you run this line, it will take more then 3 seconds.
@({sleep 1; 1}, {sleep 1; 2}, {sleep 1; 3}) | Invoke-Parallel { return $_.Invoke() }
One of the things that can be really cool: make Invoke-Parallel
work with pester tests. So I can pass describe blocks as an -InputObject
I am leveraging Invoke-Parallel to process VM actions in parallel in an Azure Automation runbook:
$AzureRMVMs | Invoke-Parallel -ScriptBlock {
$ARMVM = $PSItem
$VMStatus = $ARMVM | Get-AzureRmVM -Status | Select-Object Name, @{n = 'PowerState'; e = {($PSItem.Statuses | Where-Object Code -Like "*PowerState*").DisplayStatus}}
switch ($using:VMAction) {
'Start' {
if ($VMStatus.PowerState -ne 'VM running') {
Write-Output "VM $($ARMVM.Name) is not running - starting"
$ARMVM | Start-AzureRmVM
} else {
Write-Output "VM $($ARMVM.Name) is already running - no action triggered"
}
}
'Stop' {
if ($VMStatus.PowerState -eq 'VM running') {
Write-Output "VM $($ARMVM.Name) is running - stopping"
$ARMVM | Stop-AzureRmVM -Force
} else {
Write-Output "VM $($ARMVM.Name) is already stopped - no action triggered"
}
}
'Default' {Write-Output "VM Action $($using:VMAction) not defined in runbook, no action taken"}
}
} -RunspaceTimeout 300 -NoCloseOnTimeout
This works as expected, however, the following error is generated in the error stream:
Get-RunspaceData : Collection was modified; enumeration operation may not execute.
At Invoke-Parallell 542 char:13
Get-RunspaceData -wait
~~~~~~~~~~~~~~~~~~~~~~
Any idea what might be causing this?
Had a question from another medium, boiled down to the lack of shared state (variables).
Will look into implementing something along the lines of Boe's synchronized hash table idea, or Dave Wyatt's Lock-Object.
So I have a script that does some remote things using psexec. It looks like invoke-parallel doesn't like the psexec state. I'm assuming because it runs in a psexec process and not a powershell service and it throws a bunch og the get-runspacedata errors. It still runs really well. I would just like some way of getting some log information back to make sure everything ran smoothly. I've tried throwing in some start transcripts, but maybe I'm doing it wrong of the runspaces aren't allowing it to log correctly. Is there a way to get around the get-runspace errors when using psexec or a simpler way to get a return log for the various computers piped to the function?
Thanks for any help!
-Hunter
Invoke-parallel seems to ignore wait commands which are built-into the PS modules I am using in my script.
Here’s what I am doing
The script works against an input CSV which has multi rows and columns.
The script’s goal is to restore a VM from Azure Recovery Services– this restore can take 30 minutes and more per VM. I want to run the restore operations in parallel.
The Restore operations involves 2 main actions
a. Restore VM from Azure Recovery Vault. This can take 30 minutes and more.
b. The Azure Recovery Vault PS commands have the logic to wait for the restore to complete before proceeding to the next line. When I run the script without invoke-parallel, the sequence of steps is as expected - first finish 3b and then proceed to 3c.
c. Create a new VM based on the output from (a) above.
Currently, the script can successfully do these operations for multiple VMs, but only serially.
Questions: With invoke-parallel, the Restore VM Operations gets kicked off in parallel but for whatever reason it doesn’t wait for the restore to finish. It immediately moves ahead to Create new VM (3c) which obviously fails since the VM hasn’t yet be restored. The logic to wait is ignored when using invoke-parallel. (3c)
Anyone know how to fix this?
We have a script box that is running many scripts on a schedule and we run into situations where scripts that run on the same schedule using invoke-parallel will fail due to "The process cannot access the file 'C:\temp\log.log' because it is being used by another process." Can we make the log file optional with a switch? I've made the following edit to the copy we run.
[Parameter(Mandatory=$false)]
[validatescript({Test-Path (Split-Path $_ -parent)})]
[string]$LogFile,
Goal:
I am trying to only use PowerShell to conduct a ping sweep and I want to perform the fastest ping sweep possible.
Research:
I found a quick single line script from “https://learn-powershell.net/2011/01/31/quick-hits-ping-sweep-one-liner/” and leveraged invoke-parallel in place of foreach (%). Also, I added Where-Object {$_ -like "True"} to filter out the false and/or non-pingable IPs.
Request:
So far, I can ping sweep a /24 in about 20 to 25 seconds. I want to know if anyone else has any recommendations or tricks to cut the time down (if possible) only using PowerShell.
Reason:
I am looking into building a branch with a PowerShell based GUI that leverages invoke-parallel to ping sweep subnets and port scan live IPs to validate findings of other commercial and/or opensource networking products.
Script block:
#Very short Script block used with invoke-parallel function
#Requires change of the IP ranges to meet pinged area subnet.
#There are wildcards on each side of the word True. See bottom of the attached Fast-Ping.txt
1..254 |
Invoke-Parallel {
$IPsping = '192.168.3.'
"$IPsping$($): $(Test-Connection -count 1 -comp $IPsping$($) -quiet)" |
Where-Object {$_ -like "True"}}
Hello team,
Havign the folllowing code which raises and exception. The issue is that it is extremely hard to trace the error code, the line or the funtion once it was run with Invok-Parallel.
function RunAs-Parallel ($array, $scriptBlock) {
$array | Invoke-Parallel -ImportVariables -ImportModules -Throttle $array.Count $scriptBlock
}
$array = @(1,2,3,4,5,6,7,8,9,10)
RunAs-Parallel $array {
$item = $_
Write-Host $_
if($item -eq 4) {
haha $tmp
throw "sdfds"
}
}
The output would be like:
System.Management.Automation.CommandNotFoundException: The term 'haha' is not recognized as the name of a cmdl
et, function, script file, or operable program. Check the spelling of the name, or if a path was included, ver
ify that the path is correct and try again.
Another silly take:
try {
#$array | Invoke-Parallel -ImportVariables -ImportModules -Throttle $array.Count $scriptBlock -Verbose -Quiet
$array | Invoke-Parallel -ImportVariables -ImportModules -Throttle $array.Count $scriptBlock
}
catch {
Write-Host "Catching issue..."
$allErrors = $_
foreach($err in $allErrors) {
Write-Host "..."
$ex = $err.Exception;
while($ex -ne $null)
{
Write-Host "Error message:" -ForegroundColor green
Write-Host $ex
Write-Host "Error InnerExceptiuon:" -ForegroundColor green
Write-Host $ex.
Write-Host "Error stacktrace:" -ForegroundColor green
Write-Host $ex.InnerException
Write-Host "Position message:" -ForegroundColor green
Write-Host $err.InvocationInfo
Write-Host $err.InvocationInfo.PositionMessage
Write-Host "Script name:" -ForegroundColor green
Write-Host $err.InvocationInfo.ScriptName
Write-Host "Line:" -ForegroundColor green
Write-Host $err.InvocationInfo.LIne
Write-Host "..."
$ex = $ex.InnerException
}
}
Pretty same outcome, no clue on where the error can be.
With hundreds of functions and scripts, it is a nightmare to figure out what caused the error.
Would you suggest how to handle that with Invoke-Parallel? So that it can be possible to trace the origin of the exception to the actual function or something?
Hey guys,
Having sometimes the following exception, not too sure how to trace and fix it. Any ideas?
Get-RunspaceData : A transfer operation with the same source and destination already exists.
+ Get-RunspaceData -wait
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Write-Error], TransferException
+ FullyQualifiedErrorId : TransferException,Get-RunspaceData
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.