Parallel copy of files in NAnt

There are two main options to execute parallel copy of files, one is to user JFlower project (https://sourceforge.net/projects/jflower) to create multithread in NAnt, the other option is to copy the files using PowerShell. The JFlower option we will discard because it creates a third part dependency with no support and with no updates warranty. The PowerShell option to execute parallel copy can be implemented in different ways, here some examples:

  1. Use Copy-Item with the Invoke-Command.
$servers = Get-Content 'C:\bpi_servers.txt'

Invoke-Command -ComputerName $servers -ScriptBlock

{
    Copy-Item -Path '\\machine01\_Projectos\137\Mainline\Artifacts\Latest\XML' -Destination "E:\NetDados\teste" -Verbose

}

Requirements: WinRM, PowerShell >=3, .Net Framework 4.0 in client and server

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_remote_requirements?view=powershell-6&viewFallbackFrom=powershell-5

Note:

https://blogs.technet.microsoft.com/heyscriptingguy/2011/06/13/use-powershell-invoke-command-for-remoting

“By default, PowerShell will talk to up to 32 computers at once; if you specified more than that, it will queue them up, so that, as one computer completes, the next one in line will begin. If you have a awesome network and powerful computers, you could raise that number by specifying the -throttleLimit parameter of Invoke-Command—read the command’s help for more information.”

 

  1. User Workflow, foreach –parallel and Copy-Item together
$computers = ‘W16TP5TGT01’, ‘W16TP5TGT02’
workflow parallelcopy
{

    param ([string[]]$computername)
    foreach -parallel ($computer in $computername)
    {

        InlineScript
        {

            $s = New-PSSession -ComputerName $using:computer
            Copy-Item -Path \\machine01\_Projectos\137\Mainline\Artifacts\Latest\XML -Destination E:\Netdados\teste -ToSession $s
            Remove-PSSession -Session $s
        }
    }
}

 

Requirements: WinRM, PowerShell >=3, .Net Framework 4.0 in client and server

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_remote_requirements?view=powershell-6&viewFallbackFrom=powershell-5

Note: Good for cenários were existes the need to execute a workflow that survives reboots or when execution order matters. Implemented with WWF 4.X.

 

  1. Use Start-Job together with robocopy
$startDate = (Get-Date)
Write-Host "Start Date:" + $startDate
$serverlist = @('SSMWFVNPBCOM01','SDMWFVNPBCOM01','SSMWFVINTCOM01','SDMWFVINTCOM01','SSMWFVNPBCOM01','SDMWFVNPBCOM01','SSMWFVINTCOM01','SDMWFVINTCOM01')
$jobs = foreach ($serverName in $serverlist)
{
    Write-Host "Starting copy job for $serverName"
    Start-Job -Name "Copyjob $serverName" -ArgumentList $serverName -ScriptBlock 
{ param($serverName) if (Test-Connection -ComputerName $serverName -Count 1 -Quiet)
{ robocopy "c:\temp" "\\$serverName\netdados" teste.zip /R:0 }}
}
write-host "Waiting for job(s)"
$jobs | Wait-Job
#write-host "Job(s) output(s)"
#$jobs | Receive-Job
$endDate = (Get-Date)
Write-Host "End Date:" + $endDate
$timeSpan = NEW-TIMESPAN –Start $startDate –End $endDate 
Write-Host "This script took $($timeSpan.Milliseconds) milliseconds to run."

Requirements: PowerShell >=2 no cliente e SMB protocol between cliente and server

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_remote_requirements?view=powershell-6&viewFallbackFrom=powershell-5

Note: Did not find a throttle specification for start-job, we need to control throttling with code.

https://exchange12rocks.org/2015/05/24/how-to-limit-a-number-of-powershell-jobs-running-simultaneously/

Discussion:

There are different ways to implement parallel copy of files in Nant.

In this specific scenario, we will choose to user Start-Job together with robocopy because our infrastructure already has PowerShell 2 or higher in every machine and SMB protocol is already configured in firewall and routing rules.

Leave a Reply

Your email address will not be published. Required fields are marked *