Concurrently Uploading Files from Multiple Sources to Azure Storage using PowerShell

Certainly! Here are the steps to concurrently upload files from multiple source locations to Azure Storage using PowerShell:

Step 1: Set Up Environment and Parameters

• Ensure you have the Azure PowerShell module installed.
• Replace placeholders with your actual Azure Storage account name, container name, and destination folder path.

Step 2: Define Source Directories

Create an array containing the paths to your source directories that you want to upload.

# Define the storage account and container information
$storageAccountName = "yourstorageaccountname"
$containerName = "yourcontainername"
$destinationPath = "destination-folder-on-blob"

# List of source directories
$sourceDirectories = @("C:\path\to\source1", "C:\path\to\source2", "C:\path\to\source3")

# Scriptblock to upload files
$uploadScriptBlock = {
    param ($sourcePath, $destinationPath, $storageAccountName, $containerName)
    
    Connect-AzAccount

    $result = az storage blob upload-batch `
        --destination $containerName `
        --destination-path $destinationPath `
        --source $sourcePath `
        --account-name $storageAccountName

    # Output the result along with source directory
    [PSCustomObject]@{
        SourceDirectory = $sourcePath
        Result = $result
    }
}

# Create a runspace pool
$runspacePool = [runspacefactory]::CreateRunspacePool(1, [Environment]::ProcessorCount)
$runspacePool.Open()

# Array to store runspace results
$runspaceResults = @()

# Start concurrent uploads using runspaces
foreach ($sourcePath in $sourceDirectories) {
    $runspace = [powershell]::Create().AddScript($uploadScriptBlock).AddArgument($sourcePath).AddArgument($destinationPath).AddArgument($storageAccountName).AddArgument($containerName)
    $runspace.RunspacePool = $runspacePool

    $runspaceResults += [PSCustomObject]@{ Runspace = $runspace; Status = $runspace.BeginInvoke() }
}

# Wait for all runspace tasks to complete
$runspaceResults | ForEach-Object {
    $_.Runspace.EndInvoke($_.Status)
    $_.Runspace.Dispose()
}

# Close and clean up the runspace pool
$runspacePool.Close()
$runspacePool.Dispose()

# Display results
$runspaceResults | ForEach-Object {
    Write-Host "Upload status for $($_.Runspace.AddArgument[0]): $($_.Runspace.AddArgument[1].Result)"
}