Set-S3Acl Powershell cmdlet無法正常工作20-30k對象

我的客戶端遇到了問題，即他們意外地將1300萬個對象（文件）複製到了具有錯誤權限的S3存儲桶中。他們已經要求我的團隊修復它。我們必須使用正確的ACL更新S3存儲桶中的每一千三百萬個文件。我們正在使用下面的PowerShell腳本來解決它。但是，如果腳本在具有超過20-30k個對象的文件夾上運行，則無法設置ACL。 [它通過循環迭代，但它不會設置權限發佈20-30k對象，也不例外]Set-S3Acl Powershell cmdlet無法正常工作20-30k對象

我懷疑請求可能會被限制。有沒有人遇到過這樣的問題。請幫助我如何繼續。

我在尋找下列問題的答案： 1.如果API調用被攔截@ 20-30k對象，我該如何修改腳本以克服它。 2.對於數百萬個對象來說，修改AWS資源的腳本編制方式（如爲S3對象設置ACL權限）的最佳做法是什麼

（我不是在尋找「BucketPolicy」方法，因爲我們有用一個腳本來做到這一點，並將ACL應用到每個S3對象）

Param (
    [Parameter(Position=0,Mandatory=$true)] 
    [string]$profile, 
    [Parameter(Position=1,Mandatory=$true)] 
    [string]$switchToAccount, 
    [Parameter(Position=2,Mandatory=$true)] 
    [string]$roleName, 
    [Parameter(Position=3,Mandatory=$true)] 
    [string]$keyPrefix 
) 

#Set base AWS credentials 
Set-AWSCredentials -ProfileName $profile 
Set-DefaultAWSRegion -Region $region 

#Get and set MFA device ARN 
$userName = (Get-IAMUser).UserName 
$mfaArn = "arn:aws:iam::xxxxxxxxx:mfa/" + "$userName" 

#Configure CAA roles 
$roleArn = "arn:aws:iam::" + "$switchToAccount" + ":role/" + "$roleName" 
$roleSessionName = "xxxxxxxxxxxx" 

#Prompt for MFA token and perform CAA request 
$tokenCode = Read-Host -Prompt "Enter MFA token for $accountNumber" 
$switchRole = Use-STSRole -RoleSessionName $roleSessionName -RoleArn $roleArn -TokenCode $tokenCode -SerialNumber $mfaArn 

#Set new role for CAA 
Set-AWSCredentials -Credential $switchRole.Credentials 

#Declare access level for S3 Object ACL grantees 
$FULL_CONTROL = [Amazon.S3.S3Permission]::FULL_CONTROL 
$grants = @(); 

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxxx 
$grantee1 = New-Object -TypeName Amazon.S3.Model.S3Grantee 
$grantee1.EmailAddress = "xxxxxxxxxxxxxxxxxxx" 

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxx 
$grantee2 = New-Object -TypeName Amazon.S3.Model.S3Grantee 
$grantee2.EmailAddress = "xxxxxxxxxxxxxxxxxxx" 

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxx 
$grantee3 = New-Object -TypeName Amazon.S3.Model.S3Grantee 
$grantee3.EmailAddress = "xxxxxxxxxxxxxxxxxxxxx" 

#Create grant and add to grant list 
$grant1 = New-Object -TypeName Amazon.S3.Model.S3Grant 
$grant1.Grantee = $grantee1 
$grant1.Permission = $FULL_CONTROL 
$grants += $grant1 

#Create grant and add to grant list 
$grant2 = New-Object -TypeName Amazon.S3.Model.S3Grant 
$grant2.Grantee = $grantee2 
$grant2.Permission = $FULL_CONTROL 
$grants += $grant2 

#Create grant and add to grant list 
$grant3 = New-Object -TypeName Amazon.S3.Model.S3Grant 
$grant3.Grantee = $grantee3 
$grant3.Permission = $FULL_CONTROL 
$grants += $grant3 



#Set bucket name for S3 objects 
$bucketName = "xxxxxxxxxxxxxxxxxxxxxxxxx" 
#Get all S3 Objects in specified bucket 
$s3Objects = Get-S3Object -BucketName $bucketName -KeyPrefix $keyPrefix 




#Count for progress bar 
$totalObjects = $s3Objects.length 
$i = 1 
$fail_count = 0 
$current_count = 0 
$file_path = "C:\Users\Administrator\Desktop\Failed_Objects_new\" + $keyPrefix.Replace("/","_") + ".txt" 
$file_path_retry = "C:\Users\Administrator\Desktop\Failed_Objects_new_retry\" + $keyPrefix.Replace("/","_") + ".txt" 

     new-item $file_path -ItemType file 
     new-item $file_path_retry -ItemType file 

"Total Object Count:" + $totalObjects + "`n" | Out-File $file_path -Append 


foreach($s3Object in $s3Objects){ 

    $owner = $s3Object.owner.id 
    $s3Object.name | Write-Output 
    $current_count++ 
    #Extracts Key for each S3 object in bucket 
    $key = $s3Object.Key 

    #Logging 
    Write-Host "Setting $bucketName | $key | $grants" 

    # Pick objects that were modified on or before July 15th 

    try { 

     if (($s3Object.LastModified.month -lt 7)) { 
      Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner 
      $owner | Write-Host 
     } 

     elseif(($s3Object.LastModified.month -eq 7) -and ($s3Object.LastModified.day -le 15)) { 

      Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner 
      $owner | Write-Host 
     } 

    }catch{ 

     "Failed $bucketName | $key | $grants" | out-file $file_path -Append 
     $key | Out-File $file_path_retry -Append 
     $fail_count++ 
    } 

    Write-Host "progress: " $current_count "/" $totalObjects 
    #Update progress bar 
    $percentComplete = $i/$totalObjects 
    Write-Progress -Activity "Setting S3 Object ACL's" -Status "$i% complete" -PercentComplete $percentComplete 
    $i++ 

} 



"`n`n Total Fail Count:" + $fail_count | Out-File $file_path -Append

來源

2017-07-24 Manjunath Rao

如何用正確的權限再次複製所有文件！這可能是最快的解決方案..因爲複製將迭代發生 – Deepak

我確實推薦它。目前管理層不想去客戶端，而是希望我們通過腳本修復它。 –

步驟來調試問題：

確保如果節流問題。在for循環中;打破10k個對象後，看看是否一切正常。
另外，把try語句裏面的print語句放在if和else裏面，以確定它是否到達那裏;何時失敗？

來源

2017-07-24 17:36:04 Deepak

該腳本工作到20-30k對象。它會停止授予權限後的權限。我在try塊內有一個命令「$ owner | Write-Host」來測試它是否到達。它正在達到TRY塊。但是，它不授予權限。 –

Set-S3Acl Powershell cmdlet無法正常工作20-30k對象

回答

相關問題