讀取文件，壓縮並將壓縮的輸出傳送到S3

我正在嘗試編寫一個工具，它將壓縮目錄並將壓縮輸出流式傳輸到S3，而無需先在磁盤上進行緩存。讀取文件，壓縮並將壓縮的輸出傳送到S3

package main 

import (
    "compress/gzip" 
    "io" 
    "log" 
    "os" 
    "sync" 

    "github.com/rlmcpherson/s3gof3r" 
) 

// log.Fatal() implies os.Exit(1) 
func logerror(err error) { 
    if err != nil { 
     log.Fatalf("%s\n", err) 
    } 
} 

func main() { 
    k, err := s3gof3r.EnvKeys() 
    logerror(err) 

    // Open bucket we want to write a file to 
    s3 := s3gof3r.New("", k) 
    bucket := s3.Bucket("somebucket") 

    // Open file to upload 
    files, err := os.Open("somefile") 
    logerror(err) 
    defer files.Close() 

    // open a PutWriter for S3 upload 
    s3writer, err := bucket.PutWriter("somezipfile.gz", nil, nil) 
    logerror(err) 

    // Create io pipe for passing gzip output to putwriter input 
    pipereader, pipewriter := io.Pipe() 
    defer pipereader.Close() 

    var wg sync.WaitGroup 
    wg.Add(2) 

    // Compress 
    go func() { 
     defer wg.Done() 
     defer pipewriter.Close() 

     gw := gzip.NewWriter(pipewriter) 
     defer gw.Close() 

     _, err := io.Copy(gw, files) 
     logerror(err) 
    }() 

    // Transmit 
    go func() { 
     defer wg.Done() 

     _, err := io.Copy(s3writer, pipereader) 
     logerror(err) 
    }() 

    wg.Wait() 

}

當我編譯並運行這個，我得到沒有錯誤輸出，並在S3沒有文件。添加了一堆打印的打動了我下面的輸出，如果它是有幫助的：

files: &{0xc4200d0a00} 
s3writer: &{{https <nil> somebucket.s3.amazonaws.com /somezipfile.gz false } 0xc4200d0a60 0xc420014540 20971520 [] 0 0xc42010e2a0 0 false <nil> {{} [0 0 0 0 0 0 0 0 0 0 0 0] 0} 0xc42010e300 0xc42010e360 0xc42035a740 0 97wUYO2YZPjLXqOLTma_Y1ASo.0IdeoKkif6pch60s3._J1suo9pUTCFwUj23uT.puzzDEHcV1KJPze.1EnLeoNehhBXeSpsH_.e4gXlNqBZ0HFsvyABJfHNYwUyXASx { []} 0} 
pipewriter: &{0xc42013c180} 
gzipwriter: &{{ [] 0001-01-01 00:00:00 +0000 UTC 255} 0xc420116020 -1 false <nil> 0 0 false [0 0 0 0 0 0 0 0 0 0] <nil>} 
archive: 1283 
upload: 606

幫助讚賞！

來源

2017-03-02 starseed

您可能需要關閉s3writer。另外，除了使用管道和額外的goroutines外，你不能僅僅給's3writer'作爲'gzip.NewWriter'的參數嗎？ –

爲什麼不使用亞馬遜的Go SDK？ –

確保您的SDK是最新的。另外請確保您的數據量小於5GB，否則您需要採用多上傳方式將數據放到您的存儲桶中。 – Sam

我最終得到了一些幫助，通過另一個途徑，工作代碼：|

package s3upload 

import (
    "compress/gzip" 
    "io" 
    "os" 

    "github.com/crielly/mongosnap/logger" 
    "github.com/rlmcpherson/s3gof3r" 
) 

// S3upload streams compressed output to S3 
func S3upload(toarchive, s3bucket, object string) { 
    keys, err := s3gof3r.EnvKeys() 
    logger.LogError(err) 

    // Open bucket we want to write a file to 
    s3 := s3gof3r.New("", keys) 
    bucket := s3.Bucket(s3bucket) 

    // open a PutWriter for S3 upload 
    s3writer, err := bucket.PutWriter(object, nil, nil) 
    logger.LogError(err) 
    defer s3writer.Close() 

    // Open a compressed writer to handle gzip and pass it to S3 writer 
    zipwriter := gzip.NewWriter(s3writer) 
    defer zipwriter.Close() 

    // Open files we want archived 
    file, err := os.Open(toarchive) 
    logger.LogError(err) 
    defer file.Close() 

    // Pass opened file to compression writer 
    _, err = io.Copy(zipwriter, file) 
    logger.LogError(err) 

}

來源

2017-03-02 22:05:02 starseed

讀取文件，壓縮並將壓縮的輸出傳送到S3

回答

相關問題