How to page the IAmazonS3.ListObjects method?

Question

How to page the IAmazonS3.ListObjects method?

Navigation

#1 by (1 votes)

1

The IAmazonS3.ListObjects method you can check here limits the return in 1000 items per request however does not inform if it is possible to do pagination, I have buckets with more than 10 million files and I need to go through each of them. The description of the method in Portuguese is this:

Returns some or all (at most 1000) of the objects in a bucket. You can use request parameters as a selection criteria to return a set of objects in a bucket

Is it possible to paginate this request? I did not find anything on Google or the documentation.

My job is this:

static ListObjectsResponse GetBucketObjects(string bucketName)
{
  ListObjectsRequest request = new ListObjectsRequest();  
  request.BucketName = bucketName;
  request.MaxKeys = 1000; //Mesmo que eu coloque 90000000000, o sistema considera 1000 
  ListObjectsResponse response = client.ListObjects(request);
  return response;
}

c# aws

asked by anonymous 24.04.2018 / 20:58

1 answer

Display products from various categories How to pass parameters from one Fragment to another Fragment in the same FrameLayout?

score 1 · Accepted Answer

Following documentation from SDK to JavaScript , you must know the NextMarker so that the query returns the data after a given result.

In this way, it is not possible to query for a specific page without having previously consulted the previous one.

Here is an example in JS.:

var params = {
    Bucket: 'STRING_VALUE',
    MaxKeys: 500
}

var listObjects = function (params) {
    return new Promise((resolve, reject) => {
        s3.listObjects(params, function(err, data) {
            if (err) {
                reject(err);
            } else {
                resolve(data);
            }
        })
    })
}

async function GetBucketObjects () {
    var data = null;
    do
    {
        var data = await listObjects(params)
        // faça algo com data.Contents
        params.Marker = data.NextMarker
    } while (data.IsTruncated)
}

GetBucketObjects('')

Here is an outline of the C # version (not tested)

var listResponse = default(ListObjectsResponse);
var listRequest = new ListObjectsRequest
{
    BucketName = "STRING_VALUE",
    MaxKeys = 500
};    

do
{

    listResponse = client.ListObjects(listRequest);
    // faça algo com listResponse.S3Objects
    listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);