I am trying to read 15 MB CSV files from the s3 bucket using the following code.
ListObjectsResponse object1 = await S3Client.ListObjectsAsync("mybucket");
foreach (S3Object s3File in object1.S3Objects)
{
var response = await S3Client.GetObjectAsync("mybucket", s3File.Key);
var request = new GetObjectRequest()
{
BucketName = "mybucket",
Key = files.Key
};
using (var res = S3Client.GetObjectAsync(request))
{
StreamReader sReader = new StreamReader(res.Result.ResponseStream); //Time out here
string? line = sReader.ReadLine();
}
}
The above code is working fine with smaller files. but if the file has more 100K+ lines, the lambda function is timing out in the aws console. I want to process all the lines from s3 bucket file.
Could please let me know the best approach to implement this?