How to read a .tar file containing parquets on S3 as dataframes in Spark?

Follow this example. I hope you are using tar.gz

AWSCredentials credentials = new BasicAWSCredentials("accessKey", "secretKey");
AWSCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(credentials);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_1).withCredentials(credentialsProvider).build();
S3Object object = s3Client.getObject("bucketname", "file.tar.gz");
S3ObjectInputStream objectContent = object.getObjectContent();

TarArchiveInputStream tarInputStream = new TarArchiveInputStream(new GZIPInputStream(objectContent));
TarArchiveEntry currentEntry;
while((currentEntry = tarInputStream.getNextTarEntry()) != null) {
    if(currentEntry.getName().equals("1/foo.bar") && currentEntry.isFile()) {
        FileOutputStream entryOs = new FileOutputStream("foo.bar");
        IOUtils.copy(tarInputStream, entryOs);
        entryOs.close();
        break;
    }
}
objectContent.abort();  // Warning at this line
tarInputStream.close(); // warning at this line

scala equivalent is

    val credentials: AWSCredentials =
      new BasicAWSCredentials("accessKey", "secretKey")
    val credentialsProvider: AWSCredentialsProvider =
      new AWSStaticCredentialsProvider(credentials)
    val s3Client: AmazonS3 = AmazonS3ClientBuilder
      .standard()
      .withRegion(Regions.US_EAST_1)
      .withCredentials(credentialsProvider)
      .build()
    val s3object: S3Object = s3Client.getObject("bucketname", "file.tar.gz")
    val objectContent: S3ObjectInputStream = s3object.getObjectContent
    val tarInputStream: TarArchiveInputStream = new TarArchiveInputStream(
      new GZIPInputStream(objectContent))
    var currentEntry: TarArchiveEntry = null
    while ((currentEntry = tarInputStream.getNextTarEntry) != null) 
if (currentEntry.getName ==("1/foo.bar") && currentEntry.isFile) {
      val entryOs: FileOutputStream = new FileOutputStream("foo.bar")
      IOUtils.copy(tarInputStream, entryOs)
      entryOs.close()
    }
    objectContent.abort()
    tarInputStream.close()
  }

Update :

since you are using only tar not gzip

so you have to read like this...

val tarInputStream = new TarArchiveInputStream(new FileInputStream(
    tarFile.getObjectContent))

In your case you are passing object as a InputStream. My suggestion is to pass it as a GzipInputstream, then read entries:

val tarInputStream = new TarArchiveInputStream(tarFile.getObjectContent)

val tarInputStream = new TarArchiveInputStream(new GZIPInputStream(tarFile))
val entry: TarArchiveEntry = readEntries(tarInputStream)
def readEntries(tarInputStream: TarArchiveInputStream): TarArchiveEntry = {
  var currentEntry = Option(tarInputStream.getNextTarEntry())
  // you can use functional approach with foldLeft, reduce or something else or while loop
  // implementation details here
}

You can find how to use TarArchiveInputStream usage here

You can use GetObjectRequest to create an S3Object

val s3FullObject: S3Object = s3client.getObject(new GetObjectRequest(s3Bucket, s3TarPath))
val tis = new TarArchiveInputStream(s3FullObject.getObjectContent)
var entry: TarArchiveEntry = tis.getNextTarEntry