How to correctly handle individual exceptions in Future.sequence?

user_2159158 0
Viewed 361

Suppose I have an application that downloads files from a server and uploads them to another server.

The downloading files do not necessarily exist on the server.

So, when the file does not exist, I want to skip it, not to proceed to upload it and make sure that only existing files are downloaded/uploaded. (i.e. no single downloading exception should stop all other downloading and uploading processes)

The Future should be either successful with no results(Unit) or failure with exception along with the failed path.

What is the standard way to handle this situation?

def downloadFile(path: String): Future[DownloadFile]
def uploadFile(file: DownloadFile): Future[Unit]


Future.sequence(
  paths.map { path =>
    for {
      downloadedFile <- downloadFile(path)
      _ <- uploadFile(downloadedFile)
    } yield Unit
  }
)
scala
2 Answers

Consider

  val listOfFutures = List(
    Future(1),
    Future(throw new RuntimeException("path/foo")),
    Future(2),
    Future(throw new RuntimeException("path/bar")),
  )

  Future.traverse(listOfFutures)(_.transform {
    case Success(v) => Try(Some(v))
    case Failure(e) => Try(None)
  }).map(_.flatten) andThen { case v => println(v) }

which outputs

Success(List(1, 2))

Note Future.sequence is a simpler version of Future.traverse.

Applying the comment, consider inverting flattening like so

Future.traverse(listOfFutures)(_.transform {
    case Success(v) => Try(None)
    case Failure(e) => Try(Some(e.getMessage))
  }).map { results =>
    if (results.flatten.nonEmpty) throw new RuntimeException(s"Bad paths: ${results.flatten.mkString(",")}")
    else ()
  } andThen { case v => println(v) }

which outputs

Failure(java.lang.RuntimeException: Bad paths: path/foo,path/bar)
user_52050220

You can use Either

Future.sequence(
    Seq("path1", "path2").map { path =>
      (for {
        downloadedFile <- downloadFile(path)
        _ <- uploadFile(downloadedFile)
      } yield Right(Unit))
        .recover { case ex: Exception => Left(ex) }
    }
  )

This returns List(Right(object scala.Unit), Left(java.lang.Exception: Bad path))

Or the same with Option:

Future.sequence(
    Seq("path1", "path2").map { path =>
      (for {
        downloadedFile <- downloadFile(path)
        _ <- uploadFile(downloadedFile)
      } yield None)
        .recover { case ex: Exception => Some(ex) }
    }
  )

Returns List(None, Some(java.lang.Exception: Bad path))

The List can be filtered then.

user_27509660
Related