Transform S3 objects into hierarchical (tree) Map in Elixir

Dyey517 0
Viewed 75

I have this task where I need to extract objects from AWS S3 using ExAws and return it to Frontend in a file-tree-like structure.

From the data returned by ExAws.S3.list_objects("bucket-name", prefix: "folder-path") |> ExAws.request!

%{
  body: %{
    contents: [
      %{key: "file1.svg"},
      %{key: "FolderA/file1.txt"},
      %{key: "FolderA/file2.txt"},
      %{key: "FolderB/Layer1/file1.csv"},
      %{key: "FolderB/Layer1/file2.csv"},
      %{key: "FolderB/Layer1/GroupA/file1.pdf"},
      %{key: "FolderB/Layer1/GroupB/file1.jpg"},
      %{key: "FolderB/Layer2/file1.png"}
    ]
  }
}

I need to transform it into something like this map.

%{
  "FolderA" => %{
    "file1.txt" => "FolderA/file1.txt",
    "file2.txt" => "FolderA/file2.txt"
  },
  "FolderB" => %{
    "Layer1" => %{
      "GroupA" => %{
        "file1.pdf" => "FolderB/Layer1/GroupA/file1.pdf"
      },
      "GroupB" => %{
        "file1.jpg" => "FolderB/Layer1/GroupB/file1.jpg"
      },
      "file1.csv" => "FolderB/Layer1/file1.csv",
      "file2.csv" => "FolderB/Layer1/file2.csv"
    },
    "Layer2" => %{"file1.png" => "FolderB/Layer2/file1.png"}
  },
  "file1.svg" => "file1.svg"
}

I found this npm package, s3-tree but it does a request per 'folder'. What I need is just pure algorithm.

And these with python: Python - How can I convert S3 folders into JSON hierarchy?, s3-tree. Translating these into elixir would help.

elixir amazon-s3
2 Answers

I am here to advertise the most underrated feature of elixir, Access behaviour, again.

Access.key/2 is your friend here.

data
|> get_in([:body, :contents, Access.all(), :key])
|> Enum.reduce(%{}, fn e, acc ->
  put_in(acc, Enum.map(String.split(e, "/"), &Access.key(&1, %{})), e)
end)

That’s it.

Aleksei Matiushkin0

Nice recursion challenge

I'm not familiar with ex_aws_s3 so it may be that there is already some functionality there, but here's my attempt:

defmodule S3Tree do
  def parse(data) do
    Enum.reduce(data.body.contents, %{}, fn %{key: full_path}, acc ->
      tree = traverse(String.split(full_path, "/"), full_path)
      merge(nil, acc, tree)
    end)
  end

  defp traverse([file], full_path), do: %{file => full_path}
  defp traverse([dir | rest], full_path), do: %{dir => traverse(rest, full_path)}

  defp merge(_key, map1, map2), do: Map.merge(map1, map2, &merge/3)
end

Usage:

data = %{
  body: %{
    contents: [
      %{key: "file1.svg"},
      %{key: "FolderA/file1.txt"},
      %{key: "FolderA/file2.txt"},
      %{key: "FolderB/Layer1/file1.csv"},
      %{key: "FolderB/Layer1/file2.csv"},
      %{key: "FolderB/Layer1/GroupA/file1.pdf"},
      %{key: "FolderB/Layer1/GroupB/file1.jpg"},
      %{key: "FolderB/Layer2/file1.png"}
    ]
  }
}

S3Tree.parse(data)

Output:

%{
  "FolderA" => %{
    "file1.txt" => "FolderA/file1.txt",
    "file2.txt" => "FolderA/file2.txt"
  },
  "FolderB" => %{
    "Layer1" => %{
      "GroupA" => %{"file1.pdf" => "FolderB/Layer1/GroupA/file1.pdf"},
      "GroupB" => %{"file1.jpg" => "FolderB/Layer1/GroupB/file1.jpg"},
      "file1.csv" => "FolderB/Layer1/file1.csv",
      "file2.csv" => "FolderB/Layer1/file2.csv"
    },
    "Layer2" => %{"file1.png" => "FolderB/Layer2/file1.png"}
  },
  "file1.svg" => "file1.svg"
}
Adam Millerchip0
Related