Apache Beam : FlatMap vs Map?

user_8419892 0
Viewed 13580

I want to understand in which scenario that I should use FlatMap or Map. The documentation did not seem clear to me.

I still do not understand in which scenario I should use the transformation of FlatMap or Map.

Could someone give me an example so I can understand their difference?

I understand the difference of FlatMap vs Map in Spark, and however not sure if there any similarity?

apache-beam google-cloud-dataflow
3 Answers

Let me show you one example 

import apache_beam as beam

def categorize_explode(text):
  result = text.split(':')
  category = result[0]
  elements = result[1].split(',')
  return list(map(lambda x: (category, x), elements))

with beam.Pipeline() as pipeline:
  things = (
      pipeline
      | 'Categories and Elements' >> beam.Create(["Vehicles:Car,Jeep,Truck,BUS,AIRPLANE","FOOD:Burger,SANDWICH,ICECREAM,APPLE"])
      | 'Explode' >> beam.FlatMap(categorize_explode)
      | beam.Map(print)
  )

As you can see categorize_explode function splits the strings into categories and corresponding elements and returns iterator like [('Vehicles','Car'),('Vehicles','Jeep'),...]

FlatMap takes each element in this iterator and treats each element as a separate element in PCollection.

So the result would be:

('Vehicles', 'Car')
('Vehicles', 'Jeep')
('Vehicles', 'Truck')
('Vehicles', 'BUS')
('Vehicles', 'AIRPLANE')
('FOOD', 'Burger')
('FOOD', 'SANDWICH')
('FOOD', 'ICECREAM')
('FOOD', 'APPLE')

While Map performs one to one mapping. i.e. this iterator [('Vehicles','Car'),('Vehicles','Jeep'),...] would be returned as it is.

So the result would be for Map:

[('Vehicles', 'Car'), ('Vehicles', 'Jeep'), ('Vehicles', 'Truck'), ('Vehicles', 'BUS'), ('Vehicles', 'AIRPLANE')]
[('FOOD', 'Burger'), ('FOOD', 'SANDWICH'), ('FOOD', 'ICECREAM'), ('FOOD', 'APPLE')]

The approach I have used is somewhat similar to spark explode transform.

Hope this helps!!!

user_58449460

In simplest word,

Map transformation is "one to one" mapping on each element of list/collection. Ex -

{"Amar", "Akabar", "Anthony"} -> {"Mr.Amar", "Mr.Akabar", "Mr.Anthony"}

FlatMap transformation is usually on collection like "list of list", and this collection gets flattened to single list and transformation/mapping is applied on each element of "list of list"/collection.

FlatMap transformation Ex -

{ {"Amar", "Akabar"},  "Anthony"} -> {"Mr.Amar", "Mr.Akabar", "Mr.Anthony"}

This concept remains same across programming language and across platform.

Hope it helps.

user_18339790
Related