First, Code for storing data in google cloud platform bigQuery tables after generating simple data. Imported Apache-Beam library and used it. Runner used Google Cloud Platform Dataflow.
Here code.
from apache_beam.options.pipeline_options import PipelineOptions
import apache_beam as beam
pipeline_options = PipelineOptions(
project='project-id',
runner='runner',
temp_location='bucket-location'
)
def pardo_dofn_methods(test=None):
import apache_beam as beam
class testFunction(beam.DoFn):
def process(self, element):
result = element.split(',')
testing = {'test_column': result[0], 'test_column2': result[1], 'test_column3': result[2],
'test_column4': result[3]}
return [testing]
def finish(self):
print('finish')
with beam.Pipeline(options=pipeline_options) as pipeline:
results = (
pipeline
| 'Generating data' >> beam.Create([
'test1,test2,test3,test4'
'test5,test6,test7,test8'
])
| beam.ParDo(testFunction())
| beam.io.WriteToBigQuery(
'project-id:bigQuery-dataset.table-name',
schema='test_column:STRING, test_column2:STRING, test_column3:STRING, test_column4:STRING',
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE
)
)
pardo_dofn_methods()
It works well when run it. However, there are two warnings:
BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead.
kms_key=self.kms_key))
I don't know why there is a warning. Thank you.