I have a Beam application that runs successfully locally with directrunner and gives me all the log information i have in my code on my local console. But when I tried running it in the google cloud dataflow environment, i only see those log information on my local console but they don't show up on the Google Cloud Console for dataflow job, neither in their StackDriver logging page.
Here is what i did to run my code for dataflow runner from my local console:
mvn compile exec:java -Dexec.mainClass= ... \
-Dexec.args=... " \
-Pdataflow-runner
and all logs come back on this local console. But when I went to the Google Cloud Console on my browser and search for the logs for my dataflow job, I don't see those logs in my code LOGGER.info(msg)anywhere. I only see logs related to dataflow pipeline.
So I wonder whether my Beam application run separately in such a way that the part of the main class not inside the pipeline is run locally and only the part of pipeline code will be sent to the google cloud for execution there. And hence those logs info not inside the pipeline code will not be available on the Google Cloud Dataflow logs.