I have an airflow dag running in a VM, but in order to facilitate the event driven triggering I'm trying to set up cloud composer in GCP. However, I only see an option in cloud composer to install pypi packages.
I need rosbag package in order to run my bash script, is there any way to do that in cloud composer? Or it's better to either run Airflow in a VM or a container with Kubernetes?
You can add your own requirements in Cloud Composer
https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies
However knowing rosbag pretty well (I've been robotics engineer using ROS for quite some time) - this might not be super easy to work out the right set of dependencies. Airflow has > 500 dependencies overall and it is highly likely some of them might clash with the particular version of ROS.
Also ROS has its own, specific way of initialization and setting up all the environment variables, sourcing certain scripts - which you will have to emulate yourself, modify PYTHONPATH and possibly do some initialization.
I'd say your best bet will be to use DockerOperator and use ROS from a Docker image. This can be done even with GPU support if needed (been there, done that) and it will provide the right level of isolation - both Airflow and ROS are utilising Python and dependencies a lot, and this might be the simplest way.
