I'm using airflow 1.10.0 on Python 3.5 and encounter this error about encoding error with logging.
The operator uses default setting of output_encoding which is already utf-8.
task_compile = BashOperator(
task_id='task_compile',
retries=1,
retry_delay=timedelta(minutes=5),
bash_command='/root/docker/tools/compile.sh',
dag=dag
)
task_compile.set_downstream(task_last)
The shell script pops a docker container and runs composer install, I tested with another simple composer install task and nothing fails, the error is only with a certain set of dependencies. As shown in the trace stack the module reponsible for the exception is file_task_handler.py when it emits the line to be logged into the log file.
[2018-09-19 20:42:18,708] {bash_operator.py:111} INFO - Package operations: 134 installs, 0 updates, 0 removals
[2018-09-19 20:42:18,790] {bash_operator.py:111} INFO - - Installing ocramius/package-versions (1.3.0): Downloading (100%)
[2018-09-19 20:42:18,850] {bash_operator.py:111} INFO - - Installing symfony/flex (v1.1.1): Downloading (100%)
[2018-09-19 20:42:18,897] {bash_operator.py:111} INFO -
[2018-09-19 20:42:18,898] {logging_mixin.py:95} WARNING - --- Logging error ---
[2018-09-19 16:12:51,554] {logging_mixin.py:95} WARNING - --- Logging error ---
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - File "/usr/lib/python3.5/logging/__init__.py", line 983, in emit
stream.write(msg)
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3b6' in position 81: ordinal not in range(128)
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - Call stack:
[2018-09-19 16:12:51,557] {logging_mixin.py:95} WARNING - File "/usr/local/bin/airflow", line 32, in <module>
args.func(args)
[2018-09-19 16:12:51,557] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
[2018-09-19 16:12:51,557] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 498, in run
_run(args, dag, ti)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 402, in _run
pool=args.pool,
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1633, in _run_raw_task
result = task_copy.execute(context=context)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/operators/bash_operator.py", line 110, in execute
self.log.info(line)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/lib/python3.5/logging/__init__.py", line 1280, in info
self._log(INFO, msg, args, **kwargs)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/lib/python3.5/logging/__init__.py", line 1416, in _log
self.handle(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/lib/python3.5/logging/__init__.py", line 1426, in handle
self.callHandlers(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/lib/python3.5/logging/__init__.py", line 1488, in callHandlers
hdlr.handle(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/lib/python3.5/logging/__init__.py", line 856, in handle
self.emit(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/file_task_handler.py", line 61, in emit
self.handler.emit(record)