Dagster use dagster's custom loggers in standard python functions called from solids

user_10785101 0
Viewed 535

I can configure a custom logger (say, a file logger) which I can successfully use from within a solid from context.log.info (for example). How can I use that same logger from within a standard Python function / class ?

I am using the standard colored_console_logger so that I can directly see in the console what is happening. The idea is to swap it (or use alongside it) with another (custom) logger.

Reproducible example: test_logging.py

from dagster import solid, pipeline, execute_pipeline, Field, ModeDefinition
from dagster.loggers import colored_console_logger

from random_func import random_func


@solid
def test_logs(context):
    context.log.info("Hello, world!")
    random_func()


@pipeline(mode_defs=[
    ModeDefinition(logger_defs={"console_logger": colored_console_logger})
])
def test_pipeline():
    test_logs()


if __name__ == "__main__":
    execute_pipeline(test_pipeline,
                     run_config={
                         'loggers': {
                             'console_logger': {
                                 'config': {
                                     'log_level': 'DEBUG',
                                     'name': 'console_logger',
                                 }
                             }
                         }
                     })

random_func.py

import logging

lgr = logging.getLogger('console_logger')


def random_func():
    lgr.info('in random func')
    print('\nhi\n')
dagster pipeline logging python
1 Answers

Dagster doesn't provide any global way to get access to objects in the solid context—you'll need to have your function accept a parameter and pass the logger to it from your solid.

import logging

lgr = logging.getLogger('console_logger')


def random_func(logger=lgr):
    logger.info('in random func')
    print('\nhi\n')

@solid
def test_logs(context):
    context.log.info("Hello, world!")
    random_func(logger=context.log)
user_4914230
Related