Django Object is Not Serializable CommandError using Dumpdata with Natural Keys

Viewed 207

I am trying to use 'natural keys' for serialization (docs) in a "manage.py dumpdata" command:

python manage.py dumpdata --natural-primary --natural-foreign --indent 4 --format json --verbosity 1 > tests\test_fixtures\test_db2.json

and I am getting the following error when I use --natural-foreign on other apps that use the Project or Task model (which they all must by design):

CommandError: Unable to serialize database: Object of type Project is not JSON serializable
Exception ignored in: <generator object cursor_iter at 0x000001EF62481B48>
Traceback (most recent call last):
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\db\models\sql\compiler.py", line 1586, in cursor_iter
    cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.

If I just dumpdata from this, the 'projects' app, it works, but other apps are built with entities related to Project or Task and there the --natural-foreign option fails.

The problem occurs when a model (say Question) calls for a natural_key from Task, which includes a for a natural_key from Project.

If I use the Pycharm Python Console to access querysets of Projects or Tasks ('q' here), this works:

serializers.serialize('json', q, indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)

But if 'w' is a list of Question objects from another app that have a Task foreign key I get this error:

    Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\__init__.py", line 128, in serialize
    s.serialize(queryset, **options)
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\base.py", line 115, in serialize
    self.end_object(obj)
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\json.py", line 53, in end_object
    json.dump(self.get_dump_object(obj), self.stream, **self.json_kwargs)
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\__init__.py", line 179, in dump
    for chunk in iterable:
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 438, in _iterencode
    o = _default(o)
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\site-packages\django\core\serializers\json.py", line 104, in default
    return super().default(o)
  File "C:\Users\Andrew\anaconda3\envs\Acejet_development\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Project is not JSON serializable

The models are:

# projects.models.py
class BaseModelWithHistory(models.Model):
    history = HistoricalRecords(inherit=True)
    natural_key_fields = ('id',)  # default

    class Meta:
        abstract = True

    def natural_key(self):
        fieldlist = [getattr(self, fieldname) for fieldname in self.natural_key_fields]
        return tuple(fieldlist)
 #  natural_key.dependencies = ['projects.Project', 'projects.Task']  # serialize these first.

class Project(BaseModelWithHistory):
    """
    'Projects' group Tasks.
    """
    project_name = models.CharField(max_length=200, default="development_project")
    project_short_description = models.CharField(
        max_length=500,
        default="This is the default text.")
    target_group = models.ManyToManyField(Group, blank=True)
    objects = ProjectDiscreteManager()

    natural_key_fields = ('project_name',)

class Task(BaseModelWithHistory):
    number = models.PositiveIntegerField(default=0)
    name = models.CharField(max_length=200, default='new task')
    project = models.ForeignKey(Project, on_delete=models.CASCADE)
    target_group = models.ManyToManyField(Group, blank=True)
    app_label = models.CharField(max_length=80, choices=app_choices(), null=True, blank=True)

    objects = discrete_manager_factory('project')
    natural_key_fields = ('project', 'name', 'number')

Things that I wouldn't expect to cause this problem but I could certainly be wrong:

  • The ProjectDiscreteManager and others created by discrete_manager_factory() behave exactly as the default manager (models.Manager()), unless its called with a request from an identified user, in which case it adds a filter to see if that user is in the Group.
  • All models define the natural_keys tuple because the parent class defines it as ('id',); most models overwrite this with more representative fields.
  • With the natural_key.dependencies list set for all models to prioritize Project and then Task, I get a 'can't resolve dependencies' error for every other model. I think this ticket relates, but am not sure how to track down whether this fix is already in the Django 3.0.6 I'm using and I should just straighten up & fly right, or if my Han Solo 'this should wooork' will one day soon be rewarded. [Update: I worked out it is coming in Django 3.1.1, but I'm not sure that it is going to fix the "can't resolve dependencies" error I've created for myself.]
0 Answers
Related