How to use Datomic partitions?

Admin 1
Viewed 696

I want to use Datomic partitions to improve the scalability of my app.

First, I first created a partition in a transaction :

{:db/id "communities"
   :db/ident :communities}
 [:db/add :db.part/db :db.install/partition "communities"]

Second, I created the database schema in another transaction:

{:db/ident :person/name
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db.install/_attribute :communities}

{:db/ident :person/age
 :db/valueType :db.type/int
 :db/cardinality :db.cardinality/one
 :db.install/_attribute :communities}

{:db/ident :person/sibblings
 :db/valueType :db.type/int
 :db/cardinality :db.cardinality/one
 :db.install/_attribute :communities}

And here is an example of a simple query:

(d/q '[:find ?name ?age
       :where
       [?p :person/name ?name]
       [?p :person/age ?age]]
   db)

When I issue this query, I get the following error: 

Unhandled datomic.impl.Exceptions$IllegalArgumentExceptionInfo :db.error/not-an-entity Unable to resolve entity: :person/name {:db/error :db.error/not-an-entity}

When I replace :db.install/_attribute :communities by :db.install/_attribute :db.part/db for every attribute, the query works fine. I have the same problem for all my other queries.

Am I missing something ?

datomic clojure
1 Answers

User-defined partitions are not alternative "linkage points" for user-defined attributes. Attributes are always linked to the :db.part/db entity via the :db.install/attribute attribute1 and they always live in the :db.part/db partition.

Partitions allow you to ensure that datoms related to certain entities will occur close together in Datomic's indices by ensuring that their entity ids are allocated in a certain range. If your app is likely to access entities residing in a certain partition together, this may improve performance of your queries. It is also possible to walk datoms related to entities residing in a certain partition without encountering unrelated datoms (using seek-datoms + EAVT + entid-at).

The way to use a partition once it's defined is to pass its :db/ident to datomic.api/tempid when creating new entities (or you can use it with #db/id tagged literals in edn files):

@(d/transact connection
   {:db/id (d/tempid :some-partition)
    … …})

If you subsequently issue a query that happens to fetch some EAVT blocks related to this entity, those blocks will likely include information about other entities in the same partition (unless the entity has a huge enough number of attributes asserted on it to fill a block, I suppose… but even then you would have fetched some relevant nodes in the index tree), so your peer will be able to retrieve information about them from cache.

If you expect to see major benefits from this type of locality, then partitions may be worth looking into. If you're not sure your app would benefit, that's totally fine. Apparently this feature isn't used that much in the wild, indeed the new(-ish) client API currently doesn't expose it at all.

1 Although note that the explicit :db.install/_attribute :db.part/db line has been unnecessary since 0.9.5530: Datomic now deduces that a newly introduced entity is to be an attribute from the presence of the "attribute attributes" :db/ident, :db/valueType and :db/cardinality – this is possible because the latter two are exclusively used by attributes – and adds the :db.install/attribute linkage for you.

user_2327070
Related