I love datomic. Datalog is a definitely something noteworthy and even if you never going to use it in your projects, still it is worth getting your hands dirty.
Despite all Datomic bright sides, there are some thoughts bothering me since I started using it more extensively in my pet project.
Here they are:
Confusion on datomic peer and client libraries differences
I used to use datomic-pro api in my project and wanted to catch “anomaly exceptions” from Pedestal error handlers. That was not an easy task. It simply didn’t work in the manner I expected from the datomic code examples.
If you try to execute
(d/transact conn {:tx-data [[:this "does not" :make "sense"]]})
using peer library you’ll get
1. Caused by datomic.impl.Exceptions$IllegalArgumentExceptionInfo
:db.error/not-a-data-function Unable to resolve data function: :this
{:cognitect.anomalies/category :cognitect.anomalies/incorrect,
:cognitect.anomalies/message "Unable to resolve data function: :this",
:db/error :db.error/not-a-data-function}
error.clj: 167 datomic.error/deserialize-exception
error.clj: 160 datomic.error/deserialize-exception
peer.clj: 379 datomic.peer.Connection/notify_error
connector.clj: 155 datomic.connector/fn
connector.clj: 153 datomic.connector/fn
MultiFn.java: 234 clojure.lang.MultiFn/invoke
connector.clj: 180 datomic.connector/create-hornet-notifier/fn/fn/fn/fn
connector.clj: 175 datomic.connector/create-hornet-notifier/fn/fn/fn
connector.clj: 173 datomic.connector/create-hornet-notifier/fn/fn
core.clj: 2030 clojure.core/binding-conveyor-fn/fn
AFn.java: 18 clojure.lang.AFn/call
FutureTask.java: 266 java.util.concurrent.FutureTask/run
ThreadPoolExecutor.java: 1149 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 624 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 748 java.lang.Thread/run
(ex-data *e)
#:clojure.error{:phase :print-eval-result}
And this is what you get from datomic client api library using peer. So exceptions here are actually wrapped in ex-data info.
1. Unhandled clojure.lang.ExceptionInfo
Unable to resolve data function: :this
{:cognitect.anomalies/category :cognitect.anomalies/incorrect,
:cognitect.anomalies/message "Unable to resolve data function: :this",
:db/error :db.error/not-a-data-function,
:dbs
[{:database-id "datomic:dev://localhost:4334/login",
:t 1016,
:next-t 1017,
:history false}]}
async.clj: 58 datomic.client.api.async/ares
async.clj: 54 datomic.client.api.async/ares
sync.clj: 104 datomic.client.api.sync/eval11528/fn
protocols.clj: 72 datomic.client.api.protocols/fn/G
api.clj: 200 datomic.client.api/transact
api.clj: 183 datomic.client.api/transact
REPL: 816 stock-login.service/eval35091
REPL: 816 stock-login.service/eval35091
Compiler.java: 7181 clojure.lang.Compiler/eval
Compiler.java: 7136 clojure.lang.Compiler/eval
core.clj: 3202 clojure.core/eval
core.clj: 3198 clojure.core/eval
interruptible_eval.clj: 87 nrepl.middleware.interruptible-eval/evaluate/fn/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 667 clojure.core/apply
core.clj: 1977 clojure.core/with-bindings*
core.clj: 1977 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 nrepl.middleware.interruptible-eval/evaluate/fn
main.clj: 437 clojure.main/repl/read-eval-print/fn
main.clj: 437 clojure.main/repl/read-eval-print
main.clj: 458 clojure.main/repl/fn
main.clj: 458 clojure.main/repl
main.clj: 368 clojure.main/repl
RestFn.java: 137 clojure.lang.RestFn/applyTo
core.clj: 667 clojure.core/apply
core.clj: 662 clojure.core/apply
regrow.clj: 20 refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 84 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 56 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 152 nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
AFn.java: 22 clojure.lang.AFn/run
session.clj: 202 nrepl.middleware.session/session-exec/main-loop/fn
session.clj: 201 nrepl.middleware.session/session-exec/main-loop
AFn.java: 22 clojure.lang.AFn/run
Thread.java: 748 java.lang.Thread/run
here is ex-data of the exception. This is much more analysable!
(ex-data *e)
{:cognitect.anomalies/category :cognitect.anomalies/incorrect,
:cognitect.anomalies/message "Unable to resolve data function: :this",
:db/error :db.error/not-a-data-function,
:dbs
[{:database-id "datomic:dev://localhost:4334/login",
:t 1016,
:next-t 1017,
:history false}]}
Okay, so I need to use client library. That increases the complexity by having to have application server in form of running peer.
I understand that was totally required by datomic cloud implementation, but is it really required for me now?
Here is an excerpt on the datomic client library rationale.
Datomic’s peer library puts database query in your own application process. This provides several benefits, but at the price of a heavier dependency (both in code and in memory requirements) than a traditional client.
A smaller footprint is useful in environments that have operational limitations, or where processes are small or short-lived. The new Datomic client API addresses this need. Lightweight clients connect to Peer Servers, which are peers that run in a separate address space.
Existing peers are unchanged, and you can mix and match peer and client applications as you see fit within the same Datomic install. Clients and peers are described in detail in the new clients and peers section of the docs.
Source: https://blog.datomic.com/2016/11/datomic-update-client-api-unlimited.html
Feature parity between datomic peer api and datomic client api library
I found the following discrepancies:
- d/squiid is not supported, use (java.util.UUID/randomUUID) instead
- d/touch is not supported, use d/pull with ‘[*]
- only find-rel is supported within :find clause!
Say I can’t execute [?e …] anymore
So I’m really in doubts if the investments to moving to client libary worth all the hassle in my case.
Some more tough usecases for me
“Rollback” transaction.
There is no easy way to rollback transaction, but there is some opportunity instead.
The recommended way to do this is following.
Search datomic history for required txs, select all entities that was affected by tx and “reverse” them by adding fourth
:add parameter to complement value.
Hard to pick proper abstractions
Well, I spent a lot of time reading official sources and other people’s code.
Most prominent of these, namely: mbrainz importer examples, vase sources.
First, I have to admit that I have no idea how people can produce something like mbrainz async importer by themselves. TBH, I can hardly understand what’s going on there and how could I come up with the same implementation.
Some people doing Datomic.Entity passing as function return values, some just raw values from the d/q.
What worked well for me is to return raw entity ids from the find functions, then when required I’m able to execute pull function against returned values.
The same goes to db/ident that one tries to use as enums. Aspecially within tests you constantly trying to use :db/ident with example values to easiely reference in test checks, but at the same time your function for the prod data will return just :db/id.
Example of what I use in tests
{:db/ident :my-test-account
:account/id (d/squiid)
:account/balance 10}
so you can use just (d/entity db :my-test-account). But this is not the case for the code that will probably return :db/id instead.
I’ve found a useful snippet to address this problem
(defprotocol Eid
(e [_]))
(extend-protocol Eid
java.lang.Long
(e [n] n)
clojure.lang.Keyword
(e [n] n)
datomic.Entity
(e [ent] (:db/id ent)))
I also created the following function to flatten returned from the pull expression nested db/ident.
(defn flatten-ident [coll]
(clojure.walk/postwalk
(fn [item] (get item :db/ident item)) coll))
Testing
Some good examples on datomic boilerplate for testing I have found in vase repository and adopted like this
(def ^:dynamic *current-db-connection* nil)
(def ^:dynamic *current-db-uri* nil)
(defn new-database
"This generates a new, empty Datomic database for use within unit tests."
[txes]
(let [uri (str "datomic:mem://test" (UUID/randomUUID))
_ (d/create-database uri)
conn (d/connect uri)]
(->> (read-string (slurp "schema.edn"))
(d/transact conn)
deref)
(doseq [t txes]
@(d/transact conn t))
{:uri uri
:connection conn}))
(defmacro with-database
"Executes all requests in the body with the same database."
[txes & body]
`(let [dbm# (new-database ~txes)]
(binding [*current-db-uri* (:uri dbm#)
*current-db-connection* (:connection dbm#)]
~@body)))
I found using with-database macro with txs is much more convenient and such as having better data locality then using fixtures.
Specs and generative testing
I feel like it should naturally fall into the spec paradigm and it should be really piece of cake to generate data for testing.
But in my case I didn’t succeed with this. I couldn’t even grasp how to generate simple albums-players relation using spec/gen.
Examples are still hard to understand and follow.
Pagination and limit.
Despite datomic client support of :offset and :limit there is recommended way to do such a thing using index-pull. That requires more efforts to comprehend simple tasks.
Pull functions
Well, I faced a good amount of WTF’s to try to understand how db functions work. Simple example with xform.
The fn is either a fully qualified function allowed under the :xforms key in resources/datomic/extensions.edn, or one of the following built-ins:
Documentation was so succint, so I spent quite good amount of time to find at least one example of extensions.edn and where to keep it.
So it should be located at /resources/datomic/extensions.edn with such a content
{:xforms [mynamespace/xform]}
then you should have xform function in your namespace
(namespace mynamespace)
(defn xform [x] (str "xformed: " x))
I was thinking about usecase when you need some translations, say I keep my currency amounts in cents and want to translate to float using currency minor unit. I didn’t manage to do this with xforms.
There are not so many diagramming tools to use with Datomic at the moment.
Here is the list of known ones to me
Both were not working for me using instructions from respective READMEs.
Here is my example repo with some issues fixed for datomic pro.
Not clear when to use db.cardinality/many
So documentation says you don’t have to decide when to use many or one relation between entities, as they automatically accessible using underscore notation.
That is not 100% true. If you decide to generate sample data using generators, most probably using db.cardinality/many will be a better choice for you.
Summary
I know, the reason for my frustration is that technology is relatively new, not so many great examples on how to use it and best practices are a bit fuzzy. But most probably the root cause is my lack of experience. The amount of novelty coming with such a technology is overhelming for me. Almost literally every step required efforts to google around, how other people tend to use it.
The simplicty you get gives you much power, but at the same time great responsibility on decisions you make. I had to rewrite couple of times significant part of the backend to verify ideas and what decisions turned to be the best.
Probably, it will work better for you then it was for me, so don’t give up.
