After writing some projectile glue and
files, I thought to myself if I could write code to generate those
files. A DSL so to say. The reason is three-fold:
- Write a simple ignore file structure as Elisp code
- Allow Elisp variables for dynamic and reusable sections
- Have a hand in writing a generic DSL
Aside from my hacking sensibilities, the key concerns of doing so may be put as:
- Maintaining two ignore files when it is just really one with a different syntax
- You cannot reuse blocks of repeating sections with different files
The former is syntactic separation while the later is flexibility and
reusability. For my use case with Elisp, I have two projects: my Emacs
configuration and my org-jekyll-blogger.el blog project. Using
projectile, I have to more or less copy my ignore files and
tweak it a bit. To address this, I created my own DSL
.projectile is similar in intent but
magin was the library I had in mind as proof.
As an example, this is my .emacs.d
# Base files config.el personal.el # Base directory .cache extra # Project specific working-config.org my-macros.el # Library lib/sandbox lib/packages !lib/modules !lib/scripts # Block for gtags GPATH GTAGS GRTAGS # Block for emacs *.elc .#* *Org Src*
It is generated by this DSL:
(magin-write-to-project `(delimited ;; Context keyword that separates contexts by newlines (context ;; Represents a semantic group (comment "Base files") ;; A comment keyword (file ,(format "%s.el" (file-name-base fn/config-file))) ;; A file keyword (file ,(format "%s.el" (file-name-base fn/personal-file)))) ;; Notice I use a Elisp variable (context (comment "Base directory") ;; Generates "# Base directory" (dir ".cache") ;; A dir keyword (dir "extra")) ;; An alias to file but it indicates intends for a directory (context ; (comment "Project specific") (file "working-config.org") (file ,(format "%s.el" (file-name-base fmk/macro-file)))) (context (comment "Library") (path ;; Represents a path prefix "lib" ;; Everything underneath is generated with a "lib" prefix (context (dir "sandbox") (dir "packages")) (include (dir "modules") (dir "scripts")))) (block gtags) ;; Reusable blocks (block emacs)) user-emacs-directory ;; Rewrites =.gitginore= of `user-emacs-directory' )
- Implicit Environment
The reason why I was confident in making this DSL is due to thanks to backquoting: it mixes the use of quoting and expressions in a Lispy way. I point you to understand quoting before proceeding, but as a quick brush it is using list of symbols as data.
As an example of quoting:
(+ 1 2 3) ;; 6 '(+ 1 2 3) ;; (+ 1 2 3) (quote (+ 1 2 3e)) (subject linking-verb adjective) ;; Missing symbols '(subject linking-verb adjective) ;; As is
It also allows for lazy evaluation, the problem arises when we want to integrate Elisp variables into the equation:
(setq left 1 right 2) (+ left right) ;; 3 (+ 1 right) ;; 3 '(+ 1 right) ;; (+ 1 right) ;; Using backquotes `(+ 1 ,right) ;; (+ 1 2) `(+ 1 ,(+ left right)) ;; (+ 1 3) ;; Evaluating the actual value (eval `(+ 1 ,(+ left right))) ;; 4
The complicated approach would ask us to create a tokenizer, parser and compiler while managing text matching and all that jazz; lispy data structures allows us to bypass this through the quoting. Without needing anymore, we can create a quoted structures that integrates with Elisp without any hastle.
So in thinking of the DSL, let us understand what the domain is. Our domain of abstraction is gitignore. After some thinking and reading, this is the abstraction I want to address:
- Files and directories have no distinction
- Path context
- Reusable blocks or groups
There are other such abstraction I wish not cover:
- Quoting or escaping
- Glob keywords, as in
rx.el, for single or double asterisk
A DSL that is good enough to accomplish the above is good enough without being too abstract. Without stretching too much, this DSL has no intermediate output, it translates directly to text. This tradeoff loses flexibility in output but at least it is easier. If you wanted, the DSL can be mechanically a list of plists with a main text manipulated by several properties as such:
(magin--compiler (path "glob" (include (file "foo.bar") (path "gib" (dir "foo-bar"))))) ;; This hypothetically might yield '((:line "foo.bar" :include t :parent "glob") (:line "foo-bar" :include t :parent "glob/gib"))
Without much complexity aside, we can start with hacking with the simple syntax.
At last, we define our data structure:
;; The leaf keywords (file ,line) (dir ,line) ;; Alias for file ;; Non functional nodes (comment ,text) (newline) ;; Syntactic grouping, the equivalent of progn (context &rest ,sublines) ;; Contextual grouping, like `context' but affects everything inside it ;; Sublines have the property `:include' as `t' ;; Which tells the leaf node to add `!' as a prefix (include &rest ,sublines) ;; Sublines have the property `:path' set to `path' ;; Nesting of paths are handled (path ,path &rest sublines) ;; Block keywords ;; Create a variable (defblock ,block-name &rest block-lines) ;; Call the variable (block ,block-name) ;; Aesthetic grouping ;; Every subline is interleaved with a newline (delimited &rest ,sublines)
Those are the keywords that we must handle in our DSL. With this in mind, we can start be creating a function that handles the compilation.
(defun magin--compiler (dsl &optional env) "Compiles DSL with the environment ENV." nil)
This is our DSL handler, every node that wants to compile nodes have
to go through this dispatching function. This setup allows us to add
more keywords independently of each other. However, we are not going
to use a long
cond statement; we will use the implicit
Elisp environment to find handlers. If the compiler comes to a
keyword, it finds a function in the Elisp environment that starts with
magit--dsl- and invokes that as a handler.
(defun magin--dsl-context (dsl env) "File keyword handler.") (defun magin--dsl-file (dsl env) "File keyword handler.") (defun magin--dsl-block (dsl env) "Block keyword handler.") (magin--compiler `(context (file "a") (block emacs) (unknown) ;; Error, no `magit--dsl-unknown' ))
In the example above, the compiler finds the three functions above and
calls them but fails on the last one. So if you want to extend it
without messing with the source, one can simply define a function with
the prefix and it will be detected by the compiler. This is similar to
use-package does it with its keywords. A global dependency
injection if you will. This magic is done through
(lexical-let* ((rule-name (symbol-name rule)) (rule-handler (intern-soft (format "%s%s" magin-dsl-prefix rule-name)))) (if (null rule-handler) (error "No rule to handle %s at dsl: %s" rule-name dsl) (funcall rule-handler dsl env)))
As a side note, you can use an
hash-map or create your own
or symbol environemt if you really wanted a private space. In our
approach, you can see the symbols with the helpful
without extra work.
If you want to use
eval-sexp with the raw DSL, you have to create
env. You can remap
but you need to know when it is evaluating a DSL or a Lisp. We're not
creating a new interpreter or environment, so this is good enough.
With this simple mechanism, we can define the keywords incrementally.
So our dispatcher is a function that takes a dsl and an environment which returns a text. Without the intermediate data structure, the context is applying the dispatcher to each line and then combining it with a newline delimiter as such:
(string-join (mapcar (lambda (subdsl)(funcall #'magin--compiler subdsl env)) subdsls) "\n")
This is simply lining up the entries. Nothing complicated but how about manipulating those that manipulate the environment?
A quick way to create an environment like scoping, one can easily use
an alist and it will take care of itself. For example with the
(lexical-let ((new-env (append (list (cons :include t)) env))) (magin--dsl-context dsl new-env))
You could use plist but have to implement an
extend function for it.
Interestingly, you could also use an alist to simulate an enviroment
and lexical scoping without much problem.
It is as simple as appending the
cons entry and it is done. The
others are implemented the same way. Now how about implementing the
The implementation above forces everything to the leaf node which
really have only one important one,
file. The file has to take the
text value and format it based on the environment.
(lexical-let* ((parent (cdr (assoc :parent env))) (include (if (cdr (assoc :include env))"!" nil))) (concat include parent file))
Our friend is
cdr when manipulating an alist enviroment.
Again, it is simply a matter formatting. So writing a DSL with this
setup is actually easy.
Lastly, let's talk about defining a block variable. Like with the
dispatcher, we can use the implicit Elisp environment as the variable
space. No need to define a hash of symbols and lookups, we already
have one and we take advantage of it. The keyword
simply save the defintion into a symbol with a
and the other one,
block, looks for it. Aside from
the complimentary friend is
makunbound, which unbinds the symbol so
the value is updated properly.
(lexical-let* ((block-name (symbol-name block-symbol)) (block-def-name (intern (format "%s%s" magin-block-prefix block-name)))) (makunbound block-def-name) (eval `(defvar ,block-def-name '(context ,@block-def) ,(format "Block definition for %s" block-name))) block-def-name)
Aside from using some direct
eval magic, it is as straightforward as
it gets as well as with the
block handler. With this, we can have
our resusable blocks of code.
(magin--dsl-defblock `(defblock emacs (comment "Block for emacs") (file "*.elc") (file ".#*") (file "*Org Src*") ;; Org Src buffers ) (list))
The least important is the non-functional ones, like with
newline which are prefixed text and an empty text respectively.
delimited keyword is a bit more complex that it
interleaves each line with a
(lexical-let ((delimited-dsls (cdr (apply #'append (mapcar (lambda (dsl) (list '(newline) dsl)) subdsls))))) (magin--compiler `(context ,@delimited-dsls) env))
Take your pick in implementing
interleave but the thing here is that
it wraps the old context with a newly modified one. It may not be a
big thing but this interferes with debugging and tracing when you
don't know the original source. This is one weakness of this approach
that it interferes with the dsl itself. Nothing terrible but it is the
price of having the return value as text instead of an intermediate
With all those ideas, the DSL is easily implemented.
So with this design and implementation we have a crud but simple and dynamic DSL. You can look at the main defintion and notice the use of backquoting in how it addresses the problems mentioned. Now I can manage my ignore file with Emacs.
Now for some minor things we might have forgotten:
- Unit tests
The DSL is not too complex to warrant any of those but it is nice to
note and implement. As one final point, we could have used a
defmacro instead of a dispatch compiler but in case we want to
update our code in order to join other contexts, we have a base.
(setq left-code `(context (file "a") (file "b")) right-code `(context (file "c") (file "d"))) (magin-join left-code right-code) ;; ???
If we are dealing with text, we can't have this yet. In particular if you have an initial defintion and want to update it according some condition, then it requires changing the code to handle plist or something that fancies the data.
To reiterate, Elisp allows us to create a DSL with ease and little knowledge.
So I copied this code to create
projectile ignore file
but unless I need more functionality, I don't think I need it for now.
If another ignore file comes my way, I might refactor and do the same.
The one thing I want to really write is a DSL for SQL writing, not SQL itself. My dream is to write SQL as blocks of data.
(setq display-fields '(fields first-name last-name middle-name)) (select display-fields (from table)) ;; "SELECT firstName, lastName, middleName FROM table"
The idea in that snippet is the fields are data itself. It also extends to queries being data but the abstraction of aliasing and joining is sketchy for now. That is my dream though: to write a DSL for SQL writing. For now, ignore files are good practice.