Problem

After writing some projectile glue and .projectile and .gitignore files, I thought to myself if I could write code to generate those files. A DSL so to say. The reason is three-fold:

  • Write a simple ignore file structure as Elisp code
  • Allow Elisp variables for dynamic and reusable sections
  • Have a hand in writing a generic DSL

Aside from my hacking sensibilities, the key concerns of doing so may be put as:

  • Maintaining two ignore files when it is just really one with a different syntax
  • You cannot reuse blocks of repeating sections with different files

The former is syntactic separation while the later is flexibility and reusability. For my use case with Elisp, I have two projects: my Emacs configuration and my org-jekyll-blogger.el blog project. Using git and projectile, I have to more or less copy my ignore files and tweak it a bit. To address this, I created my own DSL magin to handle .gitignore files; .projectile is similar in intent but magin was the library I had in mind as proof.

As an example, this is my .emacs.d .gitignore: 

# Base files
config.el
personal.el

# Base directory
.cache
extra

# Project specific
working-config.org
my-macros.el

# Library
lib/sandbox
lib/packages
!lib/modules
!lib/scripts

# Block for gtags
GPATH
GTAGS
GRTAGS

# Block for emacs
*.elc
.#*
*Org Src*

It is generated by this DSL: 

(magin-write-to-project
 `(delimited ;; Context keyword that separates contexts by newlines

   (context ;; Represents a semantic group
    (comment "Base files") ;; A comment keyword
    (file ,(format "%s.el" (file-name-base fn/config-file))) ;; A file keyword
    (file ,(format "%s.el" (file-name-base fn/personal-file)))) ;; Notice I use a Elisp variable

   (context
    (comment "Base directory") ;; Generates "# Base directory"
    (dir ".cache") ;; A dir keyword
    (dir "extra")) ;; An alias to file but it indicates intends for a directory

   (context ;
    (comment "Project specific")
    (file "working-config.org")

    (file  ,(format "%s.el" (file-name-base fmk/macro-file))))


   (context
    (comment "Library")
    (path ;; Represents a path prefix
     "lib" ;; Everything underneath is generated with a "lib" prefix
     (context
      (dir "sandbox")
      (dir "packages"))
     (include
      (dir "modules")
      (dir "scripts"))))

   (block gtags) ;; Reusable blocks
   (block emacs))
 user-emacs-directory ;; Rewrites =.gitginore= of `user-emacs-directory'
 )

I do hope the DSL is easy to digest. Here are the two libraries for the interested: magin and projin. If you stay around, I'd like to discuss the following concepts in crafting it:

  • Backquote
  • Dispatching
  • Implicit Environment

Backquoting

The reason why I was confident in making this DSL is due to thanks to backquoting: it mixes the use of quoting and expressions in a Lispy way. I point you to understand quoting before proceeding, but as a quick brush it is using list of symbols as data.

As an example of quoting: 

(+ 1 2 3) ;; 6

'(+ 1 2 3)         ;; (+ 1 2 3)
(quote (+ 1 2 3e))


(subject linking-verb adjective)  ;; Missing symbols
'(subject linking-verb adjective) ;; As is

It also allows for lazy evaluation, the problem arises when we want to integrate Elisp variables into the equation: 

(setq left 1
   right 2)

(+ left right) ;; 3

(+ 1 right)    ;; 3
'(+ 1 right)   ;; (+ 1 right)

;; Using backquotes
`(+ 1 ,right)           ;; (+ 1 2)
`(+ 1 ,(+ left right))  ;; (+ 1 3)

;; Evaluating the actual value
(eval `(+ 1 ,(+ left right)))  ;; 4

The complicated approach would ask us to create a tokenizer, parser and compiler while managing text matching and all that jazz; lispy data structures allows us to bypass this through the quoting. Without needing anymore, we can create a quoted structures that integrates with Elisp without any hastle.

Domain Syntax

So in thinking of the DSL, let us understand what the domain is. Our domain of abstraction is gitignore. After some thinking and reading, this is the abstraction I want to address:

  • Files and directories have no distinction
  • Comments
  • Path context
  • Separators
  • Inclusion
  • Reusable blocks or groups

There are other such abstraction I wish not cover:

  • Quoting or escaping
  • Glob keywords, as in rx.el, for single or double asterisk

A DSL that is good enough to accomplish the above is good enough without being too abstract. Without stretching too much, this DSL has no intermediate output, it translates directly to text. This tradeoff loses flexibility in output but at least it is easier. If you wanted, the DSL can be mechanically a list of plists with a main text manipulated by several properties as such: 

(magin--compiler
 (path "glob"
       (include
        (file "foo.bar")
        (path "gib"
              (dir "foo-bar")))))

;; This hypothetically might yield
'((:line "foo.bar" :include t :parent "glob")
  (:line "foo-bar" :include t :parent "glob/gib"))

Without much complexity aside, we can start with hacking with the simple syntax.

Keywords

At last, we define our data structure: 

;; The leaf keywords
(file ,line)
(dir ,line) ;; Alias for file

;; Non functional nodes
(comment ,text)
(newline)


;; Syntactic grouping, the equivalent of progn
(context &rest ,sublines)

;; Contextual grouping, like `context' but affects everything inside it
;; Sublines have the property `:include' as `t'
;; Which tells the leaf node to add `!' as a prefix
(include &rest ,sublines)

;; Sublines have the property `:path' set to `path'
;; Nesting of paths are handled
(path ,path &rest sublines)


;; Block keywords
;; Create a variable
(defblock ,block-name &rest block-lines)

;; Call the variable
(block ,block-name)


;; Aesthetic grouping
;; Every subline is interleaved with a newline
(delimited &rest ,sublines)

Those are the keywords that we must handle in our DSL. With this in mind, we can start be creating a function that handles the compilation. 

(defun magin--compiler (dsl &optional env)
  "Compiles DSL with the environment ENV."
  nil)

This is our DSL handler, every node that wants to compile nodes have to go through this dispatching function. This setup allows us to add more keywords independently of each other. However, we are not going to use a long switch or cond statement; we will use the implicit Elisp environment to find handlers. If the compiler comes to a keyword, it finds a function in the Elisp environment that starts with magit--dsl- and invokes that as a handler. 

(defun magin--dsl-context (dsl env)
  "File keyword handler.")

(defun magin--dsl-file (dsl env)
  "File keyword handler.")

(defun magin--dsl-block (dsl env)
  "Block keyword handler.")

(magin--compiler
 `(context
   (file "a")
   (block emacs)
   (unknown) ;; Error, no `magit--dsl-unknown'
   ))

In the example above, the compiler finds the three functions above and calls them but fails on the last one. So if you want to extend it without messing with the source, one can simply define a function with the prefix and it will be detected by the compiler. This is similar to how use-package does it with its keywords. A global dependency injection if you will. This magic is done through intern-soft: 

(lexical-let* ((rule-name (symbol-name rule))
    (rule-handler
     (intern-soft
      (format "%s%s" magin-dsl-prefix rule-name))))
  (if (null rule-handler)
      (error "No rule to handle %s at dsl: %s" rule-name dsl)
    (funcall rule-handler dsl env)))

As a side note, you can use an hash-map or create your own obarray or symbol environemt if you really wanted a private space. In our approach, you can see the symbols with the helpful describe-variable without extra work.

If you want to use eval-sexp with the raw DSL, you have to create your own eval and env. You can remap eval with magin-compiler, but you need to know when it is evaluating a DSL or a Lisp. We're not creating a new interpreter or environment, so this is good enough.

With this simple mechanism, we can define the keywords incrementally.

Context Keyword

So our dispatcher is a function that takes a dsl and an environment which returns a text. Without the intermediate data structure, the context is applying the dispatcher to each line and then combining it with a newline delimiter as such: 

(string-join
 (mapcar  (lambda (subdsl)(funcall #'magin--compiler subdsl env)) subdsls)
 "\n")

This is simply lining up the entries. Nothing complicated but how about manipulating those that manipulate the environment?

Environmental Keyword

A quick way to create an environment like scoping, one can easily use an alist and it will take care of itself. For example with the include keyword: 

(lexical-let ((new-env (append (list (cons :include t)) env)))
  (magin--dsl-context dsl new-env))

You could use plist but have to implement an extend function for it. Interestingly, you could also use an alist to simulate an enviroment and lexical scoping without much problem.

It is as simple as appending the cons entry and it is done. The others are implemented the same way. Now how about implementing the leaf keywords

Leaf Keyword

The implementation above forces everything to the leaf node which really have only one important one, file. The file has to take the text value and format it based on the environment. 

(lexical-let* ((parent (cdr (assoc :parent env)))
    (include (if (cdr (assoc :include env))"!" nil)))
  (concat include parent file))

Our friend is assoc and cdr when manipulating an alist enviroment. Again, it is simply a matter formatting. So writing a DSL with this setup is actually easy.

Block Keyword

Lastly, let's talk about defining a block variable. Like with the dispatcher, we can use the implicit Elisp environment as the variable space. No need to define a hash of symbols and lookups, we already have one and we take advantage of it. The keyword defblock will simply save the defintion into a symbol with a magin--block- prefix and the other one, block, looks for it. Aside from intern-soft, the complimentary friend is makunbound, which unbinds the symbol so the value is updated properly. 

(lexical-let* ((block-name (symbol-name block-symbol))
    (block-def-name (intern
                     (format "%s%s"
                             magin-block-prefix
                             block-name))))
  (makunbound block-def-name)
  (eval `(defvar ,block-def-name '(context ,@block-def)
           ,(format "Block definition for %s" block-name)))
  block-def-name)

Aside from using some direct eval magic, it is as straightforward as it gets as well as with the block handler. With this, we can have our resusable blocks of code. 

(magin--dsl-defblock
 `(defblock emacs
    (comment "Block for emacs")
    (file "*.elc")
    (file ".#*")
    (file "*Org Src*") ;; Org Src buffers
    )
 (list))

Non-functional Keyword

The least important is the non-functional ones, like with comment and newline which are prefixed text and an empty text respectively. However, the delimited keyword is a bit more complex that it interleaves each line with a newline keyword. 

(lexical-let ((delimited-dsls
     (cdr
      (apply #'append
         (mapcar
          (lambda (dsl)
            (list '(newline) dsl))
          subdsls)))))
  (magin--compiler
   `(context
     ,@delimited-dsls)
   env))

Take your pick in implementing interleave but the thing here is that it wraps the old context with a newly modified one. It may not be a big thing but this interferes with debugging and tracing when you don't know the original source. This is one weakness of this approach that it interferes with the dsl itself. Nothing terrible but it is the price of having the return value as text instead of an intermediate value.

With all those ideas, the DSL is easily implemented.

Wrap Up

So with this design and implementation we have a crud but simple and dynamic DSL. You can look at the main defintion and notice the use of backquoting in how it addresses the problems mentioned. Now I can manage my ignore file with Emacs.

Now for some minor things we might have forgotten:

  • Unit tests
  • Logging
  • Debugging

The DSL is not too complex to warrant any of those but it is nice to note and implement. As one final point, we could have used a defmacro instead of a dispatch compiler but in case we want to update our code in order to join other contexts, we have a base. 

(setq left-code `(context
               (file "a")
               (file "b"))
   right-code `(context
                (file "c")
                (file "d")))


(magin-join left-code right-code) ;; ???

If we are dealing with text, we can't have this yet. In particular if you have an initial defintion and want to update it according some condition, then it requires changing the code to handle plist or something that fancies the data.

To reiterate, Elisp allows us to create a DSL with ease and little knowledge.

Conclusion

So I copied this code to create projin for projectile ignore file but unless I need more functionality, I don't think I need it for now. If another ignore file comes my way, I might refactor and do the same.

The one thing I want to really write is a DSL for SQL writing, not SQL itself. My dream is to write SQL as blocks of data. 

(setq display-fields '(fields first-name last-name middle-name))

(select
 display-fields
 (from table))
;; "SELECT firstName, lastName, middleName FROM table"

The idea in that snippet is the fields are data itself. It also extends to queries being data but the abstraction of aliasing and joining is sketchy for now. That is my dream though: to write a DSL for SQL writing. For now, ignore files are good practice.