Exporting anaphoric macros in common lisp packages

Question

i am having problem exporting a macro, it works when in it is declared in the same package, but not when it is imported. I use Emacs, SLIME, Clozure on Windows.

Package file

(defpackage :tokenizer
  (:use :common-lisp)
  (:export :tokenize-with-symbols 
       :current-token 
       :advanze-token 
       :peek-token
       :with-token-and-peek
       :with-token))

(defpackage :csharp-parser
  (:use :common-lisp :tokenizer)
  (:import-from :tokenizer :with-token-and-peek :with-token))

Tokenizer file

(in-package :tokenizer)

(defmacro with-token-and-peek (&body body) 
  `(let ((token (current-token tokenizer))
     (peek (peek-token tokenizer)))
     ,@body))

Parser file

(in-package :csharp-parser)

(defun expression (tokenizer node-stack)
  (with-token-and-peek
   (cond ((is-number? token) (make-value-node "number" token))
         ((is-bool? token) (make-value-node "bool" token))
         ((is-identifier? token peek) (make-identifier-node tokenizer node-stack))
         (t (make-ast-node :identifier "bla")))))

Gives the errors on compile:

csharpParser.lisp:265:3:
  warning: Undeclared free variable TOKENIZER::TOKENIZER (2 references)
           style-warning: Unused lexical variable TOKENIZER::PEEK
           style-warning: Unused lexical variable TOKENIZER::TOKEN
csharpParser.lisp:266:14:
  warning: Undeclared free variable TOKEN
etc etc etc

If i try a macroexpansion in package :csharp-parser

(macroexpand-1 '(with-token-and-peek tok))

(LET ((TOKENIZER::TOKEN (CURRENT-TOKEN TOKENIZER::TOKENIZER))
      (TOKENIZER::PEEK (PEEK-TOKEN TOKENIZER::TOKENIZER)))
  TOK)
T

Now like i said if i move the macros to the parser file, it compiles and works perfectly. But when i try to refactor it to the tokenizer file and export it via the package system it gives these errors, because it seems to internalize the symbol to the calling package. I have tried multiple ways via the colons, but can't get it to work.

If anybody could help me with this i would be very thankful.

user_7720217 · Answer

This happened because the macro with-interned-symbols expanded to code which includes symbols interned in TOKENIZER, while the cond expression has only symbols interned in CSHARP-PARSER. Any symbols (other than keywords or gensyms) that a macro expansion includes should be interned.

The following macro will intern a list of symbols to variables of the same name:

(defmacro with-interned-symbols (symbol-list &body body)
  "Interns a set of symbols in the current package to variables of the same (symbol-name)."
  (let ((symbol-list (mapcar (lambda (s)
                               (list s `(intern (symbol-name ',s))))
                             symbol-list)))
    `(let ,symbol-list ,@body)))

The macro with-token-and-peek can be redefined this way using the above to avoid this mismatch:

(with-interned-symbols (token peek)
  (defmacro with-token-and-peek (&body body) 
    `(let ((,token (current-token tokenizer))
       (,peek (peek-token tokenizer)))
       ,@body)))

Note that while anaphoric macros might be the most obvious special case here, this can happen in any macro that introduces new symbols to the expansion, with the exception of keywords as they are always in the keyword package.

Exporting anaphoric macros in common lisp packages

2 Answers