Understanding literal-produced objects in read-only memory

user_5400548 0
Viewed 94

Looking at the release notes of SBCL 2.2.7, i see an interesting new 'minor incompatible change':

minor incompatible change: literal objects (strings, in particular) in compiled code may at the discretion of the runtime be placed in read-only memory. Violations of CLHS 3.7.1 could produce memory faults. If ":PURIFY NIL" is given to SAVE-LISP-AND-DIE then no read-only memory will be used.

CLHS section on the literal objects says the following operations are considered destructive:

array: Storing a new value into some element of the array, or performing a destructive operation on an object that is already such an element. Changing the fill pointer, dimensions, or displacement of the array (regardless of whether the array is actually adjustable). Performing a destructive operation on another array that is displaced to the array or that otherwise shares its contents with the array.

So, as far as i understand, doing something like

(defun inc-char-at (idx s)
  (setf (char s idx)
        (code-char (1+ (char-code (char s idx)))))
  s)

(inc-char-at 2 "aaaaa")

can't be considered to be safe. Does this mean, that the rule of thumb is not to use string literals at all, except in situations where there are semantically immutable (like format strings or constants), preferring make-array of characters instead?

How can it be considered to be the 'minor incompatibility' then?

And, what is more important, does the standard clarify which cases would make a literal (not necessarily a string) to be considered immutable and therefore put into read-only memory?

compiler-optimization sbcl common-lisp
2 Answers

Does this mean, that the rule of thumb is not to use string literals at all, except in situations where there are semantically immutable (like format strings or constants), preferring make-array of characters instead?

Yes.

Also I'd use (copy-seq "my-literal") instead of make-array in that case, or have this value only ever be used by purely functional code.

How can it be considered to be the 'minor incompatibility' then?

It is minor in the sense that SBCL was already subject to §3.7.1 before the changes, i.e. modifying literal data had undefined consequences. It is incompatible because the runtime is allowed to do something different than before, but minor because it should not have a noticeable impact on conforming programs.

And, what is more important, does the standard clarify which cases would make a literal (not necessarily a string) to be considered immutable and therefore put into read-only memory?

Reading the paragraph mentioned above, §3.7.1 Modification of Literal Objects , it seems clear to me that no literal object should ever be modified, which makes them all immutable.

Whether or not they are put in read-only memory is up to the compiler/runtime and you have implementation-defined way to act on that (e.g. like said in the release note, by setting purify to nil). But from a standard point of view, no, you have to assume all literals might be stored in read-only memory.

Now, an "undefined behavior" can still be a useful behavior when you work on the REPL. For example, something like this works, without warnings:

> "test"
"test"
> (setf (char * 0) #\f)
#\f
> **
"fest"

But you wouldn't rely on this behavior when producing code to be used on some another Lisp environment, because, by definition, you can't expect the other environment to behave similarly.

I think in practice the strings I use are almost all produced dynamically, from user interactions, network requests, file manipulations, etc.

The possibly dangerous examples I can think of is when testing code, e.g. from the REPL or with unit tests, and for the REPL this is generally not a problem as you don't ship the code. For tests however it can be tempting to directly write:

(assert (string= (nstring-upcase "test") "TEST"))

and forget that it is non-conforming (if you wrap that in a lambda, show the output of disassemble, then execute it and disassemble again, you'll see that the code is changed: the value is upcased in the assembly code. If the function (or more precisely the string) was stored in read-only memory, this would be a problem).

Fortunately the warnings can help detect this case.

user_1243190

This is close to being a comment on coredump's excellent answer: I'm adding it here simply because comments can go away.

And, what is more important, does the standard clarify which cases would make a literal (not necessarily a string) to be considered immutable and therefore put into read-only memory?

Section 3.7.1 of the spec says

The consequences are undefined if literal objects are destructively modified.

and then goes on to give a list of operations which constitute 'destructive operations', including storing a new value into an element of an array of course. But any literal object fall into this category.

What this means is that the behaviour of any program which has ever done that has always been undefined according to the standard, or in other words, conforming programs have never been allowed to perform destructive operations on literal objects. See 1.5.2 and in particular point 3:

Conforming code shall not depend on the consequences of undefined or unspecified situations.

The reason the standard talks about 'undefined behaviour' is that it clearly places an undue burden on implementors to actually detect modifications of literals in all cases. For instance imagine an implementation on a machine without hardware write-protection for memory: such an implememtation would need to record, and check, information about every object, which would quite probably be a serious performance hit. The standard generally goes to some length to avoid requiring heroic behaviour of implementations, and it is doing just that here.

So what seems to have happened is that SBCL, which does run on machines with hardware memory protection in general, has started putting (some?) literals into read-only memory. That's a good thing I think.

user_170269340
Related