Set Emacs defaut font face per-buffer/mode

How about something like this:

(add-hook 'markdown-mode-hook (lambda () (variable-pitch-mode t))

You can then customize the variable-pitch face, and the other faces in the buffer will inherit from this instead of the default face.

Read the docs for buffer-face-mode for more customization details. (BufFace is also used for text-scale-increase and text-scale-decrease... very useful.)

I have to give a partial answer because this is too complicated to figure out on the spot and I already blew my time budget.

Face is a frame property. A frame can display multiple buffers at the same time. Mode is a buffer property. You ask how to vary the face on a per-mode basis. Combining all this, it seems that the question cannot not have a single fully-correct answer.

You can approximate the desired answer if you assume that a given frame will never display more than one buffer. You can actually accomplish that with something like this, but modified to use special-display-regexps and a set of regexps that match your markdown-mode buffer names.

(append special-display-buffer-names
        '("*VC-log*"
          "*Help*"
          ("*Completions*" 
           (height . 25)
           (font . "8x13"))))

However, this is probably not what you want. Your question seems to imply changing the face properties of a single frame.

Again assuming that a frame will never display more than one buffer at a time, you can try advising switch-to-buffer. But that might not be sufficiently low level and it might be too slow. (untested)

(defadvice switch-to-buffer (after switch-to-buffer activate compile)
  "change the frame's default face after switch-to-buffer"
  (doSomethingToChangePropertiesOfDefaultFace))

And now for my actual (incomplete) answer...

A better, albeit more complicated, approach would instruct markdown-mode to use a new face for all regions that are not already assigned one of the built-in faces. You can create a new face with copy-face and give it interesting properties with set-face-*.

Modify markdown-mode's font-lock-defaults to override the default font-lock-fontify-region-function as described in the comment block near line 946 of font-lock.el that begins, "Fontification functions". You can probably use a very slightly modified font-lock-default-fontify-region that does just one extra step immediately after it does:

  (unless font-lock-keywords-only
    (font-lock-fontify-syntactically-region beg end loudly))

The extra step parses the region similar to what font-lock-fontify-syntactically-region does, breaking the region into "interesting" sub-regions. But this time you find sub-regions that have the default face and you put-text-property those sub-regions to the new face that you previously created.

In all this feels like it should be only a couple lines of elisp in your .emacs file, plus make a copy of font-lock-default-fontify-region that has only a minor diff from the original (call one new function), plus make a copy of font-lock-fontify-syntactically-region and modify it to do your bidding (the most difficult part).

Actually, if you "after" advise font-lock-fontify-syntactically-region then you probably don't even need to modify font-lock-defaults or font-lock-default-fontify-region.

It's actually straightforward even for emacs version 22.3.1…

Just try the following:

(progn 
  (set-buffer "your buffer name here")
  (overlay-put (make-overlay 0 (buffer-size)) 'face 'your-face))