Good question! I'll divide this into parts. There's going to be a lot of material to google here, since your question touches multiple deep subjects.

1. Statements have no value

Declarations are statements. They have no value, and thus they cannot be parameters. That this code...

let a = 1

... has no value, means none of these will work:

doStuff(let a = 1)
let b = (let a = 1)
(let a = 1) + 5

The name a, or a + 5, or f(a) are expressions, and unlike statements, expressions have value. But the declaration of a itself does not.

Note that your intuition about this was not absurd: in other languages, let a = 1 is an expression that evaluates to 1. Not in Javascript.

2. Functions are objects

However, the function keyword does have value: the Function object it defines. Unlike variables, which are language constructs for your convenience, Functions are actual objects that exist in the running program. We say that functions are first-class objects.

You can do all of these:

doStuff(function f() {})
let a = function f() {}
let b = (function f() {}) + 5 // the result of this is funny

Back to your examples, then:

callback(
  "", // a String object
  function b1() { alert("empty") }, // a Function object
  function b2() { alert("not empty") } // a Function object
);

Is similar to this:

function b1() { alert("empty") }
function b2() { alert("not empty") }

callback("", b1, b2)

But not quite. Let's talk about scopes.

3. Names are defined within a scope

The scope of a name, such as a variable or function, is the section(s) of code that have that definition available.

For example:

// Top-level scope:
let a = 1

if (a == 1) {
  // Inner block scope:
  let b = 2
  console.log(a, b) // 1, 2
}

console.log(a, b) // 1, undefined

Scopes live inside larger scopes. Inner scopes can access surrounding scopes, (so a and b are visible inside the block) but not the other way round (so b is not visible outside).

When you created your function objects inside the call...

f(function a() { })

... they were trapped inside an inner scope, and cannot be referenced from outside.

4. Assignments are expressions

In your sample code, you noted that declaring a like this worked:

f(a = 5)

This is... unfortunate. A product of Javascript's history, really. In modern code, you should always use let or const to define variables.

So why does it work? Two reasons. First, because it's an assignment, not a declaration. Like so:

let x = 1
f(x = 2)

Assignments are expressions. They evaluate to the assigned value. The value of x = 2 is 2, and x changes as a side-effect.

5. There is a global scope

The second reason is the unfortunate one. When you avoid the let, var or const keywords, you're implicitly using the global scope.

This is the mother of all scopes, and names that live there are accessible from any point in the code. So, if you just do this...

f(a = 5)

... without having declared a anywhere in the current scope, it's implicitly declared in the global scope, and the assignment takes place. Think of it as this (pseudo-code):

global let a
f(a = 5)

That is, of course, not valid Javascript. But you get the point.

The shortest way I can think of explaining the difference is if we convert the named function expression argument (the context where it is called is very important) in your example:

callback("", function b1() { console.log("empty"); }, ...)

to a variable declaration argument:

callback("", var b1 = function() { console.log("empty"); }, ...)

This gives an error:

function callback(str, f1, f2) {
    if(str == "")
        f1();
    else
        f2();
};

callback(
  "", 
  var b1 = function () { console.log("empty") }, 
  var b2 = function () { console.log("not empty") }
); // Error: expected expression, got keyword 'var' 

This works fine:

function callback(str, f1, f2) {
    if(str == "")
        f1();
    else
        f2();
};

callback(
  "", 
  function b1() { console.log("empty") }, 
  function b2() { console.log("not empty") }
); // logs 'empty'

Function declarations vs function expressions

They're almost the same, the MDN reference hierarchy is quite informative for the difference:

Operators/function (expression):

function [name]([param1[, param2[, ..., paramN]]]) {
  [statements]
}

Statements/function (definition):

function name([param1[, param2,[..., paramN]]]) {
   [statements]
}

See how an expression is listed under 'operators', whereas declaration is listed under 'statements'.

Note: I made slight adjustments to the MDN syntax to show how similar they are - they both have numbers after their parameters and for both declaration and expression the statements inside are optional.

What is important is where you call them:

function b1() {console.log('empty');} // declaration

callback(
  ""
  function b1() {console.log('empty');} // expression
)

Similarly a ternary (Operators/Conditional_Operator) str == "" ? f1 : f2 is an operator/expression where as if (Statements/if...else) if (str == "") { return f1 } else { return f2 } is a statement/declaration.

function callCallback(cb) {
  cb()
}

function callback(str, f1, f2) {
  callCallback(str == "" ? f1 : f2)
}

callback(
  "", 
  function b1() { console.log("empty") }, 
  function b2() { console.log("not empty") }
); // logs 'empty'

function callCallback(cb) {
  cb()
}

function callback(str, f1, f2) {
  callCallback(if (str == "") { return f1 } else { return f2 })
}

callback(
  "", 
  function b1() { console.log("empty") }, 
  function b2() { console.log("not empty") }
); // Error: expected expression, got keyword 'if'