qmk: Make a layer respond to two modifier keys being pressed?

You have to put your specific code into the custom Quantum function process_record_user:

bool process_record_user(uint16_t keycode, keyrecord_t *record) {
    // enter your code for [Fn][Ctrl]-[α] here
    return true;
}

You can try the Quantum tab in the Firmware Builder. Even if it is end-of-life, you'll see, what is meant.

You can also set UNICODEMAP_ENABLE = yes and use the Unicode Map.

Something like this should do (half) the trick:

In function layer_state_set_user, check for layer Fn (some bit arithmetic similar to update_tri_layer_state) and the Ctrl state (get_mods() & MOD_MASK_CTRL). When in Fn layer and Ctrl is down, deactivate Ctrl (del_mods(MOD_MASK_CTRL)) and return state | x; otherwise return state & ~x.

Unfortunately, this will probably require Ctrl to be pressed before Fn.

At least that's what happens in my similar shift + layer _L4 => layer _S_L4 implementation in my keymap. It requires shift to be pressed before LT(_L4,…).

For the other direction (_L4 + shift => _S_L4), I have MO(_S_L4) on the shift keys in layer _L4, but that is somehow disabled by my layer_state_set_user.

What should "Fn + E" and "Fn + D" do (when Ctrl is not held)? For sake of example, I'll assume you want them to do PgUp and PgDn.

Here is how I would go about implementing this:

1. Enable Unicode input: In rules.mk, add UNICODEMAP_ENABLE = yes. In config.h, add #define UNICODE_SELECTED_MODES UC_WINC if you are on Windows. See the Unicode documentation for other OSs and options.
2. Define an "Fn" layer in the keymap.
3. Define a couple custom keycodes, and place them in the Fn layer at the d and e positions.
4. Handle the custom keys in process_record_user(), using get_mods() to test whether Ctrl is held. See also macros that respond to mods. Then use send_unicode_string() to type the Unicode symbol itself.

Code sketch:

// Copyright 2022 Google LLC.
// SPDX-License-Identifier: Apache-2.0

enum layers { BASE, FN, /* Other layers... */ }; 

enum custom_keycodes {
  UPDIR = SAFE_RANGE,
  FN_E,
  FN_D,
  // Other custom keys...
};

const uint16_t keymaps[][MATRIX_ROWS][MATRIX_COLS] PROGMEM = {
  [BASE] = LAYOUT(...),
  [FN] = LAYOUT(..., FN_E, ..., FN_D, ...),
  ...
};

const uint32_t unicode_map[] PROGMEM = {};

static void symbol_key(uint16_t* registered_key,
                       keyrecord_t* record,
                       uint16_t default_keycode,
                       const char* symbol,
                       const char* uppercase_symbol) {
  if (record->event.pressed) {  // On key press.
    const uint8_t mods = get_mods();
    const bool ctrl_held = (mods & MOD_MASK_CTRL) != 0;
    const bool shifted = (mods & MOD_MASK_SHIFT) != 0;

    if (ctrl_held) {  // Is the Ctrl mod held?
      unregister_mods(MOD_MASK_CTRL);  // Clear the Ctrl mod.
      // Type Unicode symbol according to whether shift is active.
      send_unicode_string(shifted ? uppercase_symbol : symbol);
      *registered_key = KC_NO;
    } else {
      *registered_key = default_keycode;
      register_code16(*registered_key);
    }
  } else {  // On key release.
    unregister_code16(*registered_key);
    *registered_key = KC_NO;
  } 
}

bool process_record_user(uint16_t keycode, keyrecord_t* record) {
  const uint8_t mods = get_mods();
  const bool ctrl_held = (mods & MOD_MASK_CTRL) != 0;
  const bool shifted = (mods & MOD_MASK_SHIFT) != 0;
  
  switch (keycode) {
    case FN_E: {
      static uint16_t registered_key = KC_NO;
      symbol_key(&registered_key, record, KC_PGUP, "ε", "Ε");
    } return false;  
    case FN_D: {
      static uint16_t registered_key = KC_NO;
      symbol_key(&registered_key, record, KC_PGDN, "δ", "Δ");
    } return false;  
  }
  return true;
}