The main requirement to keep in mind for for_each is that we always need to create a map that has one element for each instance of the resource we want to create. In this case, that means you need a map with one element per workspace per variable, because tfe_variable describes a single variable on a single workspace.
Our job then is to write an expression to project the map-of-objects value coming in via the variable to a collection that has a separate element per variable. Here's one way to get that done, using the flatten function in a way similar to an example in its documentation:
locals {
workspace_variables = flatten([
for ws_name, ws in var.custom_variables : [
for var_name, value in ws : {
workspace = ws_name
variable = var_name
value = value
}
]
])
}
The above should produce a local.workspace_variables that looks like this:
[
{ workspace = "wksp1", variable = "node_count", value = 2 },
{ workspace = "wksp1", variable = "vm_size", value = "Standard_D2_v3" },
{ workspace = "wksp2", variable = "node_count", value = 5 },
{ workspace = "wksp2", variable = "vm_size", value = "Standard_D2_v5" },
]
This now meets the requirement of having one element per desired tfe_variable instance, so our only remaining job is to project this into a map to provide unique identifiers for each element and describe how to populate the tfe_variable arguments based on these objects:
resource "tfe_variable" "custom" {
for_each = {
for wsv in local.workspace_variables : "${wsv.workspace}.${wsv.variable}" => wsv
}
key = each.value.variable
value = each.value.value
category = "terraform"
workspace_id = each.value.workspace
}
One thing I didn't contend with above, because it wasn't directly you question, is the value of workspace_id in tfe_variable. If I recall correctly, that argument is expecting a workspace id rather than a workspace name, in which case you might need a slightly more complicated expression for the workspace_id argument. If you already have a tfe_workspace resource using the workspace names as keys then something like this might work, for example:
workspace_id = tfe_workspace.example[each.value.workspace].id
If your workspaces are created in a different way then you may have to do something more complicated here, but that's getting far off the topic of your original question so I won't try to dig into that here. I'm happy to try to help with it in a separate question on this site though, if you like.