Bind select list to complex object in ASP.NET Core

Using ASP.NET Core Razor Pages, is it possible to bind a select list to a complex object without having an ID property in the PageModel?

I have these model classes:

public class Department
{
    [HiddenInput]
    public int ID { get; set; }

    [Display(Name = "Name")]
    [Required(ErrorMessage = "{0} must be specified.")]
    [StringLength(50, MinimumLength = 1, ErrorMessage = "{0} has to be {2} to {1} characters long.")]
    public string Name { get; set; }
}

public class User
{
    [HiddenInput]
    public int ID { get; set; }

    [Display(Name = "Name")]
    [Required(ErrorMessage = "{0} must be specified.")]
    [StringLength(50, MinimumLength = 1, ErrorMessage = "{0} has to be {2} to {1} characters long.")]
    public string Name { get; set; }

    [Display(Name = "Department")]
    [Required(ErrorMessage = "{0} must be specified.")]
    public Department Department { get; set; }
}

I've create a view for editing a user, including selecting department. This is my PageModel:

public class EditModel : PageModel
{
    private readonly IUserRepository _userRepository;

    [BindProperty(BinderType = typeof(UserModelBinder))]
    public User UserToEdit { get; set; }

    public SelectList DepartmentsSL { get; set; }

    public EditModel(IUserRepository userRepository)
    {
        _userRepository = userRepository;
    }

    public void OnGet(int id)
    {
        UserToEdit = _userRepository.GetUser(id);
        PopulateSelectList();
    }

    public IActionResult OnPost()
    {
        if (!ModelState.IsValid)
        {
            PopulateSelectList();
            return Page();
        }

        var success = _userRepository.UpdateUser(UserToEdit);

        if (success)
        {
            return RedirectToPage("./Index");
        }
        else
        {
            throw new Exception("Could not save the user.");
        }
    }

    private void PopulateSelectList()
    {
        var departments = _userRepository.GetAllDepartments();
        DepartmentsSL = new SelectList(departments, nameof(Department.ID), nameof(Department.Name), UserToEdit.Department.ID);
    }
}

...and this is my view:

@page
@model ProjectName.Pages.Admin.Users.EditModel
@{
    ViewData["Title"] = "Edit user";
}

<h1>Edit user</h1>

<form method="post">
    <div asp-validation-summary="ModelOnly" class="text-danger"></div>

    <input asp-for="UserToEdit.ID" />

    <div class="form-group">
        <label asp-for="UserToEdit.Name" class="control-label"></label>
        <input asp-for="UserToEdit.Name" class="form-control" />
        <span asp-validation-for="UserToEdit.Name" class="text-danger"></span>
    </div>

    <div class="form-group">
        <label asp-for="UserToEdit.Department" class="control-label"></label>
        <select asp-for="UserToEdit.Department" class="form-control" asp-items="@Model.DepartmentsSL">
            <option value="">Choose department</option>
        </select>
        <span asp-validation-for="UserToEdit.Department" class="text-danger"></span>
    </div>

    <div class="form-group">
        <input type="submit" value="Save" class="btn btn-primary" />
        <a asp-page="./Index" class="btn btn-secondary">Cancel</a>
    </div>
</form>

As you might have noticed in the PageModel, I'm using a custom modelbinder. I wrote this so that the department of the user is set correctly when the view is posted back.

public class UserModelBinder : IModelBinder
{
    private readonly IUserRepository _userRepository;

    public UserModelBinder(IUserRepository userRepository)
    {
        _userRepository = userRepository;
    }

    public Task BindModelAsync(ModelBindingContext bindingContext)
    {
        if (bindingContext == null)
        {
            throw new ArgumentNullException("bindingContext");
        }

        var modelName = bindingContext.ModelName;

        var user = new User();

        // ID.
        var idValue = bindingContext.ValueProvider.GetValue(modelName + ".ID");
        if (idValue != ValueProviderResult.None)
        {
            bindingContext.ModelState.SetModelValue(modelName + ".ID", idValue);
            var idString = idValue.FirstValue;
            if (!String.IsNullOrEmpty(idString))
            {
                var id = Convert.ToInt32(idString);
                user.ID = id;
            }
        }

        // Department.
        var departmentValue = bindingContext.ValueProvider.GetValue(modelName + ".Department");
        if (departmentValue != ValueProviderResult.None)
        {
            bindingContext.ModelState.SetModelValue(modelName + ".Department", departmentValue);
            var departmentIdString = departmentValue.FirstValue;
            if (!String.IsNullOrEmpty(departmentIdString))
            {
                var departmentId = Convert.ToInt32(departmentIdString);
                user.Department = _userRepository.GetDepartment(departmentId);
            }
        }

        bindingContext.Result = ModelBindingResult.Success(user);
        return Task.CompletedTask;
    }
}

Now, all this seems to work fine - I can change the name and the department and it is posted back correctly. The validation also works. HOWEVER, when opening this edit view the current department of the user is NOT selected in the select list.

In EditModel.PopulateSelectList() I've tried different things, but none seem to work.

DepartmentsSL = new SelectList(departments, nameof(Department.ID), nameof(Department.Name), UserToEdit.Department);
DepartmentsSL = new SelectList(departments, nameof(Department.ID), nameof(Department.Name), UserToEdit.Department.ID);
DepartmentsSL = new SelectList(departments, nameof(Department.ID), nameof(Department.Name), UserToEdit.Department.Name);

Is it possible to solve this last part?

The solution I've seen so far has been to have a separate property "DepartmentID" in the PageModel and bind to that, and then setting the Department property of User after postback. I guess that would also mean I could get rid of the custom model binder, but I'd really like to see if my current solution could be taken all the way. :)