How to custom unmarshall nested XML

I have my types as below:

type Address struct {
    Street string `xml:"street,omitempty"`
    City   string `xml:"city,omitempty"`
}

type User struct {
    XMLName xml.Name `xml:"user,omitempty"`
    Name    string   `xml:"name,omitempty"`
    Address Address  `xml:"address,omitempty"`
}

And I need to do a custom unmarshalling for them.

user := &User{}
byt := []byte(`<user><name>John</name><address><street>Domlur</street><city>Bangalore</city></address></user>`)
xml.Unmarshal(byt, user)
fmt.Printf("+%v", user)

I want to implement the unmarshaller interface. Reason is for each and every field in User, I need a specific action to be taken, so I want to have some conditional checks in there. However, the ones in the nested Address XML part can be unmarshalled directly to the Address struct.

I have something like below.

func (u *User) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
    alias := User{}
    for {
        t, err := d.Token()
        if err == io.EOF {
            break
        }

        switch token := t.(type) {

        case xml.StartElement:
            if token.Name.Local == "address" {
                addr := Address{}
                // Now here's where I am totally lost. How am I to unmarshall the Address part? I tried below line, but it doesn't work.
                d.DecodeElement(&addr, &start)
                alias.Address = addr
                continue
            }

            if token.Name.Local == "name" {
                var value string
                d.DecodeElement(&value, &start)
                alias.Name = value
                continue
            }
            
        case xml.EndElement:
            var value string
            d.DecodeElement(&value, &start)
        case xml.CharData:
            var value string
            d.DecodeElement(&value, &start)
        }
    }
    *u = User(alias)
    return nil
}