Go: Unmarshalling JSON with multiple types

I'm having an issue unmarshalling a JSON response into a struct. The problem I'm having is that the zip code can either return as a string or an integer. How do I write an unmarshal method to check if the zip is an int and force it to store it as a string?

Struct:

type CustomerAddress struct {
    Line1            string `json:"line1"`
    City             string `json:"city"`
    State            string `json:"state"`
    Zip              string `json:"zip"`
    IsPrimaryAddress string `json:"isPrimaryAddress"`
}

Example Json:

address": [
  {
    "line1": "555 ADDRESS PLACE",
    "city": "DALLAS",
    "state": "TX",
    "isPrimaryAddress": "Y",
    "zip": 55555
  }
]

After unmarshalling, the result should have the zip successfully converted into a string:

address": [
  {
    "line1": "555 ADDRESS PLACE",
    "city": "DALLAS",
    "state": "TX",
    "isPrimaryAddress": "Y",
    "zip": "55555"
  }
]

As an attempt, I tried to use a ZipWrapper.

type CustomerAddress struct {
    Line1            string        `json:"line1"`
    City             string        `json:"city"`
    State            string        `json:"state"`
    Zip              ZipWrapper    `json:"zip"`
    IsPrimaryAddress string        `json:"isPrimaryAddress"`
}

type ZipWrapper struct {
   Zip string
}

func (w *ZipWrapper ) UnmarshalJSON(data []byte) (err error) {

    if zip, err := strconv.Atoi(string(data)); err == nil {
        w.Zip = strconv.Itoa(zip)
        return nil
    }
    return json.Unmarshal(data, &w.Zip)
}

This almost worked except the zip is now a nested struct within CustomerAddress which is not what I want:

  address": [
  {
    "line1": "555 ADDRESS PLACE",
    "city": "DALLAS",
    "state": "TX",
    "isPrimaryAddress": "Y",
    "zip": {
      "Zip": "55555"
    }
  }
]

Any ideas? I feel like this is a relatively easy task but I'm a complete Go noob and haven't fully wrapped my head around how Unmarshalling works.