Parsing URL with port and without scheme

Viewed 7872

I am trying to parse a URL in Go and get host and scheme from the URL. But while parsing the URL with port and without scheme I am getting unexpected result.

u, err := url.ParseRequestURI("hello.com:81")
fmt.Println("host :",u.Host)
fmt.Println("scheme :",u.Scheme)

I am getting unexpected result

host :
scheme: hello.com

I wanted this instead

host : hello.com:80
scheme:
5 Answers

According to go doc, The general url form represented is:

[scheme:][//[userinfo@]host][/]path[?query][#fragment]

URLs that do not start with a slash after the scheme are interpreted as:

scheme:opaque[?query][#fragment]

Your URL is parsed as the second format.

You can use this method to get the result expected as yours. In the function, if there is no scheme in URL we add it then again parse it to get the expected result.

func parseRawURL(rawurl string) (domain string, scheme string, err error) {
    u, err := url.ParseRequestURI(rawurl)
    if err != nil || u.Host == "" {
        u, repErr := url.ParseRequestURI("https://" + rawurl)
        if repErr != nil {
            fmt.Printf("Could not parse raw url: %s, error: %v", rawurl, err)
            return
        }
        domain = u.Host
        err = nil
        return
    }

    domain = u.Host
    scheme = u.Scheme
    return
}

You can try it out in go playground

If you need to handle URLs that only include host and port (without scheme and other parameters) you can use this code:

   host, port, err := net.SplitHostPort("hello.com:81")
   fmt.Println("host:", host, "port:", port, "err:", err)
   // output: host: hello.com port: 81 err <nil>

Note that SplitHostPort() is not suitable for parsing standard URLs (conforming with [scheme:][//[userinfo@]host][/]path[?query][#fragment])

The format defined in the net.URL docs is one of the following:

  • [scheme:][//[userinfo@]host][/]path[?query][#fragment]
  • scheme:opaque[?query][#fragment]

The scheme: is optional, but the double slash is part of the host field. This means that a valid string for your input would be: //hello.com:81 which results in:

u, _ := url.Parse("//hello.com:81")
fmt.Println("host:", u.Host)
// Output: host: hello.com:81

You'll need to turn your input into something valid. If you know that the string never includes the scheme, you can simply prepend //. If you only sometimes have a scheme specified, you can try to conditionally manipulate the input.

Trying to parse a hostname and path without a scheme is invalid but may not necessarily return an error, due to parsing ambiguities.

— documentation for net/url.Parse

What you have passed to ParseRequestURI isn't a URI, or at least not one that means what you think it means. If you had passed http://hello.com:81/ then the host would have been set as expected.

You need to add scheme if you want to parse it with `ParseRequestURI``

acording docu :

ParseRequestURI parses rawurl into a URL structure. It assumes that rawurl was received in an HTTP request, so the rawurl is interpreted only as an absolute URI or an absolute path. The string rawurl is assumed not to have a #fragment suffix. (Web browsers strip #fragment before sending the URL to a web server.)

So hostname:81 is invalid url.

so try to do this :

u, err := url.ParseRequestURI("http://" + "hello.com:80")
fmt.Println("err:",err)
fmt.Println("host :",u.Host)
fmt.Println("scheme :",u.Scheme)

or even in your case Parse is more correct:

u, err := url.Parse("http://" + "hello.com:80")
fmt.Println("err:",err)
fmt.Println("host :",u.Host)
fmt.Println("scheme :",u.Scheme)
Related