Best way to handle security and avoid XSS with user entered URLs

If you think URLs can't contain code, think again!

https://owasp.org/www-community/xss-filter-evasion-cheatsheet

Read that, and weep.

Here's how we do it on Stack Overflow:

/// <summary>
/// returns "safe" URL, stripping anything outside normal charsets for URL
/// </summary>
public static string SanitizeUrl(string url)
{
    return Regex.Replace(url, @"[^-A-Za-z0-9+&@#/%?=~_|!:,.;\(\)]", "");
}

The process of rendering a link "safe" should go through three or four steps:

• Unescape/re-encode the string you've been given (RSnake has documented a number of tricks at http://ha.ckers.org/xss.html that use escaping and UTF encodings).
• Clean the link up: Regexes are a good start - make sure to truncate the string or throw it away if it contains a " (or whatever you use to close the attributes in your output); If you're doing the links only as references to other information you can also force the protocol at the end of this process - if the portion before the first colon is not 'http' or 'https' then append 'http://' to the start. This allows you to create usable links from incomplete input as a user would type into a browser and gives you a last shot at tripping up whatever mischief someone has tried to sneak in.
• Check that the result is a well formed URL (protocol://host.domain[:port][/path][/[file]][?queryField=queryValue][#anchor]).
• Possibly check the result against a site blacklist or try to fetch it through some sort of malware checker.

If security is a priority I would hope that the users would forgive a bit of paranoia in this process, even if it does end up throwing away some safe links.

Just HTMLEncode the links when you output them. Make sure you don't allow javascript: links. (It's best to have a whitelist of protocols that are accepted, e.g., http, https, and mailto.)

You don't specify the language of your application, I will then presume ASP.NET, and for this you can use the Microsoft Anti-Cross Site Scripting Library

It is very easy to use, all you need is an include and that is it :)

While you're on the topic, why not given a read on Design Guidelines for Secure Web Applications

If any other language.... if there is a library for ASP.NET, has to be available as well for other kind of language (PHP, Python, ROR, etc)

For Pythonistas, try Scrapy's w3lib.

OWASP ESAPI pre-dates Python 2.7 and is archived on the now-defunct Google Code.

How about not displaying them as a link? Just use the text.

Combined with a warning to proceed at your own risk may be enough.

addition - see also Should I sanitize HTML markup for a hosted CMS? for a discussion on sanitizing user input

There is a library for javascript that solves this problem https://github.com/braintree/sanitize-url Try it =)

In my project written in JavaScript I use this regex as white list:

 url.match(/^((https?|ftp):\/\/|\.{0,2}\/)/)

the only limitation is that you need to put ./ in front for files in same directory but I think I can live with that.

Using Regular Expression to prevent XSS vulnerability is becoming complicated thus hard to maintain over time while it could leave some vulnerabilities behind. Having URL validation using regular expression is helpful in some scenarios but better not be mixed with vulnerability checks.

Solution probably is to use combination of an encoder like AntiXssEncoder.UrlEncode for encoding Query portion of the URL and QueryBuilder for the rest:

    public sealed class AntiXssUrlEncoder
    {
        public string EncodeUri(Uri uri, bool isEncoded = false)
        {
            // Encode the Query portion of URL to prevent XSS attack if is not already encoded. Otherwise let UriBuilder take care code it.
            var encodedQuery = isEncoded ? uri.Query.TrimStart('?') : AntiXssEncoder.UrlEncode(uri.Query.TrimStart('?'));
            var encodedUri = new UriBuilder
            {
                Scheme = uri.Scheme,
                Host = uri.Host,
                Path = uri.AbsolutePath,
                Query = encodedQuery.Trim(),
                Fragment = uri.Fragment
            };
            if (uri.Port != 80 && uri.Port != 443)
            {
                encodedUri.Port = uri.Port;
            }

            return encodedUri.ToString();
        }

        public static string Encode(string uri)
        {
            var baseUri = new Uri(uri);
            var antiXssUrlEncoder = new AntiXssUrlEncoder();
            return antiXssUrlEncoder.EncodeUri(baseUri);
        }
    }

You may need to include white listing to exclude some characters from encoding. That could become helpful for particular sites. HTML Encoding the page that render the URL is another thing you may need to consider too.

BTW. Please note that encoding URL may break Web Parameter Tampering so the encoded link may appear not working as expected. Also, you need to be careful about double encoding

P.S. AntiXssEncoder.UrlEncode was better be named AntiXssEncoder.EncodeForUrl to be more descriptive. Basically, It encodes a string for URL not encode a given URL and return usable URL.

Allowing a URL and allowing JavaScript are 2 different things.