How to make Django slugify work properly with Unicode strings?

I'm afraid django's definition of slug means ascii, though the django docs don't explicitly state this. This is the source of the defaultfilters for the slugify... you can see that the values are being converted to ascii, with the 'ignore' option in case of errors:

import unicodedata
value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')
value = unicode(re.sub('[^\w\s-]', '', value).strip().lower())
return mark_safe(re.sub('[-\s]+', '-', value))

Based on that, I'd guess that cnprog.com is not using an official slugify function. You may wish to adapt the django snippet above if you want a different behaviour.

Having said that, though, the RFC for URLs does state that non-us-ascii characters (or, more specifically, anything other than the alphanumerics and $-_.+!*'()) should be encoded using the %hex notation. If you look at the actual raw GET request that your browser sends (say, using Firebug), you'll see that the chinese characters are in fact encoded before being sent... the browser just makes it look pretty in the display. I suspect this is why slugify insists on ascii only, fwiw.

This is what I use:

http://trac.django-fr.org/browser/site/trunk/djangofr/links/slughifi.py

SlugHiFi is a wrapper for regular slugify, with a difference that it replaces national chars with their English alphabet counterparts.

So instead of "Ą" you get "A", instead of "Ł" => "L", and so on.

I am interested in allowing only ASCII characters in the slug this is why I tried to benchmark some of the available tools for the same string:

• Unicode Slugify:

  In [5]: %timeit slugify('Παίζω τρέχω %^&*@# και γ%^(λώ la fd/o', only_ascii=True)
  37.8 µs ± 86.7 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  
  'paizo-trekho-kai-glo-la-fdo'
  

• Django Uuslug:

  In [3]: %timeit slugify('Παίζω τρέχω %^&*@# και γ%^(λώ la fd/o')
  35.3 µs ± 303 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  
  'paizo-trekho-kai-g-lo-la-fd-o'
  

• Awesome Slugify:

  In [3]: %timeit slugify('Παίζω τρέχω %^&*@# και γ%^(λώ la fd/o')
  47.1 µs ± 1.94 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  
  'Paizo-trekho-kai-g-lo-la-fd-o'
  

• Python Slugify:

  In [3]: %timeit slugify('Παίζω τρέχω %^&*@# και γ%^(λώ la fd/o')
  24.6 µs ± 122 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  
  'paizo-trekho-kai-g-lo-la-fd-o'
  

• django.utils.text.slugify with Unidecode:

  In [15]: %timeit slugify(unidecode('Παίζω τρέχω %^&*@# και γ%^(λώ la fd/o'))
  36.5 µs ± 89.7 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  
  'paizo-trekho-kai-glo-la-fdo'