Character folding for a Django haystack and whoosh

I have a django based app with haystack and whoosh search engine. I want to provide an accent and special character independent search so that I can find indexed data with special characters also by using words without special chars:

Indexed is:

'café'

Search term:

'cafe'  
'café'

I've written a provided a specific FoldingWhooshSearchBackend which uses a StemmingAnalyzer and aCharsetFilter(accent_map) as described in the following document:

https://gist.github.com/gregplaysguitar/1727204

However the search still doesn't work like expected, i.e. I cannot search with 'cafe' and find 'café'. I've looked into the search index using:

from whoosh.index import open_dir
ix = open_dir('myservice/settings/whoosh_index')
searcher = ix.searcher()
for doc in searcher.documents():
    print doc

The special characters are still in the index.

Do I have to do something additional? Is is about changing the index template?