I have created a script that scrape website: 1688.com and the problem is, the site is in Chinese so whenever i try to retrieve the text, it gives me a bunch of unicode and when i export to a CSV file, there's nothing in the file. My code:
# -*- coding: utf-8 -*-
import csv
from urllib import urlopen
from bs4 import BeautifulSoup as BS
csv_content = open('content.csv', 'w+')
writer_content = csv.writer(csv_content)
url = urlopen('https://fuzhuang.1688.com/nvzhuang?
spm=a260k.635.1998214976.1.7eqUGT')
html = BS(url, 'lxml')
container = html.find('ul', {'class' : 'ch-box fd-clr'})
offers = container.find_all('div', {'class' : 'ch-offer-body'})
lst = []
for offer in offers:
offer_box = offer.find('div', {'component-name' : '@alife/ocms-
component-1688-pc-ch-offer-pic'})
images = offer_box.find('img')['src']
title = offer.find('div', {'class' : 'ocms-component-1688-pc-ch-offer-
title-0-1-11'}).text
price = offer.find('div', {'class' : 'ocms-component-1688-pc-ch-offer-
price-0-1-14'}).text
lst.append(price)
for item in lst: writer_content.writerow([item])
print lst
The output is
[u'\n\n\n\xa5\n109.00\n\n\n\u6210\u4ea4\n329\n\u4ef6\n\n\n', u'\n\n\n\xa5\n56.00\n\n\n\u6210\u4ea4\n195\n\u4ef6\n\n\n', u'\n\n\n\xa5\n83.00\n\n\n\u6210\u4ea4\n109\n\u4ef6\n\n\n', u'\n\n\n\xa5\n69.00\n\n\n\u6210\u4ea4\n208\n\u4ef6\n\n\n', u'\n\n\n\xa5\n46.00\n\n\n\u6210\u4ea4\n204\n\u4ef6\n\n\n', u'\n\n\n\xa5\n45.00\n\n\n\u6210\u4ea4\n54\n\u4ef6\n\n\n', u'\n\n\n\xa5\n82.00\n\n\n\u6210\u4ea4\n38\n\u4ef6\n\n\n', u'\n\n\n\xa5\n48.90\n\n\n\u6210\u4ea4\n318\n\u4ef6\n\n\n']
And i have already tried encoding and decoding utf-8, i would really appreciate it if you show me how to solve this problem.