如何在url中處理urllib2.urlopen？

Question

我收到一個url：https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp®-75-desktop-virtualization-solutions;它來自BeautifulSoup。

url=u'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp\xae-75-desktop-virtualization-solutions' 

我想再次反饋到urllib2.urlopen。

import urllib2 
source = urllib2.urlopen(url).read() 

的錯誤，我得到：

UnicodeEncodeError: 'gbk' codec can't encode character u'\xae' in position 43: illegal multibyte sequence 

因此，我想：

source = urllib2.urlopen(url.encode("utf-8")).read() 

它讓網頁的源文件，但它是從原始地址從什麼不同。

originalUrl = 'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp®-75-desktop-virtualization-solutions' 
originalSource = urllib2.urlopen(originalUrl).read() 
originalSource == source 

結果是錯誤的。有沒有想法解決這個網址？如何將u'\ xae'轉換成原來的®？

Answer 1

URL必須是有效的bytestring，並且正確編碼非ASCII碼點。你需要編碼成UTF-8，那麼URL報價你的URL路徑：

import urllib 
import urllib2 
import urlparse 

originalUrl = u'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp\xae-75-desktop-virtualization-solutions' 
parsed_link = urlparse.urlsplit(originalUrl.encode('utf8')) 
parsed_link = parsed_link._replace(path=urllib.quote(parsed_link.path)) 
encoded_link = parsed_link.geturl() 
source = urllib2.urlopen(encoded_link).read() 

演示：

>>> import urllib 
>>> import urllib2 
>>> import urlparse 
>>> originalUrl = u'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp\xae-75-desktop-virtualization-solutions' 
>>> parsed_link = urlparse.urlsplit(originalUrl.encode('utf8')) 
>>> parsed_link = parsed_link._replace(path=urllib.quote(parsed_link.path)) 
>>> encoded_link = parsed_link.geturl() 
>>> encoded_link 
'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp%C2%AE-75-desktop-virtualization-solutions' 
>>> source = urllib2.urlopen(encoded_link).read() 
>>> len(source) 
68758