的Python - ValueError異常：未知的URL類型

Question

我試圖從<iframes>屬性，像這樣的提取來源：

iframes = [<iframe frameborder="no" height="160px" scrolling="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/308197184%3Fsecret_token%3Ds-VtArH&amp;auto_play=false&amp;hide_related=false&amp;show_comments=true&amp;show_user=true&amp;show_reposts=false&amp;visual=true" width="100%"></iframe>, <iframe allowtransparency="true" frameborder="0" scrolling="no" src="//www.facebook.com/plugins/likebox.php?href=https%3A%2F%2Fwww.facebook.com%2FPauseMusicale&amp;width=300&amp;height=62&amp;show_faces=false&amp;colorscheme=light&amp;stream=false&amp;show_border=false&amp;header=false" style="border:none; overflow:hidden; width:300px; height:62px;"></iframe>, <iframe allowfullscreen="" frameborder="0" height="169" src="//www.youtube.com/embed/videoseries?list=PLNKCTdT9YSESoQnj5tPP4P9kaIwBCx7F1" width="100%"></iframe>] 

但是當我嘗試將其解壓：

for iframe in iframes: 
    url = urllib2.urlopen(iframe.attrs['src']) 
    print (url) 

我得到以下錯誤：

url = urllib2.urlopen(iframe.attrs['src']) 
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen 
    return opener.open(url, data, timeout) 
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 423, in open 
    protocol = req.get_type() 
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 285, in get_type 
    raise ValueError, "unknown url type: %s" % self.__original 
ValueError: unknown url type: //www.facebook.com/plugins/likebox.php?href=https%3A%2F%2Fwww.facebook.com%2FPauseMusicale&width=300&height=62&show_faces=false&colorscheme=light&stream=false&show_border=false&header=false 

爲什麼我收到的網址沒有http的前？

有沒有一些解決方法呢？

Answer 1

why am I getting url with no http before the //www

這是爲了表示對進行以後的請求時，它應該使用相同的方案（HTTP，HTTPS，FTP，文件等）作爲當前頁面的用戶代理的常用方法。因此，例如，如果您通過https加載當前頁面，那麼那些忽略該方案的URL將通過https訪問。

Is there some workaround this?

可以使用urlparse模塊在Python 2處理這個（因爲這是你的Python版本）：

# from urllib.parse import urlparse, urlunparse # Python 3 
from urlparse import urlparse, urlunparse 

for iframe in iframes: 
    scheme, netloc, path, params, query, fragment = urlparse(iframe.attrs['src']) 
    if not scheme: 
     scheme = 'http' # default scheme you used when getting the current page 
    url = urlunparse((scheme, netloc, path, params, query, fragment)) 
    print('Fetching {}'.format(url)) 
    f = urllib2.urlopen(url) 
# print(f.read()) # dumps the response content 

如果你運行上面的代碼，你會看到這樣的輸出：

 
Fetching https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/308197184%3Fsecret_token%3Ds-VtArH&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=true 
Fetching http://www.facebook.com/plugins/likebox.php?href=https%3A%2F%2Fwww.facebook.com%2FPauseMusicale&width=300&height=62&show_faces=false&colorscheme=light&stream=false&show_border=false&header=false 
Fetching http://www.youtube.com/embed/videoseries?list=PLNKCTdT9YSESoQnj5tPP4P9kaIwBCx7F1 

它顯示默認方案已被應用到URL。