解析URL美麗

import requests 
import csv 
from bs4 import BeautifulSoup 
page = requests.get("https://www.google.com/search?q=cars") 
soup = BeautifulSoup(page.content, "lxml") 
import re 
links = soup.findAll("a") 
with open('aaa.csv', 'wb') as myfile: 
    for link in soup.find_all("a",href=re.compile("(?<=/url\?q=)(htt.*://.*)")): 
     a = (re.split(":(?=http)",link["href"].replace("/url?q=",""))) 
     wr = csv.writer(myfile, quoting=csv.QUOTE_ALL) 
     wr.writerow(a)

此代碼的輸出是，我有一個CSV文件，其中28個URL保存，但URL不正確。例如，這是一個錯誤的URL： -解析URL美麗

http://www.imdb.com/title/tt0317219/&sa=U&ved=0ahUKEwjg5fahi7nVAhWdHsAKHSQaCekQFgg9MAk&usg=AFQjCNFu_Vg9v1oVhEtR-vKqCJsR2YGd2A

相反，它應該是： -

http://www.imdb.com/title/tt0317219/

如何刪除第二個是對每一個URL，如果它包含"&sa="

因爲那麼URL的第二部分應從以下位置開始移除： - "&sa="應該被移除，以便所有網址都保存爲第二個網址。

我使用python 2.7和Ubuntu 16.04。

來源

2017-08-02 Tanja Meeren

你正在使用什麼？你在這裏使用正則表達式的目的是什麼？ – Mohamed

如果URL的每一次冗餘部分與&開始，你可以申請split()的每個網址：

url = 'http://www.imdb.com/title/tt0317219/&sa=U&ved=0ahUKEwjg5fahi7nVAhWdHsAKHSQaCekQFgg9MAk&usg=AFQjCNFu_Vg9v1oVhEtR-vKqCJsR2YGd2A' 
url = url.split('&')[0] 
print(url)

輸出：

http://www.imdb.com/title/tt0317219/

來源

2017-08-02 18:19:36

簡單而有效！非常感謝！ –

不是最好的方式，但你可以做一個更時間分割，在a後添加一行：

a=[a[0].split("&")[0]] 
print(a)

結果：

['https://de.wikipedia.org/wiki/Cars_(Film)'] 
['http://webcache.googleusercontent.com/search%3Fq%3Dcache:I2SHYtLktRcJ'] 
['https://de.wikipedia.org/wiki/Cars_(Film)%23Handlung'] 
['https://de.wikipedia.org/wiki/Cars_(Film)%23Synchronisation'] 
['https://de.wikipedia.org/wiki/Cars_(Film)%23Soundtrack'] 
['https://de.wikipedia.org/wiki/Cars_(Film)%23Kritik'] 
['https://www.mytoys.de/disney-cars/'] 
['http://webcache.googleusercontent.com/search%3Fq%3Dcache:9Ohx4TRS8KAJ'] 
['https://www.youtube.com/watch%3Fv%3DtNmo09Q3F8s'] 
['https://www.youtube.com/watch%3Fv%3DtNmo09Q3F8s'] 
['https://www.youtube.com/watch%3Fv%3DkLAnVd5y7M4'] 
['https://www.youtube.com/watch%3Fv%3DkLAnVd5y7M4'] 
['http://cars.disney.com/'] 
['http://webcache.googleusercontent.com/search%3Fq%3Dcache:1BoR6M9fXwcJ'] 
['http://cars.disney.com/'] 
['http://cars.disney.com/'] 
['https://www.whichcar.com.au/car-style/12-cartoon-cars'] 
['https://www.youtube.com/watch%3Fv%3D6JSMAbeUS-4'] 
['http://filme.disney.de/cars-3-evolution'] 
['http://webcache.googleusercontent.com/search%3Fq%3Dcache:fO7ypFFDGk0J'] 
['http://www.4players.de/4players.php/spielinfonews/Allgemein/36859/2169193/Project_CARS_2-Zehn_Ferraris_erweitern_den_virtuellen_Fuhrpark.html'] 
['http://www.4players.de/4players.php/spielinfonews/Allgemein/36859/2169193/Project_CARS_2-Zehn_Ferraris_erweitern_den_virtuellen_Fuhrpark.html'] 
['http://www.play3.de/2017/08/02/project-cars-2-6/'] 
['http://www.imdb.com/title/tt0317219/'] 
['http://webcache.googleusercontent.com/search%3Fq%3Dcache:-xdXy-yX2fMJ'] 
['http://www.carmagazine.co.uk/'] 
['http://webcache.googleusercontent.com/search%3Fq%3Dcache:PRPbHf_kD9AJ'] 
['http://google.com/search%3Ftbm%3Disch%26q%3DCars'] 
['http://www.imdb.com/title/tt0317219/'] 
['https://de.wikipedia.org/wiki/Cars_(Film)']

來源

2017-08-02 18:24:22

簡單而有效！非常感謝！ –

解析URL美麗

回答

相關問題