我想直接導入網絡報廢的數據到PostgreSQL中,而不是先導出到.csv。從網站直接導入抓取的數據到PostgreSQL
下面是我正在使用的代碼,將數據導出到.csv文件,然後我手動導入它。任何幫助,將不勝感激
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
my_url = 'http://tis.nhai.gov.in/TollInformation?TollPlazaID=236'
uClient = uReq(my_url)
page1_html = uClient.read()
uClient.close()
#html parsing
page1_soup = soup(page1_html,"html.parser")
filename = "TollDetail12.csv"
f = open(filename,"w")
headers = "ID, tollname, location, highwayNumber\n"
f.write(headers)
#grabing data
containers = page1_soup.findAll("div",{"class":"PA15"})
for container in containers:
toll_name = container.p.b.text
search1 = container.findAll('b')
highway_number = search1[1].text
location = list(container.p.descendants)[10]
ID = my_url[my_url.find("?"):]
mystr = ID.strip("?")
print("ID: " + mystr)
print("toll_name: " + toll_name)
print("location: " + location)
print("highway_number: " + highway_number)
f.write(mystr + "," + toll_name + "," + location + "," + highway_number.replace(",","|") + "\n")
f.close()
[在數據進入的PostgreSQL插入(http://www.postgresqltutorial.com/postgresql-python/insert/)閱讀。它會幫助你解決你的問題。 –