0
所以，後臺優先。我正在運行Ubuntu 14.04並運行以下腳本而沒有問題，其中我put文件在EC2實例上。 注意：我使用相同的IP來說明成功和失敗，但假裝我從兩次開始運行此腳本，每次都生成新的IP。EC2上的Python結構並行執行失敗：已更新
import boto.ec2
import os
from fabric.api import run, parallel, env, sudo
from fabric.tasks import execute
from fabric.operations import put
# file path python scripts and data
rps_file = "review_page_scraper.py"
# make sure hosts are clear before we add to them
env.hosts = []
# how many instances to start and how to split up the data frame
num_instances = 3
# EC2 access keys
access_key = 'my_access_key'
secret_key = 'my_secret_key'
# get a connection to the east region
conn = boto.ec2.connect_to_region("us-east-1",
aws_access_key_id=access_key,
aws_secret_access_key=secret_key)
# create the reservation of instances
reservation = conn.run_instances('my_ami_id',
key_name='original_key', # my original key
security_groups=['my_sec_grp'],
instance_type='t2.micro',
min_count=num_instances,
max_count=num_instances)
# get list of instances
instance_lst = reservation.instances
# get a status update and wait if the instance isn't up and running yet
for instance in instance_lst:
while instance.state != "running":
sleep(5)
instance.update()
print "%s is running" % instance.ip_address
# get username and host, add 'ubuntu' as username
hosts = ["[email protected]" + ip.ip_address for ip in instance_lst]
env.hosts = hosts # set environment variable
@parallel
def upload_scripts_data(file_name):
path = "~/amazon_proj/amazon/"
put(path + file_name, "~") # put it in the home dir of EC2 instance
# execute functions w/ rps_file
execute(upload_scripts_data, rps_file) # send review_page_scraper helpers
下面是輸出：
In [25]: execute(upload_scripts_data, rps_file) # send review_page_scraper helpers
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
Out[25]:
{u'[email protected]': None,
u'[email protected]': None,
u'[email protected]': None}
現在，問題： 我毀了我的Ubuntu的安裝和使用ssh-keygen -t rsa，我把它叫做 'original_key' 失去了我的密鑰對，生成的時候我將公鑰導入AWS。所以，我不得不重新安裝Ubuntu，我選擇了16.04。我使用ssh-keygen -t rsa生成了一個新密鑰，並分別將其保存到~/.ssh/id_rsa和〜/ .ssh/id_rsa.pub，分別用於私鑰和公鑰。
然後我導入了公鑰並用「id_rsa_pub」名稱保存。所以，現在我運行上面的同一個腳本，將key_name參數更改爲「id_rsa_pub」。另外，我根據AWS的指示跑了chmod 0400 id_rsa。該ouptut是：
In [22]: execute(upload_scripts_data, rps_file)
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
!!! Parallel execution exception under host u'[email protected]':
Process [email protected]:
Traceback (most recent call last):
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 242, in inner
submit(task.run(*args, **kwargs))
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
Traceback (most recent call last):
File "<ipython-input-22-ed18eb00cc62>", line 1, in <module>
execute(upload_scripts_data, rps_file)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 412, in execute
ran_jobs = jobs.run()
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/job_queue.py", line 168, in run
self._fill_results(results)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/job_queue.py", line 191, in _fill_results
datum = self._comms_queue.get_nowait()
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/queues.py", line 152, in get_nowait
return self.get(False)
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/queues.py", line 135, in get
res = self._recv()
TypeError: ('__init__() takes exactly 2 arguments (3 given)', <class 'paramiko.ssh_exception.NoValidConnectionsError'>, (None, 'Unable to connect to port 22 on or 54.165.186.168'))
return func(*args, **kwargs)
File "<ipython-input-19-ed4344124d24>", line 4, in upload_scripts_data
put(path + file_name, "~")
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/operations.py", line 345, in put
ftp = SFTP(env.host_string)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/sftp.py", line 33, in __init__
self.ftp = connections[host_string].open_sftp()
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 603, in connect
raise NetworkError(msg, e)
NetworkError: Low level socket error connecting to host 54.165.186.168 on port 22: Unable to connect to port 22 on or 54.165.186.168 (tried 1 time)
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
!!! Parallel execution exception under host u'[email protected]':
Process [email protected]:
Traceback (most recent call last):
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 242, in inner
submit(task.run(*args, **kwargs))
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
return func(*args, **kwargs)
File "<ipython-input-19-ed4344124d24>", line 4, in upload_scripts_data
put(path + file_name, "~")
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/operations.py", line 345, in put
ftp = SFTP(env.host_string)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/sftp.py", line 33, in __init__
self.ftp = connections[host_string].open_sftp()
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 603, in connect
raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to 54.173.57.59 (tried 1 time)
我對這樣一個詳細的問題和輸出道歉。我用盡了我的知識，沒有看到任何與我的問題完全相同的在線內容。
更新：
有幾件事要注意。我正在使用與舊密鑰對一樣使用的安全組和AMI。接下來，更令人困惑的是，如果我再次運行execute(upload_scripts_data, rps_file)命令並且它沒有錯誤地運行。
檢查安全組，例如使用ip'54.173.57.59'。它允許端口22上的傳入TCP連接？它運行的是ssh守護進程嗎？ –
謝謝@NehalJWani，我會回覆上面的更新，請看看。 –