Google PubSub返回google.gax.errors.GaxError：GaxError RPC失敗，原因是... StatusCode.UNAVAILABLE

Question

我們試圖在分佈式系統之一發生事件後，對現有主題進行簡單發佈。

代碼如下：

try: 
    dat = data.encode('utf-8') 
    topic.publish(dat) 
except: 
    <code to recover> 

如果我們用捕獲所有的，除了和打印追溯，我們得到：

google.gax.errors.GaxError: GaxError(RPC failed, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, {"created":"@1478711654.067744009","description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_status":14,"referenced_errors":[{"created":"@1478711654.067706801","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]})>

（以下完整的錯誤）

看着http://gcloud-python.readthedocs.io/en/latest/pubsub-topic.html#google.cloud.pubsub.topic.Topic.publish，它似乎並不像這個GAX錯誤是我們應該尋找的東西。但是，如果我們做捕獲錯誤並使用指數退避重試，通常這是第二次。

我找到了this discussion，雖然它說的是_gax_python中的一個潛在錯誤，但它似乎並不相關。關於我們在這裏可能會做錯什麼的想法？

完整的錯誤：

458 Traceback (most recent call last): 
    459  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner 
    460  self.run() 
    461  File "/usr/lib/python3.5/threading.py", line 862, in run 
    462  self._target(*self._args, **self._kwargs) 
    463  File "/home/pp/pp/pp/process/uploader.py", line 145, in upload_thread 
    464  topic.publish(byte_string) 
    465  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/cloud/pubsub/topic.py", line 257, in publish 
    466  message_ids = api.topic_publish(self.full_name, [message_data]) 
    467  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/cloud/pubsub/_gax.py", line 165, in topic_publish 
    468  options=options) 
    469  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/cloud/gapic/pubsub/v1/publisher_api.py", line 289, in publish 
    470  return self._publish(request, options) 
    471  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/gax/api_callable.py", line 481, in inner 
    472  return api_caller(api_call, this_settings, request) 
    473  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/gax/api_callable.py", line 158, in inner 
    474  return a_func(request, **kwargs) 
    475  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/gax/api_callable.py", line 434, in inner 
    476  errors.create_error('RPC failed', cause=exception)) 
    477  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/future/utils/__init__.py", line 419, in raise_with_traceback 
    478  raise exc.with_traceback(traceback) 
    479  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/gax/api_callable.py", line 430, in inner 
    480  return a_func(*args, **kwargs) 
    481  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/google/gax/api_callable.py", line 64, in inner 
    482  return a_func(*updated_args, **kwargs) 
    483  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/grpc/_channel.py", line 481, in __call__ 
    484  return _end_unary_response_blocking(state, False, deadline) 
    485  File "/home/pp/.virtualenvs/cv/lib/python3.5/site-packages/grpc/_channel.py", line 432, in _end_unary_response_blocking 
    486  raise _Rendezvous(state, None, None, deadline) 
    487 google.gax.errors.GaxError: GaxError(RPC failed, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, {"created":"@1478711654.067744009","description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_status":14,"referenced_errors":[{"created":"@1478711654.067706801","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]})> 

Answer 1

看起來你正在尋找相關的討論議題2683， 「Frequent gRPC StatusCode.UNAVAILABLE errors」。

你沒有做錯任何事情，它似乎像捕捉異常和重試是目前最合適的解決方法。

Answer 2

如果主題是全局變量，它將停止生成錯誤。使主題類變量，只有實例化一次 - 只有一次調用該行：

topic = pubsub.Client().topic(name) 

而且，看起來這僅適用於Python 2.7版 - 在Python 3.6重試麻木疼痛一點點。

禁用GRPC確實爲Python 3.6的伎倆 - 這可以通過設置環境變量來實現：

ENV GOOGLE_CLOUD_DISABLE_GRPC=true 

Answer 3

我設法得到一個「不那麼漂亮」的解決方法。使用在google.cloud.pubsub_v1.subscriber.policy.thread.Policy.on_exception上覆制deadline_exceeded代碼的策略。

from google.cloud.pubsub_v1.subscriber.policy.thread import Policy 
import grpc 

class UnavailableHackPolicy(Policy): 
    def on_exception(self, exception): 
     """ 
     There is issue on grpc channel that launch an UNAVAILABLE exception now and then. Until 
     that issue is fixed we need to protect our consumer thread from broke. 
     https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2683 
     """ 
     unavailable = grpc.StatusCode.UNAVAILABLE 
     if getattr(exception, 'code', lambda: None)() in [unavailable]: 
      print("¡OrbitalHack! - {}".format(exception)) 
      return 
     return super(UnavailableHackPolicy, self).on_exception(exception) 

在接收郵件的功能我有一個像

subscriber = pubsub.SubscriberClient(policy_class=UnavailableHackPolicy) 
subscription_path = subscriber.subscription_path(project, subscription_name) 
subscriber.subscribe(subscription_path, callback=callback, flow_control=flow_control) 

問題的代碼是當資源是真正暫不提供服務，我們將不知道。然而，儘管GRPC開發團隊設法解決這個問題，我們將使用這種解決方法。