爲什麼我的一些後臺作業失敗？

Question

簡要細節：

• Rails的4.0通過延遲招聘
• 後臺作業的應用程序在Heroku
• 工人賽道通過無業紡達
• 大多數的工作取得成功，一些失敗（以下情況例外追蹤）
• 工作失敗只在生產中（我無法再現這個問題在本地）
• 作業在約6秒內被Delayed Job拾起（見前面三行跟蹤）失敗
• 作業在worker/app初始化期間失敗（我的代碼甚至沒有機會執行）
• 我認爲故障是設計或它的模塊之一，在後臺作業

的Gemfile的情況下（一些相關線）：

gem 'devise', '~> 3.2.4' 
gem 'delayed_job_active_record', '4.0.1' 
gem 'workless', '~> 1.2.3' 
gem 'daemons', '~> 1.1.9' 

異常跟蹤：

16.566936 heroku[worker.2]: Starting process with command `bundle exec rake jobs:work` 
17.201669 heroku[worker.2]: State changed from starting to up 
23.375876 heroku[worker.2]: State changed from up to down 
25.734990 heroku[worker.2]: Stopping all processes with SIGTERM 
26.315290 app[worker.2]: rake aborted! 
26.315342 app[worker.2]: SignalException: SIGTERM 
26.333413 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333417 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
26.333419 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
26.333421 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333422 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bcrypt-3.1.7/lib/bcrypt.rb:16:in `rescue in <top (required)>' 
26.333424 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bcrypt-3.1.7/lib/bcrypt.rb:12:in `<top (required)>' 
26.333440 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333442 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
26.333444 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
26.333445 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333447 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models/database_authenticatable.rb:2:in `<top (required)>' 
26.333448 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models.rb:88:in `const_get' 
26.333450 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models.rb:88:in `block (2 levels) in devise' 
26.333451 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models.rb:87:in `each' 
26.333453 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models.rb:87:in `block in devise' 
26.333454 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models.rb:114:in `devise_modules_hook!' 
26.333456 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/models.rb:84:in `devise' 
26.333458 app[worker.2]: /app/app/models/user.rb:6:in `<class:User>' 
26.333459 app[worker.2]: /app/app/models/user.rb:1:in `<top (required)>' 
26.333461 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333462 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
26.333464 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
26.333465 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333466 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:330:in `require_or_load' 
26.333468 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:463:in `load_missing_constant' 
26.333469 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:184:in `const_missing' 
26.333470 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/inflector/methods.rb:226:in `const_get' 
26.333472 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/inflector/methods.rb:226:in `block in constantize' 
26.333473 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/inflector/methods.rb:224:in `each' 
26.333474 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/inflector/methods.rb:224:in `inject' 
26.333476 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/inflector/methods.rb:224:in `constantize' 
26.333477 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:535:in `get' 
26.333478 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:566:in `constantize' 
26.333480 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise.rb:297:in `get' 
26.333481 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/mapping.rb:77:in `to' 
26.333483 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/mapping.rb:72:in `modules' 
26.333484 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/mapping.rb:89:in `routes' 
26.333508 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/mapping.rb:156:in `default_used_route' 
26.333509 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/mapping.rb:66:in `initialize' 
26.333510 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise.rb:331:in `new' 
26.333512 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise.rb:331:in `add_mapping' 
26.333519 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/rails/routes.rb:221:in `block in devise_for' 
26.333520 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/rails/routes.rb:220:in `each' 
26.333522 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/devise-3.2.4/lib/devise/rails/routes.rb:220:in `devise_for' 
26.333523 app[worker.2]: /app/config/routes.rb:31:in `block in <top (required)>' 
26.333524 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/actionpack-4.0.5/lib/action_dispatch/routing/route_set.rb:335:in `instance_exec' 
26.333526 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/actionpack-4.0.5/lib/action_dispatch/routing/route_set.rb:335:in `eval_block' 
26.333527 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/actionpack-4.0.5/lib/action_dispatch/routing/route_set.rb:313:in `draw' 
26.333528 app[worker.2]: /app/config/routes.rb:1:in `<top (required)>' 
26.333530 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:223:in `load' 
26.333531 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:223:in `block in load' 
26.333542 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
26.333543 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:223:in `load' 
26.333545 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:40:in `block in load_paths' 
26.333546 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:40:in `each' 
26.333548 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:40:in `load_paths' 
26.333549 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:16:in `reload!' 
26.333551 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:26:in `block in updater' 
26.333552 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/file_update_checker.rb:75:in `call' 
26.333553 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/file_update_checker.rb:75:in `execute' 
26.333555 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:27:in `updater' 
26.333556 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/routes_reloader.rb:6:in `execute_if_updated' 
26.333557 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application/finisher.rb:69:in `block in <module:Finisher>' 
26.333559 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/initializable.rb:30:in `instance_exec' 
26.333560 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/initializable.rb:30:in `run' 
26.333562 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/initializable.rb:55:in `block in run_initializers' 
26.333563 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/initializable.rb:54:in `run_initializers' 
26.333577 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application.rb:215:in `initialize!' 
26.333579 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/railtie/configurable.rb:30:in `method_missing' 
26.333580 app[worker.2]: /app/config/environment.rb:9:in `<top (required)>' 
26.333581 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333583 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
26.333584 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
26.333585 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
26.333587 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application.rb:189:in `require_environment!' 
26.333588 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/railties-4.0.5/lib/rails/application.rb:250:in `block in run_tasks_blocks' 
26.333589 app[worker.2]: Tasks: TOP => jobs:work => jobs:environment_options => environment 
26.333591 app[worker.2]: (See full trace by running task with --trace) 
28.113879 heroku[worker.2]: Process exited with status 1 

應用/模型/ user.rb：6

devise :database_authenticatable, :registerable, :confirmable, :omniauthable, :recoverable, :rememberable, :trackable, :validatable 

配置/ routes.rb中：31

devise_for :users, controllers: {omniauth_callbacks: 'omniauth_callbacks'} 

---

第2次嘗試

我user.rb註釋掉omniauthable：

devise :database_authenticatable, :registerable, :confirmable, #:omniauthable, 
     :recoverable, :rememberable, :trackable, :validatable 

和omniauth_callbacks在routes.rb：

devise_for :users#, controllers: {omniauth_callbacks: 'omniauth_callbacks'} 

我跑〜20個職位，最後一個崩潰。這一次的異常跟蹤，只是有一點不同：

異常跟蹤：

24.233695 app[worker.2]: rake aborted! 
24.233841 app[worker.2]: SignalException: SIGTERM 
24.234293 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
24.234297 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
24.234300 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234302 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.9.0.229/lib/new_relic/agent/transaction_sampler.rb:11:in `<top (required)>' 
24.234303 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234305 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
24.234307 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
24.234309 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234311 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems /newrelic_rpm-3.9.0.229/lib/new_relic/agent.rb:94:in `<module:Agent>' 
24.234313 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.9.0.229/lib/new_relic/agent.rb:69:in `<module:NewRelic>' 
24.234315 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.9.0.229/lib/new_relic/agent.rb:40:in `<top (required)>' 
24.234317 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234319 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require' 
24.234321 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
24.234323 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234325 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.9.0.229/lib/new_relic/control.rb:24:in `<top (required)>' 
24.234327 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234329 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support /dependencies.rb:229:in `block in require' 
24.234332 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency' 
24.234334 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require' 
24.234336 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.9.0.229/lib/newrelic_rpm.rb:25:in `<top (required)>' 
24.234338 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler/runtime.rb:76:in `require' 
24.234340 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler/runtime.rb:76:in `block (2 levels) in require' 
24.234343 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler/runtime.rb:72:in `each' 
24.234345 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler/runtime.rb:72:in `block in require' 
24.234346 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler/runtime.rb:61:in `each' 
24.234349 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler/runtime.rb:61:in `require' 
24.234351 app[worker.2]: /app/vendor/bundle/ruby/2.0.0/gems/bundler-1.6.3/lib/bundler.rb:132:in `require' 
24.234354 app[worker.2]: /app/config/application.rb:12:in `<top (required)>' 
24.234356 app[worker.2]: /app/Rakefile:4:in `require' 
24.234358 app[worker.2]: /app/Rakefile:4:in `<top (required)>' 
24.234365 app[worker.2]: (See full trace by running task with --trace) 
23.305036 heroku[worker.2]: Stopping all processes with SIGTERM 
25.200583 heroku[worker.2]: Process exited with status 1 

Rake文件：4

require File.expand_path('../config/application', __FILE__) 

配置/應用。RB：12

Bundler.require(:default, Rails.env) 

Answer 1

如果你不調用SIGTERM，Heroku的通過發送SIGTERM每天每重啓賽道至少一次。 Sidekiq會自動處理這個問題，但有時候Heroku會提前關閉它，並且會導致問題 - 這意味着將超時時間從8更改爲5秒是很好的做法。延遲作業和Resque要求你在5左右處理信號。

Resque說明 https://devcenter.heroku.com/articles/queuing-ruby-resque#job-termination

延遲喬布斯 Long running delayed_job jobs stay locked after a restart on Heroku

Answer 2

你DYNOS發送一個SIGTERM命令。然後，您的進程將有10秒關閉，然後SIGKILL將發送到您的進程。如果該過程必須被強行殺死，則會拋出R12 error。我們（Heroku）除此之外不執行任何超時。

讓我知道你是否還有其他問題。或者，歡迎您打開support request以獲取有關此信息的更多信息。

來源：Dynos and the Dyno Manager：Graceful shutdown with SIGTERM