Commit 79a79d65 authored by Jonne Haß's avatar Jonne Haß
Browse files

Bye Resque. Ohai Sidekiq. 

* Dropped all references to Resque
* Moved all jobs under app/workers since that's the Sidekiq convention
* Renamed Jobs module to Worker to match new location
* Adapted all jobs to Sidekiq
* Replaced all enqueue calls with perform_async
* Dropped Resque hacks from specs and features, replaced with
  sidekig/testing in RSpec and sidekig/testing/inline in Cucumber
* Updated scripts to start a Sidekiq server
* Inline Sidekiq sinatra app
* Let Sidekiq create the actual Redis instance
* Workaround already initialized constant warnings in service models
* Resolved ToDo in one job definition by creating proper exception clases
  for some errors in receiving posts
* Added sidekiq section to configuration to make it completly
  configurable to the user
* Add Sidekiq middleware for clean backtraces
* Delay HttpMulti retry to give offline pods a chance to come back up
* Do not retry on GUID already taken and alike errors
* Be graceful about deleted posts in GatherOEmbedData
parent 3fc3b249

.foreman

+1 −1
Original line number Diff line number Diff line 
port: 3000

formation: web=1,worker=0
 
formation: web=1,sidekiq=0

Changelog.md

+97 −9
Original line number Diff line number Diff line 
# Head



## Refactor



### Replaced Resque with Sidekiq - Migration guide - [#3993](https://github.com/diaspora/diaspora/pull/3993)



We replaced our queue system with Sidekiq. You might know that Resque needs Redis.

Sidekiq does too, so don't remove it, it's still required. Sidekiq uses a threaded

model so you'll need far less processes than with Resque to do the same amount

of work.



To update do the following:



1. Before updating (even before the `git pull`!) stop your application

   server (Unicorn by default, started through Foreman).

2. In case you did already run `git pull` checkout v0.0.3.2:

   

   ```

   git fetch origin

   git checkout v0.0.3.2

   bundle

   ```

   

3. Start Resque web (you'll need temporary access to port 5678, check

   your Firewall if needed!):

   

   ```

   bundle exec resque-web

   ```

   

   In case you need it you can adjust the port with the `-p` flag.

4. One last time, start a Resque worker:

   

   ```

   RAILS_ENV=production QUEUE=* bundle exec rake resque:work

   ```



   Visit Resque web via http://your_host:5678, wait until all queues but the

   failed one are empty (show 0 jobs).

5. Kill the Resque worker by hitting Ctrl+C. Kill Resque web with:

   

   ```

   bundle exec resque-web -k

   ```



   Don't forget to close the port on the Firewall again, if you had to open it.

6. In case you needed to do step 2., run:

   

   ```

   git checkout master

   bundle

   ```



7. Proceed with the update as normal (migrate database, precompile assets).

8. Before starting Diaspora again ensure that you reviewed the new

   `environment.sidekiq` section in `config/diaspora.yml.example` and,

   if wanted, transfered it to your `config/diaspora.yml` and made any

   needed changes. In particular increase the `environment.sidekiq.concurrency`

   setting on any medium sized pod. If you do change that value, edit

   your `config/database.yml` and add a matching `pool: n` to your database

   configuration. n should be equal or higher than the amount of

   threads per Sidekiq worker. This sets how many concurrent

   connections to the database ActiveRecord allows.





If you aren't using `script/server` but for example passenger, you no

longer need to start a Resque worker, but a Sidekiq worker now. The

command for that is:



```

bundle exec sidekiq

```





#### Heroku



The only gotcha for Heroku single gear setups is that the setting name

to spawn a background worker from the unicorn process changed. Run



```

heroku config:remove SERVER_EMBED_RESQUE_WORKER

heroku config:set SERVER_EMBED_SIDEKIQ_WORKER=true

```



We're automatically adjusting the ActiveRecord connection pool size for you.



Larger Heroku setups should have enough expertise to figure out what to do

by them self.



### Other



* Cleaned up requires of our own libraries [#3993](https://github.com/diaspora/diaspora/pull/3993)

* Refactor people_controller#show and photos_controller#index [#4002](https://github.com/diaspora/diaspora/issues/4002)

* Modularize layout [#3944](https://github.com/diaspora/diaspora/pull/3944)

* Add header to the sign up page [#3944](https://github.com/diaspora/diaspora/pull/3944)

* Add a configuration entry to set max-age header to Amazon S3 resources. [#4048](https://github.com/diaspora/diaspora/pull/4048)

* Load images via sprites [#4039](https://github.com/diaspora/diaspora/pull/4039)

* Delete unnecessary javascript views. [#4059](https://github.com/diaspora/diaspora/pull/4059)



## Bug fixes



* reset comment box height after posting a comment. [#4030](https://github.com/diaspora/diaspora/issues/4030)
@@ -13,15 +110,6 @@ 
* Fix mobile view of deleted reshares. [#4063](https://github.com/diaspora/diaspora/issues/4063)

* Hide comment button in the mobile view when not signed in. [#4065](https://github.com/diaspora/diaspora/issues/4065)



## Refactor



* Delete unnecessary javascript views. [#4059] (https://github.com/diaspora/diaspora/pull/4059)

* Add a configuration entry to set max-age header to Amazon S3 resources. [#4048](https://github.com/diaspora/diaspora/pull/4048)

* Refactor people_controller#show and photos_controller#index [#4002](https://github.com/diaspora/diaspora/issues/4002)

* Modularize layout [#3944](https://github.com/diaspora/diaspora/pull/3944)

* Add header to the sign up page [#3944](https://github.com/diaspora/diaspora/pull/3944)

* Load images via sprites [#4039](https://github.com/diaspora/diaspora/pull/4039)



## Features



* Deleting a post that was shared to Facebook now deletes it from Facebook too [#3980]( https://github.com/diaspora/diaspora/pull/3980)

Gemfile

+3 −2
Original line number Diff line number Diff line
@@ -17,8 +17,9 @@ gem 'devise', '2.1.3' 


# Background processing



gem 'resque',         '1.23.0'

gem 'resque-timeout', '1.0.0'

gem 'sidekiq', '2.7.5'

gem 'sinatra', '1.3.3'

gem 'slim', '1.3.6'



# Configuration

Gemfile.lock

+19 −11
Original line number Diff line number Diff line
@@ -62,6 +62,9 @@ GEM 
    carrierwave (0.8.0)

      activemodel (>= 3.2.0)

      activesupport (>= 3.2.0)

    celluloid (0.12.4)

      facter (>= 1.6.12)

      timers (>= 1.0.0)

    childprocess (0.3.9)

      ffi (~> 1.0, >= 1.0.11)

    chunky_png (1.2.7)
@@ -81,6 +84,7 @@ GEM 
    compass-rails (1.0.3)

      compass (>= 0.12.2, < 0.14)

    configurate (0.0.2)

    connection_pool (1.0.0)

    crack (0.3.2)

    cucumber (1.2.3)

      builder (>= 2.1.2)
@@ -103,6 +107,7 @@ GEM 
    excon (0.20.1)

    execjs (1.4.0)

      multi_json (~> 1.0)

    facter (1.6.17)

    factory_girl (4.2.0)

      activesupport (>= 3.0.0)

    factory_girl_rails (4.2.1)
@@ -321,13 +326,6 @@ GEM 
    redis-namespace (1.2.1)

      redis (~> 3.0.0)

    remotipart (1.0.5)

    resque (1.23.0)

      multi_json (~> 1.0)

      redis-namespace (~> 1.0)

      sinatra (>= 0.9.2)

      vegas (~> 0.1.2)

    resque-timeout (1.0.0)

      resque (~> 1.0)

    rmagick (2.13.2)

    roxml (3.1.6)

      activesupport (>= 2.3.0)
@@ -363,11 +361,20 @@ GEM 
      multi_json (~> 1.0)

      rubyzip

      websocket (~> 1.0.4)

    sidekiq (2.7.5)

      celluloid (~> 0.12.0)

      connection_pool (~> 1.0)

      multi_json (~> 1)

      redis (~> 3)

      redis-namespace

    simple_oauth (0.2.0)

    sinatra (1.3.3)

      rack (~> 1.3, >= 1.3.6)

      rack-protection (~> 1.2)

      tilt (~> 1.3, >= 1.3.3)

    slim (1.3.6)

      temple (~> 0.5.5)

      tilt (~> 1.3.3)

    slop (3.4.4)

    spork (1.0.0rc3)

    sprockets (2.2.2)
@@ -376,10 +383,12 @@ GEM 
      rack (~> 1.0)

      tilt (~> 1.1, != 1.3.0)

    subexec (0.2.2)

    temple (0.5.5)

    terminal-table (1.4.5)

    thor (0.17.0)

    tilt (1.3.6)

    timecop (0.6.1)

    timers (1.1.0)

    treetop (1.4.12)

      polyglot

      polyglot (>= 0.3.1)
@@ -397,8 +406,6 @@ GEM 
      kgio (~> 2.6)

      rack

      raindrops (~> 0.7)

    vegas (0.1.11)

      rack (>= 1.0.0)

    warden (1.2.1)

      rack (>= 1.0)

    webmock (1.8.11)
@@ -473,8 +480,6 @@ DEPENDENCIES 
  rb-inotify (= 0.9.0)

  redcarpet (= 2.2.2)

  remotipart (= 1.0.5)

  resque (= 1.23.0)

  resque-timeout (= 1.0.0)

  rmagick (= 2.13.2)

  roxml (= 3.1.6)

  rspec-instafail (= 0.2.4)
@@ -482,6 +487,9 @@ DEPENDENCIES 
  ruby-oembed (= 0.8.8)

  sass-rails (= 3.2.6)

  selenium-webdriver (= 2.31.0)

  sidekiq (= 2.7.5)

  sinatra (= 1.3.3)

  slim (= 1.3.6)

  spork (= 1.0.0rc3)

  timecop (= 0.6.1)

  twitter (= 4.6.2)

Procfile

+1 −1
Original line number Diff line number Diff line 
web: bundle exec unicorn_rails -c config/unicorn.rb -p $PORT

worker: env QUEUE=* bundle exec rake resque:work
 
sidekiq: bundle exec sidekiq