use TheSchwartz job queue to handle Image-Resize For Catalyst App.

TheSchwartz - reliable job queue. original developped by Brad and used in LiveJournal.

My main aim is to remove 'Image::Magick' out of my Catalyst App since Image::Magick takes a lot of memories for every process under mod_perl. so we need a non-stop cron script to monitor and resize photos. and I picked TheSchwartz up for my Foorum Catalyst App.

Here comes the instructions:

1, create db 'theschwartz' by using schema: http://search.cpan.org/src/BRADFITZ/TheSchwartz-1.04/doc/schema.sql

2, write the main module Foorum::TheSchwartz::Worker::ResizeProfilePhoto use base TheSchwartz::Worker.

package Foorum::TheSchwartz::Worker::ResizeProfilePhoto;

use TheSchwartz::Job;
use base qw( TheSchwartz::Worker );
use Foorum::ExternalUtils qw/schema/;
use File::Spec;
use Image::Magick;
use Cwd qw/abs_path/;
use File::Copy;

my (undef, $path) = File::Spec->splitpath(__FILE__);

sub work {
    my $class = shift;
    my TheSchwartz::Job $job = shift;
    
    my @args = $job->arg;
    
    my $schema = schema();

    # get upload from db
    my $upload_id = shift @args;
    if ($upload_id !~ /^\d+$/) {
        return $job->failed("Wrong upload_id: $upload_id");
    }
    my $upload = $schema->resultset('Upload')->find( { upload_id => $upload_id } );
    unless ($upload) {
        return $job->failed("No upload for $upload_id");
    }
    
    # get file dir
    my $directory_1 = int( $upload_id / 3200 / 3200 );
    my $directory_2 = int( $upload_id / 3200 );
    my $file = abs_path("$path/../../../../root/upload/$directory_1/$directory_2/" . $upload->filename);
    
    # resize photo
    my $p = new Image::Magick;
    $p->Read($file);
    $p->Scale(geometry=>'120x120');
    $p->Sharpen(geometry=>'0.0x1.0');
    $p->Set(quality=>'75');
    
    my ($width, $height, $size) = $p->Get('width', 'height', 'filesize');
    
    my $tmp_file = $file . '.tmp';
    $p->Write($tmp_file);
    
    move($tmp_file, $file);
    
    # update db
    $schema->resultset('UserProfilePhoto')->search( {
        type    => 'upload',
        value   => $upload_id,
    } )->update( {
        width  => $width,
        height => $height,
    } );
    ($size) = ( $size =~ /^(\d+\.?\d{0,1})/ );    # float(6,1)
    $upload->update( { filesize => $size } );

    $job->completed();
}

sub max_retries { 3 };

1;

3, write a TheSchwartz_worker.pl and run it. it will monitor the "job" table in "theschwartz" database non-stop.

package Foorum::ExternalUtils;
# ... etc.
use TheSchwartz;
sub theschwartz {
    
    $config = config() unless ($config);
    
    my $theschwartz = TheSchwartz->new(
        databases => [ {
            dsn  => $config->{theschwartz_dsn}, # dbi:mysql:theschwartz
            user => $config->{dsn_user},
            pass => $config->{dsn_pwd},
        } ],
        verbose => 1,
    );
    return $theschwartz;
}

use Foorum::ExternalUtils qw/theschwartz/;
use Foorum::TheSchwartz::Worker::ResizeProfilePhoto;

my $client = theschwartz();
$client->can_do('Foorum::TheSchwartz::Worker::ResizeProfilePhoto');
$client->work();

4, at last, we need insert data into 'job' table then let TheSchwartz_worker.pl to handle it.

package Foorum::Controller::Profile;
use Foorum::ExternalUtils qw/theschwartz/;
sub xxx : Local {
    # ... etc.
    my $client = theschwartz();
    $client->insert('Foorum::TheSchwartz::Worker::ResizeProfilePhoto', $new_upload_id);
}

That's ALL.

The idea behind TheSchwartz is not so complicate.

• the "theschwartz" database contains 5 tables:
  
  
  
  • 'error' for recording error for every failed job. like in Foorum::TheSchwartz::Worker::ResizeProfilePhoto, $job->failed("No upload for $upload_id");
  
  
  • 'funcmap' has two columns: on is funcid and the other is funcname. It's pretty simple. when we call "$client->can_do('Foorum::TheSchwartz::Worker::ResizeProfilePhoto');" in TheSchwartz_worker.pl, it will create a new row ($auto_increasing_func_id, 'Foorum::TheSchwartz::Worker::ResizeProfilePhoto') if there hasn't
  
  
  • 'job' is the most important table here. TheSchwartz_worker.pl monitor this table every 5 secs. and if $client->insert one job, it will work_once() immediately.
  
  
  • other two tables: exitstatus and notes
  
  
  
  
• TheSchwartz_worker.pl can tell what jobs it can handle by using "$client->can_do('Foorum::TheSchwartz::Worker::ResizeProfilePhoto');". add this line means that TheSchwartz_worker.pl can do the jobs which the "funcid" in "job" table is mapping with 'Foorum::TheSchwartz::Worker::ResizeProfilePhoto'.


• TheSchwartz_worker.pl is a non-stop script to monitor the "job" table every 5 secs. if there are jobs, it will work_once() one by one.


• $client->insert('Foorum::TheSchwartz::Worker::ResizeProfilePhoto', $new_upload_id); it inserts one row in 'job' table then let worker.pl to handle it. 'Foorum::TheSchwartz::Worker::ResizeProfilePhoto' is the funcname, and $new_upload_id is the args.
  in package Foorum::TheSchwartz::Worker::ResizeProfilePhoto; we need use base TheSchwartz::Worker and write a sub work.
  

  sub work {
      my $class = shift;
      my TheSchwartz::Job $job = shift;
      
      my @args = $job->arg;
      
      # do something
      
      $job->completed();
  }



• If u want another worker:
  just write package Foorum::TheSchwartz::Worker::Another
  then in TheSchwartz_worker.pl use it and add $client->can_do('Foorum::TheSchwartz::Worker::Another');
  then in pl or pm. $client->insert('Foorum::TheSchwartz::Worker::Another', @args);
  

That's ALL. @Enjoy;