Fayland and Perl Programming

MongoDB and Perl

2012-04-26T03:17:16Z

it's really a pain to work with MongoDB in Perl. Perl has no 'type' so when you get INT value from DBI, it might be really "1" instead of int 1.

I got some data dumped from MySQL to MongoDB and found all the 'time' field is wrapped as "1335350669" instead of 1335350669.

and when you insert it from code like { time => time() }, you really have 1335350670 instead of "1335350670". it breaks the sort. it breaks the deletion. it breaks everything.

blabla, to fix that, we just need update it in mongodb like below.

PRIMARY> db.jobs.find().forEach(
function(job) {
job.time = parseInt(job.time);
db.jobs.save(job);
});

but it's still a pain. I didn't check MongoDBx::Class or Mongoose yet, but Moose's type should be able to fix it I think.

Thanks

api and android app

2012-04-14T12:20:56Z

I'm a fan of Android. even I like my iPad and MBP very much too.

I'd like to write an App for Android. and here is all the story.

first of all, I need write an API which supports JSONP. since it would be very simple, I don't want to mix it up with Dancer. at last, I wrote something based on webmachine-perl. I like the idea behind the webmachine. the chain design looks pretty nice. even I don't use that too much.

the code is at

https://github.com/fayland/findmjob.com/blob/master/api/app.psgi

https://github.com/fayland/findmjob.com/blob/master/api/lib/FindmJob/Resource.pm

and live demo as

http://api.findmjob.com/search?callback=jQuery171016797792096622288_1334322201618&q=perl&loc=&_=1334322214418

after API is done. now comes the android app part. Sorry that I don't know much about Java. and I failed to download appmobi/jqmobi (the download never ends here). so at last I picked up phonegap.

the progress went pretty smooth. mixed with jQuery Mobile, I have it out after few hours.

you can download it for fun from http://static.findmjob.com/FindmJob.apk

it's pretty simple, just with one JSONP request and not much different than the demo.

even there, I'd like to share the code with you:

https://github.com/fayland/findmjob.com/blob/master/mobile/android/assets/www/index.html

https://github.com/fayland/findmjob.com/blob/master/mobile/android/assets/www/mobile.coffee

it's very cool and I'm excited! (badly Google costs 25$ for submitting it and I don't want to do it for now)

Thanks.

sphinx search with varchar primary key

2012-04-08T01:59:10Z

usually when you index the mysql data into sphinx, you'll use id int/bigint for the primary key. but it's broken for me on http://findmjob.com/, we use uuid everywhere for the primary key.

here is the solution for it. use @id := @id + 1 for the indexer, and use sql_field_string to get the real id when matched. sample code below:

sql_query_pre = SET NAMES utf8
sql_query_pre = SET @id := 1;
sql_query = \
SELECT @id := @id + 1 AS tid, id, title, description, location, contact, inserted_at FROM job ORDER BY inserted_at DESC LIMIT 10000

sql_field_string = id

it requires latest sphinx to support sql_field_string. and the latest CPAN module too.

and Perl code for it will be normal like before. and instead you use ->{doc}, you need use the attribtues ->{id} like

my @jobids = map { $_->{id} } @{$ret->{matches}};

so now we have sphinx search supports in my new site, eg: http://findmjob.com/search/Perl.html?q=Perl

Thanks.

DBIx::Class with Moose has

2012-04-04T13:10:22Z

well, I don't know how to name it.

DBIx::Class is one of my favorite modules. for its structure, with DBIx::Class you can make all your code very well organized and clean. writing code with DBIx::Class means you can use it in framework like Catalyst or Mojo or Dancer, and you can use it in any perl script (cron usually). for me, DBIx::Class is the right model, TT2 is the right template, and framework is just for URL dispatch.

Moose is another module I like. Role, and clean OO.

by using DBIx::Class::Schema::Loader make_schema_at, with use_moose => 1 on (eg: https://github.com/fayland/findmjob.com/blob/master/script/make_schema_at.pl), we can generate very clean Result module with Moose.

sometimes you want make assessor based on the table column. like for each job in table, we all have a URL which is based on the id and title in the table, it's so related to those two fields, so we'd better to put it in the Result.

with DBIx::Class, it's very simple. (sample code)

package FindmJob::Schema::Result::Job;

....

use FindmJob::Utils 'seo_title';
has 'url' => ( is => 'ro', isa => 'Str', lazy_build => 1 );
sub _build_url {
my ($self) = @_;

return "/job/" . $self->id . "/" . seo_title($self->title) . ".html";
}

after that, we can always call $job->url after we ->search for ->find it. very neat. live demo like:

http://findmjob.com/job/Il2bUdB74RGuDaqxKQ5yzw/Senior-Perl-Developer.html

have fun. Thanks

better pagination url design in Dancer

2012-04-02T11:23:18Z

usually People do param for pager like ?page=1 or ?p=1, it maybe not that good for search engine because they may not go scrape inside. so we may come out a solution with /page=1/ or /p=1/ or even /p.1/ etc.

in Dancer, it's very tricky to do add pagination regex in all URLs. and thank God, we have 'forward' and with code like below, it becomes very simple and easy to use.

Perl code:

get qr'.*?/p\.(\d+).*?' => sub {
my $uri = request->uri;
$uri =~ s'/p\.(\d+)'';
var page => $1;
$uri =~ s/\/$//;
forward $uri;
};

get '/' => sub {
my $p = vars->{page} || 1; $p = 1 unless $p =~ /^\d+$/;

we use regex to get the page stuff, then remove it from the request uri then using forward to do a internal request.

the TT2 pager code can check at https://github.com/fayland/findmjob.com/blob/master/templates/pager.tt

live demo as: http://findmjob.com/tag/perl/ and http://findmjob.com/tag/perl/p.2/

have fun.

findmjob.com

2012-04-02T07:35:19Z

maybe to earn some money (not for fun this time), I decided to write a new website http://findmjob.com/

I sit front of my computer and coded it for 2 days and here is it. it's out.

it's very simple and without much stuff yet. and the final goal is undecided. but there it is. I'm very pleased to see it in public.

since there is no reason to keep it private, I opened source it in github: https://github.com/fayland/findmjob.com

for programmer, it's very simple to write website. but it's very hard to make it a success. so suggestions are welcome.

Thanks.

Net-GitHub-0.43_01

2012-03-30T12:43:06Z

Github is moving on with their API that "We will terminate API v1 and API v2 in 1 month on May 1st, 2012.".

I know lots of people are preferring that they want token instead of writing user/pass in the config or code.

and now we have the choice with create access_token with code instead of web flow.

it's time to kick Net::GitHub default to V3 now. so here comes the beta release. and tests/patches are welcome.

http://fayland.org/CPAN/Net-GitHub-0.43_01.tar.gz (it will be on CPAN soon too)

Tips to create access_token with script:

use Net::GitHub;
my $gh = Net::GitHub->new( login => 'fayland', pass => 'secret' );
my $oauth = $gh->oauth;
my $o = $oauth->create_authorization( {
scopes => ['user', 'public_repo', 'repo', 'gist'], # just ['public_repo']
note => 'test purpose',
} );
print $o->{token};

record the token down, and later on we can always use that token without writing down user/pass.

my $github = Net::GitHub->new(
access_token => $token, # from above
);

Thanks.

Plack::Middleware::FileWrap

2011-12-04T08:51:48Z

When you really go coding, you'll meet lots of issues. then you'll write solution for them. that's straight.

Today I have another CPAN module Plack::Middleware::FileWrap out to fit my demand: I'll have lots of plain HTML files, they'll share the same header/footer and I don't want to use stupid iframe.

under Plack, it just means wrap $res->[2] with file content or strings.

so here comes Plack::Middleware::FileWrap, very simple if you looked at the source code.

then I used it in the KinderGarden project, with snippets:

mount '/static/docs/' => builder {

enable 'FileWrap', headers => ["$root/static/docs/header.html"], footers => ["$root/static/docs/footer.html"];

Plack::App::File->new( root => "$root/static/docs" )->to_app;

Live demo as http://kindergarden.fayland.org/static/docs/TestLocally.html

now all the files under static/docs will wrapped with header.html and footer.html (header.html/footer.html itself too!)

for some advanced example, like if you just want to apply to html files, then you can code something like:

mount '/static/docs/' => builder {

enable_if { $_[0]->{PATH_INFO} =~ /\.html/ } 'FileWrap',

headers => ["$root/static/docs/header.html"], footers => ["$root/static/docs/footer.html"];

Plack::App::File->new( root => "$root/static/docs" )->to_app;

I didn't put the Content-Type check b/c I think it's better to be handled with enable_if.

Enjoy. Thanks

git submodule

2011-12-03T10:56:57Z

When you include another open source in your own project, it's usually pretty hard to keep it up to date. it becomes even more harder if you have some modification on it.

recently I have bootstrap in my KinderGarden project, it's very easy to use git submodule to handle it.

kindergarden> git submodule add https://github.com/twitter/bootstrap.git static/bootstrap
kindergarden> git add .gitmodules static/bootstrap
kindergarden> git commit -a -m "remote bootstrap"
kindergarden> git push
kindergarden> git submodule init

in delopy or other machine:

kindergarden$ git submodule init
kindergarden$ git submodule update

pretty easy and simple, and the logic behind is simple too. reference as http://help.github.com/submodules/ or http://progit.org/book/ch6-6.html

BTW, I added Live.com OAuth2 supports to KinderGarden.

Enjoy. Thanks

2011 CN Perl Advent

2011-12-01T11:16:26Z

Hi, it's time for advent again!

and here it is: http://perlchina.github.com/advent.perlchina.org/

the repos is at https://github.com/PerlChina/advent.perlchina.org

Enjoy!

KinderGarden

2011-11-27T04:43:24Z

as talked yesterday, I get it uploaded into github. well, under PerlChina. https://github.com/PerlChina/kindergarden

I really want to draw more people to add more features and fix more bugs.

and I added one new feature with Mojolicious which is http://kindergarden.fayland.org/app/whereilive

feel free to view the source and let me know what you think!

Thanks.

Dancer::Template::Xslate

2011-11-26T05:50:34Z

I'm writing some toy once again with Plack and Dancer (and Mojo later).

this time, I'm playing Plack::Middleware::OAuth and Dancer::Template::Xslate a bit. the website is http://kindergarden.fayland.org/ and I'll open source it if someone is interested. for now, it's just 'Login with ...' OAuth and nothing else. (layout is built with twitter bootstrap)

the Dancer::Template::Xslate has some bugs and I tried to submit few commits through github to fix it. (at least it's working for me now)

here is a tip to add function like gravatar into Xslate within Dancer.

problem as Text::Xslate supports function param when ->new but Dancer YAML config can't have Perl code inside. and it's very tricky or hard to fix the engine 'template' b/c we can never modify it. it has 'my $_engine;' inside code and you can't modify it at all.

after a while, I find a good solution with the module param of the Text::Xslate. it's very neat. sample code as below:

# config.yml
template: xslate
engines:
xslate:
syntax: 'TTerse'
extension: 'tt'
header:
- 'layout/header.tt'
footer:
- 'layout/footer.tt'
module:
- KinderGardenX::Text::Xslate::Bridge::KinderGarden

# KinderGardenX::Text::Xslate::Bridge::KinderGarden
package KinderGardenX::Text::Xslate::Bridge::KinderGarden;

use strict;
use warnings;
use parent qw(Text::Xslate::Bridge);

use Gravatar::URL;

my %funtion_methods = (
gravatar_url => \&gravatar_url,
);

__PACKAGE__->bridge(
function => \%funtion_methods,
);

1;

# template

user.email, size => 30) %]" />
user.email, size => 50) %]" />
user.email) %]" />

Note there is always more than one way to do it.

Thanks.

new baby

2011-11-08T11:47:34Z

I'm very happy to share the good news with all the world. my second kid, another boy, was born today. 9:45am Beijing Time, Nov 8th, 2011. 2800g. and everything is good. Thanks.

remove/add job to crontab by commandline

2011-10-13T03:30:03Z

1. add job to crontab

(crontab -u fayland -l ; echo "*/5 * * * * perl /home/fayland/test.pl") | crontab -u fayland -

2. remove job from crontab

crontab -u fayland -l | grep -v 'perl /home/fayland/test.pl' | crontab -u fayland -

3. remove all crontab

crontab -r

nothing is tricky. expect it took me 10 minutes to figure out '-' is the one I want. (- is STDOUT in Linux).

Thanks.

sphinx 0.99 bug (attributes count vs fields count)

2011-09-29T10:33:00Z

when you have 4 columns in sql_query, and you want 3 columns as attributes. you'll get a failure. 0 size sphinx files.

it's quite annoying, and it cost me almost 4 hours to figure it out. I'm so dumb and so are you, SPHINX.

a simple solution is to add a dumb col in the SELECT of sql_query like

SELECT id, radians(longitude) as long_radians, radians(latitude) as lat_radians, 'dumb' FROM table

OK. actually 'dumb' is dumb because it takes more disk than 'a'.

for a detailed issue description, please check http://sphinxsearch.com/forum/view.html?id=8345

Thanks.