Work in progress:
@public,perl,ironman
I volunteered (or was nudged) to apply to get the Enlightened Perl Organisation's grant to document/write-up some material on using DBIx::Class for SQL Hackers. The idea is to show folks that traditionally use plain DBI in their Perl code, how to do the same queries and much more, using DBIx::Class.
Still waiting for Matt to look at this, or say something about it. Will assume for now that its all fine.. ;)
I hope to post links to editable content via the EPO Wiki, helpfully both my blog/doc format and that of the Wiki are Markdown, so I can shunt things around easily.
The DBIC Tutorial is a long-time work-in-progress which I really need to get into a state/place where others can use and contribute to it. Is it a good idea to convert to markdown and then add sections on "from SQL" to complete the grant?
Almost every sizable Perl application these days needs a method of long term data storage. When the data needs to be easily retrieved as well as stored, we often use a database. Most databases can be comfortably accessed using SQL. Using the DBI module, and a DBD for the particular database, we can write SQL in our Perl code, and retrieve the results as arrays or hashes.
## Example
my $dbh = DBI->connect("dbi:SQLite:mydb.db");
my $sth = $dbh->select("SELECT artist.id, artist.name FROM artists");
$sth->execute;
my $results = $sth->fetchrow_arrayref({});
foreach my $row (@$results) {
print $row->{name};
}
There are several things we can do to make this code more usable, for example store the database connect string (DSN) in a configuration file so that users of the code can use different databases without editing the code. We can also write a separate method or module for creating and returning the $dbh, so that we don't create a lot of db connections unnecessarily.
The part we can't do much about is the SQL in the code. We can move it around, put it in libraries, but it's still there, somewhere.
Why would you not want SQL in your Perl code? For a start, it's just a string to pass to the database interpreter, there is no syntax checking at the Perl compilation level. Thus it fails late, not early. Your editor will also not syntax check what it just sees as strings of text.
Modern Perl should also leverage OO where it can. DBI is a low level library that gets things right, but returns plain hashes and arrays, not objects.
Perl ORMs still use DBI underneath, extending it to catch coding errors early (names of columns, SQL clauses), and to return the results as objects containing the database values, which work just like any other Perl object.
@public,perl,dbix-class
Several days later, what do I remember from LPW 2009?
Piers' talk on New beginnings. Interestingly, it's mostly the structure of the talk that I remember, not the content. Only a few days before, Simon Cozens had pointed out a post of Yuval's, and specifically praised it as a post with good structure: Show where there is a problem, solve that problem with a particular coding technique (or technology), dont waffle too much. Simon was so impressed he posted a meta-post on the subject.
Pier's talk was about why he left Perl for Ruby, and why he came back again. Perl has some "boilerplate" code that one typically needs when writing all methods. Instead of declaring the methods incoming arguments, one has to explicitly pull them off of the argument array, @_. Piers returned because the Perl community has been solving this and similar annoyances. In this case with Devel::Declare. And it's not even a "source filter"!
Problem; Solution; Done.
Another talk I still remember is Andy Wardley's on Template Toolkit 3. It's been many years in the making, much like Perl 6^WRakudo. It's looking pretty good however, and much evolved. I made myself some TODO notes as he was talking. Andy's talk style was the opposite of Piers', so I will have to ask him later; What problems does TT3 solve (better, more elegantly) that TT2 doesn't? Why would I use it over TT2? He also zhas some ambitious ideas to add a C library so that it can be used from other programming languages. This is another good way to spread good things that come from Perl.
That's it, most of the rest is a blur. Hallway chats, hot drinks, food, books. O'Reilly were there with a table as usual, with 35% off usual prices, so I bought "Beautiful Teams" to add to my collection of projecty books.
It was good to see friends again, to actually talk to people outside the confines of IRC. I came away somewhat more motivated to join in non-work projects than of late.
So much so that I've picked up on the EPO grants again. Having been voted on, decided, had money assigned and then listed.. They've been sitting there idle, waiting for someone to speak up and offer to complete them. Maybe its a lesson in time management, these things don't run themselves, mebbe volunteers need to be able to dedicate X amount of hours per day/week to help. Oops, minor side-rant, I'll go and write that proposal now, and assign myself time to work on it!
@diary, public, perl, lpw2009
No really, not just because I've not posted anything in.. a while. I got bored reading mst's mbox of ironman stuff, converting blog urls into atom/rss feeds, and typing the results into Plaggers yaml configuration file.
So I replaced all those manual entries with:
- module: Subscription::DBI
config:
schema_class: 'IronMan::Schema'
connect_info: ['dbi:SQLite:/home/castaway/plagger/subscriptions.db']
And wrote one of these to convert the existing junk:
#!/usr/bin/perl
use strict;
use warnings;
use YAML 'LoadFile';
use Data::Dumper;
use Data::UUID;
use IronMan::Schema;
my $yamlfile = shift;
my $dsn = shift;
my $yaml_dump = LoadFile($yamlfile);
my $schema = IronMan::Schema->connect($dsn);
my ($sub_conf) = grep { $_->{module} eq 'Subscription::Config' }
@{ $yaml_dump->{plugins} };
print Dumper($sub_conf);
my $feeds = $sub_conf->{config}{feed};
my $uuid = Data::UUID->new();
foreach my $feed (@$feeds) {
print "Feed: $feed->{url}\n";
my $fdb = $schema->resultset('Feed')->find_or_new({ id => $uuid->create_str, %$feed}, { key => 'url' });
if($fdb->in_storage) {
print "... already exists\n";
next;
}
$fdb->insert;
}
The twiddly part was setting up an app (form) for people to sign up, which now exists in the ironman repo, and runs on my local box, at the moment.
Thoughts of things to add:
Someone also needs to make the thing prettier!
Btw to install, symlink the css and image dirs from "plagger/assets/plugins/Publish-PagedPlanet/default/static" to the catalyst "root" dir.
@public,perl,ironman
I failed in the end, to Reproduce my URI problem, but that hasn't deterred me from helping to improve random parts of CPAN in general.
Recently I've been struggling with Crypt::RSA. It's API is not all that complicated, however a small typo caused me quite a bit of grief. A typo in my own code, misspelling one of the argument names for the verify function.
I had this:
$rsa->verify(
Messsage => $mess,
Key => $key,
Signature => $signature
);
Spot the problem? It took me ages, and some perl -d debugging, to realiase that Crypt::RSA happily attempts to verify an empty (undef) message. It didn't use warnings, or check it's arguments, so my extra "s" in message wasn't noticed.
I reported the problem to the author, using CPAN's RT installation, which is available to all modules, as Bug #46577.
The author responded rapidly, understood my problem, improved his code, and uploaded a new release to CPAN. You can see the fixes using the CPAN diff tool: Diff from Crypt-RSA-1.98 to Crypt-RSA-1.99. Yay!
The second (actually chronologically first) fix was related to my mumble about XML::Atom::SimpleFeed, on a previous blog entry. Aristotle got in touch with me to say "That shouldn't happen, is all sane" .. But eventually James proved him wrong with a patch including a test case, and a new release of that appeared too.
@public,perl,crypt-rsa,xml-atom-simplefeed
Odd flash of inspiration while washing up (yay for mindless activity) and also listening to a TED talk.
I often grumble at Moose. Though not actually at Moose itself, just at seemingly everyone and everything jumping on the bandwagon. I also want to use Reaction, cos Matt wrote/writes it, and from the outside it feels like a great idea.
But part of me doesn't think so. I think I know why now. Maybe I've been reading too many articles on Agile and so on. Specifically the principles of DoTheSimplestThingThatCouldPossiblyWork, and YouAintGonnaNeedIt.
Reaction is a complex piece of software, built on top of two other pieces of complex software, Catalyst, and Moose. That's a whole stack of code. As people jokingly say "It installs half of CPAN". Don't get me wrong, software re-use is good and if I actually needed a piece of software that does all the many things that these three do, I'd use them.
But I don't. Applications (and websites) don't often start complex, they gather complexity over time.
For one or two web pages, maybe even ten, I can write them by hand. Ok so I probably copy the header and footer after a few. When I come to write the eleventh one, this becomes tedious. So instead I could use something like Template Toolkit's ttree, to produce a bunch of pages out of some templates. At some point I find myself needing something dynamic, or the concept of users, then I may go fetch Catalyst.
And that's as far as I've got. Which step makes me need Reaction? The one where 90% of my website is interactive? I've not written too many of those, but the few I have, Catalyst has sufficed.
You're probably thinking now, that I just don't write the same kind of applications/websites as others do. Thats possibly true.
But again, Agile ways of doing things suggest we write the minimal amount of code we can get away with, to imlement the currently required features. That we don't plan ahead and add more code for potential features later, they will be re-thought anyway. That we release early, and often, and gather feedback for improvement, and then add new features.
Not forgetting to refactor. To upgrade, to replace the entire framework if needed.
I think, what I'm looking for is, a guide for newcomers to explain and help ths progression. How to upgrade your site or code, from a few pages, to some dynamic bits, to a full blown interactive site. Or even not, not every site wants or needs to go that far. Not all code needs to be able to use email.
There seem to exist many articles on how to write this complex code, but not enough that explain how to come to the conclusion that you need it.
Maybe this is also why there are still many many more people writing about how to do CGI scripts in Perl, than how to use Moose. The transition is missing.
(With thanks to Elizabeth Gilbert on nurturing creativity)
Right, now I'm off to move the fledgling DBIx::Class website from some hand written pages to ttree.
@public,programming,perl,thinking
It's been a while (4 months) since I created this blogging software, and I'm still happy with it. I've made a few updates to the code since I originally posted. So I thought I'd post an update.
I didn't post the RSS last time, so here's the current copy:
package Blog::RSS;
use strict;
use warnings;
use XML::Atom::SimpleFeed;
use File::Grep 'fgrep';
use Path::Class;
use Data::Dumper;
use Text::Markdown 'markdown';
use DateTime;
use Template;
use DateTime::Format::W3CDTF;
use Apache2::Const;
sub handler
{
my $r = shift;
my $filedir = dir($r->dir_config('files'));
my $feed = XML::Atom::SimpleFeed->new(
title => $r->dir_config('blog_title'),
link => $r->dir_config('blog_link'),
id => $r->dir_config('blog_id'),
author => $r->dir_config('author'),
);
## All articles which are public, and not in-progress (~, .# etc)
my @articles =
fgrep { /\@.*public/ }
map { "$_" }
sort {
($a->stat->ctime <=> $b->stat->ctime) or
($a->stringify cmp $b->stringify)
}
grep { !$_->is_dir && $_->stringify =~ m{.*/[-\w]+\.txt$} }
$filedir->children;
@articles = @articles[-18 .. -1];
# print STDERR Dumper(\@articles);
my $macros = $r->dir_config('macros');
my $tt = Template->new({
ABSOLUTE => 1,
# ENCODING => 'utf8',
# WRAPPER => $wrapper,
PRE_PROCESS => $macros,
# INCLUDE_PATH => $filedir
});
foreach my $art (@articles) {
next if(!$art->{count});
my $f = file($art->{filename});
(my $fname = $f->basename) =~ s/\.txt$/.html/;
my $dt_formatter = DateTime::Format::W3CDTF->new();
my $dt_updated = DateTime->from_epoch(epoch=> $f->stat->ctime);
## Assuming we always put all the tags on one line
my ($category_line) = fgrep { /^\@.*public/} "$f";
$category_line = (values %{ $category_line->{matches} })[0];
chomp($category_line);
my @categories = split(/,/, substr($category_line, 1));
my $tt_processed ='';
my $fh = $f->openr();
binmode($fh, ':utf8');
$tt->process($fh, { photouri => $r->dir_config('blog_photo_base') }, \$tt_processed);
## According to the spec "author" is picked up from the header if not in
## the entry. Not all atom parsers do this (see plagger!)
$feed->add_entry(
title => "Under the palm tree ($fname)",
link => file($r->dir_config('blog_link'), $fname)->stringify,
id => 'http://desert-island.me.uk:8888/~castaway/blog/' . $fname,
content => markdown($tt_processed),
updated => $dt_formatter->format_datetime($dt_updated),
author => $r->dir_config('author'),
(map { (category => $_) } @categories),
);
}
## XML::Atom::SimpleFeed outputs as us-ascii without asking!
# $r->content_type('application/atom+xml;charset=UTF-8');
$r->content_type('application/atom+xml;charset=us-ascii');
$feed->print;
return OK;
}
1;
I've just (literally minutes ago) added some extras: The link/id/author texts were in the code, I moved them into the apache location section, so now (in theory) I could have multiple blogs with different names and ids.
I also added categories (tags), which I'd failed to do originally. I caught myself mumbling at Ironman users who miss them out, then complain that their entries don't show up, so I guess I should tag my own entries!
Since there are a fair few entries now, I've added sorting and reduced the output to the most recent twenty. I hope that's enough to give new users something to read, while reducing the overall server load somewhat.
A recent entry contains a GBP currency symbol (\xA3), which as you'll note, shows up ok on the actual entry page, but not in the RSS. I tried to fix this, but as you see from my note, XML::Atom::SimpleFeed outputs it's XML as "us-ascii" no matter what the user may want. So I may have to go nudge Aristotle with a patch sometime soon.
Also I need to go patch or nudge the [File::Grep]((http://search.cpan.org/dist/File-Grep) author, as the docs claim "fgrep returns an array of matches", but actually it returns a hashref of data.
I did get around to adding images a while ago, an image can now be linked as a public thumbnail to another of my softwares, PhotoOp, using a TT macro.
Here's the apache config:
<Location /~castaway/blog>
SetHandler perl-script
PerlHandler Blog::Handler
PerlHandler Blog::RSS
PerlSetVar files /mnt/nessie/projects/gtd/stikkit/
PerlSetVar wrapper /mnt/nessie/projects/blog/wrapper.html
PerlSetVar macros /mnt/nessie/projects/blog/macros.tt
PerlSetVar author 'Jess Robinson'
PerlSetVar blog_title 'Under the palm tree'
PerlSetVar blog_link 'http://desert-island.me.uk/~castaway/blog'
PerlSetVar blog_id 'http://desert-island.me.uk/blog'
PerlSetVar blog_photo_base 'http://desert-island.me.uk:5005'
</Location>
@public,perl,blog
At work, we have a bunch of in-house written, from-scratch (more or less), software. As we're a software house, this isn't terribly surprising, you might think.
However the software I'm talking about isn't the stuff we sell. It's the basic code that holds up the website. There's an entire self-written CMS, forms system, publishing system and a host of other things. Most of this stuff evolved, from simple CGI scripts written years ago, relying on text files and databases and so on. Some of those .cgi scripts even still exist in remote corners.
Larger and frequently used systems have been refactored into more hands-off tools, optimised for minimal maintenance.
Smaller systems languish, seemingly not enough used to refactor and improve. And yet they waste the time of us workers, more than we'd like.
Managements idea is to replace everything with some off-the-shelf pre-written software, and all will be good. This is of course somewhat of a pipe-dream, one which the people actually running that project are quite aware of. It will need customising, of course.
So there's two extremes: Software we've written from scratch, and software we just install, or pay for hosted. Where is the in-between layer? I'm thinking what we need more of is mostly-done solutions, components. Things like Catalyst plugins and similar come close, but even those are often self-contained. How do I integrate, for example, MojoMojo (wiki) and Angerwhale (blog) so that both use the same user source? Ok, those are both classed as complete applications, but that's my problem. I want them as parts of a whole.
Anyway, all this rambling, what I actually want to get started on is:
We have a system for writing forms, another one that builds a calendar of events, and a third that stores training courses and their dates/prices. None of these integrate with the other, and they're all updated by hand, that is, by programmers, not by people running the events/trainings.
After stumbling around the internet for a while looking for a plausible off-the-shelf solution, I decided to just write one (this is how frustrating finding software has become..) I did find a couple of not too terrible sounding ones, but they appear to be all hosted. For eg: Tendenci, which is actually part of a CMS.
So far I've managed to write a simple (i.e. not terribly normalised, but sufficient to reproduce existing systems) DB schema, and a somewhat bare Catalyst app. Now I need to make an admin interface/controller, and one to actually display the events list and the sign-up forms.
@public,perl,ironman
Last week I talked about contributing to the URI Perl module to fix the bug I think I've found.
Eagle-eyed among you will notice that I didn't actually include any data in my example. So the first thing I need to do is reproduce the possible bug reliably, on the latest edition of the code. The reason I didn't yesterday is that I couldn't find an example, typically, it's probably buried back in one of the old Apache error logs from a few weeks ago.
After attempting to do this for a while, I've found I can't reproduce it at all. Very odd, but not uncommon. I guess I was doing something rather silly (repeatedly).
This is what testing is for, so I shall leave this endeavour for now and talk about something new.
@public,perl,uri,ironman
This is a bit of a meta-post, but bear with me.
Quite often Open Source software is written in people's spare time. Even folks who manage to make a living from customising, installing and supporting open source tools, also contribute in their "spare time". (I put this in quotes because, essentially that work time and spare time meld into one another, such that its hard to tell one from the other).
In the interests of not leaving people out, I'll also mention that so is a lot of not-open-source software, people write tools/apps/games to sell as well (see the iphone apps service).
What is spare time? The definitionn I'm using says its time that isn't paid for by an employer. Time that the person can choose to spend however they wish.
I and many other people I know choose to spend that time not only writing and contributing to, but also supporting open source (monetarily free) software. We do it because we enjoy helping people get on, because we want the software to be used, and so on.
Eventually, to my point: Supporting people is not free, it comes at the cost of supporting other people, or writing more documentation and software. If supporting one person wastes time because they don't help to find the answer themselves, or can't provide actual code that illustrates their problem, and so on, that's time wasted that could have helped other folks with clearer problems.
Similarly, writing software is not free. Writing solutions for other people, implementing their code, comes at the cost of writing code for everyone.
While I'm at it, adminning open source software websites and systems isn't free either. Configuring installs, updating planet configs, making icons and css. All this doesn't take seconds.
Looking back at the last couple of weeks, I'm amazed at how much time I haven't spent doing what I would actually prefer in my spare time (supporting users). Yet it needs doing, if we don't build up and contribute to the community, it doesn't exist. Now if Matt can start another rant about redesigning/upgrading perl-community websites, I wonder how much of that would also get done.
How To Ask Questions The Smart Way
@public,perl,ironman
There I was, the other week, attempting to solve a problem in my app. I was attempting to construct a URI consisting of the previous set of query parameters and my applications base uri.
my $uri = URI->new($ENV{REQUEST_URI};
$uri->query("");
my $queryparams = join('&', map { "${_}=" . join("&${_}=", @{ $back_data{$_} } ) } keys(%back_data));
$uri->query($queryparams);
But I kept getting odd escaping errors for some reason:
Undefined subroutine &URI::Escape::escape_char callled at /nethomes/jessrobinson/local/perl-linux/lib/perl5/site_perl/5.8.8/URI/_query.pm
I tried upgrading to the latest release of URI from CPAN, but it still errored. So I worked around the problem and moved on.
Later on I went back and looked at URI on CPAN. I looked at the reported bugs, via the "View/Report Bugs (14)" link on that page. It's an oft used module, and yet has a small pile of open bugs. Mine is not among them, now I have several options:
It looks like the author, Gisle Aas, does keep up with, or at least comment on, some of the open bugs. Maybe he's just also quite busy.
I asked about on IRC whether Gisle was about there, he's not, but I was told that he stores all his modules on github, so now I can can grab the current source code to check my bug against.
@public,perl,uri,ironman
After my last post, and having discovered that The Enlightened Perl Organisation/EPO were in fact attempting to do part of what I described, I joined up to help them herd cats. So now I'm part of the Extended Core Working Group. Has a fancy name doesn't it?
Enlightened Perl is about improving the public face of Perl, and thus encouraging companies to use it.
They have a Wiki, which doesn't have terribly much on it yet. What it does have is a list of Issues, which I read, and then tweaked a bit. The item that stuck out the most to me, was the one about CPAN not being a good tool for seasoned professionals (or in general). Why not, and why CPANPLUS isn't either, wasn't mentioned. So here's my guess as to the reasons, and some ideas for ways of solving it.
CPAN and CPANPLUS do their jobs pretty well, however those jobs are quite Perl-centric. They just build and install Perl modules and their Perl dependencies. I think what we need is something that integrates better with existing package management systems (e.g. dpkg, rpm, emerge).
Before you say "But CPANPLUS has a thingy to build .deb files!" keep listening.. I said to James, "How well do you know aptitude" and from there we discussed for an hour or so.
My initial idea was that what we need is a tool that is similar to aptitude, in that it would visually display the available packages, their dependencies and so on. The thing would then install the module, AND interact with the actual package management system of whatever OS it was on, and convince it that that module was installed. It would need several backends, for dpkg, rpm and so on.
As for figuring out actual real dependencies, for example XML::LibXML requires libxml to be installed, the tool maintainers would initially run their own CPAN server, and enhance the META.yaml or packages files to contain this info.
James however had a much better idea for the frontend. Instead of writing our own frontend, it would be saner to have the server serve the appropriate types of packages for various package managers. E.g. for dpkg, a service that can be added to /etc/apt/sources.list. And if necessary, build packages on the fly, for requested modules.
It would probably be easier to start with a subset of CPAN distributions, say for example, the one that EPO-EC decides to support.
Contribute and/or discuss (This is a Kwiki, for spam avoidance, please enter the page token 'TC' to actually edit the page)
@public,perl
