• Test::Builder2 at 10k Feet
• Finally, some Test::Builder2 examples!

Posted by Alberto Simões on Grants grants at Thu, 09 Sep 2010 10:25:29 -0000 | Permalink

For my PDX.pm presentation tonight on Test::Builder2 I threw together some quick examples of some of its killer features, in particular demonstrating changing how Test::Builder2 behaves using method modifiers and applying object roles.

First, demonstrating end-of-assert actions, there's die on fail but even cooler is DEBUG on fail! That's right, run your test in the debugger and have it automatically set a breakpoint on a failure. How cool is that?

I'm sure somebody with better debugger foo than I can make it even cooler and stop at the top of the assert stack rather than inside DebugOnFail.

The second is reimplementing Test::NoWarnings safely. TB2::NoWarnings demonstrates hooking into the start and end of the test as well as safely altering the number of tests planned by trapping the call to set_plan.

You can safely use them all together, though its a crap shoot if DebugOnFail or DieOnFail will trigger first.

While roles and method modifiers are relatively new to the Perl community, using them in lieu of designing my own event system for TB2 has two great advantages. First, I didn't have to design and debug my own event system. :) Second, rather than having to learn the quirks of a one-off system, you learn the quirks of Mo[uo]se and then can apply that knowledge all over the place.

There's a pile of stuff to be done in TB2, a lot of them are fairly small and self contained. Have a look. Patches welcome.

Posted by schwern on journal at Thu, 09 Sep 2010 08:33:42 -0000 | Permalink

Chris Nandor is changing employers, so he won't have access to the machine that hosts use.perl after Friday. He's going to take a database dump with him and turn the site into a static site so the content will still be there for a bit, but eventually it's going to disappear. This might mean that a reboot, power outage, disk failure, or something else might means the current use.perl doesn't come back to life.

The Perl community was very fortunate that Geeknet (and all the other names they went by) allowed Chris to host the site. Use.perl was a test bed for slashcode, so whatever Slashdot did or wanted to do, that's what use.perl did. It was a good thing going for a long time, and many of us owe Chris a lot of beer for use.perl, even if we didn't always like slashcode.

Now it's time to figure out what to do without use.perl and all the links to various posts in there. The main goals are:

• Links to URLs within use.perl continue to serve the same content
• People get to keep their use.perl content if they'd like to do something else with it.

Since its slash, a lot of the stuff is dynamic. Since use.perl basically ran on autopilot, Chris doesn't have the bandwidth. Since use.perl won't be connected to his new employer's tasks, he'll have even less time for it. Maybe there's someone who has the bandwidth for it.

Some possible solutions we talked about:

• Set up slashcode and serves the site dynamically. If hosting is a problem, talk to me. :)
• Set up slashcode and serves the site statically. This might mean a caching layer that stores the computed page for every URL. I'm not sure if that will handle the Ajax stuff properly though.
• Take the database dump and recreate the pages somehow.
• Crawl the entire site and store the results as static pages.

One idea was to inject everything into blogs.perl.org. I don't know how that would work or if the blogs.perl.org team want to do that. It would require a very large URL rewriting component.

Posted by brian d foy at Thu, 09 Sep 2010 05:13:45 -0000 | Permalink

Modern Perl is all the rage these days in case you hadn't noticed. But why, might you ask. I will demonstrate a scraper program that stores its results in an object database using such juicy bits as Scrappy (a web scraper that in turn uses the Web::Scraper toolkit), KiokuDB (an object store), and of course Moose.

The Goal

Search craigslist for homes matching certain criterion. For example, we'll search for cabins in Montana for sale by owner that fall within a certain price range.

The CraigHome Class

The search results give rise to natural objects where one object is a single result (ad) of the search. The search we'll study is about finding a home (cabin). Let's call the class CraigHome. It is defined like a typical (simple) Moose class:

package CraigHome;
use Moose;
use namespace::autoclean;

has id => (
    is  => 'ro',
    isa => 'Int',
);

has text => (
    is  => 'ro',
    isa => 'Str',
);

has 'link' => (
    is  => 'ro',
    isa => 'URI',
);

has amount => (
    is  => 'ro',
    isa => 'Num',
);

has search_URL => (
    is  => 'ro',
    isa => 'Str',
);

has search_keywords => (
    is  => 'ro',
    isa => 'Str',
);

has search_name => (
    is  => 'ro',
    isa => 'Str',
);

__PACKAGE__->meta->make_immutable;
1

Search and Store Results

We'll use a script to scrape the search results and then store them for later analysis.

Load needed modules

use strict;
use warnings;

use Scrappy qw/:syntax/;
use KiokuDB;
use KiokuDB::Backend::DBI;
use CraigHome;

Notice that we use Scrappy, KiokuDB, the DBI backend for KiokuDB and our custom class CraigHome.

Define the Search Parameters

The search parameters include:

• a search name
• a URL to search
• keywords to search for
• max. and min. dollar amounts

my %search_definitions = (
    1 => {
        search_name => 'montana_cabin_by_owner',
        search_URIs => [
            'http://montana.craigslist.org/reo/',
            'http://missoula.craigslist.org/reo/', 
            'http://bozeman.craigslist.org/reo/'
        ],
        keywords  => [qw/ cabin /],
        max_price => 60000,
        min_price => 1,
    },
);

Connect to Kioku Data Store and Designate Columns for Searching

In this example, we use the DBI backend and connect to a SQLite database where we store stuff. In addition, we define two attributes of our CraigHome object that we want to search on. Doing so will index the attributes (so to speak) to provide for more efficient searches. One could always just grep the object attributes from a list of all objects.

my $db = KiokuDB->connect(
    "dbi:SQLite:dbname=db/craighomes.db",
    create  => 1,
    columns => [
        search_name => {
            data_type   => "varchar",
            is_nullable => 0, 
        },
        amount => {
            data_type   => "int",
            is_nullable => 0, 
        },
    ]
);
my $scope_object = $db->new_scope;

Run the main logic

main();

sub main {
    init;
    user_agent random_ua;

    # Do the searches
    foreach my $search_definition ( values %search_definitions ) {

        # A search definition can have mulitple search URLs 
        # for the same keywords
        foreach my $search_URL ( @{ $search_definition->{search_URIs} } ) {
            process_search( $search_URL, $search_definition );
        }
    }
}

The main logic initializes a Scrappy agent then loops through each search defined (only 1 in this example) and then processes each search URL. In this example we search the (old) Montana page, then the city pages for Missoula and Bozeman.

Process the Search

This part fills out the search form then extracts each listing from the results using Scrappy. Notice the barewords Scrappy uses: init, user_agent, form_fields, loaded etc. We then pass each listing to the process_listing function.

sub process_search {
    my $search_URL        = shift;
    my $search_definition = shift;

    get $search_URL;
    my $keywords_string = join ' ', @{ $search_definition->{keywords} };
    my $search_name = $search_definition->{search_name};
    form fields => {
        'minAsk' => $search_definition->{min_price},
        'maxAsk' => $search_definition->{max_price},
        'query'  => $keywords_string,
    };

    print "Processing search: ", $search_definition->{search_name},
      " at URL: ${search_URL} with keywords: $keywords_string\n";

    # Process each listing, looking for keyword match.
    if (loaded) {
        var listings => grab 'p a', { name => 'TEXT', link => '@href' };
        var listings_textos => grab 'p', { name => 'TEXT' };
        foreach my $listing ( list var->{listings} ) {
            process_listing( $listing, $search_URL, $keywords_string, $search_name );
        }
    }
}

Process each Listing

Now that we've plucked the listings we're interested in, let's store them for later analysis. Here we build an instance of our custom CraigHome class, and then persist (store) the object using KiokuDB.

sub process_listing {
    my $listing         = shift;
    my $search_URL      = shift;
    my $keywords_string = shift;
    my $search_name     = shift;

    my $listing_amount = listing_amount($listing);
    my $listing_id     = listing_id($listing);
    my $listing_object = CraigHome->new(
        amount          => $listing_amount,
        text            => $listing->{name},
        'link'          => $listing->{link},
        id              => $listing_id,
        search_URL      => $search_URL,
        search_keywords => $keywords_string,
        search_name     => $search_name,
    );
    if ( is_new_listing_id($listing_id) ) {
        $db->store( $listing_id => $listing_object );
    }
}

Now imagine writing something like this in the 1990's or even early 2000's, ouch. Modern Perl helps get shit done with grace and elegance.

Some helper Functions

For completeness we have the following helper functions that extract the listing amount, listing id and determine if we have already have the listing on store.

sub listing_amount {
    my $listing = shift;
    my ($amount) = $listing->{name} =~ m{^\$(\d+)};

    return $amount;
}

sub listing_id {
    my $listing = shift;
    my ($listing_id) = $listing->{link} =~ m/(\d+)\.html$/;

    return $listing_id;
}

sub is_new_listing_id {
    my $listing_id = shift;

    return !$db->lookup($listing_id) ? 1 : 0;
}

Further Analysis

Now that we have the information on store, we would like to report on it. i.e. retrieve the listing objects that we just stored. That part is left as an exercise to the reader or stay tuned for a follow up article.

Credits

Thanks to and for penetrating my thick skin with an inkling of knowledge about KiokuDB.

Posted by mateu on Perl at Thu, 09 Sep 2010 02:13:34 -0000 | Permalink

After 6 or 7 months (mainly waiting around for the next "recommended upgrade instruction" from the SQLite project) the latest DBD::SQLite release should occur next week.

You can get the 1.30_06 release candidate from the CPAN, or from the following URL if your mirror hasn't synced yet.

http://svn.ali.as/cpan/releases/DBD-SQLite-1.30_06.tar.gz

Apart from the normal batch of SQLite upgrades (from 3.6.22 to 3.7.2), bug fixes, and minor enhancements, this release has two changes that may break your code.

These changes have been in the dev releases for some time, but you may want to take the opportunity to test more intensively if you use either of the following features.

1. BLOB columns with UTF8 content

- Resolved #54271: Inserting a string with utf-8 flag on
  corrupts BLOB data; now BLOB data is always stored as bytes
  (without the utf-8 flag) even if it has the flag set (ISHIGAKI)

2. FTS3 queries

- Added support for FTS3 tokenizers written in Perl. Added tests
  and documentation on how to use FTS3. Changed compilation flag
  to use the recommanded -DSQLITE_ENABLE_FTS3_PARENTHESIS
  *** MAY POSSIBLY BREAK OLD APPLICATIONS THAT ALREADY USED FTS3 ***
  (DAMI)

If you are currently using FTS3, please see DBD::SQLite::FTS3Transitional which contains a helper function for automatically upgrading old FTS3 queries to the new syntax.

Posted by Alias on journal at Thu, 09 Sep 2010 02:11:54 -0000 | Permalink

After 6 or 7 months (mainly waiting around for the next "recommended upgrade instruction" from the SQLite project) the latest DBD::SQLite release should occur next week.

You can get the 1.30_06 release candidate from the CPAN, or from the following URL if your mirror hasn't synced yet.

http://svn.ali.as/cpan/releases/DBD-SQLite-1.30_06.tar.gz

Apart from the normal batch of SQLite upgrades (from 3.6.22 to 3.7.2), bug fixes, and minor enhancements, this release has two changes that may break your code.

These changes have been in the dev releases for some time, but you may want to take the opportunity to test more intensively if you use either of the following features.

1. BLOB columns with UTF8 content

- Resolved #54271: Inserting a string with utf-8 flag on
  corrupts BLOB data; now BLOB data is always stored as bytes
  (without the utf-8 flag) even if it has the flag set (ISHIGAKI)

2. FTS3 queries

- Added support for FTS3 tokenizers written in Perl. Added tests
  and documentation on how to use FTS3. Changed compilation flag
  to use the recommanded -DSQLITE_ENABLE_FTS3_PARENTHESIS
  *** MAY POSSIBLY BREAK OLD APPLICATIONS THAT ALREADY USED FTS3 ***
  (DAMI)

If you are currently using FTS3, please see DBD::SQLite::FTS3Transitional which contains a helper function for automatically upgrading old FTS3 queries to the new syntax.

Posted by Alias at Thu, 09 Sep 2010 02:11:54 -0000 | Permalink

Google Summer of Code is a global program that offers student developers summer stipends to write code for various open source software projects. Google Summer of Code 2010 went by quickly, and much was accomplished. The Perl Foundation and Parrot Foundation took part this year, and we were lucky to get proposals from very bright and capable students. We started the summer with 10 students and had 8 students pass their final evaluations. The passing projects include:

Ryan Jendoubi -- Ctypes for Perl
Mentor: Reini Urban
Blog: http://blogs.perl.org/users/doubi/
Repo: http://gitorious.org/perl-ctypes/perl-ctypes

This project is exciting many Perl developers, because it would minimize the need to use XS, which will make many more pure-Perl modules possible. This improves portability, becaue XS-based modules are notorious for being fragile across operating systems and compiler versions. This adds up to a whole lot of WIN.

Nat Tuck -- Hybrid Threads for Parrot
Mentor: Andrew Whitworth
Blog: http://parrot.org/blog/836
Repo: http://github.com/parrot/parrot/tree/gsoc_threads

Threads allow a single program to use more than one CPU, which is becoming increasingly important these days. Even mobile phones are multicore! This work aimed at adding threading support to Parrot Virtual Machine. Much was accomplished, but this effort is still on-going. So-called "green threads" were implemented, which is a necessary step to get Hybrid threads working.

Tyler Curtis -- A PAST Optimization Framework for Parrot
Mentor: chromatic
Blog: http://parrot.org/blog/839
Repo: http://github.com/ekiru/tree-optimization

This project is about providing a framework for optimizing PASTs (Parrot Abstract Syntax Trees). This will be used by language implementors when optimizing their HLLs (High Level Languages). This framework allows all languages on Parrot to benefit from optimizations that are written once, instead of each language implementor writing their own optimizations.

Daniel Arbelo Arrocha -- NFG and single-representation strings for Parrot
Mentor: Allison Randal
Blog: https://www.parrot.org/darbelo
Repo: http://github.com/parrot/parrot/tree/gsoc_nfg

NFG stands for Normal Form Grapheme, and basically means having a standard internal representation of Unicode strings, so that very expensive conversions do not have to repeatedly take place. This makes string-heavy computations much faster and unifies a lot of code.

Carl Masak -- Add support for binary data in Rakudo
Mentor: Jonathan Worthington
Blog: http://use.perl.org/~masak/journal/
Repo: http://github.com/rakudo/rakudo

Rakudo Perl 6 now supports various binary data formats that were implemented as part of this project. Many relevant tests were also added to the Perl 6 Spec Test Suite as well as improvements and clarifications to the Perl 6 Specification.

Muhd Khairul Syamil Hashim -- Instrumentation Tool for Parrot
Mentor: Christoph Otto
Blog: http://www.parrot.org/blog/841
Repo: http://github.com/khairulsyamil/parrot-instrument

This instrumentation tool for Parrot allows one to dynamically peek into the execution of Parrot op-codes. This allows for writing profiling tools that can answer questions like "who calls functions X" and "how many Objects of type X were created."

John Harrison -- Improvements to NCI/LLVM Stack Frame Builder for Parrot
Mentor: Peter Lobsinger
Blog: http://www.parrot.org/ash
Repo: http://github.com/ashgti/parrot

This project is a prerequisite for a JIT (Just In Time compilation) framework, which is an important goal for the Parrot community, since we decided that our old JIT subsystem was broken beyond repair and removed it. Parrot has decided to use the very popular LLVM project in our rewrite of our JIT, and this project brings us a step closer on our journey.

Pawel Murias -- Mildew and SMOP on CPAN
Mentor: Daniel Ruoso
Repo: http://github.com/perl6/mu

This project involved working on Mildew and SMOP. Mildew is a Perl 6 implementation, and SMOP is the library Mildew uses for meta-object programming. You can think of Mildew as a sister to Rakudo Perl 6. Having many implemenations of Perl 6 helps to better define the Perl 6 specification. Updated versions of SMOP and Mildew are now available on CPAN.

The failing projects were:

Justin Hunter -- Rework Catalyst framework instance initialization code
Mirko Westermeier -- Bulletproofing the Mojolicious test suite

Both of these projects passed their midterms, but due to circumstances outside of the program, these students were not able to complete their goals for their final evaluation. Sometimes Real Life throws you a curve ball, like starting a new job, moving to a new city, having a baby and similar things. We wish these students the best of luck, and hope that they complete their projects outside the structure of GSoC.

I am very proud and humbled by all the students and mentors that I worked with this year. I am constantly reminded that there are very intelligent developers that are very young, and The Perl Foundation and Parrot Foundation is very lucky to attract them and have them in our communities. I firmly believe that the passing GSoC 2010 projects have made a large positive impact on our codebases and many people will benefit from them for years to come.

Rock on and keep spreading the Open Source love!

Posted by Jonathan Leto at Thu, 09 Sep 2010 00:43:55 -0000 | Permalink

Google Summer of Code is a global program that offers student developers summer stipends to write code for various open source software projects. Google Summer of Code 2010 went by quickly, and much was accomplished. The Perl Foundation and Parrot Foundation took part this year, and we were lucky to get proposals from very bright and capable students. We started the summer with 10 students and had 8 students pass their final evaluations. The passing projects include:

Ryan Jendoubi -- Ctypes for Perl
Mentor: Reini Urban
Blog: http://blogs.perl.org/users/doubi/
Repo: http://gitorious.org/perl-ctypes/perl-ctypes

This project is exciting many Perl developers, because it would minimize the need to use XS, which will make many more pure-Perl modules possible. This improves portability, becaue XS-based modules are notorious for being fragile across operating systems and compiler versions. This adds up to a whole lot of WIN.

Nat Tuck -- Hybrid Threads for Parrot
Mentor: Andrew Whitworth
Blog: http://parrot.org/blog/836
Repo: http://github.com/parrot/parrot/tree/gsoc_threads

Threads allow a single program to use more than one CPU, which is becoming increasingly important these days. Even mobile phones are multicore! This work aimed at adding threading support to Parrot Virtual Machine. Much was accomplished, but this effort is still on-going. So-called "green threads" were implemented, which is a necessary step to get Hybrid threads working.

Tyler Curtis -- A PAST Optimization Framework for Parrot
Mentor: chromatic
Blog: http://parrot.org/blog/839
Repo: http://github.com/ekiru/tree-optimization

This project is about providing a framework for optimizing PASTs (Parrot Abstract Syntax Trees). This will be used by language implementors when optimizing their HLLs (High Level Languages). This framework allows all languages on Parrot to benefit from optimizations that are written once, instead of each language implementor writing their own optimizations.

Daniel Arbelo Arrocha -- NFG and single-representation strings for Parrot
Mentor: Allison Randal
Blog: https://www.parrot.org/darbelo
Repo: http://github.com/parrot/parrot/tree/gsoc_nfg

NFG stands for Normal Form Grapheme, and basically means having a standard internal representation of Unicode strings, so that very expensive conversions do not have to repeatedly take place. This makes string-heavy computations much faster and unifies a lot of code.

Carl Masak -- Add support for binary data in Rakudo
Mentor: Jonathan Worthington
Blog: http://use.perl.org/~masak/journal/
Repo: http://github.com/rakudo/rakudo

Rakudo Perl 6 now supports various binary data formats that were implemented as part of this project. Many relevant tests were also added to the Perl 6 Spec Test Suite as well as improvements and clarifications to the Perl 6 Specification.

Muhd Khairul Syamil Hashim -- Instrumentation Tool for Parrot
Mentor: Christoph Otto
Blog: http://www.parrot.org/blog/841
Repo: http://github.com/khairulsyamil/parrot-instrument

This instrumentation tool for Parrot allows one to dynamically peek into the execution of Parrot op-codes. This allows for writing profiling tools that can answer questions like "who calls functions X" and "how many Objects of type X were created."

John Harrison -- Improvements to NCI/LLVM Stack Frame Builder for Parrot
Mentor: Peter Lobsinger
Blog: http://www.parrot.org/ash
Repo: http://github.com/ashgti/parrot

This project is a prerequisite for a JIT (Just In Time compilation) framework, which is an important goal for the Parrot community, since we decided that our old JIT subsystem was broken beyond repair and removed it. Parrot has decided to use the very popular LLVM project in our rewrite of our JIT, and this project brings us a step closer on our journey.

Pawel Murias -- Mildew and SMOP on CPAN
Mentor: Daniel Ruoso
Repo: http://github.com/perl6/mu

This project involved working on Mildew and SMOP. Mildew is a Perl 6 implementation, and SMOP is the library Mildew uses for meta-object programming. You can think of Mildew as a sister to Rakudo Perl 6. Having many implemenations of Perl 6 helps to better define the Perl 6 specification. Updated versions of SMOP and Mildew are now available on CPAN.

The failing projects were:

Justin Hunter -- Rework Catalyst framework instance initialization code
Mirko Westermeier -- Bulletproofing the Mojolicious test suite

Both of these projects passed their midterms, but due to circumstances outside of the program, these students were not able to complete their goals for their final evaluation. Sometimes Real Life throws you a curve ball, like starting a new job, moving to a new city, having a baby and similar things. We wish these students the best of luck, and hope that they complete their projects outside the structure of GSoC.

I am very proud and humbled by all the students and mentors that I worked with this year. I am constantly reminded that there are very intelligent developers that are very young, and The Perl Foundation and Parrot Foundation is very lucky to attract them and have them in our communities. I firmly believe that the passing GSoC 2010 projects have made a large positive impact on our codebases and many people will benefit from them for years to come.

Rock on and keep spreading the Open Source love!

Posted by Jonathan Leto on gsoc parrot perl rakudo gsoc2010 at Thu, 09 Sep 2010 00:00:25 -0000 | Permalink

Posted by denny at Wed, 08 Sep 2010 23:05:10 -0000 | Permalink

Chris Nandor is changing employers, so he won't have access to the machine that hosts use.perl after Friday. He's going to take a database dump with him and turn the site into a static site so the content will still be there for a bit, but eventually it's going to disappear. This might mean that a reboot, power outage, disk failure, or something else might means the current use.perl doesn't come back to life.

The Perl community was very fortunate that Geeknet (and all the other names they went by) allowed Chris to host the site. Use.perl was a test bed for slashcode, so whatever Slashdot did or wanted to do, that's what use.perl did. It was a good thing going for a long time, and many of us owe Chris a lot of beer for use.perl, even if we didn't always like slashcode.

Now it's time to figure out what to do without use.perl and all the links to various posts in there. The main goals are:

• Links to URLs within use.perl continue to serve the same content
• People get to keep their use.perl content if they'd like to do something else with it.

Since its slash, a lot of the stuff is dynamic. Since use.perl basically ran on autopilot, Chris doesn't have the bandwidth. Since use.perl won't be connected to his new employer's tasks, he'll have even less time for it. Maybe there's someone who has the bandwidth for it.

Some possible solutions we talked about:

• Set up slashcode and serves the site dynamically. If hosting is a problem, talk to me. :)
• Set up slashcode and serves the site statically. This might mean a caching layer that stores the computed page for every URL. I'm not sure if that will handle the Ajax stuff properly though.
• Take the database dump and recreate the pages somehow.
• Crawl the entire site and store the results as static pages.

One idea was to inject everything into blogs.perl.org. I don't know how that would work or if the blogs.perl.org team want to do that. It would require a very large URL rewriting component.

Posted by brian d foy at Wed, 08 Sep 2010 22:13:42 -0000 | Permalink

In a recent post, I said I wished that Perl's built in functions for array containers would work directly on references.

Rather than this (today):

# Assuming $data->{$key1}{$key2} = [ qw/foo bar/ ]
push @{ $data->{$key1}{$key2} }, @stuff;

I wanted this (in the future):

# Assuming $data->{$key1}{$key2} = [ qw/foo bar/ ]
push $data->{$key1}{$key2}, @stuff;

I've finished a draft implementation that works for push, pop, shift, unshift and splice. All existing Perl tests pass, as do new tests I've written that explore this new functionality. It will even auto-vivify as needed:

my $foo;
push $foo, @stuff; # $foo is now an arrayref

I'm still working on keys, values and each and those look much harder, but I hope to have something to share by next week.

Updated: I added example data to the "given $data..." comments to clarify that it's an example, not a recommendation about how to initialize a data structure.

Posted by dagolden on perl-programming at Wed, 08 Sep 2010 21:33:24 -0000 | Permalink

In a recent post, I said I wished that Perl's built in functions for array containers would work directly on references.

Rather than this (today):

# Assuming $data->{$key1}{$key2} = [ qw/foo bar/ ]
push @{ $data->{$key1}{$key2} }, @stuff;

I wanted this (in the future):

# Assuming $data->{$key1}{$key2} = [ qw/foo bar/ ]
push $data->{$key1}{$key2}, @stuff;

I've finished a draft implementation that works for push, pop, shift, unshift and splice. All existing Perl tests pass, as do new tests I've written that explore this new functionality. It will even auto-vivify as needed:

my $foo;
push $foo, @stuff; # $foo is now an arrayref

I'm still working on keys, values and each and those look much harder, but I hope to have something to share by next week.

Updated: I added example data to the "given $data..." comments to clarify that it's an example, not a recommendation about how to initialize a data structure.

Posted by dagolden at Wed, 08 Sep 2010 21:33:24 -0000 | Permalink

Read more of this story at use Perl.

Posted by pudge at Wed, 08 Sep 2010 20:51:00 -0000 | Permalink

I use perldoc all the time (the command-line version). I don't know how to be a productive Perl programmer without perldoc -f or perldoc Module::Name. Nor would I try.

One of Perl 5's selling points is that includes copious documentation of the core language and common tasks. If you've skim perldoc perltoc once in a while, you'll find documentation you really ought to read as well as documentation you will wish you had read previously.

Then you'll come across something like perlobj which, bless its little heart, really isn't that helpful anymore. It gives far too many implementation details along with helpful-sounding but ultimately unpleasant advice. (Explaining how to use indirect method invocation before warning not to use it in the final paragraph of the section? A lengthy discussion of garbage collection? A comaprison of Perl 5's default OO with C++? Code examples which show how to bypass method dispatch altogether as a didactic technique on how method dispatch works?)

That's not the worst file in the documentation.

Writing documentation is difficult. Writing good documentation is very difficult. Editing existing documentation is thankless and fragile and prone to arguments: someone invested hours in writing that documentation a decade ago. We should celebrate that work and offer our most sincere gratitude to everyone who's helped create the voluminous Perl documentation. Even so, when documentation has grown a layer of mold (or when our understanding of the best way to do things has changed), we need to do users the favor of giving them the best advice we have.

Sadly, we'll never see Programming Perl 4E, but we can work to improve the core documentation in terms of advice, accuracy, organization, and completeness. (Even a prominent link to the TPF Perl 5 wiki for "Object Best Practices" in the documentation could keep it relevant in seven years when so-called enterprise distributions begin to ship a Modern Perl, but that's a far different need.)

Posted by chromatic on documentation, modern perl, perl 5 at Wed, 08 Sep 2010 16:34:33 -0000 | Permalink

After a long time I finally got my own Perl::Critic policies released and uploaded to CPAN.

I have created a Task implementation bundle listing my two own policies for now (I expect to add other policies used later on). This gives you the opportunity to download the policies by themselves if they look like something you would like to use or you can even bundle them yourself with other policies you make use of.

This packaging is an experiment and is not the recommended way and so it might change in the future, so let me know if you have any feedback on the packaging, both if you are using Perl::Critic::logicLAB or the actual policies directly.

The policies which have also just been released and currently listed in Perl::Critic::logicLAB are:

• Perl::Critic::Policy::logicLAB::ProhibitUseLib
• Perl::Critic::Policy::logicLAB::RequireVersionFormat

Both are simple policies, but hopefully useful.

All should be available on CPAN by now.

jonasbn, Copenhagen

Posted by jonasbn on Uncategorized cpan perl perlcritic policies release at Wed, 08 Sep 2010 14:05:35 -0000 | Permalink

I spent several hours debugging a catastrophic test suite failure today. As our test classes take over an hour to run and the failure takes place near the end of the test run, it was a very annoying issue to debug. Unfortunately, the test class in question passed when run by itself, but not when run in the entire suite. That usually means the global state had been altered in an unexpected way, and boy had it! Seems the test was failing because it was trying to run a method that a completely separate test class had -- and those classes weren't related by inheritance. It was very confusing until I finally realized that someone had mistyped a package name at the top of the test class. A few minutes later I had a new test which verified the package name ... and found ten other misspelled package names (out of 325 packages).

You'll need to adjust this to taste (and it assumes that the first thing which looks like package declaration is the package declaration).

#!/usr/bin/env perl

use Test::Most 'bail';
use File::Find;

sub expected_package_name($) {
  my $file = shift;
  $file =~ s{^t/lib/}{};
  $file =~ s{\.pm$}{};
  $file =~ s/\//::/g;
  return $file;
}

sub found_package_name($) {
  my $file = shift;

  # we assume first package name found is actual
  open my $fh, '<', $file or die "Could not open $file for reading: $!";
  my $package;
  while ( my $line = <$fh> ) {
    next unless $line =~ /^\s*package\s+((?:\w+)(::\w+)*)/;
    return $1;
  }
}

my @files;
find(
  { no_chdir => 1,
    wanted   => sub {
      my $file = $File::Find::name;
      return if !-f $file || $file !~ /\.pm$/ || $file =~ /\.svn/;
      push @files =>
        [ $file, found_package_name $file, expected_package_name $file];
    },
  },
  't/lib/Tests'
);
plan tests => scalar @files;
for my $file (@files) {
  my ( $file, $have, $want ) = @$file;
  is $have, $want, "Package name correct for $file";
}

Posted by Ovid at Wed, 08 Sep 2010 13:27:27 -0000 | Permalink

I spent several hours debugging a catastrophic test suite failure today. As our test classes take over an hour to run and the failure takes place near the end of the test run, it was a very annoying issue to debug. Unfortunately, the test class in question passed when run by itself, but not when run in the entire suite. That usually means the global state had been altered in an unexpected way, and boy had it! Seems the test was failing because it was trying to run a method that a completely separate test class had -- and those classes weren't related by inheritance. It was very confusing until I finally realized that someone had mistyped a package name at the top of the test class. A few minutes later I had a new test which verified the package name ... and found ten other misspelled package names (out of 325 packages).

You'll need to adjust this to taste (and it assumes that the first thing which looks like package declaration is the package declaration).

  #!/usr/bin/env perl

use Test::Most 'bail';
use File::Find;

sub expected_package_name($) {
  my $file = shift;
  $file =~ s{^t/lib/}{};
  $file =~ s{\.pm$}{};
  $file =~ s/\//::/g;
  return $file;
}

sub found_package_name($) {
  my $file = shift;

  # we assume first package name found is actual
  open my $fh, '<', $file or die "Could not open $file for reading: $!";
  my $package;
  while ( my $line = <$fh> ) {
    next unless $line =~ /^\s*package\s+((?:\w+)(::\w+)*)/;
    return $1;
  }
}

my @files;
find(
  { no_chdir => 1,
    wanted   => sub {
      my $file = $File::Find::name;
      return if !-f $file || $file !~ /\.pm$/ || $file =~ /\.svn/;
      push @files =>
        [ $file, found_package_name $file, expected_package_name $file];
    },
  },
  't/lib/Tests'
);
plan tests => scalar @files;
for my $file (@files) {
  my ( $file, $have, $want ) = @$file;
  is $have, $want, "Package name correct for $file";
}

Posted by Ovid on test::class testing tests at Wed, 08 Sep 2010 13:18:01 -0000 | Permalink

Here are two examples for how the Amelia Perl logo can be used in actual marketing material. :)

The first one is a book cover for the free Perl 6 book included in the Rakudo Star distribution. (download)

The second is a quick'n dirty (i had one hour) cover for a Mojolicious flyer Marcus Ramberg will be handing out at JavaZone 2010.

P.S.: I'm looking for fun T-Shirt design ideas, leave a comment if you have one!

Permalink | Leave a comment  »

Posted by nobody at Wed, 08 Sep 2010 11:04:22 -0000 | Permalink

Awhile ago I moved my How can I troubleshoot my Perl CGI script? to StackOverflow. I'm just getting around to telling everyone about it because it was pretty far down on my to do list.

I think this has almost pushed the old location on SourceForge out of the googlebrain, but it wouldn't hurt for people to link to it in a blog post, tweet, whatever to encourage Google to find this one. Someday SourceForge will disappear and we won't have to worry about it anymore. How is it even still alive? StackOverflow has pretty good googlejuice though, maybe because Google likes StackOverflow.

Since it's on StackOverflow, this also means that I'm basically letting go of it. StackOverflow encourages people to revise the questions and answers of other to improve them, and I've given it wiki status to encourage that even more. Take a look, see what I've left out (or left in), what's new and exciting (or old and boring).

Even if you don't (or can't) edit it just yet, I'd appreciate any comments on how to bring it up to date. Maybe another StackOverflow user can make the changes if I'm too busy.

Also, sadly, the only thing keeping the bad Perl info out of StackOverflow is a small band of knowledgeable Perlers patrolling the answers (Sinan used my summer absences to pass me as the highest rated Perl user there). If you're looking for a way promote Perl in a useful way (and you actually know Perl), consider helping out. Providing good answers, voting on good answers (and against bad answers), and refining other answers helps the entire world.

Posted by brian d foy at Wed, 08 Sep 2010 04:06:59 -0000 | Permalink

Awhile ago I moved my How can I troubleshoot my Perl CGI script? to StackOverflow. I'm just getting around to telling everyone about it because it was pretty far down on my to do list.

I think this has almost pushed the old location on SourceForge out of the googlebrain, but it wouldn't hurt for people to link to it in a blog post, tweet, whatever to encourage Google to find this one. Someday SourceForge will disappear and we won't have to worry about it anymore. How is it even still alive? StackOverflow has pretty good googlejuice though, maybe because Google likes StackOverflow.

Since it's on StackOverflow, this also means that I'm basically letting go of it. StackOverflow encourages people to revise the questions and answers of other to improve them, and I've given it wiki status to encourage that even more. Take a look, see what I've left out (or left in), what's new and exciting (or old and boring).

Even if you don't (or can't) edit it just yet, I'd appreciate any comments on how to bring it up to date. Maybe another StackOverflow user can make the changes if I'm too busy.

Also, sadly, the only thing keeping the bad Perl info out of StackOverflow is a small band of knowledgeable Perlers patrolling the answers (Sinan used my summer absences to pass me as the highest rated Perl user there). If you're looking for a way promote Perl in a useful way (and you actually know Perl), consider helping out. Providing good answers, voting on good answers (and against bad answers), and refining other answers helps the entire world.

Posted by brian d foy at Tue, 07 Sep 2010 20:41:31 -0000 | Permalink

Thsi is reposted from the CPAN Testers blog:

The End of CPAN Testers 1.0

Posted by Robert at Tue, 07 Sep 2010 16:49:30 -0000 | Permalink

Thsi is reposted from the CPAN Testers blog:

The End of CPAN Testers 1.0

Posted by Robert at Tue, 07 Sep 2010 16:49:30 -0000 | Permalink

It's the end of an era. After 11 years from the first official CPAN Testers test report submission, the cpan-testers mailing list has finally closed its doors. From now on, anyone sending a test report via SMTP, will receive a bounce-back email. Most have already upgraded and reconfigured their smoker clients to use HTTP/S. For those that haven't or are new testers wanting to start afresh, please read the Quick Start page on the CPAN Testers Wiki, and join the cpan-testers-discuss mailing list to ask for help and advice.

For those wondering still wondering why we have moved from SMTP to HTTP submissions, please see my talk from YAPC::Europe, entitled "CPAN Testers 2.0 - I love it when a plan comes together". It was a talk I tried to give some background to for our reasons and motivations for the move. However, the bottom line is that we want CPAN Testers to evolve and grow further, with more diagnostics and analysis. With the old method of reporting and submission, that just wasn't possible. We have lots of ideas to expand the reporting, but first we need CT2.0 to settle in. We need to make upgrading and installation for anyone, especially our casual testers as easy as possible. There is still some fine tuning happening, but we are moving forward.

While at YAPC::Europe this year in Pisa, the White Camel Awards were announced. Despite being a reluctant recipient, I was awarded one for my work here with CPAN Testers. While I might be the glue that keeps us together, it really is all of the CPAN Testers Community that makes it all worthwhile. Though I saw the value and worth of what we do several years ago, and perhaps have had more time and motivation to promote what we do, the award really is a big big thank you to all the testers and developers who have been a part of CPAN Testers for the past 11 years. It has been a pleasure meeting many of you at conferences and workshops over the years, and hopefully I'll get to meet many more in the future. I started my involvement in CPAN Testers because of a BOF Léon Brocard gave at YAPC::Europe 2003 in Paris. In turn over the past few years I've helped to encourage others to become involved. The CPAN Testers Community has continually evolved and become a very respected community based testing team, even outside of the echo-chamber. I look forward to what we can achieve in the next 11 years.

In the final days of the SMTP submissions, I have been back mapping many of the addresses to people. Last month I also sent out the request email, for testers to help identify themselves. I had a great response, to which I thank you all, and managed to update several entries. In addition, several newer testers have followed my request and used their CPAN/PAUSE email, or have identified themselves well enough for me to spot them. In addition I took some time to work though mailing lists, whois entries, HTML source, open source code and all manner of nooks and crannies around the internet to identify many more. In all, over the last 2 months I've mapped 167 addresses to real people, of which 106 are new mappings to testers never previously identified. Many of these are brand new testers.

Because the new system of registering a profile is slightly different, the new mechanism to map testers is now also being fed from the Metabase. As such the new Administration system, that has been waiting in the wings for over a year, needs a few updates to be able to use both the new and old systems together. This shouldn't be a huge amount of change, but it does mean that it will be a little while longer before we can launch it.

One final achievement that needs noting from last month is the fact that we passed the 8 millionth accepted report submission. 11 years ago, I wonder if those first testers ever thought that we would have amassed so many reports. We were getting through roughly 1 million reports every 5-6 months back last year, but the throttling and switch over have reduced the submissions for the first half of this year. We're starting to see the numbers climb again, so I expect this time next year we should have passed the 10 millionth report submission.

Onwards and Upwards....

Cross-posted from The CPAN Testers Blog

Posted by CPAN Testers at Tue, 07 Sep 2010 12:39:18 -0000 | Permalink

I hope to showcase and talk about a great Perl project every once in a while. Hope this lasts beyond one post. :)

We recently added emailing capabilities to an in-house application. Unfortunately sending the emails adds roughly 1 minute to the runtime. I decided to take a gander at swaks and see how it's being done there.

Swaks (SMTP transaction tester) is a great application for all systems administrators. It takes the pain away from testing email setups. Instead of echo msg | mailx -n -t ... and thinking of whether it's heirloom mailx, mail, nail or which version of mail it is, or whether you're using sendmail or exim or qmail or postfix (and in comes the headache...) - you can just write swaks -t addr -f addr. You don't even need the from field, but it's just as easy to add.

Swaks saves you the trouble of tailing logs and instead just shows you everything that's going on in the transport. It's easy to work with and it supports a lot of other options: it works with piping, unix sockets, input/output files, attachments, TLS, authentication, supports 12 mail protocols (from SMTP to LMTPSA), can work as a server(!) and much more!

Swaks is truly one of the most awesome Perl projects in any sysadmin book. Let's look at the core.

The good parts:

• It uses strict!


• It's divides to subroutines (that make sense).


• The code is documented.


• The application is documented (in POD!).


• Clear code, skimmable.


• Relatively concise, compared to the amount of stuff it can do.


• Well thought-out as an application.


• I could not find a single bad thing about this program, only not-so-good stuff.

The not-so-good parts:

• Old indirect style of creating objects.


• No warnings.


• I wish more modules would be used to remove a lot of the code handling cmd opts.

(today using App::FatPacker this is no longer much of a problem)

Overall: This is an excellent project and something of which Perl should be proud, IMHO.

Posted by Sawyer X at Tue, 07 Sep 2010 08:24:16 -0000 | Permalink

I hope to showcase and talk about a great Perl project every once in a while. Hope this lasts beyond one post. :)

We recently added emailing capabilities to an in-house application. Unfortunately sending the emails adds roughly 1 minute to the runtime. I decided to take a gander at swaks and see how it's being done there.

Swaks (SMTP transaction tester) is a great application for all systems administrators. It takes the pain away from testing email setups. Instead of echo msg | mailx -n -t ... and thinking of whether it's heirloom mailx, mail, nail or which version of mail it is, or whether you're using sendmail or exim or qmail or postfix (and in comes the headache...) - you can just write swaks -t addr -f addr. You don't even need the from field, but it's just as easy to add.

Swaks saves you the trouble of tailing logs and instead just shows you everything that's going on in the transport. It's easy to work with and it supports a lot of other options: it works with piping, unix sockets, input/output files, attachments, TLS, authentication, supports 12 mail protocols (from SMTP to LMTPSA), can work as a server(!) and much more!

Swaks is truly one of the most awesome Perl projects in any sysadmin book. Let's look at the core.

The good parts:

• It uses strict!


• It's divides to subroutines (that make sense).


• The code is documented.


• The application is documented (in POD!).


• Clear code, skimmable.


• Relatively concise, compared to the amount of stuff it can do.


• Well thought-out as an application.


• I could not find a single bad thing about this program, only not-so-good stuff.

The not-so-good parts:

• Old indirect style of creating objects.


• No warnings.


• I wish more modules would be used to remove a lot of the code handling cmd opts.

(today using App::FatPacker this is no longer much of a problem)

Overall: This is an excellent project and something of which Perl should be proud, IMHO.

Posted by Sawyer X on Perl at Tue, 07 Sep 2010 06:58:57 -0000 | Permalink

A couple of years ago I put together a list of The Perl Journal articles I could find on the Dr. Dobbs website. They changed some of their URLs, so I updated those to avoid all of the redirects and in the process found several more articles. My TPJ index is on Perlmonks. You can see some of the beginnings of popular projects, such as Moose, in some of the articles.

Posted by brian d foy at Tue, 07 Sep 2010 02:37:37 -0000 | Permalink

A couple of years ago I put together a list of The Perl Journal articles I could find on the Dr. Dobbs website. They changed some of their URLs, so I updated those to avoid all of the redirects and in the process found several more articles. My TPJ index is on Perlmonks. You can see some of the beginnings of popular projects, such as Moose, in some of the articles.

Posted by brian d foy at Tue, 07 Sep 2010 02:35:03 -0000 | Permalink

If you're in the Portland, Oregon area, come to the Portland Perl Mongers September 2010 Meeting on Wednesday, September 8.

Not only will Schwern talk about Test::Builder 2 (Test::Builder is the foundation of the Perl Testing Revolution), but I'll talk about what novices must understand about Perl 5 before they can become adepts.

We're experimenting this month with a combination of novice-friendly primers and deeper subjects. We'd love to attract more people interested in Perl but not necessarily experts as well as people who've never written much Perl at all. Tell your friends.

... and for a preview of my talk, I'll explain:

• The two types of context in Perl 5
• What context really means, and why you already understand it
• What implicit variables really mean and why you already understand them
• What concision means and why it's not as scary as you think
• How to chunk Perl expressions into parts you already understand
• How to navigate the Perl 5 documentation

If you can't make the meeting, I'm sure a podcast will eventually appear. You could also read the draft of Modern Perl chapter 1 for an overview.

Posted by chromatic on modern perl, perl, perl 5, perl mongers, portland, talks at Mon, 06 Sep 2010 19:49:54 -0000 | Permalink

As of this weekend, the final switch to turn off the SMTP gateway for CPAN Testers was flipped. You can no longer post anything to the old cpan-testers mailing list, and any attempts now will result in a bounce message.

Our thanks to Robert and Ask over at the Perl NOC for looking after us all these years, and for being very patient with us while we got the HTTP gateway up and running over the last 9 months.

As a consequence, anyone wishing to still be a part of the CPAN Testers community, now needs to upgrade their test environments, to use the latest smokers and associated libraries. In the main this will involve a simple upgrade of your smoker client and the installation of 4 specific modules (which in turn will install any additional prerequisites needed). You will then need to acquire a metabase profile. For full details of the steps necessary please see the Quick Start page on the CPAN Testers Wiki.

For those casual testers, the upgrade will initially involve some manual intervention, although we hope to automate this as soon as we can. If you do have any problems, or are confused by any of the instructions, please post to the cpan-testers-discuss mailing list, where the developers and other experienced testers can help you.

The end of an era.

Posted by CPAN Testers at Mon, 06 Sep 2010 12:29:10 -0000 | Permalink

From Wikipedia, the free encyclopedia:

In game theory and economic theory, zero-sum describes a situation in which a participant's gain or loss is exactly balanced by the losses or gains of the other participant(s). If the total gains of the participants are added up, and the total losses are subtracted, they will sum to zero.

Being advocate, implementor, tester and co-designer of a new programming language, I often hear objections along the lines of you are killing $other_programming_language, combined with a mixture of fear and resentment. People are afraid that having a new player on the market will decrease market share of their own, favorite programming language.

While I can understand these thinking patterns, there is no reason for concern. The market for programming languages is not a zero-sum situation. While I don't have hard data, I have the impression that the programming job sector is growing, and the US government expects it to grow further, too.

Certainly the growth of world population sets a rapidly increasing baseline, and even if we assume a constant percentage of all people related to programming in some way, the total number of programmers rises, and will continue for quite some time.

(I'm pointing to some resources about programming jobs, and I fully realize that it's not the same as number of overall programmers; but it's easier to get data for jobs, and I do think that the general trend statements are true for both).

So as long as the total number of programmers increases, a decrease in relative market share doesn't automatically mean a loss. In fact the job trends show an increase for "scripting" languages, and while Ruby is certainly the winner in terms of growth, Python, Perl and PHP win too!

Non-job data shows for example a noisy but steady growth of uploads to the Comprehensive Perl Archive Network (CPAN) -- data from a programming language that is often perceived as a loser of ruby's and python's success.

A recent Linux distribution trend analysis fell into the same trap: it shows relative numbers of search terms, and talks about a decline for all distributions except Ubuntu. Again I don't have hard numbers (the mirror infrastructure of most Linux distributions makes it nearly impossible to get accurate download counts), but I haven't seen any evidence that total usage numbers of any of the Linux distributions actually decreased.

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

Posted by david | Permalink

Posted by david | Permalink

In Perl, subroutine and operator names determine what happens, usually not the type of the arguments. Instead the arguments are coerced to a type on which the operation makes sense:

say uc 34;      # coerces 34 to a string, and upper-cases it
say 1 + "2";    # converts "2" to a number before adding

To make things more extensible, the uc function re-dispatches to the uc method on its argument. So for the example above to work, we need an uc function in Int. And in Array, so that @a.uc works. And so on.

The original approach was to stuff all these methods into Any, the base class of the object hierarchy. Which kinda worked, but also meant that all user-defined classes ended up having some few hundreds methods to start with. Not good.

These days, the type Cool fills this niche: most built-in types (all that are meant to be used in that polymorphic way) inherit from Cool, so the uc method is actually defined in class Cool, coerces to string, and then re-dispatches to the internal logic that actually does the upper-casing.

The name either stands for Convenient object oriented loopback, or just expresses that that most built-ins are cool with an argument of that type.

If users want to write a type that can be used like a built-in type now just inherit from Cool, and define coercion methods to other built-in types. If the types don't inherit from Cool, they are more light-weight, and less magic. There's more than one way to do it.

Cool is a class (and not a role), because classes are mutable; so if you want to inject behavior into nearly all built-in types, augmenting Cool is an option (though usually considered evil, and should not be done lightly).

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

Today on IRC, Larry Wall showed this piece of Perl 6 code, which he wrote for Rosetta Code:

sub pascal { [1], -> @p { [0, @p Z+ @p, 0] } ... * };
say pascal[^10].perl
# output (reformatted for easy readbility):
# ([1],
#  [1, 1],
#  [1, 2, 1],
#  [1, 3, 3, 1],
#  [1, 4, 6, 4, 1],
#  [1, 5, 10, 10, 5, 1],
#  [1, 6, 15, 20, 15, 6, 1],
#  [1, 7, 21, 35, 35, 21, 7, 1],
#  [1, 8, 28, 56, 70, 56, 28, 8, 1],
#  [1, 9, 36, 84, 126, 126, 84, 36, 9, 1])

That's Pascal's triangle, generated in one line of Perl 6.

The ... is the series operator, which generates lists by feeding the previous value(s) (here always one array) to the generating block on its left, until it reaches the goal on the right (in this case "whatever", which means it returns a lazy, infinite list).

So for example if the previous item was the array [1, 2, 1], the code block evaluates 0, 1, 2, 1 Z+ 1, 2, 1, 0.

Z is the zip operator, Z+ is pairwise addition (ie adding the pairs that the zip operator produced). In our example that leads to 0+1, 1+2, 2+1, 1+0 or 1, 3, 3, 1.

It takes a while to get used to the meta operators and the series operator, but once you've understood them, you can do pretty neat things with them.

Posted by Moritz Lenz | Permalink

These days you often hear term Modern Perl, as something new(ish), and much improved over the old ways.

But what is it exactly? Well, there's no proper definition, but here is what that term means to me:

It's a set of tools, ideas and attitudes that help you to write better Perl programs, and allows you to have more fun while writing them.

Here are some aspects of Modern Perl

• Testing: Most modern Perl modules have extensive test suites, that make development sane and robust
• Some built-ins now come with safer forms: the three-argument form of open() allows you to open files safely with arbitrary characters in them, without any extra precautions. Lexical file handles make things safer and easier too.
• use strict; use warnings;
• Proper OO: with Perl 6 and with Moose in Perl 5, we have good object systems, that require less low-level fiddling than the standard Perl 5 object system
• Following Best Practices
• (For open source projects) Liberally handing out commit privileges. The source is stored in a version control system anyway, so low-quality changes or vandalism can simply be reverted (but that doesn't happen often in practice).
• Caring about marketing: do tell people that you built something cool and useful
• Small handy modules such as List::Util and Try::Tiny
• Development tools such as Devel::Cover and Devel::NYTProf
• (update) perlbrew and local::lib to help maintain your own perl installation and locally installed modules.

All of these techniques help to write scalable Perl programs by making proper encapsulation much easier, or by avoiding common errors, identifying performance bottlenecks etc.

Update: after watching some discussions about this post in various media, I should add a few more tools that I forgot about earlier:

• Devel::REPL, Read-Evaluation-Print Loop
• Perl::Critic, a code quality and style tester
• DDBIx::Class, a modern and very popular object relational mapper
• Catalyst, a web framework based on the MVC pattern

Posted by Moritz Lenz | Permalink

For this week's contribution to Perl 6 we ask you to implement the $*ARGFILES special variable (and underlying object) for Rakudo.

(Introduction to this series of challenges)

Background

In Perl 5, there is a "magic" way to iterate over input: while (my $line = <>) { ... }. This reads from standard input if no command line arguments were provided. If there are command line arguments, they are taken as file names, and <> iterates over the contents of these files.

In Perl 6 this magic is performed with the $*ARGFILES special variable. Please implement it!

To do that, you have to understand how file reading works in Perl 6. Once you have a file handle, there are two ways to read lines: either by calling $handle.get, which returns just one line, or by calling $handle.lines, which returns a (lazy) list of all the remaining lines.

You can obtain the command line arguments from @*ARGS, open a file with my $handle = open $filename; for reading; And finally you can use lines('filename') to read all lines from a file.

What you can do

Implement a backend class for $*ARGFILES. You can do that in normal Perl 6 code, no need to change the actual compiler. Once that's done, we will plug it into Rakudo. Your code might look like this:

class IO::ArgFiles {
    has @!filenames;
    method get() { ... }
    method lines() { ... }
    method filename() {
        # return the current filename, or '-' if standard input
        ...
    }
}
my $*ARGFILES = IO::ArgFiles.new(filenames => @*ARGS);
.say for $*ARGFILES.lines();

(Of course this is only a rough skeleton, you might need to change some details, or add some things).

Submission

Please submit your source code to the perl6-compiler@perl.org mailing list (and put moritz@faui2k3.org on CC, because the mailing list sometimes lack quite a bit).

Update:: there have been two submissions, and a mixture of both has been applied.

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

NAME

"Perl 5 to 6" Lesson 27 - Common Perl 6 data processing idioms

SYNOPSIS

  # create a hash from a list of keys and values:
  # solution 1: slices
  my %hash; %hash{@keys} = @values;
  # solution 2: meta operators
  my %hash = @keys Z=> @values;

  # create a hash from an array, with
  # true value for each array item:
  my %exists = @keys Z=> 1 xx *;

  # limit a value to a given range, here 0..10.
  my $x = -2;
  say 0 max $x min 10;

  # for debugging: dump the contents of a variable,
  # including its name, to STDERR
  note :$x.perl;

  # sort case-insensitively
  say @list.sort: *.lc;

  # mandatory attributes
  class Something {
      has $.required = die "Attribute 'required' is mandatory";
  }
  Something.new(required => 2); # no error
  Something.new()               # BOOM

DESCRIPTION

Learning the specification of a language is not enough to be productive with it. Rather you need to know how to solve specific problems. Common usage patterns, called idioms, helps you not having to re-invent the wheel every time you're faced with a problem.

So here a some common Perl 6 idioms, dealing with data structures.

Hashes

  # create a hash from a list of keys and values:
  # solution 1: slices
  my %hash; %hash{@keys} = @values;
  # solution 2: meta operators
  my %hash = @keys Z=> @values;

The first solution is the same you'd use in Perl 5: assignment to a slice. The second solution uses the zip operator Z, which joins to list like a zip fastener: 1, 2, 3 Z 10, 20, 30 is 1, 10, 2, 20, 3, 30. The Z=> is a meta operator, which combines zip with => (the Pair construction operator). So 1, 2, 3 Z=> 10, 20, 30 evaluates to 1 => 10, 2 => 20, 3 => 30. Assignment to a hash variable turns that into a Hash.

For existence checks, the values in a hash often doesn't matter, as long as they all evaluate to True in boolean context. In that case, a nice way to initialize the hash from a given array or list of keys is

  my %exists = @keys Z=> 1 xx *;

which uses a lazy, infinite list of 1s on the right-hand side, and relies on the fact that Z ends when the shorter list is exhausted.

Numbers

Sometimes you want to get a number from somewhere, but clip it into a predefined range (for example so that it can act as an array index).

In Perl 5 you often end up with things like $a = $b > $upper ? $upper : $b, and another conditional for the lower limit. With the max and min infix operators, that simplifies considerably to

  my $in-range = $lower max $x min $upper;

because $lower max $x returns the larger of the two numbers, and thus clipping to the lower end of the range.

Since min and max are infix operators, you can also clip infix:

 $x max= 0;
 $x min= 10;

Debugging

Perl 5 has Data::Dumper, Perl 6 objects have the .perl method. Both generate code that reproduces the original data structure as faithfully as possible.

:$var generates a Pair ("colonpair"), using the variable name as key (but with sigil stripped). So it's the same as var => $var. note() writes to the standard error stream, appending a newline. So note :$var.perl is quick way of obtaining the value of a variable for debugging; purposes, along with its name.

Sorting

Like in Perl 5, the sort built-in can take a function that compares two values, and then sorts according to that comparison. Unlike Perl 5, it's a bit smarter, and automatically does a transformation for you if the function takes only one argument.

In general, if you want to compare by a transformed value, in Perl 5 you can do:

    # WARNING: Perl 5 code ahead
    my @sorted = sort { transform($a) cmp transform($b) } @values;

    # or the so-called Schwartzian Transform:
    my @sorted = map { $_->[1] }
                 sort { $a->[0] cmp $b->[0] }
                 map { [transform($_), $_] }
                 @values

The former solution requires repetitive typing of the transformation, and executes it for each comparison. The second solution avoids that by storing the transformed value along with the original value, but it's quite a bit of code to write.

Perl 6 automates the second solution (and a bit more efficient than the naiive Schwartzian transform, by avoiding an array for each value) when the transformation function has arity one, ie accepts one argument only:

    my @sorted = sort &transform, @values;

Mandatory Attributes

The typical way to enforce the presence of an attribute is to check its presence in the constructor - or in all constructors, if there are many.

That works in Perl 6 too, but it's easier and safer to require the presence at the level of each attribute:

    has $.attr = die "'attr' is mandatory";

This exploits the default value mechanism. When a value is supplied, the code for generating the default value is never executed, and the die never triggers. If any constructor fails to set it, an exception is thrown.

MOTIVATION

N/A

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

After reading this blog post, I just want to say: keep going.

Gabor does really awesome things for Perl (and especially for Perl 6, which I tend to notice more): beginner's tutorials, screencasts, training courses, writes an IDE and so on. If his primary interest was his own success, his priorities would be quite different.

I hope that Gabor doesn't pay too much attention to the hostilities from parts of the Perl community, and I wish him and us all the best for his current project.

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

It works if you have a Unixish toolchain. I need to wait for the CPAN-testers reports to see if I managed not to break anything on Windows. Unfortunately, even though the lovely Mr. Alias has worked hard to make Windows machines available to developers, I found it to be just too hard to use. Even downloading my code to the Windows machine was hard, as Windows seemed to think it knew better and shouldn't download the file I told it to download. Then once I had downloaded it, Windows decided to hide it somewhere that I couldn't get to using the command line. So I gave up.

I might try again once there are some decent tools on the machines: wget, tar, and gzip at minimum, as given those I can quickly bootstrap anything else. Software development isn't just about having compilers available.

Posted by david | Permalink

First let my apologize for waiting so long to come up with a new "weekly" Perl 6 challenge - I'm running out of ideas, and the one I have left needs more time to prepare.

Instead I want to motivate you to help porting the ruby scruffy charting library to Perl 6. It can generate shiny SVG graphics (they are currently broken on their main website, hence the waybackmachine link).

There's already an initial version Perl 6 port called "tufte", but it's not running yet. It needs your help. If you know a little ruby, and want to learn some more Perl 6, join #perl6, ask for a commit bit, and translate some ruby code into Perl 6. And in the end you'll be rewarded with nice SVG charts :-).

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

Since the Rakudo Star release, there has been a noticeable increase in Perl 6 questions on perlmonks - a good sign, because it means people are using it.

• Array interpolation in Rakudo
• Rakudo build problems
• Perl6, latest Rakudo bug or error on my part? (parsing issue related to missing whitespace).
• Perl6, modifying @*INC (another ws parsing issue)
• Perl6's LWP::Simple -- seemingly an installation issue
• Perl 6, defining a hash with interpolated string

I've assembled this list by looking through the 100 newest nodes in Seekers of Perl Wisdom, which makes it 6% of the questions asked, or on average 1 per day (used to be around 1 per week).

Most of the questions are related to environmental issues (building/installing stuff), or beginner's questions related to syntax.

It's good to see the questions flowing in, and I hope that we'll soon see more questions where I can show off cool Perl 6 features in the answers :-).

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

Posted by david | Permalink

For this week's contribution to Perl 6 we ask you to implement the Hash.pick method (which does a weighted random selection) for Rakudo.

(Introduction to this series of challenges)

Background

In Perl 6 the List class has a method called pick, which randomly selects one item from a list. It has a few more options too:

<a b c>.pick;       # pick one random element
<a b c>.pick(2);    # pick two distinct, random elements
<a b c>.pick(2, :replace); # pick two random elements, it's ok
                           # if they are the same
<a b c>.pick(*);    # return a random permutation of the elements
<a b c>.pick(*, :replace); # infinite, random stream of elements

This is already implemented through several multi methods in Rakudo.

Now the specification describes such a method for hashes too (actually it talks about Bags, but Rakudo doesn't have Bags yet. Pretend it says "Hash" instead). It assumes that each value in the hash is numeric, and that the value is a weight that determines the probability of picking one value. For example

{a => 1, b => 2}.pick;  # returns 'a' with probability 1/3
                        # and 'b' with probability 2/3

{a => 1, b => 2}.pick(*);  # <a b b> with probability 1/3
                           # <b a b> with probability 1/3
                           # <b b a> with probability 1/3
{a => 1, b => 0.5}.pick(*) # dies, because the weights aren't all integers
{a => 1, b => 0.5}.pick(*, :replace)  # ok 

What you can do

Implement Hash.pick. It's ok if your patch doesn't cover all cases. It would be nice if it supported non-integer weights.

Hint: this could be done by storing a list of accumulated weights, and a list of keys.

{a => 1, b => 2.5, c => 1}

# could translate to 
my @keys = ('a', 'b', 'c');
my @accumulated_weights = (1, 3.5, 4.5);

# now pick a random number between 0 and 4.5,
# find the next-highest index in @accumulated_weights
# with a binary search, and then use that to obtain the key.

Of course other schemes are fine too.

Second hint: because it takes quite some time to recompile Rakudo, it is probably easier to implement the actual logic in a function in a normal source file first, and only later move it into src/core/Hash.pm.

Submission

Please submit your source code to the perl6-compiler@perl.org mailing list (and put moritz@faui2k3.org on CC, because the mailing list sometimes lack quite a bit).

Update: there's one submission on the perl6-compiler mailing list already, which looks pretty good.

Posted by Moritz Lenz | Permalink

During the last one and a half month, I've been working on making regex modifiers easily available in Rakudo.

The regex compiler itself has to support only a few of the adverbs that can be applied to regexes; those include :ignorecase, :sigspace, :ignoremark and :continue/:pos. NQP-rx, the regex engine that Rakudo uses under the hood, supports those (except :ignoremark), so previously you could write

if 'ABC' ~~ /:i abc/ {
    say "case insensitive match";
}

But not

if 'ABC' ~~ rx:i/abc/ {
    say "case insensitive match";
}

nor m:i/abc/, for that matter.

I've patched Rakudo to actually recognize those adverbs outside of the regex, and also for s/// substitutions.

Another category of adverbs are those that apply to regex calls, not to the compilation of a regex. Among those are :global/:g, :overlap/:ov, :nth($n), :x. I've implemented those for substitutions, but implementing them for m// turns out to be quite a bit harder.

The reason is the return value: each regex match returns a Match object, which can store positional and named parts. S05 says that regex matches with multiple results should return a single match object, with all results as positional parts. It can be distinguished from a normal match object by evaluating it in slice context... which Rakudo doesn't support yet.

Now the subst method and thus s/// are implemented by calling .match(:global, ...), and without slice context, it can't distinguish between multiple matches, and a single match with subcaptures. And so my changes to the global match broke the substitution, and I see no easy way to fix it.

Anyway, here are a few examples of what works today:

$_ = 'ab12fg34';
s:g/\d/X/;
.say; # output: abXXfgXX


$_ = 'Hello, World';
# :ii is the same as :samecase
s:ii/world/perl/;
.say; # output: Hello, Perl

$_ = 'I did not know that that work together';
s:2nd/that/they/;
.say; # output: I did not know that they work together

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

Posted by david | Permalink

Posted by david | Permalink

At YAPC::EU 2010 we had a long discussion about Perl 6, Rakudo and related matters. Here are some (very incomplete) notes of the ongoing discussions and results.

Attendees

Patrick Michaud, Jonathan Worthington, Carl Mäsak, Moritz Lenz, Gabor Szabo, and a fluctuation of other Perl 6 hackers.

Speed

What can we do to improve Rakudo's performance?

jnthn's grant proposal for a low-level meta object protocol

See http://news.perlfoundation.org/2010/07/hague-grant-application-meta-m.html. Will probably bring the biggest speed improvement of all options we have under our control

Rakudo built-in optimizations

Most Rakudo built-ins are written for correctness first, and without a good feeling for what constructs are fast and what aren't. A thorough review (and preferably profiling) could bring decent speed improvements, as the case of int ranges showed.

Garbage collector

Parrot's GC is... suboptimal. To be gentle.

Optimization framework

We will try to convince people that Tyler Curtis' optimization framework for PAST and POST should be shipped with parrot (probably compile PIRs in ext/, just like NQP-rx does it now). Using that, we can do constant folding

Moving stuff to compile time

Number parsing needs to be moved to compile time.

What do we need to keep hacking?

Brought up by Gabor

Money

We do much volunteer work, but when we get funding, we can devote more time to hacking

Travel/Conferences

We'd like to get together a few times (2? 3? 4?) a year, in real life.

Funding and organization would be very welcome

Short-time funding

It would be nice to have a way to have funding available much more quickly than through the usual grant process, which tends to be longish.

Rakudo Star feedback

Good: It worked. It did what we wanted it to.

Bad:

• It lacked a module installer (It shipped proto, but didn't install it).

• Compilation takes too much memory. pmichaud will try a hack to split the setting, which would solve that problem.

• There was some discussion about the roles + outer scopes bugs, which was way over my head. It seems to be related to the fact that parrot has two outer chains for nested blocks: one at compile time, one at runtime. Since role methods are flattened into classes, there compile time outer block is actually different than where it runs, and that screws up ... forget it, somebody else must describe it.

• Lack of modules - doesn't seem to bee a big problem

• Lack of features: not a big problem.

  Biggest complaints: missing perl6doc. Missing non-blocking IO, binary file support.

• Prefix paths with spaces are not supported :(

  jnthn: "I actually tried to write a C program that binary patches the perl6 executable to allow spaces in path names. It almost worked."

• We will try to advocate compilation to PBC, not PIR - once that's supported.

Proto/Pls

Proto needs to be end-of-life'd.

It confuses people that there are two different project lists, and the lists diverge.

We would like to decentralize the module list somehow. Still open how.

People don't release Perl 6 modules, because there's no need so far, and it's tedious to add the version name in each .pm/.pm6 file. We might need to come up with a clever idea for that.

Backend diversity

Additionally to the parrot backend, we want to run Perl 6 code on other virtual machines.

jnthn will work on a .NET/CLR port. He wants to prototype the new low-level class composition code in .NET anyway, which will provide the basic foundations for running NQP.

pmichaud wants to explore javascript on V8 as a possible backend. "I managed PIR, I'll certainly manage javascript" :-)

• Huge time sink, but still worth doing it

• Apache runtime library might be worth looking into

• risks: stalled refactors are dangerous (see: PHP 6, cardinal (the ruby-on-parrot compiler))

  We want to avoid fragmentation into many subprojects

• We want to increase the number of possible contributors to rakudo by enabling non-parrot people to contribute.

• Code for different backends will be maintained as directories in Rakudo and NQP, not as branches.

• pir:: things will be hidden behind an nqp:: abstraction layer

Attracting contributors

Moritz wants to continue with the "weekly" challenges, but runs out of ideas. Add ideas to http://svn.pugscode.org/pugs/misc/helpnow/README.

We will try to apply patches faster, thus encouraging people who already did the first step.

Documentation

• in p5 pod for now, so that people can contribute easily

• masak and szabgab expressed interest in working on pod6 tools

Posted by Moritz Lenz | Permalink

Posted by david | Permalink

Posted by david | Permalink

Posted by david | Permalink

Posted by david | Permalink

NAME

"Perl 5 to 6" Lesson 28 - Currying

SYNOPSIS

  use v6;
  
  my &f := &substr.assuming('Hello, World');
  say f(0, 2);                # He
  say f(3, 2);                # lo
  say f(7);                   # World
  
  say <a b c>.map: * x 2;     # aabbcc
  say <a b c>.map: *.uc;      # ABC
  for ^10 {
      print <R G B>.[$_ % *]; # RGBRGBRGBR
  }

DESCRIPTION

Currying or partial application is the process of generating a function from another function or method by providing only some of the arguments. This is useful for saving typing, and when you want to pass a callback to another function.

Suppose you want a function that lets you extract substrings from "Hello, World" easily. The classical way of doing that is writing your own function:

  sub f(*@a) {
      substr('Hello, World', |@a)
  }

Currying with assuming

Perl 6 provides a method assuming on code objects, which applies the arguments passed to it to the invocant, and returns the partially applied function.

  my &f := &substr.assuming('Hello, World');

Now f(1, 2) is the same as substr('Hello, World', 1, 2).

assuming also works on operators, because operators are just subroutines with weird names. To get a subroutine that adds 2 to whatever number gets passed to it, you could write

  my &add_two := &infix:<+>.assuming(2);

But that's tedious to write, so there's another option.

Currying with the Whatever-Star

  my &add_two := * + 2;
  say add_two(4);         # 6

The asterisk, called Whatever, is a placeholder for an argument, so the whole expression returns a closure. Multiple Whatevers are allowed in a single expression, and create a closure that expects more arguments, by replacing each term * by a formal parameter. So * * 5 + * is equivalent to -> $a, $b { $a * 5 + $b }.

  my $c = * * 5 + *;
  say $c(10, 2);                # 52

Note that the second * is an infix operator, not a term, so it is not subject to Whatever-currying.

The process of lifting an expression with Whatever stars into a closure is driven by syntax, and done at compile time. This means that

  my $star = *;
  my $code = $star + 2

does not construct a closure, but instead dies with a message like

  Can't take numeric value for object of type Whatever

Whatever currying is more versatile than .assuming, because it allows to curry something else than the first argument very easily:

  say  ~(1, 3).map: 'hi' x *    # hi hihihi

This curries the second argument of the string repetition operator infix x, so it returns a closure that, when called with a numeric argument, produces the string hi as often as that argument specifies.

The invocant of a method call can also be Whatever star, so

  say <a b c>.map: *.uc;      # ABC

involves a closure that calls the uc method on its argument.

MOTIVATION

Perl 5 could be used for functional programming, which has been demonstrated in Mark Jason Dominus' book Higher Order Perl.

Perl 6 strives to make it even easier, and thus provides tools to make typical constructs in functional programming easily available. Currying and easy construction of closures is a key to functional programming, and makes it very easy to write transformation for your data, for example together with map or grep.

SEE ALSO

http://perlcabal.org/syn/S02.html#Built-In_Data_Types

http://hop.perl.plover.com/

http://en.wikipedia.org/wiki/Currying

Posted by Moritz Lenz | Permalink

This week I attended my first international Perl conference, the YAPC::EU 2010 in Pisa, Italy. I very much enjoyed it.

I arrived a few days earlier, and spent the time visiting Pisa, talking with old and new friends, and did some collaborative hacking. I especially enjoyed meeting people with whom I had had only contact via Internet so far, and found all of them to be very nice in meat space.

On Tuesday, the day before the conference started officially, a group of Perl 6 hackers met and discussed topics around Perl 6 and Rakudo. I recall talking with Patrick Michaud, Jonathan Worthington, Carl Mäsak, Paweł Murias, Gabor Szabo, smash (I can't spell his full name correctly from memory, sorry for that), and we have a very productive discussion (about 5 hours or so). Notes from the discussion will be published later.

On Wednesday the actual conference started, and there were plenty of very interesting talks, and very amusing lightning talks. I generally like the humor that is widespread in the Perl community.

On Thursday I continued to attend nice and informative talks, and also gave a talk on my own. It was about physical modelling with Perl 6, and in general the feedback was very positive, and somebody even commented that while he didn't understand everything I wrote, it reminded him that it was important to learn Perl 6 now. Win \o/. (There was also some criticism, but from somebody who apparently hasn't read the abstract; the "write-only" meme seems to apply to perl bloggers, not code). You can find the slides to my talk here.

Firday was the day of my departure too - sadly I had to leave after the first talk, and missed the rest of the day, including the closing keynote by mst (I did attend a talk of his) and the traditional auction.

I especially enjoyed...

• meeting Patrick, Larry, Gloria, Aaron, Gabor and many others for the first time in real-life
• a thorough, high-bandwith discussion with Larry, Jonathan, Patrick and Carl about p6 spec questions, and the Perl 6 discussions mentioned earlier
• a lightning talk imagine you're in a data center with no connection to the outside, and you accidentally executed chmod -x chmod. What would you do?
• good Italian Pizzas (although I had to wait 50 minutes for one of them)
• A talk by Tim Bunce, where he demonstrated database access in Perl 6 both with libraries that do native calls, and through the Blizkost project using the Perl 5 DBI/DBD::SQLite modules
• the general relaxed attitude in the Perl community, where people help each other, and don't seem to be easily offended (at least in meat space :)
• being surrounded by many other geeks
• realizing that several Perl hackers brought their family to the conference

It was an overwhelming experience, and I look forward to my next YAPC!

Posted by Moritz Lenz | Permalink