Mechanical

Solving software problems

For the past year or so, I've been slowly changing my mind about how software is written. More specifically, I've begun to see how first class functions (FCFs) solve serious software problems and allow for a degree of flexibility that that few seem to appreciate¹. Part of this is due to the incredible Higher Order Perl by Mark Jason Dominus and part of this is a reaction to monolithic "solve every known problem known to man" modules available in Perl.

For those not familiar with FCFs, these are essentially functions which can be treated as data. I can assign them to variables, pass them as arguments to other functions and return them from functions. If you know C, you might think "function pointers!", but that's not the case. First class functions can be created dynamically at runtime. Further, it's generally important that they can capture the lexical scope of their surrounding environment. As a trivial example, consider the "add_to" function:

  sub add_to {
      my $number = shift;
      return sub { $number + shift };
  }

  my $seven_plus = add_to(7);
  my $four_plus  = add_to(4);
  print $seven_plus->(3); # prints "10"
  print $four_plus->(2);  # prints "6"

That example does not even begin to show how powerful these are, but it does show how they work.

FCFs are extremely powerful and when wielded by a competent programmer, can allow you to create software which can solve extremely tricky problems with a minimum of code. This has led me to create an approach to solving the "monolithic module" problem.

In a nutshell, there are some CPAN modules out there which are huge. They tackle a particular problem space and provide all of the tools necessary to handle the issues involved. This winds up with the "Microsoft Office" problem. The code is bloated and people complain that they're only using a small percentage of what's actually provided. However, as Joel Spolsky points out with Office, it may be bloated, but everyone is using a different 10 percent.

Though FCFs can't universally solve this problem, for much code, they can. I've used them a lot for specialty code, but my Class::BuildMethods module was my first attempt to utilize this technique for general-purpose code. The &validate method allowed folks to provide any validation routine they wanted for their code. The result was extremely powerful, resulted in a very small module which didn't rely on the object's implementation and, unfortunately, was not widely used. The feedback I got showed that many programmers just don't understand the power of first class functions. Further, some other minor limitations in my code cause problems in odd edge cases, but it's still a great module.

My next module which exploits this is going to be Class::CGI. I'm about to add a new method which allows people to pass arbitrary "stuff" to "handlers" which allow one to treat HTML form data as object collections. The passed "stuff" can include FCFs. The result is that it gives folks all of the functionality available in competing modules, but in a far smaller codebase. Once I saw how to implement the code, I realized that I'm providing the ten percent everyone wants and they can provide the custom ninety percent they need.

Unfortunately, it seems once again that some folks have not seen the merit of this approach and want to stick with their "old ways" of doing things. Some of the feedback I've gotten (both public and private) has been along the lines of "but X already does that". So I provide a more flexible solution in a smaller set of code and, to top it off, the basics are far easier to learn! It's the advanced stuff that confuses folks.

I'm beginning to see that the "do everything" modules, while bloated and more difficult to learn, serve newer programmers better because if they can't figure out how to do it, they can spend half an hour reading the docs and figure out a solution which is close enough to what they need. Telling them to "use a callback" (an FSF passed to code which the code can then invoke), while far more flexible and allowing them to do exactly what they need, requires that they have a better understanding of programming. So paradoxically, while I'm creating better code which is easier to use in the general case, it might be harder to use when you need to fine tune it. It's not because the fine-tuning is difficult (it's really simple), but because programmers aren't used to thinking of functions as data.

I really wish I knew how to get around this problem, but I'm stumped.


1. The problem is so ubiquitous that many languages do not support FCFs. As a result, a recent Dr. Dobbs article about Functional Programming in Java tried to simulate the the techniques, but they used full classes instead of a simple anonymous function and the result is very clumsy. That's not the author's fault, though. Java just tends to be clumsy.

Python also boasts FCFs, but it's crippled lambda implementation (statements only, no expressions) and Guido's refusal to fix it means that full-blown functional programming techniques are not available. I wonder if Guido just doesn't understand what's going on or if Python's whitespace issues make the problem insoluble in that language.

Tags: ,
Watch yourself...
(Anonymous)
... or you'll find yourself learning Lisp soon :-)

As to the problem of getting FCFs accepted - I'm just hoping that the passing of time will cure it. So far in my career I've seen garbage collection, virtual machines, object orientation go from "only weirdo academics / rocket scientists will use this" to mainstream.

Hell back in the late eighties I knew lots of people in microcomputer land who thought that you'd never use anything but assembler for serious programming :-)

I do think a lot of it has to do with language design. Look at Ruby - where passing functions very natural because of the way the language is designed.

(and as for Python - I think that Guido does understand it. It's just that all the solutions break what he considers the "Python-ness" of the language.)
The python 3000 slides from last week suggested that guido's willing to cave on lambda
Huh, I think I'm going to start using that technique my self... it will at least give me yet another method to names things correctly.
Thanks
Oh and on the but because programmers aren't used to thinking of functions as data. part, I prefer to think of it as a "you are what you eat". So in the case of programming a function returns data, so it is data.
I am so glad I read this post some months ago. It's basically given me the answer to a (fairly simple) problem I had to solve today in a nice and elegant manner.

However, I couldn't remember where I'd read it. Searching google for '"first class" functions perl' brought this post up in the top ten. :)