|« Dark||Adding complexity to reduce complexity »|
Thu, Mar 22, 2012
A truism regularly trotted out by perl users is that one of perl's biggest strengths is its ability to process text.
This was put to the test recently, following an issue at work which lead to us needing to get a lot of data out of our logs in a hurry.
Specifically, we needed to know at any given time how many web requests we were getting, what type (web page, XML request, etc), what they were for, who they were for, who they were from, etc. And we needed all this graphed so we could quickly & reliably cross-reference what was happening on one server with what was happening on others.
This had to come from a variety of log files, with anything up to 10 million lines in them - hence why we had to make the data visual & automate it: Just reading the logs simply wasn't an option.
This task landed on my desk, with the sum total of guidance on it being "I've found Chart::Gnuplot useful before"
So I had to do two things: First; turn the vast amount of logged data into a useful summary. Second; turn the summarised data into a selection of useful graphs.
The first part was pure text processing; the second was trying to quickly learn how to use a new module.
And I have to say, I was completely blown away with just how easy it was to get this task done. A couple of hours after I started, I had a useful graph. An hour or so more, and I had a multitude of useful graphs with a variety of breakdowns of the data, generated by a script that could take several helpful command-line options.
I've been using perl for two years now, on a daily basis at work & spending quite a lot of my free time on it as well - I'm a fair way through the Camel book, the Llama book, and HOP, and a reasonable way into OOP as well. I'm just about what The Definitive Guide to Catalyst would call an intermediate user (another book I'm trying to read in what little spare time I have). I still tend to feel like I'm barely even a beginner: There's so many things I'm aware I don't yet know.
After all, perl has been in active development by a multitude of very clever people all around the world for quarter of a century. It'd be surprising if it DIDN'T have a lot to learn by now.
But it's amazing just how much you can do with just the core fundamentals, quickly & easily. Perl was created for text processing, they say, but this is the first time I've really appreciated just how true it is. To be able to take millions of lines of text; use a single regex to break the useful data out of each line into discreet variables; use a foreach nested within a foreach to create a hash of hashes of hashes to store that data in; and then pass that hash to a subroutine that turns it into an array of objects with all the right names and descriptions in all the right places..
Sadly, hardly anyone I know would be capable of understanding why I was so pleased at how elegantly simple perl made it to handle the data I needed it to. But that's what blogs are for :)
And the documentation for the module I suddenly needed to learn, along with its included examples, made it incredibly easy to work out how to take the data I had and turn it into the pretty pictures I needed.
(Admittedly, it was a familiarity with sshfs made getting those images from a remote server & viewing it locally with gimp trivial. But it was a Perl conference where I first heard of sshfs, so..)
It's taken two years to stop feeling like a perl novice. I dread to think how long it'll take before I feel like a competent, advanced user. But I was genuinely amazed at how fast & easy it was for me to turn huge text files into useful sets of graphical data; using a module I'd only just discovered the existence of.
It can occasionally be a little depressing to look at how much you've learned, look at how much there still is to learn, and feel disheartened that the former is so much smaller than the latter. Especially when you notice that even after you've learned something, actually putting it into practice is so hard you're using less than half what you know, which is less than half of what there IS to know.
So it's nice when you have the other occasions, where what you know (and can put quickly into practice) is more than enough to put together something powerful, reliable, and sophisticated enough that it seems even to you to be a little bit magical.
It reminds you that although it might take you a few years to feel like you really know what you're doing, it took hundreds of people a few decades to put together what you're using.
So stick with it, because if something that's so clever you can't comprehend it today becomes something you understand completely tomorrow, just think how much smarter that'll mean you are by then.
|<< <||> >>|