Perl text document stats script

Crikey, that was a little easy, in less than a fortnight I've done a cover version of my Python script from here, now in Perl, all this from not knowing any Perl at all.

I didn't do it on my own mind, I had a textbook, Learning Perl from O'Reilly, and tutoring from Robbie and Matt off of Facebook, without their help, I'd be floored.

The program in perl that analyzes the text file of my 2001 novel Shag Times and comes up with various statistics for it.

Its a bit messy compared to the Python version, and its clunky too, there's gotta be simpler and more leet ways of doing many of the sections.

#!/usr/bin/perl

# Program to
#  open shagtimes
#  provide a word count              
#  count unique words                
#  provide top ten most popular words
#  provide all single occurance words
#  calculate average word length     
#  find longest word                 

use Text::Wrap;

sub hashValueDescendingNum {
  $occurance_list{$b} <=> $occurance_list{$a};
}

# Opening Shag Times and processing it a bit
$filename = "shagtimes.txt";
open BOOK, "<", $filename or die "Can't open '$filename': $!"; my @book = ;
close BOOK;
foreach $book (@book) {
 $allbook .= $book;
 $allbook .= " "};
$allbook = lc"$allbook";
# $allbook is a long string of Shag Times

# Code to remove full stops and commas and stuff
$regexp = '[\W]' ;
@book = split /\s+/, $allbook;
foreach $book (@book) {
 $book =~ s/$regexp//g };
print "\n=======================================\n";

# Doing the word count
foreach $book (@book) {
   $wordcount += 1 };
print "The document contains $wordcount words in total\n";
print "=======================================\n";

# Doing the unique count
foreach $book (@book) {
 $was_this_in_uniques = "no";
 foreach $uniques ( @uniques ) {
   if ($book eq $uniques){
     $was_this_in_uniques = "yes"}};
 if ($was_this_in_uniques eq "no") {
   push @uniques, $book}};
foreach $uniques (@uniques) {
   $uniquecount += 1 };
print "The document contains $uniquecount unique words\n";
print "=======================================\n";

# Finding top ten popular words
print "The top ten most used words:-\n";
foreach $uniques ( @uniques ) {
 $was_this_in_book = "no";
 foreach $book ( @book ) {
   if ($uniques eq $book) {
     $occurances += 1}};
 $occurance_list{$uniques} = $occurances;
 $occurances = 0 };
foreach $key (sort hashValueDescendingNum (keys(%occurance_list))) {
  push @ranked_occurances, "$key \($occurance_list{$key}\)"};
foreach (0..9) {
 print "$ranked_occurances[$_]\n"};
print "=======================================\n";

# Finding single use words
print "Words that were used only once:-\n";
foreach $ranked_occurances ( @ranked_occurances ) {
 $_ = $ranked_occurances;
 if (/\(1\)/) {
   s/\s\(1\)//;
   push @singles, ("$_");
   $single_use_count += 1}};
@singles = sort @singles;
foreach $singles ( @singles ) {
 $single_paragraph .= "$singles, "};
print wrap("", "", "$single_paragraph\n");
print "=======================================\n";
print "A total of $single_use_count words were used only once\n";
print "=======================================\n";

# Finding average word length
foreach $book (@book) {
   $chartotal += length($book) };
$avechar = $chartotal/$wordcount;
my $printy_avechar = sprintf "%.3f", $avechar;
print "The average word length was $printy_avechar letters long\n";
print "=======================================\n";

# Finding longest word
foreach $uniques ( @uniques ) {
 if (length($uniques) > $longlength) {
   $longlength = length($uniques)}};
print "The longest word was $longlength letters long\n";
print "These words were that long:-\n";
foreach $uniques ( @uniques ) {
 if (length($uniques) == $longlength) {
   print "$uniques\n"}};
print "=======================================\n";

Now I need to order that Ruby book, Ruby on the Rails for Dummies perhaps?

Python text document stats script

Desperately I spend the last three weeks refreshing my knowledge of Python, the easy peasy scripting language I'd once used in my youth to test CD players and hi-fi RS232 commands. I re-acquired Dive Into Python and Python For Dummies and ploughed through.

Sadly I didn't get the job I was learning it for, but instead refired my enthusiasm for learning programming languages. So, I've acquired Learning Perl and when that is done I'll be trying my hand at Ruby, just like the cool kids use.

I wrote me a wee program in Python that analyzes the text file of my 2001 novel Shag Times and comes up with various statistics for it.

I plan to re-write the same program in Perl as soon as I've finished reading Learning Perl, and then write the same program again in Ruby if that's even possible.

### Program to do the following
###  * open shagtimes.txt
###  * provide a word count
###  * count unique words        
###  * provide top ten most popular words
###  * provide all single occurance words
###  * calculate average word length
###  * find longest word         

import textwrap

# Opening Shag Times and processing it a bit
book = open('shagtimes.txt')
book = book.read()
book = book.lower()
### Code to remove punctuation
stuff_to_replace_with_space = (".", ",", "?", "/", "=", "-", ";", ":")
stuff_to_remove = ("\'", "(", ")", "\"")
for item in stuff_to_replace_with_space:
book = book.replace(item, " ")
for item in stuff_to_remove:
book = book.replace(item, "")
print "======================================="

# Doing the word count
book = book.split()
wordcount = len(book)
print "The document contains %i words in total" % wordcount
print "======================================="

# Doing the unique count
uniques = []
for item in book:
if item not in uniques:
uniques.append(item)
uniquecount = len(uniques)
print "The document contains %i unique words" % uniquecount
print "======================================="
uniques.sort()

# Finding top ten popular words
print "The top ten most used words:-"
occurancelist = []
for item in uniques:
occurances = book.count(item)
occurancelist.append((occurances, item))
occurancelist.sort()
occurancelist.reverse()
for item in occurancelist[:10]:
print item
print "======================================="

# Finding single use words
print "Words that were used only once:-"
singleuse = []
singleusecount = 0
for item in occurancelist:
c, w = item
if c == 1:
singleuse.append(w)
singleusecount+=1
singles = ""
while singleuse:
for item in singleuse:
singles = singles + (singleuse.pop()) + ", "
singles = textwrap.wrap(singles, width=70)
for i in singles:
print i
print "======================================="
print "A total of %i words were used only once" % singleusecount
print "======================================="

# Finding average word length
chartotal = 0.000
for item in book:
chartotal = chartotal + len(item)
avechar = chartotal/wordcount
print "The average word length was %.3f letters long" % avechar
print "======================================="

# Finding longest word
longlength = 0
for item in uniques:
if len(item) > longlength:
longlength = len(item)
print "The longest word was %i letters long" % longlength
print "These words were that long:-"
for item in uniques:
if len(item) == longlength:
print item
print "======================================="

Can someone recommend me a good book for learning to program in Ruby?

Olde Times

So last night on Bowlie we were chatting about things and stuff and I posted a link to that Carter USM cover, and I was suddenly overcome with how thinly spread I am, how many pies I have my fingers in. Not large pies mind, just individual portions. There's the blogging, the music, the drawings, the manufacturing, the politicing, the commenting. Its almost too much.

But not quite.

Latest delivery from Lulu came today, five more copies of the post-it notes book so I can start giving them away to anyone who wants one.

And also this.

The new hardback edition of my novel Shag Times.

Its a work of fiction what I wrote in 2001 / 2002 about Craig Illman, a student in Glasgow, who on reading an interview with comedian Ed Byrne saying he used to be a student in Glasgow and shagged hundreds of women, Craig decides to try to shag a hundred women in twelve months. Of course he fails miserably, but its kind of like a bildungsroman thing, and there's Glasgow's millennial live music scene and university and stuff.

Of course, Shag Times was originally an album by the KLF in 1988. But its also a good name for a book.

TheESSEX

Where is Wickford?
Where is Shenfield?

What do these places mean? What should they mean to me?

There's engineering work on the trainlines so I have to get a replacement bus on my way east to visit the niece, my brother and sister-in-law. Its a warm day, you are my indian summer, maybe the last of the year, the citizens of Wickford and Shenfield are out with their friend nd loved ones, drinking and holding. I am on buses and trains on my own, scribbling.

It didn't have to be this way.

Months ago I bought a new mobile phone, a Sony Ericsson K800i, with the intention of being able to send emails and update my various blogs whilst out and about

Those bastards at CarPhone Warehouse sold me a pudding. Battery life is down to 8 hours now, thats with using it for one phone call and two text messages a week, the memory is too small to surf the web and email is impossible to set up. The wee cover of the camera is badly designed, it keeps sliding open in my pocket and taking photies of my shrapnel. How can something be designed and sold so bad?

Those bastards.

Also my own fault for not taking it back to the shop.

CarPhone Warehouse = bunch of cunts

Sony Ericsson K800i = shittest phone ever

...

A girl sits next to me on the bus, lack of empty seats. She's tall, in her early 20s, blonde hair and kind of stocky. I think I'm in love with her.

Pale skin, small delicate hands, she keeps adjusting her top so it covers the way her tummy bulges over the sides of her jeans.

My mind keeps drifting to wrapping my arms round her, her body soft under my fingers, kissing her neck.

I wonder what she smells like.

Crikey I need a girlfriend.

...

Back to reading this book, The17. Bill, the narrator, has just moved to east London in it.

At exactly the same I was driving to south London from Glasgow.

What went wrong?

On dating

Recently a friend suggested I try the Guardian's Soulmates webiste, I knocked back the idea outright. Its not for me.

And here's why.

I don't struggle at meeting new people. In the course of my extra-curricular activities, both online and in real life, I'm surrounded. Thousands of people check out my websites, hundreds of people read my posts on message boards, dozens of people go to the same gigs as me. At any point I can turn the girl next to me and say "Hey, how's it going, what did you think of that last one.".

I'll accept that starting off the conversation terrifies me, but after that, its plain sailing, I can be rounded and charming enough to carry it off.

So, what is the problem?

Even when out with friends, I sit at the side, or stand at the back keeping quiet, gazing into the distance, or into my pint, sometimes my mind travels in time. It takes a cattle prod to get me communicating.

I'm not sure how Soulmates will help there.

Since I discovered my cock some time in the mid-nineties, I've have about six relationships that lasted over four months, none more than seven. It worries me, there's something very wrong. I mention it to friends on MSN, my worries. And then every so often in response to me saying something outragous, the friends on MSN will say something like "I can see why it only lasts six months".

I find it patronising and condescending, and it makes me think they're fools. They know nothing of context. I take it too seriously.

Yesterday, I was at Dr Sketchy's, the burlesque life-drawing thing. Keeping quiet at the back with my post-it pad and biro, between drawing tasks, I scrawled a wee list of those lucky lucky 4-7 month relationships and how they ended. And drew a pie chart too. The reasons they ended are diverse, only two reasons non-unique. The most popular reason for splitting up, with 2.5 hits, is 'moved to London'. This shouldn't reoccur, I've implemented corrective action. The second non-unique reason, is my refusal to come back after being dumped and then begged back.

Me saying something outrageous, has never had any bearing on the end of a relationship.

Looking at the list, the other terrifying thing is what could be called the Good Luck Chuck factor. More than half the girlfriends got engaged shortly after we broke up. Never seen the film, but I understand to break the curse I have to hump Jessica Alba. Alas she never returns my calls.

What to do?

Fuck knows.

If I don't shut anything down, or close off any of my options, the possibilities for the future are limitless.

So carry on reviewing gigs, carry on going to geeky internet meetups, carry on drawing pictures, carry on as usual with brief psychotic episodes of frustration, jealousy and anger.

Then again, the friend was also signing up to Guardian's Soulmates herself, so it could have been a round about way of asking me out. But I very much doubt it.

Those were the times

Ooh, did I mention that Fiona brought out a copy of Shag Times the other night. An actual genuine 2002 copy. I was mortified, but it was kind of neat. She had a letter I'd sent her with it saying not to leave it on a shelf collecting dust for the next five years, which I think is exactly what she did. Well, she and a succession of flatmates had read it and Dom made a start.

Is that what happened to the past five years?

Wandering round Shepherds Bush last night, I was going to meet flatmate Nick but he cancelled so I got off the underground in the 'Bush and txted Ralf to no avail, so it was just me bumbling around, fending drunken txts from 'er up north.

Gig tonight at Tufnell's again, different bands, its worth a shot. I was thinking about videoing bits of it and doing talking segments reviewing it, as well as just the usual writing thing.

Ooh, where to I take the missus out over the weekend to show her the cool places and clubs of this fair city? Any indie clubnights or just stay home roasting vegetables.