Showing posts with label bash. Show all posts
Showing posts with label bash. Show all posts

Monday, 28 January 2013

Spell checker and corrector in BASH

Why? Because it can be done.


I have used nothing but BASH.

Sample run:




I tried to make it very feature rich while keeping code to about 100 lines - this version is about 80.

Can you make it smaller?

Features:
* Auto-corrects if it can only find 1 suggested correction.
* Lists all words in dictionary that could be corrections and lets user choose or replace word.
* Suggestions only consider words with single letter deletions, additions or changes.
* Spelling errors are highlighted in yellow.
* Corrections are highlighted in green.

Todo:
* Save result to a file (Currently just displays result on terminal).

I did go a little overboard with processing word case.


#!/bin/bash

[[ -z "$1" ]] && echo "Usage: $0 " && exit 1
ERR=33; COR=32                                   # error and corrected colours
PS3="Select or manually enter a correction: "
L="a-zA-Z'"                                      # these letters are deemed part of words
FINAL=
shopt -s extglob
shopt -u nocasematch
declare -A WORD
while read W; do WORD[${W,,}]=${W,,}; done < /usr/share/dict/words

function checkSuggestion() {
    local p=$1 S=$1 C                            # uses CASE and SUGGEST from caller
    [ -z "${WORD[$p]}" ] && return 1             # matches word in dict!
    [ $CASE = PROP  ] && S=${p^}                 # change case to original word
    [ $CASE = CAP   ] && S=${p^^}   
    [ $CASE = CAPs  ] && { S=${p%%\'*};             S="${S^^}'${p#*\'}"; }   
    [ $CASE = LAT   ] && { S=${p##*\'};             S="${p%\'*}'${S^^}"; }   
    [ $CASE = OPROP ] && { S=${p%%\'*}; C=${p#*\'}; S="${S^^}'${C^}";    }   
    SUGGEST="${SUGGEST/ $S /} $S "               # add to suggestion list
}

function getSuggestions() {
    local W=$1 t l # p
    SUGGEST=; CASE=LOWER
    [[ $W =~ ^[A-Z][a-z]        ]] && CASE=PROP   # Prop
    [[ $W =~ ^[A-Z][A-Z]        ]] && CASE=CAP    # CAP
    [[ $W =~ ^[A-Z]\'[a-z]      ]] && CASE=PROP   # P'rop
    [[ $W =~  [A-Z]\'[a-z]$     ]] && CASE=CAPs   # CAP's
    [[ $W =~ ^[a-z]\'[A-Z]      ]] && CASE=LAT    # l'At
    [[ $W =~ ^[A-Z]\'[A-Z][A-Z] ]] && CASE=CAP    # C'AP
    [[ $W =~ ^[A-Z]\'[A-Z][a-z] ]] && CASE=OPROP  # O'Prop
    W=${W,,}                                     # lowercase word
    for (( t=0 ; t<=${#W} ; t++ )); do           # for each letter position of word, delete|change|insert a letter
        checkSuggestion ${W:0:t}${W:t+1}         # try deleting letter at position t
        for l in {a..z} "'"; do                  # try changing and inserting letters
            checkSuggestion ${W:0:t}$l${W:t+1}   # try changing letter
            checkSuggestion ${W:0:t}$l${W:t}     # try inserting letter
        done
    done
}

function correctWord() {                         # correct word and return in RET as well as pattern in PAT
    local W=$1 D1 D2 SUGGEST                     # uses L RESULT
        getSuggestions "$W"
        [[ ! "$RESULT" =~ ([^$L]|^)$W([^$L]|$) ]] && { printf "ERROR: Could not find [%s] in %s\n" "$W" "$LINE"; exit 1; }
        D1="${BASH_REMATCH[1]}"; D2="${BASH_REMATCH[2]}"  # get delimiters around word
        PAT="${D1:-#}$W${D2:-%}"                 # match pattern
        case $SUGGEST in
             '') echo -e "${RESULT/$PAT/$D1\e[${ERR}m$W\e[0m$D2}"   # display line with spelling mistake highlighted
                 read -p "Enter correction [$W]: " RET < /dev/fd/3  # no suggestions - let user correct
                 RET=${RET:-$W};;
            +([$L ])\ +([$L ]))                                     # multiple suggestions
                 echo -e "${RESULT/$PAT/$D1\e[${ERR}m$W\e[0m$D2}"   # display line with spelling mistake highlighted
                 select RET in "(IGNORE)" $SUGGEST; do              # get correction
                     RET=${RET:-$REPLY}                             # user entered a word instead
                     [ "$REPLY" = "1" ] && RET=$W                   # no change
                     break
                 done < /dev/fd/3;;
              *) RET=${SUGGEST// /}                                 # get single suggestion
                 printf "%b\n" "\e[31mAuto:\e[0m ${RESULT/$PAT/$D1\e[9;${ERR}m$W\e[0;${COR}m $RET\e[0m$D2}"
        esac
        COL=$COR; [ "$RET" = "$W" ] && COL=$ERR   # if unchanged, highlight in error colour
        RESULT="${RESULT/$PAT/$D1\e[${COL}m${RET}\e[0m$D2}"       # correct word
}

function correctLine(){
    local LINE="$1" W                            # uses RESULT
    while read -d' ' W; do                       # for each word in line
        [[ -z "$W" || -n ${WORD[${W,,}]} ]] 2> /dev/null && continue  # null or word in dict so ignore it
        correctWord $W
        printf ">> %b\n\n" "$RESULT" 
    done <<< "$LINE"
}

while read LINE; do  # for each line
    RESULT="$LINE"
    correctLine " ${LINE//[^$L ]/ } "            # replace punctuation with spaces
    FINAL+="$RESULT\n"                           # append corrected line to result
done 3<&0 < $1                                   # redirect stdin to /dev/fd/3 for select and read

printf "\nCORRECTED TEXT\n\n%b\n" "$FINAL"

Friday, 18 January 2013

How to Import Birthdays and Anniversaries into Google Calendar


This is a quick bash script I made that reads a file of events and creates an ical file suitable for importing into Google Calendar.

Date file supports these event formats:

day/month event description
day/month/year event description

Brief Instructions

1. Cut and paste program into a text file.

2. Save file as pc-ssv-to-ical.bash (or whatever you like)

3. In a terminal, chmod +x pc-ssv-to-ical.bash

4. Make an event file (eg. special dates) lile this:
1/1 New Year
26/1 Australia Day
4/4/1914 Someon'es Birthday
5/5/1915 Bill/Jill Wedding (1915)

5. Run ./pc-ssv-to-ical.bash special.dates > cal-import.ics 
6. Import cal-import.ics into a Google Calendar

Script

#!/bin/bash


#    This script takes events from a file and outputs in ical format 
#    suitable for importing into a Google Calendar.
#
#    Copyright (C) 2013  phil colbourn
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    A copy of the GNU General Public License can be obtained from
#    .

SSV=$1

shopt -s extglob

# load special dates as day/month[/year] event description
# eg. 1/5/2004 A Person of Interest
# eg. 24/5 Not Sure When Born


# use this to convert SSV to CVS
# cat special-dates.ssv | sed 's/ /,/' > special-dates.csv


printf "BEGIN:VCALENDAR\nVERSION:2.0\nCALSCALE:GREGORIAN\n"


IFS+=","


while read DATE EVENT; do
    [[ -z $DATE ]] && continue               # ignore null dates
    # D/M[/Y] format
    D="0${DATE%%/*}"                         # get day part
    D="${D: -2}"   
    M="${DATE#*/}"                           # get month part
    M="0${M%%/*}"
    M="${M: -2}"
    # M/D[/Y] format
    #M="0${DATE%%/*}"                         # get month part
    #M="${M: -2}"   
    #D="${DATE#*/}"                           # get day part
    #D="0${D%%/*}"
    #D="${D: -2}"
    Y="${DATE/[0-9]?([0-9])[\/][0-9]?([0-9])/}"  # get year part
    Y=${Y//[\/]/}                            # remove left over /
    [[ -z $Y ]] && Y=$( date +%Y )           # use this year if none given

cat <
BEGIN:VEVENT
DTSTART;TZID=Australia/Sydney:${Y}${M}${D}
DTEND;TZID=Australia/Sydney:${Y}${M}${D}
RRULE:FREQ=YEARLY
DESCRIPTION:${EVENT}${Y:+ ($Y)}
SUMMARY:${EVENT}${Y:+ ($Y)}
END:VEVENT
EOF

done < $SSV

printf "END:VCALENDAR\n"

unset IFS

Notes

1. Code could be simplified if a more strict date format was used. eg. DD/MM[/YYYY]
2. Code will need to be changed for M/D[/Y] dates.
3. Space separated (SSV) and CSV formats both work. Quoted and double-quoted events also seem to work.

Thursday, 26 August 2010

How to remove commas from quoted strings in csv files

I needed to remove commas (,) from within double quotes (") in a Comma Separated Variable (CSV) file.

For example, you can use cut -d, -f2,4,5 to extract fields 2, 4 and 5.

But if there is a comma in the text of a field like this "hello, world" you are generally stuck.

Also SQLite can also import csv files, but again commas in quotes cause problems.

I was stuck until I managed to create this sed script that will work in many cases.


# This bash function uses sed to remove up to 4 individual commas
# sequences from a quoted string in a csv file.


# eg. "Hello,, World, nice day." -> "Hello World nice day."



function removeCommas(){
  while read data; do
    echo "$data" | sed -e 's/^/,/g' | sed -e "s/$/,/g" \
      | sed -e 's/\(,\"[^,]*\),*\([^,]*\),*\([^,]*\),*\([^,]*\),*\(.*\",\)/\1\2\3\4\5/g' \
      | sed -e 's/^,//g' | sed -e 's/,$//g'
  done
}


A friend mentioned that the sed commands can be combined as follows:


    echo "$data" \
      | sed \
      -e 's/^/,/g' \
      -e "s/$/,/g" \
      -e 's/\(,\"[^,]*\),*\([^,]*\),*\([^,]*\),*\([^,]*\),*\(.*\",\)/\1\2\3\4\5/g' \
      -e 's/^,//g' \

      -e 's/,$//g'


How it works


The engine uses sed.


This sed statement takes input from stdin and replaces the regular expression 'from' with 'to' for any and all occurrences of 'from'.


sed -e 's/from/to/g'

First, I add a comma to the start and end of each line with this:
sed -e 's/^/,/g' | sed -e "s/$/,/g"

These make the function work for special cases and they get removed after the commas have been removed.

The 'from' string starts of by finding the beginning of a quoted string ',\"' then all non-comma characters '[^,]*'.

This pattern is enclosed in brackets '\(' and '\)' to assign the matching pattern to, in this case, part 1.

Then it matches the next one or more commas ',*'. This is not in brackets since we don't want them.

The next part of the pattern '\([^,]*\)' is like the first: it finds a string of non-comma characters and keeps the values as part 2.

Then we skip over any commas again.

This sequence can be repeated as many times as you like. I did it 3 times.

The pattern ends like it starts. But this time it reads any character including commas up to the trailing quote using '.*'. Then it reads the trailing quote '\"' and comma ','. All of this makes part 5.

This means that it will only filter the first 4 sequences of commas. To make it do more, repeat the middle pattern ',*\([^,]*\)' and increase the output parts (below).

The 'to' expression is simply the concatenation of the 5 parts of the quoted string that are guaranteed to not contain a comma (well, except possibly for the last part), where '\N' is the nth part.

\1\2\3\4\5

Testing

I tested the function with this.


function testthis(){
  echo -n "\"$1\" --> \"$2\"..."
  echo "$1" | removeCommas | grep -qE "$2" && echo "Pass" \
    || echo "FAIL"
}


# test removeCommas
testthis "this test should fail to test the 'tester'" "anything but this"
testthis "" ""
testthis "," ","
testthis ",," ",,"
testthis ",\"\"," ",\"\","
testthis ",\",\"," ",\"\","
testthis "1,2,\"3,4 5 6 7\",8,9,10" "1,2,\"34 5 6 7\",8,9,10"
testthis "1,2,\"3,4,5 6 7\",8,9,10" "1,2,\"345 6 7\",8,9,10"
testthis "1,2,\"3,4,5,6 7\",8,9,10" "1,2,\"3456 7\",8,9,10"
testthis "1,2,\"3,4,5,6,7\",8,9,10" "1,2,\"34567\",8,9,10"
testthis "1,2,\"3,,4,,5,,6,,7\",8,9,10" "1,2,\"34567\",8,9,10"
testthis "1,2,\"3,,4,,5,,6,,7,8\",8,9,10" "1,2,\"34567,8\",8,9,10"
testthis "\"tricky one where the quoted string starts the line 3,,4,,5,,6,,7,8\",8,9,10" "\"tricky one where the quoted string starts the line 34567,8\",8,9,10
testthis "1,2,\"3,,4,,5,,6,,7,8 tricky one where the quoted string ends the line \"" "1,2,\"34567,8 tricky one where the quoted string ends the line \""

Saturday, 17 July 2010

Firewall Rule Testing with BASH and TCPTraceRoute

I wanted to block the use of any DNS server except those that I select (google, openDNS and my router).

I also wanted to make sure that these DNS servers work and that others do not.

So I made a BASH script to verify my firewall rules.
#!/bin/bash
# only particular DNS servicer are allowed to be contacted. 
# this tests that this is so
LOCAL="the.IP.address.of.your.router"
ALLOW="8.8.8.8 8.8.4.4 208.67.220.220 208.67.222.222"
BLOCK="220.233.0.4 61.88.88.88 202.139.83.3 61.9.194.49 61.9.195.193 61.9.133.193 61.9.134.49 203.161.158.2"
echo "Testing DNS servers via UDP that are allowed to work..."
for d in $LOCAL $ALLOW; do
  dig @$d somename +time=1 +tries=1 +notcp > /dev/null
  [ $? -ne 0 ] && echo "FAIL: Failed to get response from $d via UDP." && exit 1
  echo "PASS: $d responded via UDP."
done

echo "Testing DNS servers via TCP that are allowed to work..."
for d in $LOCAL $ALLOW; do
  dig @$d somename +time=1 +tries=1 +tcp > /dev/null
  [ $? -ne 0 ] && echo "FAIL: Failed to get response from $d via TCP." && exit 1
  echo "PASS: $d responded via TCP."
done

echo "Testing DNS servers via UDP that are NOT allowed to work..."
for d in $BLOCK; do 
  dig @$d somename +time=1 +tries=1 +notcp > /dev/null 
  [ $? -eq 0 ] && echo "FAIL: Got response from blocked DNS server $d via UDP." && exit 1 
  echo "PASS: DNS server $d via UDP was correctly blocked." 
done 
echo "Testing DNS servers via TCP that are NOT allowed to work..." 
for d in $BLOCK; do 
  dig @$d somename +time=1 +tries=1 +tcp > /dev/null 
  [ $? -eq 0 ] && echo "FAIL: Got response from blocked DNS server $d via TCP." && exit 1 
  echo "PASS: DNS server $d via TCP was correctly blocked." 
done 
echo "" 
echo "Firewall PASSed."
Then I thought that I might be able to test other TCP blocking rules by setting the IP packet's time-to-live (TTL) to a small number and looking for ICMP time expired packets. To do this I needed to use tcptraceroute to get the core functionality. On the Mac I got this from fink.

If I get some response, the firewall is NOT blocking an outgoing port. If I get stars (* * *) then it probably is blocking the port.

#!/bin/bash
echo "Testing blocked outgoing port..."
# set TTL to 2 hops: host to ADSL router, ADSL router to ISP gateway
# if the ISP responds to TCP TTL timeouts then a blocked port should get '2  *'
# whereas an open outgoing port should get something more complicated like this:
# '2  37.1.233.220.static.exetel.com.au (220.233.1.37)  23.176 ms'
#sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 www.google.com 79 | grep '2  *' && echo "blocked"
BLOCK="135 136 137 138 139 445 593 1863 110 9000 5190 23 1503 1720 53"
VICTIM="www.some.real.site.com"
# The victim should not get any packets if the firewall rules are right.
for p in $BLOCK; do
  sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 $VICTIM $p | grep '2  \*'
  [ $? -ne 0 ] && echo "FAIL: Port $p is open for outgoing traffic." && exit 1
  echo "PASS: Port $p is blocked for outgoing traffic."
  echo ""
done

ALLOW="80 8080 443 25 21 119 22 123"
VICTIM="www.some.real.site.com"
# again, the victim should not get any packets since TTL is so small.
for p in $ALLOW; do
  sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 $VICTIM $p | grep '2  \*'
  [ $? -eq 0 ] && echo "FAIL: Port $p is blocked for outgoing traffic." && exit 1
  echo "PASS: Port $p is open for outgoing traffic."
  echo ""
done
echo "" 
echo "Firewall PASSed."


You will need to modify the script to suit your firewall rules.

Thursday, 11 March 2010

Directory to XML BASH Script

Ever wanted to get a directory into an XML structure?

Here is a quick, short and easily modifiable BASH script that works well.

To see how it works, you could run it like this:

scriptname dir | xmllint --format - | less

for dir, you can use . .. ./ ../ or / as well as subdirectories and full directory paths.

You might also be interested in this project: xml-dir-listing

How it works

The script first makes special directory specifiers easily useable. It then calls doDir with the directory.

doDir uses ls to get all files in the current directory. If the file is actually a directory and not a sym-link, it calls itself to process the subdirectory. Otherwise it outputs the file name.

Certain special directories (. and ..) are ignored to avoid infinite loops.

The program can use a lot of stack space so I increase it - I just guessed a value. It also can take a long time, so I renice the process so you can do other things.

To stop it, you may need to enter a lot (10-20) of ctrl-c's. I'm not sure why.

Sample Output


<dir>
<dirname><![CDATA[/usr/share/doc/distcc/example]]></dirname>
<file><![CDATA[init]]></file>
<file><![CDATA[init-suse]]></file>
<file><![CDATA[logrotate]]></file>
<file><![CDATA[xinetd]]></file>
</dir>
<file><![CDATA[protocol-1.txt]]></file>
<file><![CDATA[protocol-2.txt]]></file>
<file><![CDATA[reporting-bugs.txt]]></file>
<file><![CDATA[status-1.txt]]></file>
<file><![CDATA[survey.txt]]></file>
</dir>
<dir>
<dirname><![CDATA[/usr/share/doc/groff]]></dirname>
</dir>
<dir>
<dirname><![CDATA[/usr/share/emacs]]></dirname>
<dir>
<dirname><![CDATA[/usr/share/emacs/22.1]]></dirname>
<dir>
<dirname><![CDATA[/usr/share/emacs/22.1/etc]]></dirname>
</dir>
<dir>
<dirname><![CDATA[/usr/share/emacs/site-lisp]]></dirname>
</dir>
<dir>
<dirname><![CDATA[/usr/share/enscript]]></dirname>
<file><![CDATA[88591.enc]]></file>
<file><![CDATA[885910.enc]]></file>
<file><![CDATA[88592.enc]]></file>
</dir>



The Script
#!/bin/bash


# WARNING: To break this, you need to enter a lot of ctrl-c's


# heavy recursion so allow a bigger stack
ulimit -s 32768


# run with low priority so you can do other stuff while it works
renice -n +19 -p $$


function doDir {
  # directory name may contain illegal XML characters so we won't use attributes 
  #echo "<dir name=\"${1}\">"
  echo "<dir>"
  echo "<dirname><![CDATA[${1}]]></dirname>"
  # get all files and directories
  ls -Ab1 "$1/" | while read file; do
  # recursively process directories but not sym-links
  if [ -d "${1}/${file}" ] && [ ! -h "${1}/${file}" ]; then
    # don't do . and .. either
    if [ "$file" != "." ] && [ "$file" != ".." ]; then
      doDir "${1}/${file}"
    fi
  else
    # output the file
    echo "<file><![CDATA[$file]]></file>"
  fi
  done
  echo "</dir>"
}


# normalise initial directories so they all work
DIR=$1
if [ "."   == "$DIR" ]; then DIR="$(pwd)" ; fi
if [ ".."  == "$DIR" ]; then DIR=".."     ; fi
if [ "../" == "$DIR" ]; then DIR=".."     ; fi
if [ "./"  == "$DIR" ]; then DIR="$(pwd)" ; fi
if [ "/"   == "$DIR" ]; then DIR=""       ; fi


doDir $DIR