When I purchased my first Mac in 2005, my intention was to get some good hardware, avoid any Microsoft tax and run Linux.
The closest I came was Ubuntu 9.04 in 2009, but I was tied to iTunes for my Ipod Touch and a little stuck with iPhoto.
I tried again in 2010 but I could not get Ubuntu 10.04 working well enough.
More recently I purchased a Nexus One Android phone and gave away the iPod. To do this I had to divorce iTunes. This turned out to be easy. I hardly needed any apps or games and I never purchased any music. So all I had to do was copy music files from the iTunes directory into the Nexus One.
A good friend (who is really my beta tester for... well... everything) got Ubuntu 10.10 working nicely on his Mac and I quickly tagged along. I have not run OSX for 2 weeks now and I don't miss it.
Ubuntu is better than good enough.
Here is his How-To.
I am now moving my iPhoto pictures to Picasa and onto the web using 20GB of Google storage for $5/year. Then I'll clean up my files on OSX and do something else with the partition. I still have another small OSX partition to allow me to upgrade any Apple firmware and to check hardware such as the battery. It also means I have OSX should I need it.
I wonder if OSX will run under VirtualBox?
Friday, 21 January 2011
Sunday, 2 January 2011
Solar Panel Update
After a year, the performance did not seem to match what was estimated. We generated less than 2000 kWh whereas it should have been closer to 2700 based on mean solar hours. The data for the last 9 month indicated that this year would be no better.
I suspected a problem.
I contacted the supplier and they asked me to keep daily records which I more-or-less did for several weeks. I also obtained the solar radiation figures from the Bureau of Meteorology.
I was supposed to send them in, but I have not had time to do this.
Then, about a week ago, I noticed that the inverter was off - the circuit breaker had tripped and so had the one in the meter box.
Resetting them made no difference, so it seems that my inverter has died.
Last week, the inverter was replaced. It has a different UI which includes the energy generated so far for the current day down to 10Whr. Very useful.
I suspected a problem.
I contacted the supplier and they asked me to keep daily records which I more-or-less did for several weeks. I also obtained the solar radiation figures from the Bureau of Meteorology.
I was supposed to send them in, but I have not had time to do this.
Then, about a week ago, I noticed that the inverter was off - the circuit breaker had tripped and so had the one in the meter box.
Resetting them made no difference, so it seems that my inverter has died.
Last week, the inverter was replaced. It has a different UI which includes the energy generated so far for the current day down to 10Whr. Very useful.
Thursday, 21 October 2010
Barlow St. Sydney Tram Tracks
Energy Australia are doing some work in Barlow St. Sydney.
Their contractors(?) have uncovered some old Tram tracks. I managed to get a photo of one the rails and a partly visible sleeper.
I asked the friendly contractors for a small piece of rail - worth a try I think.
Their contractors(?) have uncovered some old Tram tracks. I managed to get a photo of one the rails and a partly visible sleeper.
I asked the friendly contractors for a small piece of rail - worth a try I think.
Wednesday, 20 October 2010
Blue-Green Water Results
After almost 4 weeks, our blue-green stain has not returned.
Treated Tank Water Alkalinity
The addition of 1000 mL of lime raised the pH to nearly 11. Over time this has reduced to below 10 since about 10,000 L of new rain has been collected.
Rain Water Alkalinity
I also noted that the rain water had a pH of about 6. This was a little surprising since our tank water was pH 4.5.
Stale Treated Tank Water Alkalinity
In monitoring the treated tank water I have found that the alkalinity of the water decreases to nearly 7 when left for 24 hours or more. I suspect that this is due to the water absorbing CO2 from the air and this may also explain why rain water has a higher pH compared to our tank water - if left, the pH increases as the water absorbs CO2.
CO2 Absorption of Large and Small Bodies of Water
To help confirm this theory, I left two containers of treated tank water to stand. One was over 300mL (pH=9.94) and the other was less than 60 mL (pH=9.83). Both containers had similar surface areas. The idea is that given the similar surface areas, the two systems would absorb CO2 at the same rate but due to the different amounts of water, the larger body would change pH more slowly. This is what I observed: the smaller decreased pH to 7.17 in less than 24 hours. The larger body had a pH of 7.36. Not much difference, but the temporal reading indicate that the larger body of water lags the smaller.
Note: Although the two samples should have started with the same pH I suspect the smaller sample had already absorbed significant CO2 before I measured it. This was about 15 minutes.
Boiled Water
I also boiled some treated water, let it cool and checked its pH. The water was originally about 9.6 pH. After boiling it dropped to about 8 pH. I have no theory for this at present.
Saturday, 25 September 2010
Blue-Green Water
We collect rainwater from our roof into two large 30,000 litre tanks. A pump takes the water from near the bottom of one tank and feeds the house.
From the time all the down-pipes drained into the tanks, we have not run out of water. Previously less than one third of the roof was connected to the tanks and we ran out of water three times in about three years (all in the August - October months).
The Stain
At some point we noticed that our bath would become stained with a blue-green film. Over time the thickness of the film increases. The stain was on the taps, tiles, bath-base, shower curtain, and anything left in the bath for a long time. Being white, it was most noticeable on the bath-base.
The Experiment
Late last year, we ran an experiment: we cleaned the blue-green film from the bath and switched to town water.
The stain did not return.
I had a number of theories.
1. It was a bio-slim similar to that which would grow in a sand filter. I had experimented with filtering washing machine water through a sand filter and it develops a blue-grey slim on the surface of the sand.
2. It was some other mould.
I initially thought that the town water - being high in chorine - was keeping the mould/bio-film under control. But bleach/exit mould would not shift it so these theories did not make sense.
After a recent bath cleaning, I did some more research. The only mention of blue-green stains was in the context of copper stains - but we had rainwater which was 'pure' water I thought.
Rainwater is Acidic
It turns out that rainwater is acidic due to CO2 - carbonic acid.
Acidic water dissolves the protective copper oxide layer inside the pipes and in an alkaline environment, copper hydroxide will come out of solution.
The bath is probably an alkaline environment due to the soaps and shampoos.
I checked the toilet cistern - no blue-green stain. Probably because it is not an alkaline environment.
I checked the washing machine - no blue-green stain, but there is a yellow-brown stain (could this be some other copper product?).
Data Collection
I needed to measure pH. So I purchased a pH meter, buffer solutions and de-ionised water.
I also purchased some universal paper, but this does not seem to work very well.
Tank Water
Water in tanks: 4.5 pH - acidic
Water from kitchen tap: 4.9-5.0 pH
Water from bath tap: 4.89 pH
Hot water from kitchen tap: 5.35 pH at 50 degrees C
Hot water from bath tap: 5.4 pH at 47 degrees C
Our rainwater is very acidic, but the more copper pipe it travels through the less acidic it gets.
Also, the hot water is less acidic than the cold water.
In both cases it seems that the acidity is being reduced as copper is being dissolved and this process is accelerated by high temperature.
Town Water
At meter: 7.75 pH
Stagnant Water
0 min: 5.11 pH
11 min: 5.56
13 min: 5.7
15 min: 5.7
17 min: 5.73
20 min: 5.82
It again looks like the acidity drops when the water is sitting in the pipe. I suspect that this is because the acidity is dissolving the copper pipe and it seems to happen quickly.
Research
I found that blue-green water is a mystery. There does not seem to be a single cause, but acidity, O2, CO2 and temperature are all suspects.
The CSIRO has done some research.
They suggested that Microbiologically Induced Corrosion (MIC) should be researched but they did not seem to offer a cause.
A massive Thesis by Owais E. Farooqi might be interesting to others as it contains a lot of data, statistical analysis and mathematical modelling of various forms of copper corrosion.
I did find one paper on copper corrosion in particle accelerators.
The paper shows that:
So, I need to increase pH to 7.5 or higher. A pH of 9 seems ideal. Town water seems to be above 7.5, so this indicates that the town water minimises copper corrosion.
At pH greater than 7, the corrosion due to temperature is minimised.
At pH greater than 8.5 and less than 9.5 minimises corrosion.
Treatment
My inlet is very low in the tank. This is supposed to be anaerobic so it should be low in O2 - but how low? I lowered the inlet pipe some time ago. Could this be contributing to copper corrosion by inadvertently ending up with a O2 concentration between 30 and 1000 ppb?
According to wikipedia, fresh water has 6mL per litre or 6,000,000 ppb.
After some experimentation I decided that 500ml of lime power added to one tank would be my first step.
This seemed to take the pH from about 5 to 5.8 almost immediately. But I needed the pH to be over 7.5 and ideally 9. So I added more lime and the pH went to 10.8.
I can not take the lime out, so I will monitor the pH over the next few weeks and see what it does.
Testing
The bath is virtually clear of the blue-green stain so time will tell if the stain re-appears. If it does not return I may have solved our problem.
From the time all the down-pipes drained into the tanks, we have not run out of water. Previously less than one third of the roof was connected to the tanks and we ran out of water three times in about three years (all in the August - October months).
The Stain
At some point we noticed that our bath would become stained with a blue-green film. Over time the thickness of the film increases. The stain was on the taps, tiles, bath-base, shower curtain, and anything left in the bath for a long time. Being white, it was most noticeable on the bath-base.
The Experiment
Late last year, we ran an experiment: we cleaned the blue-green film from the bath and switched to town water.
The stain did not return.
I had a number of theories.
1. It was a bio-slim similar to that which would grow in a sand filter. I had experimented with filtering washing machine water through a sand filter and it develops a blue-grey slim on the surface of the sand.
2. It was some other mould.
I initially thought that the town water - being high in chorine - was keeping the mould/bio-film under control. But bleach/exit mould would not shift it so these theories did not make sense.
After a recent bath cleaning, I did some more research. The only mention of blue-green stains was in the context of copper stains - but we had rainwater which was 'pure' water I thought.
Rainwater is Acidic
It turns out that rainwater is acidic due to CO2 - carbonic acid.
Acidic water dissolves the protective copper oxide layer inside the pipes and in an alkaline environment, copper hydroxide will come out of solution.
The bath is probably an alkaline environment due to the soaps and shampoos.
I checked the toilet cistern - no blue-green stain. Probably because it is not an alkaline environment.
I checked the washing machine - no blue-green stain, but there is a yellow-brown stain (could this be some other copper product?).
Data Collection
I needed to measure pH. So I purchased a pH meter, buffer solutions and de-ionised water.
I also purchased some universal paper, but this does not seem to work very well.
All samples were taken after running the taps for a period long enough to ensure all stagnant water was flushed.
Tank Water
Water in tanks: 4.5 pH - acidic
Water from kitchen tap: 4.9-5.0 pH
Water from bath tap: 4.89 pH
Hot water from kitchen tap: 5.35 pH at 50 degrees C
Hot water from bath tap: 5.4 pH at 47 degrees C
Our rainwater is very acidic, but the more copper pipe it travels through the less acidic it gets.
Also, the hot water is less acidic than the cold water.
In both cases it seems that the acidity is being reduced as copper is being dissolved and this process is accelerated by high temperature.
Town Water
At meter: 7.75 pH
Water from kitchen tap: 7.75 pH
Town water is alkaline.
Stagnant Water
0 min: 5.11 pH
11 min: 5.56
13 min: 5.7
15 min: 5.7
17 min: 5.73
20 min: 5.82
It again looks like the acidity drops when the water is sitting in the pipe. I suspect that this is because the acidity is dissolving the copper pipe and it seems to happen quickly.
Research
I found that blue-green water is a mystery. There does not seem to be a single cause, but acidity, O2, CO2 and temperature are all suspects.
The CSIRO has done some research.
They suggested that Microbiologically Induced Corrosion (MIC) should be researched but they did not seem to offer a cause.
A massive Thesis by Owais E. Farooqi might be interesting to others as it contains a lot of data, statistical analysis and mathematical modelling of various forms of copper corrosion.
I did find one paper on copper corrosion in particle accelerators.
The paper shows that:
- Increasing temperature increases copper solubility.
- Very low O2 (less than 30 ppb) and very high O2 concentrations ( greater than 1000 ppb) decrease copper corrosion, but lower than 30 ppb is best.
- pH less than 7 (caused by CO2) increases copper corrosion.
So, I need to increase pH to 7.5 or higher. A pH of 9 seems ideal. Town water seems to be above 7.5, so this indicates that the town water minimises copper corrosion.
At pH greater than 7, the corrosion due to temperature is minimised.
At pH greater than 8.5 and less than 9.5 minimises corrosion.
Treatment
My inlet is very low in the tank. This is supposed to be anaerobic so it should be low in O2 - but how low? I lowered the inlet pipe some time ago. Could this be contributing to copper corrosion by inadvertently ending up with a O2 concentration between 30 and 1000 ppb?
According to wikipedia, fresh water has 6mL per litre or 6,000,000 ppb.
After some experimentation I decided that 500ml of lime power added to one tank would be my first step.
This seemed to take the pH from about 5 to 5.8 almost immediately. But I needed the pH to be over 7.5 and ideally 9. So I added more lime and the pH went to 10.8.
I can not take the lime out, so I will monitor the pH over the next few weeks and see what it does.
Testing
The bath is virtually clear of the blue-green stain so time will tell if the stain re-appears. If it does not return I may have solved our problem.
Labels:
blue-green stain,
CO2,
copper corrosion,
lime,
O2,
pH,
rainwater,
treatment
Thursday, 26 August 2010
How to remove commas from quoted strings in csv files
I needed to remove commas (,) from within double quotes (") in a Comma Separated Variable (CSV) file.
For example, you can use cut -d, -f2,4,5 to extract fields 2, 4 and 5.
But if there is a comma in the text of a field like this "hello, world" you are generally stuck.
Also SQLite can also import csv files, but again commas in quotes cause problems.
I was stuck until I managed to create this sed script that will work in many cases.
# This bash function uses sed to remove up to 4 individual commas
# sequences from a quoted string in a csv file.
# eg. "Hello,, World, nice day." -> "Hello World nice day."
function removeCommas(){
while read data; do
echo "$data" | sed -e 's/^/,/g' | sed -e "s/$/,/g" \
| sed -e 's/\(,\"[^,]*\),*\([^,]*\),*\([^,]*\),*\([^,]*\),*\(.*\",\)/\1\2\3\4\5/g' \
| sed -e 's/^,//g' | sed -e 's/,$//g'
done
}
A friend mentioned that the sed commands can be combined as follows:
echo "$data" \
| sed \
-e 's/^/,/g' \
-e "s/$/,/g" \
-e 's/\(,\"[^,]*\),*\([^,]*\),*\([^,]*\),*\([^,]*\),*\(.*\",\)/\1\2\3\4\5/g' \
-e 's/^,//g' \
-e 's/,$//g'
How it works
The engine uses sed.
This sed statement takes input from stdin and replaces the regular expression 'from' with 'to' for any and all occurrences of 'from'.
sed -e 's/from/to/g'
For example, you can use cut -d, -f2,4,5 to extract fields 2, 4 and 5.
But if there is a comma in the text of a field like this "hello, world" you are generally stuck.
Also SQLite can also import csv files, but again commas in quotes cause problems.
I was stuck until I managed to create this sed script that will work in many cases.
# This bash function uses sed to remove up to 4 individual commas
# sequences from a quoted string in a csv file.
# eg. "Hello,, World, nice day." -> "Hello World nice day."
function removeCommas(){
while read data; do
echo "$data" | sed -e 's/^/,/g' | sed -e "s/$/,/g" \
| sed -e 's/\(,\"[^,]*\),*\([^,]*\),*\([^,]*\),*\([^,]*\),*\(.*\",\)/\1\2\3\4\5/g' \
| sed -e 's/^,//g' | sed -e 's/,$//g'
done
}
A friend mentioned that the sed commands can be combined as follows:
echo "$data" \
| sed \
-e 's/^/,/g' \
-e "s/$/,/g" \
-e 's/\(,\"[^,]*\),*\([^,]*\),*\([^,]*\),*\([^,]*\),*\(.*\",\)/\1\2\3\4\5/g' \
-e 's/^,//g' \
-e 's/,$//g'
How it works
The engine uses sed.
This sed statement takes input from stdin and replaces the regular expression 'from' with 'to' for any and all occurrences of 'from'.
sed -e 's/from/to/g'
First, I add a comma to the start and end of each line with this:
sed -e 's/^/,/g' | sed -e "s/$/,/g"
These make the function work for special cases and they get removed after the commas have been removed.
The 'from' string starts of by finding the beginning of a quoted string ',\"' then all non-comma characters '[^,]*'.
This pattern is enclosed in brackets '\(' and '\)' to assign the matching pattern to, in this case, part 1.
Then it matches the next one or more commas ',*'. This is not in brackets since we don't want them.
The next part of the pattern '\([^,]*\)' is like the first: it finds a string of non-comma characters and keeps the values as part 2.
Then we skip over any commas again.
This sequence can be repeated as many times as you like. I did it 3 times.
The pattern ends like it starts. But this time it reads any character including commas up to the trailing quote using '.*'. Then it reads the trailing quote '\"' and comma ','. All of this makes part 5.
This means that it will only filter the first 4 sequences of commas. To make it do more, repeat the middle pattern ',*\([^,]*\)' and increase the output parts (below).
The 'to' expression is simply the concatenation of the 5 parts of the quoted string that are guaranteed to not contain a comma (well, except possibly for the last part), where '\N' is the nth part.
\1\2\3\4\5
Testing
I tested the function with this.
function testthis(){
echo -n "\"$1\" --> \"$2\"..."
echo "$1" | removeCommas | grep -qE "$2" && echo "Pass" \
|| echo "FAIL"
}
# test removeCommas
testthis "this test should fail to test the 'tester'" "anything but this"
testthis "" ""
testthis "," ","
testthis ",," ",,"
testthis ",\"\"," ",\"\","
testthis ",\",\"," ",\"\","
testthis "1,2,\"3,4 5 6 7\",8,9,10" "1,2,\"34 5 6 7\",8,9,10"
testthis "1,2,\"3,4,5 6 7\",8,9,10" "1,2,\"345 6 7\",8,9,10"
testthis "1,2,\"3,4,5,6 7\",8,9,10" "1,2,\"3456 7\",8,9,10"
testthis "1,2,\"3,4,5,6,7\",8,9,10" "1,2,\"34567\",8,9,10"
testthis "1,2,\"3,,4,,5,,6,,7\",8,9,10" "1,2,\"34567\",8,9,10"
testthis "1,2,\"3,,4,,5,,6,,7,8\",8,9,10" "1,2,\"34567,8\",8,9,10"
testthis "\"tricky one where the quoted string starts the line 3,,4,,5,,6,,7,8\",8,9,10" "\"tricky one where the quoted string starts the line 34567,8\",8,9,10
testthis "1,2,\"3,,4,,5,,6,,7,8 tricky one where the quoted string ends the line \"" "1,2,\"34567,8 tricky one where the quoted string ends the line \""
Saturday, 17 July 2010
Firewall Rule Testing with BASH and TCPTraceRoute
I wanted to block the use of any DNS server except those that I select (google, openDNS and my router).
I also wanted to make sure that these DNS servers work and that others do not.
So I made a BASH script to verify my firewall rules.
#!/bin/bash
# only particular DNS servicer are allowed to be contacted.
# this tests that this is so
LOCAL="the.IP.address.of.your.router"
ALLOW="8.8.8.8 8.8.4.4 208.67.220.220 208.67.222.222"
BLOCK="220.233.0.4 61.88.88.88 202.139.83.3 61.9.194.49 61.9.195.193 61.9.133.193 61.9.134.49 203.161.158.2"
echo "Testing DNS servers via UDP that are allowed to work..."
for d in $LOCAL $ALLOW; do
dig @$d somename +time=1 +tries=1 +notcp > /dev/null
[ $? -ne 0 ] && echo "FAIL: Failed to get response from $d via UDP." && exit 1
echo "PASS: $d responded via UDP."
done
echo "Testing DNS servers via TCP that are allowed to work..."
for d in $LOCAL $ALLOW; do
dig @$d somename +time=1 +tries=1 +tcp > /dev/null
[ $? -ne 0 ] && echo "FAIL: Failed to get response from $d via TCP." && exit 1
echo "PASS: $d responded via TCP."
done
echo "Testing DNS servers via UDP that are NOT allowed to work..."
for d in $BLOCK; do
dig @$d somename +time=1 +tries=1 +notcp > /dev/null
[ $? -eq 0 ] && echo "FAIL: Got response from blocked DNS server $d via UDP." && exit 1
echo "PASS: DNS server $d via UDP was correctly blocked."
done
echo "Testing DNS servers via TCP that are NOT allowed to work..."
for d in $BLOCK; do
dig @$d somename +time=1 +tries=1 +tcp > /dev/null
[ $? -eq 0 ] && echo "FAIL: Got response from blocked DNS server $d via TCP." && exit 1
echo "PASS: DNS server $d via TCP was correctly blocked."
done
echo ""
echo "Firewall PASSed."
Then I thought that I might be able to test other TCP blocking rules by setting the IP packet's time-to-live (TTL) to a small number and looking for ICMP time expired packets. To do this I needed to use tcptraceroute to get the core functionality. On the Mac I got this from fink.
If I get some response, the firewall is NOT blocking an outgoing port. If I get stars (* * *) then it probably is blocking the port.
#!/bin/bash
echo "Testing blocked outgoing port..."
# set TTL to 2 hops: host to ADSL router, ADSL router to ISP gateway
# if the ISP responds to TCP TTL timeouts then a blocked port should get '2 *'
# whereas an open outgoing port should get something more complicated like this:
# '2 37.1.233.220.static.exetel.com.au (220.233.1.37) 23.176 ms'
#sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 www.google.com 79 | grep '2 *' && echo "blocked"
BLOCK="135 136 137 138 139 445 593 1863 110 9000 5190 23 1503 1720 53"
VICTIM="www.some.real.site.com"
# The victim should not get any packets if the firewall rules are right.
for p in $BLOCK; do
sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 $VICTIM $p | grep '2 \*'
[ $? -ne 0 ] && echo "FAIL: Port $p is open for outgoing traffic." && exit 1
echo "PASS: Port $p is blocked for outgoing traffic."
echo ""
done
ALLOW="80 8080 443 25 21 119 22 123"
VICTIM="www.some.real.site.com"
# again, the victim should not get any packets since TTL is so small.
for p in $ALLOW; do
sudo tcptraceroute -q 1 -w 1 -f 2 -m 2 $VICTIM $p | grep '2 \*'
[ $? -eq 0 ] && echo "FAIL: Port $p is blocked for outgoing traffic." && exit 1
echo "PASS: Port $p is open for outgoing traffic."
echo ""
done
echo ""
echo "Firewall PASSed."
You will need to modify the script to suit your firewall rules.
Tuesday, 29 June 2010
A Post from the command line using GoogleCL
One small step for Google. One giant leap for scripting.
Sunday, 20 June 2010
Google Captured Data and Passwords in Australia - an Estimate
Most would have heard by now that Google's StreetView cars have been taking pictures of our streets and at the same time collecting WiFi SSIDs. But they have also been collecting other data that has got them into trouble with Governments and privacy groups.
Google sponsored report
In Australia it is no different. Senator Conroy has been very vocal about this.
I was chatting to a friend and a quick calculation seemed to indicate that the amount of data and passwords captured must be small.
So, with lots of assumptions I have looked at two cases: the 'best' case and the 'worse' case.
To do any estimation I needed numbers. Fortunately our Bureau of Statistics (ABS) provided what I needed.
Here are my assumptions:
1. From the ABS December 2009, 5.2M ADSL and Cable subscribers. http://abs.gov.au/ausstats/abs@.nsf/mf/8153.0/
2. 114,400 TB downloaded data per year (ABS)
3. Uploaded data is 5 - 20% of downloaded data.
4. Between 50 and 80% of all subscribers use encrypted WiFi.
5. WiFi range from an indoor household Access Point is +-100m to +-250m.
6. The StreetView car samples 5 channels per second (See report).
7. Data in overlapping channels can be received in channels 1, 6, 11.
8. StreetView car travels at 30 - 50 km/h between 8:00 and 16:00. The car need bright light to take photos and for safety (fatigue) reasons they would only do 8 hour shifts - probably with breaks every 2 hours or so.
9. Uploaded data is sent evenly throughout the day from domestic homes and that between 8:00 and 16:00 the upload data rates are average.
10. Households send 1 - 10 non-secured passwords per day.
11. Household use the internet between 16 and 20 hours per day.
WiFi Reception Performance
One important variable that I did not model was the error rate of received frames that is related to distance: the further away from a WiFi access point, the lower the change of receiving a frame without error. The assumption that you can receive all frames within 100m (worse case) or within 250m (best case) is, frankly, silly and unrealistic. This will mean that any result is going to establish an upper limit for both cases.
WiFi Channels
So basically I assume the street car samples 3 channels at the rate of 5 per second. I conservatively assume that data on all the other WiFi channels can read using just these 3 channels. I doubt this is correct so again it will establish an upper limit for both cases.
Reception Period
I calculate the time the car is in range of a WiFi base is between 14 and 60 seconds and can sample between 5 and 20 seconds worth of data. The car probably travels at a speed between 30 and 50 km/h and can collect WiFi frames from any given access point for one third of the time.
Data Collected
By assuming that households transmit data continuously, and knowing the average amount of data sent each second I can estimate the amount of data collected on average per WiFi access point.
Australians downloaded 114,000 TB of data in 2009. Most web browsing is downloading, especially when we are talking about reading bank accounts and email.
Assuming uploaded data is between 5 and 20% of the amount we download, I arrived at an upload data amount of between 5700 - 23,000 TB per year or 400 - 1400 bps.
This means that they can record between 250 and 3400 Bytes of data per WiFi access point (SSID).
Password Capture
I have assumed that passwords are sent in the clear (unencrypted) between 1 to 10 times per day. Most sites (such as banks) use HTTPS and email hosting services generally support encrypted SMTP/POP/IMAP to send account and password information. But some sites may still allow access to mail and other discussion sites using unencrypted passwords. It is these passwords that Google could have captured on unencrypted WiFi access points.
The probability of capturing a password is between 0.002% and 0.14% so this means that between 17 and 3600 passwords would be captured Australia-wide as the StreetView car drove by.
Resulting Estimate
Data collected: 250 - 3400 Bytes per WiFi access point
Passwords collected: 17 - 3600 Australia-wide
The difference between my best and worse case is just over 1 order of magnitude for bytes collected and just over 2 orders of magnitude for passwords collected. This reflects the level of uncertainty on my estimates.
I think the real values will be much lower for these reasons:
1. During the day is not the peak time for downloads from households. On weekdays, on average, many family members will be at school or at work so it stands to reason that less internet activity will take place.
2. Encrypted WiFi access points seems to be closer to 80% rather than 50%. People are more aware about security and ISPs have done a lot to encourage the security of wireless networks.
3. Few services use unencrypted passwords - I can not think of any except for POP based email. All banks use some for of encryption - to do otherwise would be incompetent. Unfortunately it may be the small businesses that allow staff to access their email via unencrypted POP that are letting their employees down. GMail and Yahoo only seem to allow encrypted authentication so your account name and password are safe.
4. WiFi range is probably not even 100m and hardly 250m and the ability to pickup a transmission from a laptop/mobile at these distances is low. The further away from a WiFi access point, the higher the probability of receiving an errored frame that would contain no reliable data.
5. Uploaded data is probably less that 5% of Downloaded data. A packet containing an account name and password is small compared to the resulting page that gets downloaded.
6. Even when a WiFi access point is unencrypted, typically traffic to sites that require privacy are encrypted. So the actual unencrypted data is publicly available web pages, images, video and javascript code.
My Guess
The above best and worse case set an upper limit. But the worse case is too optimistic regarding the amount of uploaded traffic, WiFi range and the number of unencrypted access points. So I would suggest the number of passwords collected to be around 17 - say 10 to 100 - Australia-wide and the amount of unencrypted personal data to be much less - say 10 to 100 bytes per household.
http://googleblog.blogspot.com/search/label/privacy
Google sponsored report
http://www.google.com/googleblogs/pdfs/friedberg_sourcecode_analysis_060910.pdf
In Australia it is no different. Senator Conroy has been very vocal about this.
http://www.abc.net.au/news/stories/2010/05/25/2908415.htm
I was chatting to a friend and a quick calculation seemed to indicate that the amount of data and passwords captured must be small.
So, with lots of assumptions I have looked at two cases: the 'best' case and the 'worse' case.
To do any estimation I needed numbers. Fortunately our Bureau of Statistics (ABS) provided what I needed.
Here are my assumptions:
1. From the ABS December 2009, 5.2M ADSL and Cable subscribers. http://abs.gov.au/ausstats/abs@.nsf/mf/8153.0/
2. 114,400 TB downloaded data per year (ABS)
3. Uploaded data is 5 - 20% of downloaded data.
4. Between 50 and 80% of all subscribers use encrypted WiFi.
5. WiFi range from an indoor household Access Point is +-100m to +-250m.
6. The StreetView car samples 5 channels per second (See report).
7. Data in overlapping channels can be received in channels 1, 6, 11.
8. StreetView car travels at 30 - 50 km/h between 8:00 and 16:00. The car need bright light to take photos and for safety (fatigue) reasons they would only do 8 hour shifts - probably with breaks every 2 hours or so.
9. Uploaded data is sent evenly throughout the day from domestic homes and that between 8:00 and 16:00 the upload data rates are average.
10. Households send 1 - 10 non-secured passwords per day.
11. Household use the internet between 16 and 20 hours per day.
WiFi Reception Performance
One important variable that I did not model was the error rate of received frames that is related to distance: the further away from a WiFi access point, the lower the change of receiving a frame without error. The assumption that you can receive all frames within 100m (worse case) or within 250m (best case) is, frankly, silly and unrealistic. This will mean that any result is going to establish an upper limit for both cases.
WiFi Channels
So basically I assume the street car samples 3 channels at the rate of 5 per second. I conservatively assume that data on all the other WiFi channels can read using just these 3 channels. I doubt this is correct so again it will establish an upper limit for both cases.
Reception Period
I calculate the time the car is in range of a WiFi base is between 14 and 60 seconds and can sample between 5 and 20 seconds worth of data. The car probably travels at a speed between 30 and 50 km/h and can collect WiFi frames from any given access point for one third of the time.
Data Collected
By assuming that households transmit data continuously, and knowing the average amount of data sent each second I can estimate the amount of data collected on average per WiFi access point.
Australians downloaded 114,000 TB of data in 2009. Most web browsing is downloading, especially when we are talking about reading bank accounts and email.
Assuming uploaded data is between 5 and 20% of the amount we download, I arrived at an upload data amount of between 5700 - 23,000 TB per year or 400 - 1400 bps.
This means that they can record between 250 and 3400 Bytes of data per WiFi access point (SSID).
Password Capture
I have assumed that passwords are sent in the clear (unencrypted) between 1 to 10 times per day. Most sites (such as banks) use HTTPS and email hosting services generally support encrypted SMTP/POP/IMAP to send account and password information. But some sites may still allow access to mail and other discussion sites using unencrypted passwords. It is these passwords that Google could have captured on unencrypted WiFi access points.
The probability of capturing a password is between 0.002% and 0.14% so this means that between 17 and 3600 passwords would be captured Australia-wide as the StreetView car drove by.
Resulting Estimate
Data collected: 250 - 3400 Bytes per WiFi access point
Passwords collected: 17 - 3600 Australia-wide
The difference between my best and worse case is just over 1 order of magnitude for bytes collected and just over 2 orders of magnitude for passwords collected. This reflects the level of uncertainty on my estimates.
I think the real values will be much lower for these reasons:
1. During the day is not the peak time for downloads from households. On weekdays, on average, many family members will be at school or at work so it stands to reason that less internet activity will take place.
2. Encrypted WiFi access points seems to be closer to 80% rather than 50%. People are more aware about security and ISPs have done a lot to encourage the security of wireless networks.
3. Few services use unencrypted passwords - I can not think of any except for POP based email. All banks use some for of encryption - to do otherwise would be incompetent. Unfortunately it may be the small businesses that allow staff to access their email via unencrypted POP that are letting their employees down. GMail and Yahoo only seem to allow encrypted authentication so your account name and password are safe.
4. WiFi range is probably not even 100m and hardly 250m and the ability to pickup a transmission from a laptop/mobile at these distances is low. The further away from a WiFi access point, the higher the probability of receiving an errored frame that would contain no reliable data.
5. Uploaded data is probably less that 5% of Downloaded data. A packet containing an account name and password is small compared to the resulting page that gets downloaded.
6. Even when a WiFi access point is unencrypted, typically traffic to sites that require privacy are encrypted. So the actual unencrypted data is publicly available web pages, images, video and javascript code.
My Guess
The above best and worse case set an upper limit. But the worse case is too optimistic regarding the amount of uploaded traffic, WiFi range and the number of unencrypted access points. So I would suggest the number of passwords collected to be around 17 - say 10 to 100 - Australia-wide and the amount of unencrypted personal data to be much less - say 10 to 100 bytes per household.
Labels:
australia,
google,
passwords,
private data,
streetview,
wifi
IPhoto Script to Remove Missing Photos
For some reason when you try to open photos you get a big exclamation mark! This seems to happen when the actual photo is missing. Perhaps it has been deleted through the file system or perhaps iPhoto is confused or perhaps because iPhoto crashed during some operation. Who knows.
I wrote this script to find these 'photos' and to move them to the trash.
Open Script Editor, cut and paste the above script into the editor, Compile it to check for errors, and save it to a file - perhaps on your desktop but anywhere is fine.
To run it, open iPhoto, select the Photos Library, Select all photos you want to process (Edit - Select All is what I usually do), and then switch back to the Script Editor and press Run.
When it finishes you may have some missing photos in your trash. You can decide what to do with them at this point - I just empty the trash.
If you use it, write a short comment about whether it was helpful or not or whether it worked or not. If you improve the script, let me know as well.
I wrote this script to find these 'photos' and to move them to the trash.
tell application "iPhoto"
set curPhotos to selection
if (count of curPhotos) ≤ 0 then
display alert "You need to select the photos you want me to process."
else
set countPhotos to count of items in curPhotos
repeat with i from 1 to countPhotos
set thisPhoto to item i of curPhotos
try
set t to info for (image path of thisPhoto as POSIX file)
on error eStr number eNum partial result rList from badObj to expectedType
log eStr
select thisPhoto
remove thisPhoto
end try
end repeat
end if
end tell
Open Script Editor, cut and paste the above script into the editor, Compile it to check for errors, and save it to a file - perhaps on your desktop but anywhere is fine.
To run it, open iPhoto, select the Photos Library, Select all photos you want to process (Edit - Select All is what I usually do), and then switch back to the Script Editor and press Run.
When it finishes you may have some missing photos in your trash. You can decide what to do with them at this point - I just empty the trash.
If you use it, write a short comment about whether it was helpful or not or whether it worked or not. If you improve the script, let me know as well.
Subscribe to:
Posts (Atom)

