Wednesday 13 April 2022

Howto make your OS X swap file on-demand

(very old draft) I like to tweak things. With Linux this is easy but with Apple's OS X it
sudo nano /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

XML Stylesheet Pretty Printer for any XML File

(very old draft) XML files are hard to read. This is made worse when spaces and newlines are removed. To make them more readable I have made a simple stylesheet that will indent the XML and format it so it is easy to read. The XML stylesheet is very simple: To get the stylesheet, highlight it, copy it and paste it into an empty file called pcSimpleReformat.xsl (you can change the name it you like). To apply the stylesheet you need an XSL[T] processor such as xsltproc:
xsltproc pcSimpleReformat.xsl someXMLFile.xml
The output will be the easy-to-read version of the the XML file. A slightly more complicated version follows complete with comments so you can see how it works: This is the small version. It outputs will formatted text but it is a little verbose. To download, select the XML code, copy and paste it into a file called pcReformatSimple.xsl http://www.w3.org/1999/XSL/Transform"> To apply it you would use a command like this: xsltproc pcReformatSimple.xsl someInputXMLFile.xml To test that it is working we can apply it twice and compare the output at each stage. If it is working then the intermediate stage should be identical to the final stage. ie. it will reformat an xml file the xml is reformatted xsltproc pcReformat.xsl q.rng > q1.rng diff q.rng q1.rng less q.rng http://www.w3.org/1999/XSL/Transform">

Sending syslog messages to a mac

(very old draft)

I have a router that is playing-up. It locks-up about once per day. The router is a DG834GV (v1). I recently changed ISPs and the router started to lock-up - it could just be a coincidence. My new ISP resells Telstra's ADSL(1) service so the DSLAM and probably the port are unchanged. Anyway, I thought it might be nice to see if there are any syslog messages being sent just before the router locks-up. I have a mac that runs OS-X (currently 10.5.5) that has a syslog server. So I thought I could use it to capture the syslog messages. The computer said no. The apple default configuration only logs local syslog messages. A quick google search showed that other people have re-configured it to work so I followed some instructions (which I have modified) I found here:

To enable your Leopard system to receive network syslog messages edit: /System/Library/LaunchDaemons/com.apple.syslogd.plist Open the Terminal application found in Applications/Utilities. Type sudo nano /System/Library/LaunchDaemons/com.apple.syslogd.plist Find this comment: "Un-comment the following lines to enable the network syslog protocol listener". If you can not find this comment, see my note below on my upgraded Tiger system. Change the next <!-- into <!-- --> and the following --> into <!-- -->. Now save the file by holding down the control key and pressing o to write-out the file and then you can hold control and press x to exit the editor. To enable these changes you need to restart the syslog server: sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist That should allow your external device to dump it’s log into the system log. In my case I also needed to enable syslog packets through my manually configured firewall so I executed the following command: sudo ipfw add 5 allow log udp from any to any syslog in Note: This would need to be re-entered after a re-boot. I can now see syslog messages from my router being logged to my mac using the Console application found in Applications/Utilities. Here is an example of a syslog message from my router: 12/01/09 1:33:55 AM IP address is the same, and does not need to be updated! Testing From a terminal you can issue the following command to send a syslog message to your logger:
syslog -s -r yourMacsName.local "some message" or you can use the IP address of your mac: syslog -s -r 10.0.0.1 "some message" See the man page for syslog for a full description of the message format. Upgraded Tiger Systems I also have a PowerBook G4 that has been upgraded from Tiger to Leopard and it has a slightly different /System/Library/LaunchDaemons/com.apple.syslogd.plist file. The main problem is that it does not appear to have the comment I referred to. So to enable it you need to un-comment the NetworkListener key and following dict lines. Unfortunately, I can not get this syslog server to work.

Light Wells

(very old draft) There is some construction work taking place on and under the ramp at Sydney Terminal Station. The work is, I am told, to refurbish rooms under the ramp for the Railway fire department. Work above these rooms on the ramp revealed Light Wells that had been covered for years - probably over 80 years. These Light Wells consist of a metal frame filled with blocks of coloured glass which allow light to pass through into the rear of the voids under the ramp.

Simple Rolling Hash- Part 3

(very old draft) The Rolling Hash

Now that there is a way to remove the first character and append a new last character in constant time to a string hash value, we can also use this hash function suite to perform substring searches in linear time.

To find a substring of length m in a string of length n you might proceed as follows:
start at the first character of the string
compare the substring to the string for m characters.
If same, then the substring is found.
else, repeat from the next character of the string (stop when there are less than m characters left in the string)

In practice this works well since it is unlikely that more than a few character will match each round so the effective time efficiency is much less than O(n**m). But for some strings, such as DNA which have small alphabets, the number of matches in each round could be large.

But by using a rolling hash this can be reduced to linear-time.

Calculate the hash of the substring we are searching for.
Start with a hash of the first m characters of the string
If the hash values match then the substring is found.
else remove the first character of the string from the hash and add the m+1 th character to the hash.
Now the hash is of m characters from the next character of the string.

This process is O(N).

Reducing Fridge Energy Consumption

(very old draft) For about 12 months our family has tried to reduce our gas and electricity energy consumption. We use gas for hot water, wood for heating and we have no air conditioner so our main electricity consumer was our fridge. I confirmed this by measuring the energy that the fridge used. We have an old fridge. It is an upside-down model and at least 15 years old - a Westinghouse Silhouette 411L RB411B. In our situation - on the lowest [power] setting - it consumes about 1360WHr per day which consists of a constant load of 50mA (12.5W) and about 165W when the compressor is running. The power factor for the motor appears to be about 0.66 - of course these figures are rough and really just provide me with a baseline. The compressor typically operated for 11 minutes in every 41 or about 27% of the time. I began by focusing on the constant 12.5W. What is it? The fridge is not a smart fridge so why does it constantly consume 50mA? This adds up to 300WHr per day or about 22% of the total energy consumption. This fridge has a cooling plate in the upper section where the milk and eggs are kept. It seems that the design causes ice to build up on the plate which could, if not removed, reduce the efficiency of the fridge. To overcome this, a heating element was installed behind the plate to melt any ice when the compressor stops. The melted ice drips into a gutter and is drained to an evaporator above the motor. So, I ripped it out. I could have just disconnected it, but I didn't. Having removed the defrost element, I had to develop a plan to remove ice from the plate. I achieved this by placing the fridge on a timer which cut power for, initially, 3 hours from about 3AM. This didn't have a noticeable effect on the operating range of the fridge and it allowed enough time for the ice to melt. In theory, the fridge should now consume about 1060WHrs - a big difference. But shouldn't it reduce the energy consumed by more than 12.5W because the heating energy must then be pumped out the next time the compressor runs? If the compressor is like an air-conditioner, then the cooling efficiency is about 2.5 which means that it should take 5W to remove the heat from the defrost element. Therefore the reduction should be more like 17-18W or 410-430WHr per day. My records aren't the best, but I think the fridge now consumed 950WHr per day based on a duty cycle of 24%. My next thought was to insulate the fridge. As I said, it is an old fridge and has the radiator elements on the outside and about 25mm from the back of the fridge. I placed a blanket between the radiator and the fridge and I draped a doona (light weight polyester blanket) over the sides of the fridge. This also had an effect. The on-time reduced to 9 minutes in 43 minutes or about 21%. The result is an average energy consumption of 810WHr per day - down 40% from my unmodified fridge. My current challenge is to see if I can do better by controlling when the fridge can operate. The idea is that thermostat is too sensitive and operates when the door is opened and the fridge looses it's cold air. My calculations suggest that the warmer air from the room doesn't take much energy to get down to normal fridge temperatures so by making the thermostat a secondary temperature controller I hope to reduce the energy consumption further. It also allows me to have two defrost periods during a day and to ensure that the fridge has plenty of time to adjust to post meal activity as warm foods and goods are placed back into the fridge. Initial results are promising: without insulation the fridge is consuming about 800WHr per day, so with insulation it should be even lower. The fridge is operating between 4 and 9 degrees Celsius and typically between 5 and 8 degrees.

Australian Energy Use

(very old incomplete draft) How much energy does Australia consume and where is it used? According to the Australian Bureau of Statistics ABS, Australia used 5,525 petajoules (PJ) in 2004-2005. A Peta Joule is 1,000,000,000,000,000 Joules (which isn't that helpful is it?) 1 PJ is 1,000,000,000 MJ. 3.6 MJ is 1kWHr so 1 PJ is 278 Million kWHr. The transport sector (including household transport) is the largest end-user of energy, using 1,340 ... The manufacturing sector was the second highest user of energy (1,247 PJ ...). The transport and manufacturing sectors together accounted for 68% of total energy end-use. Domestic energy consumption is 433 PJ (electricity, gas, etc.) So, if we want to make a difference in terms of our energy consumption - and therefore CO2 emissions - shouldn't we focus on transport and manufacturing rather than

SNMP to XML

(very old draft - may not be complete) I wanted to be able to query SNMP data from a device and turn it into XML so that it can be queried with XPath or XQuery. The idea is to be able to script something like this:
snmpget -v2c -c$community -OXf $host $oid | ./pc-snmp2xml.sh
eg.
snmpget -v2c -cpublic -OXf 10.0.0.1 ifInOctets.1 | ./pc-snmp2xml.sh
So,
snmpget -v2c -cpublic -OXf 10.0.0.1 ifInOctets.1
might return
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets[1] = Counter32: 2606713050
whereas this
snmpget -v2c -cpublic -OXf 10.0.0.1 ifInOctets.1 | ./pc-snmp2xml.sh
returns
snmpbulkwalk works just as well to convert the whole MIB to XML. I have tested this script with Apple Time-Capsule, Nortel 425 switch, MacBook Pro, and a Billion 7404 ADSL router. I had to pre-process the MacBook Pro and Billion router output to remove newline characters and duplicate output respectively. I wrote the SNMP to XML script in AWK because I thought AWK's pattern matching and parsing would simplify the task. I'm not convinced, but it was a good learning exercise. This is the clean-up script.
#!awk # This converts snmp output like this: # .iso.org.dod.internet.mgmt.mib-2.host.hrSWRun.hrSWRunTable.hrSWRunEntry.hrSWRunName[4592] = Hex-STRING: 6A 61 76 61 00 00 00 00 00 00 00 00 00 00 00 00 # 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 # 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 # into this: # .iso.org.dod.internet.mgmt.mib-2.host.hrSWRun.hrSWRunTable.hrSWRunEntry.hrSWRunName[4592] = Hex-STRING: 6A 61 76 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0 0 00 00 00 00 00 00 # where multiple lines of data are converted into a single line by eliminating the new-line characters # process expected OID lines that start with . /^(\.).*/{ if ( FNR > 1 ) printf "\n" # terminate last OID bu not the first line printf "%s", $0 # start this OID but don't write a new-line yet next } # process other lines that we assume are actually part of previous OID { printf "%s", $0 # continue previous line } # terminate the last OID END{ printf "\n" # terminate last OID }

Howto make Virtual Machines running in VirtualBox using Physical Partitions on OS-X

(very old draft - may not be complete)

My goal is to run Linux on my new MacBook Pro 9,1.

Whilst I can sort-of can run Linux, it is not really usable until all this new hardware is understood and drivers built to manage it.

So, next best is to run Linux in a VM. I like FOSS so I will use VirtualBox.

Unfortunately, this seems hard to do, so I have document my approach for others to improve upon.

Requirements

1. Run Linux in a VM using VirtualBox running on OS-X

2. Use real/physical disk partitions so that when this laptop is better supported, I will be able to boot into these physical partitions and drop my VirtualBox and OS-X layers

Procedure 

Before proceeding, you need to understand that if you or I make a mistake, OS-X partitions, EFI partitions, and any other recovery data on our hard drives could be lost or a pain to recover.

Your setup is likely to be different to mine so don't simply copy my detail and expect your system to work.

I have re-partitioned my hard drive as follows:


rex:~ phil$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.1 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh HD            70.0 GB    disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
   4: 21686148-6449-6E6F-744E-656564454649               200.0 MB   disk0s4
   5:       Microsoft Basic Data                         20.0 GB    disk0s5
   6:       Microsoft Basic Data                         20.0 GB    disk0s6
   7:       Microsoft Basic Data                         350.0 GB   disk0s7
   8:                 Linux Swap                         19.8 GB    disk0s8

Partition 1 is my EFI partition - I don't want to touch this.
Partition 2 is my shrunk OS-X partition - again, I'd like to keep this in good working order.
Partition 3 is OS-X recovery files - probably a CD/DVD image that I also need to keep safe.
Partition 4 is something that OS-X disk manager created - I'll leave that alone too.
Partition 5 already has a mostly-working Linux root file system - I don't want to mess with this.
Partition 6 will be my new Virtual Machine root file system (and possibly swap area).
Partition 7 already is formatted ext4 and has my /home file system - I want to keep this.
Partition 8 is a Linux swap area I will look at using a bit later.


So, I only want to work with Partition 6 and 7.


rex:phys phil$ ls -la /dev/disk0*
brw-r-----  1 root  operator   14,   0 22 Jul 19:47 /dev/disk0
brw-r-----  1 root  operator   14,   2 22 Jul 19:47 /dev/disk0s1
brw-r-----  1 root  operator   14,   1 22 Jul 19:47 /dev/disk0s2
brw-r-----  1 root  operator   14,   3 22 Jul 19:47 /dev/disk0s3
brw-r-----  1 root  operator   14,   4 22 Jul 19:47 /dev/disk0s4
brw-r-----  1 root  operator   14,   5 22 Jul 19:47 /dev/disk0s5
brw-rw-rw-  1 root  operator   14,   6 26 Jul 22:37 /dev/disk0s6
brw-rw-rw-  1 root  operator   14,   7 26 Jul 21:51 /dev/disk0s7
brw-r-----  1 root  operator   14,   8 22 Jul 19:47 /dev/disk0s8

So that VirtualBox can access these partitions, I ran these commands to give everyone read and write access:

sudo chmod a+rw /dev/disk0s6
sudo chmod a+rw /dev/disk0s7

Note: I suspect that after a reboot, I will need to re-run these commands - I'll solve that later.

Create a new VM in VirtualBox.

I called mine phys, and told VB it will hold a 64-bit version of Ubuntu.
I gave it 2048 MB RAM, and told it not to worry about a start-up disk.

I now have a ~/VirtualBox/phys folder for this VM.

cd ~/VirtualBox/phys

VirtualBox needs a VMDK file to tell it how to manage each of these partitions.

sudo VBoxManage internalcommands createrawvmdk -rawdisk /dev/disk0s6 -filename d0s6.vmdk
sudo VBoxManage internalcommands createrawvmdk -rawdisk /dev/disk0s7 -filename d0s7.vmdk

Since these new files are owned by root, I ran chown to fix this - you would use your own account name here:

sudo chown phil:staff d0s6.vmdk
sudo chown phil:staff d0s7.vmdk

This makes 2 VMDK files in my current VM directory.


rex:phys phil$ ls -la
total 56
drwxr-xr-x  8 phil  staff   272 26 Jul 21:31 .
drwxr-xr-x  7 phil  staff   238 26 Jul 22:44 ..
-rw-r--r--  1 phil  staff   153 23 Jul 01:21 HOWTO.txt
drwx------  6 phil  staff   204 26 Jul 21:32 Logs
-rw-------  1 phil  staff   633 26 Jul 22:47 d0s6.vmdk
-rw-------  1 phil  staff   634 26 Jul 21:50 d0s7.vmdk
-rw-------  1 phil  staff  7524 26 Jul 21:31 phys.vbox
-rw-------  1 phil  staff  7524 26 Jul 21:28 phys.vbox-prev

Now I added these two physical disk descriptions to my VM.


You do this in Settings - Storage, select SATA, click 'Add Attachment' icon, select Hard Disk, and select your VMDK file. I did this for each VMDK file.

You will also need to add your Linux distribution ISO either using a real CD/DVD or using an ISO stored on your file system somewhere.

Don't forget to make your VM boot from CD/DVD first.

I have been terse here since those actions should be known by anyone who is experienced at creating VMs in VirtualBox.

You can make other changes to your VM if you like, or you can do it later.

You can now boot your VM.

This is where I ran into problems: I could not get Ubuntu to format my d0s6.vmdk partition during install. Instead, I ran Ubuntu's live CD and launched Ubuntu's Disk Utility. It showed 2 disks as expected. I added a GUID partition table to d0s6.vmdk, created a 4000 MB swap partition and a 16 GB ext4 root file system. I then formatted the file system.

Now I installed Ubuntu: I told it to use the swap partition, and to use my 16 GB partition for root. To do this I told Ubuntu NOT to format it again since I formatted it earlier.

I also told Ubuntu to use my existing home drive and again, to NOT format it since I want to keep its data.

My install took hours, so be patient.





Blue-Green Water - Follow up

(Very old draft)

After almost 4 weeks, our blue-green stain has not returned.

The addition of 500 mL of lime raised the pH to nearly 11. Over time this has reduced to below 10 since about 10,000 L of new rain has been collected.

I also noted that the rain water had a pH of about 6. This was a little surprising since our tank water was pH 4.5.

In monitoring the treated tank water I have found that the alkalinity of the water decreases to nearly 7 when left for 24 hours or more. I suspect that this is due to the water absorbing CO2 from the air and this may also explain why rain water has a higher pH compared to our tank water - if left, the pH increases as the water absorbs CO2.

To help confirm this theory, I left two containers of treated tank water to stand. One was over 300mL (pH=9.94) and the other was less than 60 mL (pH=9.83). Both containers had similar surface areas. The idea is that given the same surface area, the two systems would absorb CO2 at the same rate but due to the different amounts of water, the larger body would change pH more slowly. This is what I observed: the smaller decreased pH to 7.15 in less than 24 hours. The larger body had a pH of 7.49.

Note: Although the two samples should have started with the same pH I suspect the smaller sample had already absorbed significant CO2 before I measured it. I delayed measurement for 15 minutes.