Human Readable Compact Extensible Data Interchange Format

When working with Lotus Domino databases I needed a simple and compact way to pull data from one database into another.

I wanted something that was human-readable, extensible, simple, order-agnostic and simple to process. After some experimentation I settled on an old idea to use delimited fields. But, I wanted to be able to add and remove fields and have them in any order. To do this I added a field name.

The format is

Compact = Field* ~

Field = ~ fieldname : data

The only constraints were:

1. The field name could not start with a "~"
2. The field name could not contain a ":"
3. The data could not contain a "~"

Since ":" is rarely used as a field name and "~" is rarely used in data, this was a small price to pay. If you need "~" in the data then there are many ways to do this. One way we used was to append XML-like data to the end of the Compact and then extract the data in a XML-like way.

To extract a data element we used code like this:

@left( @right( compact ; "~" + fieldname + ":" ); "~" )

This gets the string right of the "~fieldname:" and then returns data left of the "~".

Similar code can be used to extract the XML data:

@left( @right( XMLData ; "<" + fieldname + ">" ); "</" + fieldname + ">" )

(It is XML-like so you need to handle escape sequences yourself.)

Here are some examples. You can see we even carried code and XML-like data as data.

~ID:TS~SubID:VP~Icon:44~SubIcon:~SLANo:~Z1RT:2~Z2RT:2~Z1CT:8~Z2CT:8~Z1ET:7.2~Z2ET:7.2~Pri:Standard~Desc:Telephone Service:VoIP Service (Ericsson)~SubForm:VP~LocSubForm:Location~FailureCategories:~PeriodDef:~ReactionDef:~PBVer:4.3;0~NItems:~<242</ConPrice><InstPrice>0</InstPrice><MACPrice>@If(movecomplexity="0";56;movecomplexity="1";111;-1)</MACPrice><FRDSCauses>TS-VP</FRDSCauses><ResponseTime>0</ResponseTime><RepairTime>0</RepairTime><EscalateTime>0</EscalateTime><PBProductCode></PBProductCode><PBPrice>360</PBPrice><ExcludeDays></ExcludeDays><BusinessHours>[08:00]:[17:00]</BusinessHours><BillingAudit>Options(OP)~VM(OP)</BillingAudit><Applications></Applications><LineSpeeds></LineSpeeds><Interfaces></Interfaces><CDIDType></CDIDType><MIMSNoDef>PL15TE01</MIMSNoDef><AssetCategory></AssetCategory>

~RCCode:AM~RCTitle:Asset Management~MCode:~MCustCode:000001~MIDs:~Discount:20~DiscountConnect:10~

~LocA:PTSA13~Location:George St 47, Level 12N Conference Rm~Lat:30.123456~Long:151.123456~SiteZone:Metro~

Fog Machine

This is one of those Now-what-am-I-going-to-do-with-it problems.

I picked up a box with a fog machine (Antari F-160) and fog fluid in it from a recent council clean-up. Since it was being thrown out I doubted that it would work. And it didn't.

Apart from a blown fuse, the fog machine heated up but the fluid pump would not work. The pump would hum when power (240VAC) was applied but it would not pump so I decided it was seized and set to dismantling it in order to clean it.

There didn't appear to be anything wrong. Nothing was blocked, it was clean and no part seemed damaged. If I loosely assembled the pump it would work, but if it was tightly assembled the pump would just hum.

I decided that someone on the net would be able to verify that I was assembling it correctly. I searched for the part number "SP-12A" and "Fog" and I came across this page which had a link to a picture showing the order of the parts.

In my pump, the washer/seal (part 5) was in a different place. I corrected my assembly and it worked nicely. With this seal (perhaps it is a spacer) in the wrong place, a little orange seal is compressed too much and grips the piston so tightly that it no longer can move and the pump can only hum. With the seal in the correct place, the piston moves freely and all is well.

The machine uses a remote control to turn the heater on and it has a switch to operate the pump. It was clear that the remote control cable was damaged so I repaired it, but the pump would not operate when the switch was depressed. I found that the remote control socket had 3 broken solder joints which I repaired.

The fog machine now worked.

Apart from entertaining the kids, what can you do with a fog machine?

• Fill the house with fog and run a fire drill (get down low and Go Go Go!)
• Check for air conditioner ducting leaks (but I don't have air conditioning)
• Look for drafts around doors and windows
• Cool party feature
• Find exhaust leaks?
  

Maybe someone else has some thoughts.

YouTube Blackout

Introduction

I sent this to a few friends in the beginning of March 2008 (One of them posted it on his Blog). It may still be of interest but the BGP data for the YouTube event may no longer be available.

You can read more about the event on Arstechnica.

The Technique I describe may be used for any event - you just need the network address and the dates surrounding the event.

YouTube Outage BGP Replay

Last week Pakistan Telecom was ordered by their government to block access to YouTube. They did this by re-directing routes containing the YouTube addresses to an internal dead-end. A mistake was made that advertised this to their peer and thus the remainder of the internet.

The IP range that was affected was 208.65.152.0/22. Pakistan added more specific routes for 208.65.153.0/24 which are a longer match and so take priority.

To see what happened go to this site and start the java applet.

http://bgplay.routeviews.org/bgplay/

To see how it should look enter this address: 208.65.152.0/22, yesterday’s date and todays date - don’t worry about the time fields. [This route should also work now: 208.65.153.0/24]

The AS (autonomous system - roughly meaning a country or large ISP or large company) 36561 is where youtube packets should generally be sent. You can see all the lines from other domains leading to this AS for this address range.

Press the play button (a small triangle) to start the animation. You should see the links occasionally changing as changes are made or transmission links break or are fixed.

Now press the New Query button.

Then enter the address that poisoned BGP: 208.65.153.0/24, 23/2/2008 and 26/2/2008 - a period around the event.

Initially the page has no lines. This is because prior to the event, this route was not used. Just imagine all the links still going to AS 36561.

Press the Play button.

Over time you see the rouge Pakistan Telecom domain 17557 start to become the priority route for all YouTube traffic until it seems to have all the routes. When the fault was fixed you can see the links moving back to AS 36561 where they should be.

Imagine how easy it is now to interrupt any domain or the whole internet? I think this risk will be fixed shortly.

Useful OS X Applications and Utilities

This is a list of OS X Applications and Utilities that I or others I know find useful. Most is Open Source or freeware.

• Wireshark 1.0
  
• Quicksilver (apparently much more than a free app launcher - two people I know can't do without it)
• Handbrake - for DVD => mp4.
• MetaX - tag mp4 files.
• VLC
• Fairmount (required VLC) mounts a DVD as though it had no CSS.
• MacTheRipper - rips DVD to disk.
• Cyberduck - GUI based ssh client.
• Text wrangler - excellent free text / code editor.

• Firefox 3 - Should be released in June 2008. Beta versions seem stable.
• Apptrap - uninstall apps when dragging to trash.
• FanControl - keep macbook cooler.
• Menumeters - cpu/network/etc. meters.
• Vmware fusion - vm.
• Google Desktop - shows local files when searching google. Also, fast way to run apps.
• Google Earth
• Open office 2.4 (aqua) - improving each month.
• Burn and Firestarter - cd burner.
• Cronnix - schedule tasks.
• Istumble - wifi scanner.
• Fink - once they release 10.5 version it will be easy to install.

Earth Hour - What happened in 2007?

Was there any significant effect and if so, was there any grid instability?

Sydney Central Business District (CBD) showed an estimated 10.2% drop in electricity consumption but only a 2.1% drop in the rest of the metropolitan areas and the grid was stable according to the Australian National Electricity Market Forum (NEMF) Minutes 78 – 26 April 2007.

http://www.nemmco.com.au/nemgeneral/057-0342.pdf

EARTH HOUR IMPACT ON THE NEM

Wayne Jackson, EnergyAustralia, presented the Earth Hour Impact on the NEM. 'Earth Hour' was the launch event of a 12 month campaign to reduce Sydney's greenhouse gas emissions.

Sydneysiders were requested to turn off their lights for one hour between 7.30 pm and 8.30 pm, on 31 March 2007. The objective is to reduce Sydney's greenhouse gas emissions by 5% over a 12 month period.

EnergyAustralia's role was to participate in Earth Hour and also to provide load data to help measure the outcomes of the exercise. Media headline results of Earth Hour are shown below:

• 65,000 individuals and over 2,000 businesses pre-registered
• survey reported 2 million households participated
• 10.2% drop in Sydney CBD Consumption
• saving of 24.86 tonnes CO2-e in Sydney CBD
• saving of 52.08 tonnes CO2-e in Sydney Metro

The impact on the Sydney CBD and the EnergyAustralia network was measured. Data was collected from 5 substations in the Sydney CBD area. Based on Saturdays in March and April from 7:30 to 8:30 pm, a typical electricity consumption profile was collected and adjusted for temperature (19.8oC on the night) and for daylight savings.

The measured load was 204.9 MWh. The typical Saturday consumption between 19.30 to 20.30 is 228.18 MWh. Therefore, 23.28 MWh of energy was saved, a reduction in energy consumption of 10.2%.

Measurement from EnergyAustralia's Sydney metropolitan BSPs shows that 48.76 MWh of energy was saved, a reduction in energy consumption of 2.1%.

On the day network stability in the Sydney metropolitan area was very good, there was no switching or load surge problems at 8:30 pm.

The WWF reported the Sydney CBD figures but didn't mention the much lower result for the greater Sydney metropolitan area.

http://wwf.org.au/news/congratulations-sydney-earth-hour-2007-results/

This year Earth Hour will be on March 29 from 8:00 PM to 9:00 PM (half an hour later).

http://www.earthhour.org.au/