Souce code is at https://github.com/mrbalky/te923.

We deal with fairly large amounts of webserver log data, so are also saving HDFS space and job i/o by using the hadoop-lzo package. It gives fast compression that retains our ability to use the data through Hive queries.

If you are only interested in compression, and have Hadoop and Hive configured appropriately, you can even mix compressed and uncompressed data in separate partitions of a Hive table.  A normal table definition will work:

CREATE EXTERNAL TABLE foo (
                       columnA string,
                       columnB string )
       PARTITIONED BY (date string)
       ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
       LOCATION '/path/to/hive/tables/foo';

One big advantage of LZO, though, is its ability to be split in map/reduce jobs. This is done by creating an index of the LZO file with the LzoIndexer tool of the hadoop-lzo project. To actually use the index, you will need to use a special input format for your Hive table:

CREATE EXTERNAL TABLE foo (
         columnA string,
         columnB string )
    PARTITIONED BY (date string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
    STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
          OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
    LOCATION '/path/to/hive/tables/foo';

Now to actually come to the point. In my case, I had already created the table, and was trying to add indexing after the fact. Hive permits changing input format with an alter statement:

ALTER TABLE foo
    SET FILEFORMAT
        INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
        OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";

But this alters only future partitions, not existing partitions. They retain their TextInputFormat. So now when I ran my Hive queries, instead of the LZO index file being used for splitting the input, it wound wind up used as table data. My results were mostly correct, but there were some result rows that were garbage.

I fixed this by dropping and recreating the table and partitions with the correct input format. Because I use EXTERNAL tables, the data itself was preserved.

While this is not a big deal, I have lost the ability to mix compressed and uncompressed data in the table. The Hive language manual claims I can alter partition metadata, which would be another way to deal with this, but so far I’ve not been able to make that work in versions 0.5 and 0.6.

Thanks to Dmitriy and Johan from Twitter for helping me understand all this.

hadoop-lzo:

https://github.com/kevinweil/hadoop-lzo

The original hadoop-gpl-compression project:
http://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/wiki/FAQ?redir=1

Hive language manual:

http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL

Discussion of Hive and table attributes:
https://issues.apache.org/jira/browse/HIVE-957

It is easily installable via apt-get from debian-multimedia.org after all. Don’t bother with the package advice on the dvdstyler site. Just add deb http://www.debian-multimedia.org lenny main to your /etc/apt/sources.list file and run apt-get update. You should then be able to install with a simple apt-get install dvdstyler

If you run a headless debian server as I do, vncserver works like a champ.
Install: apt-get install vnc4server.
Execute: vnc4server -geometry 1024×768 -depth 24.
Export your display: export DISPLAY=myserver:1.
Run: dvdstyler.
Connect a VNC client to myserver:5901 to drive dvdstyler.

See the man pages for vnc4server for more info about the display number and connecting a client.

I scheduled it to run once a day at midnight:

0 0 * * * sleep 30;php te923Status.php <notify email address>

The 30 second sleep is intended to offset the status check from the normal weather data check that happens exactly on the minute.

Debian out of the box doesn’t relay mail to external domains, so I had to do another tweak here. To enable forwarding, I reconfigured exim according to http://pkg-exim4.alioth.debian.org/README/README.Debian.etch.html.

It’s not really the proper way to create a real relay server, but since I’m behind a firewall and only using the server for this purpose, I didn’t feel it necessary to do more. But the email does look suspicious to the receiving mail system, so if you send to an address outside your domain (like gmail, for example), the mail is likely to be determined as spam. You’ll need to create whatever filters necessary at the recipient account to avoid this.

And that’s it so far. I expect I’ll delve into RRDTool and RRDWeather now to see if I can create graphs of readings Weather Underground does not (like humidity, for example).

@@ -138,7 +138,7 @@
     }
 
     else {
-        data->uv = bcd2int( buf[18] & 0x0F ) / 10.0 + 
               bcd2int( buf[18] & 0xF0 ) + 
               bcd2int( buf[19] & 0x0F ) * 10.0;
+        data->uv = bcd2int( buf[18] & 0x0F ) / 10.0 + 
               bcd2int( ( buf[18] & 0xF0 ) >> 4 ) + 
               bcd2int( buf[19] & 0x0F ) * 10.0;
         data->_uv = 0;
     }

The next thing to address is the permissions problem. To this point, the only way to get data from the station was to be root. Otherwise, you get this error:

[mcp:…/te923/te923] te923con -D
Error while setting configuration (-1).

This is a generic issue with USB devices, and I found an item on a wiki (http://wiki.openstreetmap.org/wiki/USB_Garmin_on_GNU/Linux#Fixing_Device_Permissions) about GPS units that got me going.

That page discusses how to set the group ownership on the device, as well as the permissions on the device. Long story short, I created the device rule set /etc/udev/rules.d/99-te923.rules (all on one line):

ATTRS{idVendor}=="1130", ATTRS{idProduct}=="6801",
                    MODE="0660", GROUP="plugdev"

idVendor and idProduct identify the TE923 weather station, mode tells the USB driver to give read/write permissions to the user and group that owns the device, and group tells the driver to assign group ownership to “plugdev”. My user on the machine is a member of that group so I should be OK.

Ask the system to reload the USB rulesets:

[mcp:.../te923/te923] sudo udevadm control --reload_rules

And now I can get valid data back from the unit without being root:

[mcp:.../cwanek/cronjobs] te923con
1273414489::::59:::::::::1003.1:5.0:3:0:9:0.4:0.0:11.7:215

Unfortunately, though, some readings (specifically current temp readings) are empty. Even wide-open permissions on the device don’t help. This doesn’t make sense to me, and I have not yet solved this issue, so I’m still stuck with being root to run te923con. I’d love to know why it would work only partially.

Still, I don’t really want to have root’s crontab running the script, so I configured sudo to skip the password prompt for the group plugdev for the te928con application. With visudo, add:

%plugdev ALL=NOPASSWD: /usr/local/bin/te923con

Moving on, the I was still not able to run te923con without removing the USB human interface device module (sudo rmmod usbhid). While removing it allows access to the te923 station, it would also cause any other HID like a mouse or keyboard to stop functioning. So the trick is to get the usbhid module to release just the weather station.

There are many sites that document how to get usbhid to unbind a device. I found http://lwn.net/Articles/143397/, which gave me the following command:

sudo bash -c "echo -n 2-1:1.0 > /sys/bus/usb/drivers/usbhid/unbind"

bash -c is required so the shell redirection to unbind will succeed.

Digging deeper, I learned (http://reactivated.net/writing_udev_rules.html) that you can configure the unbind to happen immediately after the device is connected, by using the “RUN=” option in the device rules file (this is all on one line in /etc/udev/rules.d/99-te923.rules):

ATTRS{idVendor}=="1130", ATTRS{idProduct}=="6801",
    MODE="0660", GROUP="plugdev", 
    RUN="/bin/sh -c 'echo -n $id:1.0 > /sys/bus/usb/drivers/usbhid/unbind'"

So now, apart from the missing data when running as a non-privileged user, the te923 weather station is coexisting with other USB devices, and I am able to schedule the upload script in my own user crontab.

There’s one last thing…

Sure, there are packages like RRD Weather that can do this too, but where’s the fun in that.

The te923con application returns most of the data included in the wunderground protocol directly, so it’s a simple matter of parsing it from the application output. Weather Underground wants imperial units, but maybe someday I’ll want proper metric units, so I made it a parse option.

The weather station apparently does not return a calculated dew point, which is part of the upload protocol. It turns out that calculating dew point is not straighforward. But many people have implemented calculators. A guy named Wolfgang Kühn implemented one in javascript that seemed very thorough, so I did a quick port of it to PHP.

The weather station also returns only the accumulated rainfall total since the station was started, so I implemented a simple array mechanism to retain historical data from the rain counter. I can then calculate the “rain in last hour” and “rain since midnight” values required by the WUnderground protocol.

From there, it’s a simple matter to build the URL string for data upload, and a file_get_contents call to upload and get the success or failure result.

I save the rain counters and the other weather data as PHP code in a cache file that is reloaded with a PHP include the next time the script runs.

WUnderground can accept data as fast as once a second, but the weather station does not update from its sensors with anywhere near that frequency, so I felt scheduling upload once a minute via cron was more than often enough. The script takes station id, password and cache file name on the command line (all on one line):

* * * * * php te923WunderUpload.php <station ID> 
                       <wunderground password> <cache file name>

Because of the permissions issues with the USB device, for now this is scheduled in the root user’s crontab.

Now on to fixing the bugs…

The problem with this station, though, is that it’s a USB station, and Linux is not supported by the software that ships with it. But I figured I could find a way around that, and indeed I could, but it wasn’t so straightforward. Along the way I learned some arcane details about USB on Debian Linux. This series of posts is something of a blow-by-blow of the process.

The hardest part was already done for me by Sebastian John who created an application that reads the current raw data from the station. The source is available at http://te923.fukz.org/index. Building the source requires libusb, and the site links to it, but I was able to build with the vanilla libusb-dev installed by apt-get:

apt-get install libusb-dev

The build creates the executable te923con, which I then installed in /usr/local/bin.

The first time I tried to run it, I got the following error message:

[mcp:…/te923/te923] te923con -D
Error while setting configuration (-1).

This indicates insufficient privileges to access the USB device, so to get things going I ran the te923con application as root. That got me to the next error:

[mcp:.../te923/te923] sudo ./te923con -D
Error while setting configuration (-16).

The dmesg shows the following error (all on one line):

[24464516.452385] usb 2-1: usbfs: interface 0 claimed 
                    by usbhid while 'te923con' sets config #1

It seems that the TE923 registers itself as a human interface device (HID) for some reason, so the OS tries to treat it a such. The author of a package of code that creates a web view for the TE923 has the solution on his blog, though without explanation of the problem or why: http://firewall.haringstad.com/TE923-Frontend/blog/TE923-Frontend-Info/TE923-Frontend%20Blog/B9EF42CC-F942-41DE-ABDF-461549FEFB46.html

He removes the usbhid kernel module:

sudo rmmod usbhid

And success:

[mcp:.../te923/te923] sudo ./te923con -D
[DEBUG] got |07|00|0a|0a|00|0a|0a|00|
[DEBUG] got |07|0a|0a|00|0a|30|00|af|
[DEBUG] got |07|3d|03|57|c0|28|00|35|
[DEBUG] got |05|00|0a|b2|00|66|00|35|
[DEBUG] got |01|5a|0a|b2|00|66|00|35|
[DEBUG] got |02|28|82|b2|00|66|00|35|
[DEBUG] got |07|37|64|c0|96|0a|00|0a|
[DEBUG] got |07|0a|00|0a|0a|00|0a|0a|
[DEBUG] got |07|00|0a|30|00|af|3d|03|
[DEBUG] got |07|57|c0|28|00|35|00|0a|
[DEBUG] got |03|b2|00|66|00|35|00|0a|
[DEBUG] TMP 0 BUF[00]=28 BUF[01]=82 BUF[02]=37
[DEBUG] TMP 1 BUF[03]=64 BUF[04]=c0 BUF[05]=96
[DEBUG] TMP 2 BUF[06]=0a BUF[07]=00 BUF[08]=0a
[DEBUG] TMP 3 BUF[09]=0a BUF[10]=00 BUF[11]=0a
[DEBUG] TMP 4 BUF[12]=0a BUF[13]=00 BUF[14]=0a
[DEBUG] TMP 5 BUF[15]=0a BUF[16]=00 BUF[17]=0a
[DEBUG] UVX   BUF[18]=30 BUF[19]=00
[DEBUG] PRESS BUF[20]=af BUF[21]=3d
[DEBUG] STAT  BUF[22]=03
[DEBUG] WCHIL BUF[23]=57 BUF[24]=c0
[DEBUG] WGUST BUF[25]=28 BUF[26]=00
[DEBUG] WSPEE BUF[27]=35 BUF[28]=00
[DEBUG] WDIR  BUF[29]=0a
[DEBUG] RAINC BUF[29]=00 BUF[30]=b2 BUF[31]=00
1272059734:22.80:37:6.40:96:::::::::986.9:3.0:3:0:10:1.6:1.3:5.7:178

Hooray! There are still a few problems to work out with the device interface, and there is a bug in the te923con application to fix, but now I have a working interface to my weather station. Next up, getting the weather data onto Weather Underground

Access was all that I had access (heh) to, and it had a “music collection” database template, so with slight modification to the template that’s what I used.

Well now, I no longer have access (heh) to Access, Access is a PITA, and I already have MySQL available on my home server, so I figured it was time to finally migrate.

I found a handy application called Navicat (fully functional trial), that will import directly from the Access .MDB file to a MySQL database.

While Navicat mostly worked, I was not quite done. The import did not properly get the auto_increment attribute on the tables’ ‘id’ column. Also, the Access database tables had spaces in their names which is kind of a drag. Fortunately both were easily solved by exporting from MySQL to text using mysqldump, editing the .sql file that resulted, and then reimporting to MySQL from the .sql file.

Sure it’s a ridiculous thing nobody in their right mind will ever have to do, but I’m not in my right mind.

Once at the top, there are some beautiful views back into Rocky Mountain National Park from the top. A spur of trail (that we didn’t take) gets you to the true summit of Crosier.

Then it’s 4.5 miles of descent, some of it pretty sketchy and loose, but still pretty fun.

I’d do it again:

Flickr set:

In February, after my 3rd knee surgery since 2005, I had no plans or expectations for my riding this summer. And when all my friends made the commitment to a Whistler, BC trip in April, I just figured I was out. But as the summer got going, I was feeling pretty good, and riding pretty clean, and got the big encouragement from Steph, so I was in after all.

Scheduling difficulties meant that I would only get 3 days there instead of the 5 my friends would, but nevertheless, I was in.

We flew into Seattle, and rented a couple of cars there. Flights to Vancouver were 2 or more times the price. It means twice the driving, and add a border crossing, but we figured it was worth it to save a couple hundred each for 7 people.

The Whistler Bike Park is incredibly well managed, and loaded with great riding. There are ride-arounds for most of the big and committing features. I spent 2 days there, and 1 day riding Vancouver’s North Shore.

It was incredible riding. There’s a reason half the photos in bike magazines are from BC. I thought maybe this was a once in a lifetime trip, but now I sure hope it isn’t.

The bike park was amazing fun, but next time I want more North Shore. The incredible amount of work done there to build trails and mind-blowing stunts was astounding.

Friends of Tom from SoCal were there at the same time we were, and very very kindly showed us around the bike park, and especially the North Shore. They were very patient with our (meaning my) slow pace.

Lessons learned:

Park bikes are very heavy and don’t really pedal at all. They’re kind of inappropriate for my riding style on North Shore rides.

So for me, park bike for park days, trail or all-mountain bike for the other stuff.

We had a condo, but the days were long and tiring, so we ate out all the time. Restaurants in Whistler are breathtakingly expensive. We could have hired a chef to cook for us in the condo and still come out ahead.

There’s no good beer there. And it’s expensive. Buy in Bellingham, WA on your way in. (In fairness, we did receive this tip from others, but were too preoccupied with getting there.)

Body armor and full-face helmets get pretty sweaty and stinky. Since it’s somebody else’s stink, rented armor and rented full-face helmets are even worse. At least buy your own helmet.

Allow extra time for the border crossing. Check the wait times.

Take lots of photos:

Day 1; Whistler Bike Park:

Day 2; More Whistler Bike Park

Day 3; Vancouver, North Shore:

Vimeo works better than Google with the AVIs from my new camera.

The log bridges on Little Raven can be tricky:

little raven 1 from charlie wanek on Vimeo.

This video notwithstanding, Casey handled the LR/SSV ride with few problems:

casey – little raven from charlie wanek on Vimeo.

ssv 2 from charlie wanek on Vimeo.

south st. vrain 1 from charlie wanek on Vimeo.

kevin – ssv from charlie wanek on Vimeo.

tyler – ssv from charlie wanek on Vimeo.

The corkscrew at Keystone looks cool, and is scary at first, but is really more of a gimmick.

Keystone corkscrew from charlie wanek on Vimeo.

I’ve been pretty happy with my home network set up, particularly with DD-WRT providing lots of nifty stuff on my gateway router. So I didn’t want to screw with that by adding VOIP. Fortunately, while Vonage prefers the adapter between the cable modem and the router, they do support putting the phone adapter behind the router.

The shipping confirmation email from Vonage very helpfully included the MAC address of the VOIP adapter, so I was able to get my router configured way ahead of time.

I started by giving the adapter a static DHCP lease. This is trivial to set up in DD-WRT. Administration -> Services:

I am not really doing any advanced firewalling on the router, so I didn’t have to do anything special for outgoing connections. But Vonage requires incoming ports 10000-20000 forwarded to the adapter. That is also trivial to set up in DD-WRT. Applications & Gaming -> Port Range Forwarding:

With this setup, I plugged in the adapter and it just worked(tm).

The most important thing to me was 911 service. Vonage offers E911 in my neighborhood. I called up the regular police phone number and was told the only to test is to just dial 911. I don’t know if it’s this way everywhere, so you’d better not take my word for it.

It was actually sort of freaky to dial 911. You just immediately say “This is not an emergency, I’m testing my VOIP”. The emergency dispatcher’s system had already popped up with my name and address, just as promised by Vonage.

A POTS line works fine during a power outage, but what about VOIP? I have my cable modem, routers and voip adapter all on a UPS. Coincidentally, I was working at the house about a week earlier, when there was a power hit long enough to freak out my TiVo and other devices in the house. But my Comcast broadband stayed up through it, so I feel pretty good about my ability to use the phone during an outage. Again, I have no idea if this is common, so don’t come running to me if yours doesn’t work that way.

The last thing I was interested in testing was if I could get voice traffic prioritized on the WAN connection. DD-WRT has some handy Quality of Service settings, and there are a variety of pages out on teh interwebs with instructions for setting it up. In the end, I just took some conservative guesses, exempting the VOIP adapter from throttling and setting bittorrent to bulk. Applications & Gaming -> QoS:

I then downloaded an Ubuntu ISO image, getting 200-300KBps down; a bit slower than normal, but for me, still a totally acceptable transfer rate. The voice quality was still good.

So everything looked good, but I was still interested in testing the quality and usability for a bit before committing to port our phone number from Qwest to Vonage. If you port at signup it’s free, but costs an extra $10 to do later, but I figured the testing was worth it.

I now have the cordless phone system on the VOIP, and put an answering machine on the old number’s line. So incoming calls are inconvenient, but outgoing calls would most often be over VOIP.

We lived with that for a couple days, and everything was just fine, so I kicked off the number porting process. The number port went right through, and it’s scheduled, but I have no idea why there’s a 10 day lead time for the switch even after all the approvals have gone through. I’m guessing that’s Qwest’s fault.

My new Pike’s steerer tube has a cap in the bottom with a threaded hole the same size as a standard headset bolt (M6). Instead of pounding a star nut into the steerer, it’s exteremely easy use this cap as the bottom of a headset locker like the Azonic Headlock or USE Safe-T.

Required parts (in addition to headset, stem, spacers, etc.):
M6 threaded rod (shortest I could buy was 3 feet)
M6 coupler nut
Total cost about $5.

Insert the rod to measure the correct length. I threaded the rod through the bottom of the steerer tube until half an inch or so was protruding out the top of the tube. This took some time to thread the 10″ of rod through; I could have done an initial cut to an estimation of length to save myself some time, I suppose.

Add coupler nut, headset cap and original stem bolt. Thread the nut on the rod first, headset cap on original stem bolt, and then stem bolt into the coupler nut as well.

Tighten down the headset cap so there is no play or extra space in the steerer stack. Mark the point where the rod emerges from the bottom of the steerer tube. Take it all apart again, and cut the rod at the mark with a hacksaw.

Put it all together with the headset cap back on original bolt, coupler to original bolt, then threaded rod into coupler. You may want to use some removable threadlocker on the coupler to make it behave more like a single bolt when it’s in the steerer. Tighten the original stem bolt tightly into the coupler; it’ll butt up against the threaded rod.

Drop the whole thing into the steerer tube, thread it into the bottom cap, then tighten the bolt to the torque specs of your headset.