We deal with fairly large amounts of webserver log data, so are also saving HDFS space and job i/o by using the hadoop-lzo package. It gives fast compression that retains our ability to use the data through Hive queries.

If you are only interested in compression, and have Hadoop and Hive configured appropriately, you can even mix compressed and uncompressed data in separate partitions of a Hive table.  A normal table definition will work:

CREATE EXTERNAL TABLE foo (
                       columnA string,
                       columnB string )
       PARTITIONED BY (date string)
       ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
       LOCATION '/path/to/hive/tables/foo';

One big advantage of LZO, though, is its ability to be split in map/reduce jobs. This is done by creating an index of the LZO file with the LzoIndexer tool of the hadoop-lzo project. To actually use the index, you will need to use a special input format for your Hive table:

CREATE EXTERNAL TABLE foo (
         columnA string,
         columnB string )
    PARTITIONED BY (date string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
    STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
          OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
    LOCATION '/path/to/hive/tables/foo';

Now to actually come to the point. In my case, I had already created the table, and was trying to add indexing after the fact. Hive permits changing input format with an alter statement:

ALTER TABLE foo
    SET FILEFORMAT
        INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
        OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";

But this alters only future partitions, not existing partitions. They retain their TextInputFormat. So now when I ran my Hive queries, instead of the LZO index file being used for splitting the input, it wound wind up used as table data. My results were mostly correct, but there were some result rows that were garbage.

I fixed this by dropping and recreating the table and partitions with the correct input format. Because I use EXTERNAL tables, the data itself was preserved.

While this is not a big deal, I have lost the ability to mix compressed and uncompressed data in the table. The Hive language manual claims I can alter partition metadata, which would be another way to deal with this, but so far I’ve not been able to make that work in versions 0.5 and 0.6.

Thanks to Dmitriy and Johan from Twitter for helping me understand all this.

hadoop-lzo:

https://github.com/kevinweil/hadoop-lzo

The original hadoop-gpl-compression project:
http://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/wiki/FAQ?redir=1

Hive language manual:

http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL

Discussion of Hive and table attributes:
https://issues.apache.org/jira/browse/HIVE-957

Of course, when speaking of MTB or Moab (or crafting a post for search results), I also need to be sure to link to my buddy Brian’s site SingleTrackRides.

So my reaction? Uh oh; I guess I’d better do something more here than just post ridiculous YouTube videos and random text.

We tried ‘em out yesterday. Just a little bit too sweet for my taste, but it’s hard to argue with a drink named after you.

We’d like to call it the Lijito, but we’re not sure if that’s something derogatory in Portugues or something, so we might just wind up calling it the Lijitini.

I thought, “sldfkjdlkjd wwerrrjrjs ttlljjjgllkkttj sldfe sdfw?” But naturally, fjwjejrb jboegj and wllejrbbbjelti3! So instead I wnne85gnns dlfkjwrng yot!

Shortly thereafter, wnern6xioa hgqqp38ng alasjtq8p6 sgklgfnqp lanffsan ouq dlfn sda gqpq dnfgga. But still, there is the matter of akl e7a z fngfn lqn fqi ore.

“IuthlzJKSf hnaa jasdj tjla5!” I said. “A hafaljsda nfqo ietyalf hld sh q5a.”

No, that will never work. Aj awfahqeol q89 ahlfqp3984 bdfalkf.

Get the new one here:

Google Home is just a collection of “Gadgets”, and it turns out it’s pretty easy to write one. So I converted the Lijit Wijit into a Gadget; the Lijit Wijit Gadget.

So if you use Google Home and want to add MrBalky search to your page, just .

It’s really just a test. Look for a better Lijit Wijit Gadget, coming soon from your good friends at Lijit Networks.

Would you like to know more? Read what Todd Vernon and Stan James have to say about PNS.

1. Go to your lijit page (mine is http://www.lijit.com/users/MrBalky)
2. Click “Customize your search” to enter your Google account info
3. Back on your lijit page, click “Get this wijit” in the search box
4. Copy the javascript wijit code.
5. Head to your WordPress admin pages -> Presentation -> Theme Editor.
6. Click “searchform.php” on the lower-right.
7. Add the wijit code. (Since there’s nothing to search on this blog, I just removed the existing search code.)
8. Click “update file>>”, and you’re done.

It’s a snap.