jra’s thoughts

I remember sitting in my databases class years ago and thinking, “This can’t possibly be the right way to store data.” It was a strange class, because it mixed theory and practice in a way that was anathema to the way I think. The theory part bored me to tears, and seemed ludicrously useless (first normal form? third normal form? who cares!). The practice part seemed ridiculously complicated and pointless. Why go to all that effort to write something down? Why not just write it? Yes, yes, I know, ACID and all that. I grudgingly learned it all, and I passed the class. But it never felt right to me. It felt like the students were putting on a performance to satisfy the teacher. It didn’t feel like Theory of Computing (i.e. Turing machines, regular expressions, the halting problem, etc), which just Felt Right.

So, you can imagine that it warms my heart immensely to read about NoSQL, a growing backlash against SQL/ACID data management.

What’s especially heart-warming about the backlash is that it’s coming from honest to god working computer scientists, not the academics, and not some bussinessmen who are trying to move SQL aside for their own profit. This is simply the next evolution of how to think about storing data: there are ways to process data where ACID is not a useful feature, and you need to have tools in your toolbox that respect that reality.

PS: A particularly philisophical reader might notice that I’m commenting here on the fundamental difference between the mathematics of computers and software engineering. Where the mathematicians lead us, the path (though steep and rocky) is usually an interesting trip. Where the software engineers lead us sometimes takes us into the malarial swamps of SQL. And the artisans of software, those who rightly reject the clunky boots and balky toolbelts that the software engineers wear? They put the sweet-smelling flowers into the meadows that the mathematical path takes us meandering by….

A friend asked me a question that reminded me of some great resources I want to mention here (in case I ever need to find them again…)

They are:

• How to Accelerate Your Internet
• Wireless Networking in the Developing World
• Wireless U

In my response to my friend, I also touched on something interesting I learned from Cricket, and from years as a consultant. Here’s the big lesson: Infrastructure projects don’t sell themselves.

Why? Because they are not sexy. Their payback is hidden down in the noise of day to day operations and (apparently) fixed costs. Once you get used to “we have to have 256 kbit/sec VSATs everywhere, and the WAN is too slow to do file sharing”, it appears to be the unchanging reality, the fixed costs that people just suck it up and pay. But if you measure a problem and show before and after pictures, then you can market the solution. It’s no secret that the farther up the management ladder you go the harder it is to explain what’s broken, why, and why it will cost $X to fix it. The charitable reasoning for why this is is that “those people are busy, they are big picture people”. The less charitable explanation is that the Peter Principle moves incompetent people up out of day-to-day work into management, where they need to be pandered to like children in kindergarten: “Ooooh, look at the pretty colors! OK, now sign here.”

We used to call it “the manager test”. Cricket passed the manager test because even a manager could understand Cricket’s output and agree to fund changes necessary to improve link utilization. At Microsoft, I can even remember one time that Cricket even managed to pass “the executive test”, when a link upgrade was going to be so expensive that a VP had to sign for it, and he only agreed to spend the money after seeing the graphs with his own eyes.

Pictures work. Everything else is a waste of time.

Here’s something else I wrote to my friend that I’d like to share:

It is the “measuring and marketing” part you need right now… Improving your bandwidth situation needs to be a stealth project hidden under the covers of some other project.

This is how you raise the profile of and fund infrastructure projects, at least in disfunctional environments where the hamsters in the wheelhouse need and want to be pandered to by bright colors and tender morsels of “synergy”.

PS: I’m keeping my friend anonymous, since we wouldn’t want his bosses to find this, on the off chance they might understand that the last sentence is talking about them.

I got interested in applying my new XSLT wizardry to the task of printing an entire blog. Like making every post into a big PDF and sending it off to a print-on-demand service. Digital backups = bad. Paper backups = good.

I thought it would be easy, just do “Wordpress export”, then write the XSLT to turn the export XML file into FO, and process into PDF. Well, it is easy, in principle. But in practice it’s not.

The first problem is that posts have HTML in them. Not a lot. It’s almost entirely in-line stuff, like bold and italics. But it’s there, and you have to format it. Which means that you have to get it into the XML document tree so that XSLT can walk the tree and convert the HTML into FO one tag at a time. (By the way, a great resource on how to do the HTML to FO is at IBM developerWorks.)

But the problem is that Wordpress exports blog entries using the <content:encoded> tag, with the stuff inside that tag is a CDATA string. Which means that your XSLT processor gets the chunk of HTML as a string, and not as nodes in the XML tree, which means you can’t process it. But if you pull off the CDATA and send the Wordpress version of the blog posts directly into your XML parser, bad things happen. It’s not valid XHTML, so you can’t process it. Lots and lots of error messages, like 20000 for my blog.

And like that, like a flash of lightening, I finally understood what all the fuss is about XHTML, etc. Data has to start structured so that if you want to process it later, you won’t have to clean it up. Duh.

The first step is to hack your Wordpress so that export does not hide the XHTML inside of a CDATA. Here’s the patch:

*** wp-admin/includes/export.php-       2009-06-16 02:22:19.000000000 -0700
--- wp-admin/includes/export.php        2009-06-17 03:35:16.000000000 -0700
***************
*** 14,24 ****
  header("Content-Disposition: attachment; filename=$filename");
  header('Content-Type: text/xml; charset=' . get_option('blog_charset'), true);

! $where = '';
! if ( $author and $author != 'all' ) {
!       $author_id = (int) $author;
!       $where = $wpdb->prepare(" WHERE post_author = %d ", $author_id);
! }

  // grab a snapshot of post IDs, just in case it changes during the export
  $post_ids = $wpdb->get_col("SELECT ID FROM $wpdb->posts $where ORDER BY post_date_gmt ASC");
--- 14,25 ----
  header("Content-Disposition: attachment; filename=$filename");
  header('Content-Type: text/xml; charset=' . get_option('blog_charset'), true);

! $where = $wpdb->prepare("WHERE post_type != 'revision'");
!
! #if ( $author and $author != 'all' ) {
! #     $author_id = (int) $author;
! #     $where = $wpdb->prepare("WHERE post_author = %d ", $author_id);
! #}

  // grab a snapshot of post IDs, just in case it changes during the export
  $post_ids = $wpdb->get_col("SELECT ID FROM $wpdb->posts $where ORDER BY post_date_gmt ASC");
***************
*** 160,165 ****
--- 161,167 ----
  <?php the_generator('export');?>
  <rss version="2.0"
        xmlns:content="http://purl.org/rss/1.0/modules/content/"
+       xmlns:excerpt="http://purl.org/rss/1.0/modules/excerpt/"
        xmlns:wfw="http://wellformedweb.org/CommentAPI/"
        xmlns:dc="http://purl.org/dc/elements/1.1/"
        xmlns:wp="http://wordpress.org/export/<?php echo WXR_VERSION; ?>/"
***************
*** 200,206 ****

  <guid isPermaLink="false"><?php the_guid(); ?></guid>
  <description></description>
! <content:encoded><?php echo wxr_cdata( apply_filters('the_content_export', $post->post_content) ); ?></content:encoded>
  <excerpt:encoded><?php echo wxr_cdata( apply_filters('the_excerpt_export', $post->post_excerpt) ); ?></excerpt:encoded>
  <wp:post_id><?php echo $post->ID; ?></wp:post_id>
  <wp:post_date><?php echo $post->post_date; ?></wp:post_date>
--- 202,208 ----

  <guid isPermaLink="false"><?php the_guid(); ?></guid>
  <description></description>
! <content:encoded><?php echo apply_filters('the_content_export', $post->post_content); ?></content:encoded>
  <excerpt:encoded><?php echo wxr_cdata( apply_filters('the_excerpt_export', $post->post_excerpt) ); ?></excerpt:encoded>
  <wp:post_id><?php echo $post->ID; ?></wp:post_id>
  <wp:post_date><?php echo $post->post_date; ?></wp:post_date>

Wow, lots of changes. Why? Here’s the rundown:

1. You have to filter out all the auto-saves that Wordpress maintains in the database. I was lazy and broke the author filtering when I made my hack. YMMV.
2. You have to add some namespaces so that your XML parser does not flip out.
3. You have to remove the CDATA thingie.

Even once you have the XHTML outside the CDATA, there’s two more layers of problems, actually. First is that Wordpress has a set of filters it uses to turn the simple input it allows into HTML, by doing stuff like adding the <p>’s for you, etc. So you need to get that filter to trigger during your export. To do that, you use the following plugin:

<?php
/*
Plugin Name: XHTML Export
Version: 1.0
Plugin URI: http://nella.org
Description: Insures output is XHTML
Author: Jeff R. Allen
Author URI: http://nella.org
Disclaimer: This Plugin comes with no warranty, expressed or implied
*/

function xhtml_export( $content )
{
  $content = apply_filters('the_content', $content);
  // there's another way to do this: see start of the stylesheet from IBM developerWorks
  $content = str_replace('&nbsp;', '&#160;', $content);
  return $content;
}

add_filter( 'the_excerpt_export', 'xhtml_export' );
add_filter( 'the_content_export', 'xhtml_export' );
?>

Put that into your wp-content/plugins directory, then go to your plugins page and activate it.

The output of the Wordpress filtering step is XHTML, so if you never ever resort to the HTML tab to hack at the HTML yourself, you’d be OK. But I do that a lot, and I also imported posts from Movable Type, which wasn’t as XHTML-aware back in the day. So I still had a massive mess on my hands. Lots and lots of XML errors.

Right about this time, in flies HTML Tidy to save the day. It can turn HTML into XHTML pretty reliably. And the Tidy-up plugin for Wordpress arranges for Tidy to do its work on each of your posts in turn and makes it super-easy to edit the posts and fix the problems.

So that’s how far I have gotten so far. I have a tidy blog, I have a big file called blog.xml which is valid XML, including all the XHTML children nodes of the <content:encoded> node. Now I am hacking away on the XSLT to turn the blog into FO to make a beautiful book. As I’m not a designer, that’s the part where things fall apart for me. But I’m hoping to find a nice design template from Lulu or someone and follow that.

Also… Tidy is also useful if you are trying to debug your FO output. Use a Makefile like this so that the error messages you get from FOP will have line numbers you can go look at instead of character offsets into the one long line that your XSLT processor emits. Tidy stupidly defaults to non-UTF8, even when you tell it it is reading an XML file, and even when the XML file has the right encoding named. So be sure to use the -utf8 flag. And don’t forget the -q flag, which shuts up the stupid advertisement the Tidy developers saw fit to put in their software. (Hello, W3C people? Yeah, we don’t care. Shut up. Thanks. okloveyabye.)

all: view

view: blog.pdf
  gnome-open $<

blog.fo: blog.xml rss-to-fop.xslt
  xsltproc --output $@ rss-to-fop.xslt $<

blog.fob: blog.fo
  tidy -utf8 -q -i -xml -o $@ $<

blog.pdf: blog.fob
  fop -fo $< $@

clean:
  rm -f blog.fob blog.fo blog.pdf

I am a little bit obsessive about checking out the document properties in PDF files I read. I can’t explain why, but there you have it.

I was sad when I noticed the PDF file being emitted by my XML has no document properties. So I figured, no problem, I can just go find the right FO tags, grep for them in the DocBook XSL and reverse engineer what stuff I need to put into my DocBook to get them set right. I figured it would be something obscure like, <author> or something. In fact, DocBook already knows who the author is, in order to format the title page nicely, so no dice there.

What I found is this page on the Apache FOP faq that explains that FOP can’t do it. WTF? This can’t be that hard, and it really seems like a natural thing that would make it into version 1.0. (Of course, FOP is version 0.94, which might explain something as well.) In their defense, it seems like this is also braindeadness in the FO spec, beacuse I found a commercial implementation of FO that says you have to resort to special extension namespaces to specify metadata in their implementation. But Apache FOP already has the “fox” namespace for this purpose, so no big deal.

The solution is to write your own Java program (real user friendly there, guys) that uses a nifty PDF library called iText to add on the metadata in a post processing step.

In case someone else needs it, here’s what I came up with:

/* based on example here:
   http://itextdocs.lowagie.com/tutorial/general/copystamp/index.php
*/

import java.io.FileOutputStream;
import java.util.HashMap;

import com.lowagie.text.pdf.PdfReader;
import com.lowagie.text.pdf.PdfStamper;

public class Stamp {
    public static void main(String[] args) {
        try {
            if (args.length != 6) {
                System.out.println("Arguments expected:");
                System.out.println("  pdf-in pdf-out title subject author keywords");
            } else {
                // we create a reader for a certain document
                PdfReader reader = new PdfReader(args[0]);

                // we create a stamper that will copy the document to a new file
                PdfStamper stamp =
                    new PdfStamper(reader,
                                   new FileOutputStream(args[1]));

                // adding the metadata
                HashMap moreInfo = new HashMap();
                moreInfo.put("Title", args[2]);
                moreInfo.put("Subject", args[3]);
                moreInfo.put("Author", args[4]);
                moreInfo.put("Keywords", args[5]);
                stamp.setMoreInfo(moreInfo);

                // closing PdfStamper will generate the new PDF file
                stamp.close();
            }
        }
        catch (Exception de) {
            de.printStackTrace();
        }
    }
}

Here’s an interesting story, well told, about an industrial process that takes refinery waste from the United States (derived from high-sulfur Mexican crude oil), cycles it through Europe, then dumps the result in West Africa.

The company running this racket (or “innovative commodity exchange”, as they call it) is Trafigura.

Learn more here:

• Coker gasoline
• A small pawn in the game
• Vast Tank sweetens gasoline
• BBC Newsnight on Trafigura’s dumping

Here’s a quick summary:

• An arbitrage opportunity exists for energy traders based on differing regulatory frameworks in rich countries and poor ones.
• A chemical process can turn a waste product in one jurisdiction into two outputs, gasoline usable in a lenient jurisdiction (West Africa), and the waste extracted from the original product.
• If you buy one ship of high quality gas, you can dissolve the waste stream from several other tankers of coke gasoline into it, meaning that you can dispose of the waste stream by getting your customers to burn it for you in their cars.
• A clever and immoral company can take advantage to squeeze profits where others just saw costs. The profits come from the externalities of burning high-sulfur gasoline (decreased longevity due to sulfur-rich smog)
• None of this is precisely against the law. Tanfigura and it’s contractors made minor infractions here and there, playing fast and loose with the rules. But what they are doing is fundamentally not illegal — though it should be.
• Trafigura was working towards, or achieved, the ability to reprocess this stuff at sea, likely to further reduce the power of regulators over their work.

How much other stuff like this is going on? Who are the people that organize and operate this kind of thing? How do they sleep at night?