Saturday, December 20, 2014

Condemned to Hack

As I've mentioned several times recently, Rhapsody recently introduced a new website design. This depends on Adobe's execrable Flash product for streaming music -- I'm not sure that is new but this is the first time I noticed a dependency. I've been running Rhapsody reliably on Ubuntu Linux, on a system which is up-to-date (14.04 LTS). The new website initially worked on this machine, but when I did a routine Ubuntu update it broke, giving me an error message that I must have Flash installed and enabled, and a URL to Adobe to "Get Flash." I spent many hours trying to figure this out, and probably made things worse along the way. Long story short, I finally got it working tonight. Still, the results are troublesome. Let me explain.

Flash (or Shockwave Flash) is proprietary (non-free) software developed and maintained by Adobe. It consists of an authoring product, which Adobe makes money on, and a player, which Adobe distributes without charge (but also without source code). Since only Adobe can compile the source code, they can choose which platforms they want to support. For a long time, they supported Linux, but in 2012 they decided to freeze Linux development at release 11.2. (They've since moved on to release 16.0 for Microsoft and Apple.) If you use Firefox go to Adobe's download website from a Linux machine, they offer you version 11.2.202.425 in various package formats. For Ubuntu you want "APT for Ubuntu 10.4+" -- Ubuntu, by the way, has since moved on to 14.04. When you click on the "Download" button, Firefox invokes the Ubuntu Software Manager to handle the package, which is identified as "adobe-flashplugin."

As I understand it, the "adobe-flashplugin" package doesn't actually include the Flash Player binary. What happens is that when you install the installer, it goes out to get the program(s) to be installed -- a bit of indirection which keeps Adobe's "crown jewels" separate from the software depositories which are used to install Linux systems. One problem here is that "adobe-flashplugin" winds up installing a slightly earlier Flash Player version (11.2.202.359) than the one advertised. That is most likely Adobe's bug. What makes this worse is that Firefox has been configured to automatically disable old versions of plugins that are believed to have security risks, and the version installed is one of those. I don't know whether the real latest version (.425) would be acceptable to Firefox. I do know that when Firefox offers a link to "Update" the offending plugin, it steers you back to Adobe's website, which gives you the wrong version again. I also know that it takes some twiddling to reinstall Adobe's "adobe-flashplugin" since Ubuntu's Software Center thinks it's already installed and up-to-date (you have to remove it then re-install it). Finally, you have to tell Firefox to allow the website to use Flash despite the security risks. (Hopefully, this is website specific, so you're not opening up a security hole for other websites.)

Now, all that's bad enough, but I had several other problems I had to figure out before I could get the above procedure to work. Linux people never have liked Flash -- even back when it was the only way to stream video and audio over the web, it was buggy, mysterious, and couldn't be fixed. So there have been many efforts to first emulate and eventually to supersede Flash. One hint I found was that Firefox was showing two Shockwave Flash plugins -- the 11.2.202.359 installed by Adobe (when I was expecting -.425), and another at 13.1.2.3 from some mysterious source. Firefox allows you to disable plugins but not to uninstall them, but I didn't get any different results from Rhapsody when I alternately disabled one or the other plugin. Finally, I took a look through the package list and uninstalled everything that looked like it had to do with Flash: namely, I removed flashplugin-installer, pepperflashplugin-nonfree and freshplayer-plugin, they verified that Firefox had no Flash plugins. Then I repeated the installation from Adobe, restarted Firefox, called up Rhapsody, and told Firefox to let me use the insecure Flash plugin. Finally, it worked.

No sooner than I got Rhapsody working again, I ran into another nasty bug. I haven't had time to comment on Francis Davis' 9th Annual Jazz Critics Poll, lately sponsored by NPR, because I've been preoccupied working on my piece of the project, which you can find here. I managed to get all the ballots counted and cross-checked by 4AM Thursday morning -- the schedule was to go live sometime Thursday but NPR didn't actually get their end together until Friday morning. However, I spent all of my time looking at my private copy of the website, and didn't notice that when I uploaded the code things broke. What happened was that any string with accented characters -- artist names like Miguel Zenón (11th) or album titles like David Virelles' Mbókó (14th) -- simply vanished. So I had to figure this out, and fix it.

Turns out that my working machine was running PHP 5.3 while the server is running PHP 5.4. One huge difference between the two is that in 5.4 the lords of PHP decided to make UTF-8 the default character set, replacing the default ISO-8859-1, which all of my data is encoded in. I've been a stickler about accents ever since college, when one of the jobs I had working on Paul Piccone's Telos was to go through the typeset galleys and use presstype to add the missing diacritical marks. When I later worked for typesetting equipment manufacturers, I specified the unified multilingual font package at Varityper, and I worked on a Japanese typesetter at Compugraphic. I later internationalized the prepress software package developed at Contex, and oversaw localization of the software for France. I saw aware of Unicode almost from the start, and I knew the guy at SCO who invented UTF-8. So in some sense I always understood that Unicode and its UTF-8 encoding would become the standard for character encoding, I found ISO-8859-1 sufficient for my own work, adopted it early, and have steadfastly stuck with it.

That's caused me increasing aggravation the last few years. I use emacs to edit my files, and it's long worked very nicely with ISO-8859-1, but it switched allegiance to UTF-8 a few years back, and that's caused me all sorts of problems. In fact, when I discovered this problem, the first thing I suspected was that emacs had saved the files using UTF-8. I've also seen MySQL move from ISO-8859-1 to UTF-8, but a simple configuration switch has allowed me to keep using ISO-8859-1 data for Robert Christgau's website. I spent hours looking for a similar configuration hack to keep PHP 5.4 from breaking not just the new code but lots of old code. While I found several candidates, I couldn't get any of them to work. Ultimately I fixed the problem by writing a wrapper for PHP's htmlentities() function, which when run under 5.4 would pass extra arguments to specify ISO-8859-1 encoding. That's not the limit of the changes, but it's the one function that I was using that was blowing up.

I've since gone back and applied this fix to the totals and ballots from 2011-13. I still need to look at 2009-10, but they are undoubtedly broken too. Updates are always a tough decision: they interrupt your regular work and often break things. As I said above, I have one Ubuntu machine that is up-to-date (the one that Rhapsody broke on), and another that is way out of date (the one with PHP 5.3). I've been meaning to upgrade the latter for some time -- mostly because Firefox has bugs handling Javascript, and those result in my browser crashing a couple times a week. (Hopefully a newer version will work better.) On the other hand, upgrading is going to be arduous. (It involves hopping through several Ubuntu releases, and any one of those hops could leave me broken, so I first need to back up all of my data -- and in this case there's a lot of that.) They I'll have to deal with software changes like PHP 5.4 (actually, more like PHP 5.6). Then I'll have the problem that I'll be ahead of the target servers for my websites. (That may be the point when I finally have to migrate tomhull.com to a new server.)

What was that line from The Godfather they liked to quote on The Sopranos? Something about trying to break out of the family business and go legit, then getting dragged back in. Looks like I'm still periodically condemned to hack.

Ask a question, or send a comment.