The WGBH.org Development Blog

Much like after the holiday season, I’m mired in the post-Drupalcon blues. I feel tired, lethargic, gassy - no, wait, I’m sorry, that’s probably just from the Chinese food I just ate.

Last week at Drupalcon they held a case-study on the Popular Science web site which, don’t you know, was just rebuilt and relaunched using Drupal. I didn’t attend the session but one of our developers, Josh, sure as heckfire did. He was quite impressed with the job they did.

Luckily, the case-study has now been published on the Drupal web site. It’s very interesting, indeed, and details all of the fine work done by the folks at pingVision to make the site a reality.

One aspect that caught my eye was the way they made use of ImageField module for image uploads, and the Drupal Markup Engine (which I’d never heard of) to allow site editors lots of flexibility in content placement on node pages. I will have to give that a gander.

We had considered using the ImageField module for adding images to content types. However, we rejected that idea because we want our editors to have the ability to add images to content nodes (e.g. articles, TV programs, etc.) - or find existing images for reuse - in the simplest way possible. Creating nodes with image fields and relating them to other nodes via node reference doesn’t seem like the best way to do this, as it forces you to select from a drop-down list to choose an existing image. Unless, of course, I’m missing something (which is always possible).

Instead, we’re planning on making use of the Image Assist module, which offers a way to search for and choose existing images or upload new ones and add them inline in the least painful way. I have not yet seen a good case study of a Drupal site that puts much emphasis on being able to find and reuse images when there are potentially lots of images already in the system.

Needless to say, if any of you jokers out there have any insight or suggestions here, please, for the love of Pete, speak up.

Oh yeah, and it looks like we’re going to plow ahead and make the leap to Drupal 6 now, rather than build on Drupal 5 and upgrade after. More on that later…

So while Phil has been blogging up a storm these past few weeks, I’ve been officially slacking in my devblog responsibilities. Truthfully, I’ve been doing things that certainly quality as blog-worthy, but between conferences (definitely blog-worthy), coding (possibly blog-worthy), and catching a pretty bad cold last week (definitely NOT blog-worthy), I’ve been either too busy or too full of Sudafed to write a decent post for the site.

Now that I’m back in the office I can relax (ha!) and translate my reams of notes into some coherent information to share on this blog.

Drupalcon was a great conference, and I think we all got a lot out of it. There were a few key sessions that I’ll probably write about in more depth, but here are some of my big takeaways form the conference:

Drupal 6 is Awesome

I admit that I wasn’t really paying attention to Drupal 6 until beta 3 was released. It caught me off guard, because I saw so many great features in the new version that I really can’t wait to start using it.

Actions and Triggers

The possibilities here are endless, especially for any web site that has a publishing process that is any more complex than simply writing a blog post and pushing submit. Actions are events that can be fired off (such as sending an email) by a trigger event (such as saving a new post). A more complex series of triggers and actions could be the backbone for a great publish/review/moderate workflow.

Theming

My prediction: 2008 is the year when Drupaling theming is really going to shine. In the past there have been a handful of Drupal sites that really stand out from a design perspective, but time and time again, I have heard people say “…but I don’t want it to look like a Drupal site.”

The improvements to the phptemplate theme engine in Drupal 6, combined with tools like Firebug (not a Drupal-specific tool, but awesome nonetheless) and the Theme Developer (included as part of the Devel module. which Phil mentioned in a previous post), are going to give themers a lot more flexibility. In fact, it was interesting to hear how often “theming” was referred to at Drupalcon, as a separate process beyond the design and development. Theming is being recognized as the complex glue that ties together the data and the presentation layers.

Performance

There’s a lot going on under the hood of Drupal 6 that I am not clued into, but the overall consensus is that Drupal 6 is fast. Javascript aggregation will speed up page loads in the same manner that CSS aggregation did in Drupal 5.x. The menu system, which has always been a complex and database heavy system, has been completely rewritten. And there is a lot of information out there about performance tuning for Drupal, database tuning, and new caching options.

Development Tips and Tricks

With so many seasoned Drupal developers in one place, it was impossible not to be exposed to new and helpful development tricks. Some key examples:

• Testing code in a separate PHP file.
  The first tow lines of index.php perform a full Drupal bootstrap. You can copy that code into a new PHP file (in the root Drupal directory) add additional testing code to that page, and test the page directly in your browser. This makes it really simple to run basic tests and write the output directly to the browser.
• Install the API module locally.
  I use api.drupal.org pretty regularly - it’s a great reference tool for finding functions and documentation. But only Drupal core is indexed on that site, no contrib modules. You can install the API module and use it on your own install to view not only Drupal core AND contrib modules that you use, but also your own custom modules.
• Finding new modules
  With so many contributed modules out there, it’s hard to know where to start when you need a specific functionality. One suggestion is to subscribe to individual RSS feeds for CVS commits, project nodes, and Planet Drupal to monitor activity on different projects. With a good feed reader, you can bookmark away those interesting projects for the day you might need them.
• Coder module
  The Coder module is a code analysis tool that is installed just like any other module. It performs a handful of functions, including:
• code style review - a rule-based engine that compares code against the established Drupal coding standards — checking for proper syntax, string concatenation, use of globals, positioning of curly brackets.
• comment review - checks for proper spacing and indentation, and for CVS $id$ tags
• SQL review - checks for proper syntax and for curly braces around table names
• upgrade review - this will flag code that may need to be modified for compatibility with the next version of Drupal, such as forms API, scheme API, menu, and .info file changes.
• security and performance review - It checks for SQL injection and some other security concerns, but this portion of Coder is rather limited in it’s current abilities.

Coder can analyze all the code on your site, or just selected modules. It does not make changes to the code for you, but it will provide line numbers and descriptions of the problematic code. Seems like this could be a very handy tool, especially when building a site with other developers.

The Future of Drupal

There is already a lot of talk about Drupal 7, and there are some really cool ideas being kicked around. Which is a little disconcerting when you are trying to build sites with Drupal, knowing how fast it is moving. But the forward momentum is good - this conference did a lot to assure me that Drupal is has a really strong future and is not going to end up in the CMS graveyard.

Drupalcon Day 4 - the last day of the con (boy, I hate that term) - was, for me, the least useful, the least interesting, the one most devoid of Drupal knowledge and tidbits.

Why? Because I wasn’t there.

This head cold my kids gave me earlier in the week blossomed overnight like tulip in spring, so I stayed home close to my tissues and decongestant.

There were several sessions that looked interesting, so I was sorry to miss them. The good news is my WGBH colleagues Pete and Josh were there and I’m sure they took some good notes. One or both of them will post their highlights here in the coming days…

If you were there and would like to add your highlights, feel free to add them in the comments.

So, there you have it! Tomorrow, it’s back to actual application building.

What can beat the excitement of Drupalcon Days 1 and 2 - why, Drupalcon Day 3, of course!

I’ll spare you the coffee update today. Safe to say I was well lubed (caffeine-wise) all day. However, I do have a nice head cold now, so the mucous was flowing and I was burning through tissues at a healthy clip. Let this be a warning to all you young folks: if you’re thinking about having children someday, be aware that they’re walking germ factories and you will spend November through March sick as a dog. So, now you’ve been warned.

The first session I attended on Day 3 was SimpleTest: Because clicking on forms is for suckers. I’m certainly familiar with unit testing and had previously heard of the SimpleTest module, which is based on (and requires) the SimpleTest PHP library. But this session gave me a nicer rundown on all that SimpleTest offers, why it’s important to make use of it, and a primer on how to get started with it.

The panelists spoke of the importance of unit testing and recommended (ideally) a unit test for each if statement in your source code. They also warned against running SimpleTest on a production database (don’t do it), and mentioned the limitations of the module, namely that it doesn’t work for testing JavaScript or CSS (i.e. the visual interface).

SimpleTest is not only good for unit testing but also has an internal web browser that can be used for functionality testing (i.e. to test functionality end users will experience through a browser).

They also mentioned the new (as of Drupal 6) SimpleTest Automator module which really simplifies functionality testing by providing (essentially) a macro builder to record user actions (e.g. filling out forms, clicking on links, etc.).

SimpleTest can also be run via the command line using drush

Official Drupal SimpleTest documentation can be found here.

The panel also mentioned that unit testing is going to be added to core in Drupal 7; functional testing will remain a contributed module. The runkit php extension may be required to support all of this.

The keynote session of the day was given by Brian Aker from MySQL, speaking about, well, MySQL and what the fine ladies and gentlemen there are up to. I can sum it up in one word: scalability. Nuff said.

We broke up the day by having lunch with some new friends that we met here at Drupalcon. One was Margaret Rosas of Quiddities, who is currently working with KUSP, a public radio station in California. We also met two nice people from WXXI, a public broadcaster in Rochester, NY: Andrew Wheeland and Janele Robinson

After lunch I checked out the birds of a feather session on Solr led by Robert Douglass. This is a new approach to search on a Drupal site based on the Apache Solr search platform. Unfortunately, I came late to this session and missed a lot of the important stuff. However, having worked with Robert last year, he previously gave us a quick backgrounder on it. Solr offers the ability to have faceted search and performance improvements over core search. It also allows for indexing multiple sites or content not in your Drupal database using the Nutch crawler. Solr also supports use of synonym lists.

It looks pretty handy.

You can see Solr in action by using the search on Rob’s own site. He’s also set up a site that lets you use Solr to seach the Drupal.org site.

Rob has written a Drupal module to integrate Solr called Apache Solr. Note that this is different from the Solr Drupal module, which is an alternative implementation.

Rob also noted that one problem with current core search is that you can’t turn off content and user searching (you can only hide it). Rob has his own module, called Core Searches, which replicates the core content and user search functionality but allows you turn them off, which you may want to do if you run Solr.

Solr is also highly scalable since it is clusterable, meaning you can run as many Solr servers as you want and put a load balancer in front of it all.

Rob also noted that Netflix and Digg both run Solr. Them’s good company.

After the mid-afternoon break and deftly avoiding being included in the Drupalcon group photo (because, good lord, I don’t need photographic evidence that I ever attended a “con” of any sort) I attended my final session of the day: Creating custom workflows for Drupal applications - taking advantage of core hooks and context. The title was a bit deceiving, as it was really more about context and how it can be more efficiently identified, passed along and used by Drupal to make decisions about display or workflow.

The panelists pointed to their site Managing News as an example of how they implemented some nice context logic.

Basically, they used the same approach that the Drupal Internationalization module uses to set the display language for a site. The key here is the function custom_url_rewrite() (renamed to custom_url_rewrite_outbound() in Drupal 6). This function actually lives in settings.php, so it gets executed before Drupal starts doing it’s work on a page request.

The idea here is to prepend the (relative) URL gets with an identifier (e.g. http://www.domain.com/en/node/12345 for English sites). When a link is clicked, this information gets read and stripped out of the URLs using the custom URL function and translated to a normal Drupal URL, hence the context is set and can be used by block/menu/theme logic when the page is generated. The nice thing here is that the context identifier is removed before Drupal starts generating the page, so it doesn’t break pages that depend on URL arguments (e.g. views, panels).

The panelists used this approach to identify and set the context on page load. This is done with a custom function that sets a static variable. Another custom function can then be called by block/menu/theme logic in a simple and clean way to affect the display. This can have nice implications for workflow by allowing you to add links/tabs to pages to give users the ability to perform and action based on context. For example, on a blog entry page, display a link to add a new blog entry. Nice.

My head swimming with another day full of Drupal knowledge I called it a day and went home.

Tomorrow, the grand finale: Day 4!

Drupalcon Day 2 started off better than Day 1. Why? Because I decided to skip the hell out of the free coffee and just buy my own at Dunkin’ Donuts. Sometimes free just ain’t worth the cost.

The morning started well when I ran into Robert Douglass, formerly of Lullabot, now of Acquia. Late last year Robert worked with us when we hired Lullabot for some consulting on our rebuild. We had only worked with Robert over the phone and via email, so it was nice to meet him in person. In addition to being a Drupal genius, he’s also a very nice guy. More on Robert in the coming days…

Today Pete and I were joined by another WGBH developer and our buddy, Josh Boughey. Josh works in the membership department and will be working with us to rebuild WGBH.org. Hopefully we’ll get Josh to blog here about his involvement at some point.

Other than that, it was more of the same. Lots of people with facial hair, lots of laptops, lots of Drupal talk, of varying degrees of usefulness. Here’s a recap:

The first session I hit was Best practices in development environments, staging, build management,
and production environments. This was pretty good, though mostly high level. There were some interesting things to note.

One of the panelists was Narayan Newton, the DBA for OSU Open Source Lab; they’re the ones who host Drupal.org. He gave us a rundown of their setup: four load balanced web servers, two database servers (one active, one passive, using replication).

Narayan told us about how they’ve made use of this database replication patch, which allows them to redirect all read queries to the passive database, thereby improving performance. He says that the new database layer for Drupal will support this functionality by default.

One person in the audience got up and spoke about SQL Relay and MySQL Proxy, both of which offer the ability to load balance requests to your MySQL database servers. This is all worth checking out.

The panelists then talked about leveraging source control. One panelist said he didn’t see the need for branching if you’re just developing for yourself (which is my opinion, also). A common code control approach is to use CVS to manage the core and contributed Drupal code, and import that all (including the CVS subdirectories) into SVN. This is the approach we plan to use for WGBH.org. The panelists differed over whether to deploy from staging to live using SVN or rsync.

They then talked about automated deployments. The Open Source Lab folks use something called Cfengine to manage the system configurations of all of their servers with a small staff.

The discussion then veered into the topic of code vs. content migration. The upshot here was that nobody seems to have a good solution to the problem of migrating database changes from staging to live environments (particularly, settings and content in the db that needs to be migrated). Again, though, the audience stepped up and somebody recommended the Publish and Subscribe modules to push from staging to live. Somebody else suggested Drupal Migraine (yes, like the headache), which is a Python-based (the scripting language, not the snake) tool for doing this sort of thing. We’ll have to check these all out.

Things wrapped up with a discussion of QA best practices. One of the attendees got up and mentioned a tool called Watir, a Ruby-based (the language, not the precious gem) tool which allows for automated cross-browser testing. It works on Windows IE, but there are also Firefox and Safari versions. The crowd was duly impressed.

The big session of the morning was by Chris DiBona of Google, who gave an entertaining and interesting speech about open source and why Google supports it. I believe that Chris will be doing two shows nightly throughout the week. Try the fish!

Following a lunch of a burrito con carnitas, I attended one of the birds of a feather sessions, about the problems with upgrading staging environments to production systems. People kicked around potential solutions for pushing configuration and content changes that are stored in the database from the staging to the live database.

The most common solution involved writing any number of scripts to dump DML and content from the staging database and import them into the live database. This approach makes some people uncomfortable as it sidesteps the usual Drupal APIs (either using the GUI or programming it on the back end).

Another solution is to enable binary logging for MySQL. Binary logging causes all SQL statements that updated (or could have updated) data (e.g. no select or show statements) as executable commands. You can then extract those commands (filtering in/out the ones you want/don’t want) using MySQL tools and writing the commands out to a text file of SQL commands that can then be executed against the live database.

The main problem with both of these approaches is the potential for overlapping ID numbers. Some people get around this by allocating the first X number of auto increment IDs (like the first 1,000 or 10,000) for use on the development server only. That sounds clunky to me and I’d rather not do that.

After a break for coffee and a large chocolate chip cookie, all of us GBHers headed into the Theming with the Theme Developer module session, led by Moshe Weitzman, who wrote the new (as of Drupal 6) theme developer module.

This module looks mighty slick.

It’s part of the Devel module distribution. The big thing here is that there is now support for sub-themes, meaning that you can base a theme on an existing theme and simply override the parts of the parent theme that you want to change without having to replicate all of the theme code. Wow-wee! Themes now have .info files, in which you can define a base or parent theme.

Another great thing about this module is that is makes use of Krumo (which you need to first download into the Devel directory). Krumo allows nice functionality like being able to highlight and click on page elements and see what file or function is controlling its style! It tells you how you can override the base theme’s styling. It will also show you all variable and parameters passed into the appropriate theming function and let’s you browse all of the data available to that theming function. It looks as sweet as that big-ass cookie I ate just before the session.

Bottom line is this new module looks like it’ll make theming much, much, much easier.

After that, I called it day and headed home to help put the kids to bed.

Today was Day 1 of Drupalcon 2008 here in Boston. Pete and I are attending, representing WGBH Online. It’s being held at the big, beautiful and publicly financed Boston Convention and Exposition Center. Your tax dollars (or, at least, mine) at work!

The location of the BCEC bought back memories for me, as it’s near one of my former places of employment. Visions of many past lunch destinations danced in my head.

Needless to say, the conference is nerd central which, sadly, includes me. Laptops, cell phones and XY chromosomes as far as the eye can see (organizers say attendees are 98% men).

Here now are the highlights of Day 1, as I saw them:

• Ran into an old friend - A former boss of mine (from here) who I hadn’t seen in a few years was there. We had a nice chance to catch up during lunch.
• Free coffee - I got one free cup before it ran out.
• Free long sleeve t-shirt -It’s got a dorky logo on it, but it’s a nice shade of navy and, hell, it’s free.
• Free lunch coupon - A $10 value!
• Some interesting Drupal info - More on this below

Now, the lowlights (again, just my opinion):

• Bad free coffee - Good thing it ran out or I would’ve drank more.
• Overpriced lunch - Does it count as overpriced if I didn’t have to pay for it? I guess not. But $10 for a weak turkey sandwich, chips, pickle (it was a good pickle) and a soda seems a bit much.
• Unreliable Wi-Fi - Couldn’t get a connection that worked most of the afternoon.

As for the actual conference, it started off a little slow, for me, but there were still some good nuggets of information. To wit (whatever that means):

The first session I attended was The Road from Newbie to Ninja: Drupal Tools and Techniques. This session covered a lot of basic Drupal developer information that I was already familiar with, but also some that I wasn’t't. I’ve already used the Devel module but mainly for gathering and displaying query information for each page. There’s a bunch of functionality that I haven’t tried yet, but will, like variable inspection and editing, clearing the cache, catching outgoing emails, and error backtracing.

I also learned more about the API and Coder modules, neither of which I’ve used. The former will generate documentation for core code, so that you can have a local copy (as opposed to relying on api.drupal.org) and can also generate documentation for contributed modules. The latter is a handy collection of developer tools, like a code reviewer and support for Drupal coding standards. I’ll be checking out both of these modules.

The keynote speech by Drupal founder Dries Buytaert was interesting, as he gave us a preview of the thinking behind Drupal 7, which is due to be released next year. Dries laid out his vision for Web 3.0 and the semantic web. The crowd seemed to eat it up. I, on the other hand, was preoccupied with thoughts of lunch.

The next session was Panels 2 and Nodequeue. Node Queue is a module that I hadn’t heard of before, but sounds potentially pretty useful, as it allows one to create queues or grouping of nodes. This can be used to allow content producers to more easily manage lists of feature stories or a lead story, that sort of thing. Sounds interesting! I was also inspired to get up to speed with the Panels module, which I’ve yet to really try out.

After finding a Starbucks in the adjacent hotel and loading up on caffeine, I finished the day by attending OpenID and Identity in Drupal: the future of user.module. Very interesting, also. OpenID is something else I really need to come up to speed on.

Tomorrow: Day 2! More information, more bad coffee, more excitement!

I’ve been up to my eyes, ears, nose and throat building out TV schedule related views. Things are moving along, I’ve got a nice working version of our Programs A-Z list and the daily schedules by channel. None of this involves any real theming work just yet; I’m focusing on functionality first. I’ll give more details on these pages next week.

Of more immediate concern, Drupalcon Boston is coming up next week and Pete and I will be attending, as will several others from WGBH. We’re looking forward to learning lots of Druaplish stuff, meeting folks and, of course, drinking free coffee. I mean, there is going to be free coffee, right?

Anyhow, there are lots of excellent sessions planned, across several different tracks. The full session list is here.

I’ll be spending most of time at the Site Building and Design & User Experience tracks. Here are some of the sessions that I’m planning to attend. If you see me there be sure to say hello.

• The Road from Newbie to Ninja: Drupal Tools and Techniques
• Performance tuning and optimization of high traffic Drupal sites
• Drupalcon Tutorial: Best practices in development environments, staging,
  build management, and production environments
• Theming with the Theme Developer module
• Drupal Security - Best Practices and Process Discussion
• Creating custom workflows for Drupal applications - taking advantage of core hooks and context
• Scalable Theming: Theming for 100s of node types, CCK fields and views

I hope to see you there! I also hope there are free donuts.

Pete and I will be attending Drupalcon right here in Boston, from March 3-6. It’s sure to be chock full of Drupaly goodness!

If you’re planning on going, be sure to leave a comment and let us know so maybe we can meet up!