The WGBH.org Development Blog

For those of us in the northern hemisphere, fall has arrived! In between raking up and burning piles of leaves (and useless 401K statements), we here at WGBH Online have continued to fine tune our new(ish) TV Programs and Schedules pages.

As you may recall, not long after launch in August, we began to revisit the whole notion of how we’re tagging our TV programs and episodes. The main reason was to improve the way we generate lists of related programs, so as to suggest to visitors other shows they might like. Our initial approach was simple: just tag the programs (not individual episodes) and use a Drupal view to generate a list of up to three related programs.

But this soon proved restrictive. Sure, Frontline is a News and Public Affairs program, but individual shows in the series can be about different things (technology, politics, science). So, we wanted to be able to capture this more detailed level of information and use it to generate more useful lists of related programs for our visitors.

After much thought and discussion (not to mention headaches), we came up with an expanded tagging scheme and more sophisticated program matching logic, which has now been implemented on the site. Here’s what we did:

We renamed our existing TV Program Genre vocabulary to TV Program Primary Genres.

The terms remained the same (a small set of high level classifications) and these are still only applied at the program level.

We then added a new vocabulary that can be applied to both TV programs and episodes: TV Program/Episode Secondary Genres.

This secondary list has many more terms that now allow for a more sophisticated level of classification. tags applied at the program level apply to all episodes in a series. Tags applied at the episode level are only applicable to that particular episode.

Once we had that in place we then had to think about how, using these tags and given a single program episode, we would define rules for identifying “related” programs and episodes.

This is where the aforementioned headaches started to kick in.

Once you started to think about it, all sorts of questions cropped up, like, which carries more weight, matching primary genre tags or secondary genre tags (or should they count equally)? Or, assuming two related programs have the same tags as the target episode, how to break the tie? Or, do we match an episode within one series to other episodes in that series or restrict it to episodes of other series?

Pass the aspirin, because I’m getting a headache just thinking about it again.

Luckily, we have some fine folks working here who sat down and really noodled through this to come up with some matching logic. When written out, the matching rules looked something like this:

1. Match at the episode level
2. Cull only from upcoming or recently-aired episodes
3. Look for most tag matches, with all tags equally weighted
4. Only allow one episode per program/series to appear in “You Might Also Like” box
5. In a tie, give priority to episodes with same “Program Primary” tag
6. If still a tie, give priority to episodes with exact same tag makeup (i.e. both have only one Primary tag)
7. If still a tie, give priority to the episode with soonest upcoming airing.

The idea was then to use the tags and these rules to generate up to three matches for each episode to display in the “You might also like” block in the right hand rail.

Well, up to three matches, unless there were more than three episodes with the exact same tag structure as the target episode. In that case, we will display up to five such matches.

No sweat!

In order to actually implement this, we could no longer just spit out the results from a view. Nope. Instead, we had to jump through a whole bunch of hoops. Here’s the thumbnail sketch of the implementation:

1. Given the tags for a target episode, query a view of TV programs, fetching all programs that match at least one Program Primary or Secondary tag.

2. Filter this list of programs, including only programs with an airing in our schedule data window (one week back, two weeks ahead).

3. Then count the exact number of tag matches and calculate a matching score for each program, based on the above rules. Then store the program in an array.

NOTE: I won’t go into the exact matching score formula here. Suffice it to say we came up with a formula that encapsulates the above matching and ordering rules. Please pass the aspirin again…

4. Next query a view of TV episodes, fetching all episodes that match at least one Episode Secondary genre of the episode in question.

5. Filter this list of episodes, including only those with an airing in our schedule data window. For each one count the exact number of matching genre tags for the episode and calculate the matching score. See if the episode’s parent program is already in the array of matching programs. If so, replace it with this episode if the matching score is higher.

6. Given the final array of matching episodes, reorder the array by the matching scores and display the top three (or five) entries!

The resulting PHP code to implement all of this ran to about 240 lines and looked a little something like this:

All that just to generate this on the front end:

Anybody know the limit on the number of aspirin you can take in one day?

One of the great things about open source software is, obviously, that there are all of these great people out there writing code and making it available to everyone. In the world of Drupal this means that there are tons of great contributed modules that are pretty much invaluable to a site like WGBH.org, like Views and CCK.

It also means that those of us who have to maintain a Drupal site need to keep up with the improvements and changes to all of this code by periodically upgrading the code bade. Often times this usually just means grabbing the updated code and running the upgrade script. Things usually go pretty smoothly.

Except when they don’t.

Take, for example, the other day when I saw (thanks to the CVS Deploy module) that there was a new release candidate version of CCK (from 6.x-2.0-rc6 to 6.x-2.0-rc7). I went ahead and grabbed the new code and ran the upgrade script against my development installation (don’t want to do this on the live system!). Then I checked out our TV schedules grid and it looked like so:

Using my years of web development experience and my highly developed technical acumen I quickly deduced that something was wrong! The quesiton was, what?

So I went to the issue queue for CCK and found that, indeed, the latest RC version of CCK required the latest development version of Views.

OK, since I’m reluctant to use development snapshots of modules, I figured that there would probably be a new RC version of Views coming out soon with the required fixes to allow CCK and Views to once again play nice together. Sure enough, in another day or two there was a new version of Views (6.x-2.0-rc1 to 6.x-2.0-rc2). I then upgraded Views (again, on the development suite) and got this:

Hmmm. Better, but still not quite right. Basically, there seemed to be a problem with passing in date arguments to the view used to generate the schedule grid; there was a similar problem with the full day schedules.

Soooo, I then went to the Views issue queue and, sure enough, found out there was a problem with the new RC release of Views and it’s interaction with the Date module. Namely, that the date filters normally available to Views were now missing, which is what I saw.

Once again I figured that, rather than go to the development version of Date, I’d just hang tight and see if a new RC of Date was released soon. Bing, bang, boom - the next day it was! After then upgrading Date from 6.x-2.0-rc2 to 6.x-2.0-rc3 (and also upgrading Views again, as a new RC was released in the meantime, to 6.x-2.0-rc3) and checking the site I saw…. this!

Whew! Everything was back to normal.

The lesson here? Some might look at this and say, boy, what a pain in the rear this open source stuff is! But not me. On the contrary, I think it demonstrates the greatness of open source and the community of people out there responding to problems, fixing bugs and generally making my life much easier. Sure, patience is sometimes required, but that’s a very small price to pay, in my opinion.

So, a big thanks to all the folks who build Drupal and Views and CCK and Date and all of those other modules! Well done, folks.

The new WGBH TV Programs and Schedules module has been up and running on Drupal for almost a month now and - knock on wood - everything is working great! In fact, things have been going so well, operationally, at least, that all has been … quiet!

Quiet is good.

Now that this phase of the project is all done we are turning our full attention towards the real goal: porting all of WGBH.org to Drupal and completely overhauling the information architecture and user interface. We’re currently busy doing content audits, wireframes, schedules and all that sort of fun site redesign stuff. Nothing is ready yet for actual development.

In the meantime, we’re also addressing a few small desired functionality changes to TV programs and schedules that we chose not to address during the build. At the top of the list is the way that we generated the list of related (i.e. You might also like) programs on our episode pages.

Currently, this list is generated automatically using tags applied at the program (series) level. We developed a simple TV Program Genre vocabularly to potentially apply to each program.

Content producers ultimately have the ability to override the automatically generated list if they like, but, for the most part, what you see is generated on the fly based on the tags.

The upshot here is that by only applying tags at the program/series level, each episode of a given series (e.g. all NOVA episodes) will display the same set of related programs. So while NOVA may generally be a science program, a given program may be focused on, say, a physics problem, but this isn’t reflected in the related programs list. Our tagging scheme doesn’t currently allow us to relate programs on a more granular level than the simple genres we’ve defined.

Initially, we had planned to support tagging at the episode level for the initial build for use in generating the related program list. However, when we sat down to hash out how it should work it quickly became clear that using tags at both the program and episode level made things far more complex.

For example, right now, with tags only at the program level, it’s pretty simple. On a given episode page, we fetch the tags on the parent program, then using a view, generate a list of other programs with the same tag(s) and display three of those. Easy-peasy!

But once you throw episodes into the mix you now have to make decisions on issues, like, do matches on program or episode level tags count more? Should we weigh episodes with matching tags that are part of the same series more - or less - than similarly tagged episodes from other series? Etc and etc.

So, we tabled the issue for the first release and just went with program level tags. Now we want to start tagging episodes and come up with rules to generate a more granular list of related programs. That’s something we’re hoping to work out this week.

Have you run into a similar problem? Any thoughts on how best to do this? Speak now!

Like Big Brown coming down the stretch at the Belmont Stakes last week, us coders here at WGBH Online are bearing down on the finish line that is the rebuild of the TV Programs and Schedules module, well ahead of the pack… er, on second thought, maybe Big Brown at Belmont isn’t the best analogy.

In any case, deadlines are looming! We’re scheduled for a database freeze at the end of the month, by which time we need to have principal development completed and the code ready for some serious testing and tuning. At this point Pete is busy working on the search functionality and I’m tying up as many loose ends as possible.

I’d like to use today’s post to discuss one particularly funky and challenging bit of functionality that is finally working - the episode page sidebar!

Now tell me the name alone - episode page sidebar - doesn’t send a chill down your spine! It does mine, at least.

Let’s take a look-see at this little puppy to see what we’re talking about:

As you can see episode page sidebars will display information related to the series or episode (e.g. related programs or events) , as well as a search box, and some promotional content (e.g. shop, support). Nothing out of the ordinary here. We do the same thing on our current episode pages.

So how would this be implemented in Drupal? Easy, you say! Each item could be managed as a block. Voila! Hardly any technical expertise at all required here.

Not so fast, partner. There are a few additional requirements.

(1) There needs to be a default set of these items, that would display on all episode pages, unless…

(2) An editor chooses to create a series-specific version of a particular sidebar item, in which case s/he needs to be able to create it, relate it the series and have that override the default item on all episode pages for that series, unless…

(3) An editor chooses to create an episode-specific sidebar item, in which case s/he needs to be able to create it, relate it to the episode and have it override any default or series-specific version of that item.

OK, maybe it’s not so simple.

Here’s what we’ve noodled up to make this all happen.

(1) Since the search box is the one sidebar item which will always be there at the top and cannot be overridden, that is indeed managed as a stand alone Drupal block and positioned at the top of the heap.

(2) All other items in the sidebar are managed using a custom (CCK) content type called a content box. Content boxes are pretty simple and have the following attributes: title (for internal use only), a body (which contains all of the content to display, including any optional title or header), and an optional parent content item (like a TV program or episode), managed as a node reference. Also, using the Scheduler module, each content box can have specific publish and expiration dates. Content boxes will be able to have images, which will be added to the body using the Image Assist module.

(3) In order to determine where a content box displays in the sidebar we defined a fixed taxonomy vocabulary (Content Box Location), which is a hierarchical set of terms (e.g. TV -> Programs -> Sidebar -> Events). The display order of the sidebar items is determined by the order of the terms within the vocabulary. The idea here is that content boxes will eventually be used on other parts of the site once it’s all on Drupal. At that point we’ll be able to define additional terms in the vocabulary for other locations on the site (e.g. Radio Home Page -> Coming Up).

(4) Default sidebar content boxes are then defined as content boxes that are assigned a TV -> Program -> Sidebar tag, are not explicitly related to a series or episode node and which have the word default in their (internal) title.

(5) Series sidebar content boxes are defined as content boxes that are assigned a TV -> Program -> Sidebar tag and are explicitly related to a series node.

(6) Episode sidebar content boxes are defined as content boxes that are assigned a TV -> Program -> Sidebar tag and are explicitly related to an episode node.

Got all that? Good, because this will be on the final.

Given that set up, here’s how we actually construct the sidebar for a given episode.

We have a block view that selects all of the default episode sidebar content boxes. In the theme file for that view display, we then call a second block display for the sidebar view that takes a parent node id as an argument. We call this first to get any sidebar views related to the parent series and then again to get any sidebar views related to the episode. All of these content boxes that we’ve gotten back are then merged according to the rules above and the final list is displayed.

Almost forgot, the Related Programs sidebar content box adds one additional layer of complexity. The contents of that box are generated using a related programs view, which finds up to three related series using another taxonomy vocubulary (TV Program Genres). It’s pretty straightforward, unless…

See, easy peasy lemon squeezy, as my kids say!

Happy May!

Hard to believe we’re already into the shortest (letter-wise) month of the year, isn’t it? After a fine week off last week to spend quality time with the kids (is there any other kind?), I’ve been back in the saddle working away on this little Drupal project we’ve got going on (think of it as the Big Dig of coding).

While I was away my friend Pete got himself fully immersed in the fun and made some great progress on implementing our theme. He’s spent this week working on the TV schedule grid. So far he hasn’t run away screaming. i’ll get him to write about that in a future post.

Once the post-vacation glow and feeling of calm and relaxation faded away (i.e. by 9:01am Monday), I was right back it. First task was to figure out why the Update Status module (which is now part of core) on my install wasn’t telling me anything about the contributed modules I had installed, like it’s supposed to. Well, turns out that if you are downloading this code via CVS, rather than a tarball - as we are - then, in order for update status to properly work you need the CVS Deploy module.

Who knew? Not me.

Anyhow, i deployed CVS Deploy and now Update Status works as planned. Using that, I went and upgraded a bunch of modules that had been updated whilst I was away. Most notably, the Views 2 module went from an alpha to a beta release, so I was anxious to see what was going on with that. In particular with the argument handling issue that I mentioned previously.

The Views 2 Alpha was completely devoid of the the Views 1 argument handling logic. So, to make it work, I jimmied and jiggered things around and did the following:

Set the view_php view attribute by hand via some code like the following

$view_name = 'tv_episode_airings_viewable';
$view = views_get_view($view_name);
$view->view_php = '‘;
$view->save();

Then, in order to have that code properly executed by Views, I added some old Views 1 code to views/includes/view.inc

// Execute the initial PHP code that a view can have.
if (!empty($this->view_php)) {
ob_start();
$result = eval($this->view_php);
if (is_array($result)) {
$this->args = $result;
}
ob_end_clean();
}

Bingo bango - that worked!

Now, in the Views 2 Beta they’ve added the ability to add some PHP code on a per-argument basis, which seems to be the way of the future (rather than one single block of code to handle all the arguments). However, currently, it provides functionality to set a default value for an argument or to validate an argument. I see no way yet to manipulate an incoming argument.

So, for now, we’re keeping things as is. Once Views 2 is officially released we’ll probably have to reengineer things a bit to handle arguments properly.

There’s one other big problem we’ve been wrestling with this week - memory, or lack thereof. Don’t recall if I’ve mentioned this before but during certain activities on the system PHP throws a fatal error, saying it has used up its alloted memory. It first showed up when I tried to run the nightly cron tasks, which mainly consists of importing the PBS TV Guide schedule data. Each night in the middle of this run that error is thrown. It also shows up when I try to use the Drupal 6 Theme Editor. Pete also ran into when trying to install the first version of our custom WGBH TV Schedules module.

At the moment we’re not really sure what’s up. In php.ini I’ve got memory_limit = 256M, which should be plenty of memory for what we’re doing. I’ve even boosted the allotted memory to 512M and we still hit the error. Stay tuned…

Last but definitely not least we’ve fully staffed up and now have an official project manager and designer for this whole project. Please join me in a around of applause for these new team members Louise and Tyler! Welcome to the party!

I don’t know about where you live, but this week coming up is school vacation week around here. As the father of two of bundles of joy, I’ll be taking next week off have some Q.T. with the family as we parade around northern New England, doing what families do (e.g. goofing off, yelling, getting chocolate out of the kids’ hair, etc.).

That means, sadly, that I’ll have to tear myself away from Drupal 6 for a week (really, not a problem, believe me). In preparation for the week off I’ve been busy as all get out trying to reach the milestone that we’ve dubbed the first code merge. By that I mean it’s the first chance for Pete and I to merge the work we’ve been doing on the first phase of our rebuild, the new TV Programs and Schedule module.

Up until now, we’ve working separately (a separate peace, if you will), developing on our own Drupal installs on our development server. Pete has been creating a theme to match the current WGBH.org template (look and feel changes to the site will come in phase 2). I’ve been busy getting the PBS/TV Guide XML data import running, as well as working up the basic functionality of our Programs A-Z page and the Full Day Schedules by Channel pages.

Anyway, it’s high time that we finally incorporated each other’s changes into our own working install. So, this week was devoted to working out a process, not just for a one time code merge, but for rolling out our development work to each other, as well as to a staging area and to live installs going forward.

Currently, we’ve basically got a theme and a custom module for TV schedules. Managing the code is easy; everything is managed in a local SVN repository. Core Drupal and contributed modules and themes are pulled down via CVS and checked into SVN. So the entire code tree can be easily managed across multiple developers and pushed up to staging and live environments via SVN commands.

The tricky part - scratch that - the headache inducing part - no, sorry, scratch that too - the bang-your-head-against-the-wall-in-frustration part - is managing all of the information that’s stored in MySQL. How do we track changes to database structures (DDL) and the actual content that needs to be shared across installs (DML)?

Good question. The answer? Well, there is no easy answer that I’m aware of. The first guess is do a database dump (that’s what it’s called, I swear) of all of the structures and content. In theory that file can be managed via SVN or CVS. Problem there is you usually have lots of junk in your development database that you don’t want to replicate to staging or live environments, or even other development installs. So then you have to start weeding stuff out of the dump, which gets messy fast.

What to do, then? Here’s the strategy we’re going to use, and which i spent most of the week working on:

(1) Create a custom Drupal installation profile. This is used when first setting up a Drupal site. The WGBH profile installs a bunch of modules, sets the default theme (our WGBH.org theme) as well the administration theme (currently Zen Classic, a sub-theme of Zen - nice and clean and simple).

(2) TV Schedules module installation file. This is where most of the action is. Here we do a whole of bunch things like:

• Create CCK content types (e.g. TV Channel, TV Program, etc.)
• Create nodes for certain content types (e.g. channels)
• Create Views (e.g. Programs A-Z)
• Create a number of blocks and position them on our themes
• Add menu links
• Tweak a number of variable settings

I’d like to use the module install and upgrade files to manage changes to database objects and content. The install and upgrade files can be managed in SVN and used to roll out changes to development/staging/live environments. The tricky part is extracting the relevant changes from the database. That’s where it gets messy.

For this initial code merge, I relied mainly on the CCK export/import functionality to get the content type definitions. In theory, this is pretty straightforward. I used the export function on my development install to generate the code to recreate the content types. I added the code to module install file. That should have been it for the content types.

Except that it wasn’t.

CCK (like Views) on Drupal 6 is still in an alpha state, meaning there are unfinished bits, bugs, and general unforeseen shenanigans involved in using it. I found that the export/import process was a bit lacking. It copied the basic content type and field definitions just fine. However, field groupings were lost, and certain field attributes didn’t survive the transfer.

In the end what I did was perform the export/import in the module install file to create the basic types and fields (and related database objects) and then overwrite the data in the CCK definition tables using a database dump of that content from my development database. That part was put into a SQL file which, for now, has to be manually run in MySQL after the module is loaded.

It’s a bit clunky and not ideal, but I figure that part can go away once CCK for D6 is finished.

The good news is the Views export/import process worked much more smoothly. I was able to take the export code and run it from the module install file and the views were created as expected. No (or minimal) data tweaking was necessary. Yay!

Aside from that, creating actual nodes, blocks and menu links in the install file was all pretty straightforward. All in all, though, it took the better of the week to work it all out and get the process in good enough shape to hand off the Pete for testing.

Going forward, it will continue to be a bit of work to continue this process of tracking database changes. But, for now, it’s the best way I know of to proceed. If anybody has a better suggestion, I’m all ears! Please share.

Oh yeah, one annoying issue I’ve had with Views 2 is the lack of support for argument handling. Where the heck did that go? Am I missing something obvious? I found that the views object has a view_php attribute/method which seems to function the same way (I put my D5 view handling code in there), but there is no place in Views UI that I could find to add that code to a view. I had to do it manually.

Again, if you know more about this, let me know! That little problem alone gave 5 or 6 new gray hairs.

There you have it! For now, I need a break from the code and all that cursing.

While I take a quick break for a large shot of Ibuprofen in order to douse the big headache that Drupal 6 (contributed) code is giving me, allow me to toss out two quick thoughts:

(1) Views 2 needs - to put it mildly - a LOT of work, which, to be fair, is not shocking since it is only in an alpha state.

(2) Enough with the drag and drop and fancy JS controls and functionality in the D6 admin screens. There’s already way too much of it for my liking.

Right. So, back to pounding my head against the wall, I mean coding.

Forgive me, Father, for it’s been 12 days since my last post, mainly because we’ve been very busy here at WGBH Online. The good news is all of that busy-ness has paid off because…

(drum roll please)

the WGBH.org rebuild has been completed! That’s right! Check her out.

Ok, I’m lying. It’s a lame attempt at an April Fools joke.

But I wasn’t lying about the being busy part. Busy we sure have been.

Since our last chat we now have our new development server up and running. With the help of IT I have configured a new development environment, where by core and contributed Drupal code is being downloaded from the Drupal CVS repository and managed locally in our new SVN repository.

Wheeeee!

Pete and I are now working away in this new environment. Pete is working at replicating the current look and feel and styles of WGBH.org in a WGBH theme. I’ve been porting the TV Programs and Schedules code that I had already written in Drupal 5 to Drupal 6.

Building on Drupal 6 has already proved to be an adventure (think Survivorman - but more dangerous), as we’re currently working with the HEAD versions of CCK and Views (and a few other modules). Needless to say, I’ve already had a few bang-my-head-against-the-wall moments with buggy code. But we will persevere!

Oh yeah and we’ve also decided (for now) to go with Zen Classic as our administration theme. I like the clean, plain simpleness of it. It’s also close to the look our of current CMS, so should be less of a shock to the systems of our content producers when they’re introduced to it. This choice, of course, is subject to change…

Meanwhile, IT continues to build out the new hardware and reconfigure the old hardware that will all be used for the new version of WGBH.org. Specification meetings continue apace and, oh yeah, we’re still looking for a project manager for the whole rebuild.

All that and Spring has sprung!

That’s about where we’re at.

I’ve been up to my eyes, ears, nose and throat building out TV schedule related views. Things are moving along, I’ve got a nice working version of our Programs A-Z list and the daily schedules by channel. None of this involves any real theming work just yet; I’m focusing on functionality first. I’ll give more details on these pages next week.

Of more immediate concern, Drupalcon Boston is coming up next week and Pete and I will be attending, as will several others from WGBH. We’re looking forward to learning lots of Druaplish stuff, meeting folks and, of course, drinking free coffee. I mean, there is going to be free coffee, right?

Anyhow, there are lots of excellent sessions planned, across several different tracks. The full session list is here.

I’ll be spending most of time at the Site Building and Design & User Experience tracks. Here are some of the sessions that I’m planning to attend. If you see me there be sure to say hello.

• The Road from Newbie to Ninja: Drupal Tools and Techniques
• Performance tuning and optimization of high traffic Drupal sites
• Drupalcon Tutorial: Best practices in development environments, staging,
  build management, and production environments
• Theming with the Theme Developer module
• Drupal Security - Best Practices and Process Discussion
• Creating custom workflows for Drupal applications - taking advantage of core hooks and context
• Scalable Theming: Theming for 100s of node types, CCK fields and views

I hope to see you there! I also hope there are free donuts.