nptech.info

• The Unpardonable Absence of a Democratic “Attack Machine”
• “Time to Start Taking The Internet Seriously” by David Gelernter Great read.
• Openness can make citizens collaborators with officials
• Day Traders and Campaign Plans
• Obama Finally Keeps His Word
• Darwin Foes Add Warming to Targets – NYTimes.com Alliance of know-nothings. Sigh.

Slurm!!
Originally uploaded by jonstahl

Molly and I are in love with these nightlights we just bought. Slurm, anyone?

Copenhagen was doomed to failure from the day that Obama decided to do health care in 2009 and not climate legislation. Enviros should have recognized this and responded by de-emphasizing the strategic importance of Copenhagen, and focusing their limited resources to prepare for 2010.

Last June, I wrote up some initial thoughts about how Plone could to more to help folks with limited experience preparing images for the web.    It got quite a bit of favorable response, but then I went off on sabbatical and haven’t followed up, until now.  I’m more convinced than ever that this is a good idea, and I really want to build some energy and resources around it.

Here’s some updated thinking on what I think could be done, both in core Plone and in a new add-on product, along with a simple wireframe and some implementation notes.

Making core Plone smarter

Plone 4 includes a frequently-requested feature that makes images much smarter: through-the-web configurable image scales.  Previously, these were hard-coded into ATContentTypes.  Now, we have a new “Image Scales” control panel, and site admins can add, edit and remove image scales.  This makes it much easier to provide the image scales site users need, and to eliminate the ones they done.  Nice work!

There’s one more change that I think should happen here:

Let’s make the amount of JPEG compression that PIL applies be easily configurable.  Right now, it’s hardcoded into our zope.conf, and it’s really, really high.  When we prep images for the web here at Groundwire, we typically use a compression quality of 60.  I think that it would make a lot of sense to expose this variable through-the-web in the Image Scales control panel configlet.

Rough Spec for an “Image Optimizer” Add-on Product

Ok, now here’s where the cool stuff starts.  Here are my updated thoughts on an add-on product that I think would really make life easier for sites that have lots of poorly-optimized images.

I imagine an “Image Optimization” control panel configlet that will present a UI that allows the user to:

1. Walk the site catalog, looking for all objects that are Image-ish and in JPEG, PNG or GIF format.
2. Look at the pixel dimensions and the filesize of each image.
3. Find images where the “bytes per pixel” (filesize/*(height*width)) is higher than a certain “reasonable” value.  (0.5 bytes per pixel seems about right to me, based on the idea that a 150X150 pixel JPEG image shouldn’t weigh much more than 10kb.  (Obviously, these settings should be user-editable.)
4. Present the user with a list of “suspiciously large” images, along with their pixel dimensions, filesize, bytes per pixel and a preview.
5. For images with dimensions that are larger than user-specified value, offer to resize them.
6. For images that are too “heavy per pixel,” offer to apply more compression (for JPEGs) or convert to JPEG (for GIFs/PNGs)
7. Apply changes via a separate process with AJAX updates so that Zope doesn’t bog down.
8. Afterwards, offer to rebuild Plone’s auto-resized images afterwards from this new original.  Purge caches as necessary.

Here’s a wireframe of what I’ve got in mind:

A few more implementation notes/questions:

• Alec Mitchell recommended using ImageMagick, because it offers much better quality than PIL.   I think this also seems sensible, because it would offer an easy way to spawn the intensive image manipulation work off into a separate process.  An outside-of-Zope processing approach such as zc.queue is probably worth looking at as a model.
• Resize, then convert, then compress.
• JPEGs should be compressed via ImageMagick.
• PNGs could potentially be compressed by running through OptiPNG.
• GIF’s can’t be compressed
• Use AJAX to refresh the screen on scan and as optimization proceeds.  Show optimization results by crossing out weight/dimension/density and replacing with new values.  (Hide unselected images when you hit Optimize.)  Total up savings at the bottom when optimization is complete.
• Can Plone’s image scale rebuild be run in the background somehow so we don’t tie up a Zope process?
• This should target both Plone 3 and Plone 4.
• For extra bonus points, warn the user of unoptimized images right after upload, and offer to fix it “on the spot.”

Somewhat randomly, I also wanted to take note of Kurt Bendl’s useful recipes for limiting the max KB and pixel dimensions of uploaded AT Images and Files.  This might be useful and/or worth exposing in a TTW setting.

Next steps

Doing a mediocre job on this would probably be pretty easy, but it will take some focused effort to really nail the details that will make this sing.  I think this is a project worth raising a little bit of money around, and hiring someone good to really own it.

If your clients would benefit from a tool like this, I encourage you to think about whether you can raise some cash from them for a bounty fund.  I don’t have a hard total in mind just yet, but I’m thinking that $3000-$5000 might be a reasonable range.

I’ve got a couple of people in mind that I’d like to approach as possible developers on this, but if this description strikes a chord with you and you’ve got both the passion and skills, let me know!

This also might make a great Google Summer of Code project, in which case I’d love to supplement that by paying an experienced Plonista to be a strong, hands-on mentor and ensure that the project gets done and done right.  (This worked really well last summer with Martin Aspeli and Timo Stollenwerk on plone.app.discussion!)

I’m willing to PM this and be “the customer” and Groundwire can very likely help seed the bounty fund as well.  Let’s make this happen!

I’ve had a Squeezebox as my digital music system for over three years now; it’s still one of the best few hundred bucks I’ve ever spent on electronics.

The Squeezebox’s server software runs on a server.  Since 2006, I’ve been using an old Dell Dimension 2350 that we had lying around.  But this weekend, I upgraded (slightly) to a Dell Dimension 3000 that my employer Groundwire excessed from its inventory.   Wow, what a difference!

The old server had 512MB of RAM and a 2GHZ Celeron. My “new” server has a 1.25GB of RAM and a 3GHZ Pentium 4.  My music library is about 24,000 tracks (that’s large-ish, but not huge by community standards, it seems).   While my old box never swapped, it was often running pretty close to its RAM limits, and the web interface could be sluggish at times.  Under the new server, it’s smooth and fast.  I suspect it’s additional RAM rather than the faster processor that’s making most of the difference.  I also swapped in a set of optimized MySQL caching settings (thanks Squeeze community!), which allows the system to take more advantage of its increased RAM.

Bottom line: should upgraded the RAM years ago. 

Shortly before dashing out the door for Pycon 2010, David Glick pushed out a 1.0 release of Popup Forms for Plone, which he and Steve McMahon built on top of Steve’s excellent Pipbox and PloneFormGen products.

Popup Forms for Plone makes it point-and-click easy to create timer-driven javascript popup forms anywhere in your Plone site.   You can see a simple example in action at Washington Conservation Voters.   It’s amazingly simple: just build your form in PloneFormGen.  (If you just want to popup a static HTML page or an image, you can use PloneFormGen’s “Form Prologue” and skip adding any form fields!)  Then, you use Plone’s portlets mechanism to assign the form to a page or folder on your site, and to configure an optional time-delay.  That’s it!  No programming, no javascript, no fuss, no bother.

Popup Forms are great for email capture forms, action alerts, user surveys, and many other calls-to-action.  If that’s what you need in your Plone site, I encourage you to check it out!

Here’s a pattern I’ve observed. Many website design clients, especially those who have never been responsible for a website project before, expect to a process that goes roughly like this:

1) Talk about requirements

2) Do a complete graphic design

3) Fully implement the design in the site

4) Then move on to building out the functional elements of the site and the content.

Unfortunately, modern web development processes don’t usually work that way.

The process usually needs to go more like this:

1) Identify requirements

2) Do sitemaps and wireframe mockups to get the functional elements and information architecture right

3) Implement the technical bits

4) Do a design to make it pretty

Lots of missed expectations can easily ensue. Educating clients to the point where they can understand a wireframe can be a big — and costly — challenge.

We’ve got two open positions at Groundwire right now: one for a CRM database consultant and one for a manager for our “Groundwire Labs” innovation program.  Both are incredible opportunities for a creative, entrepreneurial social change technologist who wants to join one of the most accomplished teams in the nonprofit sector.

I’ve been here for nearly 14 years, so I’m happy to field any questions if you’re thinking about applying!

CRM Consultant

We need an experienced CRM Consultant to build customized databases that transform the effectiveness of the environmental movement. Our ideal candidate brings to the table client-facing consulting experience, project management experience, and a technical understanding of database design & development.

Read the job description and apply online

Groundwire Labs Manager

We are now looking for someone to run Groundwire Labs. As the Groundwire Labs Manager, you’ll lead Groundwire’s R&D investments and define the cutting edge of how we use technology to help organizations to do a better job of engaging their communities. It’s all with an eye to our mission of building a sustainable society.

Read the job description and apply online

• Faux Friendship – The Chronicle Review – The Chronicle of Higher Education
  Wow, fascinating and damning indictment of “online friendship.”
• How Google Buzz Is Disruptive – Open Data Standards – NYTimes.com
  I would like this to be true. Oh please.

• Worldchanging: Bright Green: Where Did We Go Wrong on "Green Jobs"?
  Some really interesting, thoughtful conversation here.
• David Glick: Reflections on building a member directory using Plone and Salesforce.com
  
• Correcting the Oil Industry’s Errors
  
• Jon Stewart Disembowels, Eviscerates, Then Destroys The Blogosphere (VIDEO) | TPM LiveWire
  Funny and on point. Boom!
• State Senate votes to ban chemical BPA in baby bottles
  
• Daily Kos: State of the Nation
  Republicans are actually as batshit as we think they are. I weep for this nation.

• Access Denied: Abuses and Failures Under BC's FOI Act — Dogwood Initiative
  Great report from Dogwood Initiative
• Putting Kemper Freeman Jr. in his place
  
• What Should SR-520 Be?
  

• Recession Forces Nonprofits to Consolidate – WSJ.com
  Hmm, more anecdotes, but no hard data on the actual number of nonprofits merging or closing. Does anybody track this?
• Amazon, Macmillan: an outsider's guide to the fight – Charlie's Diary
  Totally fascinating.
• A "shock jock" in Montana has a great fall — High Country News
  Good news for decency.

Do you tend to see innovations as massive paradigm-shifting change, or as the incremental achievements built on longer-term trends?  Obviously, there’s some truth in both perspectives, but I think that most people have a tendency to lean one way or the other in how they interpret what’s going on in the world.

If you know me, you know I tend to be an incrementalist.  I think that real honest-to-god paradigm shifts happen, but that they’re pretty uncommon.  And they’re usually much slower than you’d think.

I’m often frustrated by what I perceive as a tendency of consultants and media types (especially new media types!) to frame every innovation as revolution.   I can understand how one might think it makes better copy, makes you a more dynamic speaker, makes you seem smarter and more interesting.  But really, I don’t think it does.  The best talk I’ve ever seen about open source is the one in which Eben Moglen weaves the story of open source into the centuries long struggle for freedom, dignity and human rights. Context is sexy.

Also, as someone whose professional practice is all about helping people make and embrace change, I think it’s usually more effective to connect new ideas to the already-familiar, rather than trying to motivate people with the fear of being “obsolete” or missing out on “what everyone else is doing.”  One of the most valuable things we can do as consultants is to help people fit confusing new chunks of knowledge into a longer-term framework of ideas and trends.  That builds capacity and confidence.

I’ve been thinking a bunch about the challenges of making cultural transformation in the organizations I work with here at Groundwire.  It’s a tough challenge.  The first step, it seems, is about naming the changes we want to help folks make.

Here are some rough notes that popped out as I was gathering my thoughts for a meeting.  I’d love to know what thoughts they provoke for you.

From –> To

1. Broadcast –> Dialogue
2. Formal –> Conversational
3. Organizational voice –> Personal voice
4. Goals –> Values
5. Centralized communications –> Distributed through many channels
6. Intuitive decisions –> Data driven decisions
7. Master planned –> Continual refinement toward clear big picture goals
8. Set the agenda –> Respond to what’s hot that fits your goals & values
9. Always the center of collaborations –> Partner more, and more informally

• The Six Americas of Climate Change
  
• The Facts of Cap and Trade
  

Center for a New American Dream has a nicely done “Alternative Gift Registry” tool (currently the #4 Google result for “gift registry”!) that allows you to create gift registries that de-emphasize consumerism (used goods, donations to charity, experiences rather than stuff, etc.).   This is a great example of a nonprofit advocacy group coming up with a valuable public-facing service that is grounded in its mission and expertise to bring people into the circle of engagement.

Continuing on the theme of quick-and-dirty benchmarking of the forthcoming Plone 4 release, I decided to revisit an experiment I did about a year ago in which I looked at the memory usage on startup of Plone 3 vs. Plone 4.  In December 2008, I found that Plone trunk used 36% less memory on startup than Plone 3.1.7.

But back then, Plone 4 was still a long way off, so I wondered if things had changed lately.

So I fired up my 2Ghz MacBook with 2GB RAM, I started up Plone 3.3.4 and Plone 4’s current trunk, and Activity Monitor reported memory usage as follows:

That’s a 29% decrease in RAM usage for Plone 4, from 142.5MB to 102.5MB.

Sweet!  Plone is not only a lot faster than Plone 3, it’s also much more RAM-efficient.  Truth is, though, it’s not really Plone’s fault.  One of the big changes in Plone 4 is that it now uses Python 2.6 instead of Python 2.4.  Python 2.6 is quite a bit more memory efficient, and that’s where Plone’s gains are coming from.

So, thanks, Python team!

Update: See comments below from Hanno.  Turns out Python 2.6 is better at garbage collection, so Plone’s memory usage will increase less as it runs, but Python 2.6 actually causes Plone to use slightly more RAM at startup than it would under Python 2.4 (which Plone 4 does not support!).  But, thanks to all of the amazing work that the Plone 4 team has done to make Plone leaner and meaner than ever, Plone 4 more than compensates for Python’s slightly increased drag.  Whee!

Bonus: If you’re wondering why I clocked lower overall RAM usage when I tested this last year (97MB for Plone 3, and 62MB for Plone 4), you can blame Snow Leopard for that.  Running 64-bit Python makes it use more memory than 32-bit, and Snow Leopard, new since last year, is now (mostly) 64-bit.  You can avoid this by running your production sites on a 32-bit Python.  I’m just too lazy to do it right now.

Extra special bonus: One of the many benefits of Plone 4’s new “blob storage” infrastructure for handling files and images is that it is way, way more RAM efficient.  We don’t have hard benchmarks on this yet, but in one very large, heavily used intranet site with 16.5 GB of most documents and media files, the team at Jarn was able to cut RAM usage from 10GB to 3GB with substantial increases in performance.  We expect that most sites with large archives of binary files will see similar resource usage improvements as they migrate to Plone 4.

• Profiting From Haiti’s Crisis
  
• The limits of innovation
  
• New York Times to Charge Frequent Readers of Web Site – NYTimes.com
  NYTimes plans a modern, flexible paywall. I actually hope it succeeds, quality journalism needs a good business model.
• The Seattle Talks
  
• Speaker Chopp unfurls his tax roadmap
  
• “See you lata’” is the new “beta.” Exploring an epidemic of opting out.
  
• New York Times Ready to Charge Online Readers — Daily Intel
  I for one would be happy to pay.

If you didn’t already think that one of the most important long-term issues in our society is the twin hydra of corporate personhood and campaign finance reform, today’s Supreme Court ruling should be a shocking eye-opener for you.  Make no mistake, this is serious, worrisome stuff.

It’s time for all of the groups in the “progressive” spectrum to come together to mount a major, long-term campaign to end the misguided, extra-constitutional notion of corporate personhood.  It seems to me that everyone has thought this is “someone else’s issue.”  No longer.  If we don’t get our act together here, pretty soon we’re not going to have a democracy left to argue about.  It is well past time to see a serious infusion of resources and organizing around these issues.

Good starting points if you’re new to this whole issue area are POCLAD’s book “Defying Corporations, Defining Democracy“and CELDF’s “Democracy School.”

Plone 4 is about to leave its alpha testing phase and enter beta testing prior to a final release.  One of the many things the Plone team has worked really hard on in this release cycle is improving Plone’s performance.   Plone core developer Hanno Schlichting has blogged about this a number of times, and deserves a tremendous amount of individual credit for digging deep into Plone’s innards to find and fix inefficiencies.

When I read Hanno’s most recent performance post, in which he shows Plone 4.0 alpha 3 serving up 22 pages/second without caching on his personal laptop, I  started to wonder how this compares with some of the other common CMS platforms out there.  I was pretty sure none of them could deliver more than 10 pages/second with zero caching or performance tuning.  So I did a bit of quick-and-dirty speed benchmarking.

I’ll start with results, then explain my methods.  As I suspected, Plone 4 is faster out of the box than some of the most common PHP platforms.  Lots faster.

That’s right:  Out of the box, Plone 4 alpha 3 served up 14.5 pages/second, that’s over three times faster than Drupal 7 alpha 1’s 4.1 pages/sec, Joomla! 1.5.15’s 3.6 pages/sec or Wordpress 2.9.1’s 4.5 pages/sec.  Plone 3.3.4, our currently shipping release, turned in a snappy 9.4 pages/sec, over twice as fast as its PHP competitors.  Whee!

Testing methodology

Here’s how I tested.  Hardware was my 2GHz MacBook with 2GB RAM.  I installed Plone 4.0 alpha 3 using buildout, and used MAMP 1.7.2 to run the PHP products.  I did a default install of each product, no add-on modules, I used whatever default content and initial configuration each product provided.  (Joomla! gives you the option whether or not to install sample content, which I accepted.)  No caching was installed or configured for any system.   I measured performance of the homepage as an anonymous user with trusty ol’ ab -n 10, ran it a few times to get the systems warmed up, then noted the value where the runs stabilized.

Lies, damn lies and statistics

So, what does all this mean?  Well, honestly, not much.  (Although Hanno clearly has a faster laptop than I do!) This is obviously a crude benchmark.  I didn’t load up each CMS with realistic sample content.  I tested two pre-release products (Plone 4.0 alpha 3 and Drupal 7 alpha 1) against three production releases (Plone 3.3, Joomla! and Wordpress).   All of these CMSes can easily pump out hundreds of pages per second with a little bit of tuning, reverse proxy and/or database caching (although many real-world users don’t bother with performance optimization!).  So, this benchmark is definitely not an accurate measure of the real-world performance of a site.  (Matt Hamilton recently explored this a bit.)

But I think that Hanno and the rest of the Plone team can be very, very proud of Plone’s raw speed.  And as Hanno points out, there are even bigger gains just around the corner; he’s targeting 50  pages/second (without caching), as his goal for Plone 5.