Zotero and the Bazaar: What Zotero Should Learn From Successful Open Source Projects
For awhile now I’ve been watching the Zotero project, and admiring their ability to deliver a compelling application that users (including me) have been hungering for. I appreciate that they have built the tool around one of the preeminent open source applications (Firefox), and that they have relied on open standards and development tools where possible. Finally, the code is all freely available and unencumbered. In all these ways, the Zotero team has gotten it right.
But … is this really enough to take it to the next stage?
Consider that there is no community involvement in Zotero development planning. The Zotero coders add features, based on their own internal assessment, adopting their own solutions, without wider public input or discussion. Third-party developers find out about what they are doing only after they release the code as more-or-less a fait accompli.
The latest news about word-processor integration is a perfect case in point. As a co-project lead for the OpenOffice bibliographic project, I’m trilled to see them say elsewhere they intend to support similar functionality there, but also perplexed they plan to do this without apparent consultation with our project. I can only assume this because I have yet to see any evidence to the contrary.
Put simply, Zotero looks more and more like a proprietary project dressed in free software clothing.
My problem with this does not just reflect some hopeless idealism, though. The issues of process and governance have practical consequences. The Zotero dev list has been essentially dead since its opening, which I have to believe is a consequence of the fact that developers simply do not feel included in the process.
And Zotero needs outside developers. Innovation in this space depends on standardization of data formats and representations, document encoding, APIs, and so forth. And a lot of code ought to be able to be shared between projects. Without the collaboration that makes that possible, users will end up boxed into the same kind of corner that truly proprietary applications like Endnote have long painted us into.
Free software is not just about cost; it’s about an alternative development methodology. Eric Raymond famously described traditional closed development models as cathedral-like, and the free software revolution inaugurated by Linux as a bazaar. Raymond makes clear that the cathedral approach has its place, particularly in the early stages of a project. But beyond that, the advantages of truly open development are so compelling that they are hard to avoid. Indeed, Mozilla and Firefox are themselves a product of this methodology. That Zotero exists at all is because they have smartly built off of this free software infrastructure. Yet one gets the sense of a project at the portals of the cathedral gazing out at the bazaar, but not yet ready to step out the door.
Now—as they look to transition from a compelling first release to more advanced functionality—is a perfect time for the Zotero team to move from the hermetic world of the cathedral, to the open world of the bazaar. If done right, it will get us all where we want to go more quickly, and with better results. This has to mean cultivating a more collaborative and interactive community, particularly with developers. It has to mean publicly documenting and discussing what they want to do before they do it, so that other developers can give useful feedback, and in turn plan for forthcoming changes.
I should add that I hesitated to post this. But I’ve already gently pushed on this in both public and private, on more than one occasion, without much effect (though they did open up their code repository). I present this, then, in the spirit of constructive criticism.
Well put, Bruce. I wonder if the sponsors of Zotero, including the Alfred P. Sloan Foundation, the Andrew W. Mellon Foundation, and the Institute for Library and Museum Services, are aware of how much further their investment would go towards achieving their stated purpose if a fully open bazaar style of development were adopted for Zotero. As is, the ones who benefit the most from Zotero’s cathedral style of development are those at the Center for History and New Media at George Mason University who may wish to make a career of Zotero, and the intellectual property scavengers and hoarders at Thompson (”our business is information”), who would be well advised to patent everything in site for the sake of protecting market share of EndNote et al, before it gets documented on a truly open source project as prior art.
Indeed. I’ve been warning the Zotero crew that one of the reasons they need to do everything in the open is to guard against patent problems later. Apple has already tried to patent fairly obvious ideas in this space that could have far-reaching impacts.
BTW, I don’t attribute any ulterior motive whatsoever to the current development approach at Zotero. Most of these guys are academics, and so–like me–a typical example of scratch-your-itch user-led development. But I do think it needs to be changed, and soon.
[...] Over the weekend, Bruce D’Arcus wrote a blog post about Zotero in which he said that “one gets the sense of a project at the portals of the cathedral gazing out at the bazaar, but not yet ready to step out the door.” This has sparked a bunch of discussion among us Zoterons over the past few days, and I wanted to distill a bit of it here. [...]
I just posted a longish missive on my own blog about this, but I’d like to chime in to this comment thread with a few directed points (speaking for myself, not on behalf of Zotero as a whole):
First, I think we’re being unfairly characterized as an opaque box - for months, anyone interested in engaging with development planning has been able to access our internal tracking system and SVN; the ticket queue is the single best representation of our development plans, and at this point virtually no code gets written without being described there first. We’ve definitely not been great about regular vision updates, but there’s only so much time, and our focus has gone toward improving the software.
Secondly, beyond visibility, if anyone wants to participate in the process, we welcome all comers. Much like other open source projects, you can contribute code and, once you’re vetted, hack away. If there are particular places that people want to see us elaborate or features they want implemented, they’re more than welcome to do so. For those who don’t write code, check out the forums; I’ll hold up our record of responsiveness there against any other open source project.
Third, I’m particularly wounded by Paul’s assertions above; at this point everything is open - code, documentation, the works. We aren’t claiming ownership over anything in the “property” sense that he uses to conflate us with Thompson, and we’re not trying to make our careers on Zotero - as Bruce kindly pointed out, we got started on this because we just wanted a tool we could use, and were lucky to find funding support from institutions who agreed on the broader need.
I guess the thing I’d love to see is more constructive criticism - beyond general nods toward open governance, are there concrete things that you think we can/should do ? What mechanisms do we not have that you want to see?
Josh — I don’t think the “we’re busy writing code, don’t have time for conversation” argument is convincing. We’re all busy, but communication is critical for what we’re try to do here.
I recognize that you have your own perspective on the degree of openness of your project, but my post reflected the position of a number of developers looking at this from the outside. I stand by my proposition that the measure of the health of an open source project is not the traffic on user forums, but rather on the dev list. Indeed, most of the developers that would contribute to Zotero would themselves be users.
I agree that Paul was harsh, but I hope you’re not asking me for specific constructive suggestions. I’ve been giving them to you guys for the past six months. Open up the repository and bug tracker (which you’ve now done for both; kudos), document your development plans and discuss them on the dev list.
“are there concrete things that you think we can/should do ? What mechanisms do we not have that you want to see?”
Josh, I’ve mentioned this already on your blog post:
Personally, I’d wish that the Zotero developers would discuss their visions and do their early-stage planning in the open. Let us third-party developers participate on your decision forming process. Let’s not only talk about site-specific translators and short-term goals, but let us collaborate on standards-based features, as e.g. microformats (i.e. embedded RDF or the like), unAPI, RSS/Atom, OpenSearch & SRU, and the planned Zotero server/client API (think synching).
It’s true that your forums are active but it’s really hard to get developer-related questions answered there. And currently, the dev list (or direct mail) is not any different. Many thanks for taking these issues into account.
[...] Such promise generates excitement and early evangelists. One of those evangelists, however, frustrated with the pace and degree of responsiveness from the program, went public with a call to better embrace Raymond’s “bazaar” aspects of open source (see this reference and its comments). That, in turn, with its comments, spurred another response from one of the Zotero team leaders speaking unofficially called Cathedrals and Bazaars, also in reference to Raymond’s treatise. [...]