darcusblog

Another year, and another $99 bugfix release from ISI Reearchsoft.

Last night I decided to take another look at Endnote and its XML support. Here’s the thing:

1. The XML that Endnote creates is itself not valid against their own DTD! I counted 18 errors in the XML exported with demo “Paleo” library, which is astoundingly bad.

2. The file includes a root element of “xml,” which is not included in the DTD.

3. Likewise, because the DTD is designed in such a way that element order is crucial, it leaves huge room for exactly these mistakes.

4. Now try this: export Endnote data as XML, then open the file in some XML editor and pretty print it (e.g. indent the tree). Now re-import the file into Endnote. Guess what: it doesn’t work!!

Sorry ISI, but this is not XML support.

In comments to the above linked article, one poster writes:

The inability of Thomson to deliver a competent version of Endnote for the Mac OSX user may be the single largest reason for the decline of the Mac in the scientific community. Though a great many of us still persist, the truth of the matter is that Endnote in OSX with the latest versions of Office has been largely dysfunctional. At large research institutes such as where I work, scientists must chose the best tool to help them complete a job. Unfortunately, that answer is more frequently found on the Windows platform. I now do all of my writing on an IBM laptop because I need seamless, reliable integration of Endnote with Word. If Apple wants to make a statement of their support for the scientific community, they should buy or license away Endnote for OSX from Thomson and make this product work.

The commenter is right about just how bad an application Endnote has become. I’d say it has become so bad that ISI must be violating some law or another by continuing with its practice of not fixing major problems without forcing users to pay a full upgrade price. Really, if Microsoft was this negligent, my guess is the Justice Department would be all over them.

However, the problem is bigger than ISI and Endnote, and it would be a mistake for someone to try to buy — or blindly copy — the application. I keep saying it, but will repeat: with contemporary open standards and protocols and free software tools, it’s really not that hard to do much better than the commercial alternatives. It just takes a critical mass of people and institutions willing to dedicate some effort or time or money to solving the problems.

Indeed, just this morning someone posted a note to the CiteULike list inquiring about using my CiteProc formating system with the service. That’s exactly the sort of thinking that we need right now! Applications like Endnote are dinosaurs!

My last post seems to have stirred some controversy, which is good. My purpose is mainly that the next person that thinks hmm .. maybe I ought to write an open source bibliographic application? will think twice about how to do so.

In comments, Mark Grimshaw wrote:

[A] programmer designing conversions from bibliographic databases for such styles (as wikindx does) HAS to know what type a particular resource is as the presentation of the resource entry for a particular style very often depends directly on the type of resource. A journal article is displayed quite differently to a newspaper article, to a chapter in a book, to an article on the web or to a proceedings article.

I’ve had the same discussion recently with Paul Tremblay, and let me get quickly to the point and say that this wrong. I put together a demonstration awhile back that shows my argument. Moreover, I recently put together an XSLT stylesheet that fairly successfully proves that it works.

A data model and formatting system that is based only on bibtex-like typing will fall down once it has to handle the needs of many scholars. Data will be inaccurate or vague and formatting styles will fail to format any record type not explicitly defined.

The solution is a system that has a rigorous generic fallback system based around structural class, and only secondarily on genre/type. This is nothing radical; it’s just an extension of the existing models in Endnote and Reference Manager.

update: ok, here’s a compromise example:

1. use typing for main layout, but list of types is extensible in schema
2. require definitions for article, book, and chapter, which serve as the generic fallbacks
3. some logic then associates other record types with their appropriate fallback
4. rendering of name roles and genre and media description is moved into separatee elements

This has the advantage of being more-or-less familiar and author-friendly, while also being quite flexible.

<citationstyle>
  <info>
    placeholder for metadata
  </info>
  <content>
    <name-roles>
      <role>
        <term>editor</term>
        <renderas>
          <single>Ed.</single>
          <multiple>Eds.</multiple>
        </renderas>
      </role>
    </name-roles>
    <genres>
      <genre>
        <term>dissertation</term>
        <renderas>PhD Dissertation</renderas>
      </genre>
      <genre>
        <term>letter</term>
        <renderas>letter</renderas>
      </genre>
    </genres>
    <media>
      <medium>
        <term>cdrom</term>
        <renderas>CD-ROM</renderas>
      </medium>
    </media>
    <citation>
      <author-year>
        <names>
          <firstname/>
          <middlename/>
          <lastname/>
        </names>
        <entry>
          <creator/>
          <year before=", "/>
          <point before=": "/>
        </entry>
      </author-year>
    </citation>
    <bibliography>
      <names>
        <firstname/>
        <middlename/>
        <lastname/>
      </names>
      <entry>
        <reftype name="book">
            <creator/>
            <date before=" (" after=") ">
              <year/>
            </date>
            <title font-shape="italic" after=", "/>
            <origin>
              <place after=":"/>
              <publisher/>
            </origin>
            <physical-location before=", "/>
            <url before=", "/>
        </reftype>
        <reftype name="chapter">
            <creator/>
            <date before=" (" after=") ">
              <year/>
            </date>
            <title/>
            <container before=", In ">
              <creator after=", "/>
              <title/>
              <origin after=", ">
                <place/>
                <publisher before=":"/>
              </origin>
              <part-details>
                <pages/>
              </part-details>
            </container>
            <physical-location/>
            <url/>
        </reftype>
        <reftype name="article">
            <creator/>
            <date before=" (" after=") ">
              <year/>
            </date>
            <title before="“" after="”"/>
            <container>
              <title/>
              <origin/>
              <part-details>
                <volume/>
                <issue before="(" after=")"/>
              </part-details>
            </container>
        </reftype>
      </entry>
    </bibliography>
  </content>
</citationstyle>

Just under a year after releasing its previous “major upgrade” (bugfix), Endnote maker ISI Researchsoft has released another. The major functionality improvement they will seek to convince users to pay $100 for upgrading to? Unicode support.

Sigh … it would be nice to see some smart institution out there take the money they’d spend on this upgrade, and put it into open source development. It would not take much to best Endnote with a free offering. Per previous discussions, if you want to make this happen, see RefDB and OOoBib.

Indeed, a guy working in an optics lab has been working on what he calls an “advanced interface” for RefDB using PHP. He’s even using XHTML Strict and CSS for the HTML stuff. Am looking forward to seeing the results.

The majority of the energy I’ve put into open source bibliographic tools and standards over the past two years is a direct consequence of being frustrated with Word vX and its bibliographic companion Endnote. Why was I angry with Endnote producer ISI Researchsoft, in particular? Because Endnote 6 was:

• very late
• added no compelling features, and removed others
• did not initially work with Panther

So … a year later they release what is basically a bug-fix as a full paid upgrade, and guess what? It doesn’t work with the new Office 2004 release!

The is exactly the sort of reason the free software movement exists to begin with!

Update: Official comment (by way of MacFixIt) from ISI:

Due to changes introduced by Microsoft in the newly released Office 2004 for Mac OSX, EndNote’s Cite While You Write (CWYW) tools are not currently compatible with Word 2004…. At this time there are no plans to make EndNote 7 or older versions compatible with Word 2004.

The arrogance of this company is absolutely unbelievable!

Art Rhyno has an interesting blog on library-oriented technology, and some thoughts on some of the pieces necessary for an Endnote-killer.

Having thought about this deeply for the past year or so, I think it best to start fresh. Certainly it is important to learn from existing solutions, but it would be a mistake to think there are not better ideas floating around.

My key argument would be that it is time to get rid of monolithic applications. Like Word, Endnote is a monolithic application. It is a database for reference data, it is an online search/and query tool, and it is a citation formatter. These functions need not all be included in a single application, and we’ll get to where we need to go quicker and with greater flexibility if we can not just recognize that, but exploit it.

So here are the pieces we need:

1. A formatting engine independent of any particular data model. Whether the thing is written in Perl, or C, or Java, or – as Bibliofile (soon to be renamed BiblioX) is – XSLT, the sole job of this engine ought to be to suck in XML data, and spit out formatted citations. The formatting style specification must itself be XML. Aside from that it ought to be open to any format for which one can write an input driver. If anyone has XSLT skills and is willing to contribute, Bibliofile is the best place to start.
2. Rich metadata serialized in XML and representable in RDF. BibTeX is not the place to start here; its data model is simply wrong. Likewise, the data model in applications like Endnote is also broken.
3. An online query/download utility. Like the above, this should focus on acquiring metadata in XML if possible, and passing it to other applications and tools.

All of the above have the common theme that they are small and task-focused, and based on open metadata. They also must be open source. Different projects – OpenOffice, Chandler, etc. – can contribute unique end-user experience by drawing on those tools.

One frustration I have with all current bibliographic systems – both free and commercial – is their metadata models.

Consider the following scenario:

I go to an archive and find a report in a collection.

Question: How do I represent this in existing standards?

Answer: Awkwardly.

In Endnote, for example, there is – as in most of these standards – a report reference type. So I can code most of the information there. However, there are no fields to represent archival holdings, which are typically handled by a separate reference type of “manuscript.” Yet this makes little sense; a report is still a report, whether it is in an archive or not.

What about a postcard from the same archive? Here one generally falls back on a “miscellaneous” reference type, and so lose metadata in the process.

How about a newspaper or magazine article republished in a book? Here all of the metadata models – DocBook, Endnote, BibTeX, etc. – fall apart.

By contrast, MODS is cleanly and rigorously structured, and so can handle all of these examples gracefully. The first example is just a report contained in a collection. The second has the exact same structure, but with different resource type and genre values. The third is just an article contained in a book, but originally contained in a newspaper.

I have been trying to convince people of the central importance of the model for bibliographic metadata. Thankfully, it seems I am having some success. To wit, the bibliographic project at OpenOffice plans to adopt MODS as its primary metadata format. Likewise, Chris Putnam is moving to MODS as his XML format for his bib conversion tools.

There’s still a lot more to be done, though. How about an elegant form-based web interface for mods data entry, for example?

For any academic thinking of alternatives to Word, they are immediately stuck with the issue of bibliographic data. Endnote developed as a well-integrated bibliographic plug-in to Word. It makes the task of managing complex citations and bibliographies manageable.

However, Endnote was bought by ISI Researchsoft a few years back, which now holds a monopoly of the commercial bibliographic manager market. Not only do they now own Endnote, but also Reference Manager and ProCite.

The result: glacial innovation, buggy software, and regular paid updates. Indeed, version 6 of Endnote – the first compatible with Mac OS X – came very late, buggy, and missing key features. Endnote 7 is released roughly a year later as a full paid upgrade that a) fixes bugs, and b) adds as a “new” feature the ability to process RTF documents; functionality that had been around for years before ISI Researchsof opted to drop it in the previous release.

Enough!

Nevertheless, for anyone who uses these two applications together, it becomes quite a challenge to find alternatives.