Ebook production: chaos and opportunity

Chaotic_mixingAugust was a busy month both in and out of the office. In house, we published the French Kindle edition of the ERG 2012, GMU 2012 : Guide Facile, and we successfully shepherded the English iBook edition through Apple’s tedious review process. We also learned, surprise!, that it takes at least as long for a change in the product description to get through Apple’s review process as it does the for the actual book to be reviewed. No quick fixes with Apple, ever.

Out on the road, I had the opportunity to meet face-to-face with a few specialty publishers who are either currently publishing ebook editions or are evaluating that option. Along with the phone conversations I had earlier, I’ve now had a whirlwind tour of the current state-of-the-art for small publishing houses.

The good news, for readers, is that most publishers are either on-board the ebook train or making their reservations as we speak. They know their readers want ebook editions. They are finding ways to supply those editions.  And they aren’t letting marketplace or technical uncertainties hold them back.

The bad news, for publishers, is that there’s no consensus yet on a best practice for producing those ebooks.  Each publishing house has had to, essentially, put together a DIY ebook production process

In my notes from the last five specialty publishers I’ve talked with at any length, I count a total of six separate production strategies. One house has experimented with two different processes and isn’t particularly satisfied with either yet.  Another has a ‘single’ strategy but it involves separate, parallel processes for producing their Kindle and Google Play editions.

Of course, here at Sherprog, we use two completely different processes but we produce two quite different types of ebooks.  The ERG is a reference book built up primarily from files of raw data.  For it we have a customized, highly automated process that lets us do most of our review and quality assurance while the ‘book’ is formatted as HTML.  The novels and short stories we publish for my mom, Elizabeth Gunn, are much more straightforward.  We’ve been using a process that starts with exporting the Word doc to HTML.  But we will probably switch to using Calibre’s new ability to accept docx files now that it is available. However, the two processes do share their final step: using Calibre to produce the final mobi (Kindle) and epub (every other marketplace) files

On the other hand, the publishers I’ve interviewed share a starting point: they are converting books that were originally laid out for print in Adobe InDesign. That’s why I find the lack of a consistent process so surprising. Why doesn’t Adobe just bake a good ebook conversion capability into InDesign and put folks out of their misery? I’m sure that’s what publishers want and are hoping for.

To be fair, InDesign does supply a variety of options that should assist in ebook conversions. And the marketplaces are contributing their own tools, too.  Among the combinations I found in my small survey are:

  • InDesign -> export to HTML -> Dreamweaver (for reformatting and corrections) -> Calibre to produce final epub and mobi files
  • InDesign -> export to epub -> Sigil (for reformatting and corrections) -> final epub -> conversion to mobi via an Amazon-supplied commandline tool
  • Make an InDesign clone of print version -> reformat and correct selected sections to deal with known ebook formatting problems -> export via the Amazon-supplied InDesign plugin (to produce just the Kindle version)
  • InDesign -> generate low resolution version of the print PDF -> send to Google Play (for conversion by Google to epub)

I bet everyone involved, on the technology side, is hard at work trying to produce better, more capable tools to make ebook production more of a push-button process.  But there are serious obstacles to that actually happening any time soon:

  • The technology  and market targets are moving in three dimensions at once.
    • The Adobe toolset continues to evolve in ways that have nothing to do with ebooks.  As I write, InDesign is in the midst of its ascension to the Creative Cloud, along with the rest of the Creative Suite tools.  But I know for a fact there are publishers out there who have backlist content they are considering for ebook conversion that still lives in PageMaker files.  The switch from PageMaker to InDesign happened a decade ago for Adobe.  User habits and re-usable content change much more gradually and always will.
    • The eReader marketplace is not a stable one.  New vendors with new readers enter the market (Kobo).  Existing vendors adopt new file formats (Amazon).  And, even for a single vendor such as Amazon, the use of eReader apps, implemented for a variety of different hardware platforms, is great for expanding sales opportunities and awful for ensuring a consistent reading experience.
    • eReaders exist as just one part of a larger electronic publishing ecosystem and that landscape is being carved anew all the time.  One big issue facing the whole electronic publishing industry now is adoption of the EPUB 3.0 standard. Although the final specification was approved almost two years ago, adoption has been, as it always is with these things, slow and uneven. Toolmakers, from the well-funded Adobe to the all-volunteer Sigil and Calibre projects, have to balance support for new, much wanted features with the burden of backward compatibility.
  • So much of the problem isn’t about technology, it’s about design. Ebooks are different from paper books, from PDFs, from web pages. They present huge opportunities for the reader (I can control the size of the type and make it bigger when my eyes are tired) and huge obstacles for the designer. Note the recurring manual step in the list of DIY processes above: reformat and correct. Trust me, ebook conversions don’t introduce spelling errors, they introduce design problems.  Some of these will have straightforward technical solutions. Some won’t.  It will take a person, with design skills, to work through the trade-offs.  How should pictures and text be ordered when you can’t guarantee they will be viewed side-by-side or even on the same page? Where and how can I break up this big table into smaller units that still convey the information in a useful way?

My money is on ebook conversion turning out to be a classic ‘semi-structured task’. That’s a programming distinction so fundamental I’m not even able to find a reasonable citation for it. While the difference between a structured task and a semi-structured one is easy to understand in theory, it is also painfully easy for technologists to overlook in practice.

Whenever you have a process where parts of it are so tedious or so overwhelmingly technical that they just can’t be done fast enough or reliably enough with a manual process, you clearly have a need for automation.  The question is, can you automate the whole process and make it truly push-button? If so, it’s a structured task. Think large scale inter-bank transaction processing.  Or do you need to focus on automating parts of the process but leave other parts open to human intervention/judgement/creativity? Think business planning with a spreadsheet.

In practice, the distinction is anti-fractal — that it, the answer changes depending on the scale you are looking at.  You can bet that somewhere, in the deep, dark recesses of a bank data processing center, there sits and waits a very human, if not green-eyeshaded, financial analyst to whom problem cases are brought for sleuthing and reconciliation.

And programmers like me can find useful and interesting structured tasks to work on in and around every semi-structured task we encounter in the real world, including ebook conversions.  There is a huge opportunity to contribute to the tool sets as long as we don’t have the hubris to think our code can replace that person who needs to make those critical design decisions along the way.

Back here in the office, we’ve got a small project coming up where we will get a chance to try out a couple of variants on the ebook conversion processes for ourselves. I think we’ll check out, for one option, the InDesign -> epub -> Sigil process.  I dismissed Sigil a while ago because it doesn’t support automation in the same way Calibre does. But it does seem to offer the design-oriented professional some nice features, so I’m eager to try it out on a real project. We’re still trying to decide on the other variant we want to work with although the InDesign export to HTML process seems to make sense for this particular client.

If you have a process to recommend, based on your own experience, please let me know.


3 Replies to “Ebook production: chaos and opportunity”

  1. Bill Adams

    I agree that the difficult production problem is design, especially for books that want to use graphics in a significant way. Kindle Format 8 for Kindle Fire (Amazon) addresses this problem to some extent.

    I’ve used Smashwords.com successfully. They automatically convert a prepared .doc file to all other formats (epub, mobi, html, pdf, you name it). The process clearly demonstrates that file format conversion is so highly structured that it can be entirely automated, as long as the input is well-controlled. The result is bare bones in terms of formatting, but it exists.

  2. ag Post author

    Thank you for chiming in.

    I’ve read good things about the Smashwords conversion from Word files. For an individual author with books that are largely text, it may be the way to go.

    But it is not a great option for many of the small publishers I’ve been talking to.

    Their books are in InDesign, which may have had an export to Word format at some point in the past but no longer does. They do have the option of exporting to epub but, as I understand it, the Smashwords conversion from epub is still very much a work in progress, not nearly as dependable as their Word conversion.

    And ‘using graphics in some significant way’ is very much at the heart of the matter for professional book designers. So they are in the hunt for a process that gives them some amount of efficiency and simplicity but doesn’t require they to be satisfied with a totally bare-bones result.

  3. Pingback: Self publishing: a flawed but useful overview · Sheridan Programmers Guild

Leave a Reply

Your email address will not be published. Required fields are marked *