Getting Organized: Practical Guidelines for Documentation Scalability

Editor's Notes

1. Thank you all so much for being here. My name is Richard Rabil, and I am a Sr. Tech Writer at Opower. It’s great to see tech writers come together to geek out about this topic tonight. As I go along, feel free to stop me at a particular slide or point and ask a question or make a comment. I’ll also make time for questions at the end, so feel free to wait as well.
2. To set the context for our discussion, let me tell you about our company. CLICK. Basically, we are a software company that wants to save the world. To do this, we partner with utilities to help their customers save energy. On average, people spend about 5 minutes thinking about their energy usage A YEAR. Opower is on a mission to change that. We motivate people to pay attention to their energy consumption and change their behavior. How do we do this? CLICK. We make products such as Home Energy Reports, email reports, and a personalized web portal. Communications like these tell you how much energy you use compared to your neighbors, highlight when you are using the most, and provide energy saving tips. So… you might wonder… does it work?
3. Absolutely. As of July of this year, we’ve saved 5 terrwatt hours of energy. To put that into perspective... CLICK THROUGH BULLETS. Utilities around the globe are paying us top dollar to help them achieve these outcomes. Which means Opower has been growing rapidly as a company.
4. Hiring: Opower is now a public company, and we’re hiring a lot. Multi Prod Lines: We have 14 products, with more in the works. Iterative: We work in an Agile environment, which means software updates are happening every three weeks. Internal Clients: We’re supporting over 90 utilities in 8 countries, and we have new international clients and offices in other time zones. We’re serving more than 10 million households (it’s probably closer to 20 million now) and 1 million small and medium businesses. Specialized: And we have specialized teams across the company who need varying levels of detail about how our products work.
5. What does Opower’s growth mean specifically for our Documentation team? Let’s take a look at our audiences and deliverables. CLICK. Sales and Internal Staff > Internal Knowledge Base: We support sales and other internal staff who need to know how our products work so they can sell them, make millions of dollars, and launch them for various clients and keep our clients happy. Our wiki as a whole has thousands upon thousands of pages, and within that wiki is a special knowledge base (KB) that our team is responsible for, which has over 800 pages—and counting! We currently support about 600 employees with this high-quality internal documentation. CLICK. Utility Support Staff > Online Help and User Guides: We also support several hundred customer support representatives (CSRs) with guides that describe how our products work so they can answer customer questions. We have online help and print guides, with hundreds of topics. CLICK. Utility Technical Staff > Technical Briefs and Data Standards: We support utility technical staff with technical briefs and data transfer specifications so they know how to set up our program, how to send us data, and do other technical things like that. CLICK. Third-Party Developers > API Docs: And we support third-party developers who use our APIs to get data from us and integrate with our applications. I believe we have somewhere in the range of 50 APIs and we’re documenting. CLICK. My Focus. What I will be focusing on tonight are these three categories, since all this documentation is generated through MadCap Flare, and we’ve had to tackle quite a few scalability challenges related to that. If you’re not familiar with MadCap Flare, it’s a special software program specifically designed for writing help content in a single source, and then publishing that content in multiple formats, like PDF, Word, HTML, and mobile outputs.
6. And of course the other thing is that we have to deliver this documentation in multiple languages. Currently we have four languages (sv, fr, jp, zk) to support in our documentation, and we expect this to grow very soon as we expand into more countries. Raise your hand if you’re in a similar boat. You probably know how extremely challenging, costly, and time-consuming this is. It’s a major challenge to rapid growth and scalability.
7. So what did we do to handle this? CLICK. With the buy-in from our leadership (and we were very fortunate in this regard), we responded like any reasonable docs team. CLICK. We went from two writers to five writers and one writer contractor in less than a year. Some of my teammates are here, and so is our contractor Ryan. And we wrote tons and tons of content! And today we STILL get to write a lot of cool stuff about a lot of cool products. But of course, that’s not the end of the story.
8. Of course, we faced no challenges. CLICK. Only challengetunies. What were they? CLICK. We had to onboard and teach new people, which takes precious time. CLICK. Knowing who changed what, and when: What’s been changed since the last version? Who changed it and why? Can we restore the previous version? CLICK. Content inconsistencies (different styles, structures, voices, names, etc.): Some of our cover pages had a title only; others had a title and subtitle. Our writers were naming sections differently and used different structures and tone of voice. How could we ensure we a consistent voice, structure, and branding across our hundreds of pages of content and over 50 document deliverables? CLICK. Cumbersome editorial maintenance: We found ourselves making similar updates in many places, since we weren’t following the same standards across our projects. CLICK. Content findability problems: We often had trouble finding pieces of content that we knew someone on our team wrote, since the files were scattered and followed different naming conventions. CLICK. Content duplication, not reuse: We realized we had copies of the same graphic floating around. Would one author know that we needed to update it in multiple places if they received a corrected version? These aren’t trivial problems. Cumulatively, they take away valuable time and money.
9. Here’s another way of looking at it. When we started off as a small team, it was easy to make choices, answer questions about where a file was, and hold each other to the same standard of consistency and quality. CLICK. But then what happens if your company builds a ton more products? CLICK. And needs a ton more documentation? CLICK. You hire more people to manage your existing docs and your new docs. My estimate is that we currently have ownership of somewhere between 1,500-2,000 pages of documentation. CLICK. At minimum that’s 250 pages of documentation each person is responsible for on three-week release schedule. And that’s a conservative estimate, and doesn’t count the fact that we have quite a few documents in multiple languages. CLICK. The challenge is not only how the documentation for each product stays consistent. CLICK. The challenge is how you maintain the same level of quality, accuracy, clarity, and so on ACROSS your whole team and hundreds of pages of content.
10. There are probably other ways we could diagnose and classify these challenges, but ultimately I think they all have this in common: CLICK. scalability problems.
11. But what I do mean by that? CLICK. It’s more than a dermatology crisis. In fact, I think we often consider scalability an engineering problem, such as the ability to quickly import millions of rows of data, or handle huge server loads all of a sudden, like when there’s a huge upsurge of tweets on Twitter, and Twitter’s servers crash. CLICK. But actually there’s a more generic definition that I think applies to us. Scalability is the ability of a system – such as our processes, folders, files, and standards -- to handle growing amounts of work in a capable manner. In other words, in a way that minimizes duplication of effort or wasting of time as we create more and more files over time. We don’t want to be doing repetitive tasks—such as hunting around for a file, or resolving minor inconsistencies, or fixing broken references--that add minimal value to our business.
12. The main point I want to get through is very simple: CLICK. Scale happens. You might be in a place where you’re growing fast, and you have a ton of work to do AND you’re given the budget to do it – or maybe not. But you’re adding tons and tons of files and growing your team in order to meet demand. When you face this situation, proactive planning and design is the cure.
13. I know that’s vague, so let’s unpack it. CLICK. Same Page. When it was just me and Rebecca, my Docs Manager, making decisions was easy, and we could keep each other in the loop and have oversight of a manageable number of files. But with more people in the mix, you lose consistency and face chaos because no one is following the same standards. CLICK. Proactive Planning. When you face these decisions, you can be conscious and proactive about them, which tends to more order; or you can be near-sighted and informal about them, passing them off as bothersome and irrelevant, which will tend towards more inconsistency and chaos. CLICK. Long-Term Benefits. When you design your folders and files for consistency, scannability, and so on, you achieve small increases in time savings, which have enormous cumulative benefits. CLICK. My Goal. My goal is to prepare and empower you to be proactive in the choices you make. A lot of it comes down to anticipating future challenges, weighing pros and cons of different choices, and developing clarity and resources your team can use to be consistent and hold each other to the same set of standards.
14. Now again that’s all very high-level, so get let’s get more specific and talk about three big categories of decisions you’ll face. CLICK. First, you’ll face questions at a very broad level about how you’re going to collaborate and share files. Do you use a version control repository or a shared drive? How will you set up your team’s shared file repository to optimize order, findability, and long-term growth? CLICK. Second, you’ll face project architecture choices, which simply means design choices about how to optimally set up and organize your projects. How do you set up your project folders and files so that there is minimal manual effort involved in maintaining them, and maximum capability to expand them in a scalable way? CLICK. Third, I’ll get to the most granular level, namely, choices about file management WITHIN your projects, getting into things like file naming conventions. The emphasis here is going to be on how to manage your writing projects and files in a scalable way. Our team uses MadCap Flare as our primary authoring tool, so many of my examples are based on our experience with that tool. Even so, much of this stuff should still be more widely applicable.
15. Let’s start at the very highest level. How will you collaborate and share files? This is the necessary starting point because it lays the foundation for everything else. CLICK. The big related question here is, will you stick with a shared drive or a version control repository? If you’re not familiar with what version control is, it’s just a central repository for all folders and files that tracks changes over time. It is incredibly useful for collaboration. Raise your hand if you use a version control system like Git, SharePoint, or Apache Subversion. If you use version control, you can go back and restore previous versions, etc., whereas you can’t do that with a shared drive. CLICK. Now you might wonder: does it really matter? I’m doing fine without it. To which I respond, for how long, and for how many people? I would argue that if you’re dealing with a team of two or more writers, you’re working across multiple products and deliverables, and you expect a lot of future growth, then yes, it matters a great deal. CLICK. In fact, you might go crazy without it, because as you grow, you will increasingly need to have a way of seeing WHO has changed WHAT, at WHAT TIME, and WHY. It is a critical way of holding each other accountable to the same standards and having a safety plan if something bad happens and you need to restore a previous version. CLICK: Think of it this way. It’s as useful as mind control, but less creep. I strongly recommend you adopt it if you’re not already using it.
16. Regardless of what you choose about version control, you have to ask, how will you organize your shared repository as you grow? I am talking about the very top level of your repository. This too is an inevitable series of choices your face, and which you can tackle them in a strategic orderly way, or an ad-hoc, non-orderly way. CLICK. Ad HOC Design. Here’s an example of ad-hoc repository design. Pretend all of these first level items are folders, and everything with a file extension is a folder. This organization grew organically as people joined the team, with little thought to optimal organization. There are inconsistencies in naming and hierarchy. Some folders have spaces and underscores, others don’t. There’s obviously chaos. Imagine a new writer joins your team and is looking at these folders. Imagine a new team member ADDING to this mess without any guidance. That’s a big risk for adding chaos to an already-chaotic situation. CLICK. Enhanced Design. Here’s a draft design for updates to that repo. The naming is much more consistent to facilitate scanning, which is a small increase in time savings that has a good long-term effect. Folder names are shorter and simpler, which makes maintaining and editing them a little easier. And locale folders have been added in to support localization, to accommodate future business growth in i18n locales.
17. Obviously those were very specific examples; they won’t fit your situation. So what are some general guidelines for making your own organized repo? CLICK. Draft an Outline! Easily this is the most important point. Make a draft using the actual folder names your anticipate creating, and incorporate any naming conventions you want your team to adopt. Simulate how the folder would actually look, with alphabetical ordering. This is as easy as making a bulleted list in a document. Add in examples of actual files that will reside in the folders. This will really aid the discussion you have with your team. CLICK. Flat Hierarchies. Flatter is better, and you should plan this from the beginning. But why? This is an important theme that will emerge again and again, so hang tight, we’ll come back to this. CLICK. Folder Names. We chose to avoid capitalization altogether, use hyphens at the top level for folders, and keep them as consistent as possible. For example, don’t use a phrase for some folders, but nouns for the others. Why? Because this generally facilitates scanning, reduces cognitive burden, and is easier to maintain over time, because there’s less you need to remember when editing or creating new folders, and fewer keyboard combinations to hit. It’s a small choice that has long-term, cumulative positive impact. I’ll show examples and talk in more depth about naming conventions later. CLICK. Software Access. In our case, we needed to work with the engineering team to plan for the fact that a software program—namely, a Jenkins job--would need to look in this directory, go to a specific locale folder, and pull out the appropriate help content for deployment with the appropriate web application. It’s just part of the deployment process. If this is something you anticipate, talk to your engineers, because you will need to make sure the file path/directory structure that you create makes sense to them and their software architecture. CLICK. Locale Folders. I suggest creating folders based on locale codes, even if you only have one language right now. In that case just put everything under en_us, because it’s likely you’ll need these folders if your company expands internationally in the future. CLICK. Technical Constraints. Talk with your technical SMEs up front about this so they can help you identify any applicable technical constraints with your folder system and naming conventions. For example, maybe your engineering team prefers dashes instead of underscores due to potential limitations with a given version control system. Also, we had a case where Github didn’t like some of the file extensions that Flare automatically created, so we had to work with an engineer on a workaround, and eventually it was such a huge pain that we reverted back to Apache Subversion. Time-consuming. Costly. Incredibly annoying.
18. Let’s drill a bit deeper into the flatter is better argument, because this is a recurring theme. I started a forum topic on linkedin with the madcap flare users group, and out of the six people who replied, only one person really liked folders. There are pros and cons, which we should make explicit so we can make more informed design choices. CLICK THROUGH AND EXPLAIN. NOTE: At scale, I think the pros easily outweigh the cons. Just adding one more level seems so simple and easy for you in the moment, but it rarely brings gains in the long-term. At the very least, be wary of it, and seriously examine if and why you need it.
19. Now let’s talk about project architecture. This just means how you organize your projects and link them together to make them as future-proof as possible. So the theme of this category will be to design your projects with the future in mind. Of course, that’s the theme of this whole presentation, but it will have special emphasis in these slides. I will be focusing a little bit more on help authoring tools like Flare, but there are still some good general principles for everyone here.
20. So if you’re using an advanced authoring tool like Flare or Robohelp or even Microsoft SharePoint, you have to ask a crucial question: at what granularity do you create your project folders or project repos? For example, should your whole team share a single master Flare or RoboHelp project with subfolders for individual projects? Or should you create multiple project repositories that are not embedded within a master project? I realize not everyone is in this boat, but you very well could be in the future. To help empower you to make this decision, let’s discuss the pros and cons. Then I’ll tell you what we chose and why.
21. CLICK THROUGH PROS AND CONS.
22. Let’s say you’ve decided to go with multiple projects, which is our situation. Somewhere in your shared team repository, you will need to choose where the source content for your projects will go. Here are some guidelines for doing that in a scalable way. CLICK. Keep a flat list and use acronyms for the project folder names, to keep directory paths shorter. CLICK. Your projects may need to be associated with a particular language or release. It’s a good idea to design for this and plan with your engineering team if you need to. CLICK. Find the scheme that’s best for you. I recommend organizing project folders by the products your company makes, because you’ll probably have many different deliverables and audiences associated with it. CLICK. Here’s an example of the kind of hierarchy. We use. Imagine this is our root directory, and then there’s a projects source. Within that we have source projects in different languages. And within THAT, we have a flat project list. Obviously this won’t work for everybody. You need to find the one that works best for you, given factors like localization, release versions, and future growth.
23. Some of your content, such as your headers, footers, logos, product definitions, will be—universal across your projects. How do you make sure you reuse those resources and thereby save yourself time and maintain consistency? One way of doing this is a technique that Flare calls “global project linking.” So what is it? CLICK AND EXPLAIN. The big benefit here is that you can update these common resources in one location and reuse it elsewhere. CLICK. But there are some Scalability Considerations you need to keep in mind. Guidelines. Our team had to write guidelines about when to create a snippet, image, or other reusable file in the global project versus a child project. Renaming. Know that renaming a folder or file in the global project might create complications for wherever that updated file is imported. Explosion. Child project explosion is very tempting in this arrangement. It’s easy to simply create a new child project for all your unique scenarios. The danger is that this increases the number of places where content is stored and makes cross-project updates much harder. So you should seriously consider whether you can just write the new content in an existing project, rather than adding too many child projects. NOTE: If you don’t have global project linking, I still encourage you to set up a global resources folder where you can store common content for your projects.
24. There are a number of things you can do to save time for your team and maintain consistency. Some of these ideas are generic, others are specific to tools like Flare. CLICK. Information Models: Basically this just means detailed outlines and rules that guide the structure and flow of your documentation. For example, what sections should go in a typical user guide, and what should the standard tone and flow be? Experts in the field recommend information models; I don’t have time to get into the specifics, but I can recommend good resources on how to make them. CLICK. Variable Sets: Variables are incredibly useful. A variable lets you update a product name in a single place ONCE, and then it gets updated everywhere. And rather than having a single list of variables that explodes with time, you could have multiple sets. This is an idea I got from Lynn Carrier at an MC Flare conference. CLICK. TOC Templates: This is another cool idea I’ve heard about. Create TOC templates to demonstrate how to organize topics, section breaks, and so forth. The benefit is increased consistency across your growing docs set. CLICK. Target Templates: With target templates, you make it easier for you whole team to be using the proper settings when they generate outputs. CLICK. Written Policies and Guidelines: Our team has internal guidelines on how to use Flare and reuse content so we’re trying to be consistent. These are critical things to define as your team grows. Otherwise you create Flare projects in a vacuum and run the risk of lots of inconsistencies.
25. Lastly, file management. The key idea I want to get across in these slides is that if you are not proactive about managing your files, they will inevitably get out of control, and it will seem like they are managing you and your precious time.
26. We’ve looked at the project level already, so now let’s go a level deeper and ask, how will you organize folder and files in a scalable way at the sub-project level? I’m going to use a lot of examples from MadCap Flare at this point, but I still think much of this widely applicable. CLICK: Some Already Organized. The good news is that if you’re using an advanced help authoring tool, some folders are already created for you, and you don’t have to worry about organization too much at this level. The real challenge is how to organize your topics, images, and snippets, since you will create a lot of these files, and they can quickly explode out of control. CLICK. Avoid Levels: Constantly assess how badly you need additional folder levels, because this can have a real impact on the length of your file path and long-term maintenance and content findability. If at all possible, I would avoid going two or more levels deep with your folders. CLICK. Weigh Pros and Cons: I’ve heard of a bunch of different schemes for organizing your topics, and I hate to say it, but there is no single good answer. The list here shows some common dimensions by which people organize their topics. You need to plan for and design for the one you think works best for you. NOTE: Some schemes work better for different content types. For example, I think “by product” is more useful for snippets and images. I don’t recommend organizing topics by deliverable type , locale, or audience because there’s usually so much overlap between them. CLICK. Example: Here’s an example of how we organize topics and snippets. You can see we have folders for main sections of the document. As for snippets, there are folders for each product acronym. We don’t go any deeper than that, and it works for us. CLICK. Organizing by Tag: I think this could potentially be helpful. Upside: avoids folders. Downside: maintaining the tags could be very difficult.
27. At last we’ve come to naming conventions. This is incredibly important in a rapidly growing environment. You have a lot of options on how to name your files. For example… CLICK THROUGH OPTIONS. The fact is there’s no silver bullet. I’m not going to recommend that one of these is necessarily better than the other. Each and every one of these has benefits and downsides. Maybe one day I’ll write a quick reference document that considers all these variations and weighs the pros and cons. But today I will give you some practical guidelines on factors you need to consider when making this choice.
28. CLICK. Length: The length of a file name determines how many total characters are in a file path, and this could result in errors in your tool. CLICK, I bring this up because of the dreaded error message you can get when building an output in Flare. In light of this, using camel case is very attractive. CLICK. Readability. You want to make your file name understandable, distinct, and usable in case someone new comes along and is trying to determine what the file name means. Of course, this conflicts with the first bullet, and so you must strive for a balance. CLICK. Consistency / Scannability: Consistency is one of the most important factors for scalability. For example, see this screenshot from one of the MadCap Software application file lists. Because of the naming consistency, this list is much easier to scan even though it’s long. Why? Because consistency greatly simplifies filtering or scanning lists of folders or files. It improves how well files appear when organized next to each other and as we’ll see it also helps the localization process. So we have to keep consistency as a top goal, because the business value is real time savings. CLICK. Technical Constraints: Technical constraints are things like having being required to avoid special characters if your version control system or operating system makes you. For example, Flare recommends avoiding spaces in your output files names in UNIX systems. Another consideration is that spaces can end up substituting breakout characters, resulting in messy URLs. CLICK. Recommendation: Different Conventions for Different File Types. Not all files are the same. In flare, there are images, topics, variables, targets, and snippets. You don’t have to rigidly apply the same convention to every file type. Maybe one of the conventions on the previous slide works very well for image file names, whereas another convention works very well for your published PDF and Word file names. We have different conventions for different files types; you should consider doing the same.
29. Whatever you decide, it’s important to document your naming conventions clearly and in way that’s accessible to everyone on your growing team. “Clearly” and “accessibly” are key here. Don't shortchange your team. Think about when new people start. Think about when you're in a hurry and need a reminder. Having a documented reference point is incredibly important and useful in those moments. Use principles of good writing. CLICK. Here’s an example of how we’ve documented our conventions. I’ll run through some good practical ideas on how to do this in an optimal way. CLICK THROUGH.
30. You can make all these great decisions along the way, but if you don't take the time to document them clearly and in a way that your whole team can find and understand, it's as if they were never made. And if you don’t enforce—if you don’t hold each other accountable—they’re completely useless.
31. I know it can seem like all of this is mind-numbering minutiae. But I think we should embrace these choices as a part of the art of technical communication. What do I mean by that? CLICK. Take a look at this quote from a pulp fiction writer in the ‘50s. I would argue this is as true for technical writers as it is for engineers. We thrive when it comes to making order out of chaos, and we love to obtaining clarity out of complexity. So in many ways, I think is just one more extension of what we already do, and we shouldn’t be afraid to tackle it head on.
32. So now, I’m going to open the floor for questions. Does anybody have any questions?