Specs: We'll write these if we have to

The good news about writing functional specifications is that you’ll most likely have plenty of examples to read for reference and a template to use. The bad news is that you might have to write it. Even if you don’t write it you’ll have to read and understand it. You’ll have to read the business requirements and understand those too. As you read these, you should be thinking about the technical specs, what prototyping will need to be done, and where the holes are. One more thing, make sure the eventual users of the system are engaged and sign off on all functional decisions. If they blow off meetings and are not involved in the process, they’ll be back later and will demand changes.

Large companies with a lot of procedures, lawyers, regulations, and outsourcing will undoubtedly require that technical specs be written before starting development. This is fine for about 80% of the project, however the technical details during the development, testing and deployment will change. The technical specs will need to be reworked to reflect this change at the end of the project. If you start out with this expectation, you won’t get as frustrated writing the specs without all the details. Be sure to write down in the assumptions that you expect to rewrite the specs after the project goes live.

As you solve the technical issues keep the following questions in mind:

• Are the User’s of the system served well by this design?
• Are there too many details written down at this stage?
• Are there features which are training issues vs. customizations?
• How much continuous discovery will this project have?
• Will the development or testing be outsourced?
  
  

Are the User’s of the system served well by this design?
Any reservations should be considered and detailed here. In many cases the writers of the specs don’t really know Documentum well enough to fully understand possible scope creep issues and the technical ramifications of writing something like “this system will support nine languages…”
I don’t know how many times the specs were written in a vacuum. The User’s were too busy to fully detail the business requirements so the spec writer reads the manual and fills the functional specs with existing functionality.

Are there too many details written down at this stage?
I’ve read functional specs that have been bogged down by details which will change. Here’s where the lumpers and the splitters show there true colors. The lumpers will write too little detail, while the splitters will describe too much. It’s my experience that approximately 10 to 20% of what you need to develop a comprehensive application will not be known at the time that the functional spec written. Make this assumption, record it and move on. You can really slow a project down by trying to perfect all the details at this point.

Are there features which are training issues vs. customizations?
This question is usually asked if the initial discovery phase was done by folks who did not know the Documentum application well enough. The business requirements would reflect this lack of knowledge, would cascade to the specs. You have to gage how much continuous discovery this project might have. Was the business unit as engaged in the project as it should be, or will they perk up during the testing phase and really start thinking about how they’re gong to use this?

Will the development or testing be outsourced?
I had to bring this up because in these times of going the cheaper route, the project just may be going over seas. I wouldn’t worry though, because every Documentum project that I’ve done is way too involved to ship the whole thing to a third party vendor. Outsourcing pieces of the project may make sense. If you have to outsource get ready to write a lot more to describe what to do. Also, prepare to hand hold the third party through some of the more complex details of the development or testing.

The bottom line is that there’s a balance between the time you take to write or correct quality specs and the time it takes to develop. In some cases, it may make more sense to prototype and write the specs afterward. You’ll have to make this decision and get buy in up front. Don’t just do it; if it ends up taking longer to prototype, you’ll be on the hook to explain why.

Prototyping and proof of concept: This is fun!

This is fun if project managers get that POCs are crucial to the decision making process:

• When someone on the team says “The way I did it at company X…”, but this way doesn’t feel right, prototyping should be considered to detail the pros and cons of the solution.
  
• Every project has at least a few gray areas where the business users aren’t quite sure what they want and say they’d like to “see” something.
  

The fun part is actually working out some of the pieces of the puzzle. It’s also satisfying to prove to yourself and others that your gut feeling was right, or that it was wrong, but not as wrong as everyone thought. Here are some of the pieces to the puzzle that you may be able to figure out:

• Is the customization possible? What is the level of complexity?
• How many resources are needed and what type to achieve the functionality requested?
• Is the estimated amount of servers of the infrastructure correct?
• Will the application solution perform to the user’s expectations?
• Will the schedule slip because of the functionality required?
  
  

Setting a POC sandbox also allows for installation experience and a preview into some of the issues that you’ll have to solve later. For example, when was the last time an installation of a Documentum product went exactly according to the installation documentation? Is your database administrator new to Documentum databases? This would be a good chance to get through the DBA headaches.

What about storage requirements? After loading up some dummy data, does the total disk space estimate make sense? When the dev, test, and production environments are taken into account including extra room for migrations, is there still enough space?

TBO vs. WDK customizations
I really like TBOs. They are very useful to making changes to attributes during all content operations. There will be times when you look at a customization to a drop down in the UI to achieve some kind of functionality and it occurs to you that it could be done with a TBO. Prototyping will help you work through the reality of that idea.

The bottom line is that you will be dealing with scope creep if you don’t prototype solutions when it makes sense to do so. When someone says “we need to customize” , make sure through a test of the functionality of the application that it’s not just a configuration. The WDK can be very flexible in terms of configuration and TBOs are great to use for customizations.

WP installation and configuration: This can be painful.

First of all there’s very little detail in the installation manual on how to actually configure Webpublisher to work correctly. For example, here’s how you first set up a template:

1. To import your first Template file, first you have to know what an example template file should be, then you have to know that to import you must create a new “category” whatever that’s supposed to be, go into that new category and then import. Patched together functionality or what?
2. To edit your first Template, you need a rules file. So you go to the “Rules” area and create one. You go back to the Template, check it in (because it was checked out when you to edit it the first time), View/Associations and click on Rules, and Add the new Rule. Is this intuitive enough yet?

Second, once you’ve installed the WP, you have to install and configure Site Cache Services (SCS). Configuring the website publishing using Documentum Administrator is relatively straight forward, however, when it comes to figuring out what happens during a SCS publish, you’re kind of on your own. One thing to check is the SCS configuration in WP. To do this, click on “Web Cabinets”, select the cabinet that you are publishing, and click on “View” and select “Web Cabinet Overview”. This should show you the publishing configurations that were configured earlier using DA. Here’s what happens during a Site Cache run:
1. Either the SCS job executes or WP promotes a doc to Staging or Active which invokes the dm_webcache_publish method.
2. SCS service reconciles modified files with the configuration and the source SCS tracking table.
3. A properties.xml file is generated and sent over to the target SCS server.
4. Content is transferred via http or https to the target server.
5. On the target, when all of the files have arrived, target SCS processes database inserts and deletes if there’s a database, it expands zip_html files, it copies the files from the temp directory into the live website, and much more.

File extensions: htm vs. html
The first time you site cache the html renditions you’ll notice that the files are published with the “htm” extension. To get the files to publish with an “html” extension, you’ll have to make changes to the webcache.ini file located on the content server in the “/Documentum/dba/config” folder.

exclude_formats=xml
use_docbase_formats=true

You’ll also have to change the extension of the html format using Documentum Administrator. It took me a day of part-time rummaging through the forums to piece together this configuration fix.

There will be times when you want to throw up your hands and give into a beginner WP class, but by investigating this issues that arise first by browsing the forums at EDN and Bluefish and other site, you’ll learn as you go and more importantly remember what you did when you need to configure WP again.

WebPublisher Business Requirements: Who Are We Writing These For?

If someone tells you that Business Requirements are not that important to your role as a developer, ask them what happens if you need to make a change to them? Get involved in this phase if you can because BRs set the tone for the whole project, and you need to meet and get to know all the team players. Most developers get pigeon-holed into roles that suggest that they just do what you tell them to do; follow the specs… It might feel comfortable to do this, but you’ll always feel frustrated during the process, especially if you need to push back on the BRs.

Know Webpublisher
As you sit through the Discovery phase (if you get invited) you’ll gather incite into the issues that really matter for the project and who will be looking for what functionality. What happens during the business requirements writing phase is that everyone becomes an “expert” in Webpublisher. The requirements reflect what the business thinks WP can do OOB, not necessarily what it’s potential is. To be fair, the main objective is to use WP OOB as much as possible, but also to find the middle ground in terms of configuration vs. customization.

Consultants Who Know Everything
If there are outside consultants, especially those newly graduated from our fine Ivy League schools, they will commandeer the discussions about requirements and do breakout sessions to fine tune their own understanding of the requirements and their skills in Excel. Business school graduates are fond of Excel and can achieve incredible feats in expanding BRs to new levels of complexity. These are well thought out, but are usually overkill for OOB WP functionality. Make sure the consultants know WP functionality inside and out before they start writing BRs. The more the requirements grow in size and complexity the less likely that the business users will read all the requirements and sign off on a document they can completely grasp.

Think about the whole application
It’s easy to get bogged down by concentrating too much on the details. For example, there were many times when we drilled down on the requirements only to realize that it was close enough to OOB functionality that we didn’t need to write it all up. By “whole application” I mean:
- What’s the end to end flow of content through the system?
- Who will access the content and when?
- What level of support can be sustained?
- How much training will be needed?
- Do you have enough resources to satisfy all the requirements?

Whiteboard the end to end flow
One absolutely necessary step in the requirements process is to draw out the end to end flow of content so that everyone is on the same page. This flow diagram will serve as a milestone of the group’s understanding of the application. It will be altered and republished, but the core flow should not change. This flow should have a combination of technical and visual elements. It should be of use for the business side as well as the IT side. Everyone needs a visual representation of the application and now is the time to create it.

There will be times when an issue can not be resolved by the people within the team. This is the time when you spin off pros and cons of the possible solutions to the problem. This is also the most frustrating time of the project, when egos are flying and a lot of time is wasted. A few types of issues I’m talking about are:
- Configuration vs. customization
- Integration vs. manual data entry

Configuration vs. Customization
It’s easy to get excited about developing new functionality to WP mainly because WP has many foibles that are usually corrected during the first release of the project. However, you have to weigh customizations with wanting a smooth upgrade path. One of the WP projects I was a part of was so heavily customized that management waited until the version was no longer supported and then had to plan the upgrade to which was scheduled to talk over a year to accomplish. This type of customizing indulgence is usually avoidable if the right balance of Configuration and Customization is achieved.

The typical pros and cons of these issues involve:
- Cost
- Length of time to develop
- Testing
- Functional worth
- Priority
- Impact on upgrading

Integrations
There are all sorts of integrations that you can do with Documentum, the one I’d like to talk about is one with Documentum Compliance Manager (I’ll carry this example throughout this series on WP). The pharmaceutical company I worked for wanted to be able to run certain regulated web content to DCM. A content flow diagram was created and a prototype was developed during the requirement writing phase mainly because no one knew for sure if integrating WP with DCM was even possible. By architecting the solution with a TBO we were able to integrate with DCM without disrupting any of DCM’s functionality.

“Nice to have” vs. an absolute must
When you have the business users in the room they will say everything they bring up is necessary. But, when pressed, they’ll start to admit that some functionality is “nice to have”. WP offers many features that can be configured to get close to most requirements around publishing to a website. I’ve migrated from a custom Documentum web application to WP and the biggest complaint was the amount of versioning that goes on in WP. They also had very complex ideas on how to organize the content. These ideas carried over into WP as categories. WP categories are versatile and powerful when used in the right way, but, in our case, we went way overboard. We had over 10k nodes, multiplied by 10 locales. This was and continues to be a nightmare.

Two thorns of WP publishing
There are two issues that plaque WP: Security and Navigation. Customizations gravitate toward these two monsters. Why hasn’t EMC purchased or built a web server / app server yet? Sitecache is nice, it’s decoupled, it scales. However what happens when the content gets to the website? The coupling of the decoupled security and navigation is a sink hole for bad performance and enormous support issues.

Discovery: do they really know what they want?

As a Developer, depending on the size of your company and how enlightened the management, you’ll most likely get involved with the Discovery phase of the project. If you can try to influence who’s on the Discovery team by suggesting other resources which you trust will “do the right thing” try to do it before the first meeting.

Don’t solve the issues: listen
The first few meetings during this phase are spent listening to the Project Manager(s) go over what they believe are the questions that should be asked and the corresponding answers from the Business Manager(s). Developers: don’t try to solve these complaints in the meeting, just listen and answer questions when asked. Be careful not to ask detailed questions at this point, as the discussion will dive way too deep into the nitty gritty and could get religious. The managers need to hear themselves talk. If you create too many disruptions at this point, they’ll write you off as a pain.

Know OOB functionality cold
I’ve been in situations where I knew the functionality being asked for what hard to implement, but wasn’t 100% sure and couldn’t back up my gut feeling with proof. If I had known WebPublisher OOB functionality better I would have been more effective in arguing my point. One way I dealt with not knowing the answer to a technical question was to suggest that I prototype the functionality and get back to the team with an answer the next day. This allowed me to get proof (or not) of my gut feeling and work through possible solutions without getting shutdown immediately. This also saves lots of time and effort in the long run by exploring the ramifications of a small decision which could balloon out the project (80x20 rule).

Where does the content come from?
The content of the website project was already designed and developed, but was being hosted on a tradition web hosting site. The website content was FTP’d to the site and manually published. The users wanted “everything we have now” and more. The “everything we have now” was a black hole which needed to be dealt with asap. What can you do with a statement like that without going through a huge process of requirements gathering? What we came up with was punch list of what WP does and presented it to the business.

Lumpers and Splitters
On the project team, there will be people who like to drill down on every piece of functionality, not only to understand it but to drive others crazy. They like to split issues into smaller pieces and understand them better that way. Others like to lump issues together and hand off the large understanding to those who understand the software (WP) better. During each meeting the lumpers and splitters will go at it. If you can catch this happening before it gets too heated, call for another meeting to hash out the particulars. It’s important to keep the focus on manageable chunks of the project.

Who’s poking holes in the boat?
During these meetings, at first it was hard to figure out who was aligning with whom in terms of making the hard decisions about what functionality was in and what was out. Over the course of a few meetings, it became clear and it also was obvious who was fully invested in the project and who didn’t care. The folks who didn’t care reacted by saying things like:
- “Let’s cross that bridge when we get there”
- “We don’t have any expertise in that area”
- “We don’t have the time or money to do things the right way”
- “Let’s get a consultant in to look at this, I know someone…”
- “What are the risks involved with this decision?”

Sense of Urgency
Remember that managers like to drive the project to their own schedules. During the discovery phase, if you don’t push back when necessary, you’ll be setting the stage for long hours down the project lifecycle line. Managers want to push the project along no matter what issues arise. Let’s say you’re trying to decide whether to use WP’s import screen the way it is or to customize it. This kind of exercise requires time and attention to detail. If something is missed, you’ll pay for it later.

Your homework
As the high-level, prioritized issues are ironed out you’ll get a better picture of the scope of what’s being asked for. During the priority step, you should fill in a matrix that you make up on your own. Columns that I’ve used in the past are Requirement, Priority, Scope, Effort, Custom/Config, Cost:

- Requirement: This is the specific functional piece of functionality that is being asked for by the business.
- Priority: The weight given to the requirement by the business.
- Scope: All the WebPublisher, technical and project-related ingredients of the requirement.
- Effort: Gut feeling on how much time it will take to accomplish.
- Custom/Config: Are there any customizations? Is it really all configuration? Try to find any holes that are not being discussed
- Cost: Any potential sink holes that will be expensive.

Example:
Requirement: Automate website approval process
Priority: 1
Scope: Design workflow template, notification design, prototype, user testing, rework, application migration from dev, to test, to integrated test, to training, then to production.
Effort: Moderate
Custom/Config: Mostly configuration, possible customization: deal with aborted workflows.
Cost: Developer training, consultants

Cognitive Dissonance
Take note of Business folks and members of your team who do not admit when they are wrong or have made a mistake. These are the people who will sometimes unwittingly take the project of course and will need to be reigned-in when they do so. They are also the people who pull some of the team aside and have “off the record” discussions about the project. They occasionally need ego boost by show boating an experience in the past that they think is important: “the way I did that at was to implement with .

The following are some paths of discovery decisions and where it took WP customizations:

Security
- Implemented RSA Cleartrust
- Had to add 2 servers per environment: 10 in total
- Had to hire an RSA consultant for initial installations
- Spent numerous cycles with basic documentation and endless trial and error troubleshooting.
- Customized the login, denied, and logout pages
- Held up the project as authentication became the critical path during the initial testing phase

Multiple Locales
- Used Categories to manage multiple languages for the website navigation.
- Created 10 sitecache configurations, times 5 for each environment.
- Redesigned the web cabinet structure to accommodate the new content submission groups.
- Trained publishers how to add translations and publish them.
- Created a custom application to integrate with outside translation vendors.
- Dumped huge amounts of money on developing an automated way to process the sitecache target content and metadata.

Automated Navigation
- Designed a simple way to publish navigation changes based on folders and linked content.
- Developed a TBO and customized WP to prototype the functionality.
- After the User Demo, we rewrote the design and developed a much more complex solution.
- This went on for a few more cycles until we had to stop and move on to other aspects of the project to meet our schedule.
- The business said they wouldn’t make any changes to the site folders for a year. After the site went live using WP, within a month they were changing the navigation.

Discovery is a term used for exploring what the business would like to have. What it really means is finding out what is realistic to have now and what is going to be released at a later date. There will always be disappointed members of the team who will never be satisfied and other who will feel over worked. Hopefully you’ll have a balance of these types with the more motivated types who enjoy what they do.

Documentum In the Trenches: WP

I have been working with EMC Documentum's Webpublisher long enough to know a few things about the got yas during the phases of a website project.

1. There is no guide that provides a soup to nuts view into how a website publishing project is designed, developed, and implemented.
2. Webpublisher will have issues, it's just a reality.
3. As with most websites the business requirements will focus mainly on the look and feel of the site, not the mechanics of how you get the content on the site.
4. If you like to follow project schedules closely then you better pad in time because you're going to need it.

With these four observations in mind, I'd like to take you on a ride down Webpublisher memory lane with a project that I recently completed (no projects are ever really done, who's kidding who?). The structure of my remembrances will following the usual project lifecycles:

1. Discovery: do they really know what they want?
2. Business requirements: who are we writing the requirements for?
3. WP installation and configuration: This can be painful.
4. Prototyping and proof of concept: This is fun!
5. Specs: We'll write these if we have to.
6. WP Development: Try not to Thrash.
7. Deployment: That wasn't in the spec!
8. Training: That was easy; until they actually start to use it.
9. Stabilization: Is anyone using this?
10. Support: Who's doing it?