Infinite Blend : Info Management

"With Liberty and Justice for Some" applied to ECM

2012-01-28T10:20:00.000-05:00

Glen Greenwald's book, "With Liberty and Justice for Some" can be applied to the world of enterprise information management. In my 20 years of watching content management projects gain political momentum inside companies, be rammed through the process usually ignoring the requirements of key groups, finish with fanfare and pomp, and then IT moves onto the next big project. This is not new, however Greenwald's point that we need to resolve the issues of the past in order to move on fairly and responsibly is crucial.

No one wants to look back on an ECM project's successes and failures, especially given the fast pace of software technologies. When I'm done with a project, the core software has been patched and a major upgrade is approaching fast. If the teams involved with designing, developing, deploying, and testing the project could sit down and work through the issues (because there were no doubt issues: always are) retrospectively the whole company is that much more richer in understanding and agile in their pursuits.

I know, this is called a "post mortem". All companies do this or at least try halfheartedly. What I'm really getting at is true analysis of who the bullies are in the process, which groups got shafted, who broke the rules for personal reasons, etc. I concerned about the characters involved; what they did that was constructive and what they did that obstructed the process. In most companies a "post mortem" is done by the IT team, maybe including a few doers from the business side. Management does not get that involved, unless they want to fire someone or control the situation (I know pessimistic) .

If management understood the underlying negative impact that some of the projects and software used impose on their workers, they would demand closer scrutiny of every project. Instead, management seems to be more keen on the next big thing, leaving the workers to scrap together new ways to design and develop the mess they were just handed on their own. I say mess because information projects never really end; they have a "long tail". Some information management projects eventually fail because they were not nurtured and fed the right amount of emphasis and attention that they rightfully deserve. Just as Greenwald says that the elite have impunity with the rule of law, so to companies and managers as whole need to take their share of the blame, retrospectively, when an information management project fails.

Demos: Not as easy as it sounds

2012-01-13T14:31:00.003-05:00

Creating a plain vanilla software demo is easy. You follow the tutorial if you haven't used the software before and create a simple solution to show a potential client. That worked years ago. Now demos are supposed to be multi-tiered catering to new users and experienced users alike.

What is usually missing during demos is a focused solution which solves the business problems that have brought the potential client to see what the demo is all about. They have most likely seen the same vanilla demo over and over again if they are being pitch an xCP solution.

What the client is looking for is your understanding of their problems, that you've been there. Some can see the potential of the demo to solve their problems. Others will want a business analyst to give the details of how the requirements of the business will be matched with the functionality of the software.

The problem is that every business is unique and are seeking software to fix issues with their processes or content management or collaboration. They have gone to the trade shows, they have read the books, they want to move up in their companies. Now you have to somehow convince them that the software can get them their promotion, or at least make them look good.

If you have a fast talking sales guy in the room make sure you get a feel quickly if the clients are comfortable with his energy and fluff. If not, have the demo talk. If that doesn't work, bring out the business analyst who has been in the trenches. If they chew him up, walk away.

Quick Overview of Designing a Basic DCTM Solution

2011-12-22T20:22:00.002-05:00

Make sure the businessrequirements are complete and describe use cases on how the users plan to usethe system. The users might say they want everything the "old" system has, butwork with them to be as detailed as possible.

Content

List out the attributes whichdescribe the assets.

Custom objects types should becreated where the attributes are not standard and need to be entered in,validated and searched by the users via forms on the application server.

Security: Users/Groups/ACLs

Access Control must be detailedout and matched to the custom object types. It may be that you will only haveone custom type and one main ACL.

Users need to be mapped togroups
Groups to ACLs
ACLs to folders/objects

Lifecycle

These are used to automatechanging attributes on content as it goes through stages of development fromDraft, Review, Approve, Obsolete, etc.

Business Process

Using the Use Cases as a guide,create workflows which have activities that follow the procedures forpublishing and/or storing assets.

The key to find content is todescribe it well enough so that users can find it better than they do now.

Folder Structure

Keep folder structure simple,2-3 levels deep max if that, and rely more on search to find content.

Most important steps:

Documentfunctional and technical specs

Build prototypes for the users.

FAQ for Information: Frequently Needed Information (FNI)

2011-10-20T13:09:00.004-04:00

How many times have you vaguely remembered a project you worked on a few years ago and wanted to review it as a template for what you are working on now? Frequently Needed Information is a need, like tags and metadata, but a need that is thoughtful and idiosyncratic to be put in a bucket for you only.

When you need it, you know where to look. Like a drawer in the middle of the kitchen for quick access of useful utensils. The drawer has implicit shared knowledge as well, as members of the family know to use it as well. This is not Enterprise 2.0 or social media hype, this is essential information readiness.

Is there a feature like this in Webtop, CenterStage, or xCP? Not yet. Most frequently used is not necessarily what I need to look at for the next project. Should this be automatic or tagged? Should I only have to tag the content in the authoring program and not the from the content management system application UI? There's room for improvement.

Jeremy Rifkin knows how to build the pillars of the "Third Industrial Revolution". If energy creation is going the way of distributed green energy, then why can't we figure out distributed ECM? Is it that centralized control and production of ECM software is still too proprietary and entrenched in our hierarchical businesses?

Decentralized pods of information gathering and projection could emerge as a possible solution. Every business unit would be on its own, but have to play within the rules and make sense to the whole or they will fail. Business units already have relationships, and control patterns in their normal daily tasks, now they need to fashion their use of technology to these habits.

The "man" is turning into the "people", we just don't realize it yet. Facebook realizes it with relationships, now it's time for business to let go of the reigns of information and to start the new information revolution within their companies. Rules and regulations will still dictate process and procedures in some aspects of information gathering, however individuals and groups will slowly gain more and more license to gather their own utensils in the kitchen drawer and use them effectively. Enterprise 2.0 should really be Distributed Information 1.0.

DCTM Outage Scenarios

2011-09-24T19:43:00.003-04:00

Outage Windows
Typically, weekend outage times are acceptable to users of the system. These times will be used to deploy most fixes or upgrades.

Upgrades
Each server has specific requirements for OS upgrades as well as application upgrades. These upgrades may require downtime. Separate evaluations will have to be done by reviewing the risk matrix to determine the amount of integration dependencies.

Server Failures and VM Clones
Services on servers fail for a variety of reasons. Each server should have a recovery policy associated with it. For example, a clone of each server could be maintained for fast recovery of that particular server.

Routine Maintenance
Occasionally, patches will be applied to DCTM software installations. These patches may require restarting the services.

Break/Fix
For problems with individual applications on servers, a procedure for fixing the issue in development, testing in Validation, and deploying to Production will be followed.

Failover
The SLA required by the GxP rules state that 4 hours is acceptable. This means that HA for the DCTM is not required.

The risks for service outage

2011-09-22T16:03:00.002-04:00

The risks for service outage can be broken down into threecategories:

Server: each server has services which are vulnerable tooutage. These servers are the Content Server, Index Server, Application Server,Database Server, and the Storage Server.

Systemic: The dependency of each server’s integration(s) witheach other is vulnerable to outage. For example, if the content server goesdown, the application will be out; if the database or storage goes out, thecontent server is down, etc.

Disaster: This would mean that the whole server room is down.The disaster scenario would cause the DR system to synch and start up.

The risks of services going down are real and happen most often at the server level. User complaints occur during times when performance is slow which may be a sign that a service is in trouble. Many times integration between DCTM and other services are risky because it is assumed that the other services are always up. If a company is growing, the network will be changing, databases will stumble, even electricity circuits will blow, so keep all of this in mind and in your recovery plans regardless of assurances that this "will never happen".

Risk Matrix by Server

Scope	Server Outage	Description	Integration Dependency	Risk Level	Monitoring
Systemic	Storage App	Storage Services	Database, Content, Index, App	Low (If HA, redundancy)	monitoring scripts
Systemic	Oracle	Database Server	Content Server	Low (If HA, redundancy)	monitoring scripts
Systemic	LDAP Server	LDAP	App/Content Server	Low (If HA, redundancy)	monitoring scripts
Systemic	DNS Server	DNS	All Servers	Low (If HA, redundancy)	monitoring scripts
Server	DCTM	Repository Services	App Servers, Index Servers	Med (If standalone)	monitoring scripts
Server	DCTM	Java Method Server (JBoss)	Index agents, Jobs, workflow	Med (If standalone)	monitoring scripts
Server	Application	Tomcat		Med (If standalone)	monitoring scripts
Server	Index	xPlore Servers and Agents	App Server Search	Med (If standalone)	monitoring scripts

Disaster Recovery systems are replicated systems which constitute a low but viable risk.

In the Aftermath of EMC Sales and Sales Engineers

2011-09-06T14:35:00.002-04:00

Any consultant who has landed a project after EMC sales and salesengineers have "sold" the DCTM software suite knows that resettingthe client's expectations can be a challenge. The motivations ofsales and implementation are two completely difference animals. EMC Sales wantslicenses and commissions, Consultants want to design, develop, and deploy thebest possible solution (ideally). The intersection of these two perspectives isthe customer who, more times than not, ends up feeling deceived and gipped.

So how do we accommodate the claims of EMC sales? First, acceptthat the client will want more than the software can deliver. For example, ifthe sales engineer said InputAccel for invoices can learn automatically how topick up line items from an invoice, then you need to immediately explain infuller detail what validation means and the steps taken for IA to actually “learn”the layout of an invoice.

Another example would be that it takes a few weeks to implement anenterprise wide solution for content management. If you installed the vanillaproducts and walked away maybe, but the client would be left with a car withouta clue how to drive it and no roads to follow.

Second, do not make promised that you know you can’t keep. If youbid low to get a project, get ready to pay the consequences. Be honest and ascomprehensive as possible. Show the client the details where they will have topay more to accomplish what EMC sales had envisioned for them. The client wantsa great deal and everything for free, but it is your job to bring them back toreality.

The Intersections of Metadata, eDiscovery, Taxonomy, and Records Management

2011-04-24T19:26:00.000-04:00

Designing and implementing systems which manage content (outside of creating/reviewing/approving content) such as metadata, ediscovery, taxonomy, and records management can be a challenge if done in a vacuum. Each of these systems of content description and rules has intersection points with one another.

I have witnessed what happens when one system is designed without taking the others into account: change management nightmares. Think about freezing a file plan for RM and then having to change it… Add to this, CMIS and other web services which try to do similar actions on content and we have a web of interactions that collide and push and pull on each other.

Why do monolithic ECM companies have to apply layer upon layer (apps mentioned above) of abstraction and rules and xml configurations to do very core things to content? Because it sells new products and brings in market share from other vendors: it allows ECM companies to grow and to please their shareholders. So where do the specialized products and do-it-all products meet? At the following components of description and action:

Import

Tip: Plan for developing ways to migrate large amounts of information into the repository.

A mechanism to get content into the repository and to describe it as table entries in a database that points to the file location.

Attributes

Tip: plan for inevitable change with a metadata repository.

Describe the important/pertinent aspects of the content for the purposes of discovery, 21 CFR part 11, ISO-15489, MOREQ2, SAS 70, etc. Describe for the rules and regulations not for the applications. Describe for the audit. Describe for the User trying to process invoices through approval and payment.

Finding

Tip: Google is Google for a reason, don’t try to copy them.

Searching for content especially lots of content is a major challenge for large repositories. For lawyers trying to find content pertinent to a class action suit this can be good or bad depending on the company’s strategy and how it weighs the fines for not auditing correctly vs. finding self-damaging evidence. The key is how to handle results and this is still not in its infancy.

Browsing

Tip: think about the ramifications of completely changing the folder structure or overlaying it with multiple ways of “seeing” the information.

Many companies discount how powerful the folder hierarchy metaphor still is. They through content into repositories and hope they can find it through searching. Only later do they figure out that folders can be thought of as virtual in the sense that they can change in structure and labeling without disrupting other ways to find the content they need.

Export

Tip: be careful of file names and deep folder paths.

Get the content and metadata out of the system for discovery, migration, or long-term storage. The issues are getting the attributes and audits together and maintaining the context of the content with locations and original modification dates, user data, and validation.

Disposition

Tip: here’s where poor metadata and lazy content management really cost a company huge bucks in maintaining backups of worthless content.

Delete unwanted content period. If you are like some pharma or financial companies and send all of your old content to Iron Mountain, you are a hoarder and should seriously look at your retention policy.

Parsing the xCP Buzz Words

2011-04-16T16:05:00.001-04:00

Taking all of the buzz words out of the Documentum xCP pitch we’re left with “accelerated”, “content” and “platform”.

Accelerated

If a client would accept building a TaskSpace application from a napkin in their production environment, then I’d say this accelerates the application build process, but it no client’s IT department would allow it. TaskSpace apps are built in development and deployed to test and prod via Composer, quick right? Well, what about requirements and functional aspects of the solution? Is that made quicker? No, and here’s why:

Let’s say we have a solution where we scan/capture invoices, process them, and finally report on them. Easy, just like the end to end slick demo that EMC sales did, right? Install InputAccel and you’re done? Install Forms and Process Builder and slap together a workflow? I don’t think so.

The problem with smoke and mirrors is that we as solution architects and developers get blamed for how slow it takes to build a solution that the sales guys touted as a piece of cake, 3 months to build max. The 3 month schedule should be more like 6 to 9 months. The sales guys are long gone and the customer is annoyed and start to cover their own asses as the bean counters are tapping their fingers.

These products might be easier to use for cookie cutter solutions, but what about that 20% of a solutions that doesn’t fit the mold? You need requirements which take time, you need functional specs to setup up the configurations for scanning, forms, processes, use-cases, etc. This takes more time than is usually allotted. This is not accelerated.

Content

Where is going to happen with the old content? The legacy stuff needs to be migrated. Where are the requirements for this? What are the new attributes and object model for the new system? What is the mapping of old to new attributes? The sales guys didn’t talk about this. This is not part of the acceleration.

Platform

The “platform” is still a mashed up combination of export connections (InputAccel) and xml integrations between Forms Builder and TaskSpace and the Content Server. One application has variables, the other has attributes. One can parse scanned pages, the other reads a whole document. In order to put the whole solution together you have to be part developer, part UI designer, and part lucky. The reporting aspect of this platform is an afterthought and with BAM can bog the whole Java Method Server down to a halt.

Next generation of xCP

The next generation of xCP needs to address the following:

Better coupling between requirements and functional specs, configuration, and validation of configurations.
A smoother ride when developing/configuring the pieces of the solution puzzle in terms of common language of computing as well as nomenclature in manuals and tutorials.
Build on open source platforms which are in common use, take a tip from Alfresco.
Slowly eliminate the bottlenecks of configuration, for example, on a large project for each product there will be experts who are assigned to work on their one piece, however they always seem to hurry up and wait for others in the config chain to finish, or make changes.

Documentum's 2009 "Roap Map": Optimism to Reality

2011-03-28T16:00:00.001-04:00

Looking back on EMC Documentum’s product “Road Map” announcements and the hype revolving around them shows how marketing works and how it tries to scare or lure customers into upgrading and/or buying more products.

I read an article entitled EMC World 2009: Beyond D6.5, A Product Roadmap written by Pie in May, 2009. It was very informative with all the buzz words and I’m sure accurate for that moment in time. However, there were statements like “D7 not 2009, will be 2010”. This gives the impression that EMC is moving quickly on its major releases.

It is March, 2011 and we’re at D6.6. I’m not going to bet that D7 comes out this year. I remember reading Pie’s article and being excited to work with DFS and CMIS. I was also surprised at how many installations were still at 5.3sp2. I’m working on an upgrade right now from 5.3sp2 to 6.6. This is a large installation which requires lots of planning and coordination. A year ago, I looked at a 4.3 installation still going strong...

The owners of legacy DCTM systems wait until post support ends and then they leapfrog over many releases. Which technique is best suited for your company depends on many factors, but I’m willing to bet that they save a lot of money by waiting. There is less disruption of User experience as well. The IT department suffers because the technology is antiquated by the time of the upgrade. Three CIOs have probably swung through the company.

Taken with a grain of salt, these “road maps” usher in new excitement about technology X.0. These unveilings show us what we want to hear, they excite us, they allow us to dream of new interactions, new trends, new connections. I enjoy these “road maps”, but I would rather they were called “dreamscapes” instead. Like most companies that are strapped these days and cutting IT budgets, EMC needs to figure out ways not to over sell. They need to tell it like it is, which is a map with changing roads and the distances will always be longer, the scale will morph, and their products will slip in and out of relevancy to the hype.

EMC Documentum's longevity has to do with the product's orginal vision, not the over selling antics of sales and marketing.

Kicking the Share Drive Habit

2011-03-25T14:42:00.000-04:00

Business productivity in the world of “just get this thing done already” means using whatever MS Excel can offer to track the content out there and report on it. This process usually equates to lots of manual double checking, lots of verification, and lots of busy work. Lots of human intervention is not a bad thing, however at some point the “finding” and “versioning” become unwieldy. Some type of content management system must be purchased.

The purchasing, deployment, and education of the first system is the most crucial step in setting the stage for the future health of the organization’s information. It was not that long ago when IT Directors were saying that content management systems did not belong in the enterprise services stack. Some industry IT shops still harbor misguided and regrettably wrong impressions about the complexities of information and especially content and process automation.

So here’s a guide to moving content off the share drive and into a content management system.

One of the key concepts that is missed by most ECM vendors is that each bulk import/migration needs to be executed with a certain percentage of customization in order to get the best results. You could bulk import with an off the shelf tool, but don’t expect to get all of your content into the target content management system.

Remember: the cost of exceptions could add up to more than the cost of the off the shelf tool.

Analyze Your Content and Metadata

Use a file listing app (whether it’s off the shelf or home grown) to build inventory Excel/CSV files with the following criteria:

• Absolute file path for folder and file location

• File names

• File properties such as creation date

• Create drop down lists based on the target content management system

User names
Fixed values like state or vendor name
Destination folder paths

Use or Create an Import/Migration Application

• Incorporate the work done in the Analysis

• Design it based on the specific requirements of the import or migration at hand

• Pre-Flight of Excel track files

• Does the file exist?

• Are the date values valid?

• Do User names exist?

• Are Illegal characters handled during the migration?

Make Validation Simple

Using the absolute path values of the source content, make sure the target has an attribute for that same value. This will make is much easier to not only validate the results, but to recover from failures where rerunning thousands of imports (for one failure) would be incredibly inefficient.

Fear and Loathing in Content Management

2010-12-22T15:46:00.000-05:00

You know when an atmosphere of fear and loathing has a grip on an ECM team when no content gets deleted, all customization is outsourced, communication is restricted to your immediate manager, and there are no formal business requirements. Boiko’s “Laughing at the CIO” comes to mind. It could be my present to the CIO this year…

Nothing Gets Deleted

“Oh no, we can’t delete anything. The boss is afraid to delete anything, he came from quality and we really wouldn’t know what to delete anyway.” Have you heard this before? Are there too many issues to tackle before even thinking about records management and retention schedules? Fear of unknown reprisals should a vital record be deleted is very real if there are no principles and understood requirements around the disposition of them.

All Customization is Outsourced

This is a cover your ass maneuver under the guise of saving money. If you fear being responsible for actually understanding and building solutions to company specific issues like integrations and metadata management, why not outsource. The long-term effects of this are a skeleton crew of implementation and support workers who are board and loath the lack of process and career potential. When most technical design decisions are made outside of the ECM group, there’s usually one manager who knows most of the details and hoards the knowledge for fear of losing his job. His employees loath his shortsighted design decisions.

Communication Restrictions

“There’s a certain protocol here. If you have an idea on how to improve something, you talk to your direct boss, who talks her direct boss, who talks to his direct boss, and so on.” This is a symptom of fear of being made obsolete by contributors who are smarter than you are. Ideas and complaints, both should be shared freely. Communication is tipping in the desert at best.

Business Requirements

Are there requirements for the ECM system? Have they been updated? Have the business users been doing whatever they have wanted without any standards for metadata, templates, workflows, etc.? You’d be surprise how many ECM systems are still dumping grounds for whatever the business want to through in there. If users are complaining about performance and search then chances are very good that users should be self loathing because they’d brought these issues on themselves.

An Approach to Classification Change

2010-12-17T13:12:00.000-05:00

Ease Change to Assure Adoption
Approach using the new classification on the homepage of the portal or ECM system an option rather than a mandate, that is, provide the old homepage in parallel with the new look and let the Users decide which is better. Users will either adopt it or not. Either way you’ve reduce the risks of a wholesale change. Chances are good that the new classification will be faster and more amenable to the way User’s think about the company’s information.

Align Classification Schemes and Labels
Research all of the classification schemes in your company and attempt to conform them to a matrix where comparisons can be made. Look at common values. Look at classification labels which repeat themselves. Weigh the priority of following the lead of initiatives that have momentum, for example, if SharePoint is being adopted as a platform, look at how the tabs are labeled and try to conform to those. The goal is to work together for a common label structure for many reasons beyond the portal or ECM system. The goal is to create a reference for integration of search, records management, security, etc. This is one way to slowly achieve continuity of classification.

Mapped Metadata to Metadata Repository
In large organizations there are many reports, spreadsheets, databases, websites, etc., which proliferate different ways of describing content and information. These cause confusion and turf wars among groups responsible for applications. The integration groups are caught in-between trying to map one value to another in attempts to patch and process content flows. Mapping metadata is a stop gap approach, but does not deal with the larger issues of working toward a central metadata repository.

Taking the Long-Term Approach

Design and develop an enterprise metadata and classification model based on industry standards and integration requirements.
Pick the project that has the most traction and momentum and make sure they are classifying and describing their data according to the enterprise model.
Work in the adoption and changes to other applications as the latest "killer app" is maturing.
Integrate all applications at the metadata and classification levels, assuring bi-directional change interfaces.

Records Management comparison of Sharepoint vs. Opentext vs. Documentum

2010-12-16T09:21:00.003-05:00

Here's a rough records management functional and design comparison matrix based on Sharepoint 2010's offering out-of-the-box:

Function	SharePoint	Open Text Content Server	Documentum 6.x
Document IDs	New in 2010	DTree Dataid Docid in dversdata	r_object_id, i_chronicle_id
Document Sets	Tab in UI, custom page, properties of set and content list	Folders, Projects, Collections, Virtual Folders, Reports, Web Reports, Custom View	Folders, Change Sets, and WDK customizations Virtuals Docs Relationships
Auto Tag hierarchy	Library > Content Types	Folder > RM Classifications Categories	Folders > Object types
Taxonomy	Configurable, multi lingual, terms easily changed	Classifications and categories/metadata	Categories and translations
Content Organizer	A “transparent process” which routes content based on attribute values Folder Provisioning	RM Classifications	Customization or DCM or Taskspace or xCP solution Smart Folders File Plan in RM
Tag doc properties during creation	Transparent while writing in MSWord, term suggestions	Picks up basic MS Word properties	Picks up basic MS Word properties
In place Records Management	Declare from list	Declare from properties on standard UI	Declare from attributes or dropdown on standard UI
File Plan	Site level, content number, policy, content types	RM Classification on folders, categories, and content	Configuration in RPS or RM
Multi Stage Policy	Review Cycles Apply retention to folders (libraries?) or content types	Record Series Identifiers are configured to handle actions by certain criteria in doc properties	Lifecycle actions, folder inheritance, content type, TBO/SBO, webservices
Applying Holds	Search and apply In place or move	Part of RM module	RPS
Failover	Clustered, sticky session	Session is lost when a node goes down	Clustered, sticky session

Groupon and How We Work

2010-12-01T08:23:00.001-05:00

I’m going to take Daniel Lyons’s Newsweek article, “Click and Save” which is an introduction to Groupon to the next logical level. Groupon has subscribers who sign up for a service or merchandise and save by the scale of how many subscribers are interested and the deal that the merchandiser makes. This type of give and take can be applied as new way to doing work within IT.

So let’s apply this type of methodology to our “agile” or “waterfall” projects in a corporation. The project would start out as an idea or pain point which would be described by the person who has a budget and needs work done. Its details would feed into a sourcing cycle which matches the pain point with subscribers who know how to build/deliver the solution. There would have to be a deal making component which could resolve the processes around the sponsor deciding on who should do the work, the scope, for how much money and time, and for the subscriber/vendor to answer the requests and seal the deal.

So you ask, where’s the manager in this process? There would be minimal need. The project sourcing app is directly focused on the needs of the business and the solutions of the vendors or in-house developers. There’s a certain amount of critical mass that needs to be involved with the whole application for it to work, but this is the portfolio distribution process directly connecting the dots -- no middle person influencing decisions, playing politics, back stabbing, protecting their jobs, etc. Man, I’m not jaded am I?

So Gartner where does this fit into your ECM quadrants? Transactional content management? Could be. There’s a real need within companies to streamline the whole process of sorting out content creation and building applications that resource and process them. Content Infrastructure? Maybe. Portfolio management built into workflows that push knowledge from the creative brains to the consumers. We’ve heard all of this before, but what is new is that a small company like Groupon can blend a need for efficiency with the collective motivation of the crowd cutting away the profits of the middlemen. If this could be applied to the many issues of why IT projects fail wouldn’t that be a step forward?

The IT Cycle of Optimism and Hubris

2010-11-19T16:00:00.000-05:00

Once you have reach the twenty year mark in your IT career, look back, and chances are likely that you’ve experienced a few cycles of optimism from your leadership regarding using the latest software tools to become more competitive.

However, there’s a big difference when the optimism comes from the CEO vs. CIO. If it’s coming from the top it’s in the form of high level goals which are translated “logically” down to the tools to implement them. If the optimism comes from the CIO, the chances of it working are much lower and that’s because the CIO is throwing tools at a systemic problem that will eased a bit but not solved by them.

We’re at least at the second IT cycle of hype with “Knowledge Management” and Enterprise 2.0. As these cycles of optimism hit wide-eyed leaders who were not present during the last cycle enthusiasm breaks out on the presentation circuit with promises of curing the culture gap of knowledge sharing within companies.

Will Sharepoint solve the communication issues between different groups during the lifecycle of ideas, memes, and products? Did eRoom solve the issues in the late 90’s? Did ECM suites try for the past 10 years to make it easy for groups to share information? Wasn’t email or isn’t twitter going to help us? The point is that we need to go through these cycles to fail and get better at automating certain menial aspects of processing the information in our heads. The governance of engagement will prove to be a challenge with any attempts to fully electronically explore sharing of ideas within a company.

Dr. SIDDHARTHA MUKHERJEE talking about optimistic cancer “cures” that prove to a momentarily hubris in a long battle to finding cures:
“They said well, how can we possibly run a trial on something that we know has got to work? This story carries the memory of the kind of optimism that very quickly tips into hubris, which is so much part of the story of cancer.”

Tom Davenport, blogging a few years ago on Ent 2.0:
”I admit to a mild hostility to the hype around Enterprise 2.0 in the past. I have reacted in a curmudgeonly fashion to what smelled like old wine in new bottles. But I realized after hearing Andy talk that he was an ally, not a competitor. If E2.0 can give KM a mid-life kicker, so much the better. If a new set of technologies can bring about a knowledge-sharing culture, more power to them. Knowledge management was getting a little tired anyway.”

David Weinberger on Davenport’s 2007 blog above:
“But it's reasonable to think that the technology, when taken up and used, will affect enterprises directly and indirectly…”

Flip this ECM Stack

2010-11-05T13:52:00.002-04:00

Mike Alsup’s SharePoint and Records Management presentation got me thinking why not take the traditional ECM application stack and flip it upside down. I don't mean to be flip, but it's time to morph ECM into another set of solutions altogether. You’d have storage, records management, and archiving control through rules on top. Now think of these applications as goals, requirements and metadata which make a blueprint for the enterprise. The principle of this flip would be to drive consistency and efficiencies down through the enterprise of applications based on rules.

There’s no such thing as an enterprise application, an enterprise of apps is governed by new mechanisms like Boiko’s entities which have metadata repositories, identity, and apps which run models of rules, all of which are mutually exclusive, yet related and agile. These islands of entities are related to each other and connected by services at varying levels of complexity. CMIS is focused on inoperability, but is an add-on standard to the existing stacks of ECM. It’s time to unwind the stacks even more and create flexible models for governance, retention schedules, rules like 21 CFR part 11, etc.

ARMA and Enterprise 2.0

2010-11-02T09:56:00.000-04:00

Ron Miller is right on when he’s reporting on the ARMA and Enterprise 2.0 shows and how their relationship to each other is getting closer. I agree with most of his analysis, but as far as the responsibility and control of this relationship happening at the content management application level I disagree. ECM vendors have been trying to reign in this relationship for a number of years now and do not seem to the have the agility or will to invest in the fast moving Ent. 2.0 software realm.

Take, for example, Documentum’s records management offering. Their federated records management (FRM) solution is interesting and heading in the right direction, but falls short of commitment from EMC. The technology behind the solution was created by a third party. Tell me any storage management company that owns an ECM software suite has a strong records management solution which is backed by top quality professional services? It is in the company’s interest to provide half hearted attempted at solving one of the biggest issues that legal and information management teams have: how to find content and get rid of it in a rules-base, systematic way. Do oil companies really want to provide alternative energy solutions?

ECM: The Perfect Storm for Business and IT

2010-10-21T14:11:00.003-04:00

The blizzard of ’78 conjures feelings of helplessness and anticipation of the thaw. Everyone knew it was coming and yet they went to work anyways which meant lots of folks were caught on highways and airports. The lack of communication as to the severity of the storm was obvious in hindsight. Likewise, with ECM, the lack of governance and communication usually culminates into a perfect storm of control and blame between the business and IT.

With so much marketing information on “best practice” solutions to legal holds, records management, learning management, business process management, etc., it’s easy for individual business units to fall in love with a solution before IT even knows about it. Good governance would never allow this wild west atmosphere, but growing companies are usually governance challenged. A chronic lack of IT resources over time creates easy ways for business units to push their own tools to innovate in areas the IT does not “understand”.

All this back and forth with whining on both the business and IT sides leaves an easy scapegoat: the ECM solution. It becomes easy for the business to exclaim, “The UI is so 1990’s”, or “I can’t find anything”, or “We don’t know what to delete so we keep everything”. Then IT chimes in, “It will take us over a year to upgrade!”, and “Why can’t we just use SharePoint, it’s free!”, or “We don’t have any functional specs, so what does the business expect?”.

The middle ground where the requirements meet software functionality and architecture is where the weak link sometimes is. This link is where the projects are vetted, resourced, and funded. The problem is that this middle ground is stuck between being not too business experienced and not too technical. The middle ground is constantly changing, but can be called portfolio management or business relations. If governance is not at the enterprise level for content management, then project priorities and resources are constantly stressed and projects fail.

Storm Formation

Sales professionals with hidden agendas sell the business on slick demos and solution X.0 capabilities. The business goes to conferences and comes back psyched to use the latest tools, only to find the dull, boring ECM solution in place. The business writes up requirements and gives them to their portfolio manager. The project doesn’t get approved. A year later the business pleads with their boss’s boss. The boss’s boss talks to upper management in IT and a pilot project is born.

History vs. Change
ECM Directors have with hidden agendas too. They have relationships with vendors, they know the software functionality. They are comfortable being experts with the system. They want to ride it out as long as possible. Change is too risky and fraught with functional gaps which the business won’t like, but meanwhile the business is secretly hoping for a new system. The license agreement has not expired and the cost to migrate is huge. There are always many reasons not to change if your vision is one to two years out.

Storm Aftermath
The period after the big storm affords some down time; hindsight to revisit any failures or damages sustained. Standards and rules have a chance to be implemented because there should be opportunists who understand architecture and governance who can exclaim, “I have a way out of this mess”. A new CIO is usually hired and new models are sold to IT. The new models promote ties with requirements and functions, budgets and accomplishments, and business to IT communication. The ECM system will most likely morph into something that is more suitable to the requirements, maybe a new application server (Sharepoint) or focusing the “E” of ECM to a system which archives and manages retention.

Open Text 2010 Sets Up Defensive Services

2010-10-12T14:54:00.003-04:00

Open Text’s marketing of their new 2010 ECM suite and share services points to four distinct areas of focus: Lifecycle, Process/Transaction, Engagement, and Shared Services. Looking at these stacks, it is clear that OT’s has set up them up to deal SharePoint’s strengths and weaknesses. The Library services layer would have been revolutionary 10 years ago, now it’s just another ECM suite trying to be the archive and retention layer in an enterprise.

Also the initial release of ECM 2010 is for Windows 2008 R2 64bit until the 32bit is available. If this isn’t a concession to Microsoft, I’m not sure what is. Oracle gets a boost too with the 11g requirement.

“Management content and metadata in a consistent manner” is such a cliche at this point.

Lifecycle (“strong” hold): these are all set to be broken out as services.

Document Management – lots of potential here, but each of these need to be services…
Records Management – classification layer on top of document classes. One of the biggest potential services that could be developed, but currently is very embedded in the core CS product.
Rights Management – a service which is used to encrypt/decrypt content.
Digital Asset Management – “I want you tube”
Archiving – this needed to be highlighted a long time ago. Flexible storage allocation was huge before EMC bought Documentum. Now everyone is following their lead.

Process/Transaction (decoupled service from content server for enterprise implementation)

Business Process Management – separated from the core content server, this service is offered as a single instance which will integrate with many other repository services, including the content server.
Capture and Imaging – This is separated out as well with connectors to repositories.

Engagement (up for grabs, here you go SharePoint, please don’t take anything more…)

Collaboration: is this a joke?
Social Media: right?
Web Content Management: hmm, I don’t think so.
Rich Media Management: If I have too.
Mobility: get in line.

Shared Services

Library Services (too little, too late)
ECM suite, SAP, SharePoint, File system, Email system
Enterprise Process Services – BPM above

So what would be a better alignment of services to appeal to experienced and frustrated folks? How about tossing out the authoring and collaborating side until Microsoft succumbs to Open Source formats and focusing on tagging, classifying, and storing information. In other words, focus on where the current pain points are which are not at the document level anymore, they are at the information architecture level and SOA projects are all custom after the designing part is done. Hmm, maybe Open Text can open itself up to services that really help information get organized instead of trying to compete with Microsoft in collaboration or publishing to website…

What happens when ECM is renamed to DMS?

2010-10-05T13:50:00.000-04:00

When IT governance, steering committee, or strategy team renames a whole IT group from ECM to DMS (document management system) it is no small change. This change has many ramifications: changes to application software, data architecture, integrations, requirements, functionality, etc. A divide and conquer rationalization is presented. A “let’s look at all requirements from the stand point of: do we really need this?” Ok, but it’s not that easy.

What is lost?
Talent: it takes time to dismantle an ECM system into to smaller, more focused parts and personnel with attention deficit disorder will get antsy and want to jump ship.
Sense of cohesiveness in the lifecycle of content creation through disposition: application silos will start to appear.
Unlimited playground for the business to experiment in: the business will have more work to do to figure out what they really want to do and get measured by it.
Central administration of users and groups, access control, attributes, taxonomy, retention schedules, search, backup.
Source of record: How do you keep track of and archive content that need is mission critical and could be audited many years hence?

What is gained?
Broader understanding of the content lifecycle: because of more integration requirements, the business will have to know exactly where their content lives and is dispose of.
Divide and conquer mentality: separate groups tackling the tough business communication issues.
Better governance: now the strategy of cohesiveness of all the desperate applications will
Better data architecture
Better standards definitions and adherence: assuming there are strong governance and data architecture, standards should thrive and push integrations to new heights.
A distributed system which reduces the “too big to fail” issues of a centralized repository system.

Recycling the Same Issues
To what extent are we recycling the same content architecture, management, and publishing issues? Collaboration issues will be deferred to SharePoint, workflow will be spread among all of the applications, archiving will further muddled, group access control will be splintered further, regulation paper work will increase by the new amount of applications.

Content and Information Architecture Design
I have hope that this will be the golden age of the content architecture frameworks, that we will witness true policies which are governed by blueprints which can withstand the issues of silos of information, de-normalized data, historical metadata and the like. The framework would have to be malleable, yet strong. It would have to be clear and forward thinking to lay the ground work for all content and information evolution to come. Ok, I can dream can’t I?

Embedded (shadow) IT: Where are you?

2010-09-17T17:30:00.000-04:00

If you have ever had a position in IT where there was a “dotted” line in your responsibility to another group, you’ve worked for embedded IT. If you have developed an application on your desktop to support your team, you have been embedded IT.

CIO’s always want to centralize most of embedded IT to achieve the cost savings of common services, infrastructure, and architecture. I’ve been embedded and centralized, both of which have their faults.

Here are some aspects of embedded IT that could be potentially missing from a centralized model:

Well understood business requirements: ability to read between the lines
Trust in your teammate’s ability to deliver
Efficiency of work and play is a given
Hard work is observed and appreciated by the business
Governance is organized and strong: no dotted lines needed
Content is fully described to suit the project at hand
Sneaker net workflow works well

Here are some aspects of centralized IT that could be potentially missing from an embedded model:

Automated workflow
Consistent metadata and values which describe content
Governance which is tuned into the goals of IT as a whole
Consolidation and sharing of duplicated services
Lack of trust in your teammate’s ability to deliver on a project due to being spread thin
Business requirements that are read literally

Creative Destruction of SharePoint

2010-09-13T14:19:00.002-04:00

SharePoint is trying to transform, or split up ECM into pieces, similar to what Schumpeter referred to as “creative destruction”. Schumpeter’s use of this term implies that in order to have innovation the current paradigm has to be superseded.

What aspects of ECM will be crushed in order for SharePoint to innovate? Will ECM actually be destroyed? I have doubts. The main “innovations” of SharePoint over ECM software are based on shortcomings of the software suites, not the solutions or platforms. SharePoint is not as much innovation as it is a fix to chronic ECM problems:

A more unified architecture, as opposed to the patch work of typical ECM suites
Easier to configure and build sites from templates (unless you have to customize, which is most of the time)
Ease of Use: by definition SharePoint is easier to learn and use, unless your thousands of users already use another software suite.
Already in the house as an OS and email system, thus foundationally integrated, which is the key business driver for Microsoft-based systems.
Less expensive, in the short run, but the long run expenses will depend upon reducing the need for customizations and traditional library management services.

ECM suites will have to develop and innovate on their strengths, which in some cases are slowness to change with the times. Most small mistakes made in implementing content management systems were made on a continuing basis for cost and lack of direction reasons, not because of the obvious flaws in the software. The market of ECM has created the dinosaurs of ECM suites, not the other way around. All large enterprises, which own the largest amount of licenses, can not and will not move at social media speeds.

Social media has many more iterations to go through in order to develop the depth of ECM requirements. SharePoint will run in parallel of the ECM suites for some time to come, maybe nibbling off bits of functionality and previous integrations. In the mean time, ECM will hopefully rewrite their core software to be more open source and model their applications and added value on innovations rather than bandaids.

Litigation Response as Wedge in IT Issues

2010-09-13T11:18:00.000-04:00

Let’s say you have litigation response that the legal department has mandated. It becomes clear that the overall cost to outsource the discovery of self-organized content repositories with limited metadata and historical changes in metadata values, let alone file shares and email, is staggering.

The Legal Hold project revealed issues in many other areas of the company, namely that there was no central data management architecture, records management was not automated enough, and central governance of the embedded IT groups was ineffectual.

In this case, the Early Case Assessment was controlled by outsourced personnel instead of an automated tool. The mandate acted as a wedge that opened up the heart of the issues that almost every enterprise has to some degree: namely the lack of coherent standards which span all applications and are well understood and governed.

Floodlight vs. Spotlight in eDiscovery Solutions

2010-08-31T11:21:00.002-04:00

Floodlight
I had the privilege to be entertained and enlightened by a few recent demonstrations by leading eDiscovery/Legal Hold software vendors. After listening and looking at smoke and mirrors, each vendor revealed that their core competency still remains their strength and eDiscovery is an extension or add-on rather than a full fledged solution. The “floodlight” approach of these vendors to eDiscovery was not achievable given the current state of the offerings.

A true eDiscovery/Legal Hold vendor has yet to emerge. The add-on vendors flooded the market by buying the pure plays, but couldn’t have imagined the difficulties of integration that these add-ons have created. The issues behind efficient eDiscovery are squarely in the business’s camp and not in IT. Not one of the vendors mentioned the integral importance of common and historical metadata management. If the doc is not described well enough, good luck discovering it. If Record Management is not fully functionally incorporated into the business daily routine, docs will not be disposed of according to compliance mandates.

Search Vendor
Core: Index and Search Results
Platform: Most
Hold: Export out of repository to file system
Vision: Nice long-term story if affordable
Agent: stealth/regular desktop agents
Connectors: Most applications with varying levels of customization needed
Index: Best in class, but dependent on the source content's metadata accuracy
Collection: Kind of fuzzy given that the exports from repository are done without built in abilities to validate source content accuracy, ie. ways to test discrepancies.

Forensic Vendor
Core: chain of custody of collection
Platform: Windows
Hold: Office docs on file system by using extra control files for tracking
Vision: Deal with Windows file system content
Agent: no client agent
Connectors: Windows and SQL
Index: Dependent on other’s indexing
Collection: Searching and collecting are secondary to browsing and selection.

ECM Vendor
Core: Workflow, doc lifecycle, storage management
Platform: Most
Hold: Export out of repository to file system with xml metadata file. This potential removes the source context, auditing, and access control information from exported content. What about a hold inside the repository?
Vision: Partner with eDiscovery solutions and let them deal with the issues.
Agent: stealth/regular desktop agents
Connectors: Many, but disappointing connector to own system
Index: Does not compare to robust search engine eDiscovery product
Collection: Built-in export is marginal out-of-the-box and third party tools need customization for legal department export and metadata requirements.

Spotlight
When will eDiscovery solution providers design the foundation of their offerings around metadata/records management prerequisites and controlling the chain of custody? Of course this is not easy for any enterprise to achieve, but focusing a “spotlight” on specific areas and building the solution each area at a time will get this done. This “spotlight” approach will force the business unit that is most motivated to solve their organizational issues around RM and metadata. The policies and procedures can be created after the first spotlight is finished. Subsequent spotlights will be dependent on the quality of governance and policies in place.

The spotlight approach would show the strength of pure play solutions over floodlight enterprise solutions. The forensic vendor would shine when testing Windows file system content only which would be a bad start. The holy grail of all of these solutions is the ability to introduce a layer (like RM solutions) where the intelligence gained by indexing, searching, and collecting could be written back on the content for better future classifications and entrance to the semantic qualities of future systems.

Single Point of Failure

2010-07-10T14:48:00.001-04:00

I’m going philosophical for a moment here. Why do we have a technical systems oriented concept of “single point of failure”? Aren’t we all “single points of failure” and the recovery from our absence is proportional to the much we shared of our experiences? The risk of losing a connection, process, services, invocation, etc is proportional to how much we know about system’s interactions and what their limits are.

Redundancy is nothing but an attempt to backfill a single point of failure. Redundancy is really just a double point of failure if you think about it. How many points of redundancy is really risk free? If you have one hot backup system to failover to, then what do you have if that fails? It’s better than nothing, but not fail safe. Enough of this sophomoric banter, let’s try to figure out a list of single points to check and test.

Any call for failover and redundancy is mandated by a Service Level Agreement between the business and IT. This agreement has flexibility factors built into it and is most likely an extension of a more comprehensive regulatory or industry standard. It usually happens when an old policy didn’t work according to plan and the C’s freaked out.

Can you perform an in-place upgrade a system without downtime? If not, what services in the application stack are not redundant enough? Here’s a common list of systems and issues to consider:

Application Servers
Should be clustered for failover, however do sessions really failover gracefully or do Users get errors during the failure?

Database Servers
Should be setup with a hot failover like an Oracle RAC system. This would allow for backup recovery. Also, as storage expensive as transaction logs are some cases these could be a life saver.

Service accounts: Local vs. Domain
I understand that applications need to be secured between services and file systems, but to only allow domain accounts for an application sets up a potential for one account to run services that could all fail if there is an account lockout of that one domain account, or Active Directory. Local admin accounts serve the same purpose and in most cases can be substituted for the domain account.

Storage Backup/Restore
Storage itself is relatively straight forward; backing it up and restoring it is a whole different story. Backups are usually done on a rolling basis: full on the weekend, incremental on the weekends. But, what happens when a restore from tape is needed? The normal backups get delayed because the same system is used for both. Do you know how long it takes to restore from the backup? Are the backups redundant? How close is the backup machine to the application? Is there a whole DR off sight solution?

Free Export/Import Tool Registered tables, groups, ACLs, and Users (for DCTM)

2010-07-07T07:58:00.001-04:00

When it comes time to designing the community, information, and structures of the repository, these Java tools will come in handy. They facilitate deployment between environments. They also allow for editing in Excel and porting back into the repository.

Forcing Structure by Leveraging a Crisis

2010-06-23T16:17:00.002-04:00

Leveraging a crisis happens all the time in politics and the financial industry. Why not leverage a crisis for the sake of organizing content? Let's face it unstructured content will remain as such unless a fire is lit under the businesses responsible for making the mess in the first place. Here are five possible avenues to pursue if the opportunity arises. The idea here is to build a foundation of detail and context, the "who, what, and where" of Zachman's framework. This will provide a foundation to then apply certain tools to go to the next levels of acting on the content in ways that structure, permission, and dispose.

Legal
Let's say there's a lawsuit against your company and the legal department is breathing down IT's neck to find, collect, and produce information and content pertinent to the case. You're not too happy with the 3rd party vendor who is helping collect the information. The business is somewhat removed from any searching activities. Most of the burden is on IT. But why? Who created this chaos call "content management"? What you could do is use the litigation request as a method to push back detailing content and context to the business under the guise of needing this information for legal counsel.

Cost
When cost cutting is running it's course through your company, use this to apply a cost charge structure where the business units bear the burden of paying for all aspects of IT. For example, if you inventory your ECM user groups, I bet you'll find a few groups which use larger percentages of the system in terms of licenses and storage capacity. When these groups feel the pain of actually paying out of pocket for IT's services and storage, they will quickly want to figure out what the records management policies are and dispose of content. They will also want to structure their content to better understand what they have and who owns it.

Security
If there is a breach in the ECM system, unauthorized access to sensitive information, the business will want to clamp down on access control. This will prompt the business units to figure out what they have and who can see it. It would be a good time to apply metadata and better folder structure to the content as they are applying ACLs.

IT Capabilities: Limits of DR and SLA
The next time a system goes down because of backup time during an upgrade, or recovering from a flooded data center, the business will be tick off, but receptive. This could be an opportunity to mandate standards of ECM organization and use.

Resources
If the work requests of the ECM system are back logged and the business is complaining, try pushing back an agenda of redesign and restructuring the content and information architecture. Too many customizations to a system could mean it's time to upgrade and apply standards that help integrations and interoperability.

Unbundling the silos within ECM

2010-05-28T08:51:00.002-04:00

About 10 years ago, all you heard about in IT was consolidate, consolidate, consolidate. Back then consolidate meant economies of scale, bring all website together and save money by utilizing less resources and infrastructure. This really worked in the short run, but news flash, the silos are back: they never went anywhere. The silos I’m talking about are brain trusts of business units that work as a team but don’t really share. The walls of the silos are still there; they are just within the ECM now. ECM itself has built its own silos: silos of services.

So I was reading an article entitled, “How to Save the News” in the Atlantic. It describes the downfall of the traditional newspaper revenue model with the help of the Internet and Google. But Google says it’s on the newspaper’s side. Google’s assessment of the journalism is that the “Bundling was the idea that all parts of the paper came literally in one wrapper—news, sports, comics, grocery-store coupons—and that people who bought the paper for one part implicitly subsidized all the rest…” The internet is forcing newspapers to unbundle their sections, thus their revenue cows.

The idea of unbundling got me thinking about a platform in the enterprise that is allowing business units to manage their content in “silos” without worrying about the cost anymore. The platform is doing to ECM what Google and others are doing to the newspaper industry. The platform integrates well with the OS, productivity software, and the email system. The platform offers most of what ECM offers. The platform licensing model makes sense. The platform is SharePoint.

OK. So how do ECM solution providers unbundled the stack and content that has taken years to design, develop, and deploy? Answer: One business unit at a time. Webservices and CMIS will allow for the slow migration away from the great “consolidation” ideas of the early 2000s. As migration happens, some of the standards and best practices that were too expensive to implement in the past will be implemented. Business units will be able to do what they want in their information “silo”, but will be using standards of metadata, taxonomy, security, business process, and records management. Farming for knowledge will be possible by virtue of these standards. The days of the ubiquitous file share will come to a close, the new platform that will be taken for granted is SharePoint.

ECM vendors will split up their services and sell them separately. For example, workflow services are in desperate need of enterprise integration. ECM software can and should pursue this. Also, records management should be an integrated service with SharePoint. Not the underlying repository. Unbundling and expanding ECM services will be key to ECM’s software strategy.

Decoupling ECM

2010-05-27T15:48:00.001-04:00

As your enterprise content migrates from one repository vendor to another, you will feel the pain of not following decoupling standards that I’m sure were implemented but not fully. By “not fully” I mean in 2001 you tried an interoperable integration only to cancel it because the APIs were limited and the performance was slow. To add fuel to the fire, CMIS is being pushed on us and that means if you’ve been following standards of information architecture with metadata, linking from external sources, training application integration, workflow, taxonomy, and databases, you’ll be fine. However, who has been able and/or could afford to follow all of the best practices? Anyone?

Below is a list that starts to detail what needs to be looked at when thinking about decoupling and preparing for CMIS or a migration.

Metadata
Is the object model and attributes designed well? Are there issues with attribute that have one name, but are used for other functionality? Will system date attributes like creation date and modified date trigger any unexpected actions? For example, if you have a retention schedule based on creation date and you migrate your content (creating a new creation date), what’s the real creation date?

Intranet/Portal
Links from a website via a URL that points to content in the repository will most likely be broken during a migration. Portlet Integrations that use APIs to query content in the repository will have to be checked. Content that is published to a separate website will have couplings with attributes from the repository which will break.

Training
Links to content (SOPs) as training material that is triggered by changes in modification dates will have to be redesigned.

Taxonomy
Relationships of folder structures with attributes will have to be reviewed.

Web services
Any dependencies with attributes and content ids will be broken.

Database Integration
Redesign any dependence the integration with repository database has on system generated dates.

Free DCTM Object Model Tool

2010-05-14T18:59:00.001-04:00

As a developer and solution architect you know it's a pain to document the object model and attributes deploy it and then document it again after minor changes -- and there will be changes. This tool allows you to select any dm_sysyobject child type and show its attributes and up to four levels of that object's child objects. Comes in handy to visualize a repository's object and metadata structure.

http://webpages.charter.net/webberj/products.htm

Legal Discovery Drill

2010-05-12T06:40:00.000-04:00

If you ever want to really drive home the need for records management and info architecture in your enterprise, run a discovery drill on the production systems. Drive it from Legal and send in “auditors” for outside counsel and work through the IT processes, testing, term searches, extracts and exports, org charts, the works.

In the least it would tick off the admins and DBA’s who have to do the grunt work. But, it could open up all sorts of discussions around the right ways to structure metadata and content such that the discovery process is more straightforward and less risky. Preserving content is half the battle; the other half is finding it.

There will always be a divide between managers who want to identify and separate content as “records” and those who recognize that the process of record identification is too much work and that all content should be considered part of the record albeit in a “big bucket” sense. For example, email that is not considered a record by a small group of records managers and is deleted could be later deemed extremely relevant to an investigation and a judge could decide to slap an adverse judgment on the evidence meaning that the jury should assume the worse possible conclusion about the email’s contents. The risk of this is weighed by counsel…

Mapping your Records Retention Schedule

2010-05-07T11:54:00.000-04:00

OK, so you had a consultant meticulously categorize and detail every type of record your company has and how long to keep them. The consultant looked at all of the paper documents for the passed 20 years which have been stored at Iron Mountain. This you thought is the source of all records, but did the consultant cross reference these “best practice” categories/file plan with the actual ECM repository(s) in use today? Did the RRS get vetted by a representative sample of business groups from the company? Is the RRS in a form that can be easily modified during the years it might take to implement it and the constant tweaks made to it on an ongoing basis?

Let’s assume that some of these questions were considered and that you have a reasonably well put together RRS. Now let’s look at the RRS to figure out how to apply it to an ECM application. Mapping the RRS columns of information may not be that obvious at first glance. The column headers will be generic like “record class name” and “customer record class code”. Which one should be what the Records Officer sees and what should the Users of the ECM see? The “code” is usually distinct thus it should be used as the File Plan number and Record Series Indicator. The File Plan corresponds to the folder or classification and the Record Series Indicator corresponds to the retention schedule object.

When you see the columns “Event” and “Period” these correspond to the status that triggers the status date and how long to keep the content in that state. For example, if the period states ACT+INA 6, this means keep the content if it’s active and dispose of it when the content is inactive for 6 years. The event could be when the content was created, or if the content is superceded. The schedule is attached to the Record Series Indicator and might have multiple schedules per series for ACT, INA, ARC, CLO, etc.

The “description” section might have extra information which could be useful for classification terms or record series types. It should also be added to the configuration to help describe both the File Plan and the Record Series Indicator.

Do you have content types and lifecycle states?
Doc type, status, and status date are essential to configuring your RRS. These may be obvious based on how the ECM object model was designed or how workflows push content through their states, however it assumes that all your content is processed this way and most important that it is consistently processed this way. A tall order for even the best designed repositories.

User Adoption
As you map and configure your RRS keep User adoption in the back of your mind. Think about seamless use for tagging records. Should all content be a record or just some designated by the Records Officer? My opinion is that all content needs to be disposed of eventually so all content is a record. There are different levels of records importance to the company and these will be retained, monitored, and secured accordingly.

ECM Acquisitions: the Paradigm is Shifting

2010-04-28T12:13:00.000-04:00

To hear that Open Text bought Burntsand, a professional services company with expertise in MS SharePoint (MS Gold Certified) and EMC Documentum (Signature Partner), the paradigm is shifting... What is the strategy behind this type of acquisition? Take out the legs of SharePoint proliferation? Or, partner with the competition now to ease the eventual shift to lighter weight, federated repositories with agile, modular services of functionality of the future? Or, is this a Canadian acquisition among friends? It’s just strange at first glance.

As the Facebook-like mashup wars continue, this may be one of many interesting acquisitions. How can a traditional ECM repository layer be repurposed into a service layer that underlies SharePoint as a frontend? A fully integrated (not EMC’s SharePoint web parts, or OT’s CLM or Wingspan’s Docway) backend, maybe a “faster” performing CMIS? Hey all the pundits said 2010’s the year of CMIS…

One obvious synergy is the need for OT to integrate its whole suite of applications and solutions with SharePoint or get fully cut out of the picture in the next five years. EMC does this sound familiar? Did you hear the shock and awe salvo being launched when SP 2007 was released? Can you see the rockets heading toward Documentum as SP 2010 is getting hyped?

As these huge suites of software get split and repurposed, ECM as we know it will evolve into Enterprise Content Services, ECS. Content will remain as king, it’s the “Management” that will be replaced by agile services which when mashed up will be greater than the sum of its individual purposes. Plug and play like iPhone apps. What will be the platform for this: SharePoint.

Some Tensions of ECM

2010-04-26T14:54:00.000-04:00

Navigation vs. Search

Gerry McGovern says it well when he says that navigation is just as important as search: navigate first to a sub division of content, then search the index of that section. Some repositories have no rhyme or reason to their folder structures: governance and standards were nonexistent during the proliferation of folders and content. This happens when the software product was oversold and the budget for rollout planning and information architecture was pilfered for the sake of getting the ECM solution deployed and stable. The same groups who were negligent are now forming strategy groups to figure out how to reorganize the navigation and content, as well as fine tune search results. Sound familiar? The people responsible for the mess are now tasked with cleaning it up. Do they understand metadata standards? I hope so. Will they decide to reclassify existing content, migrate and tag content to a new repository, or design virtual folders? They’ll have to start somewhere.

Legal Preservation vs. Information Architecture

Understandably, the Legal department wants to limit the amount of metadata to as little as possible to reduce exposure to liability. The Records Manager wants to tag content according to preservation rules and schedules. The Users want to be able to find their content by typing in metadata values which they are accustomed using to describe their content. Doc Type, Area, Owned By, Modified Date, Reviewed By, etc. These are fundamental to all repositories, but Legal would prefer to cut them to the point of obscurity; least amount of description equals the least amount of risk.

Agile Business Process vs. IT Stability

Users, trying to keep current with their changing business processes, are constantly pressured to modify their workflows. Meanwhile, the IT department cannot keep up with all of the change requests in a timely fashion. This creates a constant battle of change management between the users and the supporters of the system. Support is typically under staffed to fully handle all of the necessary changes in an agile and stable way. Also, architects usually consider change management and support an after thought to the main design of the repository and integrations.

“Too Big to Fail” Resonates with ECM

2010-04-16T13:48:00.000-04:00

The “Enterprise” part of ECM conjures up visions of comprehensive coverage of content, metadata, and economies of scale. Like our “too big to fail” financial institutions, some ECM systems are too large and cumbersome to deal with lack of metadata standards and content hoarding that exists in many large corporations. Can an enterprise afford to bail out a monolithic ECM system when it eventually fails? Sooner or later the system will need to go through a major upgrade or a migration to another vendor. Because the system is centralized, it will cost a lot of money and resources to plan and implement this transition.

Federating ECM
Let’s make a distinction here between having one monolithic repository from implementing agile repositories which are small and more focused on “Silos” of business unit functionality and “case” type processing. “Silos” of information have long been wrongly accused by vendor marketing of being the issue with inconsistent business process and integration. The real issues are lack of governance and enterprise architecture reviews of applications. This is why Sharepoint is infiltrating: it fills the need for business units to create their small silos and integrate them with overall standards of the enterprise. Of course the lack of governance and architecture with Sharepoint deployments will result in the same issues of “Silos” with a capital “S” as vendors have been successfully selling on.

Change Management
If change management is not considered up front during the design and rollout of the ECM system, the emphasis on standards of metadata and integration will be pushed out to later phases. As the ECM system fills with process and content without policing the expansion of content/metadata and process, the cycle of complexity of these “internal” silos of information continues. This comes to a head when the system bogs down or the centralized resources can not keep up with the business’s demands for new functionality.

The Cost Effective Way
Typically, IT will focus on hardware and software upgrades to make ECM faster or give it more capacity: add more machines to scale it, the biz will be happy with the speed. Hire offshore resources to build workflows. IT has to control costs by centralizing content management; enterprise architecture traditionally puts ECM in a storage box; metadata is controlled by silos of business. This is part of the overall issue with ECM. There’s a gulf of misunderstanding between IT and the business. Save money by consolidating, then blow tons of money in four years trying to upgrade or migrate. Does this make sense? “The cost effective way” is a myth with a short-term vision that ECM IT managers tend to by into.

Derivatives
I think of UIs and add-ons as the equivalent of financial instruments like derivatives. When a vendor introduces a new look and feel and different functionality, this is an attempt to create an illusion of a new application. But, in reality, the underlying information is still the same. A new UI cannot mask inefficient governance and lack of regulations. For example, as DCTM’s webtop was waning in popularity, EMC decided to create Taskspace. This UI was promoted as an easier way for a business power user to configure UIs to suit their process needs. However, Taskspace relies on sound information architecture and group/permission structures. If these aren’t implemented, a vendor will promote creating a whole new info architecture next to the exisiting ECM one. So now we have two different, competing metadata structures. Professional Services will promote the new arch as being for case management—leave the old one for regular content. Can you see where this is leading?

Default Swaps
In ECM, these are like assurances that previous failed attempts at governance and info architecture will be covered. If one project fails, we’ll buy the latest technology to swap it out on. Enterprise 2.0 will solve the current issues regarding lack of business agility and limited IT resources. Don’t think so. Application Default Swaps are common and only push the issues into the next cycle of gadgets.

Credit Ratings Companies
Lastly, let’s look at Gartner as the research companies that put ECM apps in their quandrants. They slice and dice the application’s functionality, technology stack, scalability, etc., but this obfuscates the issues underneath the hood of a company’s ailments. What good is the best enterprise search application without a consistent metadata and access control standard in place enterprise wide? Is any part of an ECM application research company’s revenue derived from the vendors? Indirectly? Hmmm.

eDiscovery Solution Drill Bits are Dull

2010-04-05T11:31:00.002-04:00

After reading Reed Irvin’s 10 Steps to Litigation Readiness in the March/April Infonomics, I felt like I was glossed over with good intentions and a sense of sales urgency, but I was annoyed at the same time. When will Infonomics stop writing the steps and start dealing with real fundamental and complicated issues in enterprise content, process, and people? I know it’s easy to poke at articles like this which serve as an introduction litigation readiness and try to weave in experience as well. The issue I have is that it is written by a promoter of a certain product. The writing expands on the core functionality of the software and services which his company is trying to sell. It’s generic for the purposes of a “drill baby drill” mentality meant to sell services.

Here are my comments on the “ 10 Steps”:

First of all, where are some of the new prerequisites in this piece for effective enterprise litigation? Put an “effective” in front of all of these…

Metadata Repository
Enterprise Architecture
Enterprise Search/Index
Identity Management
Content Architecture Standards
Integration Services
Role based Retention Schedules
Enterprise Process Management

These requisites are essential to defending a lawsuit in court. Imagine investigating a crime, but not knowing how to drive or read a road map, you wouldn’t even know what type of crime occurred? Are the above not included because they haven’t been developed yet into the toolset? Or, because they are “too technical” for the readers? Or, not focused enough on litigation? I hope not. For example, eDiscovery products assume a great deal is working effectively with the above technology areas. Tell me one company that has IT running all of these initiative flawlessly…

Second, the big bucket, then smaller buckets approach is not addressed here because it negates a lot of the need to create an inventory, automate retentions and dispositions at a detailed level. If you don’t need to tag and control all of the insects in the jungle, then why would you need our tools? If you could start with a big bucket compliance policy and slowly identify and weed out the more sensitive content wouldn’t that make more sense? When you see an ant in your kitchen do you have to identify it to decide whether you are going to step on it?

Datamap: to make legal search easy, determine data source, figure out the who, what and why of content. This is all good, but needs a few more details of potential pitfalls to round it out. One huge issue is the legal department itself. Legal wants the map, but also has to make sure that vendor and the software being used is vetted out. This means that the process of getting approval to actually use the tools might take a lot longer than anticipated. Having an effective enterprise metadata repository, architecture and index/search in place would greatly benefit the datamap effort. And vice versa, the datamap would open the eyes of IT to the needs for enterprise level processes and frameworks.

In step 9, Reed mentions having a flexible process which is spot on, but there is no discussion of historical analysis of the data and process in order to develop a framework for change. It’s not enough to say there will be a periodic audit of the map and processes, you have to know what to expect based on the company’s cultural changes in the past, present, and future.

Step 2 is a good start but needs to be tested with scenarios. These scenarios are just as important as the policies. Using a Zachman framework may help with figuring out the details of the whens which are not covered here. In other words, what is the roadmap of this readiness? What are the different ways to roll this out? Also, access control policies and identity management are components of the policies which are usually left for the IT department to figure out, but should be detailed up front.

Getting buy in and executive support for Step 3 is right on. This is absolutely necessary and should be step one. However, the legal department as a sponsor may not have the clout needed to push all of the agenda through the organization, secondary buy in from governance groups would also be necessary. What would help tremendously is a previous litigation or a few drills that permeate through the whole company and highlight the need to get the house in order and more importantly save in huge costs associated with manual labor.

Step 4 and Step 7 are at odds with one another. On the one hand, know your users and on the other take them out of the process. Adding Step 6 (automating the process) here, what needs to happen is development of an automated process for preservation and disposition which is seamless to end users, but is very clearly understood and vetted by end users. One way of doing this is to create a pit stop mechanism which implements these policies at the integration level not the UI level. Also, if there’s an IT compliance department in your company then why not drop the process of applying retention there and add it to the other steps to adding storage to applications. A business partner within an enterprise has to justify adding storage and why not add it step to detail the retention schedule for the integration service?

Step 5, Build your team, should be a framework of roles not the actual team members which will form and disperse depending on the type of suit against the company. The in-house IT folks will need to shift priorities and delay other projects. This could create a wave of ramifications through the interrelated projects that are in progress.

The bottom line is that no one product suite or solution can possibly handle the myriad of issues involved with eDiscovery. The best case would be to develop scenario based approaches and frameworks with the knowledge that a hybrid solution will inevitably be developed. A comprehensive datamap could take years and is really an exercise in futility. The prerequisites listed above are necessary to get to the point where a map will make sense. The map will have to be a framework more than a blue print—the exact specifications will change based on the litigation at hand. Building integration services that regulate retention policies and control content processing will make dealing with the human intervention factor more straight forward and less of a burden on “repeatable” processes. Building and morphing a set of litigation scenarios all honing your processes to these will go a lot farther than buying one-size-fits-all software solutions. This may be where EMC’s Case Management and Microsoft’s Sharepoint projects collide into interesting hybrid (mashup) solutions.

Why Financial Services and ECM is a Challenge

2010-03-27T11:27:00.003-04:00

Most Financial Services companies large and small have really screwed up the ECM model. Years ago when they purchased Documentum, IT folks looked at the technology and installed the “docbase” server with HA and DR, and that was it. They are heavily into databases and the interrelationships of client IDs and Customer IDs, products and services, etc. IT looked around at the time and saw programmers, DBAs, and web tech guys and thought we’ll build our own frontends to Documentum, our permissions and rules for metadata are too complex, too fancy for the WDK. This was partly true as EMC Professional and Product Services never caught up to the specific requirements and functional needs of the finance industry as they did with life sciences.

Typical scenario at a Financial Company

The client applications director has been somewhat satisfied for the past four years with the way IT has maintained custom web applications integrating with Documentum on the back end. The director is getting more heat from above to look at cheaper, OOTB Documentum solutions. The IT department development (even outsourced to India) is getting too expensive. IT has been planning and executing an upgrade for the past year and a half. They are still at 5.3 (not supported) for the repository and are upgrading the website DFCs to 6.0. Why not 6.5? Because at the time of planning 6.0 was just coming out and the client development started. This is way too expensive and lethargic, but what are the alternatives?

BSAs with their head in the sand

The Business Systems Analyst Manager, with his MBA says we need to think about the big picture, but deal with the low hanging fruit now. In other words, there’s no bridge in communication between operations and IT. This is where a Deloitte is usually helpful in terms of getting the too sides to agree on long-term goals. I know consulting firms can be devastating, but in this case, all the MBAs need to get together and show off their power points. It’s not about just fixing one issue then the next. It’s about getting the CIO to wake up and figure out ECM should be handled like it’s ERP. There is usually a disconnect between the Client App Business Ops vision and the IT technical vision. This will not get fixed at the director level, as much as the C’s would like it.

IT is RESTful with their plans

Historically, the IT Internet/Extranet group has run the show. Choosing how to architect the ECM solution as a by product of their database and web application server architecture plans. Just figure out the storage of content and keep it forever. Here’s the argument: we already have custom applications for Documentum that work. All they care about is the “harvesting and production” side of content. They have no clue what the business is actually doing. How do we get IT to feel the pain of the business? A strong Client app director who cares about business process and content management would help. Get the IT managers out of the web services cloud and either do something enterprise wide with CMIS and other integrations or work with the OOTB products.

EMC Professional Services Where Are You?

The problem with EMC Documentum professional services is that it is not engaged fully during the pre sales activities of the sales engineers who are building POCs and demoing new software solutions. If they are they’re spread too thin and are not effective enough. They can’t keep up with actual work let alone consult on goals and vision. Most companies form their opinions on the can do all sales guys. The client is overwhelmed with products not solutions. They see scanning solutions, but not high end UIs which solve the complex financial relationship issues. A WDK or Taskspace form is a child next to the maturity of what IT has created for their highly customized UIs. EMC, what’s the solution here?

Sample Financial Solution Patterns

There are many solution patterns within the financial applications that EMC could solve with a product.

Center a User Interface around Client and Customer Ids. The pre vs. post sales information gathering activities.
Deal with the historical changes in attributes and their values. Financial services databases are always current and don't necessarily handle change management in content and metadata very well.
Like the case management solution, figure out the foundational use cases of financial services (beyond scanning and OCR) and build product and solutions from them.

eDiscovery’s Promise Is Dependent on Your Enterprise’s Laziness

2010-03-22T15:02:00.000-04:00

eDiscovery solutions are predicated on your company’s lack of cohesive and best information management practices. If you read any eDiscovery software solution’s proposal it will not mention how to fix the issues of inadequate metadata structures, lack of auditing, risks of pre-ediscovery spoliation. They will not tell you to get organized and clean house. They’ll say information governance is impossible to keep track of, that information growth is out of control. They’ll all be lawyers and MBAs and be able to pontificate extremely well about your issues, but do they really want you to help you where you really need it?

If I was selling software they was really a search tool and a legal CYA process would I be interested in telling how to create an Enterprise Metadata Repository? Do creditors want you to save money? Managing your content requires standards, conventions, governance, and compliance. MoReq is an EU set of standards for records management that basically outlines the keys to eDiscovery freedom.

Without the discipline and perseverance that is necessary to push these throughout your company you’ll be stuck with an eDiscovery application infiltrating your email, ECM repositories, file shares, etc. It’s a matter of how you want outsiders to carte blanch access to your information. If you know where everything is then you’ll be able to clearly define and execute the information discovery without a whole sale fishing expedition on all of your seemingly sensitive information. Most importantly you’ll know what needs to be disposed of!

So what’s it going to be a house of hoarders, or an extreme home makeover?

Architecture without an Effective Enterprise

2010-03-19T15:05:00.006-04:00

So what are some of the issues that arise when a large company does not have an Enterprise Architecture group? How could this happen, especially in a company which is dependent on validated systems and many different and competing standards?

Business Partners are frustrated with the lack of governance and execution of projects that matter to them. The project pipeline for smaller projects is large. The large projects get funding to trump the smaller ones, pushing the smaller deployments back by years. The Business threatens and sometime does build its own solutions.
Project liaisons like Business Relationship Managers, Project Managers, and Business System Analysts are overwhelmed and focus mainly on timelines, milestones, and deliverables without knowledge of other projects and synergies that might be available from an architecture perspective. There will be different levels of training and experience across application specific groups.
The Service Delivery Team lines up resources and figures out who can do what, when, and thinks everything else will just have to work. The issue is that other outside team resources may not be considered by the Service Delivery Team. For example, the infrastructure team may be working on 10 other different build outs and there’s no priority.
Solution Architects will clash with Network and Storage Architects who are blindsided by project changes and overwhelmed by lack of resources. The documentation of each group is not consistent. There will be no overarching document template standards across the application stacks. There are 15 different icons in Visio diagrams that represent servers.
Large Consultants will infiltrate easily into the heart of ERP and ECM and try to get into Legal and Records Management. These consultants will suck big bucks out of IT’s coffers. Knowledge transfer will be an after thought and spotty at best. The stability of process and deployed applications will take much longer to achieve.
The Integration Team will be very busy patch holes with webservices and other protocol, but there is no EA team looking at the whole wall of patches to determine where the future cracks will be. Each patch is customized with different metadata requirements, no common values, no Moreq2-type analysis. The testing of upgrades across integrations is too little, too late. For example, there is a general lack of awareness of the full impact of changes made in Active Directory.
There is no overall orchestration of applications, integrations, standards, ROI, and cost controls. The budgets for projects and their corresponding SLAs are out of balance. The downtime for upgrades blows away the accepted downtime window.
Priorities of projects lack coordination of architects experienced with building platforms which scale. IT Governance touts its own top 10 projects, while ignoring the priorities of the business partner’s projects and initiatives. For example, deploying Sharepoint 2010 with complete confidence in the first release vs. holding off on a major ECM upgrade until the first or second point release, even though the business is clamoring for the new ECM functionality, eg. Center Stage… Another example would be launching a large project with two versions of the same application because in the time it takes to deploy a new version is up and user are clamoring for its features.
Records Managers and Legal Departments will be forced to drive metadata standards at the enterprise level. They will most likely be stonewalled at every turn, especially by IT.
The Infrastructure architects are explaining too much and doing too little. One person questions the amount of storage required because of cost and another says don’t worry about the money. The racks in the server room have blades tucked into spaces that are not shelves. The power supply of the server room is maxed out. There is no coherent Storage strategy for expansion. Records Management and thus disposition is far down the road.
There is a high churn rate of smart developers and architects. There aren’t enough incentives to keep them. They are too busy putting out fires so they quit and move to companies with EA teams in place.
These are some of the issues that arise when a medium-sized company grows into large one. However the EA team should be in place from day one. All companies should recognize the importance of standards and centralized governance of IT architecture.

Sharepoint and the “Others”

2010-03-18T17:12:00.003-04:00

If you watch the TV show “Lost” you’ll know what I’m talking about when I say that Sharepoint is one of the survivors of Oceanic flight 815 and Documentum, Open Text, IBM, Oracle are the “Others”: the ones who are indigenous in the ECM space, the island. The Others are adept at moving around on the island. They know the lay of the land and the dangers that lurk at every turn on the trail.

Sharepoint is poised to set up camp on the easy to reach, safe locations like the nice beaches, or in the collaboration and project oriented areas of the enterprise. The Others will lurk in the deep jungle holding onto their strongholds. As Sharepoint becomes more equipped to fight the Others and outsmart them, the old patterns of ECM will evolve; Sharepoint propogates and splits into factions; there is no central goal or common standards in the way Sharepoint is used and spread.

This Sharepoint scenario is very similar to the internet boom. Remember what happened when the first wave of intranets hit? Individual business units spun their own websites overnight and they catered to their own unit’s requirements. Everyone was happy among themselves in their own insulated groups until the Others took out their sticks and started asking questions like, “what’s your metadata model?”, or “does this site scale?”, and “Can you search this thing”, “do you have a review and publishing process?”, and so on.

These “shadow IT” groups were slowly consolidated, shunned, taken off island, until at last a centralized ECM solution was implemented. The ECM solution was great for IT, with its centralized support and architectures. The problem was it took an act of god to add new functionality and it was expensive. The more centralized the ECM solution got, the more business units wanted to break away and fulfill their own needs. Enter Sharepoint: small and uncoordinated at first. But now with SP 2010 the Others are no longer that dangerous. The Others needed to evolve, but did not morph fast enough. The Others now are hiding in the strong holds of records management, index/search, business and forms processing, case management, scanning, and license agreements. The Others will be forced out eventually to face Sharepoint. Google and open source will follow in pursuit as well.

Meanwhile as business units go wild with Sharepoint and architects run behind them trying to put RM, metadata standards, storage in place, Sharepoint will be dealing with its own issues. The silos of information will not be fixed by technology, by search. Info silos will only be fixed by continuously evolving and releasing easier ways to institute standards and conventions around organizing/normalizing data and content. As soon as the structure of lists in Sharepoint is deployed, the structure will have to change. Sharepoint may well go the way of individual intranets. Getting the business as a whole to agreed on certain rules and policies and to comply with best practices is the real issue and IT can't do it alone.

Let’s hope the user’s obsession with the UI will go away and a new vision of usefulness and knowledge building and sharing will arise. Knowledge Management of the 90’s is just morphing into collaboration and search. Who knows what, where, when, how, and why is the framework to design to. Integration of desperate services by common interoperable factors (CMIS) is one of the keys. The Others have known this, but have been driven to distraction by profits and the friuts of the jungle. Now, as the island is rumbling for change, the Others will have to change their tactics and listen to the island for future instructions.

Documentum vs. Open Text Content Server

2010-03-07T09:42:00.003-05:00

Open Text Content Server and Documentum grew up together during the same time, but in different neighborhoods. Sometimes I wonder how certain architectural design decisions were made, or how lack of funding may have affected the level of integration between the products in their respective software suites. When compared with DCTM, OT is like an old Mustang and DCTM is a newer Toyota with recall issues. This is by no means a comprehensive comparison of the two software suites. It points to some important ECM aspects and tries to expose some of the issues and precularities.

Session Thread Model
OT has an understood session thread limit of 5 per server. Can you imagine this with DCTM? OT will fix this with their 10X release, but customers have suffered for years with this limitation. Session management in DCTM is mature, but has its own issues. So in OT the way to scale for many users is to build out Livelink servers (5 session threads) per server. If you had the potential for 30 concurrent sessions, then you’d have to build out 6 server instances. These OT instances would have to have all of the modules, patches/fixes, on each. This is 6 times the deployment effort. It took OT 10 years to scale concurrent sessions on their base server kernel?

DocBroker vs. Nothing
Here’s a real big issue with OT: failing over one repository to another. OT just can’t unless you want to purchase archive server which is not really part of the LL server and not pertinent to this. OT relies entirely on load balancers for distributing client requests. Yes, the docbroker has its issues too, but at least it can be used to failover one repository to another.

Object Model
Because Open Text doesn’t have the foundation classes that Documentum has it is hard to conceptualize the inheritance model and relationships between the underlying database tables.

dm_sysobject : DCTM’s main table which holds the chronicle and object ids of most of the objects in the repository (dm_type, dm_user, dm_acl are related by not derived from dm_sysobject). Document objects extend from this table in the DFC and are related to separate doc tables per configured doc type. Both architectures have main tables which are related to be most other tables. This has its pros and cons. Let’s take a cursory look at both models and try to compare some key areas.

Dtree: OT’s main table which holds the data id of all objects. If you look at the schema, it’s like a wheel with spokes all leading to the dtree hub.

Version Trees
If you ask an experienced OT administrator if there is a parent id to a document id, chances are he would not know the answer or he might even ask why is this important? Version in OT is not an option. In DCTM at least there’s a way to control versioning during the work in progress states.

dm_sysobject: The parent id of a version is the chronicle_id which holds all of the object_ids. The chronicle and object_id are in the dm_sysobject table.

version tree: dm_sysobject chronicle_id and object_id

dtree: The parent id of a version is the docid which is in the dversdata table and the version ids are in that table as well. The key between the dtree and dversdata table is the dataid which is the docid

version tree: dtree dataid = dversdata docid then versioned

Given this simple example of versions, you can see that OT’s table join for version look ups of basic system level content attributes may not scale as well as DCTM’s.

Permissions and ACL Inheritance
In DCTM there are separate tables and objects which manage permissions of folders and content, as well as system objects themselves. In OT, there is no such concept: permissions are inherited by default from folders only. In DCTM, ACLs can be inherited by object_type, user, or folder.

OT: folder only
DCTM: object type, User, Folder

If you think about the advantages that DCTM’s model has over OT’s look at how limiting it is for workflows which may rely on changing permissions to an attachment as it goes through the review and approval lifecycle states. In DCTM you set the ACL and you’re done. In OT you set the permissions for each user and group at every state. What a change management nightmare!

Object Types vs. Categories
When I first worked with OT’s categories and understood how they related to content and served as metadata tables, I liked them (kind of a cross between custom object types and aspects in DCTM). Then I learned that if you add an additional value to a dropdown selection, not only do you add it to the category, but the system has to update all of the content metadata as well. Each document is coupled with the UI’s potential values in a dropdown selection box. This is mind boggling. OT’s solution to this is the turn off the modification trigger during the category “upgrade”. This makes me worship DCTM’s value assistance.

DCTM’s metadata inheritance model: the custom object type table inherits attributes from dm_sysobject. This makes the doc type, plus an ACL is associated to each object. UI metadata and document metadata are decoupled.

OT’s categories: all content is dtree content. A custom category is an “object type”. The category metadata and the content metadata are coupled.

As you can imagine this does not scale. One attribute value change in the category has to “upgrade” potentially millions of documents without actually modifying any of the content’s metadata.

Development/IDE
I never thought I’d say this but compared to LiveBuilder, Composer (with all of its bugs) is hands down a superior development environment and deployment tool.

DCTM: Eclipse/Composer
OT: LiveBuilder

UI customizations: WDK/TBO vs. Modules
MVC development in OT is more streamlined than DCTM but it’s more manual and awkward. Deployment of OT’s UI customizations straight forward, like DCTM it requires a web server restart. A bonus of OT is that you don’t have to shag after caches between deployment environments. Despite OT’s cryptic Oscript, I bet the learning curve to be productive in developing it is faster than the WDK.

As far as extending core methods in OT, you’re out of luck. As far as I can tell, TBO/SBO extension customization concepts were never introduced into the OT environment.

DCTM TBO, SBO, WDK: JSP/Configuration/Java Behavior class
OT Modules: Oscript/HTML

Webservices
DCTM wrote the iECM spec for service interoperability. Like DCTM, OT plans to expand and eventually only use webservices for all MVC requests and responses. CMIS adoption and use will slowly rise as large companies sunset their old repositories and bean count the new services into their road maps.

Users/Groups:
OT’s users and groups are keyed by a Number, in DCTM the key is the user/group name or String. Using a number allows OT to manage the changes in User names easier, however there is no such concept as a federation of repositories. OT is designed around a central repository user concept. Users created in one repository will have a different id than the same users in another repository, however the synchronization of AD between environments mitigates this issue.

OT KUAF: Number
DCTM User: String

Potential for Home Grown Mess
The maturity of a repository is directly related to how much resources and attention to ECM standards were governed during the repository’s expansion. If the users were in control of the folders, permissions and content, then chances are you have a mess of projects, permissions that do not scale, and outside URLs that point to the repository’s contents from portals and embedded in documents.

Somehow companies with DCTM seem to be more adept at realizing the limitations of performance and scale when it comes to the difference between consumers and contributors. Maybe it was Webpublisher’s adoption success over OT’s lame attempt at the same.

The concept of decoupling the consumer interface from the contributor’s is what web 2.0 is all about. But both camps have an uphill battle to convert repositories in flexible content architectures.

Storage Management
What can I say EMC owns Documentum and it was a beautiful purchase in terms of storage management. I leave it at that though. OT’s archiver is an after thought compared to the power of taking BCV snaps of servers. Of course only large companies can afford EMC storage solutions…

Developer Community
DCTM developer resources cannot be touched by OT’s. I’m glad I learned DCTM’s solution before learning OT’s. If you are like me and learn the most by examples, DCTM is heaven. Part of the issue is the OT didn’t seem to know that Oscript was a very limiting factor in terms of adoption of OT as a whole. They didn’t get that in-house OT administrators need ways to develop their skills beyond following installation and admin guides. Learning Oscript meant you were locked into OT’s vision and world, but DCTM, Oracle, IBM, etc were out there as well.

Each section above could be a chapter in a book, but this a blog and I only have so much time to elucidate some of the points of comparison. It's hard to compare a teenage to an adult, but it's my opinion that OT hasn't been able to mature as fast as DCTM. This puts it at a disadvantage in terms of competing with Sharepoint and 2.0 technical advances and User's expectations in of UI functionality. So, the ability to be nimble and move with the technical changes in the world will prove which repository is more ubiquitous in 5 to 10 years. Also, our friends the bean counters will help slow adoption of new technology and especially new ways to thinking and expressing ourselves within the corporation.

Documentum Talent Revisited

2010-02-18T11:22:00.003-05:00

I recently searched google for documentum talent and came across an article from a fellow ECM blog entitled, “Documentum Talent”. The issue I have with this article is that although it is informative, it only scratches the surface in terms of understanding what takes to create true Documentum Architect talent. Java: yes; OO design: absolutely; ECM experience at any level: priceless.

First of all, the tone of the article is conceited and has one of those boss man attitudes which says, “Technically, I know it all, so don’t question what I’m saying. You are kind of a smart guy, but you are all minions to my brilliance, mere duck tape to my superior abilities…” Some architects are arrogant, but most are mere mortals, stuck between a rock and a hard place, trying to please the users and the bean counters.

Ok, whatever the tone, let’s focus on what is missing:

Business Requirements
I’m sorry, but Java and OO design as core to Documentum developers is obvious. If you can’t find this core in a developer, don’t hire them period. What is much harder to find (and more core) is a developer with business requirements gathering experience. Hello, what does the User want? This is why User’s are flocking to Sharepoint, “Talent” as defined here have been too technical and not business savvy enough. They thrive on xml payloads and not enough UI, search, and ease-of-use. Not mentioning “business” or “User” anywhere in his article on Documentum Talent explains the disconnect that has been going on for years between implementation and design. Talent is not knowing webservices only, it is knowing the “why” behind integration needs? Why CMIS has taken so long to emerge. The reasons behind needing to integrate applications are just as enlightening as the technical solutions that make it happen. Business units fund these applications, IT builds them, architects integrate them, but what decisions were made in the past that required two document management apps, for example, or why was Sharepoint introduced?

Knowledge of Competition
Sharepoint didn’t just appear this year to scoop up ECM. It has been around for years, honing its functionality. Taskspace was a joke, it was buying time for Center Stage which was a few years late. Now we have Sharepoint and Alfresco taking business away in circumstances where the EMC licensing contracts are up and the entrenched Documentum Users are fed up. Technology focused architects have helped create this mess. Just make it faster and improve the SLA, the users will be happy…

Long-term, Goal Oriented
Beyond the politics within an organization, how projects are funded, who is technocrat of what services delivery group, lays the long-term vision of the solution at hand. What role does the solution have to the ECM vision in three years? In five years? What are the overarching goals of IT? Knowing how to work with these, question them, and design to them is very hard to find in our technically “superior” architects. Knowing how the historical impact of ECM has influenced the current design and plans for the future. Who is promoting collaboration and why? These are key to talent. In politics, bringing in Sharepoint is the equivalent of promising jobs to your constituents who have been suffering from EMC’s lack of attention to your needs.

Change Management
I’ve seen many Documentum installations that are OOTB with shortsighted object models, security, folder structure, workflows, integrations, etc. These are the result of profit hungry partners smirking as they write their SOWs. Some partners purposely put their blinders on when it comes to scoping their projects. They know full well that as soon as the current project is over the client will be begging for improvements. These partners purposely take shortcuts because they rely on their client signing on for more work. Technically, the project could be architected well, but the human interaction issues with the software may not have been fully addressed, leaving the user scrambling to duck tape their processes to a UI that is too constrained. Knowing when to give the users something for free is a key sign of good talent.

Sharepoint vs. Documentum: Analyst Perspective

2010-02-15T11:02:00.001-05:00

It depends on who's perspective you want to take when you compare Sharepoint with Documentum. Every person has a unique take on the Web 2.0, its marketing and hype, but the bottom line is that it is cheaper than proprietary software and can be an open source application that eclipses both MS and EMC. When I think of Sharepoint, I think of projects and ease of use. When I think of Documentum, I think of content lifecycle management.

Resource Delivery Director: wants one solution, centralized with easy configuration. This person is usually religious on a programming language so depending which one .net or Java, this will be part of the decision. This person has pressure to deliver applications balancing this against resource expense.

Architect: wants scale, performance, and integration, plus local developers. This could be a problem for Sharepoint as good developers work for partners, they are not mature enough to contract on their own. Also, good luck out sourcing to India for Sharepoint folks. That will happen, but not yet. Integration and CMIS are also considerations. What apps are integrated, ease of webservice integration, identity management, workflow, etc.

Chief Bean Counter: wants a good ROI story. If a CIO has a preferred technology, she will find/procure a research study which is a proponent of it.

Users: want something that looks different than what they have, but they want everything they had. Do you put a new skin on an old application, or do you buy a whole new app?

Elephant in the Room: Open Source is coming, so EMC and MS are really on the same side here. Alfresco is not going away, neither are the other players. Google might trump everyone once it can’t grow everywhere else, why not tackle the source?

I have witnessed the demo difference between the two platforms and Sharepoint wins hands down. Partners selling these two products are the front line in this discussion. Partners who sell both are selling one short and hedging their bets. Find a partner who sells either EMC or MS and get their pitches, then write the matrix and gap analysis. Also, throw a third party in there for balance.

The main issues you have to matrix are business requirements and the key stake holders. Also, who’s a member of the steering committee.

There Is No Such Thing As "Unstructured" Information

2010-01-23T13:04:00.002-05:00

All information, content, documents, chunks have implicit and explicit context and metadata associated with them, therefore they are structure to a certain degree. One of the major issues of keeping tracking and finding content can be solved if the “who” in the equation is dealt with.

Take Records Management. First, the concepts are agreed upon; second, “we have to get something out” requirements are decided on; third, the tools are picked; finally, the issue of how to tag the information is arrived upon. Oh yeah, the creators of the content. How to get then to change the way they tag information, their habits of thinking around identifying their content for world consumption?

The legacy content is another issue altogether. Are you going to force authors to tag and reorganize their whole c: drive? Are you going to auto categorize the share drive and hope your taxonomy is comprehensive enough? Are you going to set up a master/proxy that layers a meta metadata structure on top of everything? Will there be a hybrid approach to humanly sort out the who, what, when, where, how, why (Zachman).

The point here is that it’s the Users, the corporate culture, that have to change their modus operandi. Sound familiar KM? How many times do we have to come up with applications that work around the ultimate issue of training users to think in terms of the group and the community, of sharing not hording ideas and knowledge? Social applications like Facebook will slowly seep into the minds of corporate Users and break this road block wide open.

Let’s hope that Center Stage grabs this trend and really takes off. If it’s easy to configure during a customer demo like SharePoint is, then maybe it has a chance. The overworked, whittled down IT departments out there want easy to administer enterprise software, that can be mostly configured. As soon as Documentum builds in a more comprehensive configuration tool (yeah I know Composer, right, not quite) the issues of customizations breaking or being too focused will be a problem of the past.

File shares in corporations are not “unstructured”. They actually are very structured, but to an indexing application with a taxonomy, they are “unstructured”. The real issues are change management of User behavior and ECM applications, governance with a small stick, and software that IT can be lazy and cheap about. SharePoint is already the next dumping ground, but User’s have learned just a baby step more about tagging their content so that others can find and understand it.

Diagnosing an ECM illness

2010-01-15T14:47:00.003-05:00

As ECM projects and systems grow through awkward teenager years and start to show signs of age, there inevitably comes a time when symptoms of an illness show themselves and can’t be ignored any longer. The existing management is usually blind to the issues. The issues are inherent in how they make decisions and thus restructuring the governance and budget process is usually part of the treatment.

Symptom:

Lots of Testing on Production and extended down times for maintenance.

Diagnosis:

Lack of QA environment that is exactly like Production to validate test scripts
The more testing, the more risk of extended downtime and possibility of rollback
Relaxed emphasis on testing and more attention to deadlines

Treatment:

Create a QA environment which matches Production down to the hardware (to test server swaps).
Establish a mandatory QA and Validation process for the QA environment, thus reducing the risk of running major tests in Prod.
Schedule more time than anticipated for testing in QA
Budget extra resources for testing

Symptom:

Unorganized folder structures and search results that are useless.

Diagnosis:

No taxonomy standards
Lax governance
Fuzzy goals for business requirements
No discipline in term of awareness of others who are outside of the team
No training of common principles of organization
Poor change management
No organized Taxonomy: lack of governance, training, standards
Bad search results: bad metadata, values are too haphazard

Treatment:

Establish goals and standards
Create a governance structure that can force business units to change
Study current structures and come up with a plan to rebuild them with the acceptance from the key stake holders.
Train superusers on information architecture

Symptom:

A call for “re-branding” an implementation

Diagnosis:

Managers did not listen to the subtle whispers of dissatisfaction with the product.
UI complaints accumulate, they don’t just happen.
There are other factors that prevent upgrades to existing software or new software to replace outdated solutions.

Treatment:

Illicit feedback and make changes continually
Identify the lack of awareness of management to issues of business productivity
Identify why the system lasted so long in its current condition without treatment
Create mandates to leap frog to new technology every few years
Design and architect for change

Symptom:

Extended Downtimes in Production due to performance and scaling issues

Diagnosis:

No Failover or HA to fall back on during inevitable times of failure
Cutting corners on budgeting and resources
No QA environment for performance testing
Lack of enterprise architecture plan and resources

Treatment:

Budget for failover contingency in hardware and software
Create a QA environment
Understand the lessons learned from the whole solution development lifecycle: where things went wrong and steps to take to remedy the issues.

Symptom:

Postponing/rescheduling work execution due to emergency, “all hands on deck” system downtime triage

Diagnosis:

Poor program and project management
Poor resource management
Resources that are not trained and experienced enough
Sign of a series of mistakes during the whole project, no one resource is to blame

Treatment:

Have clear goals stated up front
Do not cut corners to meet bogus timelines
Make sure there is enough time allotted for testing

Symptom:

Slow performance

Diagnosis:

No architecture plan for scaling of the application
Lack of performance testing in QA
Too much leeway given to power users who are not trained enough and only care about their own content
Poor budgeting process
No standards for change management

Treatment:

Transfer the management of the application if this has been going on for a long time
Architect a solution from the ground up which is flexible and scalable enough to deal with the use and capacity issues
Think about dividing up the applications if the current one is monolithic

Symptom:

Stealing infrastructure for other higher priority projects

Diagnosis:

Poor communication from management of goals and forecasts
Management is feeling pressure to succeed after a few failures
Infrastructure procurement process is too slow to be effective
Resources are frustrated from infrastructure and architecture to implementation teams

Treatment:

Budget and purchase hardware well in advance
Do not accept a project if it entails cannibalizing hardware and resources from other projects unless the whole teams is fully on board with the sense of urgency

Symptom:

Infighting during deployment planning meetings

Diagnosis:

Lack of cohesive governance
Favoritism within the teams or from higher up
Lack of trust
Lack of common understanding of goals or standards

Treatment:

Create common understanding of goals and standards
Communicate often and effectively with the team(s)

ECM Framework Equals Aligned Goals

2009-12-18T14:45:00.003-05:00

All organizations need an ECM architectural framework in order to move in unison toward common goals. I know all about the phased, lumbering approach of ECM deployments in the enterprise and everyone’s intersecting plans of what to do about it, but I have been through this before a few times and I have to underscore some ideas and questions.

I’m going to use a content management software upgrade as an example shock to the ECM system. My focus is on the results of a lack of clear goals around ECM.

Why use an architecture framework?

Silos of ideas need to be reviewed and galvanized into a commonly understood direction and focus. We all have visions, but are they all in line, fully clued into each other? There are budget forecasts, collaboration strategies, content management strategies, roadmaps of what will be executed and when, top 10 projects, ECM strategic meetings, but is everyone looking at the same picture?

Budgets show the how many bucks and what quarter.
Strategies based on Gartner or McKinsey reports overwrite each other.
Roadmaps show what, who and when.
But where’s the why and how?
What are the goals???

For example, at the functional ECM level, how will an upgrade be impacted by SharePoint? What is the anticipated impact of a Usability report, can we start thinking about executing the obvious changes now? Have the mission critical regulation focused groups been frustrated with our services, why aren’t we working with them to implement their submissions to the FDA, shouldn’t this be our number one priority?

Due Diligence in a Vacuum
At the document management level, an upgrade should be done “in place” with limited scope and repositories should be eventually consolidated to save money and centralize control. However, at an ECM level, the approach to the upgrade is magnitudes more complicated and interrelated to other strategic and governance factors.

Ideas and questions around the upgrade and restructuring at an architectural level

Will consolidation of repositories meet the imperative cultural goals of the company? What is the SLA threshold for upgrading a system? One week of downtime? Zero downtime? Plan for a year or execute in 2 months? Wouldn’t it make sense to isolate mission critical content by business unit and their specific SLA’s for FDA submission, audits, security, etc?
How does the upgrade fit into a more tactical approach based on commonly understood ECM goals / strategies (framework)? If the long-term vision is to migrate away from LL, then let’s face it strategically. If we thought of ECM in terms of services like business process, records management, FDA submission, publishing, migration, security, library, retention etc., would this help us define our goals?
Should the upgrade methodically move non-regulated content to a parallel upgraded system to free up the rest of the company from the rules and regulations of a validated system? Should we assume that Sharepoint will consume all of projects eventually, and if so, are we being methodical about this?
Should the controlled repository be migrated to another upgraded repository, or should we separate repositories by service, for example, business process management/SOP (current controlled upgraded in place), file share sandbox and portal (new repository), etc.
Should each mission critical business unit have its own repository (within the framework of the ECM stack) to be unencumbered by the goals/governance of other mission critical content of other business units?
At one company I worked at the content, structure, security and business process were consolidated around common business goals and anticipated use: finance, portal, mission critical business units and remote sites each had their own repository with workflows and migration tools for integration between them. If agility in change management is one of our goals this distributed approach should be considered.
What are the primary goals, mission critical drivers of ECM at the Enterprise level, Business Unit, and Service levels?

The mere fact that there are so many questions around an upgrade is why you should absolutely consider a framework for change management/deployment of services.

Here are a few among many: Zachman (IBM traditional) and TOGAF (ECM focused and complicated, but comprehensive).

eDiscovery Due Diligence Approach

2009-12-08T10:36:00.002-05:00

Requirements, Requirements, Requirements

Define your legal hold requirements, not at a high level, but at very specific level. Create a few scenarios. For example, the chain of custody of a copy of information vs. “frozen” information in context will present itself very differently in court. These requirements, like records management requirements, filter down to all information in the organization. It is worth the effort to understand the intricacies of each existing system and their integrations.

As-Is Systems

Determine what information is needed for the Legal Hold tool to function correctly, for example, assuming identity management is important to discovery then is IDM/AD in good shape? If it isn’t, when will it be? Does the Information Security group have enough resources to deal with this new software?

Inventory the existing information repository vendors to determine if they have eDiscovery add-ons which might be adapted to or used out right. For example, Open Text has an eDiscovery module tailored to its Livelink software.

Integrations

Interaction of the proposed software solution with existing systems is very important. For example, how well does EMC’s solution adapted to Open Text repositories? A few more: Email holds? File system shares? Identity Management?

Tradeshows and Research Analysts Analysis

At tradeshows the players with the deepest pockets are going to wow the audience with all of the bells and whistles. “Yeah, we can do that”, but it’s a customization… Even the demos are usually canned and not real. The reality is that it’s up to your specific requirements. Gartner’s quadrant could be based on a pure play model not an integrated one. I agree with Gartner that solutions are still in their awkward stage, which is more the reason to develop specific requirements.

Search vs. eDiscovery focused

Autonomy is an excellent search tool, however, is it going to integrate well with our other systems specifically around the access control aspects? This could dovetail nicely into an Enterprise Search tool effort…

Security Group Participation

The Information Security group must be an integral part of this whole approach. Without their participation and buy-in from the start, it will be an uphill battle. They will obviously work in tandem with Legal to perform eDiscovery activities. They need to be comfortable with driving their cruiser.

Professional Services after the purchase

I’m not sure of the overall percentage of services vs. software in Legal Hold software, but I’d say it is substantial. Weighing the specific requirements against the software’s out-of-the-box offerings will be worth the effort.

When An Interview Becomes Consulting

2009-11-16T22:04:00.004-05:00

When you're interviewing with a potential employee and he asks a specific question about a current issue that his team is having with their content management system, how are you suppose to answer? If you answer vaguely he might think you don't really know what you're talking about, but if you answer completely and might cut the interview short and send you home in order to run down to his team and tell them how to fix the problem.

These days financial companies are the worst because they were used to living fat off the hog for the past few years spending big bucks on contractors to do their development work. Now they have a work force of implementors, but no one experienced enough with development to take on the new custom work. So what do they do? They go ahead and try the development tasks and get into trouble along the way. In the meantime they are creating an environment in IT that is toxic to developers so anyone with talent moves on and the manager is stuck in a never ending interview cycle. But, hey I have a great idea, let's get some really experienced developers in here and interview them and get their advice on our issue! Brilliant! Short sighted, just like this latest rise in the stock market.

So how does you avoid free consulting? Here are some tips:

Confuse the hell out of them, laugh and walk out
Say you want to meet with the developers themselves to solve this issue
Start answering the question, but stop short of answering saying they have to hire you to find out
Answer the question, but introduce other aspects of the issue and build in explanations of risk
Ask if they've ever tested their disaster recovery system, fully
If the manager is an ass, ignore his questions, talk around them until you get kicked out
Ask how large their IT compliance team is and whether they are hiring
Start saying "just kidding" after every other sentence

The bottom line is don't give sleaze balls free consulting. We're valuable and they know it. Now all we have to do is wait until they pay us the big bucks again in 2012.

Composer and ACHells

2009-10-16T10:52:00.000-04:00

Hosed by Composer again! This time trying to install Permission Set from Dev to Test Repositories:

Environments:

Dev Repository name: LoserDev
Dev Repository Domain User: LoserDev
Test Repository name: LoserTest
Test Repository Domain User: LoserTest

Created a Permission set in Dev, which set the ACL domain as LoserDev which corresponded to the LoserDev install parameter in Composer.
Installed it into LoserDev and everything unit tested fine.
Went to install the dar file to the LoserTest repository and got an error: “user LoserDev does not exist in Repository”.
** Don’t be tempted to create the user in the repository. This will allow the dar to install, but will really confuse the UI with ACLs that kind of work, but not really.
Went back to the LoserDev Project and opened the LoserDev install parameter, typed in “dm_dbo” into the default value box, saved it, and created another dar.
Went to install the dar file to LoserTest: same error. What the?
Went back to LoserDev Composer project, check the LoserDev install parameter and the default value was blank. Hmmm.
Type the default value into the user parameter value of dm_dbo again and hit the enter key. Ahha! Saved it, created the dar and the install to LoserTest worked.

Bottom line: In Composer make sure you have an asterix * in the tab, to guarantee that your work is getting saved to the underlying xml file which is used to create the dar file.

Documentum Composer Wrestles with Lifecycles

2009-10-16T09:23:00.002-04:00

Hosed by Composer again! Whoever thought Composer was ready for primetime with Lifecycle management was really in the clouds. Here’s what happened:

I created a Composer Project (call it Poser) and a new lifecycle (let’s call it DOA) along with many other artifacts.
I installed the Poser.
DOA had issues with ACLs and was not working correctly
I thought it would be better to create a delta dar for modifying just DOA
I created Composer Project 2 (Hoser) and imported DOA from the repository
I fixed DOA and installed Hoser.
Now Webtop showed 2 DOA lifecycles.
Naturally I deny that anything wrong is happening and I choose the wrong DOA, try it and get frustrated.
I DQL, I look at ACLs, I search Powerlink, I download Doc App Builder 5.3sp65
So I go back to Hoser, import the original DOA and click the “uninstaller” checkbox to uninstall both DOAs.
I install the Hoser again.
Now Webtop showed the original DOA still installed…What the? What got uninstalled?
At this point I installed Documentum App Builder, created a docapp, imported the lifecycles and uninstalled them. I fixed the DOA and had no other issues. This is still the true work horse!

Bottom line: once you create a lifecycle and actions, stick with that project, don’t create new projects using the original artifacts.

Content Architecture using Memetics

2009-10-05T13:31:00.002-04:00

Virus of the Mind: The New Science of the Meme by Richard Brodie, describes the foundation of memes as being distinctions, associations, and strategies. When applied to Enterprise Content Management there are some interesting comparisons. Advertisers know how to push our buttons to drive sales, just as content architectures know the best ways to describe and find content, or do they?

Distinctions

Example: It’s snowing, or our content is all on a share drive.

These are the ways to describe content that are particular to the business unit, the company, and the industry. This metadata is vital for survival of the content, in other words, can a User find it among thousands or millions of other pieces of content? What key information can be drawn out of the content file or its context to direct successful search result?

Associations

Example: Snow is dangerous when driving, or without metadata I get thousands of search results.

Relationships among content and its environment are key the understanding the thoughts (memes) behind the content. A taxonomy helps by categorizing a business unit’s way of thinking for its search or retention purposes. This taxonomy would have to fit into the enterprise as a whole. The issue here is to start at the level where the content is created and is useful to the local users, then expand the levels out in a way that doesn’t disturb the functional aspects of the original group. Too much of an imposition will get rejected or worse slowly ignored.

Rules and Regulations come in to play for controlling and focusing content for common delivery to people and interfaces outside of the company’s mindset.

Fuzzy vs. Absolute: Users want to be able to fill out metadata and find that exact content later. This means the content architecture has to balance the business unit’s requirements with the enterprise's.

Strategies

Example: If I have an all wheel drive I’ll make it through the snow, or with a taxonomy I can make sense of complex organizations.

Repetition: this is used to drill home the importance of certain ways of thinking (memes). For example, a naming convention will reinforce ways of thinking about content and its context (association meme).

Cognitive Dissonance: this is used to reward a User for taking the time to fill out metadata correctly. For example, filling out metadata and associations is rewarded with less change management, less hassle in the future when ways of organizing content changes.

Content Silos

However you want to attack the issues of content silos, they will always exist. The strategy memes of the business unit will always differ in meaning and scope from the enterprise. I’ve come to the conclusion that the best way to make importing and/or changing content the easiest on the User is the hardest to figure out in terms of scale and performance on the system. This means that finding the right balance of splitting up the system’s resources for each business unit weighing the content demands, the ability to find content, the access control, and the application of the latest rules and regulations. This balance when seen visually will make sense, but the challenge is to get agreement from all the parties involved, the governance. This is where strategy memes make inroads: they help far removed executives understand the long term benefits when seen from the past, present and future.

Why Test Scripts Suck

2009-09-16T12:43:00.003-04:00

I’ll trade one motivated business user in for five IT testing professionals for making sure an application works as designed. Why? Because that business user has all of her day to day requirements, pain points and frustrations invested into the new application. On the other hand, the testing professional has to make sure the test script is executed flawlessly, that’s it, on to the next project.

Below is the typical best practice, software development mantra that project managers will promote. I’ve added some notes under each of them and few new mantras.

The software implements the required functions.

Have the requirements been aloud to change during the project? Flexibility is key to the perceived success of any project. If concessions can not be made without huge push back and it’s a pain to change requirements from the business’s perspective, the project should be stopped and re-evaluated for its purposes (and management). This is very pronounced with large projects.

Added to normal Project Manager’s software development lifecycle list:

Prototyping

Prototyping functionality for business users to experience (see) what’s been talked about and promised. If a third party or internal team cannot commit to show their application during development for fear of “giving a bad impression” or “scaring” the business then there are trust and lack of communication issues going on that need to be dealt with now rather than during the full User Acceptance Testing.

The software has passed unit testing.

Make sure the developers know the architecture of the application as a whole, its requirements and the importance of the unit testing and integration into the larger application or service. If one developer is slacking the project is at risk of failure. Project managers should have a good idea of who can perform and who needs help at this point in the project timeline.

Added to normal Project Manager’s software development lifecycle list:

Code Reviews

A junior developer cannot possibly know all of the ins and outs of the application if they are focused on coding specific components or services. All code, at least initially, should be reviewed by senior developers and architects to assure efficiency and scalability.

The software source code has been checked into the repository.

This can be a pain in the ass if the project is small, however necessary if you are developing with others and integrating into a larger repository. This also is a good check for senior developers to do quick code reviews.

The software has been compiled into the current build (for compiled systems) and deployed into the appropriate test environment.

Without proper safe guards, one developer’s code could break a whole series of other test scripts. Smoke testing is highly advised before fully committing the code.

The team has developed appropriate test scripts that exercise the software's capabilities.

These scripts are usually end to end tests that are few clueless testers, not irrational business users, who change their minds, cancel, go to lunch, upload their whole hard drive, etc.

The software has passed integration and system testing, and been deployed into the user acceptance test environment.

Many large projects are desperate for true testing environments are usually skimp on resources for them. This poses an issue when the new build is supposed to be deployed and fully functional in a test environment that has kludges.

Added to normal Project Manager’s software development lifecycle list:

Performance and Scalability Testing

Normal third party developers comment during the development phase of the project that they wonder if this will scale or how it will perform under a load. These developers talk to the project manager and usually the discussion ends there. If this is brought up at this time in the project with no time allotted to it then forget about it. Also, why would the third party be motivated to do this when this is a typical reason that they get called back in to do more business?

The users have had an opportunity to use and respond to the software, and their change requests have been acknowledged and implemented where appropriate.

Again, this is important, but there should be no surprises at this time. The users should have had their requirements changed and prototyped and testing by this time.

The software has been documented in accordance with whatever standards your project follows.

Have you ever seen a test script written for documentation accuracy?

If documentation is not ongoing during the whole project this document will be worthless. I have not worked on a project where the design document is perfect after being signed off on. During development and fixing, the design doc needs to be corrected, changed, or expanded on.

Also, the deployment and knowledge transfer documentation should be complete and tested.

What the F’ Happened to the Customer’s Vision?

2009-09-13T21:43:00.000-04:00

It seems every new technology or architecture or new way of looking at the complexities of content is like building a new platform on quicksand. It eventually sinks below the surface and then a new “genius” comes along with a solution that gets sold to our “shock and awe” addicted users.

The customer used to always be right, now they are sold what’s “right”. What is sold to the customer is pretty and “easy-to-use” technology which is over their heads. They become reliant on experts to build the solution and to come up with language that makes the Manager/Director look good to his superiors.

Once ECM is in place, the users look at it and inevitably want their old system back. After a while they become more comfortable to the new ways of doing things. Then they want continuous improvement. By the time this happens a new version is released, new bugs cause the experts to come in and fix them. The continuous improvement requirements get scaled down by technology issues of performance and content growth. The IT department thinks they own the system. The Business Units get frustrated with IT. Yada, yada, yada.

At this point “shadow IT” starts its cycle again. In the late 90’s it was websites popping up every where as intranets via easy to use, inexpensive website publishing tools. Now it’s Sharepoint portals. These portals are what the customers want. They want messy rooms (unstructured content spaces) where they can play with content and ideas, not technology. Metadata, security, taxonomy, workflow, lifecycles, retention, etc. need to be worked into these “messy rooms”, periodically cleaning them up, organizing the useful content and throwing away the building blocks. EMC’s CenterStage, like Sharepoint, is trying to fulfill the need for users to produce, edit (collaborate) content while the systems handles structuring and storing behind the scenes.

This introduces the big gray area of ECM: the void between structure and unstructured content. Let’s say an invoice is structure content because it originated from a database and has a number. The problem is that this invoice was printed to paper, signed, scanned, and place back into a content repository. The number is still there, but the systems are different. Even though the systems are interoperable there is no source of record anymore. What is more important the financial aspect of the content or the actual scanned proof of purchase? It depends who you ask.

Solution Pattern for OOTB Webtop and Search

2009-06-04T11:23:00.003-04:00

Scenario

The client requires a multi-tiered custom object model with most of the attributes at the child level.
There are 10 distinct types of content which share some common attributes, but have very specific attributes as well.
The client want to search for attribute/value pairs across all of the children documents easily by selecting one parent object, typing in search criteria, and executing.
All imports, checkins, new templates, and properties interfaces need to be tailored to the client’s specific requirements for conditional attribute population in specific order.
All content types have some mutually exclusive required attributes that cannot be null.
And of course, the client does not want a lot of Webtop customization.

Possible Solution Routes

Traditional: Customize Webtop import, checkin, new doc, properties, and search pages.
New: TBO with common attributes pushed to the parent object type, with limited WDK customization.

Design Road less Traveled By

We isolated the absolutely necessary WDK components that needed to be developed to satisfy display and functional requirements. There is some functional requirements work here to decide on which attributes are common and have the most impact finding content in the repository. Searching across all child object types was a critical requirement so we focused the design of the object model toward shared or common attributes. The design of the object model looked like this:

Parent (common_attr 1, common_attr2)

Child 1(common_attr 1) --- Child2(common_attr 1, common_attr2) --- Childx(common_attrx)…

Every time a document is saved, the common attributes of that child doc type is replicated to the common attributes of its parent. Besides display customization of search results and grid columns, there is no need to customize the query builder of the search page. The query performs better because the attributes are at the parent level, there are no database table unions happening behind the scenes during the execution of the search.

Developing the TBO

The main purpose of the TBO is to override the Save and Checkin methods that are triggered during the use of Webtop. This TBO basically gets the common attributes of the content being saved or checked in and sets their values to the parent attributes. This TBO is deployed to override the parent object type.

Developing the Webtop Components

The customizations are to the object and doc lists. The “documentum.webtop.webcomponent.objectlist” and “doclist” components have the common attributes added to them in order to show the same view of the attribute columns for browsing folder contents and search results. This could have been a preference setting, however there was a bug with the particular version at the time in terms of sorting the columns so we had to customize these. The “search60” component was also changed to show only the parent object type and its children object types in the search.

Max Session: Obscure Documentum server.ini key saves the day!

2009-05-04T16:51:00.006-04:00

Environment
Windows D6.5 installation using Webtop, some WDK customizations, and a TBO for major customizations. The TBO has to create a superuser session to do some work.

Issue
Max sessions are reached on the application server after a few hours of use even though the “Active” sessions are far below the configured threshold. In other words, the Tomcat server is counting “Active” and “Inactive” sessions in it determine of “max” sessions.

Keep in mind that there is a lot of customization in the TBO and this required creating and releasing a session manager for a superuser during each save of the document. This is what is building up the "Inactive" sessions.

Patch Solution Given
The initial solution was to jack the max sessions up to 100,000. This caused the Tomcat service to die once a week or so, basically maxing out the memory allocation to the process.

Max Sessions Investigation
I opened the Documentum Content Server Administration Guide and searched for “max” or “session”.

I listed the variables in the session being maxed out

Application versions
Property files
Custom code

I looked into implicit and explicit sessions

Hint: For testing superuser in TBOs, I used a different superuser than the install owner account.

Application Server

Measured by Active and Inactive Sessions
dfc.properties Key/Value Pair: dfc.session.max_count = 1000 (default)
DQL: execute show_sessions
DA: Administration > User Management > Sessions (All)
web.xml: HTTP session timeout is set in the \app server\ web.xml (default is 30 min): 30
Hint: To find leaks set dfc.diagnostics.resources.enable = true (default is false)

Content Server

Measured by Active Sessions
server.ini Key/Value Pair
concurrent_sessions = 100 (default is 100, max is 1024). These sessions are “Active” sessions from the content server’s perspective
history_sessions = (how many timed out sessions show in list_sessions)
history_cutoff = (default is 240 minutes)
client_session_timeout: default is 5 min
check_user_interval: frequency in seconds which the CS checks the login status for changes.
Default is 0, meaning it checks only and connection time.
login_ticket_timeout: length of time a ticket is valid, default is 5min
DQL: execute list_sessions

Final Solution

I added “history_cutoff = 5” in the Content Server’s server.ini file.The “history_cutoff” key controls the longevity of the inactive sessions. The default value of this key is 240 minutes (4hrs). This would explain why only on occasionally the max session is hit.

My testing has shown that if you set the “history_cutoff” key to a value much smaller like 5 to 30 minutes, that this allows for the inactive sessions to clear reasonably soon, so as not to fill the max sessions of the Tomcat server.

To test this I set the following:

Set up WDK Automated Test Framework to run the same tasks over and over again to build up Active and Inactive sessions.

Set up the Content and Tomcat servers with these base line settings:
server.ini file: concurrent_sessions = 20
dfc.properties in the webtop/WEB-INF/classes: dfc.session.max_count = 30

Result: The Tomcat server fails when the total of Active and Inactive sessions exceeds 30.

Then set up the Content and Tomact servers with these settings:
Settings with history_cutoff changed
server.ini file: history_cutoff = 5 and concurrent_sessions = 20
dfc.properties in the webtop/WEB-INF/classes: dfc.session.max_count = 30

Result: The Tomcat server fails only if the number of active sessions exceeds 20, thus relieving it of the inactive session burden.

Creating and Deploying Templates Using API and DQL Scripts

2009-04-25T20:08:00.003-04:00

Recently I thought I was finished working on a project that had some templates in the design and deployment. We were “done” which meant the budget was depleted and the customer wanted us gone; no more billable time. I’m not sure what happened to the “customer is always right”, but this statement and sentiment is coming back in popularity. In this economy it makes more sense to bend over backwards to please a client, than bicker over getting paid for our own mistakes.

The issue with the templates eluded my developer and me. My developer had created an API script (see below) to load in the templates. The script created the doc objects, set the content, and set the i_folder_id of the doc to the “/Templates” cabinet object id. The templates were “linked” into the cabinet and seemed to function as desired.

However, as the in-house developer at the client site found out later, the templates were not truly linked to the “/Templates” cabinet. The in-house developer had the advantage of sorting this out over a much longer period of time than we as consultants had. That being said, I should have figured this out, but I was confident that my developer’s script was correct, plus a dump of the template object looked okay.

Here’s the one attribute of the template object that we missed: “i_reference_cnt”. It was “0” instead of “1”. The “i_folder_id” was correct, but the “i_reference_cnt” was not set correctly. The script was setting the i_folder_id value when it didn’t have to. The object gets linked to the home cabinet of the session account be default. A follow up DQL can be run to move the object from the home cabinet to the ‘/Templates’ folder.

There’s a support note on Powerlink which describes how to create custom templates from using DQL and copying using DA. You can also try the following API and DQL script that was modified to work correctly for deploying template files.

API Script

create,c,dm_document

set,c,l,object_name
Test template

set,c,l,owner_name
dmadmin

set,c,l,a_is_template
true

set,c,l,title
Test Template

setfile,c,l,C:\temp\test.doc,msw8

save,c,l

DQL Script

update dm_document objects move to '/Templates' where object_name = 'Test template'

go

Documentation: What and When

2009-03-17T10:09:00.002-04:00

In software development we test everything but the project’s documentation. I can’t tell you how many times I’ve had to scour a project’s documentation for information that should be organized for quick reference and be up-to-date with the latest configuration and customizations. Instead the documentation is usually missing some crucial bit of detail that forces me to search for answers and waste time, mine and the client’s.

So, to get back to putting more emphasis on verify or using documentation: how do we do this, besides test scripts and installation docs, or design and requirements docs?

One approach is to log more diligently all of the issues that happen during the development and deployment of the project. These logs have vital setup information and deployment hurdles that will never get documented formally. The testers and developers are the keepers of this knowledge and need to document it as they work through problems encounters.

The problems lead to the most important aspects of the project’s success. The problem’s solutions will suffice for the time being, but they will strike again in a similar fashion, in a pattern. These patterns are what need to be understood.

For example, you deployed a workflow with an auto activity that timed out during the QA testing. The timeout setting was increased, but no one documented it. When the workflow gets deployed to Production the same thing happens, but users see the workflow has paused and are now concerned and annoyed. The first thing you do is read through the documentation which has no reference to timeout changes. Then you look at logs and see that a method has failed with no reason why. The workflow supervisor’s inbox is filled with errors but you don’t know that because no one documented how to occasionally check that user’s inbox. No one even considered a fast system with a few workflows timing out.

I think the point here is that documenting is not only writing about the design of the system, its configuration and customizations, but detailing the pitfalls and hurdles of the process as well. There could be two sets of documents, one for the client and one for your sanity when things are wrong, which they will, it’s just a matter of time. Next time you'll be more prepared with a cheat sheet and quick references to previous issues and complex configuration and deployments.

Documentum Maintenance/Procedure Checklist

2009-03-02T16:52:00.007-05:00

After the initial Documentum installation and rollout of the first phase, it is essential to
follow a maintenance/procedure checklist to assure maximum system performance and stability.

Documentum Administrator
Many of the maintenance procedures and jobs are configured or accessed through Documentum
Administrator (DA):

Server and Repository configurations
LDAP configuration
Users, Groups, Roles
Security (ACLs)
Storage (Locations, Storage, and Filestores)
Index Agent’s failed index list should be understood and resubmitted if necessary

Maintenance

Logs to Monitor
It is highly recommended to check all logs periodically for errors and warnings.

Application Server
Name: stdout_yyyymmdd.log (example: stdout_20090218.log)
Location: \Program Files\Apache Software Foundation\Tomcat 6.0\logs
Purpose: shows warnings and errors from Webtop and TBOs.

Content Server Repository Log

Name: DocbaseName.log
Location: C:\Documentum\dba\log
Purpose: Shows the repository startup output and any warnings or errors.

Java Method Server Log

Name: access.log and DctmServer_MethodServer_DocbaseName.logLocation:
C:\Documentum\bea9.2\domains\DctmDomain\servers\DctmServer_MethodServer\logs
Purpose: tracks access and status of the Java Method Server

Index Server Log

Name: access.log and DctmServer_IndexAgent.log
Location: C:\Documentum\bea9.2\domains\DctmDomain\servers\DctmServer_IndexAgent\logs
Purpose: tracks access and status of index agent

Disk Space Management

The Content Server has a state of the docbase job (dm_StateOfDocbase) which monitors
this. Also the data drive should be monitored.

The SQL Server transaction log should be monitored
The Webtop cache files should be monitored
The Index data drive should be monitored
Database Maintenance and Logs
Disk space should be monitored
Transaction logs should be monitored
CPU and RAM usage patterns

Jobs

Some of the jobs below are not active OOTB. They have to set to active and started on a schedule. Be sure to set the run times so that they do not conflict other jobs and backup
schedules.

dm_ContentWarning

Purpose: Warnings for low availability on DM content/fulltext disk devices
Method args: -window_interval 720, -queueperson, -percent_full 85
dm_DMClean: Executes dmclean on a schedule Method args: -queueperson, -clean_content TRUE, -clean_note TRUE, -clean_acl TRUE,
-clean_wf_template TRUE, -clean_now TRUE, -clean_castore FALSE, -clean_aborted_wf FALSE, -window_interval 1440
Note that the "-percent_full" value is "85" which you may want to lower for a more lead time to deal with diskspace.

dm_LogPurge

Purpose: Removes outdated server/session, and job/method logs Method
args: -queueperson, -cutoff_days 30, -window_interval 1441
Note the "cutoff_days" parameter should be set to a reasonable number of days, balancing compliance and trouble shooting issues.

dm_StateOfDocbase

Purpose: Lists docbase configuration and status information
Shows: Number of docs and Total size of content, among many other stats.

dm_AuditMgt

Purpose: Removes old audit trail entries A key parameter is the cutoff in days, basically how many days worth of audits to keep.
args: -queueperson, -custom_predicate r_gen_source=1, -window_interval 1440,
-cutoff_days 1
Note the "cutoff_days" parameter should be set to a reasonable number of days, balancing compliance and trouble shooting issues.

dm_QueueMgt

Purpose: Deletes dequeued items from dm_queue
args -queueperson, -cutoff_days 90, -custom_predicate, -window_interval 1440

dm_UpdateStats

Purpose: Updates RDBMS statistics and reorgs tables (if RDBMS supports)
args: -window_interval 120, -queueperson, -dbreindex READ, -server_name SQL2\SQL2005

dm_ConsistencyChecker

Purpose: Checks the consistency and integrity of objects in the docbase

dm_DataDictionaryPublisher

Purpose: Publishes data dictionary information

dm_LDAPSynchronization

Purpose: One-way synchronization of LDAP users and groups to Docbase Method
args -window_interval 1440, -queueperson, -create_default_cabinet true, -full_sync
false

dm_FTStateOfIndex

Purpose: State of Index dm_FTIndexAgentBoot Boot Index Agents Method
args -window_interval 12000, -queueperson dmadmin, -batchsize 1000,
-writetodb_threshold 1000000, -serverbase F, -usefilter F, -dumpfailedid F,
-matchsysobjversion F, -matchallversion F

dm_GwmTask_Alert

Purpose: Sends email alert if task duration is exceeded

dm_GwmClean

Purpose: Cleans all the orphan decision objects

DQLs to run to check on audit trails and dmi_queue_items

The following statements are some of the DQLs that EMC support had us run to determine the
number of audit trails and queue items that were in the repository:

Select count(*) from dmi_queue_item

Select count(*) from dm_audittrail

Backup Procedures

Ideally, the Content Server should be shutdown prior to running the back up of the SQL Server database and started back up afterward. This will reduce any likelihood of the repository becoming out of synch with the database and the content files.

OS and Software Upgrades/Patches

Before applying any patches or upgrades to any of the Documentum suite and supporting applications, be sure to check for compatibility. Apply any patches or upgrades to the dev and QA environments and test them first.

Network Connectivity Interruption

If any network interruption occurs, then service logs should be checked for compromised activity. The Content Server and Tomcat server may need to be restarted. The logs of the application and content servers should be periodically monitored for errors and warnings.

Performance

RAM and CPU Utilization Maxed Out

If RAM is filled or CPU utilization is maxed out then the service responsible should be checked. If the service is a Documentum service, it should be restarted and root cause should be determined. Utilization should be monitored and any anticipated spikes in use or
additional services need to be load tested and analyzed. What should you do if Tomcat performance slows? If the concurrent users reach EMC’s limit of 20, EMC will recommends adding a second Tomcat server.

Further Java Memory Allocation settings to consider.

EMC Support gives the basic JVM settings to cover for common exceptions and crashes. There
are a number of other settings to add as more traffic occurs on the Tomcat server. From the
DCM Installation Guide:“To achieve better performance, add these parameters to the application server startup command line:

-server-XX:+UseParallel01dGC

Document caching can consume at least 80MB of memory. User session caching can consume approximately 2.5 MB to 3 MB per user. Fifty connected users can consume over 200 MB of VM memory on the application server. Increase the values to meet the demands of the expected user load.”

Monitor Sessions

Location: Under Administration > User Management > SessionOrDQL: execute show_sessions (to show all active and inactive sessions)

DQL

execute list_sessions(to show active sessions)

Via docbasic ebs script

Purpose: set this script at a command line prompt to output how many active and inactive sessions are current on the content server. Set the interval between output and how many loops to run.

Troubleshooting Max Sessions error

Before restarting Tomcat:
Try logging into the content server from docweb using the Doc App Builder
application. If you can, then this isolates the max session error to the Tomcat/Webtop
server.

Using DA, look at how many “active” users sessions are currently in the repository.
How many “inactive” sessions.
Try reducing the session timeout value in the web.xml on the Tomcat server to see if
the inactive sessions get cleared out faster.

Security and Server Access Maintenance

Test users and test content should be deleted out of Production
The database schema owner account should be locked down
The Documentum install owner, “dmadmin” should be locked down
Only scheduled, authorized access to the Production should be allowed for all
servers of the system.
Repository audit trails should be configured for certain events, such as deleting of
content.

Long Term High Availability and Scaling Recommendations

As more users access the system, it may become necessary to create a second Tomcat
(clustered) instance to ease the load on just one application server.
As more content get added to the system, more disk space will need to be added to
the filestore drive.
Set up failover services for all key components
Add more Java Method Servers if lifecycle processing overwhelms the existing one.
A comprehensive content archiving plan will need to be designed and implemented.
Setup a disaster recovery site if the system’s service level agreement (SLA) is
sooner than a new system could be built with backups.

Applying Malcolm Gladwell to a Documentum Project

2009-02-22T17:44:00.002-05:00

In Outliers, Malcolm Gladwell’s stories of successful people show a few concepts that can be applied to Documentum projects. These concepts explain how some individuals succeed and some fail. A successful project can point to these concepts as part of a successful pattern.

“Concerted cultivation”
Taking an honest interest in what the business customer really needs out of a Documentum content management system (beyond your personal business/monetary objectives) is what this is about. Fostering the open disclosure of issues the business would like to solve in an honest and transparent manner will go miles to making the project successful. If the issue is lack of accountability in tracking their processes, then they need workflow, not just Webtop and we'll see in the future if they need workflow. If the issue is that they want to convert from paper and be more productive, then they need scanning and basic classification, not just Author Integration Services.

Cultivate what the client will need to learn in order to appreciate the solution, do not hold back information in hopes establishing an on going support contract. If you don’t give enough knowledge transfer, the client will resent the issues it encounters and might blame you for not training them enough. If you don't document all of the potential issues that might come up, then the customer could point their finger at you and say "fix this for free".

Being the right person present at the right time
Chances are it wasn’t that you or your company won a Documentum project because of your raw talent, it was more likely that you had the right architect give the pre-sales demo or a contact at the company who you knew from a previous engagement, or some other connection to the company that won the work. Chances are even better that you happened to have previous experience that fitted well with what the company was looking for.

This brings up one of my issues with pre-sales. Most of us get paid when we’re working on a project, if we’re not we’re on the bench slaving over a state of work or a response to an RFP. Why don’t consulting companies realize that they need a research and development group just as much as a software company? It doesn’t have to be for product development, but solution development. Researching issues and developing solutions is what we do, but most of us solve issues on the project, write documentation and then we’re on to the next project. We don’t have what we need which is an occasional sabbatical to put the solution together and be able to truly work through the requirements, content, functionality, testing implementation, architecture, and most importantly the lessons learned. Once I was asked what types of content was part of a solution that I had only helped technically with, I couldn't answer for sure what types of content there were. This was because my time on the project was specifically allocated to deploy it to production, not to fully understand what the content was. This was sobering. I'm in this business to help solve content management issues and I was so caught up in the technology that I didn't even know what content was being managed...

“Mitigated speech”
It’s no secret that developers and architects from India are well trained and very talented in the computer science fields. What has taken me a while to understand is how a developer will communicate to an architect or project manager. He might ask a question like, “do you think the system is a little slow?” You might respond, “It’s not bad considering that it’s a QA environment. The developer might be downplaying what is being said, that is, he might really be saying “do some load testing you idiot or this solution might crash production when it’s released”. This is how “mitigated speech” works and it needs to be dealt with during the beginning of the project. Make sure you’ve created a few avenues for communication that wring out concerns clearly and effectively.

“It is those who are successful…who are most likely to be given the kinds of special opportunities that lead to further success.”
Obviously, this applies to all project outcomes, however, you may think you’ve succeeded when in fact the business feels coerced into paying you the final amount due. During the final stage of a project, which includes stabilization and knowledge transfer, there may be times when giving a few extras days for free gives a nice feeling to the customer. If the stabilization of the project is bumpy, like most are, beware of saying “this is not in the budget, we won’t do it without getting paid.” This kind of approach during a time when the customer is stressed out and any problems are easily blamed on the solution provider will not result in the kind of return business that would be achieved if you went the extra mile for free. Letting the customer push you around a little at the end may help extend the support contract until a further project comes along. These days a customer needs to fell like they’re getting a deal on your services. Make them feel that way…

The success of a project is not because of the one person or the architect or the design, it is because of the overall efforts of everyone involved, the timing, the company culture, the governance, mutual trust, and confidence in each other. One project’s success could catapult your whole company into phenomenal success, but it was a combination of experience, who you know, how much you’ve practiced, and whether you were in the right place at the right time. And luck.

Users, Groups, and Roles with ACLs and Presets

2009-01-18T16:11:00.003-05:00

At some point in the evolution of any repository’s design and implementation, the issues of managing users, groups and roles, and of reducing object type clutter rise to the top of the priority list. Also, finding a common way in Webtop of displaying attributes for documents listed in folders and search results becomes vital to maintaining a consistent user experience and getting a grip on endless customization costs.

You can talk to the business about users, groups, and roles, but the ramifications of your design will not click with them until they see Webtop in action and what presets and ACLs are actually doing to their user experience. For example, a user has the ability to import content and as the owner of the content they can promote it even if they have Read access only: this is hard to understand unless it was shown during a demonstration.

Here are some tips on how to design and implement users, groups, and roles with ACLs and presets:

Manage User Group Membership from LDAP or Active Directory Groups
I strongly recommend managing group membership using LDAP integration, otherwise you’ll have to run a script to add members to groups the first time and then manage group membership one by one with Documentum’s tool instead of a more robust user directory tool management tool.

Either use or exclude the dm_world default group from accessing your content
For tighter control of security, create a custom group that is your company’s base group from which to build. This base group should have a consistent level of minimal access to all content in the repository, such as Read access.

Using Presets to reduce Action in Webtop
Now that you have a base custom group, you can exclude from that group any actions, such as “Create New Documents” or “Import” from Webtop. Review the manual on what the basic client capabilities of Consumer, Contributor and Coordinator are before trying to add these to Presets.

Create Test Users that have Inline Passwords, as well as Users from LDAP and Local Domain Users
Sometimes LDAP will be “slow” or “failover” (I know User Directories never fail, but they do and you need to be able to access the Repository when and if it ever does. Chances are that your install owner account (“dmadmin”) will be a local account (I hope). So if domains or LDAP or the network is down at least you’ll be able to access the repository with inline or local accounts.

dmadmin: superuser best practice
- Add docu group as delete, full privilege to all ACLs
- Don’t include docu with the custom world group
- Don’t use dmadmin to test
- Don’t have dmadmin in any other groups, especially ones controlled by presets because the link to Webtop administration of presets may vanish and you’ll have to create or move a new user into the preset group.

Create contributor Roles that are responsible for importing and creating new content for each group of content types
For a financial company, this means create a group for Tax, Treasury, Marketing, Operations, Accounting, and so on. Each group is responsible to for a certain group of content, for example, Tax imports tax returns, withholding docs, etc and Treasury has capital calls and distributions.

Create Presets for each Role that include the content types they import only
This controls the object type dropdowns when importing, checking in, and creating new content from templates. Be sure to include a parent type that will not be used, but will be the first in the dropdown, so that custom attributes will be refreshed when the custom type is selected from the dropdown.

Scope Search Object Type Dropdown by Object Type Parent
Users of Webtop usually need to limit the amount of object types to search for in advanced search. There’re only so many types that users need to look at for searching. Less is more in the case of the advanced search object types.

Scope Object and Doc Lists by Role
The display of custom attributes in the columns of search results and when browsing folders should be customized by Role to establish consistency and attribute value listing expectations. Custom attributes will be the important ones to show and sort in these circumstances.

For searching across all object types think about replication
First, you’ll need to figure out the common attributes across all of the object types. Second, although you might be able to roll up a few of the custom attributes into the parent type, different user interface requirements will probably force the design to repeat certain attributes in every child object type. One solution is to develop a TBO that replicates some of these common child attribute values to the corresponding parent attribute in order to search for them from the one parent object type in advanced search.

Composer the Poser

2008-12-15T19:57:00.002-05:00

When a teenager wears skateboarding shoes and a brand named hoody but doesn’t skateboard, he is referred to by real boarders as a “poser”. Unfortunately, the same can be true for EMC’s Composer. It wears all of the key features of the doc app builder, but it ain’t no DAB.

Don’t get me wrong, like a good EMC citizen, using a DAB built system, I created Composer projects from Production to QA and Dev. I worked through some initial bugs in the Composer build and was determined to deploy the next phase of the project using D6 sp1 Composer. I had heard that D6.5 was not going to use DAB, so I thought I’d better get on the bandwagon and get used to Composer.

I read through Jason Duke’s article on the Blue Fish site and was assured that I’d get through the process with a few issues, but no show stoppers. I figured out that all of the directory structures needed to be consistent in each environment. I also worked through a deployment plan for multiple, decoupled dar files for easier development coordination and implementation. I rewrote my design document to follow the new and “improved” configuration UI of Composer.

The first road block was value assistance for a query. The xml file created by Composer was missing the “complete_list = true” key/value pair for the query element. This meant that the dropdown was fixed at 20 pixels. EMC support sent a fix for webtop (not Composer) which fixed the width of the dropdown, but the dropdown graphic was the new look and feel graphic. This was glaringly bad for users to see. What were they thinking?

The second road block was that if you include groups and roles in the Composer project during deployment they will overwrite the existing groups and members if you don’t remember to change the install instructions each time. There is no global way to tell Composer to ignore existing artifacts, thus each time you deploy, you have to set the install option again for each artifact. When the project install fails which it will over and over again, you’ll be clicking and clicking. Bottom line is that this needs a lot of work for it to be production ready.

The third road block was that value assistance for query based dropdowns changes the rendering of the select box as compared to the fixed value dropdown. Hello, a customer can tell the difference between the two and won't like it. The fix is the go into DAB and save it again, but this is no excuse. Gotta wonder what level of regression testing was done...

After chasing a few more bugs with Composer D6 sp1, the patch came out, but that didn’t help much. Common sense won out and we’ve gone back to using DAB for deployment. As clunky as it is, at least it isn’t a poser. By the way, DAB was released for D6.5. Gee I wonder why?

LDAP Unwrapped

2008-11-09T22:01:00.001-05:00

Every time I configure a Documentum repository for LDAP Synchronization I start out impressed with the new features that have been added to the latest version, but end up frustrated with the myriad of problems to solve to get the synch to actually work.

The first thing you have to do is download the latest bug fixes, to avoid what always happens to me: shagging after errors on EMC Developer Network and searching and searching and finally finding some reference to bugs and fixes. The ftp address for D6 sp1 fixes is ftp://dev_pre:qa5.grN6@ftp2.lss.emc.com/sustaining/Content_Server/6.0_SP/LDAP .

The appendix below lists the bugs that are currently fixed. These fixes will make the synchronization actual work if you have more than one configuration set up to run in succession.

Start with the User Directory
You need to engage the LDAP administrator in order to determine whether the group membership structure of LDAP and if it makes sense to use it, that is, are the groups setup with content management in mind or are they a mixture of file system security and ad hoc assignments.

Also, have the user accounts been setup consistently? For example, Active Directory works fine with accounts set up like “smith,john” and “smith.joe”, however Documentum throws an error while processing the “smith.joe” because of the period.

You’d think managing group membership from one source application is easier than from multiple applications, however decoupling LDAP groups with Documentum groups may be more practical given the differing purposes and rules within each application. The classification rules (driven in part by group/role definitions) for a content management system are usually different and definitely more robust than those of a file system.

Automatic creation of user’s cabinets from LDAP
It’s not obvious how to do this from the LDAP configuration form. From the LDAP Server Configuration Properties’s mapping tab you have to add the following:
- Property: default_folder
- Type: dm_user
- Map To: /${cn}
- Map Type: Expression

This mapping will create a cabinet for each LDAP user as they are added to the repository. The cabinet will be owned by the user and will be private to that user. This cabinet will be the user’s home cabinet.

Appendix: D6 SP1 LDAP Bug Fixes

Bug fixes in LDAP_DOCAPP_HOTFIX
153994 - Unable to create LDAP config object on AIX/BEA as it throws java.lang.NoClassDefFoundError (It applies to all OS/Appserver combinations)
153355 - The Diretory type property is nto properly set while creation of LDAP config object
153322 - DA hangs on the Mapping tab while creation of LDAP config object
151022 - Cannot create LDAP config object with out setting proper OU info
150397 - LDAP config object cannot pull the SamAccountName attribute in AD2000 and AD mixed mode
151570 - Cannot create proper LDAP config object

Bug fixes in D6SP1_dmldap_hotfix
145896 - When using "\" in the CN values for LDAP, the LDAP synchronization propagates the users to Documentum properly but group membership synchronization fails because of this special character.
149446 - LDAP Synch fails when using subtype of dm_user
151224,154269 - LDAP Users with apostrophes in their names (i.e Robert O’Leary) have been successfully imported to Documentum. However, due to the apostrophe in the user name a DQL query is failing.
149443 - LDAP Synch fails when mapped attribute is null on Directory server
154399 - LDAP Sync jobs tries to deactivate the user even though the user is not present in the docbase.
154511 - If dm_ldap_config is to configure to map LDAP attributes to DocBase attributes of a subtype of dm_user, running LDAP sync failes to populate the additional attributes introduced in my subtype with LDAP data.
154704 - LDAP Sync throws DmLdapException:: THREAD: [ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'; MSG: home_docbase - Property not found; ERRORCODE: ff; NEXT: null
155432 - The LDAPSync job appears to be querying older versions of the serverconfig object to get the ldap_config_id. If the older versions do not have a value set, the job will fail.
156347 - LDAP Sync job fail to remove a sub group when a sub group is deleted from AD.
157177 - LDAP Sync throws NullPointerException
157650 - com.documentum.fc.common.DfException constructor throws Exception:
java.lang.StringIndexOutOfBoundsException: String index out of range: 90
157759 - LDAP Sync throws Exception [DM_GROUP_E_UNABLE_TO_SAVE_EXISTING]error: "Cannot save group geoda-grp because a user already exists with the same name"; ERRORCODE: 100; NEXT: null
158197 - LDAP job fail to create the users cabinet though the argument create_default_cabinet set to true.
158766 - Running LDAP Sync throws the following LDAP NamingException:
javax.naming.NamingException: [LDAP: error code 1 - 000020EF: SvcErr:DSID-020513B8, problem 5012 (DIR_ERROR), data 8333]; remaining name 'CN=GAU2876SPE,OU=Security Groups,OU=Groups,OU=USPTO,DC=prepro,DC=local'"; ERRORCODE: 100; NEXT: null
153709 - Request for LDAPSynch job to have a switch to ignore case when comparing usernames
162830 - Ldap Sync job removes users default_group attribute
162684 - LDAPSynch leaves trailing spaces when setting user attributes with mapped custom values
164164 - LDAPSynch fails with DM_LDAP_SYNC_E_EXCEPTION_ERROR "String index out of range: 32" on getTruncatedString adding a member to a group if the username has trailing blank spaces that exceed the 32-character length limit.

Blame Everything on Testing

2008-10-21T13:02:00.001-04:00

Your project is executing according to the schedule; the code development is done and unit tested, the Documentum applications have been installed and smoke tested; and UAT is about to begin. The designated tester on the team has been testing and writing scripts on the Test system. There’s a week allocated for testing the system in test. Oh, by the way, there’s a migration of legacy content being tested as well.

Yessss, I’ll be able to take my planned vacation
The testing is going excellent in Test. What do you think could go wrong? Well, first there are LDAP users in Prod and not in Test. Second, there are slight undocumented differences in the ACL’s between the two environments. Third, the migration will not be fully tested until it is run in the Prod environment. Fourth, UAT Users were not fully engaged during the testing and script writing phase of the project. Fifth, basic performance tests were run on Test, but real use tests to soak the system have not been executed. Sixth, a key developer is rolled off the project to start a new project at another company. Finally, seventh the sponsor of the project left the company and a new more technical sponsor is critiquing the application.

Can I get a refund for my vacation?
LDAP Users are in Production: The user that are imported will need to be added to groups if they aren’t from LDAP. The user logins will need to be tested. Renaming users will need to be tested.

ACLs: It is extremely tempting to make changes in Dev, QA and then Prod for ACLs manually without deploying them through a docapp or scripts each time. This inevitably causes slight differences between the environments.

Migration: When the migration is finally run in prod more focus will be on the results. The real users will be searching and finding bugs. Rerunning the migration a few times to prod should be built into the schedule.

UAT: No matter how much you tell the UAT testers that they need to be engaged and thinking about testing while reviewing the test scripts, they’ll find a ton of issues with the application during UAT. Showing them demos and prototypes during the development of the application helps, but they still won’t be able to really test until they sit down and seriously try to figure things out.

Performance: The Test environment was not tested for performance and there’s not enough money in the budget for a separate performance env. You’re left with trial by fire and it’ll be challenging.

Key Developer Rolls Off: This happens a lot. The Project Manager schedules a developer to be available for a few days after Unit testing. The best documentation of code and customizations could exist, but if the developer is no longer on the project, the learning is huge for someone else to fix bugs quickly during UAT.

New Sponsor: In this time of layoffs, you’ll get a number of sponsors who are reassigned of layed off. This means that the new person will come in with preconceived notions as to what needs to be done. This new person will want to make difference and a quick win is to delay the project while reviewing what it is suppose to do. What business problems are being solved, let’s look at the bigger picture here, why is this over budget, can we consolidate more, etc?

Stacation
The testing phase of projects will give you a few gray hairs. There are so many potential pot holes. How ever many days are scheduled for UAT, double it no matter what. If you want to go on vacation after, schedule it for next year.

Docapp Distractions

2008-09-30T16:37:00.003-04:00

I know Composer is out there and useful in many installations, however it’s tempting to still use the Docapp Builder to build real time object models and once you start you kind of have to keep going. Also, there are still a ton of 5x installations out there that need to be maintained and eventually upgraded.

Different versions of Docapp Builder and Docapp Installer
I was recently in a situation where one developer was developing on D6 sp1 with a Docapp Installer version 6 and another was developing using 5.3 sp5 update1. The D6 developer was creating docapp archives for Forms and Process Builder which when installed using the 5.3 sp5 update1 installer left out certain form properties and functionality. It was a real waste of time trouble shooting these issues.

Creating a workflow docapp with auto activities for lifecycle promotion
If you create a workflow and have an auto activity for promoting a lifecycle state regardless of whether the lifecycle is added to the Docapp, the lifecycle will get added to the Docapp archive and installed on the target docbase.

ACLs with non dm_dbo domains
If you have set up ACLs with an owner that is not the dm_dbo (docbase user account) then these permission sets will not show up in the Docapp’s Lifecycle, Action, Add permission set drop down list.

Versions
If you are splitting the docapp work between a few developers, get ready to deal with a lot of versions of the docapp and all of the workflows, lifecycles, activities, etc.

ACL inheritance from Folder
If your docbase is set up with ACL inheritance from folders be sure to set the “System/Applications” folder ACLs. If you don’t do this, everytime you deploy a docapp its components, such as a lifecycle will inherit the ACLs of the docapp folder and may or may not work in the application.

Scoping of Display configurations
This is very strange when it happens. Let’s say you start out with a small object model of two levels and configure the display. Then you need to add a new parent level and redo the display. Sometimes the scope remains with the original model and object type. The only way I found to fix this was to dissect the database scope config and cut out the disease.

Mandatory attribute changes with old references
If you ever decide to change an object type’s attribute from required to not required, sometimes this change is held in a reference or cache somewhere. The only way I’ve been able to clear it is to delete the attribute using DAB, closing DAB, clearing the cache and inserting the attribute again.

Importance of having one source for docapp creation
I can’t stress it enough that Docapps should be created and control from one source, with the same versions of the Docapp Builder and Installer on the source and target machines. Configure the object type’s display only when the attributes and object model are final. Add workflows in with lifecycles. Try not to version docapps or its objects unless absolutely necessary. Finally, beware of the permissions that are inherited by the docapp installation.

Solution Demos: Catch 22

2008-09-19T15:51:00.002-04:00

Damned if you demo, damned if you don’t, especially if your client is new to Documentum. You can talk about functionality, show the vanilla installation of Webtop, and still get surprised looks when you show the final solution. On the flip side, you can demo the solution when it’s not fully tested and have to dance around errors and incomplete development.

Demos of a client’s project are essential if the client reads the Functional Requirements Spec and the Tech Design without too many comments. This means they don’t understand and/or don’t have enough time to review it. This can be frustrating, but if you demo the solution in front of those same future users, you’ll get invaluable feedback and will eventually have a more successful launch of the solution.

Steps to successfully demonstrate your solution:

Test the part of the solution that will be demo’d
Plan talking around any part of the solution that is not completed
Have an agenda of what will be shown during the demo
Repeat to the clients/users that this is not a finished solution, that there will be bugs
Try to have fun with your audience, this should not be too serious
Have other technical resources there to take notes and comment on questions that pertain to their customizations
Listen you your client’s comments and take their feedback seriously
Make notes of comments that could form future opportunities, like “does this mean we have to sign on to another application each time we want to access it?”. This could mean that single sign on is in the future. Or, “I already fill out this information in the database, why do I need to do this twice?” This could mean an integration possibility to ease the transition to using Documentum, not to mention make finding content easier with more integrated attributes.
Schedule a few demos to reduce the risk of disillusionment after an error causes one demo to fall short of expectations.
Try to impress the client each time by showing something extra in the product that they might not have seen.
Whatever you do, do not take screenshots and dummy up the solution. This will lead to more questions and anxiety over the progress of the overall solution.

From Functional Requirements to Tech Design

2008-08-22T17:43:00.002-04:00

The most important part of designing a Documentum solution is to understand and translate the business and functional requirements into Documentum configurations and customizations. You have to create maps of all of the major components of requirements, matching each requirement to solution. You might use a traceability matrix or you might organize your specification similar to the functional requirements. It depends on the project scope and business expectations. My opinion is that if you need a traceability matrix the solution is too wide in scope or there are some trust issues with the business.

Document Types to Object Types

Scope of project phase
When reviewing the list of doc types in the functional specification do not take for granted that these are the final types of documents. A business analyst cares about capturing all of the possible documents, not necessary thinking about the scope of the project when he does so. Be sure to explicitly describe the scope of the project in your design. For example, the doc types may cover all content in the enterprise in order to create a comprehensive object model, however the scope of the current phase of the project may be a subset of those types. Do not over commit to the business what content will be covered in the first phase. When you demo the functionality, only show that phase’s content being imported and published. Try to reduce scope creep by repeating the project’s phase expectations and assuring that all content will eventually be brought into the system. This is a good opportunity to talk about a roadmap of the phases required to fully actualize the business’s total content management with Documentum.

Model review and rework
Roll up attributes as you work through each object type, finding commonalities across object types. Think about all the UI, TBO, ACL ramifications of your design. If the object model is huge, try to consolidate, look into using aspects as an alternative to many object types or many attributes.

Implementation design
Depending on the scope, you may be releasing the object types in phases. If this is the case, make sure the phase’s object types cover all of the document types, business processes and security requirements. You may want to prototype some of the object types and their relationships to importing and folder linking. For one-offs in a dev environment, Documentum Application Builder is the fast way to do this. For more methodical approaches, Composer is more portable and best practice going forward.

Metadata to Attributes

Match with OOTB attributes first
Don’t reinvent the wheel, search for a suitable existing attribute first then create a custom one. If there’s a requirement for a comments field, use the log_entry attribute. If there’s a status field, use the r_version_label.

Source of values

Value assistance could be used for the easily maintained attribute values, which would be maintained using Doc App Builder or Composer. This uses docbasic.
You could get a custom object to query for attribute values, which would be maintained by the business via the UI.
You could query registered tables or views which could integrate with linked database tables
You could read and parse a properties file maintained on the server.

UI constraints: repeating attribute, large text boxes
Think about how the users will react to the WDK repeating attribute dialog box. For large strings, you’ll have to put a text area in the forms.
Taxonomy to Folder Structure

Search ramifications
Giving context to search results is an important way to transfer knowledge. Repeating key attribute/value pairs (for searching) with folder hierarchy labels (for browsing) usually makes sense. If it doesn’t, maybe you are trying to put too much metadata on the object.

Simplify where ever possible
If you are going deeper than four levels, you better have a good reason. At a certain level there’s a “project” or “deal” or “product” level which repeats over time. This level should be closer to the top level, have some automated way to build new folders as new “projects” are added, and have an automated or easy way to archive, thus reducing the clutter of old content over time.

Business Process vs. share drive organization
Folder structure may be organized in many ways. A business process oriented structure may make more sense than strictly following the share drive approach of silos of data structured by business group. Avoid organizing content by date only. If there’s an end of quarter date use it, but don’t rely on it solely to find content.

Mix of common, evolved classification with more structured rules
Chances are good that you are moving from one established way of structuring content to another. Doing this right so the users know what they are doing in the new system without hours of training can be difficult, especially because the old way is usually not the best way. If the company is large, your headache is merging structures, if the company is small your headache is reorganizing silos of information. Sometimes both.

Modify according to folder/object type map
As the reality hits the road and you’re building folders and automating the linking of content to them, expect changes to the design, additions to the attributes needed, and delays to build and customize the right way.

Folder Structure mapping to Object Type plus Attributes

Match Object Type and Attribute Key/Value pair Combinations to Folder paths
Prototype the search and browse functionality
Do subfolders follow the document templates?

Business Process to Workflow
This is where you’ll read the functional requirements of a process of getting content approved and have to figure out the specific activities involved for a workflow. Sometimes, Common business processes may not translate into a formal workflow. In this case, describe the steps of using the application and how the work can get done without a formal workflow. You’ll have to determine how to create decision point that splits the workflow if there’s a yes/no decision to be made. A common example of an auto-activity is the requirement to send a notification email with a link to the workflow’s package

Importing content
Check all assumptions involved with importing content into the system. It might be assumed that all content related to the first phase should somehow get into the system. The problem with this is that the contributors of the related content may not be ready to participate in the Documentum system. They may not be in the first phase.

The Art of the Custom Documentum Object Model

2008-05-18T20:56:00.004-04:00

Before designing a Documentum Object Model, you’ll need to take a litmus test of the culture of trust between the sponsor of the project and the IT organization. If the company is large there will be multiple levels of politics. You’ll need to judge from how the requirements gathering sessions went to figure out your approach to the object model design. Part of gathering requirements is educating your client on the types of objects that make up the content management system without getting too technical and wrapped up in explanations that are too long winded and lost on the client. During this education, ask questions like:

How do the different business units communicate with each other?
Do they share information, is there emphasis on security?
Are there databases that they use to look up information?
How effective are Marketing and Sales at driving the accumulation of knowledge into content published to consumers?

The answers to these types of questions help determine the meaning of the object model’s hierarchy levels. As an architect of the content management system, you are the only one who is qualified to make non-biased design decisions and hopefully would not have an agenda in your design. The most common object model hierarchy has an enterprise object as a child of dm_document, dm_folder, etc. and then children objects underneath it. The ramifications of your design magnify at the second level of the hierarchy. Here are some scenarios of what happens when the architect gets influenced in the wrong ways:

Forced to design without all the requirements
Have you been given enough time to really design a model that reflects the whole organization? If the IT Manager on the project says, “Don’t worry about the whole organization, we have three business units in front of us now, this is all we need to worry about for now,” you know there will be issues with the design if you create a model without knowing the bigger picture. Scalability, performance and reporting all suffer when a design is not drawn from the full foundational background.

Influenced by the wrong folks in IT
Most notorious for screwing up an object model would be the turf war database architects who don’t understand object-oriented design. They demand to know what the relationships are between all of these tables, where’s the schema, etc. They will flip out over table unions and joins if you tell them too much. They could care less how much the object model is integrated with the UIs and security. So when you say that the enterprise level attributes are reserved for only the most far reaching attributes across the whole company like retention period, stick to your guns when they push back and shake their heads. If you need more than 3 levels of custom object levels, make your case is as sound and simple as possible, try to include monetary impacts on potential customizations if it isn’t followed.

Confusing Department Security with Content Functionality
Most companies are set up by departments. Each department shares some information and restricts access to the rest. This doesn’t mean that the object model has to follow the org chart of the company. It may make more sense in the long run to figure out the function of each content type in the enterprise and really study the use cases of the content that is most critical to the company’s success. For example, for a government organization which has vital records (like birth certificates) to scan, index, and store, it makes sense to design an object model around function, in this case vital records, instead of the name of the agency that keeps track of the vital records. Agencies and departments will reorganize over time, functions such as birth records will not.

One repository vs. many
In many cases multiple repositories designed around one global registry makes sense. There’s more flexibility built into this design through out the technology stack, as well as with the changes in the business units over time. This however does not mean that each repository should have autonomy in its object model design. In fact, there should still be an enterprise level custom object for each object type being customized. The object model should be the same in each repository. You’ll have to be more diligent with migrating docapp archives between repositories, especially with the install options.

Decoupling Internal Business Process and External Publishing
If the end goal of the content is to publish it to a portal, there will be conflicts between the internal structure of the content (how the business works with each other) and external structure (how consumers view and search the content). Do not underestimate how long it will take to weed out the navigational systems for each side, the security and identity management, the functional driving attributes, etc. In the best case there will be enough decoupling of the objects and their attributes that the design can provide decoupled and scalable solutions to the conflicts between content management and content publishing.

Some General Rules of Object Model Design

Determine if content types are functional or departmental in nature
The security model of the repository whether it’s user, object, or folder based may have an overriding influence
Build in flexibility to enable the object model to expand in all directions
Move attributes that span all of object types up one level if possible

Polluting the ECM Ecosphere

2008-02-09T14:36:00.001-05:00

You know those email spams that fill up your inbox? Well what about the trail of junk that Content Management Systems leave behind as they forge ahead solving complex business problems?

As I’ve work on large, small, and medium sized CMS’s, I’m always amazed at how polluted they are with logs, audit files, orphaned work items, queue items, reports, ACLs, versions, etc. The out-of-box clean up jobs focus on getting rid of unwanted versions, orphaned content, logs, and queue items, to an extent.

The problem is, with some business reporting requirements, they are too good at deleting files that may be of use for historical analysis. Business users get nervous when you say “we have to clean up things up to maintain performance”. They say, “Can we wait for a while until we really need to do this? What are the risks? What if you delete something that we need at the end of the quarter, or year, or in ten years?”

M. Scott Roth’s “Seven Jobs Every Documentum Developer Should Know and Use” article details the use of seven Documentum jobs: DMClean, DMFilescan, LogPurge, ConsistencyChecker, UpdateStats, QueueMgt, and StateOfDocbase. There’s a job to trim versions, but for whatever reason Scott didn’t include it in his job list. These jobs are all essential to keeping your repository clean and performing the way you expect, but what do you do about ACLs, workflow history, and versions if the deletion is not specific enough?

Content pollution is rampant in all industries and is a direct result of rushed design and over ambitious technical solutions to relatively simple business problems. Take a regulated content management system for example. This system most likely creates new versions of content for every change to its file or its metadata. There also could an audit trail which records every version’s change, a backup of the file system and the database for nightly and weekly data security, disaster recovery with off-site replication, multiple renditions, and multiple language versions.

The upshot is that the proliferation of versions and logs, and backups is great for storage “archive” companies, but can lead to confusion and a false sense of security. Who’s making the design decisions? Most likely it’s a business user who doesn’t want change, thus forcing an over worked IT Manager and ECM Architect to work out the solution which puts garbage control on the back burner. “We’ll deal with logs and versions later, right now we have to roll out the project on time and within budget.”

So how do we design with conservation in mind? For one, we think a year or two in the future and try to extrapolate the effects of thousands or millions of scraps of content floating around in the CMS, slowing done queries and filling up the more expensive hard disk space. Here are some more ideas:

Design to recycle:
For each log and object type ask how will this be created, versioned, and disposed of. What is the purpose of this content? How long will it be useful? Efficiency Think of conservative approaches to logging events, to versioning content (regardless of OOTB functionality). Upgrades, New Development, and Performance Testing Logs, database temp space, temporary migrated content files can pile up everywhere during special testing and migrations. These files are often “hidden” and sometimes move along to production systems only to clog things up later.

Site Cache temp files and orphaned site files:
Site Cache Services is notorious for leaving stray temp files, logs, and orphan folders all over the place especially during fail publishing attempts.

Docapp messes:
Docapps, when not performed carefully, can leave references to old lifecycles, workflows, object types, and attributes. These orphaned objects can not only clog the system, they can corrupt production environments with hardcoded references to filestores and none existent owner names.

Repository and LDAP synch logs:
Every time an LDAP synch job runs logs get stored in the repository and on the Content Server file system. Every time a repository starts up a new log starts for it. These logs fill up the server file system which is usually not a large disk.

DFC traces:
During development and testing, trace logs are essentially to tracking down bugs and slow performance. These files are usually forgotten and build into huge space choking surprises when you least expect it.

Environments such as Sandbox, Dev, Test, Performance, Staging, Prod, DR, Off shore, Business Continuance:
All these environments double, triple, xduple the amount of disk space needed for solutions. Think about ways to migrate subsets of needed content without versions perhaps. Reduce logging in environments not used very often or that are dormant for a period of time.

Integrations with other applications:
Many integrations of systems require multiple renditions of content for presentation. For example, email messages from Outlook get saved as .msg files in Documentum. Even when EMC’s email Xtender is installed, an integration with Outlook requires copies of the original email to be imported into Documentum’s repository.

Queue Item Maintenance

2008-01-04T15:17:00.000-05:00

Issue:
The recording of task events are tracked in tables, namely the dmi_queue_item table. These build up proportionally to the number of tasks executed. As time goes on and performance potentially slows down, these queues will need to be deleted.

Requirements:
Periodically delete old records from the dmi_queue_item table, but keep track of a document’s workflow history for a certain amount of time beyond the dmi_queue_item cleaning.

Solution:
The first thing to determine is whether the out-of-box dm_QueueMgt job can do what’s required. This job runs a method that can delete queue items older than a cutoff date. This is useful, however, we want to keep the workflow history of a document. This is also useful because this table keeps a record of many other events in the repository which need to be deleted on a scheduled basis. The solution was to create a custom table which holds the requirement queue items to maintain the workflow history of documents, and to create a new job and method to populate it before as part of queue management.

Solution Details:
First, create a custom table using DQL (Note: this table has the same columns as the dmi_queue_item table):

UNREGISTER TABLE wf_history_s
EXECUTE exec_sql WITH query='DROP TABLE wf_history_s'
EXECUTE exec_sql WITH query='CREATE TABLE wf_history_s
(
r_object_id VARCHAR2(32),
object_type VARCHAR2(32),
id_1 VARCHAR2(32),
string_5 VARCHAR2(200),
string_4 VARCHAR2(200),
string_3 VARCHAR2(200),
string_2 VARCHAR2(200),
string_1 VARCHAR2(200),
workflow_id VARCHAR2(32),
policy_id VARCHAR2(32),
registry_id VARCHAR2(32),
audit_signature VARCHAR2(255),
audited_obj_vstamp INTEGER,
user_name VARCHAR2(32),
time_stamp_utc DATE,
audit_version INTEGER,
chronicle_id VARCHAR2(32),
controlling_app VARCHAR2(32),
object_name VARCHAR2(255),
audited_obj_id VARCHAR2(32),
version_label VARCHAR2(32),
acl_domain VARCHAR2(32),
attribute_list_id VARCHAR2(32),
host_name VARCHAR2(128),
user_id VARCHAR2(32),
i_audited_obj_class INTEGER,
event_source VARCHAR2(64),
event_name VARCHAR2(64),
r_gen_source INTEGER,
owner_name VARCHAR2(32),
time_stamp DATE,
event_description VARCHAR2(64),
session_id VARCHAR2(32),
current_state VARCHAR2(64),
application_code VARCHAR2(64),
acl_name VARCHAR2(32),
attribute_list VARCHAR2(2000),
i_is_archived VARCHAR2(32),
id_5 VARCHAR2(32),
id_4 VARCHAR2(32),
id_3 VARCHAR2(32),
id_2 VARCHAR2(32)
)'

REGISTER TABLE dm_dbo.wf_history_s
(
r_object_id CHAR(32),
object_type CHAR(32),
id_1 CHAR(32),
string_5 CHAR(200),
string_4 CHAR(200),
string_3 CHAR(200),
string_2 CHAR(200),
string_1 CHAR(200),
workflow_id CHAR(32),
policy_id CHAR(32),
registry_id CHAR(32),
audit_signature CHAR(255),
audited_obj_vstamp INT,
user_name CHAR(32),
time_stamp_utc TIME,
audit_version INT,
chronicle_id CHAR(32),
controlling_app CHAR(32),
object_name CHAR(255),
audited_obj_id CHAR(32),
version_label CHAR(32),
acl_domain CHAR(32),
attribute_list_id CHAR(32),
host_name CHAR(128),
user_id CHAR(32),
i_audited_obj_class INT,
event_source CHAR(64),
event_name CHAR(64),
r_gen_source INT,
owner_name CHAR(32),
time_stamp TIME,
event_description CHAR(64),
session_id CHAR(32),
current_state CHAR(64),
application_code CHAR(64),
acl_name CHAR(32),
attribute_list CHAR(2000),
i_is_archived CHAR(32),
id_5 CHAR(32),
id_4 CHAR(32),
id_3 CHAR(32),
id_2 CHAR(32)
)

update dm_registered object
set owner_table_permit = 15,
set group_table_permit = 15,
set world_table_permit = 15
where table_name = 'wf_history_s'

Second, create a custom Documentum method to be executed by the custom queue management job. This class should have the following methods and logic:

a. Populate Workflow History Table according to criteria. Here’s an example dql:

"insert into dm_dbo.wf_history_s " +
"(r_object_id, event_name, time_stamp, user_name, audited_obj_id, string_4, workflow_id, string_3) " +
"SELECT '0000000000000000' as r_object_id, task_name as event_name, date_sent as time_stamp, sent_by as user_name, r_object_id as audited_obj_id, name as string_4 , router_id, task_state as string_3 " +
"FROM dmi_queue_item " +
"WHERE r_object_id not in (select audited_obj_id from dm_dbo.wf_history_s) " +
"AND router_id != '0000000000000000' " +
"AND date_sent < DATEADD(Day, -"+sCutOffDate+", date(today)) " +
"AND delete_flag = 1";

b. If the Workflow History Table gets populated successfully, delete the dmi_queue_item rows according to criteria. Here’s an example dql:

"DELETE dmi_queue_item objects " +
"WHERE router_id != '0000000000000000' " +
"AND date_sent < DATEADD(Day, -"+m_cutoff+", date(today)) " +
"AND delete_flag = 1";

c. Write the job report to the repository.

Third, create the custom queue management job.

Documentum Workflow: Reporting History by Document

2007-12-13T11:04:00.000-05:00

Issue:
Tracking workflow status by document from Webtop’s Properties/History tab.

Requirements:
Allow a user to select a document in Webtop and click on Properties/History to view the document’s workflow history.

Solution History:
Webtop: Out-of-box workflow functionality which requires running the dm_WFReporting job and auditing events.

Report: Properties/History
Location: Select Workflow task and go to Properties/History
You’ll get a list of the work activities, there date/time stamps and who performed it.

Report: Workflow Reporting
Location: Tools/Workflow/Workflow Reporting
Attributes: Workflow Name, Status, Active, Task Name, Performer, Supervisor
Edit Report:
User – select a user
Document – select a document by drilling down folders
Template – select a workflow template
Show overdue, all, running, or completed workflows

Report: Historical Report (Process or User)
Location: Tools/Workflow/Historical Report/Process or User
Form options:

Display statistics for business process running
From: To:
Include only process where these conditions are met
Process Template name contains: type in text
Or User contains: type in text
Workflow Supervisor is: select user
Duration: select operater and type in days, hours, minutes
Cost: select operator and type in number
Location is: select folder

WorkQueue Monitor
Location: Select Workflow task and go to Properties/History
Functionality: You’ll get a list of the work activities, there date/time stamps and who performed it.

Solution:
Create a custom Package_History table
Purpose: This is necessary because the link between the workflow instance and the package id (the content id) is lost after the workflow is complete. This provides the track record of workflow’s packages during the workflow activity and after the workflow is complete.

Create a custom Workflow_History table
Purpose: To keep track of workflow queue items. We decided to store all workflow related queue events in a custom table to assure that the historical information for the workflow activities would not be deleted by the queue management job.

Query the workflow history based on the following steps

Look up the workflow instance ID from the custom package history table based on the Object ID.
Select attributes from the custom workflow history table
Union these results with a selection from the dmi_queue_item table

Present the User with the following information about the document’s workflow history
Date: Timestamp of action
Work Queue: Name of the work queue or activity
Performer: Name of the user who performance the task
Action: The event status of the activity

Changes to Webtop
Component changes
· Make a copy of the history_component.xml to the custom folder and extend for the original configuration file, and point the behavior to the new class
· Set “String_3” to true: true so it will show up on the jsp page.

Create a custom component behavior class
The main customization is with the query string method:

protected String getQuery(String strVisibleAttrs, ArgumentList args)
{
if(m_strSelectedVersionObjectId == null)
m_strSelectedVersionObjectId = m_strObjectId;
String strWhere = m_strQueryConditionFormat;
strWhere = StringUtil.replace(strWhere, "{r_object_id}", "'" +
m_strSelectedVersionObjectId + "'");
StringBuffer buf = new StringBuffer(128);
buf.append("SELECT ");
buf.append(strVisibleAttrs);
buf.append("'1' as dummy");
buf.append(" FROM dm_dbo.wf_history_s WHERE ");
buf.append(strWhere);
buf.append(" OR workflow_id in (select r_workflow_id ");
buf.append(" from dm_dbo.package_history ");
buf.append(" where r_component_id = '"+m_strObjectId+"' ) ");
buf.append(" UNION ");
buf.append(" SELECT sent_by as user_name, task_name as event_name,
task_state as string_3, date_sent as time_stamp, '1' as dummy ");
buf.append(" FROM dmi_queue_item");
buf.append(" WHERE router_id in ");
buf.append(" (select r_workflow_id ");
buf.append(" from dm_dbo.package_history ");
buf.append(" where r_component_id = '"+m_strObjectId+"' ) ");
buf.append(" ORDER BY 4 ASC");

System.out.println("Query: "+buf.toString());
return buf.toString();
}

Component Presentation Changes
The history.jsp file is copied and moved the custom folder in /custom/webcomponet/library/history

Documentum Workflow: Queue Management

2007-12-07T16:27:00.000-05:00

Overview
In Workflows, each work item or task has to be managed, maintained and tracked. As workflows are executed and performed, there are certain aspects of managing these tasks that are not straight forward and are definitely not offered out-of-the-box with Webtop.

Management:

Issue:
Work queue tasks could be acquired then left unattended unless a scheduled job runs to determine whether the user who acquired the task is still using the system or has left for the day.

Solution History:

First Attempt:
The first fix was to customize the logout functionality of Webtop to set the work item (task) to the “putback” auto-activity. The “putback” auto-activity then gets the work item’s sequence number and subtracts 1 from it and sets the work item back to the previous activity.

First Attempt Issues:
This does not take into account user’s who X out of their browser without logging out. It also doesn’t scale.

Follow Up Attempt:
We decided to create a scheduled job which would execute a java method that “unacquired” the work items, like the “unassign” functionality in the Work Queue Monitor page on Webtop. This required some reverse engineering of the existing unassign class to figure which services and api’s were used and the objects and jars required. This also required looking at DA’s User Session functionality and bringing that in as well. Here are the components of this solution:

Documentum Objects
dm_job
dm_method

Java classes

Class: public class TimeoutPutBackMethod implements IDmMethod

Methods:

execute(Map params, OutputStream output)

- This is the main dm_method execution method which excepts parameters sent from the job and an output stream which ends up as a report written to the “Temp/Jobs// log files.

getAllActiveSessions(session, output)

- This method uses DfSessionCommand object to get all of the active sessions on the repository.
- This also queries the a_held_by (Username who acquired it) values of the work items to figure out which work items should be unassigned.

unAssignWorkflowTasks(IDfSession session, String UserName, String sDCTMDocbase, OutputStream output)

- Calls the getWQName method to lookup the work queue name based on the activity name.
- Calls UnassignQueuedTask.unassignTasks

getWQName(IDfSession session, String ActivityName)
- queries the dm_activity table looking for the performer (work queue name) based on the activity name.

Class: public class UnassignQueuedTask

Method:
unassignTasks(IDfSession m_session, String m_taskId, String m_queueName, String m_docbase, OutputStream output)

- This method constructs a work queue manager service and work queue object to unassign the work item.

Non-standard APIs used
import java.io.OutputStream;
Used for outputting strings to a job report file.
import com.documentum.services.workqueue.IWorkQueue;
This is used to construct a workqueue and unassign it
import com.documentum.services.workqueue.IWorkQueueMgmt;
This is used to construct a workqueue manager service
import com.documentum.mthdservlet.IDmMethod;
Used to implement this method and to be able to execute it as a dm_method object
import com.documentum.admin.commands.DfSessionCommand;
This is used to get all of the active sessions in the repository

Fixing the Deprecated api exec for Lifecycles

2007-10-18T14:18:00.000-04:00

A few months ago I came across a deprecated api that I was trying to use to uninstall a dm_policy or lifecycle. What I needed to do was uninstall the LC apply a new ACL to the LC object and install it. I searched EMC's dev discussions and found a few references to this, but no real solutions. I knew that the App Builder dealt with lifecycles, so I took a look at the DDS.jar files to see if there was something of interest. There was.

Here's the deprecate code I was trying to fix:

obj = (IDfSysObject)m_session.getObject(idObj);
m_objectId = idObj.toString();
// Uninstall this LifeCycle Object
m_session.apiExec("uninstall",m_objectId);
obj.setACLName(sAcl);
IDfACL idAclObj = (IDfACL)(m_session.getObject(getDocbaseObjectIDIfExists
("dm_acl WHERE ( object_name = '" + sAcl + "' )")));
sAclDomain = idAclObj.getDomain();
obj.setACLDomain(sAclDomain);
obj.save();
m_session.apiExec("install",m_objectId);

In the 5.3 DFC eclipse IDE, the "apiExec" was crossed out. I could compile the code, but the runtime environment needed to be very controlled. After digging into the DDS.jar file of the App Builder I fould a "policy" object which looked useful.

The following is the fixed code:

import java.beans.PropertyVetoException;
import com.documentum.policy.data.*;
try{
obj = (IDfSysObject)m_session.getObject(idObj);
m_objectId = idObj.toString();
// Uninstall this LifeCycle Object
IDfPolicy policy = new DfPolicy();
policy.setDocbaseContext(m_session);
try{
policy.open(new DfId(m_objectId), 1);
policy.uninstall();
obj.setACLName(sAcl);
IDfACL idAclObj = (IDfACL)(m_session.getObject
(getDocbaseObjectIDIfExists("dm_acl WHERE ( object_name = '" + sAcl +
"' )")));
sAclDomain = idAclObj.getDomain();
obj.setACLDomain(sAclDomain);
obj.save();
policy.validate();
policy.install();
}catch(IOException ioe){
}catch(PropertyVetoException pve){ }
}catch(DfException e) { }

DCM Configuration Nightmare

2007-10-10T11:38:00.000-04:00

I was involved in a DCM implementation that was highly organized, but riddled with validation issues. This was in a highly regulated pharmaceutical company with strict qualifications from the development to QA to Produciton environments. Here's an overview list of the issues:

IQs and OQs were dry run by the same users.

Developers were asked to run some of the OQs.

When moving from Dev (where the app worked) to QA, some of the DCM configuration was entered in manually. Inevitable human error caused many late nights figuring out what changed.

So for issue one, we eventually broke out the OQ dry runs to other users. For issue two, we outsourced all responsibility for running the IQ/OQs for validation. Issue three, was never figured out completely until after the project went to production.

DCM Configuration Issues

Why was configuring DCM an issue? Well, for one reason TBO customizations had certain hard coded conditions with relied on the specific Roles and ACLs, Lifecycle states relied on specific ACLs to function as expected, and the UI configuration relied on specific Roles and ACLs as well. Docapps could bring over Roles, but you can't modified them in the App Builder tool. Also, the App Builder doesn't recognize ACLs that are created in a repository without using Permission Templates.

Within DCM you have the following components (among others) to configure:

Business Applications (which are part of the DocumentClass object)
Document Classes (coordinators, contributors, templates)
Auto Name Schemes
Lifecycle State Extensions
Roles
ACLs
Auto Processes

With each of these components comes varying amounts of configuration details. The headaches come with Roles, ACLs, and LC Extensions. All you need is a misspelled acl or a forgotten role assignment to cause all sorts of havoc. The question becomes "how do you compare environments?" You dump all the objects and attributes. You can't run DQLs to determine the differences. You have to create an application which uses DFC interfaces to enable access to all the Documentum objects of DCM and expose all of their attributes.

Extracting DCM objects as XML

What we ended up developing is an XML bridge which allowed us to extract all (or a subset) of the DCM objects to XML. The schema allowed us to compare XML extractions from one environment to another. We would then take the extraction and run the configuration end of the application on a new docbase which had the base DCM installed and the docapps applied. This saved weeks of time manually configuring the application. Over the course of building and validating four environments the weeks saved, turns into months of effort saved by using this tool.

The Configurator Utility is available for you needs. If you would like to find out more about this utililty please email me at webber_john@charter.net.

Specs: We'll write these if we have to

2007-08-30T13:59:00.000-04:00

The good news about writing functional specifications is that you’ll most likely have plenty of examples to read for reference and a template to use. The bad news is that you might have to write it. Even if you don’t write it you’ll have to read and understand it. You’ll have to read the business requirements and understand those too. As you read these, you should be thinking about the technical specs, what prototyping will need to be done, and where the holes are. One more thing, make sure the eventual users of the system are engaged and sign off on all functional decisions. If they blow off meetings and are not involved in the process, they’ll be back later and will demand changes.

Large companies with a lot of procedures, lawyers, regulations, and outsourcing will undoubtedly require that technical specs be written before starting development. This is fine for about 80% of the project, however the technical details during the development, testing and deployment will change. The technical specs will need to be reworked to reflect this change at the end of the project. If you start out with this expectation, you won’t get as frustrated writing the specs without all the details. Be sure to write down in the assumptions that you expect to rewrite the specs after the project goes live.

As you solve the technical issues keep the following questions in mind:

Are the User’s of the system served well by this design?
Are there too many details written down at this stage?
Are there features which are training issues vs. customizations?
How much continuous discovery will this project have?
Will the development or testing be outsourced?

Are the User’s of the system served well by this design?
Any reservations should be considered and detailed here. In many cases the writers of the specs don’t really know Documentum well enough to fully understand possible scope creep issues and the technical ramifications of writing something like “this system will support nine languages…”
I don’t know how many times the specs were written in a vacuum. The User’s were too busy to fully detail the business requirements so the spec writer reads the manual and fills the functional specs with existing functionality.

Are there too many details written down at this stage?
I’ve read functional specs that have been bogged down by details which will change. Here’s where the lumpers and the splitters show there true colors. The lumpers will write too little detail, while the splitters will describe too much. It’s my experience that approximately 10 to 20% of what you need to develop a comprehensive application will not be known at the time that the functional spec written. Make this assumption, record it and move on. You can really slow a project down by trying to perfect all the details at this point.

Are there features which are training issues vs. customizations?
This question is usually asked if the initial discovery phase was done by folks who did not know the Documentum application well enough. The business requirements would reflect this lack of knowledge, would cascade to the specs. You have to gage how much continuous discovery this project might have. Was the business unit as engaged in the project as it should be, or will they perk up during the testing phase and really start thinking about how they’re gong to use this?

Will the development or testing be outsourced?
I had to bring this up because in these times of going the cheaper route, the project just may be going over seas. I wouldn’t worry though, because every Documentum project that I’ve done is way too involved to ship the whole thing to a third party vendor. Outsourcing pieces of the project may make sense. If you have to outsource get ready to write a lot more to describe what to do. Also, prepare to hand hold the third party through some of the more complex details of the development or testing.

The bottom line is that there’s a balance between the time you take to write or correct quality specs and the time it takes to develop. In some cases, it may make more sense to prototype and write the specs afterward. You’ll have to make this decision and get buy in up front. Don’t just do it; if it ends up taking longer to prototype, you’ll be on the hook to explain why.

Prototyping and proof of concept: This is fun!

2007-08-27T16:36:00.000-04:00

This is fun if project managers get that POCs are crucial to the decision making process:

When someone on the team says “The way I did it at company X…”, but this way doesn’t feel right, prototyping should be considered to detail the pros and cons of the solution.
Every project has at least a few gray areas where the business users aren’t quite sure what they want and say they’d like to “see” something.

The fun part is actually working out some of the pieces of the puzzle. It’s also satisfying to prove to yourself and others that your gut feeling was right, or that it was wrong, but not as wrong as everyone thought. Here are some of the pieces to the puzzle that you may be able to figure out:

Is the customization possible? What is the level of complexity?
How many resources are needed and what type to achieve the functionality requested?
Is the estimated amount of servers of the infrastructure correct?
Will the application solution perform to the user’s expectations?
Will the schedule slip because of the functionality required?

Setting a POC sandbox also allows for installation experience and a preview into some of the issues that you’ll have to solve later. For example, when was the last time an installation of a Documentum product went exactly according to the installation documentation? Is your database administrator new to Documentum databases? This would be a good chance to get through the DBA headaches.

What about storage requirements? After loading up some dummy data, does the total disk space estimate make sense? When the dev, test, and production environments are taken into account including extra room for migrations, is there still enough space?

TBO vs. WDK customizations
I really like TBOs. They are very useful to making changes to attributes during all content operations. There will be times when you look at a customization to a drop down in the UI to achieve some kind of functionality and it occurs to you that it could be done with a TBO. Prototyping will help you work through the reality of that idea.

The bottom line is that you will be dealing with scope creep if you don’t prototype solutions when it makes sense to do so. When someone says “we need to customize” , make sure through a test of the functionality of the application that it’s not just a configuration. The WDK can be very flexible in terms of configuration and TBOs are great to use for customizations.

WP installation and configuration: This can be painful.

2007-07-30T15:10:00.000-04:00

First of all there’s very little detail in the installation manual on how to actually configure Webpublisher to work correctly. For example, here’s how you first set up a template:

1. To import your first Template file, first you have to know what an example template file should be, then you have to know that to import you must create a new “category” whatever that’s supposed to be, go into that new category and then import. Patched together functionality or what?
2. To edit your first Template, you need a rules file. So you go to the “Rules” area and create one. You go back to the Template, check it in (because it was checked out when you to edit it the first time), View/Associations and click on Rules, and Add the new Rule. Is this intuitive enough yet?

Second, once you’ve installed the WP, you have to install and configure Site Cache Services (SCS). Configuring the website publishing using Documentum Administrator is relatively straight forward, however, when it comes to figuring out what happens during a SCS publish, you’re kind of on your own. One thing to check is the SCS configuration in WP. To do this, click on “Web Cabinets”, select the cabinet that you are publishing, and click on “View” and select “Web Cabinet Overview”. This should show you the publishing configurations that were configured earlier using DA. Here’s what happens during a Site Cache run:
1. Either the SCS job executes or WP promotes a doc to Staging or Active which invokes the dm_webcache_publish method.
2. SCS service reconciles modified files with the configuration and the source SCS tracking table.
3. A properties.xml file is generated and sent over to the target SCS server.
4. Content is transferred via http or https to the target server.
5. On the target, when all of the files have arrived, target SCS processes database inserts and deletes if there’s a database, it expands zip_html files, it copies the files from the temp directory into the live website, and much more.

File extensions: htm vs. html
The first time you site cache the html renditions you’ll notice that the files are published with the “htm” extension. To get the files to publish with an “html” extension, you’ll have to make changes to the webcache.ini file located on the content server in the “/Documentum/dba/config” folder.

exclude_formats=xml
use_docbase_formats=true

You’ll also have to change the extension of the html format using Documentum Administrator. It took me a day of part-time rummaging through the forums to piece together this configuration fix.

There will be times when you want to throw up your hands and give into a beginner WP class, but by investigating this issues that arise first by browsing the forums at EDN and Bluefish and other site, you’ll learn as you go and more importantly remember what you did when you need to configure WP again.

WebPublisher Business Requirements: Who Are We Writing These For?

2007-07-25T10:39:00.002-04:00

If someone tells you that Business Requirements are not that important to your role as a developer, ask them what happens if you need to make a change to them? Get involved in this phase if you can because BRs set the tone for the whole project, and you need to meet and get to know all the team players. Most developers get pigeon-holed into roles that suggest that they just do what you tell them to do; follow the specs… It might feel comfortable to do this, but you’ll always feel frustrated during the process, especially if you need to push back on the BRs.

Know Webpublisher
As you sit through the Discovery phase (if you get invited) you’ll gather incite into the issues that really matter for the project and who will be looking for what functionality. What happens during the business requirements writing phase is that everyone becomes an “expert” in Webpublisher. The requirements reflect what the business thinks WP can do OOB, not necessarily what it’s potential is. To be fair, the main objective is to use WP OOB as much as possible, but also to find the middle ground in terms of configuration vs. customization.

Consultants Who Know Everything
If there are outside consultants, especially those newly graduated from our fine Ivy League schools, they will commandeer the discussions about requirements and do breakout sessions to fine tune their own understanding of the requirements and their skills in Excel. Business school graduates are fond of Excel and can achieve incredible feats in expanding BRs to new levels of complexity. These are well thought out, but are usually overkill for OOB WP functionality. Make sure the consultants know WP functionality inside and out before they start writing BRs. The more the requirements grow in size and complexity the less likely that the business users will read all the requirements and sign off on a document they can completely grasp.

Think about the whole application
It’s easy to get bogged down by concentrating too much on the details. For example, there were many times when we drilled down on the requirements only to realize that it was close enough to OOB functionality that we didn’t need to write it all up. By “whole application” I mean:
- What’s the end to end flow of content through the system?
- Who will access the content and when?
- What level of support can be sustained?
- How much training will be needed?
- Do you have enough resources to satisfy all the requirements?

Whiteboard the end to end flow
One absolutely necessary step in the requirements process is to draw out the end to end flow of content so that everyone is on the same page. This flow diagram will serve as a milestone of the group’s understanding of the application. It will be altered and republished, but the core flow should not change. This flow should have a combination of technical and visual elements. It should be of use for the business side as well as the IT side. Everyone needs a visual representation of the application and now is the time to create it.

There will be times when an issue can not be resolved by the people within the team. This is the time when you spin off pros and cons of the possible solutions to the problem. This is also the most frustrating time of the project, when egos are flying and a lot of time is wasted. A few types of issues I’m talking about are:
- Configuration vs. customization
- Integration vs. manual data entry

Configuration vs. Customization
It’s easy to get excited about developing new functionality to WP mainly because WP has many foibles that are usually corrected during the first release of the project. However, you have to weigh customizations with wanting a smooth upgrade path. One of the WP projects I was a part of was so heavily customized that management waited until the version was no longer supported and then had to plan the upgrade to which was scheduled to talk over a year to accomplish. This type of customizing indulgence is usually avoidable if the right balance of Configuration and Customization is achieved.

The typical pros and cons of these issues involve:
- Cost
- Length of time to develop
- Testing
- Functional worth
- Priority
- Impact on upgrading

Integrations
There are all sorts of integrations that you can do with Documentum, the one I’d like to talk about is one with Documentum Compliance Manager (I’ll carry this example throughout this series on WP). The pharmaceutical company I worked for wanted to be able to run certain regulated web content to DCM. A content flow diagram was created and a prototype was developed during the requirement writing phase mainly because no one knew for sure if integrating WP with DCM was even possible. By architecting the solution with a TBO we were able to integrate with DCM without disrupting any of DCM’s functionality.

“Nice to have” vs. an absolute must
When you have the business users in the room they will say everything they bring up is necessary. But, when pressed, they’ll start to admit that some functionality is “nice to have”. WP offers many features that can be configured to get close to most requirements around publishing to a website. I’ve migrated from a custom Documentum web application to WP and the biggest complaint was the amount of versioning that goes on in WP. They also had very complex ideas on how to organize the content. These ideas carried over into WP as categories. WP categories are versatile and powerful when used in the right way, but, in our case, we went way overboard. We had over 10k nodes, multiplied by 10 locales. This was and continues to be a nightmare.

Two thorns of WP publishing
There are two issues that plaque WP: Security and Navigation. Customizations gravitate toward these two monsters. Why hasn’t EMC purchased or built a web server / app server yet? Sitecache is nice, it’s decoupled, it scales. However what happens when the content gets to the website? The coupling of the decoupled security and navigation is a sink hole for bad performance and enormous support issues.

2007-07-23T16:30:00.000-04:00

Discovery: do they really know what they want?

As a Developer, depending on the size of your company and how enlightened the management, you’ll most likely get involved with the Discovery phase of the project. If you can try to influence who’s on the Discovery team by suggesting other resources which you trust will “do the right thing” try to do it before the first meeting.

Don’t solve the issues: listen
The first few meetings during this phase are spent listening to the Project Manager(s) go over what they believe are the questions that should be asked and the corresponding answers from the Business Manager(s). Developers: don’t try to solve these complaints in the meeting, just listen and answer questions when asked. Be careful not to ask detailed questions at this point, as the discussion will dive way too deep into the nitty gritty and could get religious. The managers need to hear themselves talk. If you create too many disruptions at this point, they’ll write you off as a pain.

Know OOB functionality cold
I’ve been in situations where I knew the functionality being asked for what hard to implement, but wasn’t 100% sure and couldn’t back up my gut feeling with proof. If I had known WebPublisher OOB functionality better I would have been more effective in arguing my point. One way I dealt with not knowing the answer to a technical question was to suggest that I prototype the functionality and get back to the team with an answer the next day. This allowed me to get proof (or not) of my gut feeling and work through possible solutions without getting shutdown immediately. This also saves lots of time and effort in the long run by exploring the ramifications of a small decision which could balloon out the project (80x20 rule).

Where does the content come from?
The content of the website project was already designed and developed, but was being hosted on a tradition web hosting site. The website content was FTP’d to the site and manually published. The users wanted “everything we have now” and more. The “everything we have now” was a black hole which needed to be dealt with asap. What can you do with a statement like that without going through a huge process of requirements gathering? What we came up with was punch list of what WP does and presented it to the business.

Lumpers and Splitters
On the project team, there will be people who like to drill down on every piece of functionality, not only to understand it but to drive others crazy. They like to split issues into smaller pieces and understand them better that way. Others like to lump issues together and hand off the large understanding to those who understand the software (WP) better. During each meeting the lumpers and splitters will go at it. If you can catch this happening before it gets too heated, call for another meeting to hash out the particulars. It’s important to keep the focus on manageable chunks of the project.

Who’s poking holes in the boat?
During these meetings, at first it was hard to figure out who was aligning with whom in terms of making the hard decisions about what functionality was in and what was out. Over the course of a few meetings, it became clear and it also was obvious who was fully invested in the project and who didn’t care. The folks who didn’t care reacted by saying things like:
- “Let’s cross that bridge when we get there”
- “We don’t have any expertise in that area”
- “We don’t have the time or money to do things the right way”
- “Let’s get a consultant in to look at this, I know someone…”
- “What are the risks involved with this decision?”

Sense of Urgency
Remember that managers like to drive the project to their own schedules. During the discovery phase, if you don’t push back when necessary, you’ll be setting the stage for long hours down the project lifecycle line. Managers want to push the project along no matter what issues arise. Let’s say you’re trying to decide whether to use WP’s import screen the way it is or to customize it. This kind of exercise requires time and attention to detail. If something is missed, you’ll pay for it later.

Your homework
As the high-level, prioritized issues are ironed out you’ll get a better picture of the scope of what’s being asked for. During the priority step, you should fill in a matrix that you make up on your own. Columns that I’ve used in the past are Requirement, Priority, Scope, Effort, Custom/Config, Cost:

- Requirement: This is the specific functional piece of functionality that is being asked for by the business.
- Priority: The weight given to the requirement by the business.
- Scope: All the WebPublisher, technical and project-related ingredients of the requirement.
- Effort: Gut feeling on how much time it will take to accomplish.
- Custom/Config: Are there any customizations? Is it really all configuration? Try to find any holes that are not being discussed
- Cost: Any potential sink holes that will be expensive.

Example:
Requirement: Automate website approval process
Priority: 1
Scope: Design workflow template, notification design, prototype, user testing, rework, application migration from dev, to test, to integrated test, to training, then to production.
Effort: Moderate
Custom/Config: Mostly configuration, possible customization: deal with aborted workflows.
Cost: Developer training, consultants

Cognitive Dissonance
Take note of Business folks and members of your team who do not admit when they are wrong or have made a mistake. These are the people who will sometimes unwittingly take the project of course and will need to be reigned-in when they do so. They are also the people who pull some of the team aside and have “off the record” discussions about the project. They occasionally need ego boost by show boating an experience in the past that they think is important: “the way I did that at was to implement with .

The following are some paths of discovery decisions and where it took WP customizations:

Security
- Implemented RSA Cleartrust
- Had to add 2 servers per environment: 10 in total
- Had to hire an RSA consultant for initial installations
- Spent numerous cycles with basic documentation and endless trial and error troubleshooting.
- Customized the login, denied, and logout pages
- Held up the project as authentication became the critical path during the initial testing phase

Multiple Locales
- Used Categories to manage multiple languages for the website navigation.
- Created 10 sitecache configurations, times 5 for each environment.
- Redesigned the web cabinet structure to accommodate the new content submission groups.
- Trained publishers how to add translations and publish them.
- Created a custom application to integrate with outside translation vendors.
- Dumped huge amounts of money on developing an automated way to process the sitecache target content and metadata.

Automated Navigation
- Designed a simple way to publish navigation changes based on folders and linked content.
- Developed a TBO and customized WP to prototype the functionality.
- After the User Demo, we rewrote the design and developed a much more complex solution.
- This went on for a few more cycles until we had to stop and move on to other aspects of the project to meet our schedule.
- The business said they wouldn’t make any changes to the site folders for a year. After the site went live using WP, within a month they were changing the navigation.

Discovery is a term used for exploring what the business would like to have. What it really means is finding out what is realistic to have now and what is going to be released at a later date. There will always be disappointed members of the team who will never be satisfied and other who will feel over worked. Hopefully you’ll have a balance of these types with the more motivated types who enjoy what they do.

Documentum In the Trenches: WP

2007-07-20T13:46:00.000-04:00

I have been working with EMC Documentum's Webpublisher long enough to know a few things about the got yas during the phases of a website project.

There is no guide that provides a soup to nuts view into how a website publishing project is designed, developed, and implemented.
Webpublisher will have issues, it's just a reality.
As with most websites the business requirements will focus mainly on the look and feel of the site, not the mechanics of how you get the content on the site.
If you like to follow project schedules closely then you better pad in time because you're going to need it.

With these four observations in mind, I'd like to take you on a ride down Webpublisher memory lane with a project that I recently completed (no projects are ever really done, who's kidding who?). The structure of my remembrances will following the usual project lifecycles:

Discovery: do they really know what they want?
Business requirements: who are we writing the requirements for?
WP installation and configuration: This can be painful.
Prototyping and proof of concept: This is fun!
Specs: We'll write these if we have to.
WP Development: Try not to Thrash.
Deployment: That wasn't in the spec!
Training: That was easy; until they actually start to use it.
Stabilization: Is anyone using this?
Support: Who's doing it?