Why Financial Services and ECM is a Challenge

Most Financial Services companies large and small have really screwed up the ECM model. Years ago when they purchased Documentum, IT folks looked at the technology and installed the “docbase” server with HA and DR, and that was it. They are heavily into databases and the interrelationships of client IDs and Customer IDs, products and services, etc. IT looked around at the time and saw programmers, DBAs, and web tech guys and thought we’ll build our own frontends to Documentum, our permissions and rules for metadata are too complex, too fancy for the WDK. This was partly true as EMC Professional and Product Services never caught up to the specific requirements and functional needs of the finance industry as they did with life sciences.

Typical scenario at a Financial Company

The client applications director has been somewhat satisfied for the past four years with the way IT has maintained custom web applications integrating with Documentum on the back end. The director is getting more heat from above to look at cheaper, OOTB Documentum solutions. The IT department development (even outsourced to India) is getting too expensive. IT has been planning and executing an upgrade for the past year and a half. They are still at 5.3 (not supported) for the repository and are upgrading the website DFCs to 6.0. Why not 6.5? Because at the time of planning 6.0 was just coming out and the client development started. This is way too expensive and lethargic, but what are the alternatives?

BSAs with their head in the sand

The Business Systems Analyst Manager, with his MBA says we need to think about the big picture, but deal with the low hanging fruit now. In other words, there’s no bridge in communication between operations and IT. This is where a Deloitte is usually helpful in terms of getting the too sides to agree on long-term goals. I know consulting firms can be devastating, but in this case, all the MBAs need to get together and show off their power points. It’s not about just fixing one issue then the next. It’s about getting the CIO to wake up and figure out ECM should be handled like it’s ERP. There is usually a disconnect between the Client App Business Ops vision and the IT technical vision. This will not get fixed at the director level, as much as the C’s would like it.

IT is RESTful with their plans

Historically, the IT Internet/Extranet group has run the show. Choosing how to architect the ECM solution as a by product of their database and web application server architecture plans. Just figure out the storage of content and keep it forever. Here’s the argument: we already have custom applications for Documentum that work. All they care about is the “harvesting and production” side of content. They have no clue what the business is actually doing. How do we get IT to feel the pain of the business? A strong Client app director who cares about business process and content management would help. Get the IT managers out of the web services cloud and either do something enterprise wide with CMIS and other integrations or work with the OOTB products.

EMC Professional Services Where Are You?

The problem with EMC Documentum professional services is that it is not engaged fully during the pre sales activities of the sales engineers who are building POCs and demoing new software solutions. If they are they’re spread too thin and are not effective enough. They can’t keep up with actual work let alone consult on goals and vision. Most companies form their opinions on the can do all sales guys. The client is overwhelmed with products not solutions. They see scanning solutions, but not high end UIs which solve the complex financial relationship issues. A WDK or Taskspace form is a child next to the maturity of what IT has created for their highly customized UIs. EMC, what’s the solution here?

Sample Financial Solution Patterns

There are many solution patterns within the financial applications that EMC could solve with a product.

1. Center a User Interface around Client and Customer Ids. The pre vs. post sales information gathering activities.
2. Deal with the historical changes in attributes and their values. Financial services databases are always current and don't necessarily handle change management in content and metadata very well.
3. Like the case management solution, figure out the foundational use cases of financial services (beyond scanning and OCR) and build product and solutions from them.

eDiscovery’s Promise Is Dependent on Your Enterprise’s Laziness

eDiscovery solutions are predicated on your company’s lack of cohesive and best information management practices. If you read any eDiscovery software solution’s proposal it will not mention how to fix the issues of inadequate metadata structures, lack of auditing, risks of pre-ediscovery spoliation. They will not tell you to get organized and clean house. They’ll say information governance is impossible to keep track of, that information growth is out of control. They’ll all be lawyers and MBAs and be able to pontificate extremely well about your issues, but do they really want you to help you where you really need it?

If I was selling software they was really a search tool and a legal CYA process would I be interested in telling how to create an Enterprise Metadata Repository? Do creditors want you to save money? Managing your content requires standards, conventions, governance, and compliance. MoReq is an EU set of standards for records management that basically outlines the keys to eDiscovery freedom.

Without the discipline and perseverance that is necessary to push these throughout your company you’ll be stuck with an eDiscovery application infiltrating your email, ECM repositories, file shares, etc. It’s a matter of how you want outsiders to carte blanch access to your information. If you know where everything is then you’ll be able to clearly define and execute the information discovery without a whole sale fishing expedition on all of your seemingly sensitive information. Most importantly you’ll know what needs to be disposed of!

So what’s it going to be a house of hoarders, or an extreme home makeover?

Architecture without an Effective Enterprise

So what are some of the issues that arise when a large company does not have an Enterprise Architecture group? How could this happen, especially in a company which is dependent on validated systems and many different and competing standards?

• Business Partners are frustrated with the lack of governance and execution of projects that matter to them. The project pipeline for smaller projects is large. The large projects get funding to trump the smaller ones, pushing the smaller deployments back by years. The Business threatens and sometime does build its own solutions.
• Project liaisons like Business Relationship Managers, Project Managers, and Business System Analysts are overwhelmed and focus mainly on timelines, milestones, and deliverables without knowledge of other projects and synergies that might be available from an architecture perspective. There will be different levels of training and experience across application specific groups.
• The Service Delivery Team lines up resources and figures out who can do what, when, and thinks everything else will just have to work. The issue is that other outside team resources may not be considered by the Service Delivery Team. For example, the infrastructure team may be working on 10 other different build outs and there’s no priority.
• Solution Architects will clash with Network and Storage Architects who are blindsided by project changes and overwhelmed by lack of resources. The documentation of each group is not consistent. There will be no overarching document template standards across the application stacks. There are 15 different icons in Visio diagrams that represent servers.
• Large Consultants will infiltrate easily into the heart of ERP and ECM and try to get into Legal and Records Management. These consultants will suck big bucks out of IT’s coffers. Knowledge transfer will be an after thought and spotty at best. The stability of process and deployed applications will take much longer to achieve.
• The Integration Team will be very busy patch holes with webservices and other protocol, but there is no EA team looking at the whole wall of patches to determine where the future cracks will be. Each patch is customized with different metadata requirements, no common values, no Moreq2-type analysis. The testing of upgrades across integrations is too little, too late. For example, there is a general lack of awareness of the full impact of changes made in Active Directory.
• There is no overall orchestration of applications, integrations, standards, ROI, and cost controls. The budgets for projects and their corresponding SLAs are out of balance. The downtime for upgrades blows away the accepted downtime window.
• Priorities of projects lack coordination of architects experienced with building platforms which scale. IT Governance touts its own top 10 projects, while ignoring the priorities of the business partner’s projects and initiatives. For example, deploying Sharepoint 2010 with complete confidence in the first release vs. holding off on a major ECM upgrade until the first or second point release, even though the business is clamoring for the new ECM functionality, eg. Center Stage… Another example would be launching a large project with two versions of the same application because in the time it takes to deploy a new version is up and user are clamoring for its features.
• Records Managers and Legal Departments will be forced to drive metadata standards at the enterprise level. They will most likely be stonewalled at every turn, especially by IT.
• The Infrastructure architects are explaining too much and doing too little. One person questions the amount of storage required because of cost and another says don’t worry about the money. The racks in the server room have blades tucked into spaces that are not shelves. The power supply of the server room is maxed out. There is no coherent Storage strategy for expansion. Records Management and thus disposition is far down the road.
• There is a high churn rate of smart developers and architects. There aren’t enough incentives to keep them. They are too busy putting out fires so they quit and move to companies with EA teams in place.
• These are some of the issues that arise when a medium-sized company grows into large one. However the EA team should be in place from day one. All companies should recognize the importance of standards and centralized governance of IT architecture.

Sharepoint and the “Others”

If you watch the TV show “Lost” you’ll know what I’m talking about when I say that Sharepoint is one of the survivors of Oceanic flight 815 and Documentum, Open Text, IBM, Oracle are the “Others”: the ones who are indigenous in the ECM space, the island. The Others are adept at moving around on the island. They know the lay of the land and the dangers that lurk at every turn on the trail.

Sharepoint is poised to set up camp on the easy to reach, safe locations like the nice beaches, or in the collaboration and project oriented areas of the enterprise. The Others will lurk in the deep jungle holding onto their strongholds. As Sharepoint becomes more equipped to fight the Others and outsmart them, the old patterns of ECM will evolve; Sharepoint propogates and splits into factions; there is no central goal or common standards in the way Sharepoint is used and spread.

This Sharepoint scenario is very similar to the internet boom. Remember what happened when the first wave of intranets hit? Individual business units spun their own websites overnight and they catered to their own unit’s requirements. Everyone was happy among themselves in their own insulated groups until the Others took out their sticks and started asking questions like, “what’s your metadata model?”, or “does this site scale?”, and “Can you search this thing”, “do you have a review and publishing process?”, and so on.

These “shadow IT” groups were slowly consolidated, shunned, taken off island, until at last a centralized ECM solution was implemented. The ECM solution was great for IT, with its centralized support and architectures. The problem was it took an act of god to add new functionality and it was expensive. The more centralized the ECM solution got, the more business units wanted to break away and fulfill their own needs. Enter Sharepoint: small and uncoordinated at first. But now with SP 2010 the Others are no longer that dangerous. The Others needed to evolve, but did not morph fast enough. The Others now are hiding in the strong holds of records management, index/search, business and forms processing, case management, scanning, and license agreements. The Others will be forced out eventually to face Sharepoint. Google and open source will follow in pursuit as well.

Meanwhile as business units go wild with Sharepoint and architects run behind them trying to put RM, metadata standards, storage in place, Sharepoint will be dealing with its own issues. The silos of information will not be fixed by technology, by search. Info silos will only be fixed by continuously evolving and releasing easier ways to institute standards and conventions around organizing/normalizing data and content. As soon as the structure of lists in Sharepoint is deployed, the structure will have to change. Sharepoint may well go the way of individual intranets. Getting the business as a whole to agreed on certain rules and policies and to comply with best practices is the real issue and IT can't do it alone.

Let’s hope the user’s obsession with the UI will go away and a new vision of usefulness and knowledge building and sharing will arise. Knowledge Management of the 90’s is just morphing into collaboration and search. Who knows what, where, when, how, and why is the framework to design to. Integration of desperate services by common interoperable factors (CMIS) is one of the keys. The Others have known this, but have been driven to distraction by profits and the friuts of the jungle. Now, as the island is rumbling for change, the Others will have to change their tactics and listen to the island for future instructions.

Documentum vs. Open Text Content Server

Open Text Content Server and Documentum grew up together during the same time, but in different neighborhoods. Sometimes I wonder how certain architectural design decisions were made, or how lack of funding may have affected the level of integration between the products in their respective software suites. When compared with DCTM, OT is like an old Mustang and DCTM is a newer Toyota with recall issues. This is by no means a comprehensive comparison of the two software suites. It points to some important ECM aspects and tries to expose some of the issues and precularities.

Session Thread Model
OT has an understood session thread limit of 5 per server. Can you imagine this with DCTM? OT will fix this with their 10X release, but customers have suffered for years with this limitation. Session management in DCTM is mature, but has its own issues. So in OT the way to scale for many users is to build out Livelink servers (5 session threads) per server. If you had the potential for 30 concurrent sessions, then you’d have to build out 6 server instances. These OT instances would have to have all of the modules, patches/fixes, on each. This is 6 times the deployment effort. It took OT 10 years to scale concurrent sessions on their base server kernel?

DocBroker vs. Nothing
Here’s a real big issue with OT: failing over one repository to another. OT just can’t unless you want to purchase archive server which is not really part of the LL server and not pertinent to this. OT relies entirely on load balancers for distributing client requests. Yes, the docbroker has its issues too, but at least it can be used to failover one repository to another.

Object Model
Because Open Text doesn’t have the foundation classes that Documentum has it is hard to conceptualize the inheritance model and relationships between the underlying database tables.

dm_sysobject : DCTM’s main table which holds the chronicle and object ids of most of the objects in the repository (dm_type, dm_user, dm_acl are related by not derived from dm_sysobject). Document objects extend from this table in the DFC and are related to separate doc tables per configured doc type. Both architectures have main tables which are related to be most other tables. This has its pros and cons. Let’s take a cursory look at both models and try to compare some key areas.

Dtree: OT’s main table which holds the data id of all objects. If you look at the schema, it’s like a wheel with spokes all leading to the dtree hub.

Version Trees
If you ask an experienced OT administrator if there is a parent id to a document id, chances are he would not know the answer or he might even ask why is this important? Version in OT is not an option. In DCTM at least there’s a way to control versioning during the work in progress states.

dm_sysobject: The parent id of a version is the chronicle_id which holds all of the object_ids. The chronicle and object_id are in the dm_sysobject table.

version tree: dm_sysobject chronicle_id and object_id


dtree: The parent id of a version is the docid which is in the dversdata table and the version ids are in that table as well. The key between the dtree and dversdata table is the dataid which is the docid

version tree: dtree dataid = dversdata docid then versioned

Given this simple example of versions, you can see that OT’s table join for version look ups of basic system level content attributes may not scale as well as DCTM’s.

Permissions and ACL Inheritance
In DCTM there are separate tables and objects which manage permissions of folders and content, as well as system objects themselves. In OT, there is no such concept: permissions are inherited by default from folders only. In DCTM, ACLs can be inherited by object_type, user, or folder.

OT: folder only
DCTM: object type, User, Folder

If you think about the advantages that DCTM’s model has over OT’s look at how limiting it is for workflows which may rely on changing permissions to an attachment as it goes through the review and approval lifecycle states. In DCTM you set the ACL and you’re done. In OT you set the permissions for each user and group at every state. What a change management nightmare!

Object Types vs. Categories
When I first worked with OT’s categories and understood how they related to content and served as metadata tables, I liked them (kind of a cross between custom object types and aspects in DCTM). Then I learned that if you add an additional value to a dropdown selection, not only do you add it to the category, but the system has to update all of the content metadata as well. Each document is coupled with the UI’s potential values in a dropdown selection box. This is mind boggling. OT’s solution to this is the turn off the modification trigger during the category “upgrade”. This makes me worship DCTM’s value assistance.

DCTM’s metadata inheritance model: the custom object type table inherits attributes from dm_sysobject. This makes the doc type, plus an ACL is associated to each object. UI metadata and document metadata are decoupled.

OT’s categories: all content is dtree content. A custom category is an “object type”. The category metadata and the content metadata are coupled.

As you can imagine this does not scale. One attribute value change in the category has to “upgrade” potentially millions of documents without actually modifying any of the content’s metadata.

Development/IDE
I never thought I’d say this but compared to LiveBuilder, Composer (with all of its bugs) is hands down a superior development environment and deployment tool.

DCTM: Eclipse/Composer
OT: LiveBuilder

UI customizations: WDK/TBO vs. Modules
MVC development in OT is more streamlined than DCTM but it’s more manual and awkward. Deployment of OT’s UI customizations straight forward, like DCTM it requires a web server restart. A bonus of OT is that you don’t have to shag after caches between deployment environments. Despite OT’s cryptic Oscript, I bet the learning curve to be productive in developing it is faster than the WDK.

As far as extending core methods in OT, you’re out of luck. As far as I can tell, TBO/SBO extension customization concepts were never introduced into the OT environment.

DCTM TBO, SBO, WDK: JSP/Configuration/Java Behavior class
OT Modules: Oscript/HTML

Webservices
DCTM wrote the iECM spec for service interoperability. Like DCTM, OT plans to expand and eventually only use webservices for all MVC requests and responses. CMIS adoption and use will slowly rise as large companies sunset their old repositories and bean count the new services into their road maps.


Users/Groups:
OT’s users and groups are keyed by a Number, in DCTM the key is the user/group name or String. Using a number allows OT to manage the changes in User names easier, however there is no such concept as a federation of repositories. OT is designed around a central repository user concept. Users created in one repository will have a different id than the same users in another repository, however the synchronization of AD between environments mitigates this issue.

OT KUAF: Number
DCTM User: String

Potential for Home Grown Mess
The maturity of a repository is directly related to how much resources and attention to ECM standards were governed during the repository’s expansion. If the users were in control of the folders, permissions and content, then chances are you have a mess of projects, permissions that do not scale, and outside URLs that point to the repository’s contents from portals and embedded in documents.

Somehow companies with DCTM seem to be more adept at realizing the limitations of performance and scale when it comes to the difference between consumers and contributors. Maybe it was Webpublisher’s adoption success over OT’s lame attempt at the same.

The concept of decoupling the consumer interface from the contributor’s is what web 2.0 is all about. But both camps have an uphill battle to convert repositories in flexible content architectures.

Storage Management
What can I say EMC owns Documentum and it was a beautiful purchase in terms of storage management. I leave it at that though. OT’s archiver is an after thought compared to the power of taking BCV snaps of servers. Of course only large companies can afford EMC storage solutions…

Developer Community
DCTM developer resources cannot be touched by OT’s. I’m glad I learned DCTM’s solution before learning OT’s. If you are like me and learn the most by examples, DCTM is heaven. Part of the issue is the OT didn’t seem to know that Oscript was a very limiting factor in terms of adoption of OT as a whole. They didn’t get that in-house OT administrators need ways to develop their skills beyond following installation and admin guides. Learning Oscript meant you were locked into OT’s vision and world, but DCTM, Oracle, IBM, etc were out there as well.

Each section above could be a chapter in a book, but this a blog and I only have so much time to elucidate some of the points of comparison. It's hard to compare a teenage to an adult, but it's my opinion that OT hasn't been able to mature as fast as DCTM. This puts it at a disadvantage in terms of competing with Sharepoint and 2.0 technical advances and User's expectations in of UI functionality. So, the ability to be nimble and move with the technical changes in the world will prove which repository is more ubiquitous in 5 to 10 years. Also, our friends the bean counters will help slow adoption of new technology and especially new ways to thinking and expressing ourselves within the corporation.