Monday, May 4, 2009

Max Session: Obscure Documentum server.ini key saves the day!

Windows D6.5 installation using Webtop, some WDK customizations, and a TBO for major customizations. The TBO has to create a superuser session to do some work.

Max sessions are reached on the application server after a few hours of use even though the “Active” sessions are far below the configured threshold. In other words, the Tomcat server is counting “Active” and “Inactive” sessions in it determine of “max” sessions.

Keep in mind that there is a lot of customization in the TBO and this required creating and releasing a session manager for a superuser during each save of the document. This is what is building up the "Inactive" sessions. 

Patch Solution Given
The initial solution was to jack the max sessions up to 100,000. This caused the Tomcat service to die once a week or so, basically maxing out the memory allocation to the process.

Max Sessions Investigation
I opened the Documentum Content Server Administration Guide and searched for “max” or “session”.

I listed the variables in the session being maxed out

  • Application versions
  • Property files
  • Custom code

I looked into implicit and explicit sessions

Hint: For testing superuser in TBOs, I used a different superuser than the install owner account.

Application Server

  • Measured by Active and Inactive Sessions
  • Key/Value Pair: dfc.session.max_count = 1000 (default)
  • DQL: execute show_sessions
  • DA: Administration > User Management > Sessions (All)
  • web.xml: HTTP session timeout is set in the \app server\ web.xml (default is 30 min): 30
  • Hint: To find leaks set dfc.diagnostics.resources.enable = true (default is false)

Content Server

  • Measured by Active Sessions
  • server.ini Key/Value Pair
  • concurrent_sessions = 100 (default is 100, max is 1024). These sessions are “Active” sessions from the content server’s perspective
  • history_sessions = (how many timed out sessions show in list_sessions)
  • history_cutoff = (default is 240 minutes)
  • client_session_timeout: default is 5 min
  • check_user_interval: frequency in seconds which the CS checks the login status for changes.
  • Default is 0, meaning it checks only and connection time.
  • login_ticket_timeout: length of time a ticket is valid, default is 5min
  • DQL: execute list_sessions

Final Solution

I added “history_cutoff = 5” in the Content Server’s server.ini file.The “history_cutoff” key controls the longevity of the inactive sessions. The default value of this key is 240 minutes (4hrs). This would explain why only on occasionally the max session is hit.

My testing has shown that if you set the “history_cutoff” key to a value much smaller like 5 to 30 minutes, that this allows for the inactive sessions to clear reasonably soon, so as not to fill the max sessions of the Tomcat server.

To test this I set the following:

Set up WDK Automated Test Framework to run the same tasks over and over again to build up Active and Inactive sessions.

Set up the Content and Tomcat servers with these base line settings:
server.ini file: concurrent_sessions = 20 in the webtop/WEB-INF/classes: dfc.session.max_count = 30

Result: The Tomcat server fails when the total of Active and Inactive sessions exceeds 30.

Then set up the Content and Tomact servers with these settings:
Settings with history_cutoff changed
server.ini file: history_cutoff = 5 and concurrent_sessions = 20 in the webtop/WEB-INF/classes: dfc.session.max_count = 30

Result: The Tomcat server fails only if the number of active sessions exceeds 20, thus relieving it of the inactive session burden.