ERPM General Maintenance Tasks

Follow

Version: 2.1
Rev Date: August 23, 2016

Background

Enterprise Random Password Manager (ERPM) is a product designed to manage privileged identities in a variety of ways and respond to incidents involving the disclosure or malicious use of these credentials. Privileged credentials take the form of passwords, SSH keys, group memberships, password lists, privileged sessions and more. Over time, certain things need to be done to keep ERPM and more specifically its underlying hardware (VMs) in tune.

This guide will discuss items to check over time such as VM, data store, etc.

General

Most customers, even the largest customers, typically review ERPM system status every three to four weeks. A typical check includes looking at the jobs monitor for over all job run health status (did unexpected things fail?) and looking at DB indexes. In many cases, customers are using Microsoft System Center Operations Manager (SCOM) to also help examine the overall health of the ERPM ecosystem. Lieberman Software provides a SCOM management pack on the ERPM download page (https://liebsoft.com/products/enterp...duct-download/) to help monitor the overall health of ERPM.

Data Store Data

The longer ERPM runs, the more data and jobs it will accumulate. Some data is relevant to running operations and some is not. For example, a one-time job run for password re-randomization that will never occur again can likely be removed. Job logging data from five years prior can likely be removed. Your data retention policies regarding security logs may only span two years. All of this data can be managed automatically by ERPM.

To have ERPM automatically prune database information that may no longer be required, go to Settings | Data Store Configuration | App Data Store Maintenance. App Data Store Maintenance allows ERPM to prune old data based on the settings you define. The settings cover:

  • Operation Messages - logging collected during a job run such as items propagated to, which systems were changed. The existence of these messages do not affect the status of the job and knowing whether or not a job succeeded.
  • Web Audit Logs - these are the logs for every operation performed within the website or web service. These should follow typical security data retention guidelines for your company.
  • Obsolete Jobs - gets rid of jobs that will never run again, like password re-randomization jobs following recovery or jobs set with a schedule of "immediately". These could be pruned on a monthly basis or at least as frequently as you audit the product.
  • Item References - these are various reference items such as account usage for systems, random permissions for accounts that no longer exist and so on. These could be pruned on a weekly basis or more frequently depending on the item.
  • Remove Auto Created Indexes - ERPM can auto-create indexes to maintain performance over time. If the index is unused it could be deleted. This item should not be enabled if you maintain your DB indexes directly on the SQL server (rather than within ERPM).
  • App Operation Metrics - ERPM can track statistics about DB connection and query times as well as website and web service and deferred/zone processor responsiveness. If App Operation Metrics are being captured, they only need to be kept around for as long as you need to reference the data as they are only used for reporting. If App Operation Metrics are not being captured, this setting may be ignored.
  • Management Set Updates - this information is captured every time a management set updates. Generally, this is not security sensitive information. It captures the added and removed systems along with any other errors and operational messages. Typically, this information could be pruned at a more frequent interval than management set updates occur. For example, if the management sets auto-update every 30 days, you might keep the data for 29 days. This information can add many rows to a database for very large management sets.
  • Network Scan Data - this is the information performed during a network scan prior to performing a management set update. In this case, these are scans beginning with an IP scan to find SNMP, SSH, Telnet, SQL Browser, TNS objects, etc. This scan data is typically not configured when scanning AD, LDAP, DB or static lists. This information is useful for helping to generate future queries for new management sets or for examining the XML data to see what was actually captured. This is not information that is easily accessed except the management set properties and is not used again once the scan is performed. Typically, this information could be pruned at a more frequent interval than management set updates occur. For example, if the management sets auto-update every 30 days, you might keep the data for 29 days. This information can add many rows to a database for very large management sets.

App Data Store Maintenance can be scheduled to run or run interactively. Once the settings are defined for the above items, set this job to run daily or weekly to keep your database data in tune. Note that more frequent modification of the data will also necessitate re-indexing and regeneration of statistics.

Pruning this data will also impact the amount of data captured for the built-in compliance reports. Less data means faster captures and report generations.

Data Store Tuning

Under Settings | Data Store Configuration are more controls for:

  • Auto Index Tuning - this calls the SQL server to see what indexes it recommends based on the statistics it keeps. Check this once a month or when performance visibly degrades.
  • Index Defragmentation - this calls the SQL server to check how fragmented the indexes on the tables are. Fragmented databases can slow down operations. Check this once a month or when performance visibly degrades.
  • Generate Stats Fullscan - forces SQL server to regenerate usage statistics on the tables to improve its utilization and access plans. Run this once a month or when performance visibly degrades.

All three of the above items can be run from the ERPM console manually or can be performed directly in the ERPM database.

Data Store Backup

Backup your database frequently. It contains the keys to your kingdom. Depending on the deployment you may have multiple online copies of the database managed through availability groups, mirrors, replication, log shipping, clustering, etc. You may even have further redundancy provided by your VM infrastructure and back-end SAN. These infrastructure elements will impact the need to backup more frequently if they are not present. Typical customers perform a full backup nightly.

When performing your backups, don't forget to prune the transactions logs as they will grow at a much faster rate than the actual ERPM data store! It has been seen where a 260MB database has a 48GB transaction log when that transaction log is not maintained.

General ERPM Host System Maintenance

ERPM runs on Windows and uses Windows infrastructure such as IIS and MS SQL Server. These systems should all be maintained regularly as well per Microsoft and your company's guidelines.

Patching

Typically, Microsoft patches and service packs do not impact ERPM. As such, patch as frequently as your policies mandate.

Performance Metrics

  • CPU: ERPM is multi threaded. If all CPUs (and there needs to be multiple CPUs) are constantly above 90%, add more CPU cores or upgrade the host running ERPM. Possibly examine offloading components (like the website, web service, zone processors) to alternative hosts. It is best to check this metric over time and during periods of user action or when all your jobs are queued to run. If the CPU goes above 95%, ERPM will hold/pend processing until more CPU resources become available so as to not starve the system.
  • Disk: ERPM and its web site/service is not typically disk bound. However, the database most definitely is. If the value of Avg. Disk Queue Length exceeds twice the number of spindles, then you are likely developing a bottleneck. With a volume set, a queue that is never shorter than the number of active physical disks indicates that you are developing a bottleneck. This should be tracked on the ERPM host and physical host if using a VM. An average high disk queue on the database will bring ERPM to a crawl and possibly cause database timeouts resulting in failed operations. It is best to check this metric over time and during periods of user action or when all your jobs are queued to run.

ERPM Data Logs

ERPM automatically logs everything it does to text logs. These logs are typically kept at %programdata%\lieberman\pwc\version 4.x\.

Main logging options can be controlled from the console at Settings | Logging Options.

  • PWCLog.txt - the main program log. Anything done in the console, interactively, is logged here. Clear from the console anytime.
  • PWCJobLog.txt - the deferred processor log. Any launching of jobs is logged here. Clear from the console anytime.
  • RouletteWebApplicationLog.txt - website log. Basic operations and website errors are logged here. Clear from the console anytime if website is local or by stopping the website COM application and deleting the log if on a remote server.
  • RouletteAppServiceSupport.txt - web service log. Basic operations and web service errors are logged here. Clear from the console anytime if web service is local or by stopping the web service COM application and deleting the log if on a remote server.
  • JLxxx.txt - job logs. Anything run by the deferred processor is logged to these job logs. XX is the job ID. Can safely be deleted anytime.
  • \Diagnostics\AppPerfControl -{Component}-{Date}.txt - kept when diagnostic logging is enabled. Not typically recommended to enable. Use when troubleshooting. Can safely be deleted anytime.
  • \Extension{ExtName}\Extension_{ExtName}-{Date}.txt - created when using CLR extension components. Can safely be deleted anytime.
  • \Archive\{LogName}-Archived-{Date}.txt - when log archiving is enabled, these compressed files are created from the live logs and the live logs are truncated. Can safely be deleted anytime.

Text log files are for review and typically only used in troubleshooting or periodic review. If your ERPM review cycle is once a month, review the text logs to check for any errors potentially not visible in the ERPM console (like errors happening between maintenance intervals where the job ran multiple times) and clear the logs by hand.

Logging is also optional for all components. Some customers, though not many, disable logging text logging until errors occur. Then text logging is enabled. This is not generally recommended though it can save disk space and disk I/O. To disable logging, in the console, go to Settings | Logging Options and clear the checkbox next to Enable Logging for the specific component.


Applies To

  • Enterprise Random Password Manager (ERPM)
Was this article helpful?
0 out of 1 found this helpful

Comments

Powered by Zendesk