OpenVMS Guide to System Security

November 1996

Revision/Update Information: This manual supersedes the OpenVMS Guide to System Security, Version 6.2

Software Version: OpenVMS Alpha Version 7.1
OpenVMS VAX Version 7.1

Digital Equipment Corporation Maynard, Massachusetts

Digital Equipment Corporation makes no representations that the use of its products in the manner described in this publication will not infringe on existing or future patent rights, nor do the descriptions contained in this publication imply the granting of licenses to make, use, or sell equipment or software in accordance with the description.

Possession, use, or copying of the software described in this publication is authorized only pursuant to a valid written license from Digital or an authorized sublicensor.

Digital conducts its business in a manner that conserves the environment and protects the safety and health of its employees, customers, and the community.

© Digital Equipment Corporation 1996. All rights reserved.

The following are trademarks of Digital Equipment Corporation: Bookreader, DECamds, DECdns, DECdtm, DECnet, DEC Notes, DECserver, DECwindows, Digital, OpenVMS, OpenVMS Cluster, VAX, VAX DOCUMENT, VAXcluster, VMS, VMScluster, and the DIGITAL logo.

Display POSTSCRIPT is a registered trademark of Adobe Systems Incorporated.

Motif is a registered trademark of the Open Software Foundation, Inc.

All other trademarks and registered trademarks are the property of their respective holders.

ZK6346

The OpenVMS documentation set is available on CD-ROM.

Preface

Intended Audience

This guide is designed for users and for administrators responsible for protecting operating systems from tampering, observation, or theft of services by unauthorized users. The term security administrator is used in this guide to refer to the person or persons responsible for system security.

Document Structure

This guide contains the following information:

• Part 1: Overview
  Gives security administrators an overview of security issues, conceptual design features, and security features specific to OpenVMS systems.
  • Chapter 1 discusses levels of security requirements and describes three sources of security failures.
  • Chapter 2 introduces the reference monitor concept of security design and provides an overview of the operating system's security features.
• Part 2: Security for the User
  Describes security actions and features for the general user.
  • Chapter 3 provides information for the general user about the login and logout processes and the responsible use of passwords.
  • Chapter 4 and Chapter 5 describe object protection features in detail.
• Part 3: Security for the System Administrator
  Describes security actions and features for the security administrator.
  • Chapter 6 describes the general tasks of a security administrator.
  • Chapter 7 describes methods of controlling system access.
  • Chapter 8 describes methods of controlling access to system data and resources.
  • Chapter 9 describes security-auditing features.
  • Chapter 10 describes how to recognize when a system is under attack and how to protect and defend your system.
  • Chapter 11 describes security-related actions specific to clustered systems, such as setting up common system files and synchronizing authorization data.
  • Chapter 12 describes security considerations for systems using networking.
  • Chapter 13 describes how to set up and manage protected subsystems.
  • Appendix A provides a summary of all the user privileges available on the operating system and describes who may need them.
  • Appendix B lists the protection codes and ownership that Digital provides for critical system files.
  • Appendix C describes how to operate OpenVMS systems in a Division C, Class 2 (C2) security environment.
  • Appendix D provides examples of security alarm messages.
• The Glossary provides definitions of security-related terms introduced in this guide.

Related Documents

The OpenVMS Guide to System Security assumes you are familiar with the reference material in the OpenVMS System Management Utilities Reference Manual pertaining to the following security-related utilities:

• Access control list editor (ACL editor)
• Accounting utility
• Audit Analysis utility
• Authorize utility
• Backup utility
• System Management (SYSMAN) utility

You might find helpful the amplified security information in the following manuals:

• OpenVMS DCL Dictionary
• OpenVMS System Manager's Manual
• OpenVMS Cluster Systems

Users with a specific interest in networking should be familiar with the DECnet for OpenVMS Networking Manual (Phase IV) or DECnet/OSI Network Management (Phase V).

For a complete list and description of the manuals in the OpenVMS documentation set, see the Overview of OpenVMS Documentation.

For additional information on the Open Systems Software Group (OSSG) products and services, access the Digital OpenVMS World Wide Web site. Use the following URL:

http://www.openvms.digital.com 

Reader's Comments

Digital welcomes your comments on this manual.

Print or edit the online form SYS$HELP:OPENVMSDOC_COMMENTS.TXT and send us your comments by:

Internet	openvmsdoc@zko.mts.dec.com
Fax	603 881-0120, Attention: OSSG Documentation, ZKO3-4/U08
Mail	OSSG Documentation Group, ZKO3-4/U08 110 Spit Brook Rd. Nashua, NH 03062-2698

How To Order Additional Documentation

Use the following table to order additional documentation or information. If you need help deciding which documentation best meets your needs, call 800-DIGITAL (800-344-4825).

Conventions

The name of the OpenVMS AXP operating system has been changed to the OpenVMS Alpha operating system. Any references to OpenVMS AXP or AXP are synonymous with OpenVMS Alpha or Alpha.

VMScluster systems are now referred to as OpenVMS Cluster systems. Unless otherwise specified, references to OpenVMS Clusters or clusters in this document are synonymous with VMSclusters.

In this manual, every use of DECwindows and DECwindows Motif refers to DECwindows Motif for OpenVMS software.

The parts of this guide are intended for different audiences, and the meaning of you differs accordingly:

• Part 1 addresses primarily security administrators although all users may find the concepts useful.
• Part 2 addresses general users.
• Part 3 addresses security administrators.

The following conventions are also used in this manual:

Ctrl/ x	A sequence such as Ctrl/ x indicates that you must hold down the key labeled Ctrl while you press another key or a pointing device button.
[Return]	In examples, a key name enclosed in a box indicates that you press a key on the keyboard. (In text, a key name is not enclosed in a box.)
...	Horizontal ellipsis points in examples indicate one of the following possibilities: Additional optional arguments in a statement have been omitted. The preceding item or items can be repeated one or more times. Additional parameters, values, or other information can be entered.
. . .	Vertical ellipsis points indicate the omission of items from a code example or command format; the items are omitted because they are not important to the topic being discussed.
( )	In command format descriptions, parentheses indicate that, if you choose more than one option, you must enclose the choices in parentheses.
[ ]	In command format descriptions, brackets indicate optional elements. You can choose one, none, or all of the options. (Brackets are not optional, however, in the syntax of a directory name in an OpenVMS file specification or in the syntax of a substring specification in an assignment statement.)
{ }	In command format descriptions, braces surround a required choice of options; you must choose one of the options listed.
text style	This text style represents the introduction of a new term or the name of an argument, an attribute, or a reason. This style is also used to show user input in Bookreader versions of the manual.
italic text	Italic text emphasizes important information and indicates complete titles of manuals and variables. Variables include information that varies in system messages (Internal error number), in command lines (/PRODUCER= name), and in command parameters in text (where device-name contains up to five alphanumeric characters).
UPPERCASE TEXT	Uppercase text indicates a command, the name of a routine, the name of a file, or the abbreviation for a system privilege.
-	A hyphen in code examples indicates that additional arguments to the request are provided on the line that follows.
numbers	All numbers in text are assumed to be decimal unless otherwise noted. Nondecimal radixes---binary, octal, or hexadecimal---are explicitly indicated.

• Sources of security failures (Section 1.1)
• Levels of security requirements (Section 1.2)
• Reference monitor concept of security design (Section 2.1)
• Security features of the operating system (Section 2.2)

Effective operating system security measures help prevent unauthorized access and theft of computer time and any kind of sensitive information, such as marketing plans, formulas, or proprietary software. These measures can also protect equipment, software, and files from damage caused by tampering.

This chapter provides security administrators with an overview of security measures available with the operating system. Part 3 provides more specific information regarding site security policies, the tasks of security administrators, and methods of protecting site computer systems and resources.

1.1 Types of Computer Security Problems

On any system there can be two types of users: authorized and unauthorized. Any person authorized to use the computer system has the right to access the system and its resources according to the authorization criteria set up by the site security administrator. Usage criteria may include the time of day, types of logins, use of different resources like printers and terminals, and so on. Unauthorized users have no right to use the system at all or only at a given time of day, or they have no right to use certain system resources.

On a computer system, security breaches usually result from one of four types of actions:

• User irresponsibility refers to situations where the user purposely or accidentally causes some noticeable damage. One example would be a user who is authorized to access certain files making a copy of a key file to sell.
  There is little that an operating system can do to protect sites from this source of security failure. The problem frequently lies in application design deficiencies or inconsistent use of available controls by users and the security administrator. Sometimes the failure to enforce adequate environmental security unwittingly encourages this type of security problem.
  Even the best security system will fail if implemented inconsistently. This, along with the failure to motivate your users to observe good security practices, will make your system vulnerable to security failures caused by user irresponsibility. Chapter 3 discusses what users can do to help maintain system security.
• User probing refers to situations where a user exploits insufficiently protected parts of the system. Some users consider gaining access to a forbidden system area as an intellectual challenge, playing a game of user versus system. Although intentions may be harmless, theft of services is a crime. Users with more serious intent may seek confidential information, attempt embezzlement, or even destroy data by probing. Always treat user probing seriously.
  The system provides many security features to combat user probing. Based on security needs, the security administrator implements features on either a temporary or permanent basis. See Chapter 4 for information on protecting data and resources with protection codes and access control lists.
• User penetration refers to situations where the user breaks through security controls to gain access to the system. While the system has security features that make penetration extremely difficult, it is impossible to make any operating system completely impenetrable.
  A user who succeeds in penetrating a system is both skilled and malicious. Thus, penetration is the most serious and potentially dangerous type of security breach. With proper implementation of the OpenVMS security features, however, it is also the rarest security breach, requiring unusual skills and perseverance.
• Social engineering refers to situations in which an intruder gains access to a system not by technical means, but by deceiving users, operators, or administrators. Potential intruders may impersonate authorized users over the phone. Potential intruders may request information that gains them access to the system, such as telephone numbers or passwords, or they may request an unwitting operator to perform some action that compromises the security of the system.
  As the technical security features of operating systems have strengthened in recent years, social engineering has been a factor in a growing percentage of security incidents. Operator training, administrative procedures, and user awareness are all critical factors to ensure that access is not inadvertantly granted to unauthorized persons.

The following chapters explain how to avoid these problems:

• Chapter 8 explains how to augment the protection of system files and resources.
• Chapter 7 describes the intrusion detection system and how to set its parameters.
• Chapter 9 explains how to monitor system activity and be notified by malicious activity.
• Chapter 10 suggests how to handle system intrusions.
• Chapter 3 and Chapter 6 list topics to include in your site training programs.

Each site has unique security requirements. Some sites require only limited measures because they are able to tolerate some forms of unauthorized access with little adverse effect. At the other extreme are those sites that cannot tolerate even the slightest probing, such as strategic military defense centers. In between are many commercial sites, such as banks.

While there are many considerations in determining your security needs, the questions in Table 1-1 can get you started. Your answers can help determine the levels of your security needs. Also refer to Section 6.2 for a more specific example of site security requirements.

Table 1-1 Event Tolerance as a Measure of Security Requirements

Question: Could you tolerate the following event?	Level of Security Requirements Based on Toleration Responses
	Low	Medium	High
A user knowing the images being executed on your system	Y	Y	N
A user knowing the names of another user's files	Y	Y	N
A user accessing the file of another user in the group	Y	Y	N
An outsider knowing the name of the system just dialed into	Y	Y	N
A user copying files of other users	Y	N	N
A user reading another user's electronic mail	Y	N	N
A user writing data into another user's file	Y	N	N
A user deleting another user's file	Y	N	N
A user being able to read sections of a disk that might contain various old files	Y	N	N
A user consuming machine time and resources to perform unrelated or unauthorized work, possibly even playing games	Y	N	N

If you can tolerate most of the events listed, your security requirements are quite low. If your answers are mixed, your requirements are in the medium to high range. Generally, those sites that are most intolerant to the listed events have very high levels of security requirements.

When you review your site's security needs, do not confuse a weakness in site operations or recovery procedures as a security problem. Ensure that your operations policies are effective and consistent before evaluating your system security requirements.

1.3 Building a Secure System Environment

There are two sources of security problems outside the operating system domain: employee carelessness and facility vulnerability. If you have a careless or malicious employee or your facility is insecure, none of the security measures discussed in this guide will protect you from security breaches.

Most system penetration occurs through these environmental weaknesses. It is much easier to physically remove a small reel of tape than it is to break access protection codes or change file protection.

Digital strongly encourages you to stress environmental considerations as well as operating system protection when reviewing site security.

This book discusses operating system security measures. When deciding which of these measures to implement, it is important for you to assess site security needs realistically. While instituting adequate security for your site is essential, instituting more security than actually necessary is costly and time-consuming.

When deciding which security measures to apply to your system, remember the following:

• The most secure system is also the most difficult to use.
• Increasing security can increase costs in terms of slower access to data, slower machine operations, and slower system performance.
• More security measures require more personnel time.

The operating system provides the basic mechanisms to control access to the system and its data. It also provides monitoring tools to ensure that access is restricted to authorized users. However, many computer crimes are committed by authorized users with no violation of the operating system's security controls.

Therefore, the security of your operation depends on how you apply these security features and how you control your employees and your site. By first building appropriate supervisory controls into your application and designing your application with the goal of minimizing opportunities for abuse, you can then implement operating system and site security features and produce a less vulnerable environment. For an example of one organization's security plan, see Chapter 6.

If you require your system to meet the United States government rating of a C2 secure operating system, please refer to Appendix C in this manual.

If you need a higher level of computer security for your OpenVMS secure system, Digital offers SEVMS, which is the security enhanced version of OpenVMS that provides mandatory access controls to enforce a system-wide security policy.

SEVMS is a U.S. Department of Defense B1-rated secure operating system.

This chapter presents the concepts that guided the design and implementation of the security features and mechanisms incorporated into the operating system. The intent is to provide a framework for thinking about your total system security picture. Subsequent chapters present details about the security features and their use.

2.1 Structure of a Secure Operating System

In the late 1960s, a great deal of research and development was dedicated to the problem of achieving security in multiuser computer systems. Much of the development work involved attempts to find all the things that could go wrong with a system's security and then to correct those flaws one by one. It became apparent to the researchers that this process was ineffective; effective system security could result only from a basic model of the structure of a secure computer system. The reference monitor concept was proposed as such a model and gained wide acceptance.

2.1.1 Reference Monitor Concept

According to the reference monitor concept, a computer system can be depicted in terms of subjects, objects, an authorization database, an audit trail, and a reference monitor, as shown in Figure 2-1. The reference monitor is the control center that authenticates subjects and implements and enforces the security policy for every access to an object by a subject.

Figure 2-1 Reference Monitor

The following table describes the elements shown in Figure 2-1:

Item	Element	Description
1	Subjects	Active entities, such as user processes, that gain access to information on behalf of people.
2	Objects	Passive repositories of information to be protected, such as files.
3	Authorization database	Repository for the security attributes of subjects and objects. From these attributes, the reference monitor determines what kind of access (if any) is authorized.
4	Audit trail	Record of all security-relevant events, such as access attempts, successful or not.

2.1.2 How the Reference Monitor Enforces Security Rules

The reference monitor enforces the security policy by authorizing the creation of subjects, by granting subjects access to objects based on the information in a dynamic authorization database, and by recording events, as necessary, in the audit trail. In an ideal system, the reference monitor must meet the following three requirements:

• Mediate every attempt by a subject to gain access to an object
• Provide a tamperproof database and audit trail that are thoroughly protected from unauthorized observation and modification
• Remain a small, simple, and well-structured piece of software so that it is effective in enforcing security requirements

These are the requirements proposed for systems that are secure even against penetration. In such systems, the reference monitor is implemented by a security-related subset, or security kernel, of the operating system.

2.2 Implementation of the Reference Monitor

While the OpenVMS operating system does not implement the reference monitor as a security-related subset, or security kernel, its interface to users and system managers does mirror the basic structure dictated by the reference monitor concept. Experience shows that incorporating such a structure is the best way to build a system resistant to probing and to most attempts at penetration.

The following sections describe the OpenVMS operating system's implementation of the reference monitor model.

2.2.1 Subjects

Subjects are the users or user agents (the user processes) that access information and, in some cases, may be prevented from accessing information. Subjects can be interactive processes, network processes, or batch jobs, and:

• Must pass security controls
  • During process creation
  • During information access
• Require identification
  • User names
  • Passwords
  • User identification codes
  • Rights identifiers

When a user logs in to use the operating system interactively or when a batch or network job starts, the operating system creates a process that includes the identity of the user. That process gains access to information as the agent for the user, as described in Chapter 4.

Processes are vulnerable to security breaches while they are being created and while they are accessing information. The system manages process access to information by using its authorization data and internal mechanisms, such as hardware controls. Because process creation has many areas of security vulnerability, many operating security features concentrate on the area of process (or subject) creation.

  6346P.HTM
  OSSG Documentation
  22-NOV-1996 13:04:48.38

Copyright © Digital Equipment Corporation 1996. All Rights Reserved.

Legal