DRAFT
CETI
Operations and Support Plan
Executive Summary
11 December 1997
Prepared For:
CETI Infrastructure Committee
Prepared By:
Operations and Support Team
Dennis Edstrom, Co-Leader
Richard Sol, Co-Leader
Larry Harmon
Marty Knizner
Kevin Ralston
Jeff Shelmire
Carl Vigil
DRAFT
EXECUTIVE SUMMARY
OPERATIONS & SUPPORT
1.0 Introduction
The prime objective of CETI Operations and Support will be to offer services on new products and technologies purchased through the CETI Partnership, and all equipment installed as part of the infrastructure buildout. CETI will also be postured to provide service on those products and technologies currently in place. The intention is to offer CSU a single source solution that will provide substantial cost and quality improvements over current support options.
In order to accomplish this goal, CETI must establish an operational organization capable of providing service to any product at any campus location within the state. A basic strategy of providing quality service is to utilize qualified personnel. For all areas that require CETI support, CETI will utilize current CSU support and call center personnel that are qualified and positioned to meet CSU service requirements. There may also be locations where the support of certain systems or technologies creates a burden on local CSU staff. For these situations, CETI will offer service options that may allow the CSU to better utilize their resources.
The CETI Operations Management Structure must be evolutionary to meet the anticipated needs and requirements of the CSU customer base in which 23 distinct and unique campuses function in various educational and community support missions. It must also be able to deliver exceptional customer satisfaction, while achieving the ITS vision of "anytime, anywhere access" for a growing CSU customer base.
Because of this diversity, the intention of the CETI operations structure will be to provide an infrastructure that will offer a single source solution option for all CSU campuses to utilize. It is understood that some campuses may elect to perform some services themselves. CETI sees this as a strength of this unique partnership. The concept is to establish a presence and understanding that whatever services are required, CETI will be there to assist and will provide a point of contact (POC) for authorized users to utilize.
It is anticipated this organization will grow significantly in size as the network infrastructure is installed and brought on line, and CETI sponsored products are sold to the CSU community. Individual campus service requests could also have an impact on the staffing requirements. It is proposed that this organization reside in a location that will have adequate resources to accommodate current and future needs. Consideration should also be given to collocation of this organization with the primary Help Desk, and Network Operations and Control Center. This will provide economies of scale, enhanced service delivery and improved problem escalation.
The following chart profiles the proposed CETI operations structure:
2.1 CETI Support Services (CSS) Profile
CETI has chosen to implement a single point of contact support center that will be the focal point for all service requests from CSU faculty, staff, and students. The CSS will log and track trouble tickets until resolution and assess the appropriate level of support to complete the request. CSS support will be provided in two levels, and will include hardware support, operating systems support, application software support, and network support. The CSS will assume responsibility for follow-up on all incidents opened with the Help Desk until resolution is completed and verified.
Benefits of this approach include:
Definition of Level 1 Support. Level 1 support will be provided by CSS Support Representatives. They will be the primary interface with end users and will verify user/equipment entitlement. They are capable of providing end users with user assistance, basic hardware and software troubleshooting, setting priorities, escalating to the appropriate second level support, tracking, and providing status to accomplish the desired results.
Definition of Level 2 Support. Level 2 support will be provided by Support Analysts located in the CSS. They will have advanced support capabilities to isolate and resolve problems remotely. They will determine the need for additional support including advanced technical support, on-site personnel, outside contractors, repair depots, or network technical support personnel.
Definition of Level 3 Support. Level 3 support will be the coordination of services provided by Support Analysts supporting specific applications that they have either created or specialized in. This would also include coordinating support from Other Equipment Manufacturer’s (OEM’s) and/or other vendor’s.
CSS will maintain a database of all end users and/or equipment entitled to the services provided by CETI. CSS will utilize this database to verify entitlements, manage support requests, to provide monthly reports, and to build a knowledge base. The sources used to maintain this database will come from the CSU and from the Unified Messaging initiative.
2.2 Services Profile
The Services organization will oversee and direct the delivery of all service offerings performed on the campuses. This function has been initially divided into two regionalized entities to provide improved focus to the CSU customer based on geographical diversity. Experience has shown that by decentralizing field management efforts, a services organization is able to render the most timely and effective support to the customer. Additional regions could be established due to increased level of effort, or a need to provide a more focused field structure to service unique requirements, such as the "coastal" and "valley" campuses.
The manager of these organizations will also act as the liaison/point of contact for all CETI/CSU requirements, concerns, and service delivery issues. They will have the knowledge and flexibility to provide rapid assessment and resolution of any unique campus equipment problems, and can overcome those issues related to the unique geography of their areas and the individual campuses. Regional Managers control all of the resources necessary to assure prompt response to user needs. Personnel, material, documentation, and technical support can be assigned to any campus or FSR within a region to promptly repair a system failure and/or to assist a user with the resolution of any unusual problem.
It is anticipated that many of the larger campuses will also have on-site managers or supervisors to oversee the daily activities performed at their respective locations. Field Service Representatives (FSR) would provide Remedial Maintenance (RM), Preventive Maintenance (PM), and Installation services for all equipment entitled under the CETI/CSU agreement. Field Service personnel would be a combination of on-site (residential) and on-call. Qualified CSU personnel who are currently performing this function would be utilized wherever possible. New FSRs will be hired where necessary to meet the service requirements of the CSU.
All Services management and FSRs will be equipped with pagers and cellular phones to ensure instant availability for information flow and problem escalation. The Services organization works in concert with the CSS and other support organizations to achieve the established repair and restoral requirements. Managers and Supervisors will visit with campus representatives to discuss service performance and any issues of concern for CSU personnel.
2.3 Network Operations Profile
The Network Operations and Control Centers (NOCC) will be structured to provide the management and performance needed to support and monitor current and the future CSU Inter-Campus WAN, the Intra-Campus LANs, and the network components and integrity within each of the buildings supported by the network. Systems management support activities shall include network administration, system fault management, system performance management, system configuration management, and network security management (as applicable).
It is recommended that there be two NOCCs to provide uninterrupted support capabilities during any disaster or crisis situation. It is anticipated that one NOCC continue operations from the current 4CNet NOC at WestEd, and the second be located at one of the HUB sites in Northern California. This will offer the CSU infrastructure the highest level of survivability and support. Both NOCCs will have the infrastructure and capability to provide all required monitoring, diagnostics, and technical support for the CSU network.
Network Operations will have a Network Engineering staff capable of assessing current 4CNet and campus network performance and requirements, and will work closely with the Infrastructure Buildout initiative to evaluate and integrate the future needs of the CSU network. The total CSU network will benefit from the capabilities of this organization to monitor, evaluate/assess, and optimize the performance of the network and all its communications components. The optimization and its implementation will be assured through the use of state of the art hardware and software tools, such as Solstice Enterprise Manager, HP Openview DM platform, CISCO Works, Synoptic Optivity, various database tools, and network sniffers.
The NOCC will also act as the interface to circuit providers such as GTE, AT&T, Sprint, and MCI. All actions will be tracked through the CETI Support Services Help Desk.
This function will focus on the training requirements of the Operations and Support staff. Proficiency and effectiveness in the performance of support activities will be monitored against industry metrics to determine when and what training programs are required.
2.5Implementation and Planning Profile
This group will be the focal point for all activities related to the ongoing CSU requirement to install, deinstall, upgrade, integrate, and relocate desktop hardware, network equipment, operating systems, and software applications. As required and tasked by CSU, CETI will plan and implement these changes for currently installed systems and new equipment required to meet CSU objectives. This will include the installation and implementation efforts required to meet the system refresh as required by the agreement.
The CSU infrastructure will provide the basis for meeting the short and long-term objectives of the university. Technology advancements may continue to offer significant improvements to the methods for delivering education and achieving a leadership role in community relations. This group will be responsible for working closely with CSU IT personnel to determine the needs of the campuses and seeking the most effective means to deliver the product. They will also keep abreast of technology and recommend products and services that could help CSU achieve their objectives. When it is determined that the CSU has interest in new technology, this organization will plan, integrate, test, and implement the new technology. This group will also provide input to CSU regarding refresh options.
CETI may be asked by CSU or individual campuses to accept service requirements that are under contract to an outside vendor. This organization will be responsible reviewing current contracts and managing them to the best interests of CSU service delivery, and CETI business objectives. They will also work with other departments and CSU campuses to develop new subcontracts where it is advantageous to the partnership. This group will have responsibility to administer, track, and monitor the performance of these contracts during the established period of performance.
The CETI organization will develop policies and processes that will be compliant with ISO 9000 quality standards as well as ISO 14000 Environmental Standards. ISO is the International Organization for Standardization and its objective is to promote the development of standards, testing, and certification in order to encourage the trade of goods and services while establishing and maintaining sound environmental practices.
Benefits from quality processes include:
Achieving superior quality in performance will be one of the first major challenges of the CETI organization. This group will comprise the management, administration and staff necessary for ensuring quality processes are met and maintained.
3.0 Service Offerings
As a baseline for support expectations, the following service offerings are defined as products of CETI. The tasks listed are intended to encompass all possible activities related to the agreement. Depending on CSU requirements, each of these services can be made available on a 7X24 basis.
Help Desk: CETI Support Services (CSS) will provide user assistance, administrative functions, troubleshooting, and service coordination for:
Remedial Maintenance: Remedial maintenance applies to malfunctioning resources, including hardware and software in any segment of the CSU network that a user has requested service from the CSS Help Desk. This includes determining why such malfunction occurred and either repairing or replacing it so that full functionality is restored. This level of maintenance is applicable to all products covered by CETI.
Preventive Maintenance: Preventive maintenance is regularly scheduled activities to assure the correct operation of both hardware and software items within the environment in which they are normally expected to operate. Preventive maintenance is to be performed at times coordinated between the CSU user and CETI. It will also be in accordance with established vendor (OEM) published guidance if they exist.
Installation Services: This category includes software upgrades and package enhancements which may be part of a preestablished agreement, as well as physical relocation, installation, deinstallation, and expansion of systems or components included in the CETI/CSU agreement.
On-Site (residential) Service: Residential maintenance will be provided to all campuses with a level of effort that justifies on-site support during established Principle Periods of Maintenance (PPM). This service can be delivered to both the campus and network user communities. From the campus perspective the on-site service may be desirable for large telecom systems or installed desktop base, whereas the network user may select the option for an NOCC or some other lesser network management facility. Maintenance performed will include remedial, preventive, and installation services as required. FSRs will be dedicated to service the requirements of the campus to which they are assigned.
On-Call Service: On-call maintenance will include preventive, remedial, and installation services that will be performed during established Principle Periods of Maintenance (PPM) at those campuses without on-site FSRs, and Outside Principal Periods of Maintenance (OPPM) at all campuses. On-call maintenance is initiated by the receipt of a call at the CSS Help desk from an end user. If determined by the Support Representative (SR) that on-site service is required, the call will go to the Services Organization for dispatch of the appropriate FSR.
Engineering: A structured approach to the planning and design of systems within the CSU network. This support will include the design engineering of inter-campus network, intra- campus LANs, Telephony communications, and any other traffic or performance level requirements. Based on the individual demand this may be included as part of a package or may be a "for fee" arrangement
Depot Maintenance: Depot maintenance is a full repair service plan performed at strategically located Depot Repair Facilities (DRF) throughout the CSU. They will be equipped to provide "Carry In", "Over the Counter", and "mail-around" depot maintenance services for equipment covered by the CETI agreement.
Network Operations and Control: 7X24 support to monitor and evaluate the performance of the CSU network(s), and LAN and WAN components. This service will be provided 365 days/year at one or both NOCCs to ensure CSU network availability metrics are met. Services to include monitoring, analyzing, tracking, and assessing the performance of the backbone network elements, routers, bridges, switches, and associated telecommunications components.
Network Administration: Support providing on-site and/or remote administration of the network’s physical and logical resources: users, peripherals, and transmission media. On-site Network administration at the campus level will be based on the level of effort required for each campus. These services would be performed under the direction of the responsible CETI NOCC and could be performed by CETI personnel or by CSU Network Administrators that are currently performing these tasks. Services shall include adding and removing users; assigning access rights to network resources; making application and peripheral resources available to users; providing periodic network backup; and monitoring network capacity and resource usage in order to plan and carry out expansion.
System Fault Management: Support service to provide on-site and/or remote fault management including detection, isolation, and correction of irregularities in the CSU network and other system component operations. Services provided will include monitoring, analyzing, tracking, and assessing the overall and detailed performance of hardware components in respect to fault management. The NOCC will also detect, diagnose, track, and correct major and minor alarms, faults, troubles, outages, and performance deterioration.
System Performance Management: Support service that provides on-site and/or remote capability to monitor and evaluate the performance of the system, network and layer entities. Services include monitoring, analyzing, tracking, and assessing the overall performance of hardware components, communications facilities, interfaces of the Inter-Campus WAN, Intra-Campus and Intra-Building LANs, voice, and video teleconferencing centers to ensure the network is meeting established performance standards.
System Configuration Management: CETI will have responsibility to provide on-site and/or remote support for configuration control of the network, system, or layer entities. The NOCC will generate and conduct configuration management for various hardware, software, and service components which constitute the Inter-Campus WAN, Intra-Campus and Intra-Building LANs, voice, and video teleconferencing centers. This also includes tracking hardware and software components located at workstations, servers, bridges, routers, hubs, and data transport facilities and generate reports as required.
CETI has a responsibility to deliver timely service on all products covered by the partnership. This service response is intended to provide all CSU users with the ability to perform in their specific functions and meet the vision of "anytime, anywhere access". The following chart depicts some of the general services provided as a part of this agreement. These service delivery metrics are the targets that CETI intends to drive toward. We will be monitoring these metrics and fine tuning our service offerings throughout the implementation phase of the CETI initiative.
Service Delivery Metrics
| Network Support | Performance Objective | 100% Availability |
|
2 Hours on Site | 24 Hours/365 Days/Year |
|
4 Hours | 24 Hours/365 Days/Year |
| Desktop Support | Performance Objective | 98% Availability |
|
4 Hours on Site | 9 Hours/Monday-Friday |
|
4 Hours | 9 Hours/Monday-Friday |
|
3 Day Turnaround | 9 Hours/Monday-Friday |
| Recall Rate | 80% Problems First Call | No Recurrence of Failure |
| Help Desk | Performance Objective | 100% Availability |
|
30 Seconds | 24 Hours/365 Days/Year |
|
60 Minutes | 100% of the Time |
|
45 Minutes | 90% of the Time |
and Close Ticket |
1 Business Day | 90% of the Time |