Table of content

Description

At the TF-NOC meeting during fall 2010 we agreed to collect information about what the TF-NOC participants thought would be interesting to share from their organisation, as well as information they would be interested in receiving from other organisations. The aim is to summarize that collected information on this page.

Summary of interesting topics

BELNET
What BELNET would like to present:

• lessons learned from outsourced incident & change management
• approach & tools: how the different departments & the NOC's work together
• scripting & naming convention: making the network more consistent for the NOC
• introduction of our customers to the outsourced NOC's: how to change their attitude
• keeping the quality of the NOC's in check over time: means & experiences

What we would like to hear from other participants:

• procedures: presentation of the NOC's current procedures & who is responsible for maintaining them
• authentication: how do you verify the caller / e-mail sender
• reponsibility of your NOC: how far does it go?
• good & bad experiences with outsourcing
• distributed NOC's (as opposed to a NOC on a central location): pro's & con's

DANTE
Other basic information

• Organization (tiers, and which tiers are in-house or outsourced);
• Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting);
• Hours of operation (office and on-call);
• Network type (IP only, hybrid, switched/transmission only, LAN only);
• Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc

Of most interest to learn from others

• Main Monitoring systems and alarm thresholds;
• Target repair times;
• Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists
• Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers
• Core equipment vendor/make, approx numbers;

CARNet
We would like to include in our presentation topics about:

• CARNet network, users & Core/Access nodes - how to treat different users and different parts of network
• CARNet NOC - organization, tiers, team operating hours, knowledge sharing
• NMS - our tool functionalities / developement
• prodecudes and documentations - what should be strictly put "on paper" (level of knowledge and procedures) from our experience ...

Topics about what we would like to learn from others are:

• NOC key performance indexes - how to measure/to improve ?
• Knowledge management and NOC staff training
• NMS - tools (functionalities)
• NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours
• How to work proactively in NOC (active/passive monitoring)

PIONIER
Five topics PSNC would like to present about NOC:

• networks PIONIER (NREN) and POZMAN - technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions
• procedures used in our NOC
• our TTS - public part, experience, missing functions
• NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC
• Experiences in joint management application design & deployment with commercial company

Five topics (or questions) that PSNC is interested in knowing how other NOCs handle:

• how to minimize users calls (public available FAQ? tools? documentation?)
• reports - what? how often?
• data and tools integration - how? recommended tools?
• serving users calls - how to automate it, distributing calls among different NOC administrators
• (D-)DoS - how to detect network attacks

ARNES
ARNES could present:

• Pros and contras for dedicated NOC personnel
• NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES would like to hear from others about:

• NOC staff job description
• NOC assesment and KPI – measuring NOC performance
• NOC to user communication (communication with more or less experienced users, how to keep user contact info up-to-date, personalized web interface for NOC to user communication)
• NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
• How many customers do you have? (depends on if you are counting persons, networks, hosts,..)
• What kind of agreements do you have with service providers and customers (SLA,..)? Without financial info, but just roughly descriptions of responsibilities.

NORDUnet
Things NORDUnet could present:

• Troubleticketing system, experiences from building our own.
• Managing user interfaces (using people outside the NOC to aggregate user requests)
• Using external community tools (letting the community answer questions instead of the NOC).
• NOC organisation (staffing, internal escalations, handover and schedule)
• Reaching your network from anywhere (out-of-band, in-band and VPN)

Things NORDUnet would like to hear about from others:

• How is knowledge dissemination handled between groups and new NOC employees?
• How is monitoring and alarm correlation handled between different systems?
• How are new services and users provisioned?
• What kind of routines are used in regards to change management (e.g. in planned maintenance)?
• What other services than the network does your NOC operate and how do they differentiate from the network service?

GNOC at Indiana University
Five topics to introduce the Global Research NOC at Indiana University:

• Global Research NOC Service Desk<http://globalnoc.iu.edu/tour.html> provides 24x7x365 technical call center support, trouble ticket management, and workflow support. The service desk is housed in a state-of-the-art call center in Indiana University's Informatics and Communications Technology Complex (ICTC) on Indiana University's Indianapolis campus. This call center features a fully customizable 30-foot screen used to monitor and troubleshoot the current health of the various networks supported. We provide NOC support for the most advanced research and education networks in the country. http://globalnoc.iu.edu/tour.html
• Willingness to assist network customers with other services and requests outside of traditional front-line network operations, such as network redesign planning and utilizing our tool set.
• A network knowledgeable tier-one Service Desk that can assess network issues intelligently and can provide engineering with helpful information as they begin troubleshooting.
• Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html
• International networking efforts

Five topics the GlobalNOC at Indiana University would like to know about other NOC's:

• What methodologies and tools do you use to understand complex cross-network relationships and dependencies?
• How do you organize and update your internal documentation environment for easy access and search, as well as providing clear and accurate information?
• How do you integrate the different groups within your NOC (Service Desk-Tier One, Engineering-Tier Two & Three, Systems Engineering, etc) so each has clear expectations of what each group does, with all working as a unified team?
• With R&E networking moving towards having more regional connectedness between smaller networks, RON's, universities, etc, are you planning for a new phase in networking where specific operational differences between NOC's may negatively affect the reliability of the services provided? (for example: different response levels from hardware and fiber vendors, vastly different maintenance windows, NOC's not being 24x7, etc.)
• What type of training methodologies do you use to bring new employees up-to-speed quickly so they can start making a valuable contribution to the operation?

SWITCH
The topics SWITCH could tell about:

• network, equipment, services, customers
• management of fibers - with segments from different providers
• staff
• outsourced services
• work flows
• tools
• experiences and things we like to improve

The topics SWITCH is interested to know from other NOCs:
The same points like above with special focus on:

• incident handling
• tools and especially
• tools to support the distributed handling of incidents

GRNET
GRNET is keen in presenting one of the topics trailed below:

• Efficient administration of many Linux based servers,
• Providing VPS service,
• Network database (GRNnet-DBII),
• Visualisation tools,

On the other hand, we would like to hear how our colleagues tackle the followings:

• NOC key performance indexes,
• Automated SLA monitoring tools,
• Network monitoring tools (may be different than the ones above),
• operation of CERT team and security policies

RedIRIS
Five topics to introduce RedIRIS NOC:

• Type of network. Equipment and type of lines.
• Out of band network.
• Trouble ticket system.
• SLAs with vendors and providers and type of contact.
• Organisation of our NOC.

Five topics RedIRIS would like to know about other NOCs:

• Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
• Monitoring tools.
• Acquiring, management and operation of dark fibre. Procedures. Optical NOC differenciated from IP NOC?
• Services managed by the NOC.
• Network database and integration with other monitoring tools.

FCCN
Five topics to introduce FCCN's NOC:

• Deployed network (Optical/Ethernet/IP)
• Out-of-band systems (MRV using GSM/GPRS)
• Service Desk & Automated reporting (per member)
• Deployed probes network (QoSMetrics solution)
• Past horror stories: What you can try to avoid & Murphy's Law.

Five topics FCCN would like to know about other NOCs:

• How is the 24x7x365 NOC service guaranteed?
• Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
• Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
• Do you have/use a fixed maintenance window?
• How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.

CERN
CERN would be glad to present from the following list of topics:

• Transforming 2 layer network support into 3 layer support with outsourcing
• Integration of network and telecom support
• Creation of a knowledge repository for network/telecom support
• Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)
• The challenge of using multiple ticketing systems

CERN would be interested in hearing about:

• Roles and responsibilities (structure, who does what)
• Relation and communication between the network operations team and other internal entities (design, deployment)
• How operation is done outside working hours including weekends
• How common experience is handled: specialized vs. universal team members
• How procedures are created and stored

RESTENA
Five topics to introduce RESTENA's NOC:

• Infrastructure building and operations (dark fiber, net devices,server-rooms, POPs, servers)
• An "all-in-one" NOC: structure/organization/tools
• 24x7x365 service support
• Coverage of user support (helpdesk)
• Services operated (from lambdas to large scale e-mail accounts)

Five topics RESTENA would like to know about other NOCs:

• User (customer) information exchange: channels and content
• NOC activities/coverage and staff (network, services, community details)
• Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
• Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
• NOC awareness and identification: internally and among user community

SURFnet
Items we from SURFnet can present our experiences

• acquring and managing dark fiber
• managing an outsourced NOC
• procedures - the good and bad things

Things we would like to hear about from other NOC's:

• how to inform/communicate with customers
• branding of an outsourced NOC
• ticketing systems
• CMDB experiences

CESCA
And these are topics that we would like to introduce about our NOC:

• Virtualization of the network.
• NOC tools and how to adapt them to our needs (development).
• Users management. How to approach our users, depending on the action (change, incident, request,..).
• Survey for our services, what to ask and how?
• Out-of-band access to our network.

These are the topics that we would want to know from other NOCs:

• Ticketing Systems, how to optimize their use.
• Integration of diferent tools.
• Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
• Escalation of incidents. How many levels?
• Documentation: what can and can not be documented.

HEAnet
5 important bullet points introducing our NOC(s)

• Heuristic Failure analysis (how we use heuristics to determine SLA calculations and where they are and are not appropriate.) (Also interested in discovering if others are doing this.)
• Ticketing, including automatic ticket generation for incidents.
• Contract and supplier management - how HEAnet does it and how well it works (or doesn't work.)
• Reporting to your clients. How do NRENs report to clients? Periodical and Real-time reporting. On demand reporting.
• 24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?

5 questions what you want to know about other NOCs

• How are staff hired for the NOC? Skillsets? Interview processes?
• Standards compliance and process adoption. Have other NOCs adopted ISO9000? ITIL? PRINCE2? APM? ISO20000? Six Sigma? Other standards?
• What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
• How do NOCs source support and SLAs? What about penalty clauses? What's reasonable, what works and how are they enforced?
• How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?

Interesting topics mapped to the work items

Taxonomy of NOC organisations, internal processes, workflows

NOC front-end development and improvement

NOC tools, interworking/interfacing issues, and automation

Efficient communication/collaboration tools and practices

Facilitate the collection/creation of best practice documents

Investigate and liaise with other communities

Flash presentation template

Purpose

The purpose of the flash presentations going to be held at the TF-NOC meeting in February 2011 is the following:
To get an overview of how other NOCs participating in TF-NOC work.
To give some food for thoughts to what is interesting to include in the upcoming TF-NOC survey.
To give some answers to questions that some NOCs are interested to hear how others are handling.

Topics

Slide 1 - Network

• What infrastructure have your organization deployed?
  (e.g. fiber + equipment)
• What services are your organization offering and maintaining?
• Which tools are you using to manage/monitor the network?

Slide 2 - NOC structure

• Which roles and coverage and responsibility does your staff have?
  (e.g. dispatchers or experts, 24/7 or daytime)
• How is your NOC organized?
  (e.g. centralized, distributed or out-sourced)
• How does the NOC work with different kinds of tools?
  (e.g. tool integration)

Slide 3 - Front end

• What types of users are using your network and services?
• How does the SLAs or agreements you have with your customers look like?
• Which tools are used to communicate with and keep track of users?

Slide 4 - Inter NOC communication

• How do your NOC communicate internally and with other groups in your organization?
• How does your NOC communicate NOCs outside your organization?
  (e.g. provider and/or customer NOCs)
• Which tools are used to facilitate inter-NOC communication?

Slide 5 - Documentation

• What information do your NOC document?
• Which tools are used to create and update documentation?
• Do you have any Best Practice Documents available to share?