Wiki Markup |
---|
h2. Table of content |
...
Table of Contents |
---|
Description
...
{toc} h2. Description At the TF-NOC meeting during fall 2010 we agreed to collect information about what the TF-NOC participants thought would be interesting to share from their organisation, as well as information they would be interested in receiving from other organisations. The aim is to summarize that collected information on this page. |
...
h2. Summary of interesting topics |
...
BELNET
What BELNET would like to present:
- lessons learned from outsourced incident & change management
- approach & tools: how the different departments & the NOC's work together
- scripting & naming convention: making the network more consistent for the NOC
- introduction of our customers to the outsourced NOC's: how to change their attitude
- keeping the quality of the NOC's in check over time: means & experiences
What we would like to hear from other participants:
- procedures: presentation of the NOC's current procedures & who is responsible for maintaining them
- authentication: how do you verify the caller / e-mail sender
- reponsibility of your NOC: how far does it go?
- good & bad experiences with outsourcing
- distributed NOC's (as opposed to a NOC on a central location): pro's & con's
DANTE
Other basic information
- Organization (tiers, and which tiers are in-house or outsourced);
- Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting);
- Hours of operation (office and on-call);
- Network type (IP only, hybrid, switched/transmission only, LAN only);
- Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc
Of most interest to learn from others
- Main Monitoring systems and alarm thresholds;
- Target repair times;
- Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists
- Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers
- Core equipment vendor/make, approx numbers;
CARNet
We would like to include in our presentation topics about:
- CARNet network, users & Core/Access nodes - how to treat different users and different parts of network
- CARNet NOC - organization, tiers, team operating hours, knowledge sharing
- NMS - our tool functionalities / developement
- prodecudes and documentations - what should be strictly put "on paper" (level of knowledge and procedures) from our experience ...
Topics about what we would like to learn from others are:
- NOC key performance indexes - how to measure/to improve ?
- Knowledge management and NOC staff training
- NMS - tools (functionalities)
- NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours
- How to work proactively in NOC (active/passive monitoring)
PIONIER
Five topics PSNC would like to present about NOC:
- networks PIONIER (NREN) and POZMAN - technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions
- procedures used in our NOC
- our TTS - public part, experience, missing functions
- NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC
- Experiences in joint management application design & deployment with commercial company
Five topics (or questions) that PSNC is interested in knowing how other NOCs handle:
- how to minimize users calls (public available FAQ? tools? documentation?)
- reports - what? how often?
- data and tools integration - how? recommended tools?
- serving users calls - how to automate it, distributing calls among different NOC administrators
- (D-)DoS - how to detect network attacks
ARNES
ARNES could present:
- Pros and contras for dedicated NOC personnel
- NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
ARNES would like to hear from others about:
- NOC staff job description
- NOC assesment and KPI – measuring NOC performance
- NOC to user communication (communication with more or less experienced users, how to keep user contact info up-to-date, personalized web interface for NOC to user communication)
- NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
- How many customers do you have? (depends on if you are counting persons, networks, hosts,..)
- What kind of agreements do you have with service providers and customers (SLA,..)? Without financial info, but just roughly descriptions of responsibilities.
NORDUnet
Things NORDUnet could present:
- Troubleticketing system, experiences from building our own.
- Managing user interfaces (using people outside the NOC to aggregate user requests)
- Using external community tools (letting the community answer questions instead of the NOC).
- NOC organisation (staffing, internal escalations, handover and schedule)
- Reaching your network from anywhere (out-of-band, in-band and VPN)
Things NORDUnet would like to hear about from others:
- How is knowledge dissemination handled between groups and new NOC employees?
- How is monitoring and alarm correlation handled between different systems?
- How are new services and users provisioned?
- What kind of routines are used in regards to change management (e.g. in planned maintenance)?
- What other services than the network does your NOC operate and how do they differentiate from the network service?
GNOC at Indiana University
Five topics to introduce the Global Research NOC at Indiana University:
...
*BELNET* What BELNET would like to present: - lessons learned from outsourced incident & change management - approach & tools: how the different departments & the NOC's work together - scripting & naming convention: making the network more consistent for the NOC - introduction of our customers to the outsourced NOC's: how to change their attitude - keeping the quality of the NOC's in check over time: means & experiences What we would like to hear from other participants: - procedures: presentation of the NOC's current procedures & who is responsible for maintaining them - authentication: how do you verify the caller / e-mail sender - reponsibility of your NOC: how far does it go? - good & bad experiences with outsourcing - distributed NOC's (as opposed to a NOC on a central location): pro's & con's *DANTE* Other basic information - Organization (tiers, and which tiers are in-house or outsourced); - Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting); - Hours of operation (office and on-call); - Network type (IP only, hybrid, switched/transmission only, LAN only); - Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc Of most interest to learn from others - Main Monitoring systems and alarm thresholds; - Target repair times; - Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists - Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers - Core equipment vendor/make, approx numbers; *CARNet* We would like to include in our presentation topics about: - CARNet network, users & Core/Access nodes - how to treat different users and different parts of network - CARNet NOC - organization, tiers, team operating hours, knowledge sharing - NMS - our tool functionalities / developement - prodecudes and documentations - what should be strictly put "on paper" (level of knowledge and procedures) from our experience ... Topics about what we would like to learn from others are: - NOC key performance indexes - how to measure/to improve ? - Knowledge management and NOC staff training - NMS - tools (functionalities) - NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours - How to work proactively in NOC (active/passive monitoring) *PIONIER* Five topics PSNC would like to present about NOC: - networks PIONIER (NREN) and POZMAN - technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions - procedures used in our NOC - our TTS - public part, experience, missing functions - NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC - Experiences in joint management application design & deployment with commercial company Five topics (or questions) that PSNC is interested in knowing how other NOCs handle: - how to minimize users calls (public available FAQ? tools? documentation?) - reports - what? how often? - data and tools integration - how? recommended tools? - serving users calls - how to automate it, distributing calls among different NOC administrators - (D-)DoS - how to detect network attacks *ARNES* ARNES could present: - Pros and contras for dedicated NOC personnel - NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) ARNES would like to hear from others about: - NOC staff job description - NOC assesment and KPI -- measuring NOC performance - NOC to user communication (communication with more or less experienced users, how to keep user contact info up-to-date, personalized web interface for NOC to user communication) - NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) - How many customers do you have? (depends on if you are counting persons, networks, hosts,..) - What kind of agreements do you have with service providers and customers (SLA,..)? Without financial info, but just roughly descriptions of responsibilities. *NORDUnet* Things NORDUnet could present: - Troubleticketing system, experiences from building our own. - Managing user interfaces (using people outside the NOC to aggregate user requests) - Using external community tools (letting the community answer questions instead of the NOC). - NOC organisation (staffing, internal escalations, handover and schedule) - Reaching your network from anywhere (out-of-band, in-band and VPN) Things NORDUnet would like to hear about from others: - How is knowledge dissemination handled between groups and new NOC employees? - How is monitoring and alarm correlation handled between different systems? - How are new services and users provisioned? - What kind of routines are used in regards to change management (e.g. in planned maintenance)? - What other services than the network does your NOC operate and how do they differentiate from the network service? *GNOC at Indiana University* Five topics to introduce the Global Research NOC at Indiana University: - Global Research NOC Service Desk<[http://globalnoc.iu.edu/tour.html]> provides 24x7x365 technical call center support, trouble ticket management, and workflow support. The service desk is housed in a state-of-the-art call center in Indiana University's Informatics and Communications Technology Complex (ICTC) on Indiana University's Indianapolis campus. This call center features a fully customizable 30-foot screen used to monitor and troubleshoot the current health of the various networks supported. We provide NOC support for the most advanced research and education networks in the country. [http://globalnoc.iu.edu/tour.html |
...
] - Willingness to assist network customers with other services and requests outside of traditional front-line network operations, such as network redesign planning and utilizing our tool set. |
...
- A network knowledgeable tier-one Service Desk that can assess network issues intelligently and can provide engineering with helpful information as they begin troubleshooting. |
...
- Internally developed tools such our network database [http://globalnoc.iu.edu/grnoc-tool-set.html |
...
] - International networking efforts |
...
Five topics the GlobalNOC at Indiana University would like to know about other NOC's: |
...
- What methodologies and tools do you use to understand complex cross-network relationships and dependencies? |
...
- How do you organize and update your internal documentation environment for easy access and search, as well as providing clear and accurate information? |
...
- How do you integrate the different groups within your NOC (Service Desk-Tier One, Engineering-Tier Two & Three, Systems Engineering, etc) so each has clear expectations of what each group does, with all working as a unified team? |
...
- With R&E networking moving towards having more regional connectedness between smaller networks, RON's, universities, etc, are you planning for a new phase in networking where specific operational differences between NOC's may negatively affect the reliability of the services provided? (for example: different response levels from hardware and fiber vendors, vastly different maintenance windows, NOC's not being 24x7, etc.) |
...
- What type of training methodologies do you use to bring new employees up-to-speed quickly so they can start making a valuable contribution to the operation? |
...
SWITCH
The topics SWITCH could tell about:
- network, equipment, services, customers
- management of fibers - with segments from different providers
- staff
- outsourced services
- work flows
- tools
- experiences and things we like to improve
The topics SWITCH is interested to know from other NOCs:
The same points like above with special focus on:
- incident handling
- tools and especially
- tools to support the distributed handling of incidents
GRNET
GRNET is keen in presenting one of the topics trailed below:
- Efficient administration of many Linux based servers,
- Providing VPS service,
- Network database (GRNnet-DBII),
- Visualisation tools,
On the other hand, we would like to hear how our colleagues tackle the followings:
- NOC key performance indexes,
- Automated SLA monitoring tools,
- Network monitoring tools (may be different than the ones above),
- operation of CERT team and security policies
RedIRIS
Five topics to introduce RedIRIS NOC:
- Type of network. Equipment and type of lines.
- Out of band network.
- Trouble ticket system.
- SLAs with vendors and providers and type of contact.
- Organisation of our NOC.
Five topics RedIRIS would like to know about other NOCs:
- Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
- Monitoring tools.
- Acquiring, management and operation of dark fibre. Procedures. Optical NOC differenciated from IP NOC?
- Services managed by the NOC.
- Network database and integration with other monitoring tools.
FCCN
Five topics to introduce FCCN's NOC:
- Deployed network (Optical/Ethernet/IP)
- Out-of-band systems (MRV using GSM/GPRS)
- Service Desk & Automated reporting (per member)
- Deployed probes network (QoSMetrics solution)
- Past horror stories: What you can try to avoid & Murphy's Law.
Five topics FCCN would like to know about other NOCs:
- How is the 24x7x365 NOC service guaranteed?
- Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
- Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
- Do you have/use a fixed maintenance window?
- How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.
CERN
CERN would be glad to present from the following list of topics:
- Transforming 2 layer network support into 3 layer support with outsourcing
- Integration of network and telecom support
- Creation of a knowledge repository for network/telecom support
- Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)
- The challenge of using multiple ticketing systems
CERN would be interested in hearing about:
- Roles and responsibilities (structure, who does what)
- Relation and communication between the network operations team and other internal entities (design, deployment)
- How operation is done outside working hours including weekends
- How common experience is handled: specialized vs. universal team members
- How procedures are created and stored
RESTENA
Five topics to introduce RESTENA's NOC:
- Infrastructure building and operations (dark fiber, net devices,server-rooms, POPs, servers)
- An "all-in-one" NOC: structure/organization/tools
- 24x7x365 service support
- Coverage of user support (helpdesk)
- Services operated (from lambdas to large scale e-mail accounts)
Five topics RESTENA would like to know about other NOCs:
- User (customer) information exchange: channels and content
- NOC activities/coverage and staff (network, services, community details)
- Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
- Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
- NOC awareness and identification: internally and among user community
SURFnet
Items we from SURFnet can present our experiences
- acquring and managing dark fiber
- managing an outsourced NOC
- procedures - the good and bad things
Things we would like to hear about from other NOC's:
- how to inform/communicate with customers
- branding of an outsourced NOC
- ticketing systems
- CMDB experiences
CESCA
And these are topics that we would like to introduce about our NOC:
- Virtualization of the network.
- NOC tools and how to adapt them to our needs (development).
- Users management. How to approach our users, depending on the action (change, incident, request,..).
- Survey for our services, what to ask and how?
- Out-of-band access to our network.
These are the topics that we would want to know from other NOCs:
- Ticketing Systems, how to optimize their use.
- Integration of diferent tools.
- Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
- Escalation of incidents. How many levels?
- Documentation: what can and can not be documented.
HEAnet
5 important bullet points introducing our NOC(s)
- Heuristic Failure analysis (how we use heuristics to determine SLA calculations and where they are and are not appropriate.) (Also interested in discovering if others are doing this.)
- Ticketing, including automatic ticket generation for incidents.
- Contract and supplier management - how HEAnet does it and how well it works (or doesn't work.)
- Reporting to your clients. How do NRENs report to clients? Periodical and Real-time reporting. On demand reporting.
- 24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?
5 questions what you want to know about other NOCs
- How are staff hired for the NOC? Skillsets? Interview processes?
- Standards compliance and process adoption. Have other NOCs adopted ISO9000? ITIL? PRINCE2? APM? ISO20000? Six Sigma? Other standards?
- What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
- How do NOCs source support and SLAs? What about penalty clauses? What's reasonable, what works and how are they enforced?
- How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?
Interesting topics mapped to the work items
Taxonomy of NOC organisations, internal processes, workflows
...
Topics participants are interested in sharing information about
...
Topics participants are interested in getting input on
...
Network
...
Network
...
Infrastructure building and operations
(dark fiber, net devices, server-rooms, POPs, servers)
...
Core equipment vendor/make, approx numbers
...
Deployed network (Optical/Ethernet/IP)
...
...
Network type (IP only, hybrid, switched/transmission only, LAN only)
...
...
Acquiring and managing dark fiber
...
Acquiring, management and operation of dark fibre.
...
Management of fibers - with segments from different providers
...
...
Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc
...
...
Network, users & Core/Access nodes
- how to treat different users and different parts of network
...
...
Reaching your network from anywhere
...
...
Out-of-band access to our network
...
...
Out-of-band systems (MRV using GSM/GPRS)
...
...
NOC functions
...
NOC functions
...
Survey for our services, what to ask and how?
...
Services managed by the NOC.
...
Services operated (from lambdas to large scale e-mail accounts)
...
What scope of services does the NOC support? Networking?
Servers? Value added services like DNS, web hosting, etc.? HPC?
...
Providing VPS service
...
...
Responsibilities (e.g. monitoring, configuring/provisioning,
direct or indirect repairs, DNS updates, RIPE updates,
POP access and deliveries, reporting);
...
Responsibility of your NOC: how far does it go?
...
...
What other services than the network does your NOC operate
and how do they differentiate from the network service?
...
...
Which services and incidents are covered by the NOC?
How do you deal with requests/incidents that are outside your NOC's scope?
...
...
Connected networks: Types of organization and approx numbers for
subscribers, peers and upstream IP providers- Roles and responsibilities
(structure, who does what)
...
...
Operation of CERT team and security policies
...
...
Number (approx) of ticket mail lists the NOC is on,
and approx number of tickets received on these lists
...
NOC structure
...
NOC structure
...
Staffing, internal escalations, handover and schedule
...
NOC activities/coverage and staff (network, services, community details)
...
24x7x365 service support
...
...
24 x 7 NOC cover. Do you provide it? What SLA?
How do you hire staff and plan rotas, etc?
Is there a real client demand for it?
...
How is the 24x7x365 NOC service guaranteed?
...
Hours of operation (office and on-call)
...
How operation is done outside working hours including weekends
...
Pros and contras for dedicated NOC personnel
...
How common experience is handled: specialized vs. universal team members
...
A network knowledgeable tier-one Service Desk that can
assess network issues intelligently and can provide engineering
with helpful information as they begin troubleshooting.
...
NOC organization and procedures: Tier1-N functions,
# of persons in shifts and in total, work quality control, peak hours
...
Structure and roles in a multi-layer NOC operating multi-type networks
(campus network, international research network, internet exchange point)
...
Staff organization, i.e. services support strategy
(per person/team/NOC) with their respective pros/cons
...
Global Research NOC Service Desk provides 24x7x365 technical call center
support, trouble ticket management, and workflow support. The service desk
is housed in a state-of-the-art call center in Indiana University's Informatics
and Communications Technology Complex (ICTC) on Indiana University's
Indianapolis campus. This call center features a fully customizable 30-foot
screen used to monitor and troubleshoot the current health of the various
networks supported. We provide NOC support for the most advanced
research and education networks in the country.
...
With R&E networking moving towards having more regional connectedness
between smaller networks, RON's, universities, etc, are you planning for a
new phase in networking where specific operational differences between
NOC's may negatively affect the reliability of the services provided?
(for example: different response levels from hardware and fiber vendors,
vastly different maintenance windows, NOC's not being 24x7, etc.)
...
Technical structure (DWDM, switches, routers) and its influence to
NOC structure, organizational structure: Layer 1,2 (NOC) and
Layer 3 (IP NOC), employees and their functions
...
...
...
Distributed NOC's (as opposed to a NOC on a central location): pro's & con's
...
...
Optical NOC differentiated from IP NOC?
...
Transforming 2 layer network support into 3 layer support with outsourcing
...
Organisation and roles of NOC members. Number of levels.
Outsourced vs. staff-in NOC. Pros and cons. Functions.
Experience with externalized NOC.
...
Tiers, and which tiers are in-house or outsourced
...
Partially outsourced NOCs (24x7, weekends,...): what kind of information
do you give them, how do they interact with you and your users?
...
Managing an outsourced NOC
...
Branding of an outsourced NOC
...
Lessons learned from outsourced incident & change management
...
Good & bad experiences with outsourcing
...
Setting up and maintaining a NOC
...
Setting up and maintaining a NOC
...
Knowledge sharing
...
Knowledge management and NOC staff training
...
...
NOC staff job description
...
...
How are staff hired for the NOC? Skillsets? Interview processes?
...
NOC assessment
...
NOC assessment
...
NOC key performance indexes
- how to measure/to improve?
...
...
How do you assign value to what the NOC does?
Do you cross charge for projects that will be handed over to the NOC?
How to you prove to upper management that the NOC is good value for money?
...
NOC workflows
...
NOC workflows
...
Procedures used in the NOC - the good and bad things
...
Presentation of the NOC's current procedures &
who is responsible for maintaining them
...
Workflows
...
...
...
How do you deal with performance issues/incidents and e2e issues
regarding traffic that flows through your network.
...
...
What kind of routines are used in regards to change management
(e.g. in planned maintenance)?
...
...
Do you have/use a fixed maintenance window?
...
...
How are new services and users provisioned?
...
...
How does your NOC take care of incident handling?
...
...
(D-)DoS - how to detect network attacks
...
...
What are your target repair times?
...
...
How to work proactively in NOC (active/passive monitoring)
...
...
How are new services and users provisioned?
NOC front-end development and improvement
NOC tools, interworking/interfacing issues, and automation
Efficient communication/collaboration tools and practices
Facilitate the collection/creation of best practice documents
...
*SWITCH*
The topics SWITCH could tell about:
- network, equipment, services, customers
- management of fibers - with segments from different providers
- staff
- outsourced services
- work flows
- tools
- experiences and things we like to improve
The topics SWITCH is interested to know from other NOCs:
The same points like above with special focus on:
- incident handling
- tools and especially
- tools to support the distributed handling of incidents
*GRNET*
GRNET is keen in presenting one of the topics trailed below:
- Efficient administration of many Linux based servers,
- Providing VPS service,
- Network database (GRNnet-DBII),
- Visualisation tools,
On the other hand, we would like to hear how our colleagues tackle the followings:
- NOC key performance indexes,
- Automated SLA monitoring tools,
- Network monitoring tools (may be different than the ones above),
- operation of CERT team and security policies
*RedIRIS*
Five topics to introduce RedIRIS NOC:
- Type of network. Equipment and type of lines.
- Out of band network.
- Trouble ticket system.
- SLAs with vendors and providers and type of contact.
- Organisation of our NOC.
Five topics RedIRIS would like to know about other NOCs:
- Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
- Monitoring tools.
- Acquiring, management and operation of dark fibre. Procedures. Optical NOC differenciated from IP NOC?
- Services managed by the NOC.
- Network database and integration with other monitoring tools.
*FCCN*
Five topics to introduce FCCN's NOC:
- Deployed network (Optical/Ethernet/IP)
- Out-of-band systems (MRV using GSM/GPRS)
- Service Desk & Automated reporting (per member)
- Deployed probes network (QoSMetrics solution)
- Past horror stories: What you can try to avoid & Murphy's Law. :-)
Five topics FCCN would like to know about other NOCs:
- How is the 24x7x365 NOC service guaranteed?
- Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
- Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
- Do you have/use a fixed maintenance window?
- How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.
*CERN*
CERN would be glad to present from the following list of topics:
- Transforming 2 layer network support into 3 layer support with outsourcing
- Integration of network and telecom support
- Creation of a knowledge repository for network/telecom support
- Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)
- The challenge of using multiple ticketing systems
CERN would be interested in hearing about:
- Roles and responsibilities (structure, who does what)
- Relation and communication between the network operations team and other internal entities (design, deployment)
- How operation is done outside working hours including weekends
- How common experience is handled: specialized vs. universal team members
- How procedures are created and stored
*RESTENA*
Five topics to introduce RESTENA's NOC:
- Infrastructure building and operations (dark fiber, net devices,server-rooms, POPs, servers)
- An "all-in-one" NOC: structure/organization/tools
- 24x7x365 service support
- Coverage of user support (helpdesk)
- Services operated (from lambdas to large scale e-mail accounts)
Five topics RESTENA would like to know about other NOCs:
- User (customer) information exchange: channels and content
- NOC activities/coverage and staff (network, services, community details)
- Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
- Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
- NOC awareness and identification: internally and among user community
*SURFnet*
Items we from SURFnet can present our experiences
- acquring and managing dark fiber
- managing an outsourced NOC
- procedures - the good and bad things
Things we would like to hear about from other NOC's:
- how to inform/communicate with customers
- branding of an outsourced NOC
- ticketing systems
- CMDB experiences
*CESCA*
And these are topics that we would like to introduce about our NOC:
- Virtualization of the network.
- NOC tools and how to adapt them to our needs (development).
- Users management. How to approach our users, depending on the action (change, incident, request,..).
- Survey for our services, what to ask and how?
- Out-of-band access to our network.
These are the topics that we would want to know from other NOCs:
- Ticketing Systems, how to optimize their use.
- Integration of diferent tools.
- Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
- Escalation of incidents. How many levels?
- Documentation: what can and can not be documented.
*HEAnet*
5 important bullet points introducing our NOC(s)
- Heuristic Failure analysis (how we use heuristics to determine SLA calculations and where they are and are not appropriate.) (Also interested in discovering if others are doing this.)
- Ticketing, including automatic ticket generation for incidents.
- Contract and supplier management - how HEAnet does it and how well it works (or doesn't work.)
- Reporting to your clients. How do NRENs report to clients? Periodical and Real-time reporting. On demand reporting.
- 24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?
5 questions what you want to know about other NOCs
- How are staff hired for the NOC? Skillsets? Interview processes?
- Standards compliance and process adoption. Have other NOCs adopted ISO9000? ITIL? PRINCE2? APM? ISO20000? Six Sigma? Other standards?
- What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
- How do NOCs source support and SLAs? What about penalty clauses? What's reasonable, what works and how are they enforced?
- How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?
h2. Interesting topics mapped to the work items
h3. Taxonomy of NOC organisations, internal processes, workflows
|| Topics participants are interested in sharing information about || Topics participants are interested in getting input on ||
|| Network \\ || Network ||
| Infrastructure building and operations \\
(dark fiber, net devices, server-rooms, POPs, servers) | Core equipment vendor/make, approx numbers |
| Deployed network (Optical/Ethernet/IP) | |
| Network type (IP only, hybrid, switched/transmission only, LAN only) | |
| Acquiring and managing dark fiber | Acquiring, management and operation of dark fibre. |
| Management of fibers - with segments from different providers | |
| Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc | |
| Network, users & Core/Access nodes \\
\- how to treat different users and different parts of network | |
| *Reaching your network from anywhere* | |
| Out-of-band access to our network | |
| Out-of-band systems (MRV using GSM/GPRS) | |
|| NOC functions || NOC functions ||
| Survey for our services, what to ask and how? | Services managed by the NOC. |
| Services operated (from lambdas to large scale e-mail accounts) | What scope of services does the NOC support? Networking? \\
Servers? Value added services like DNS, web hosting, etc.? HPC? |
| Providing VPS service | |
| Responsibilities (e.g. monitoring, configuring/provisioning, \\
direct or indirect repairs, DNS updates, RIPE updates, \\
POP access and deliveries, reporting); | Responsibility of your NOC: how far does it go? |
| | What other services than the network does your NOC operate \\
and how do they differentiate from the network service? |
| | Which services and incidents are covered by the NOC? \\
How do you deal with requests/incidents that are outside your NOC's scope? |
| | Connected networks: Types of organization and approx numbers for \\
subscribers, peers and upstream IP providers\- Roles and responsibilities \\
(structure, who does what) |
| | Operation of CERT team and security policies |
| | Number (approx) of ticket mail lists the NOC is on, \\
and approx number of tickets received on these lists |
|| NOC structure || NOC structure ||
| Staffing, internal escalations, handover and schedule | NOC activities/coverage and staff (network, services, community details) |
| 24x7x365 service support | |
| 24 x 7 NOC cover. Do you provide it? What SLA? \\
How do you hire staff and plan rotas, etc? \\
Is there a real client demand for it? | How is the 24x7x365 NOC service guaranteed? |
| Hours of operation (office and on-call) | How operation is done outside working hours including weekends |
| Pros and contras for dedicated NOC personnel | How common experience is handled: specialized vs. universal team members |
| A network knowledgeable tier-one Service Desk that can \\
assess network issues intelligently and can provide engineering \\
with helpful information as they begin troubleshooting. | NOC organization and procedures: Tier1-N functions, \\
\# of persons in shifts and in total, work quality control, peak hours |
| Structure and roles in a multi-layer NOC operating multi-type networks \\
(campus network, international research network, internet exchange point) | Staff organization, i.e. services support strategy \\
(per person/team/NOC) with their respective pros/cons |
| Global Research NOC Service Desk provides 24x7x365 technical call center \\
support, trouble ticket management, and workflow support. The service desk \\
is housed in a state-of-the-art call center in Indiana University's Informatics \\
and Communications Technology Complex (ICTC) on Indiana University's \\
Indianapolis campus. This call center features a fully customizable 30-foot \\
screen used to monitor and troubleshoot the current health of the various \\
networks supported. We provide NOC support for the most advanced \\
research and education networks in the country. | With R&E networking moving towards having more regional connectedness \\
between smaller networks, RON's, universities, etc, are you planning for a \\
new phase in networking where specific operational differences between \\
NOC's may negatively affect the reliability of the services provided? \\
(for example: different response levels from hardware and fiber vendors, \\
vastly different maintenance windows, NOC's not being 24x7, etc.) |
| Technical structure (DWDM, switches, routers) and its influence to \\
NOC structure, organizational structure: Layer 1,2 (NOC) and \\
Layer 3 (IP NOC), employees and their functions | |
| | Distributed NOC's (as opposed to a NOC on a central location): pro's & con's |
| | Optical NOC differentiated from IP NOC? |
| Transforming 2 layer network support into 3 layer support with outsourcing | Organisation and roles of NOC members. Number of levels. \\
Outsourced vs. staff-in NOC. Pros and cons. Functions. \\
Experience with externalized NOC. |
| Tiers, and which tiers are in-house or outsourced | Partially outsourced NOCs (24x7, weekends,...): what kind of information \\
do you give them, how do they interact with you and your users? |
| Managing an outsourced NOC | Branding of an outsourced NOC |
| Lessons learned from outsourced incident & change management | Good & bad experiences with outsourcing |
|| Setting up and maintaining a NOC || Setting up and maintaining a NOC ||
| Knowledge sharing | Knowledge management and NOC staff training |
| | NOC staff job description |
| | How are staff hired for the NOC? Skillsets? Interview processes? |
|| NOC assessment || NOC assessment ||
| \\ | NOC key performance indexes \\
\- how to measure/to improve? |
| | How do you assign value to what the NOC does? \\
Do you cross charge for projects that will be handed over to the NOC? \\
How to you prove to upper management that the NOC is good value for money? |
|| NOC workflows || NOC workflows ||
| Procedures used in the NOC - the good and bad things | Presentation of the NOC's current procedures & \\
who is responsible for maintaining them |
| Workflows \\ | |
| | How do you deal with performance issues/incidents and e2e issues \\
regarding traffic that flows through your network. |
| | What kind of routines are used in regards to change management \\
(e.g. in planned maintenance)? |
| | Do you have/use a fixed maintenance window? |
| | How are new services and users provisioned? |
| | How does your NOC take care of incident handling? |
| | (D-)DoS - how to detect network attacks |
| | What are your target repair times? |
| | How to work proactively in NOC (active/passive monitoring) |
| | How are new services and users provisioned? |
h3. NOC front-end development and improvement
h3. NOC tools, interworking/interfacing issues, and automation
|| <!-- body, div, table, thead, tbody, tfoot, tr, th, td, p { font-family: "Nimbus Sans L"; font-size: x-small; } -->| *Topics participants are interested in sharing information about* | || <!-- body, div, table, thead, tbody, tfoot, tr, th, td, p { font-family: "Nimbus Sans L"; font-size: x-small; } -->| h4. *Topics participants are interested in getting input on* | ||
| <!-- body, div, table, thead, tbody, tfoot, tr, th, td, p { font-family: "Nimbus Sans L"; font-size: x-small; } -->| h5. Monitoring tools | | | h5. Monitoring tools |
| CARNET: NMS - our tool functionalities / developement | DANTE: Main Monitoring systems and alarm thresholds |
| PIONIER: NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC | | CARNET: NMS - tools (functionalities) | |
| | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | | | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | |
| | GNOC IU: Internally developed tools such our network database [http://globalnoc.iu.edu/grnoc-tool-set.html|http://globalnoc.iu.edu/grnoc-tool-set.html] |
| |
| |
| | | SWITCH: tools and especially |
| SWITCH: tools | GRNET: Network monitoring tools (may be different than the ones above) |
| GRNET: Visualisation tools | | RedIRIS: Monitoring tools. | |
| CESCA: NOC tools and how to adapt them to our needs (development). | FCCN: Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated? |
| | *Multidomain tools: Distributed handling of incidents* | | *Multidomain tools: Distributed handling of incidents* |
| | GNOC IU: What methodologies and tools do you use to understand complex cross-network relationships and dependencies? |
| | | SWITCH: tools to support the distributed handling of incidents | |
| | *Ticketing systems* | | | *Ticketing systems* | |
| NORDUNET: Troubleticketing system, experiences from building our own. | SURFNET: ticketing systems |
| | CESCA: Ticketing Systems, how to optimize their use. |
| *Knowledge management: Configuration Management Database/Network Database* | *Knowledge management: Configuration Management Database/Network Database* |
| GNOC IU: Internally developed tools such our network database [http://globalnoc.iu.edu/grnoc-tool-set.html|http://globalnoc.iu.edu/grnoc-tool-set.html] | RedIRIS: Network database and integration with other monitoring tools. |
| | RESTENA: Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively) |
| | SURFNET: CMDB experiences |
| *SLA reporting and monitoring tools* | *SLA reporting and monitoring tools* |
| ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) |
| | | GRNET: Automated SLA monitoring tools | |
| *Chat/communication/coordination tools* | *Chat/communication/coordination tools* |
| NORDUNET: Using external community tools (letting the community answer questions instead of the NOC). | |
| *Integration of tools* | *Integration of tools* |
| BELNET: approach & tools: how the different departments & the NOC's work together | PIONIER: data and tools integration - how? recommended tools? |
| ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) |
| RESTENA: An "all-in-one" NOC: structure/organization/tools | | RedIRIS: Network database and integration with other monitoring tools. | |
| | CESCA: NOC tools and how to adapt them to our needs (development). | | CESCA: Integration of diferent tools. |
h3. Efficient communication/collaboration tools and practices
h3. Facilitate the collection/creation of best practice documents
h3. Investigate and liaise with other communities<!-- body, div, table, thead, tbody, tfoot, tr, th, td, p { font-family: "Nimbus Sans L"; font-size: x-small; } -->| *NOC tools* | |
| *Topics participants are interested in sharing information about* | *Topics participants are interested in getting input on* |
| *Monitoring tools* | *Monitoring tools* |
| CARNET: NMS - our tool functionalities / developement | DANTE: Main Monitoring systems and alarm thresholds |
| PIONIER: NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC | CARNET: NMS - tools (functionalities) |
| ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) |
| GNOC IU: Internally developed tools such our network database [http://globalnoc.iu.edu/grnoc-tool-set.html|http://globalnoc.iu.edu/grnoc-tool-set.html] | SWITCH: tools and especially |
| SWITCH: tools | GRNET: Network monitoring tools (may be different than the ones above) |
| GRNET: Visualisation tools | RedIRIS: Monitoring tools. |
| CESCA: NOC tools and how to adapt them to our needs (development). | FCCN: Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated? |
| *Multidomain tools: Distributed handling of incidents* | *Multidomain tools: Distributed handling of incidents* |
| | GNOC IU: What methodologies and tools do you use to understand complex cross-network relationships and dependencies? |
| | SWITCH: tools to support the distributed handling of incidents |
| *Ticketing systems* | *Ticketing systems* |
| NORDUNET: Troubleticketing system, experiences from building our own. | SURFNET: ticketing systems |
| | CESCA: Ticketing Systems, how to optimize their use. |
| *Knowledge management: Configuration Management Database/Network Database* | *Knowledge management: Configuration Management Database/Network Database* |
| GNOC IU: Internally developed tools such our network database [http://globalnoc.iu.edu/grnoc-tool-set.html|http://globalnoc.iu.edu/grnoc-tool-set.html] | RedIRIS: Network database and integration with other monitoring tools. |
| | RESTENA: Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively) |
| | SURFNET: CMDB experiences |
| *SLA reporting and monitoring tools* | *SLA reporting and monitoring tools* |
| ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) |
| | GRNET: Automated SLA monitoring tools |
| *Chat/communication/coordination tools* | *Chat/communication/coordination tools* |
| NORDUNET: Using external community tools (letting the community answer questions instead of the NOC). | |
| *Integration of tools* | *Integration of tools* |
| BELNET: approach & tools: how the different departments & the NOC's work together | PIONIER: data and tools integration - how? recommended tools? |
| ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) | ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) |
| RESTENA: An "all-in-one" NOC: structure/organization/tools | RedIRIS: Network database and integration with other monitoring tools. |
| CESCA: NOC tools and how to adapt them to our needs (development). | |
| | CESCA: Integration of diferent tools. | |