Troubleshooting Cisco LAN QoS

QoS on the LAN is something that a lot of people think is not very necessary.  When everything is running at 1Gb or 10Gb, we something think that QoS isn’t needed.  Let me tell you this isn’t true, as I came to see while troubleshooting a very difficult problem this week (well I thought it was difficult at least)
Our sys admins use a tool called Icinga for monitoring of their servers.  It’s a very similar tool to Nagios.  It uses SNMP to gather information about servers and reports their status. There was one site were they were not getting any information back from the servers.
We checked the basics first as usual.  Things link, can you ping the servers, can you SSH to the servers, is there anything blocking SNMP and finally what does a TCP dump show?
The answer to the last question was what kicked of the QOS investigation, the TCP dump showed something very interesting.

The Icinga server would query the remote servers and the remote servers would respond.  However the last two packets would never make it.  A  TCP dump on the Icinga server for the return traffic was always missing the last two UDP packets.  This happened every single time the Icinga server would poll the remote sites.
This is when I started looking at the QOS config on the devices in question.  Without getting into too much detail I ended up narrowing down the problem to the Cisco 3750 where the offending servers were connected.  We have a pretty simple QOS configuration on all our switches which classifies (marks) traffic inbound on every switch port, then the sites WAN router does prioritization onto the WAN based on those classifications.  The 3750 has a 1Gb uplink to the WAN router and it was on this uplink that I noticed a whole bunch of output drops.

GigabitEthernet1/0/47 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 0006.f623.a441 (bia 0006.f623.a441)
  Internet address is 10.10.248.1/32
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, 
     reliability 255/255, txload 2/255, rxload 8/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3181875
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 3288000 bits/sec, 491 packets/sec
  30 second output rate 964000 bits/sec, 351 packets/sec
     7139931692 packets input, 1912938538729 bytes, 0 no buffer
     Received 977217303 broadcasts (11568 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 977217015 multicast, 0 pause input
     0 input packets with dribble condition detected
     5751712193 packets output, 2914640075478 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
 
Notice the amount of total output drops on this interface. That was increasing a fair bit every time we kicked off a SNMP poll from Icinga. Normally the cause of output drops is a congested interface, but this is not really the case here, its a 1GB interface which normally sits around 20 – 30 Mb so over utilization didn’t seem to be an issue. However bursty traffic can also cause output drops, and it seemed that this could be a likely cause. Micro bursts of traffic can use all of the buffers available to an interface which can cause drops to occur.

So I started looking into what was happening with SNMP traffic. It gets classified inbound with a class-map that looks for SNMP traffic, then a policy-map marks it as CS2.

Here is a cut down view of the relevant QoS config

#show class-map
 Class Map match-all low-latency
   Match access-group name qos-low-latency

Class Map match-all video 
   Match access-group name qos-video

 Class Map match-all ssh
   Match access-group name qos-ssh

Class Map match-all others
   Match access-group name qos-others

#show ip access-list qos-others
 ip access-list extended qos-others
  remark telnet
  permit tcp any any eq telnet
  permit tcp any eq telnet any
  remark snmp
  permit udp any any eq snmp
  permit udp any eq snmp any

#show policy-map 
  Policy Map qos-ingress-marker
   Class low-latency
      set ip dscp ef
   Class others
      set ip dscp cs2
 
Next I needed to know what was happening to traffic marked with CS2 as if left the switch. CS2 is the same as DSCP 16 decimial or 010000 in binary. So to see what is happening to DSCP 16 when it leaves the switch I needed to know which queue and thresholds are being assigned. To view this I run:

#show mls qos maps dscp-output-q 
   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9 
     ------------------------------------------------------------
      0 :    02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 
      1 :    02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01 
      2 :    03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 
      3 :    03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 
      4 :    01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01 
      5 :    04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 
      6 :    04-01 04-01 04-01 04-01

This command shows me that DSCP 16 is assigned to queue 3 threshold 1. To read the table above, take the first number of your DSCP in decimal, in this case 1, and find that number in the D1 column. Then get the next number (6) and find that on the D2 row, then find where they meet

This tells me that DSCP 16 traffic is using queue 3 threshold 1. Each interface has 4 software queues. As above different classes of traffic are sent to different queues. Each interface is assigned a ‘queue-set’ which is just a collection of settings for the output queue. Queue-set 1 is the default and it is shown below:

#show mls qos queue-set 1
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400
 
So here we see that the four queues all have pretty much the same settings. So my goal here was to give queue 3 some more breathing room. I didn’t want to change queue-set 1 since that is assigned to every interface, so luckily there is queue-set 2 which can be modified, then assigned to a specific interface. First I gave queue 3 some more buffers:
 
(config)#mls qos queue-set output 2 buffers 20 10 50 20

This gives queue 3 50% of the available buffer space
Then i wanted to change the queue-set 2, queue 3 threshold. Both Threshold 1 and Threshold 2 are mapped to 3100 so that they can pull buffer from the reserved pool if required.
 
(config)#mls qos queue-set output 2 threshold 3 3100 3100 100 3200

So queue-set 2 now looks like this:

#show mls qos queue-set 2
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      20      10      50      20
threshold1:     100     200    3100     100
threshold2:     100     200    3100     100
reserved  :      50      50     100      50
maximum   :     400     400    3200     400
 
The last step is to assign this queue set to the interface where I saw the drops

(config)#int g1/0/47
(config-if)#qu
(config-if)#queue-set 2
And as luck would have it, this fixed the problem. The busty nature (and rather large packet size) was using all the available buffers in queue 3, hence causing the packet loss. For normal TCP this could have been OK as the packets would have been re-transmitted, but for UDP it caused a total loss of data and caused the monitoring to fail.
It took me a fair bit of mucking around to fix this, but in the end I got there.

show ip route

Displays the entire IP route table, a summary of the routing table or route information for specific IP addresses, network masks or protocols.
Syntax: show ip route [[ipAddress [mask]] [bgp | connected | mpls [ipAddress] [ipAddress/PrefLen] [ipAddress mask] [detail] | isis | ospf | static | summary | multicast |unicast]

ipAddress
Specify an IP address for the IP routes to display.
mask
Specify an IP address mask for the specified IP address.
PrefLen
Specify a Prefix Length for the specified IP address. Valid values = 0 - 31.
Description: Use the show ip route command with no arguments to display all IP routes.
Use the show ip route command with the address argument to display routes to a specific IP address.
Use the show ip route command with the mask argument to display routes with a specific network mask.
Use the show ip route command with the bgp, isis, ospf keyword to display summary information about all routes for the specified protocol.
Use the show ip route multicast command to display active routes used by Multicast protocols.
Use the show ip route unicast command to display active routes used for unicast forwarding.
Use the show ip route connected command to display summary information about all directly connected routes.
Use the show ip route mpls command to display all tunnel related route statistics.
use the show ip route mpls detail to display detailed information of all the tunnels by which each route/prefix is reachable. The detail keyword can optionally be used with all ipAddress variants of mpls.
Use the show ip route static command to display summary information about all statically configured routes.
Use the show ip route summary command to display summary information about all IP routes.
Factory Default: None.
Command Mode: Executive and privileged.
Example 1: In the following example, the show ip route command with no arguments displays all routes:


router>show ip route

Codes: C - connected O - OSPF i - IS-IS

S - static UD - Up/Down bit 1 L1 - level-1

B - BGP E1 - external type 1 L2 - level-2

M - MPLS E2 - external type 2

* - candidate default

m - route's metric

d - administrative distance

S * 0.0.0.0/0 via 10.200.0.1 [d:1 m:0]

S 9.9.9.9/32 via 127.0.0.1 [d:1 m:0]

C 10.200.0.0/16 directly connected to Ethernet 0

i UD 12.2.41.19/32 via 55.55.55.1 [d:115 m:35]

i UD 12.2.41.20/32 via 55.55.55.1 [d:115 m:35]

i UD 12.2.41.21/32 via 55.55.55.1 [d:115 m:35]

i UD 12.2.41.22/32 via 55.55.55.1 [d:115 m:35]

i UD 12.2.41.23/32 via 55.55.55.1 [d:115 m:35]

.

.

.

C 55.55.55.0/24 directly connected to GBE 1/19/2

C 55.55.55.3/32 directly connected to Null 0

S 127.0.0.0/8 via 127.0.0.1 [d:0 m:0]

C 127.0.0.1/32 directly connected to Null 0

i L1 191.191.191.191/32 via 55.55.55.1 [d:115 m:28]

i L1 191.194.1.0/24 via 55.55.55.1 [d:115 m:28]

C 193.193.193.193/32 directly connected to Loopback 0

i L1 194.10.1.0/24 via 55.55.55.1 [d:115 m:28]

C 194.10.2.0/24 directly connected to POS 1/14/2

C 194.10.2.1/32 directly connected to POS 1/14/2

C 194.10.2.2/32 directly connected to Null 0

S 200.1.1.0/24 via 127.0.0.1 [d:1 m:0]

S 200.1.2.0/24 via 127.0.0.1 [d:1 m:0]

S 200.1.3.0/24 via 127.0.0.1 [d:1 m:0]

.

.

.



The following table describes the fields displayed by the show ip route command.

Table 1-8. Fields Displayed by show ip route 
Field Description
connected
Specifies the route was learned as a result of configuring the interface.
static
Specifies the route was explicitly configured using the ip route command.
BGP
Specifies the route was received from BGP protocol advertisements.
MPLS
Specifies this route carries MPLS data.
external type 1
Specifies the route was imported into the OSPF area from an Autonomous System Boundary Router (ASBR). Further, this route used the cost associated with the link between the ASBR and this router as part of the cost of the route.
external type 2
Specifies the route was imported into the OSPF area from an Autonomous System Boundary Router (ASBR). Further, this route does not use the cost associated with the link between the ASBR and this router as part of the cost of the route.
isis
Specifies this route was received from IS-IS protocol advertisements.
N.N.N.N/nn
Network address and mask of the remote network.
via N.N.N.N
Indicates the next router in the remote network.
directly connected to
Indicates the network is directly connected to a local interface.
UD
Indicates the Up/Down bit is set and that the route was leaked from L2 into L1.
d:
The administrative distance assigned to this route.
m:
The metric assigned to this route.
Example 2: In the following example, the show ip route static command displays information about statically configured routes:


router>show ip route static

S 0.0.0.0/0 via 10.5.0.1 [w:1 m:0]

S 127.0.0.0/8 via 127.0.0.1 [w:0 m:0]

S 131.1.1.0/24 via 127.0.0.1 [w:1 m:0]

S 131.1.2.0/24 via 127.0.0.1 [w:1 m:0]

S 131.1.3.0/24 via 127.0.0.1 [w:1 m:0]

S 131.1.4.0/24 via 127.0.0.1 [w:1 m:0]

Example 3: In the following example, the show ip route summary command displays summary information about all IP routes:


router#show ip route summary



Route source Networks

connected 12

static 2

bgp 100654

ospf intra-area 1785

total 102453

The following table defines the fields displayed in the show ip route summary command:

Table 1-9. Fields Displayed by show ip route summary 
Field Description
connected
These routes were learned as a result of configuring the interface
kernel
These routes were in the kernel before the routing task started running.
static
The number of routes that were explicitly configured using the ip route command.
bgp
The number of routes received from BGP protocol advertisements.
ospf intra-area
The number of routes learned from within this area.
ospf inter-area
The number of routes received from other OSPF areas.
ospf external-1
These routes were imported into the OSPF area from an Autonomous System Boundary Router (ASBR). Further, these routes use the cost associated with the link between the ASBR and this router as part of the cost of the route.
ospf external-2
These routes were imported into the OSPF area from an Autonomous System Boundary Router (ASBR). Further, these routes do not use the cost associated with the link between the ASBR and this router as part of the cost of the route.
isis
The number of routes received from IS-IS protocol advertisements.
total
The total number of routes in the IP route table.
Related Commands: ip address
ip route

Cisco 1941 Password Recovery

We always receive such kinds of answers “I have a Cisco router that i forgot the password, i need to rest it if someone could help me?” or “Could everyone explain the procedure for password recovery associated with Cisco routers?” It’s no doubt that Cisco password recovery is one of the most popular problems

Cisco router password recovery








during the process of using Cisco products.Recently, many friends asked me about steps of popular Cisco 1941 password recovery, now, let’s share it here:

1) Attach a terminal or PC with terminal emulation to the console port of the Cisco 1941 router. And please refer to these documents for information on how to cable and connect a terminal to the console port or the AUX port:
Cabling Guide for Console and AUX Ports
Connecting a Terminal to the Console Port on Cisco Catalyst Switches
Connect a Terminal to Catalyst 2948G-L3, 4908G-L3, and 4840G Series (if without those devices, replace it with the latest related switches.
9600 baud rate
No parity
8 data bits
1 stop bit
No flow control

2) If you can access the router, type show version at the prompt, and record the configuration register setting. See Example of Password Recovery Procedure in order to view the output of a show version command.
Note: The configuration register is typically set to 0×2102 or 0×102. If you can no longer access the router (because of a lost login or TACACS password), you can safely assume that your configuration register is set to 0×2102.

3) Use the power switch in order to turn off the router, and then turn the router back on.

4) Press Break on the terminal keyboard a couple of times after you see the message program load complete, entry point: 0×80008000, size: 0x6fdb4c in order to put the router into ROMMON.
Note: The values of entry point and size are subjected to the routers.
If the break sequence does not work, refer to Standard Break Key Sequence Combinations during Password Recovery for other key combinations.
If unable to break into ROMMON mode, perform these steps:
Remove flash.
Reload the router. Router will end up in ROMMON mode.
Insert flash.
Perform standard procedure for Cisco 1941 password recovery.Cisco 1941 k9 password recovery5) Type confreg 0×2142 at the rommon 1> prompt in order to boot from Flash.
This step bypasses the startup configuration where the passwords are stored.

6) Type reset at the rommon 2> prompt.
The router reboots, but ignores the saved configuration.

7) Type no after each setup question, or press Ctrl-C in order to skip the initial setup procedure.

8) Type enable at the Router> prompt.
You are in enable mode and should see the Router# prompt.

9) Type configure memory or copy startup-config running-config in order to copy the nonvolatile RAM (NVRAM) into memory.
Warning: Do not enter copy running-config startup-config or write. These commands erase your startup configuration.

10) Type show running-config.
The show running-config command shows the configuration of the router. In this configuration, the shutdown command appears under all interfaces, which indicates all interfaces are currently shut down. In addition, the passwords (enable password, enable secret, vty, and console passwords) are in either an encrypted or encrypted format. You can reuse encrypted Cisco router passwords. You must change encrypted passwords to a new password.

11) Type configure terminal.
The hostname (config) # prompt appears.

12) Type enable secret in order to change the enable secret password. For example:

hostname (config)#enable secret cisco

13) Issue the no shutdown command on every interface that you use.
If you issue a show ip interface brief command, every interface that you want to use should display up.

14) Type config-register. Where configuration_register_setting is either the value you recorded in step 2 or 0×2102. For example:

hostname(config)#config-register 0×2102

15) Press Ctrl-z or end in order to leave the configuration mode.
The hostname# prompt appears.

16) Type write memory or copy running-config startup-config in order to commit the changes.

CISCO 877 troubleshooting guide



There are many reasons why your Digital Subscriber Line (DSL) connection may not be functioning properly. The goal of this section is to isolate the cause of the failure and repair it. The first troubleshooting step is to determine which layer of your Asynchronous Digital Subscriber Line (ADSL) service is failing. There are three layers in which the failure can occur.
  • Layer 1 – DSL physical connectivity to your ISP's Digital Subscriber Line Access Multiplexer (DSLAM)
  • Layer 2.1 – ATM connectivity
  • Layer 2.2 – Point-to-Point Protocol over ATM (PPPoA), Point-to-Point Protocol over Ethernet (PPPoE), RFC1483 Bridging, or RFC1483 Routing
  • Layer 3 – IP
The easiest way to determine which layer you should begin troubleshooting is to issue the command show ip interface brief. The output of this command differs slightly depending on your configuration.
827-ESC#show ip interface brief
Interface     IP-Address     OK?     Method   Status     Protocol
ATM0          unassigned     YES     manual    up         up
ATM0.1        unassigned     YES     unset     up         up
Ethernet0     10.10.10.1     YES     manual    up         up
If the statuses of ATM0 and ATM0.1 are up and the protocol is up, begin troubleshooting at Layer 2.
If the ATM interfaces are down, or if they keep coming up and then going down (they don't stay up and up), begin troubleshooting at Layer 1.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Layer 1 Issues

Is the carrier detect (CD) light on the front panel of the Cisco DSL Router on or off?

If the CD light is on, go to the Layer 2 Issues section of this document.
If the CD light is off, continue with the next question.

Is your ISP using a DSLAM that supports the Alcatel chipset?

Verify this information with your ISP.

Is the DSL port on the back of the Cisco DSL Router plugged into the DSL wall jack?

If the DSL port is not plugged into the DSL wall jack, connect the port to the wall with a 4-pin or 6-pin RJ-11 cable. This is a standard telephone cable.

Is the ATM interface in an administratively down state?

Issue this command in enable mode on the router in order to determine if the ATM0 interface is administratively down.
Router#show interface atm 0
ATM0 is administratively down, line protocol is down
<... snipped ...> 
If the ATM0 interface status is administratively down, issue the no shutdown command under the ATM0 interface.
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#interface atm 0
Router(config-if)#no shut
Router(config-if)#end
Router#write memory

Is the cable pinout correct?

If the ATM0 interface status is down and down, the router does not see a carrier on the ADSL line. This generally indicates one of two issues:
  • The active pins on the DSL wall jack are incorrect.
  • Your ISP has not turned up a DSL service on this wall jack.
Cisco DSL Router xDSL Port Pinouts
The RJ-11 connector provides an xDSL connection to external media via a standard RJ-11 6-pin modular jack.

Pin
3 ------ XDSL_Tip
4 ------ XDSL_Ring

In order to determine if the ATM0 interface is down and down, issue the show interface atm 0 command from enable mode of the router:
Router#show interface atm 0
ATM0 is down, line protocol is down
<... snipped ...>
If the ATM interface is down and down—not administratively down—check the pinout of your DSL wall jack. The DSL Router uses a standard RJ-11 (4-pin or 6-pin) cable to provide the ADSL connection to the wall jack. The center pair of pins on the RJ-11 cable is used to carry the ADSL signal (pins 3 and 4 on a 6-pin cable, or pins 2 and 3 on a 4- pin cable).
If you are sure you have the right pins on the wall jack and the ATM0 interface is still down and down, replace the RJ-11 cable between the ADSL port and your wall jack. If the interface is still down and down after you replace the RJ-11 cable, contact your ISP and have the ISP verify that DSL service has been enabled on the wall jack you use.
If you are not sure what pins on your wall jack are active, ask your ISP.

Do you have the correct power supply for the Device?

If you have verified that your ADSL cable is good and that you have the correct pinouts, the next step is to make sure you have the correct power supply for the 827.
Note: The 827 does not use the same power supply as other 800 Series Routers.
In order to determine if you have the correct power supply, on the back of the power adapter look for Output +12V 0.1A, -12V 0.1A, +5V 3A, -24V 0.12A, and -71V 0.12A. If your power supply is missing the +12V and -12V feeds, then it is for a different Cisco 800 Series Router and does not work on the 827. Note that if you use the wrong power supply, the Cisco 827 powers up but is unable to train up (connect) to the ISP DSLAM.

Is the DSL operating-mode correct?

If everything up to this point in the Layer 1 troubleshooting procedure is correct, the next step is to make sure you have the correct DSL operating mode. Cisco recommends that you use dsl operating-mode auto if you are not sure what DMT technology your ISP uses. These are the commands to configure operating-mode autodetection:
Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#interface atm 0
Router(config-if)#dsl operating-mode auto
Router(config-if)#end
Router#write memory

Is the circuit tested/provisioned correctly?

Obtain this information from your ISP or telephone company.

Layer 2 Issues

Do you have the correct Permanent Virtual Circuit (PVC) values (VPI/VCI)?

Complete these steps in order to determine whether you have the correct virtual path identifier/virtual circuit identifier (VPI/VCI) values configured on the router.
  1. Verify your version of Cisco IOS® software.
    Important: This does not work with Cisco IOS Software Release 12.1(1)XB.
    Router#show version 
    
    !--- Used to determine your Cisco IOS software release.
    
    
    Cisco Internetwork Operating System Software
    IOS (tm) C820 Software (C820-OSY656I-M), Version 12.1(3)XG3, 
    EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
    
    !--- The two lines immediately preceding appear on one line on the router.
     
    TAC:Home:SW:IOS:Specials for info
    Copyright (c) 1986-2000 by cisco Systems, Inc.
    Compiled Wed 20-Dec-00 16:44 by detang
    Image text-base: 0x80013170, data-base: 0x80725044
    <... snipped ...>
  2. Configure the router for debug logging.
    Router#configure terminal 
    Enter configuration commands, one per line. End with CNTL/Z.
    Router(config)#logging console
    Router(config)#logging buffer
    Router(config)#service timestamp debug datetime msec
    Router(config)#service timestamp log datetime msec
    Router(config)#end
    Router#write memory
    Building configuration...
    [OK]
    Router#terminal monitor
    
  3. Enable debugging on the router.
    Router#debug atm events 
    ATM events debugging is on
    Router#
    2d18h:
    2d18h: RX interrupt: conid = 0, rxBd = 0x80C7EF74 length=52
    2d18h: Data Cell received on vpi = 8 vci = 35
    
    !--- Your VPI/VCI.
    
    2d18h:
    2d18h: RX interrupt: conid = 0, rxBd = 0x80C7EEC0 length=52
    2d18h: Data Cell received on vpi = 8 vci = 35
    2d18h:
    2d18h: RX interrupt: conid = 0, rxBd = 0x80C7EECC length=52
    2d18h: Data Cell received on vpi = 8 vci = 35
    2d18h:
    2d18h: RX interrupt: conid = 0, rxBd = 0x80C7EED8 length=52
    2d18h: Data Cell received on vpi = 8 vci = 35
  4. Make sure you have debug ATM events that run on the Cisco DSL Router, and then go to a working Internet connection and begin to ping the IP address your ISP statically assigned to you.
    It does not matter whether you have configured this IP address on the Cisco DSL Router. What is important is that your ATM interface is up/up and that you ping the IP address your ISP gave you. If you do not see the expected output after the ping test, contact your ISP for support.
  5. Disable debugging on the router.
    << wait 60 seconds >>
    Router#undebug all 
    
    !--- Used to turn off the debug events.
    
    
    All possible debugging has been turned off.
    Verify your VPI/VCI values, and then make the necessary changes to your configuration. If you do not see output during the 60 seconds of debugging, contact your ISP.

Can you ping the default gateway?

In a bridged environment, pinging the default gateway is a good test of connectivity. In general, if you can ping to your default gateway, you know that Layer 1 and Layer 2 services function properly. Issue the ping command:
Router#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
.!!!! 
Success rate is 80 percent (4/5), round-trip min/avg/max = 44/44/48 ms
Router#

or

Router#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!! 
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/44/48 ms
Router#
A successful ping takes one of two forms. The first form shows an 80 percent success rate. The first ping packet that is sent is lost (.!!!!). This is a successful ping—the first packet is lost while a Layer 2 to Layer 3 binding is created via address resolution protocol (ARP). The second form of the ping is a 100 percent success rate, which is indicated by five exclamation points (!!!!!).
If your success rate is 80 to 100 percent, try to ping a valid Internet address (198.133.219.25 is www.cisco.com). If you can ping the default gateway from the router but you cannot ping another Internet address, make sure you have only one static default route in the configuration (for example, ip route 0.0.0.0 0.0.0.0 192.168.1.1).
For the previous example, if you already have a correct static default route and cannot ping Internet addresses, contact your ISP to resolve the routing issue.
If the ping test fails (a ping success rate of 0 percent), you see output similar to this:
Router#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
..... 
Success rate is 0 percent (0/5)

Router#

BGP Aggregate-Address

The BGP aggregate-address can be used to summarise a set of networks into a single prefix. In the diagram below, I’m going to use R2 to summarise prefixes inside AS20 & AS30, then send the summary to R3.

Let’s hop onto R2 and set this up.
R2(config)#router bgp 10
R2(config-router)#aggregate-address 206.25.128.0 255.255.128.0 summary-only
We configure the summary using the aggregate-address command.  The significance of the summary-only statement is there to ensure that longer-prefixes inside of the aggregate address are suppressed before sending BGP updates out to R3. Without it, longer prefixes are advertised as well as the aggregate-address.
The only rule for advertising an aggregate-address is that we must have at least one of the longer prefixes inside of our BGP table.  Obviously, we meet this because R1 & R4 are advertising the specific prefixes to R2.  Anyhow, let’s have a look at R2’s BGP table now we have configured the aggregate-address.
R2#sh ip bgp
BGP table version is 8, local router ID is 195.26.10.9
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 195.26.10.0/30   0.0.0.0                  0         32768 i
*> 195.26.10.4/30   0.0.0.0                  0         32768 i
*> 195.26.10.8/30   0.0.0.0                  0         32768 i
*> 206.25.128.0/17  0.0.0.0                            32768 i
s> 206.25.160.0/19  195.26.10.5              0             0 20 i
s> 206.25.224.0/19  195.26.10.1              0             0 30 i
The output also shows an s> by two of the networks; and from using the status codes at the top we can see this means these routes have been suppressed.  This is because the “summary-only” keyword on the aggregate-address command forces longer prefixes to be suppressed.  We can verify this by checking R3’s BGP table.
R3#sh ip bgp
BGP table version is 7, local router ID is 195.26.10.9
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 195.26.10.0/30   195.26.10.10             0             0 10 i
*> 195.26.10.4/30   195.26.10.10             0             0 10 i
*> 195.26.10.8/30   0.0.0.0                  0         32768 i
*                   195.26.10.10             0             0 10 i
*> 206.25.128.0/17 195.26.10.10             0             0 10 i

R3#sh ip bgp 206.25.128.0
BGP routing table entry for 206.25.128.0/17, version 6
Paths: (1 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  10, (aggregated by 10 2.2.2.2)
    195.26.10.10 from 195.26.10.10 (2.2.2.2)
      Origin IGP, metric 0, localpref 100, valid, external, atomic-aggregate, best
This output also highlights some other important points when looking at aggregation. We can now see that we have lost some path information about our two networks from AS20 and AS30. The aggregate shows these routes are now in AS10 (Although this sentence isn’t technically correct, this is what it looks like when first glancing at the information in the BGP table above; a technical explanation is given later).
Because the atomic-aggregate attribute is there; it signifies that the NLRI should not be de-aggregated, as the advertising router (2.2.2.2)  has chosen to advertise a less-specific prefix for a range of networks. It also means that some of the paths to the destination may not follow the AS_PATH that has been advertised.
As you know, the AS_PATH is used in BGP as a loop prevention mechanism.  So because this information is now lost, we could inherit a loop.  Cisco has provided a solution to this situation by breaking down the AS_PATH attribute into two tuples (sub-components of the AS_PATH).  These are the AS_SEQUENCE and AS_SET:
  • AS_SEQUENCE.  This is the ordered list of AS’s that were used in the path to reach the destination.
  • AS_SET.  This is an unordered list of AS’s that were used in the path to reach the destination.
When a BGP speaker aggregates multiple AS_PATH’s, he will select the longest, most common shared chain of AS_SEQUENCE tuples and put this into the AS_PATH attribute for the aggregated prefix (in our topology this is just AS10).  Any AS’s that are uncommon to each prefix are put into the AS_SET attribute. The end result is that any BGP speaker will be able to detect a loop because each AS that has been traversed, will either be in the AS_PATH or AS_SET; and if the speaker see’s his AS in any of these attributes, he can be confident a loop has occurred.  Let’s enable this feature now.
R2(config-router)#aggregate-address 206.25.128.0 255.255.128.0 summary-only as-set

R3#sh ip bgp
BGP table version is 8, local router ID is 195.26.10.9
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 195.26.10.0/30 195.26.10.10 0 0 10 i
*> 195.26.10.4/30 195.26.10.10 0 0 10 i
*> 195.26.10.8/30 0.0.0.0 0 32768 i
* 195.26.10.10 0 0 10 i
*> 206.25.128.0/17 195.26.10.10 0 0 10 {20,30} i

R3#sh ip bgp 206.25.128.0
BGP routing table entry for 206.25.128.0/17, version 8
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
Not advertised to any peer
10 {20,30}, (aggregated by 10 2.2.2.2)
195.26.10.10 from 195.26.10.10 (2.2.2.2)
Origin IGP, metric 0, localpref 100, valid, external, best
On R2 the most common AS_SEQUENCE towards networks to 206.25.160.0/19 and 206.25.224.0/19 was just AS 10. So this was inserted into the AS_PATH attribute. The uncommon AS’s to both of these networks is AS20 & 30.  These AS’s were inserted into the AS_SET (as shown by the numbers in the squiggly brackets).  So there we go.  We have loop detection with an aggregated prefix.

Reading and Understanding the OSPF Database

Introduction

OSPF, being a link-state protocol, allows for every router in the network to know of every link and OSPF speaker in the entire network. From this picture each router independently runs the Shortest Path First (SPF) algorithm to determine the best path through the network. All of this information is stored in the "Link State Database" (LSDB). Every network engineer has seen the LSDB at some point by running show ip ospf database but few actually know how to read the details. By looking only at the LSDB we should have enough information to draw a topology diagram from scratch.


Link State Advertisements

OSPF uses "Link State Advertisements" (LSAs) to provide information about links and link-costs to neighboring OSPF speakers. OSPF defines multiple LSAs, which all serve a different purpose.
LSA TypeWho Generates the LSA?What is Accomplished?
Type 1 -
Router LSA
Every router in every areaHow routers advertise their connected interfaces
Type 2 -
Network LSA
DRs on all non-point-to-point linksThe DR collects all the Type 1 LSAs and sends out a single Type 2 representing all of the routers on the link. This is used to build the Shortest Path Tree
Type 3 -
Network Summary LSA
Area Border Routers (ABRs)ABRs send a single LSA representing all of the Type 1 and Type 2 LSAs in an area. This reduces the number of LSAs on the routers in other areas.
Type 4 -
ASBR Summary LSA
ABRs connected to an area where external routes (Type 5) are originatedType 4 LSAs are sent to other areas to build the Shortest Path Tree to an ASBR.
Type 5 -
AS External LSA
Routers with the redistribute command that are not in a NSSA areaThis represents any external routes redistributed into OSPF.
Type 7 -
NSSA External LSA
Routers with the redistribute command that are in a NSSA areaType 5 LSAs are not allowed in Stub Areas. Type 7 LSAs allow external information to pass through NSSA areas.


Building the Topology

Starting on a router named r120 we can get a high level overview of the network (or at least our Area).
First, who are we (what is our Router ID)?
r120#show ip ospf data
            OSPF Router with ID (10.0.0.120) (Process ID 1)
Next, who are the other routers in our area?
                Router Link States (Area 1)
Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.111      10.0.0.111      600         0x8000023A 0x0092B3 1
10.0.0.112      10.0.0.112      1246        0x80000234 0x009CAC 1
10.0.0.113      10.0.0.113      148         0x8000022C 0x004399 3
10.0.0.120      10.0.0.120      152         0x80000240 0x0046CB 1
This tells us there are four routers in Area 1. The router with RID 10.0.0.113 has 3 links in Area 1, every one else has only 1 link.
Next, who are all of the DRs in this Area? What network segments do they represent?
                Net Link States (Area 1)
Link ID         ADV Router      Age         Seq#       Checksum
192.168.1.112   10.0.0.112      1862        0x80000237 0x00D860
192.168.7.113   10.0.0.113      12          0x80000001 0x00E8F5
Routers 10.0.0.112 and 10.0.0.113 are the router IDs of the DRs for two segments. 192.168.1.112 and 192.168.7.113 represent the IP address of the DR on that segment. We will see later that if we were to look at the Router LSA for 10.0.0.112, for example, we would see 192.168.1.112 as one of the interfaces owned by that router.
The Summary Network LSAs (Type 3) are generated by the ABRs and will give us information about every segment in the network, outside of our Area. Type 1 and Type 2 LSAs are not flooded beyond an ABR. The ABR is responsible for taking all of the information in Type 1 and Type 2 LSAs and repackaging them into Type 3 LSAs.
                Summary Net Link States (Area 1)
Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.119      10.0.0.111      1215        0x8000022A 0x00A845
10.0.0.119      10.0.0.112      1862        0x80000229 0x00A449
192.168.0.0     10.0.0.111      1215        0x80000234 0x00D842
192.168.0.0     10.0.0.112      1862        0x80000233 0x00D446
192.168.2.0     10.0.0.111      1215        0x80000234 0x0027E7
192.168.2.0     10.0.0.112      1862        0x80000233 0x0023EB
192.168.3.0     10.0.0.111      1215        0x80000232 0x008481
192.168.3.0     10.0.0.112      1862        0x80000232 0x007E86
192.168.4.0     10.0.0.111      1215        0x80000232 0x00798B
192.168.4.0     10.0.0.112      1862        0x80000232 0x007390
192.168.5.0     10.0.0.111      1215        0x80000232 0x006E95
192.168.5.0     10.0.0.112      1862        0x80000232 0x00689A
192.168.6.0     10.0.0.111      1215        0x80000231 0x00C930
192.168.6.0     10.0.0.112      1862        0x80000231 0x00C335
From this we know Area 1 has two ABRs with RIDs 10.0.0.111 and 10.0.0.112. We also see a total of 7 segments in the entire OSPF network. Each network is seen twice because each ABR generates its own LSA. Two ABRs means two LSAs.
Type 4, Summary Network LSAs are next. These are generated by the ABRs (10.0.0.111 and 10.0.0.112) to represent any routers or ABRs outside of our Area that is passing along Type 5 (external LSAs). This may be a little confusing at this point but it will make more sense when we start working through the topology.
Summary ASB Link States (Area 1)
Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.114      10.0.0.111      1215        0x80000232 0x00E915
10.0.0.114      10.0.0.112      1862        0x80000232 0x00E31A
Finally the external routes are represented by Type 5 LSAs.
                Type-5 AS External Link States
Link ID         ADV Router      Age         Seq#       Checksum Tag
172.16.0.113    10.0.0.113      631         0x80000001 0x00F006 0
172.16.0.118    10.0.0.114      678         0x80000001 0x009775 0
Here we see two different prefixes being redistributed into OSPF. The ADV Router is either the configured with the redistribute command (10.0.0.113) or an ABR connected to a NSSA area where redistribution is happening.
With this information let's build some high level topologies. First, start with what we know from the Type 1 LSAs (all the routers in our area)
area1.png
We will skip the Type 2 Network LSAs for now and go to the Type 3 Summary Network LSAs. This gives us the other subnets in the network and the ABRs for Area 1
abrs.png
The Type 4 ASBR Summary LSAs let us know that 10.0.0.114 is also an ABR on Area 0
asbr-summary.png
and finally, the Type 5's tell us about the externals. Using the "ADV Router" field we can figure out where those routes come from.
externals.png
Just from looking at the summary information in the database we've been able to put a lot of information together. Now we can start looking into the LSAs to get an idea of what the connectivity of Area 1 looks like.
We will start with router r120. Since we don't know anything about the links r120 has, we start with a router with no connections.
r120.png
To see the connections on r120 we'll need to look at the Router LSA that is generated by r120 (remember: Router LSAs are represented by the Router ID)
r120#show ip ospf database router 10.0.0.120
            OSPF Router with ID (10.0.0.120) (Process ID 1)
                Router Link States (Area 1)
  LS age: 408
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.120
  Advertising Router: 10.0.0.120
  LS Seq Number: 8000023C
  Checksum: 0x815
  Length: 36
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.7.113
     (Link Data) Router Interface address: 192.168.7.120
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

From this we know a few things:
1.) There is a single interface with IP 192.168.7.120
2.) There is a least one other router on this segment with IP 192.168.7.113, and this is the DR
2a.) Because there is a DR, we know this interface is multi-access (not point-to-point)
3.) The Router ID (10.0.0.120) is not advertised in OSPF (becuase there is no link information representing the router ID)
4.) The metric we are advertising is 10
We have an IP and a DR, but we don't know the subnet mask or which router in Area 1 owns the DR IP address. There is where the Type 2 LSA comes in. Remember the Type 2 is generated by the DR for a segment, and represents that segment, so we look for the segment DR.
r120#show ip ospf data network 192.168.7.113
            OSPF Router with ID (10.0.0.120) (Process ID 1)
                Net Link States (Area 1)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 93
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 192.168.7.113 (address of Designated Router)
  Advertising Router: 10.0.0.113
  LS Seq Number: 80000004
  Checksum: 0xE2F8
  Length: 32
  Network Mask: /24
        Attached Router: 10.0.0.113
        Attached Router: 10.0.0.120
A lot of great information here. First, we see the advertising router, which is the Router ID of the DR. In this case it's 10.0.0.113. Now we know who r120 is attached to. We also see the network mask (/24) and all of the routers on the segment. In this case only r120 and 10.0.0.113 are on the segment. If there were other routers on this segment we would see their Router IDs in the "Attached Router" list. So let's update the topology diagram.
first-segment.png
Now we can look at the Router LSA of 10.0.0.113
r120#show ip ospf data router 10.0.0.113
            OSPF Router with ID (10.0.0.120) (Process ID 1)
                Router Link States (Area 1)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 395
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.113
  Advertising Router: 10.0.0.113
  LS Seq Number: 80000256
  Checksum: 0x5465
  Length: 60
  AS Boundary Router
  Number of Links: 3
    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.0.0.113
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.7.113
     (Link Data) Router Interface address: 192.168.7.113
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.1.112
     (Link Data) Router Interface address: 192.168.1.113
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
Here we see three interfaces: 10.0.0.113, 192.168.7.113 and 192.168.1.113. We see that 10.0.0.113 is a "Stub Network". This does not have any relationship to a Stub Area, a "stub network" is simply an interface with no OSPF neighbors on it. We can also see that we are not the DR on the segment for 192.168.1.113. Let's take a look at the Type 2 for that segment. Remember, the Type 2 is represented by the DR for that segment.
r120#show ip ospf data network 192.168.1.112
            OSPF Router with ID (10.0.0.120) (Process ID 1)
                Net Link States (Area 1)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 161
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 192.168.1.112 (address of Designated Router)
  Advertising Router: 10.0.0.112
  LS Seq Number: 80000261
  Checksum: 0x848A
  Length: 36
  Network Mask: /24
        Attached Router: 10.0.0.112
        Attached Router: 10.0.0.111
        Attached Router: 10.0.0.113
We know that routers 10.0.0.112, 10.0.0.111 and 10.0.0.113 are all attached to this segment. Now we can look at the Router LSAs for routers 10.0.0.112 and 10.0.0.111. This will provide us with their interface IPs as well as any Stub Networks we haven't seen yet.
r120#show ip ospf data router 10.0.0.111
            OSPF Router with ID (10.0.0.120) (Process ID 1)
                Router Link States (Area 1)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 1004
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.111
  Advertising Router: 10.0.0.111
  LS Seq Number: 80000264
  Checksum: 0x3EDD
  Length: 36
  Area Border Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.1.112
     (Link Data) Router Interface address: 192.168.1.111
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
r120#show ip ospf data router 10.0.0.112
            OSPF Router with ID (10.0.0.120) (Process ID 1)
                Router Link States (Area 1)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 1444
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.112
  Advertising Router: 10.0.0.112
  LS Seq Number: 8000025E
  Checksum: 0x48D6
  Length: 36
  Area Border Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.1.112
     (Link Data) Router Interface address: 192.168.1.112
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
Now we know that the DR, 192.168.1.112, is router 10.0.0.112. We know that router 10.0.0.111 is also connected to the 192.168.1.0/24 segment with IP 192.168.1.111. We now know the entire topology for Area 1.
area1-complete.png
We know everything there is to know about Area 1. There is nothing to learn from router 10.0.0.113, since all of the links on that router are discovered. The next point to continue mapping the network would be on one of the ABRs. We will start with 10.0.0.112, or r112. Since r112 is an ABR it will have Type 1, 2, 3 and 4 information for both Area 1 and Area 0. We will want to focus on the Area 0 information. Let's get started by looking at our own Router LSA
       OSPF Router with ID (10.0.0.112) (Process ID 1)
                Router Link States (Area 0)
  LS age: 720
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.112
  Advertising Router: 10.0.0.112
  LS Seq Number: 80000262
  Checksum: 0x20FD
  Length: 36
  Area Border Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.0.111
     (Link Data) Router Interface address: 192.168.0.112
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
Looks like we only have link 192.168.0.112 in this area, and we are not the DR on this segment. Now we take a look at the Type 2 LSA for this segment.
r112#show ip ospf data network 192.168.0.111
            OSPF Router with ID (10.0.0.112) (Process ID 1)
                Net Link States (Area 0)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 388
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 192.168.0.111 (address of Designated Router)
  Advertising Router: 10.0.0.111
  LS Seq Number: 80000261
  Checksum: 0x759F
  Length: 36
  Network Mask: /24
        Attached Router: 10.0.0.111
        Attached Router: 10.0.0.110
        Attached Router: 10.0.0.112
We see that the advertising router represents the Router ID of the DR, or 10.0.0.111, who we already know is the other ABR for Area 1. We also see that there is a third router on this segment with Router ID 10.0.0.110.  Let's get the interface information from 10.0.0.111
r112#show ip ospf data router 10.0.0.111
            OSPF Router with ID (10.0.0.112) (Process ID 1)
                Router Link States (Area 0)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 700
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.111
  Advertising Router: 10.0.0.111
  LS Seq Number: 80000268
  Checksum: 0x1605
  Length: 36
  Area Border Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.0.111
     (Link Data) Router Interface address: 192.168.0.111
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

One interface here, with IP 192.168.0.111. This is the same segment as r112 and Router 10.0.0.110. Finally, let's look at 10.0.0.110

r112#show ip ospf data router 10.0.0.110
            OSPF Router with ID (10.0.0.112) (Process ID 1)
                Router Link States (Area 0)
  LS age: 1232
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.110
  Advertising Router: 10.0.0.110
  LS Seq Number: 80000263
  Checksum: 0x4E09
  Length: 48
  Number of Links: 2
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.0.111
     (Link Data) Router Interface address: 192.168.0.110
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.2.110
     (Link Data) Router Interface address: 192.168.2.110
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
We see that 10.0.0.110 owns the IP 192.168.0.110. We also see a second interface, 192.168.2.110. On this second segment 10.0.0.110 is the DR. Let's up the topology diagram for Area 0:
area0-first-segment.png
Let's keep moving down. First, we look at the Type 2 from 192.168.2.110, then we'll look at the Type 1 LSAs from the other routers on this segment.
r112#show ip ospf data net 192.168.2.110
            OSPF Router with ID (10.0.0.112) (Process ID 1)
                Net Link States (Area 0)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 781
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 192.168.2.110 (address of Designated Router)
  Advertising Router: 10.0.0.110
  LS Seq Number: 80000261
  Checksum: 0x1779
  Length: 32
  Network Mask: /24
        Attached Router: 10.0.0.110
        Attached Router: 10.0.0.114

And now the Type 1 for 10.0.0.114
r112#show ip ospf data router 10.0.0.114
            OSPF Router with ID (10.0.0.112) (Process ID 1)
                Router Link States (Area 0)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 889
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.114
  Advertising Router: 10.0.0.114
  LS Seq Number: 80000265
  Checksum: 0x1178
  Length: 48
  Area Border Router
  AS Boundary Router
  Number of Links: 2
    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 192.168.3.0
     (Link Data) Network Mask: 255.255.255.0
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.2.110
     (Link Data) Router Interface address: 192.168.2.114
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
We have two links again, one connected to the segment with 10.0.0.110 and a new segment. Again, notice that the segment 192.168.3.0 is a Stub Network, so there are no other OSPF speakers on this link. Now, before we think we've finished up, we haven't looked at the Type 3 LSAs that are generated by ABRs. We don't know if there is another ABR in Area 0, so let's look
                Summary Net Link States (Area 0)
Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.113      10.0.0.111      397         0x80000264 0x004394
10.0.0.113      10.0.0.112      17          0x80000264 0x003D99
10.0.0.119      10.0.0.114      1276        0x80000258 0x00715F
192.168.1.0     10.0.0.111      397         0x80000263 0x006F7B
192.168.1.0     10.0.0.112      792         0x80000262 0x006B7F
192.168.4.0     10.0.0.114      1036        0x80000261 0x0040A6
192.168.5.0     10.0.0.114      1036        0x80000261 0x0035B0
192.168.6.0     10.0.0.114      1036        0x80000260 0x00904B
192.168.7.0     10.0.0.111      397         0x80000265 0x008D4B
192.168.7.0     10.0.0.112      17          0x80000265 0x008750
Before addressing the new routes here, you can see the Type 3 LSAs in Area 0 that are generated by the two ABRs, 10.0.0.111 and 10.0.0.112. These routes here are all of the routes in Area 1, that we just described. This is how an ABR hides the details of an Area from the rest of the network.
We see four new networks all coming from the ABR 10.0.0.114. Now we can update our topology diagram of Area 0.
area0-complete.png
Again, we need to jump to our ABR to see what's going on in the rest of the network.
On r114 things get interesting. Looking at the LSAs we see that r114 is in 3 areas.
r114# show ip ospf data | i States
                Router Link States (Area 0)
                Net Link States (Area 0)
                Summary Net Link States (Area 0)
                Summary ASB Link States (Area 0)
                Router Link States (Area 2)
                Net Link States (Area 2)
                Summary Net Link States (Area 2)
                Router Link States (Area 3)
                Net Link States (Area 3)
                Summary Net Link States (Area 3)
                Type-7 AS External Link States (Area 3)
                Type-5 AS External Link States
But things look a little fishy in Area 2 and Area 3. Notice that Area 2 has no "Summary ASB Link States" (Type 4). Also notice that Area 3 has "Type-7 AS External Link States".
Let's start with Area 2.
If we are in an area that does not have any Type 4 LSAs, that area can not have external routes. OSPF works by linking the information carried in a Type-4 LSA to the information carried in the Type-5 LSA to build a tree. OSPF Stub areas do not allow any external information, matching this description. We can assume that Area 2 is a Stub Area.
Again, let's get a lay of the land by looking at the Router LSA summaries.
                Router Link States (Area 2)
Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.114      10.0.0.114      23          0x8000026C 0x00B250 1
10.0.0.115      10.0.0.115      1584        0x80000269 0x00B350 1
10.0.0.116      10.0.0.116      745         0x8000026E 0x00F225 2
10.0.0.119      10.0.0.119      706         0x8000026B 0x0074E7 2
We can see there are four routers in the area. 10.0.0.116 and 10.0.0.119 both have two links.
Now let's look at our Router LSA
r114#show ip ospf data router 10.0.0.114
               
     Router Link States (Area 2)
  LS age: 1234
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.114
  Advertising Router: 10.0.0.114
  LS Seq Number: 80000265
  Checksum: 0xC049
  Length: 36
  Area Border Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.5.114
     (Link Data) Router Interface address: 192.168.5.114
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
Now the Type-2, Network LSA
r114#show ip ospf data net 192.168.5.114
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Net Link States (Area 2)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 1312
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 192.168.5.114 (address of Designated Router)
  Advertising Router: 10.0.0.114
  LS Seq Number: 80000263
  Checksum: 0xD429
  Length: 36
  Network Mask: /24
        Attached Router: 10.0.0.114
        Attached Router: 10.0.0.115
        Attached Router: 10.0.0.116
Next, the Router LSA (Type-1) of our Attached Routers
r114#show ip ospf data router 10.0.0.115
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Router Link States (Area 2)
  LS age: 1000
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.115
  Advertising Router: 10.0.0.115
  LS Seq Number: 80000263
  Checksum: 0xBF4A
  Length: 36
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.5.114
     (Link Data) Router Interface address: 192.168.5.115
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
r114#show ip ospf data router 10.0.0.116
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Router Link States (Area 2)
  LS age: 269
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.116
  Advertising Router: 10.0.0.116
  LS Seq Number: 80000268
  Checksum: 0xFE1F
  Length: 48
  Number of Links: 2
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.5.114
     (Link Data) Router Interface address: 192.168.5.116
      Number of MTID metrics: 0
      TOS 0 Metrics: 10
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.6.119
     (Link Data) Router Interface address: 192.168.6.116
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
Let's digest this here. First 10.0.0.115 has a single interface with IP 192.168.5.115. Next we see 10.0.0.116 with two interfaces, 192.168.5.116 and 192.168.6.116. Let's update the topology and then take a look at the segment with DR 192.168.6.119. This must be the router 10.0.0.119, the only router in Area 2 we haven't looked at yet.
area2-1.png
Remember that routers 10.0.0.116 and 10.0.0.119 both had two links. We have discovered the two links on 10.0.0.116, but we still have one link on 10.0.0.119 to find, so let's look at the Type-1 for 10.0.0.119
r114#show ip ospf data router 10.0.0.119
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Router Link States (Area 2)
  LS age: 1272
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.119
  Advertising Router: 10.0.0.119
  LS Seq Number: 8000026B
  Checksum: 0x74E7
  Length: 48
  Number of Links: 2
    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.0.0.119
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.6.119
     (Link Data) Router Interface address: 192.168.6.119
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

Now we have discovered all of the links in Area 2. Because Area 2 still receives Type-3 LSAs, it will know about all of the links in the OSPF network. The only thing it will not have will be the external routes that are injected into OSPF.
area2-complete.png
Remember that r114 was in three Areas: Area 0, Areas 2 and Area 3. Before digging into Area 3, let's take another look at the LSAs that exist in Area 3.
r114#show ip ospf data | i Area 3
                Router Link States (Area 3)
                Net Link States (Area 3)
                Summary Net Link States (Area 3)
                Type-7 AS External Link States (Area 3)
We see Type 1 (Router Link States), Type 2 (Net Link State), Type 3 (Summary Net Link) and Type 7 (Type-7 AS External). Similar to Area 2, we do not see Type 4 or Type 5 LSAs. However we see Type-7 LSAs, which only exist in Not So Stubby Areas (NSSA). In a normal Stub area external route information is not allowed. NSSA areas allow us to have all of the features of a Stub area (no externals from other parts of the network) while still allowing external information to be originated in this area. To accomplish this, NSSA areas do not allow Type-5 (normal external LSAs) and use a special Type-7 LSAs. When the Type-7 arrives on the ABR (r114 in this case), the ABR must convert this Type-7 to a Type-5 for the rest of the network. We'll take a look at this process in a little while.
First, let's see how many routers and links are in Area 3
               Router Link States (Area 3)
Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.114      10.0.0.114      1610        0x80000335 0x00B37A 1
10.0.0.117      10.0.0.117      1344        0x80000333 0x00A881 1
10.0.0.118      10.0.0.118      802         0x80000332 0x00AE77 1
We have 3 routers, each with 1 link. Now, as always, take a look at our Type-1
r114#show ip ospf data router 10.0.0.114
                Router Link States (Area 3)
  LS age: 723
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.114
  Advertising Router: 10.0.0.114
  LS Seq Number: 80000334
  Checksum: 0xB579
  Length: 36
  Area Border Router
  AS Boundary Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.4.117
     (Link Data) Router Interface address: 192.168.4.114
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
A single link with IP 192.168.4.114 and DR 192.168.4.117. Now the Type-2
r114#show ip ospf data net 192.168.4.117
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Net Link States (Area 3)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 635
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 192.168.4.117 (address of Designated Router)
  Advertising Router: 10.0.0.117
  LS Seq Number: 80000330
  Checksum: 0xCE50
  Length: 36
  Network Mask: /24
        Attached Router: 10.0.0.117
        Attached Router: 10.0.0.114
        Attached Router: 10.0.0.118
Here are three routers attached to this segment. r114, the DR and a third router. Now the Type-1 LSAs for the other routers.
r114#show ip ospf data router 10.0.0.117
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Router Link States (Area 3)
  LS age: 794
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.117
  Advertising Router: 10.0.0.117
  LS Seq Number: 80000333
  Checksum: 0xA881
  Length: 36
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.4.117
     (Link Data) Router Interface address: 192.168.4.117
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
r114#show ip ospf data router 10.0.0.118
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Router Link States (Area 3)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 257
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.118
  Advertising Router: 10.0.0.118
  LS Seq Number: 80000332
  Checksum: 0xAE77
  Length: 36
  AS Boundary Router
  Number of Links: 1
    Link connected to: a Transit Network
     (Link ID) Designated Router address: 192.168.4.117
     (Link Data) Router Interface address: 192.168.4.118
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
We see IPs 192.168.4.118 and 192.168.4.117. Here's the topology for Area 3.
area3-1.png
But let's not forget about the Type-7 LSAs we saw earlier. Because these are Type-7, we are not looking at external LSAs but nssa-external LSAs
r114#show ip ospf data nssa-external         
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Type-7 AS External Link States (Area 3)
  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 952
  Options: (No TOS-capability, Type 7/5 translation, DC)
  LS Type: AS External Link
  Link State ID: 172.16.0.118 (External Network Number )
  Advertising Router: 10.0.0.118
  LS Seq Number: 800000FF
  Checksum: 0xEC13
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 192.168.4.118
        External Route Tag: 0
This is the LSA representing the external network 172.16.0.118/32. Within this area the routers should send traffic for this destination to the Forwarding Address of 192.168.4.118. Once the LSA arrives on the ABR, r114, it will be converted into a Type-5 LSA and sent to all other areas (that aren't stubs, meaning that Area 2 will not see this LSA). We can confirm this by looking at the Type-5 LSAs
r114#show ip ospf data external 172.16.0.118
            OSPF Router with ID (10.0.0.114) (Process ID 1)
                Type-5 AS External Link States
  LS age: 146
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 172.16.0.118 (External Network Number )
  Advertising Router: 10.0.0.114
  LS Seq Number: 80000101
  Checksum: 0x9477
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 20
        Forward Address: 192.168.4.118
        External Route Tag: 0

Here we see the Type-5 originated not by 10.0.0.118, like the Type-7, but by 10.0.0.114. This is due to the Type-7 to Type-5 conversion. Since r114 is generating a new LSA it sets itself as the Advertising Router. You'll also notice that the Forwarding Address has remained the same. When other routers in the network build the tree to reach this external destination they will build to the best ABR to reach this network (since it would be part of a Type-3 LSA). For more information on Forwarding Addresses, there is a great doc on Cisco.com.
With all of this information we can finish the topology for Area 3
area3-complete.png
and then for the entire OSPF network.
complete-topo.png

Summary

Hopefully reading the OSPF topology is a little more clear now. The less obvious takeaways are how OSPF scales by hiding topology information. You noticed that in an area we have a large number of Type 1 and Type 2 LSAs. Outside of that area there is only a single Type-3 LSA generated by each ABR. We can also use Stub Areas to hide external information, keeping even less information in the LSDB of the routers in those areas.
Finally, think about how each LSA type links together. OSPF's SPF algorithm links different pieces of information together. For a router in Area 1 to reach the external route in Area 3, it has to look at the Type-5 that represents the external route. Then it has to look at the Type-4 representing the ABR on the area that the ASBR lives in. Then we have to look at the Type-3 to get to that remote ABR. Finally we look at the Type-1 and Type-2 LSAs in our area to determine how to get to our closest ABR.
Each LSA serves a specific purpose and they all fit together to supply end-to-end connectivity.