About Me

My photo
This blog has been created to share technical information, interesting innovations that I notice on daily basis and Architectural/Consulting overview of various technologies. My areas of interest, on which I would be blogging, are VMware, Microsoft and Citrix Technologies. I hope you will enjoy this blog and share your experience with me.

BugCheck CB on Windows 2003 Service Pack 2

Recommendation:
===============

==> In this case TCPIP.SYS driver was the cause for the BugCheck as the IRP was stuck at the TCPIP.SYS.
==> Apply MS KB 962238 to update TCPIP.SYS and AFD.SYS
==> Disable TCP Chimney Feature of Windows Server 2003 SP2. You can follow the article below to disable TCP Chimney either using the package mentioned in the article or you can refer to the workaround section for disabling it manually.

Debug Trace:
============

DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS (cb)
Caused by a driver not cleaning up completely after an I/O.
When possible, the guilty driver's name (Unicode string) is printed on
the bugcheck screen and saved in KiBugCheckDriver.
Arguments:

Arg1: b7bf92f2, The calling address in the driver that locked the pages or if the
                IO manager locked the pages this points to the dispatch routine of
                the top driver on the stack to which the IRP was sent.
Arg2: b7bfa933, The caller of the calling address in the driver that locked the
                pages. If the IO manager locked the pages this points to the device
                object of the top driver on the stack to which the IRP was sent.
Arg3: 8975d618, A pointer to the MDL containing the locked pages.
Arg4: 00000003, The number of locked pages.

Debugging Details:
------------------

PEB is paged out (Peb.Ldr = 7ffd500c).  Type ".hh dbgerr001" for details

PEB is paged out (Peb.Ldr = 7ffd500c).  Type ".hh dbgerr001" for details

FAULTING_IP:
afd!AfdAllocateMdlChain+bf
b7bf92f2 8b45dc          mov     eax,dword ptr [ebp-24h]

DEFAULT_BUCKET_ID:  DRIVER_FAULT

BUGCHECK_STR:  0xCB

PROCESS_NAME:  cognos.cgi

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 8084cc7b to 80827c83

STACK_TEXT: 
b49b2c38 8084cc7b 000000cb b7bf92f2 b7bfa933 nt!KeBugCheckEx+0x1b
b49b2c68 8094c4eb ffffffe5 898cbcb8 f1256323 nt!MmCleanProcessAddressSpace+0x551
b49b2cf0 8094c569 f1256323 b49b2d4c 8082e0d6 nt!PspExitThread+0x5f1
b49b2cfc 8082e0d6 898cbcb8 b49b2d48 b49b2d3c nt!PsExitSpecialApc+0x1d
b49b2d4c 80889867 00000001 00000000 b49b2d64 nt!KiDeliverApc+0x1ae
b49b2d4c 7c8285ec 00000001 00000000 b49b2d64 nt!KiServiceExit+0x56
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012f3bc 00000000 00000000 00000000 00000000 0x7c8285ec

STACK_COMMAND:  .bugcheck ; kb

FOLLOWUP_IP:
afd!AfdAllocateMdlChain+bf
b7bf92f2 8b45dc          mov     eax,dword ptr [ebp-24h]

SYMBOL_NAME:  afd!AfdAllocateMdlChain+bf

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: afd

IMAGE_NAME:  afd.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  48f76164

FAILURE_BUCKET_ID:  0xCB_afd!AfdAllocateMdlChain+bf

BUCKET_ID:  0xCB_afd!AfdAllocateMdlChain+bf

Followup: MachineOwner
---------

==>> Now, use the !search extension to find the virtual addresses that are associated with the MDL pointer.

13: kd> !search 8975d618
Debuggee is a kernel summary dump, some physical pages may not be present.
Searches will miss hits from those pages.
Searching PFNs in range 00000001 - 0022FFFE for [FFFFFFFF8975D618 - FFFFFFFF8975D618]

Pfn      Offset   Hit      Va       Pte     
- - - - - - - - - - - - - - - - - - - - - - - - - - -
000008A6 0000054C 8975D618 808A654C C0404530
00105EA6 0000016C 8975D618 876A616C C043B530
                876a6160+0xc     : Irp   -- Io, IRP packets
00105EF2 00000828 8975D718 876F2828 C043B790
                876f2000+0x828   : Io   (Free)  -- general IO allocations
00106EE8 00000F70 8975D698 886E8F70 C0443740
                886e8f60+0x10    : AfdC (Protected)  -- Afd connection structure
00107E22 000008DC 8975D698 896228DC C044B110
                896228a8+0x34    : File (Protected)  -- File objects
00107FC3 00000EE4 8975D698 897C3EE4 C044BE18
                897c3e18+0xcc    : TCPA  -- TCP/IP network protocol
00107FC3 00000EEC 8975D698 897C3EEC C044BE18
                897c3e18+0xd4    : TCPA  -- TCP/IP network protocol
00107FC3 00000EF4 8975D698 897C3EF4 C044BE18
                897c3e18+0xdc    : TCPA  -- TCP/IP network protocol
00107FC3 00000F04 8975D698 897C3F04 C044BE18
                897c3e18+0xec    : TCPA  -- TCP/IP network protocol
00107FC3 00000F1C 8975D698 897C3F1C C044BE18
                897c3e18+0x104   : TCPA  -- TCP/IP network protocol
001085BD 00000B68 8975D698 89DBDB68 C044EDE8
                89dbda78+0xf0    : TCPT  -- TCB pool
001085BD 00000BEC 8975D698 89DBDBEC C044EDE8
                89dbda78+0x174   : TCPT  -- TCB pool
00108E75 00000034 8975D618 8A675034 C04533A8
                8a675000+0x34    : Dump (LargePool)  -- Bugcheck dump allocations
00109211 00000CD8 8975D618 8AA11CD8 C0455088
                8aa11cc8+0x10    : MmLk  -- Bugcheck dump allocations
0010A79A 00000318 8975D618 8BF9A318 C045FCD0
                8bf9a2f0+0x28    : TCPr  -- TCP request pool
001DDAE0 0000082C 8975D618 B49B282C C05A4D90
001DDAE0 000008BC 8975D618 B49B28BC C05A4D90
001DDAE0 00000C2C 8975D618 B49B2C2C C05A4D90
001DDAE0 00000C4C 8975D618 B49B2C4C C05A4D90
Search done.

==>> For each virtual address (VA) found, look for an IRP signature. Do this by using the dd command with the VA minus one DWORD.

13: kd> dd B49B2C4C-4 l4
b49b2c48  b7bfa933 8975d618 00000003 87ac10e0

13: kd> dd B49B2C2C-4 l4
b49b2c28  b7bfa933 8975d618 00000003 00000000

13: kd> dd B49B28BC-4 l4
b49b28b8  b7bf92f2 8975d618 00000000 01330180

13: kd> dd B49B282C-4 l4
b49b2828  b7bfa933 8975d618 00000003 bad8b242

13: kd> dd 8BF9A318-4 l4
8bf9a314  876a6228 8975d618 00000000 00000000

13: kd> dd 8AA11CD8-4 l4
8aa11cd4  87708550 8975d618 007dd000 00000003

13: kd> dd 8A675034-4 l4
8a675030  b7bfa933 8975d618 00000003 45474100

13: kd> dd 89DBDBEC-4 l4
89dbdbe8  b7bfaf81 8975d698 00000000 00000000

13: kd> dd 89DBDB68-4 l4
89dbdb64  b7bfb4bb 8975d698 00000000 00000000

13: kd> dd 897C3F1C-4 l4
897c3f18  b7bfaf81 8975d698 00000000 00000000

13: kd> dd 897C3F04-4 l4
897c3f00  b7c08f11 8975d698 00000000 00000000

13: kd> dd 897C3EF4-4 l4
897c3ef0  b7bfb4bb 8975d698 00000000 00000000

13: kd> dd 897C3EEC-4 l4
897c3ee8  b7c077bb 8975d698 b7bfb4bb 8975d698

13: kd> dd 897C3EE4-4 l4
897c3ee0  b7bf9ebb 8975d698 b7c077bb 8975d698

13: kd> dd 896228DC-4 l4
896228d8  00000000 8975d698 00000000 00000000

13: kd> dd 886E8F70-4 l4
886e8f6c  00111030 8975d698 891005e0 8aa33030

13: kd> dd 876F2828-4 l4
876f2824  61436d4d 8975d718 00000000 808ab4d4

13: kd> dd 876A616C-4 l4
876a6168  01d80006 8975d618 00000000 00000000  ==>> $ Matches signature

13: kd> dd 808A654C-4 l4
808a6548  b7bfa933 8975d618 00000003 00000000

==>>     After a VA with an IRP signature has been found, use the !irp extension to find out what driver is pending on this IRP.

13: kd> !irp 876a6168
Irp is active with 4 stacks 3 is current (= 0x876a6220)
Mdl=8975d618: No System Buffer: Thread 00000000:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                                                Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                                                Args: 00000000 00000000 00000000 00000000
>[  f, 8]   0 e1 8aa33030 88c532a0 b7bfbb72-8acf31b0 Success Error Cancel pending
                       \Driver\Tcpip             afd!AfdRestartBufferReceiveWithUserIrp
                                                Args: 00002000 00010020 00000000 00000000
[  e, 5]   5  0 8ac68030 894e4c28 00000000-00000000   
                       \Driver\AFD
                                                Args: 00002000 00000000 20000020 00000000

==>> From the stack below, TCP Chimney feature is enabled on the server, disabling it will be good step to disable that.

13: kd> dd tcpip!DisableTCPChimney L1
b7c950d0  00000000

13: kd> dd tcpip!DisableRSS L1
b7c950d4  00000000

13: kd> dd tcpip!UseDma L1
b7c95fa0  00000101

13: kd> dd tcpip!DisableTaskOffload L1
b7c95e54  00000000

Reference Microsoft KB articles:

MS09-008: Vulnerabilities in DNS and WINS Server could allow spoofing
http://support.microsoft.com/kb/962238

An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
http://support.microsoft.com/default.aspx?scid=kb;EN-US;948496

Don’t forget to leave your comments :)

Thanks,
Aresh

No comments:

My Blog List