Tuesday, June 23, 2015

Samsung deliberately disabling Windows Update the way the user intends it to

Last updated 6/26/2015 - 4:49 PM EST.

-- Windows Update will be abbreviated as "WU" in text from myself.

First of all, I had this included in my post since the get-go, but it was overlooked as it wasn't at the beginning of the post. With that said, I'm moving it here, and clarifying a bit more. I was not the sole person involved, it was a multiple-person discovery. Here were the people involved:

wavly - The user that had the problem, and the reason we had anything to even discover in the first place.
BrianDrab - Assisted wavly in their Windows Update problem, and investigated with us why it was resetting and disabling the user from keeping it the setting they wanted to.
niemiro - Was largely involved in the discovery by investigating/reverse engineering SW Update.
zcomputerwiz  - Was largely involved in the discovery by suggesting registry auditing.
tom982 - Was largely involved in the discovery by investigating/reverse engineering SW Update.
Tekno Venus -  Was largely involved in the discovery by investigating/reverse engineering SW Update.
Me (Patrick Barker) -  Was involved in the discovery by further reverse engineering and investigating SW Update and its behavior after the above people, and creating the blog post.

I've also seen a few (very few) articles even say I was the individual who was helping with the Windows Update issue(s) wavly was having. For the record, I personally don't know a damn thing about the technicalities of Windows Update, how to fix broken updates, etc. The user that was assisting wavly with the Windows Update issue(s) was BrianDrab, as I had mentioned in this post, just apparently not mentioned enough (or clearly enough). I merely further investigated and reverse engineered SW Update, and brought Disable_Windowsupdate.exe and its silent behavior to light.

Onto the post...

On my home forum Sysnative, a user (wavly) was being assisted with a WU issue, which was going well, aside from the fact that wavly's WU kept getting randomly reset to "Check for updates but let me choose whether to download or install them" after every single reboot of Windows. It was figured out eventually after using auditpol.exe and registry security auditing (shown below later) that the program that was responsible for resetting WU was Disable_Windowsupdate.exe, which is part of Samsung's SW Update software.

SW Update is your typical OEM updating software that will update your Samsung drivers, the bloatware that came on your Samsung machine, etc. The only difference between other OEM updating software is, Samsung's disables WU from working as the user intends it to.

SW Update will install on:

Windows XP (all Service Packs) - Update service will not be installed whatsoever.
Windows Vista (x86/x64)
Windows 7/SP1 (x86/x64)
Windows 8/8.1 (x86/x64)

Do note that it does check for a Samsung environment, and if one is not detected, the program will in general run really buggy. A lot of its features won't drop or work as intended either, which is why a lot of manual work needs to be done to investigate this program.

What devices does SW Update run on?

Samsung notes:
SW Update allows you to download and install the newest drivers, updates, and software for your Windows PC.
So most likely only desktop and laptop type devices that run the Windows OS.

Uninstalling SW Update

UPDATE:  I've received confirmation from a Samsung NP350V5C-A06UK user (Windows 8.1) that uninstalling SW Update via the Programs and Features list does in fact remove all of its installed parts, including the service. With that said, it does indeed stop resetting Windows Update's settings after reboots. So the solution to having SW Update constantly reset your Windows Update settings and disabling it from working as you intended, is to simply uninstall SW Update.

-- Initially today I had this saying it did not stop it from resetting, but wavly got back to me and said they were mistaken.

First off, here's how it was found:

 A registry value was modified.  
      Security ID:          SYSTEM  
      Account Name:          PURGED  
      Account Domain:          WORKGROUP  
      Logon ID:          0x3E7  
      Object Name:          \REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update  
      Object Value Name:     UpdatesAvailableForDownloadLogon  
      Handle ID:          0xecc  
      Operation Type:          Registry value deleted  
 Process Information:  
      Process ID:          0x5c  
      Process Name:          C:\Windows\System32\svchost.exe  
 Change Information:  
      Old Value Type:          REG_DWORD  
      Old Value:          0  
      New Value Type:          -  
      New Value:          -  

And then shortly after...

 A registry value was modified.  
      Security ID:          SYSTEM  
      Account Name:          PURGED  
      Account Domain:          WORKGROUP  
      Logon ID:          0x3E7  
      Object Name:          \REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update  
      Object Value Name:     UpdatesAvailableForDownloadLogon  
      Handle ID:          0x135c  
      Operation Type:          New registry value created  
 Process Information:  
      Process ID:          0x5c  
      Process Name:          C:\Windows\System32\svchost.exe  
 Change Information:  
      Old Value Type:          -  
      Old Value:          -  
      New Value Type:          REG_DWORD  
      New Value:          0  

      Object Server:          Security  
      Object Type:          Key  
      Object Name:          \REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update  
      Handle ID:          0x144  
      Resource Attributes:     -  
 Process Information:  
      Process ID:          0x1ae4  
      Process Name:          C:\ProgramData\SAMSUNG\SWUpdate\Temp\Packages\BASW-A0394A05\64\Disable_Windowsupdate.exe  
 Access Request Information:  
      Transaction ID:          {00000000-0000-0000-0000-000000000000}  
      Accesses:          DELETE  
                     Query key value  
                     Set key value  
                     Create sub-key  
                     Enumerate sub-keys  
                     Notify about changes to keys  
                     Create Link  
      Access Reasons:          -  
      Access Mask:          0xF003F  
      Privileges Used for Access Check:     -  
      Restricted SID Count:     0  


There were other Object Value Names, such as:
  • CachedAUOptions
  • InstallInProgress,
  • UpdatesAvailableForInstallLogon 
  • UpdatesAvailableWithUiLogon 
  • UpdatesAvailableWithUiOrEulaLogon
  • FirmwareUpdatesNotDownloaded
  • FirmwareUpdatesNotInstalled
Anyway, moving on, let's take a look!

 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\AuthorizedCDFPrefix: ""  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Comments: "SW Update Setup"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Contact: "Samsung Electronics CO., LTD."  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\DisplayVersion: "2.2.9"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\HelpLink: "http://www.samsung.com"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\HelpTelephone: ""  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\InstallDate: "20150623"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\InstallLocation: ""  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\InstallSource: "C:\ProgramData\Samsung\SWUpdate\Temp\"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\ModifyPath: "MsiExec.exe /I{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Publisher: "Samsung Electronics CO., LTD."  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Readme: ""  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Size: ""  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\EstimatedSize: 0x00008172  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\UninstallString: "MsiExec.exe /I{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\URLInfoAbout: "http://www.samsung.com"  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\URLUpdateInfo: ""  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\VersionMajor: 0x00000002  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\VersionMinor: 0x00000002  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\WindowsInstaller: 0x00000001  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Version: 0x02020009  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\Language: 0x00000409  
 HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{AAFEFB05-CF98-48FC-985E-F04CD8AD620D}\DisplayName: "SW Update"  

Here's its basic information from a comparison of registry changes after installation.

 HKLM\SOFTWARE\Samsung\CurrentPath\20000: ""C:\Program Files\Samsung\SW Update\sManager.exe""  
 HKLM\SOFTWARE\Samsung\SW Update\AgentPath: "C:\ProgramData\Samsung\SW Update Service\SWMAgent.exe"  
 HKLM\SOFTWARE\Samsung\SW Update\InstallPath: "C:\Program Files\Samsung\SW Update\sManager.exe"  
 HKLM\SOFTWARE\Samsung\SW Update\TrafficDecentralize: "Y"  
 HKLM\SOFTWARE\Samsung\SW Update\LastORCAServerUpdateDateTime: "2015-06-22T02:28:42"  
 HKLM\SOFTWARE\Samsung\SW Update\AgentSleepSec: "300"  
 HKLM\SOFTWARE\Samsung\SWMCommon\FirstAgentExecDateTime: "2015-06-23T01:47:42"  
 HKLM\SYSTEM\ControlSet001\Services\SWUpdateService\Type: 0x00000110  
 HKLM\SYSTEM\ControlSet001\Services\SWUpdateService\Start: 0x00000002  
 HKLM\SYSTEM\ControlSet001\Services\SWUpdateService\ErrorControl: 0x00000001  
 HKLM\SYSTEM\ControlSet001\Services\SWUpdateService\ImagePath: "C:\ProgramData\Samsung\SW Update Service\SWMAgent.exe /SERVICE"  
 HKLM\SYSTEM\ControlSet001\Services\SWUpdateService\DisplayName: "SW Update Service"  
 HKLM\SYSTEM\ControlSet001\Services\SWUpdateService\ObjectName: "LocalSystem"  
 HKLM\SYSTEM\CurrentControlSet\Services\SWUpdateService\Type: 0x00000110  
 HKLM\SYSTEM\CurrentControlSet\Services\SWUpdateService\Start: 0x00000002  
 HKLM\SYSTEM\CurrentControlSet\Services\SWUpdateService\ErrorControl: 0x00000001  
 HKLM\SYSTEM\CurrentControlSet\Services\SWUpdateService\ImagePath: "C:\ProgramData\Samsung\SW Update Service\SWMAgent.exe /SERVICE"  
 HKLM\SYSTEM\CurrentControlSet\Services\SWUpdateService\DisplayName: "SW Update Service"  
 HKLM\SYSTEM\CurrentControlSet\Services\SWUpdateService\ObjectName: "LocalSystem"  

Here we can see some more information, such as its agent's sleep is set to 300 seconds, its first execution timestamp, and the creation of the "SW Update" service. I'll break down the service stuff:

Type (0x00000110): As far as I know, this implies it's a Win32 program that can be started by Windows' Service Controller, and that it obeys the service control protocol. This type of Win32 service runs in a process by itself.

Start: (0x00000002): This implies it's set to load or startup automatically for all startups, regardless of the service type. Its loader is the Service Control Manager, where as the 0x0 (boot) would be the kernel, and 0x1 (system) would be the I/O Subsystem.

ErrorControl: (0x00000001): This implies if the driver fails to load or initialize, proceed regardless with startup, however display a warning.

We note that its ImagePath is:


If you show hidden files & folder and navigate here, you have two folders - "SW Update Service", and "SWUpdate". If you actually have a Samsung machine, you instead have two "SWUpdate" folders, and they both contain XML files. If we take a look at one (BASW-A0394A05_1B33BCEB.xml):

 <?xml version="1.0" encoding="UTF-8"?>  
 <InstallPara1>/pbr /na</InstallPara1>  
 <Str>Windows Configuration</Str>  
 <Str>Windows Configuration</Str>  
 <Str>This program helps your windows configuration settings.</Str>  
 <Str>이 프로그램은 Windows configuration 프로그램입니다.</Str>  
 <TargetCISCode> </TargetCISCode>  
 <BulletineDate>2015-05-12 17:12:43</BulletineDate>  
 <Value>HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update</Value>  
 <Value>HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update</Value>  
 <FromDate>1900-01-01 오전 12:00:00</FromDate>  
 <ToDate>1900-01-01 오전 12:00:00</ToDate>  

Note its installer file.

We can see now how Disable_Windowsupdate.exe begins the process to its "drop", which is downloading the zip its contained in from:


I find this string excerpt particularly funny:

 <Str>This program helps your windows configuration settings.</Str>  

Once the zip is dropped, we can inspect its contents as well:

If we check the config file for the installer file:

 ;HowTo : The registry location of the installed language....  
 ;%CD%\ = Current Folder Location Variable  
 ;%WinDir% = Windows Folder               ex) C:\Windows C:\Winnt  
 ;%ProgramFiles% = Program Files Folder     ex) C:\Program Files, C:\Archivo de program, C:\Programme  
 ;HowTo : The registry location of the installed language....  
 ;LangID     Lang / Export to  
 ;0412     KOR / KOR  
 ;0409     ENG / UK, HKG  
 ;040C     FRN / FRN  
 ;0407     GER / GER  
 ;0411     JPN / JPN  
 ;0404     CHT / CHT  
 ;0804     CHS / CHS  
 ;0C0A     SPA / SPA  
 ;0816     POR / POR  
 ;0419     RUS / RUS  
 OSConditional= TRUE  
 ShowWin = FALSE  
 RunInAuditMode     = TRUE  
 Setup1=xcopy 32\Disable_Windowsupdate.exe "%ALLUSERSPROFILE%\Samsung\" /y  
 Setup2=schtasks /create /XML "%CD%\Dis_AU.xml" /tn "Dis_AU"  
 Setup1=xcopy 64\Disable_Windowsupdate.exe "%ALLUSERSPROFILE%\Samsung\" /y  
 Setup2=schtasks /create /XML "%CD%\Dis_AU.xml" /tn "Dis_AU"  

We can see its using the xcopy command to inevitably "drop" Disable_Windowsupdate.exe in \ProgramData\Samsung. %ALLUSERPROFILE% is an environment variable for \ProgramData on >Vista, and \Documents and Settings\All Users on XP.

We can confirm this by checking ourselves:

Note that the exe is actually signed by Samsung themselves:

So a big thing is the question as to how this persistently resets Windows Update from working after you change it and reboot, and it's actually not SW Update. SW Update is basically just there to genuinely do its job, which is to update Samsung's drivers, software, etc.

What's actually causing Windows Update to persistently become reset and not allow the user to set it the way they want it to, is the fact that Disable_Windowsupdate.exe creates a scheduled task that runs at every logon to ensure that Windows Update is indeed consistently reset to "Check for updates but let me choose whether to download or install them".

We can see the task's contents below:

 <?xml version="1.0" encoding="UTF-16"?>  
 <Task version="1.2" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">  
   <LogonTrigger id="145a3a6c-a630-4ec0-985d-1280512f0ba8">  
   <Principal id="Author">  
  <Actions Context="Author">  

Let's see it in action

So first off, as I noted earlier in the post, if you're trying to run the Samsung update software + disabler, etc, on a non-Samsung environment, it's really buggy. My VM was going through convulsions trying to just take screenshot examples after frequent restarts, etc, so there's a few minutes in between each screenshot.

Here's what WU looks like directly after installing SW Update:

Note that it's set to 'Check for updates but let me choose whether to download and install them'.

Let's change it to 'Install updates automatically (recommended)':

Cool, let's restart and check again.

Oh, this doesn't look right. Let's check the settings:


There's a bit more to it that I'd like to get to eventually, but I suppose this is enough to get the point across. Anyway, with this known, I decided to try Samsung's chat to see if they knew of it:

You are now chatting with 'Rep'. There will be a brief survey at the end of our chat to share feedback on my performance today.
Your Issue ID for this chat is *purged*.
Rep: Hi, thank you for reaching out to Samsung technical support. How may I assist you?
ringzero: Hi Rep, I have a question regarding your SW Update software.
Rep: Hi Ringzero, please go ahead with your question.
Rep: I'll be glad to assist you.
ringzero: Thanks Rep! My question is, why does this software actively monitor the registry and deliberately cripple Windows Update by forcefully disabling it?
Rep: SW Update tool helps in automatically detecting the hardware on the laptop and installs the supporting drivers for them. I am afraid; this tool has directly no effect on the registry of your laptop or Windows Updates.
ringzero: Rep, I am afraid that you're incorrect. SW Update drops an exe named "Disable_Windowsupdate.exe"
ringzero: When SW Update is installed, Windows Update is always disabled. If it's enabled, or set to a setting of your liking, it'll be re-disabled on reboot.
ringzero: If SW Update is uninstalled, Windows Update stays enabled persistently throughout reboots.
Rep: Thank you for waiting. I'll be with you in just a moment.
ringzero: Sure.
Rep: When you enable Windows updates, it will install the Default Drivers for all the hardware no laptop which may or may not work. For example if there is USB 3.0 on laptop, the ports may not work with the installation of updates. So to prevent this, SW Update tool will prevent the Windows updates.

So thanks to Rep over at Samsung, we now know Samsung's motive to disabling WU.

OEMs, come on... has Superfish taught us nothing?

Upload/report this as malware to Microsoft/MSRC, etc, because that's exactly what it is. Why would you ever tamper with WU in such a fashion (or in general), in a way a generic user cannot control, leaving them vulnerable?

x86 MD5


x64 MD5


x86 SHA-256


x64 SHA-256


Small edit: I edited out the Samsung rep's real name to just 'Rep'. It was clearly a tier 1/2 support just doing their job, and I of course don't want them getting in any trouble since this appears to be blowing up. After all, as I said, this isn't their fault at all.


According to a few news articles, here's Samsung's latest statement:
"It is not true that we are blocking a Windows 8.1 operating system update on our computers. As part of our commitment to consumer satisfaction, we are providing our users with the option to choose if and when they want to update the Windows software on their products," said Samsung.
"We take product security very seriously and we encourage any Samsung customer with product questions or concerns to contact us directly at 1-800-SAMSUNG."
I don't understand what this statement is implying, and it may have been a loss in translation between whichever article reporter/editor got the statement from Samsung, because I never implied it specifically blocked a "Windows 8.1 OS system update", just that their SW Update software is preventing Windows Update from automatically installing updates, and forcing the user to have it set to "let me choose whether to download and install". If you attempt to change it, it'll switch right back on a reboot. Microsoft has openly stated that they do not like the fact that it's persistently changing, or even existing in the first place without the user's consent. It's disabling Windows Update from working as the user intends it to.

However you look at this, Samsung's solution to what we can guess is a device driver workaround was not done in the best way, or a safe way. I mean, come on, the exe is named Disable_Windowsupdate.exe. In any case, if it appears I am acting as an enemy to Samsung, I'm not. I'm just a 22 year old cashier with a love for Windows internals that found a security risk for Windows' Samsung users with a few others. That's it.

Update #2

According to a few news articles, here's Samsung's latest statement:
“Samsung has a commitment to security and we continue to value our partnership with Microsoft. We will be issuing a patch through the Samsung Software Update notification process to revert back to the recommended automatic Windows Update settings within a few days."
I'm very glad Samsung is committed to implementing a resolution to this issue so soon. Ultimately, in a perfect world, I hope OEMs will learn from Superfish/SW Update, as it would be disheartening to see a similar issue occur in the future. I feel OEMs need to disclose whatever they intend with their users with their software, and if possible, giving them a choice.

If this is done, it's not "under the table" anymore so to speak. If Samsung's users were notified in the first place that their Windows Update settings were being actively modified, then even though it still potentially may have been a question of poor implementation/methods, it probably wouldn't have been seen as malicious or questionable behavior in the first place as it would have at least been known.

Wednesday, May 20, 2015

FwpsStreamInjectAsync0 bug/leak - Bitdefender (0x4A)

Today I'll be investigating an issue involving Bitdefender, which is turned out to be a Windows bug/issue more than Bitdefender, although there are developmental changes that could be made aside from a hotfix to stop this issue. Bitdefender's 0x4A bug check issue has been prevalent for quite awhile now, but there's little to no documentation on solving it or what's causing it, just a few things to try like updating Bitdefender, uninstalling it, etc
First off, taking a look at a non-verifier enabled kernel dump, here's our bug check as discussed:

 10: kd> .bugcheck  
 Bugcheck code 0000004A  
 Arguments 00000000`77a1dc2a 00000000`00000002 00000000`00000000 fffff880`13695b60  

0x4A bug check, essentially implying that the thread which was previously involved in a system call  attempted to return to user mode at an IRQL higher than PASSIVE_LEVEL (zero [0] on x86 and x64).

 10: kd> !irql  
 Debugger saved IRQL for processor 0xa -- 2 (DISPATCH_LEVEL)  

In this case, at the time of the crash, the IRQL was DISPATCH_LEVEL (Two [2] on x86 and x64).


The process involved in the IRQL raise was vsserv.exe, Bitdefender's main active protection process.

Let's also go further and dump the address of the system function involved:

 10: kd> !address 0000000077a1dc2a  
 Usage:         VAD  
 Base Address:      00000000`779d0000  
 End Address:      00000000`77b79000  
 Region Size:      00000000`001a9000  
 VA Type:        UserRange  
 VAD Address:      0xfffffa8020d1f830  
 Commit Charge:     0xd  
 Protection:       0x7 [ReadWriteCopyExecute]  
 Memory Usage:      Section [\Windows\System32\ntdll.dll]  
 No Change:       no  
 More info:       !vad 0x779d0000  

We can see its VA type is UserRange, its protection is 0x7 which implies it's R/W/X (or E).

If we run !vad on the VAD Address field, we can see frequent mention of Bitdefender:

 10: kd> !vad 0xfffffa8020d1f830  
 fffffa8018c95a70 ( 2)  7fee7980 7fee79c8     6 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\mimepack.dll  
 fffffa802278cb70 ( 3)  7fee79d0 7fee7aa2    69 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asregex.dll  
 fffffa8018c944e0 ( 0)  7fee7ab0 7fee7bb9     9 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asmcocr.dll  
 fffffa804d4baa50 ( 2)  7fee7bc0 7fee7dea    259 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asunicode.dll  
 fffffa8022f25450 ( 3)  7fee7df0 7fee810b    89 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asemlthin.mdl  
 fffffa8050690e60 ( 1)  7fee8110 7fee8333    86 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asemlrtr.mdl  
 fffffa8018c0b1e0 ( 2)  7fee8340 7fee8442    81 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\ascore.dll  
 fffffa804f4f7970 ( 3)  7fee8550 7fee8622    69 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\asregex.dll  
 fffffa804e14ff80 (-1)  7fee8630 7fee885a    259 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\asunicode.dll  
 fffffa804f509640 ( 2)  7fee8860 7fee89ef    82 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttprbl.mdl  
 fffffa804f4bfc80 ( 3)  7fee89f0 7fee8cc9    88 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttpph.mdl  
 fffffa80232c0460 ( 1)  7fee8cd0 7fee8dca    82 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttpdsp.mdl  
 fffffa804ce69120 ( 3)  7fee8dd0 7fee8edb    81 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttpbr.mdl  
 fffffa804ce52bb0 ( 2)  7fee8ee0 7fee8fe2    81 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\otcore.dll  

Let's use !address and -v together to get nice verbose PTE/PFN/VAD information:

 10: kd> !address -v -map 0x779d0000  
 PXE:  fffff6fb7dbed000 [contains 02e0000763d62867]  
     Page Frame Number: 763d62, at address: fffffa80162b8260  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     0000000000763e3c  
     Attributes:     M:Modified,Cached  
     Usage:       PPEs Process fffffa8020d96b10 [vsserv.exe], Entries:5  
 PPE:  fffff6fb7da00008 [contains 18700007641ad867]  
     Page Frame Number: 7641ad, at address: fffffa80162c5070  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     0000000000763d62  
     Attributes:     M:Modified,Cached  
     Usage:       PDEs Process fffffa8020d96b10 [vsserv.exe], Entries:31  
 PDE:  fffff6fb40001de0 [contains 0370000764ab6867]  
     Page Frame Number: 764ab6, at address: fffffa80162e0220  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     00000000007641ad  
     Attributes:     M:Modified,Cached  
     Usage:       PTEs Process fffffa8020d96b10 [vsserv.exe], Entries:159  
 PTE:  fffff680003bce80 [contains 82a000079e2d4025]  
     Page Frame Number: 79e2d4, at address: fffffa8016da87c0  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     000000000079ec0c  
     Attributes:     P:Prototype,Cached  
     Usage:       MappedFile CA:fffffa801f3a3010 [\Windows\System32\ntdll.dll]  
 Type:  Valid  
 Attrs: Private,NormalPage,NotDirty,NotDirty1,Accessed,User,NotWritable,NotWriteThrough  
 PFN:  79e2d4  

Overall, we can see vsserv.exe is listed as active and valid within the page regarding its location, as well as ntdll being involved with memory usage:

 10: kd> !vad 0x779d0000  
 VAD       level   start   end  commit  
 fffffa8020d1f830 (-1)    779d0   77b78    13 Mapped Exe EXECUTE_WRITECOPY \Windows\System32\ntdll.dll  

Throughout all of the 0x4A Bitdefender related crashes, the NT kernel was labeled as the fault:

 Probably caused by : ntkrnlmp.exe  

Given we're seeing ntdll, we can likely imagine the reason for the NT kernel being blamed as being the fault of the crash is because most of the API from ntdll is implemented in the NT kernel variants, with this being ntkrnlmp.exe because this system has a multi-processor without physical address extension configuration.

 10: kd> vertarget  
 Windows 7 Kernel Version 7601 (Service Pack 1) MP (12 procs) Free x64  
 Product: WinNt, suite: TerminalServer SingleUserTS  
 Built by: 7601.18839.amd64fre.win7sp1_gdr.150427-0707  
 Machine Name:  
 Kernel base = 0xfffff800`03064000 PsLoadedModuleList = 0xfffff800`032ab730  
 Debug session time: Fri May 15 09:00:27.644 2015 (UTC - 4:00)  
 System Uptime: 0 days 16:20:00.892  

So regarding processor #10, that's probably as far as we're going to go considering it's the bug check thread and there's no information really whatsoever.

 10: kd> k  
 Child-SP     RetAddr      Call Site  
 fffff880`13695928 fffff800`030d7e69 nt!KeBugCheckEx  
 fffff880`13695930 fffff800`030d7da0 nt!KiBugCheckDispatch+0x69  
 fffff880`13695a70 00000000`77a1dc2a nt!KiSystemServiceExit+0x245  
 00000000`2798f908 00000000`00000000 0x77a1dc2a  

All we can see if we're exiting user-mode code using the KiSystemServiceExit function, and we go off the rails right there - KiSystemServiceExit+0x245. This function is in charge of handling the various call-styles used to enter kernel-mode, and then returning to user-mode.

With that said, let's switch to the other processor within the system that was involved and see what's going on at the time of the crash. To find out the active processors on the specific system, we'll use !running:

 10: kd> !running  
 System Processors: (0000000000000fff)  
  Idle Processors: (00000000000003ff) (0000000000000000) (0000000000000000) (0000000000000000)  
     Prcbs       Current     (pri) Next      (pri) Idle  
  10  fffff880038c9180 fffffa8021c85b50 ( 9)            fffff880038d41c0 ................  
  11  fffff8800393b180 fffffa8021c91060 ( 9)            fffff880039461c0 ................  

We can see our processors are #10 and #11. We've explored #10, so let's check #11. The reason 0-9 aren't listed is because they're idle.

 11: kd> knL  
  # Child-SP     RetAddr      Call Site  
 00 fffff880`03969e20 fffff880`0584f75e e1c62x64+0x558e  
 01 fffff880`03969e50 fffff880`0584ff1f e1c62x64+0x2075e  
 02 fffff880`03969ec0 fffff880`0584fb43 e1c62x64+0x20f1f  
 03 fffff880`03969f70 fffff880`0584fa49 e1c62x64+0x20b43  
 04 fffff880`03969fa0 fffff800`0301f62f e1c62x64+0x20a49  
 05 fffff880`03969fe0 fffff880`01a0c600 hal!HalBuildScatterGatherList+0x203  
 06 fffff880`0396a050 fffff880`0584ffb2 ndis!NdisMAllocateNetBufferSGList+0x110  
 07 fffff880`0396a0f0 fffff880`05850649 e1c62x64+0x20fb2  
 08 fffff880`0396a150 fffff880`0585028e e1c62x64+0x21649  
 09 fffff880`0396a1b0 fffff880`01ac84f1 e1c62x64+0x2128e  
 0a fffff880`0396a1f0 fffff880`01a0c4d4 ndis!ndisMSendNBLToMiniport+0xb1  
 0b fffff880`0396a250 fffff880`05c6d6b8 ndis!NdisFSendNetBufferLists+0x64  
 0c fffff880`0396a290 fffff880`05c6d92c bdfndisf6+0x16b8  
 0d fffff880`0396a2f0 fffff880`05c6df4b bdfndisf6+0x192c  
 0e fffff880`0396a380 fffff880`01a0c4d4 bdfndisf6+0x1f4b  
 0f fffff880`0396a480 fffff880`00c16199 ndis!NdisFSendNetBufferLists+0x64  
 10 fffff880`0396a4c0 fffff880`01a0c419 pacer!PcFilterSendNetBufferLists+0x29  
 11 fffff880`0396a5c0 fffff880`01ac85d5 ndis!ndisSendNBLToFilter+0x69  
 12 fffff880`0396a620 fffff880`01c60eb6 ndis!NdisSendNetBufferLists+0x85  
 13 fffff880`0396a680 fffff880`01c67895 tcpip!IpNlpFastSendDatagram+0x496  
 14 fffff880`0396aa30 fffff880`01c68450 tcpip!TcpTcbSend+0x495  
 15 fffff880`0396acb0 fffff880`01c671a8 tcpip!TcpEnqueueTcbSendOlmNotifySendComplete+0xa0  
 16 fffff880`0396ace0 fffff880`01b30267 tcpip!TcpEnqueueTcbSend+0x258  
 17 fffff880`0396ad90 fffff880`01b35f5d NETIO!StreamInjectRequestsToStack+0x287  
 18 fffff880`0396ae60 fffff880`01b376b4 NETIO!StreamPermitDataHelper+0x5d  
 19 fffff880`0396ae90 fffff800`030e41dc NETIO!StreamPermitRemoveDataDpc+0x84  
 1a fffff880`0396af00 fffff800`030db335 nt!KiRetireDpcList+0x1bc  
 1b fffff880`0396afb0 fffff800`030db14c nt!KyRetireDpcList+0x5  
 1c fffff880`13abf190 fffff800`0312371c nt!KiDispatchInterruptContinue  
 1d fffff880`13abf1c0 fffff800`030c2aec nt!KiDpcInterrupt+0xcc  
 1e fffff880`13abf350 fffff880`01b383aa nt!KeInsertQueueDpc+0x1dc  
 1f fffff880`13abf3e0 fffff880`01b3b468 NETIO!StreamPermitData+0x13a  
 20 fffff880`13abf450 fffff880`01b3b99a NETIO!StreamInternalClassify+0x1a8  
 21 fffff880`13abf520 fffff880`01b3bd8e NETIO!StreamInject+0x1ca  
 22 fffff880`13abf5f0 fffff880`01b91df3 NETIO!FwppStreamInject+0x12e  
 23 fffff880`13abf680 fffff880`05c9aaf1 fwpkclnt!FwpsStreamInjectAsync0+0xcf  
 24 fffff880`13abf6e0 fffff880`05c9bce3 bdfwfpf+0x2af1  
 25 fffff880`13abf780 fffff880`05ca469c bdfwfpf+0x3ce3  
 26 fffff880`13abf7c0 fffff880`05ca4d0a bdfwfpf+0xc69c  
 27 fffff880`13abf840 fffff880`05c9ebb3 bdfwfpf+0xcd0a  
 28 fffff880`13abf8a0 fffff800`033f3e47 bdfwfpf+0x6bb3  
 29 fffff880`13abf8d0 fffff800`033f46a6 nt!IopXxxControlFile+0x607  
 2a fffff880`13abfa00 fffff800`030d7b53 nt!NtDeviceIoControlFile+0x56  
 2b fffff880`13abfa70 00000000`77a1dc2a nt!KiSystemServiceCopyEnd+0x13  
 2c 00000000`27a9f928 00000000`00000000 0x77a1dc2a  

I used knL as opposed to the other stack dump commands as I wanted to get the frame # feature for reference reasons.

Starting at frame # 2a, we can see the NtDeviceIoControlFile function calls IopXxxControlFile. The latter function appears to be undocumented, so I'm unsure as to what it does. What I do know is, the NtDeviceIoControlFile function is ultimately used to build descriptors for a driver. I imagine it's using the IopXxxControlFile function to aid in passing such to the driver.

Also, for what it's worth, although NtDeviceIoControlFile has since been superseded by DeviceIoControl, the former native function provides more information that may be beneficial to the caller (especially for debugging purposes). This is likely why Bitdefender chose to use the former function instead.

 11: kd> ln nt!IopXxxControlFile  
 (fffff800`033f3840)  nt!IopXxxControlFile   
 (fffff800`033f4650)  nt!NtDeviceIoControlFile  
 Exact matches:  
   nt!IopXxxControlFile (<no parameter info>)  

If we disassemble this function, we can wade through some of the stuff and find some of the interesting tidbits:

 11: kd> u fffff800`033f3840 fffff800`033f4650  
 fffff800`033f3956 e845c5ffff   call  nt!ProbeForWrite (fffff800`033efea0)  
 fffff800`033f39b7 e81498fdff   call  nt!ObReferenceObjectByHandleWithTag (fffff800`033cd1d0)  
 fffff800`033f3b05 e84688cfff   call  nt!IoGetRelatedDeviceObject (fffff800`030ec350)  
 fffff800`033f402a e8d130cdff   call  nt!IoGetAttachedDevice (fffff800`030c7100)  
 fffff800`033f3c01 e88a7bcfff   call  nt!IoAllocateIrp (fffff800`030eb790)  
 fffff800`033f40d1 e82af9cfff   call  nt!IoAllocateMdl (fffff800`030f3a00)  

So after neatly putting together this disassembly of sorts, we can see that this is indeed how the NtDeviceIoControlFile function is passing on the buffer and such to the driver.

The IoAllocateMdl function in this specific case is used to ultimately associate the MDL with an IRP, which is why we call into the IoAllocateIrp function, to of course assign the IRP. IoGetAttachedDevice is called likely to return a pointer to the devobj, with help from the IoGetRelatedDeviceObject function to probably obtain the devobj from the file system driver stack.

ObReferenceObjectByHandleWithTag is called to increment the reference count of the object, and to write a four-byte value known as a "tag" so it can support object reference tracing for debugging purposes. Finally, the ProbeForWrite function is called to ensure that a user-mode buffer meets the following:

  • Resides in the user-mode portion of the address space.
  • Is writeable.
  • Is correctly aligned.

As all appears to have went well, we can see the driver we were ultimately building and passing descriptors to/for was bdfwfpf.sys, which is Bitdefender's firewall filter driver. As it's a driver in charge of a firewall, it of course uses the WFP API (Windows Filtering Platform) to achieve its goals (not just filtering and monitoring).

We can confirm this easily by looking at the very first driver/function call after Bitdefender's firewall, which is fwpkclnt.sys. Specifically, Bitdefender's firewall driver called it to inject new/cloned data to the data stream. Directly afterwords we have calls from the Network I/O Subsystem to continue the injecting, which is because fwpkclnt.sys exports kernel-mode functions, as opposed to fwpuclnt.dll which exports and handles the user-mode side.

To handle and/or continue the injection into the data stream, it looks like DPC(s) are used to handle it by calling KeInsertQueueDpc to create a queued DPC for execution.

 11: kd> !dpcs  
 CPU Type   KDPC    Function  
 10: Normal : 0xfffffa806a7b7cb0 0xfffff88001b37630 NETIO!StreamPermitRemoveDataDpc  

-- After discussion with a fellow kernel-debugger friend of mine, we also thought that the IRQL was possibly DISPATCH_LEVEL due to the multiple injections, etc, therefore Windows deferred it to a DPC. Given this possibly being the case, when the DPC was to be worked on, the system service finished but the IRQL is still DISPATCH_LEVEL. Since that was the case, we get a bug check.

We continue through netio.sys' functions regarding the data stream injection, ultimately injecting the request to the stack and going through a few tcpip.sys functions.

To continue sending the data along, NDIS' NdisSendNetBufferLists function is called, and NDIS' filter driver (which I believe is pacer.sys), called NdisFSendNetBufferLists to send the list of network data buffers back to Bitdefender's firewall driver.

Bitdefender's firewall driver then calls into NDIS' network data buffer sending functions to send the list to the user's network miniport driver, e1c62x64.sys (Intel(R) 82579V Gigabit Network Connection). The network miniport driver then calls NDIS' NdisMAllocateNetBufferSGList function to obtain a scatter/gather list for the network data for the associated NET_BUFFER structure.

In order to do so, NDIS needs to call the HAL, which we can see through the function HalBuildScatterGatherList. What is supposed to happen next is, the HAL builds the scatter/gather list, and we go on through various registered miniport functions. However, this did not happen, and we go off the rails on frame #00 with a call to the miniport driver.

So, where's our problem? Frame #23:

 23 fffff880`13abf680 fffff880`05c9aaf1 fwpkclnt!FwpsStreamInjectAsync0+0xcf  

FwpsStreamInjectAsync0, the function in charge of injecting TCP data segments into a TCP data stream, is the issue. How so? Well, let's get dirty once again.

Using the NDIS debugging extension (!ndiskd), we can get a lot of information to help us here. On its lonesome, !ndiskd isn't too special. However, when we use !ndiskd.miniport, it gets fun.

 11: kd> !ndiskd.miniport  
   MiniDriver     Miniport      Name                   
   fffffa8020c71cd0  fffffa8018c281a0  RAS Async Adapter  
   fffffa801f844cd0  fffffa801f8771a0  SonicWALL NetExtender Adapter  
   fffffa801f862840  fffffa801f86b1a0  WAN Miniport (SSTP)  
   fffffa801f84bb70  fffffa801f8671a0  WAN Miniport (PPTP)  
   fffffa801f837c30  fffffa801f8631a0  WAN Miniport (PPPOE)  
   fffffa801f8409b0  fffffa801f85e1a0  WAN Miniport (IPv6)  
   fffffa801f8409b0  fffffa801f85a1a0  WAN Miniport (IP)  
   fffffa801f8409b0  fffffa801f8561a0  WAN Miniport (Network Monitor)  
   fffffa801f835cd0  fffffa801f8411a0  WAN Miniport (L2TP)  
   fffffa801f82f820  fffffa801f83d1a0  WAN Miniport (IKEv2)  
   fffffa801f664020  fffffa801f7b81a0  Intel(R) 82579V Gigabit Network Connection  
   fffffa801f5cb9e0  fffffa801f5e61a0  Teredo Tunneling Pseudo-Interface  
   fffffa801f5cb9e0  fffffa801f5e21a0  Microsoft ISATAP Adapter #2  
   fffffa801f5cb9e0  fffffa801f5de1a0  Microsoft ISATAP Adapter  
   fffffa801f5cb9e0  fffffa801f5d61a0  Microsoft 6to4 Adapter  

So we know that our miniport involved in all of this was the Intel Gigabit, so let's look at that one:

 11: kd> !ndiskd.minidriver fffffa801f664020  
   Ndis handle    fffffa801f664020  
   Driver Context   NULL  
   DRIVER_OBJECT   fffffa801f7b6e70  
   Driver image    e1c62x64.sys  
   Registry path   \REGISTRY\MACHINE\SYSTEM\ControlSet001\services\e1cexpress  
   Reference Count  2  
   Flags       [No flags set]  
   fffffa801f7b81a0 - Intel(R) 82579V Gigabit Network Connection  

If we take a look at the miniport address:

 11: kd> !ndiskd.miniport fffffa801f7b81a0  
   Intel(R) 82579V Gigabit Network Connection  
   Ndis handle    fffffa801f7b81a0  
   Ndis API version  v6.20  
   Adapter context  fffffa801f990000  
   Miniport driver  fffffa801f664020 - e1cexpress v12.6  
   Network interface fffffa8019c8c870  
   Media type     802.3  
   Device instance  PCI\VEN_8086&DEV_1503&SUBSYS_849C1043&REV_06\3&11583659&0&C8  
   Device object   fffffa801f7b8050  More information  
   MAC address    e0-3f-49-78-a1-dd  

   Miniport      Running  
   Device PnP     Started  
   Datapath      Normal  
   Interface     Up  
   Media       Connected  
   Power       D0  
   References     0n10  
   Total resets    0  
   Pending OID    None  

   Protocol list   Driver       Open        Context        
   RSPNDR       fffffa8021b39cf0  fffffa8021b608d0  fffffa8021b62010  
   LLTDIO       fffffa8021b1a8f0  fffffa8021b528d0  fffffa8021b361b0  
   TCPIP6       fffffa801d05c2c0  fffffa801fb13010  fffffa801fb0b010  
   TCPIP       fffffa8019c7b890  fffffa801fb08580  fffffa801fb03ba0  
   Filter list    Driver       Module       Context        
   WFP LightWeight Filter-0000  
             fffffa801f59f010  fffffa801faff660  fffffa801faff400  
   QoS Packet Scheduler-0000  
             fffffa801f5ab930  fffffa801fb00780  fffffa801f9d3010  
   BitDefender Firewall NDIS6 Filter Driver-0000  
             fffffa801f574d40  fffffa801fb04c80  fffffa801fb04850  

We get a lot of good information, and can see that Bitdefender's firewall filter driver is/was involved with this miniport. We know this, because we saw it all happening in the stack, but this just confirms it.

Anyway, what's next? Well, let's check for any pending NBLs (NET_BUFFER_LISTS):

 11: kd> !ndiskd.pendingnbls fffffa801f7b81a0  
 PHASE 1/3: Found 23 NBL pool(s).           
 PHASE 2/3: Found 512 freed NBL(s).                    
   Pending Nbl    Currently held by                      
   fffffa80593c82c0  fffffa801f7b81a0 - Intel(R) 82579V Gigabit Network Connection [Miniport]            
 PHASE 3/3: Found 1 pending NBL(s) of 789 total NBL(s).             
 Search complete.  

Ah ha, we have one held by the miniport driver that was involved in passing data to Bitdefender's firewall filter driver. Let's look at the pending NBL:

 11: kd> !ndiskd.nbl fffffa80593c82c0  
   NBL        fffffa80593c82c0  Next NBL      NULL  
   First NB      fffffa80593c83f0  Source       fffffa801fb08580 - TCPIP   

From here we can take a direct look at the NBL:

 11: kd> dt _NET_BUFFER_LIST fffffa80593c82c0  
   +0x000 Next       : (null)   
   +0x008 FirstNetBuffer  : 0xfffffa80`593c83f0 _NET_BUFFER  
   +0x000 Link       : _SLIST_HEADER  
   +0x010 Context     : 0xfffffa80`593c84a0 _NET_BUFFER_LIST_CONTEXT  
   +0x018 ParentNetBufferList : (null)   
   +0x020 NdisPoolHandle  : 0xfffffa80`1cfe6080 Void  
   +0x030 NdisReserved   : [2] (null)   
   +0x040 ProtocolReserved : [4] 0x746c6100`00000001 Void  
   +0x060 MiniportReserved : [2] 0xfffffa80`1f990000 Void  
   +0x070 Scratch     : (null)   
   +0x078 SourceHandle   : 0xfffffa80`1fb08580 Void  
   +0x080 NblFlags     : 0  
   +0x084 ChildRefCount  : 0n0  
   +0x088 Flags      : 0x100  
   +0x08c Status      : 0n0  
   +0x090 NetBufferListInfo : [19] 0x00000000`00220015 Void  

What appears to be happening here is multiple NBLs in a chain are being passed, the FwpsStreamInjectAsync0 function is called to pass Bitdefender's data, and then the chain is broken as the call goes on (see the NBL next member is zeroed out/null).

Possibly a fix (in Bitdefender's case) is to avoid multiple injections inside the stream callout routine, possibly taking NBLs in a chain and calling the FwpsStreamInjectAsync0 function just ONCE for each callout routine execution. Unsure, kernel development isn't my strong point : )

A fix for user's is to install this hotfix and hope it works, as it should. Overall, maybe Bitdefender instead of making any developmental changes could just raise awareness for this issue, like creating a well explained documentation page with a link to the hotfix. I think developmental changes would be a better workaround.