Delivery Optimization Internals – Researching the Logs

Delivery Optimization (DO) is what Microsoft defines as a self-organizing distributed cache, available in Windows 10 and later, or as we like to call it, a P2P solution. DO is enabled by default in Windows 10 and later, Windows Server 2019 and later, and can use multiple sources like Microsoft CDN servers, local cache servers, or peers for content. All at the same time too. Before diving into the main topic for this post, meaning researching the logs, let me give you a crash course in DO content support and a short rant on network impact.

Note #1: With DO, servers only peer to servers, and clients only peer to clients. DO doesn't do peering between server and clients (though they can all use a connected cache server). I should probably mention that while peering is enabled by default in Windows 10 and Windows 11, peering is disabled by default on servers (and on IOT and HoloLens). Thanks, Phil Wilcock, for pointing that out.

Note #2: Please don't try to be clever and completely disable the Delivery Optimization service by unsupported registry hacks, it will only force Windows to use BITS to download the content instead. If you want to configure Delivery Optimization, please use the supported group policy methods or the equivalent Intune settings.

Delivery Optimization Content Support

Delivery Optimization only works with certain types of content. For example:

• Windows Updates, including Drivers
• Native Application Updates in Windows 10 and Windows 11
• Win32 Apps from Microsoft Intune
• Microsoft 365 Apps, Installation, and Updates
• Microsoft Store applications

Note: Delivery Optimization does not support peering of ConfigMgr content. For that, you have to use BranchCache (recommended) or Peer Cache.

Delivery Optimization Network Impact

Organizations have been surprised, to say the least, by the sheer number of connections and amount of data that Windows 10, and Windows 11 clients generate. We even had customers having to replace their Internet firewalls because they could not handle the load. These days some organizations we work with have DO use more than 50 percent of their network transfers in terms of connections and data, which is a lot. The good news is that with a good DO configuration, and effective visualization/control of DO (cough, StifleR), you can quickly reduce the network impact DO will have on your environment.

Understanding the Delivery Optimization cmdlets

Depending on your Windows version you have access to different sets of PowerShell cmdlets for Delivery Optimization. Some of these cmdlets will require you to add a -verbose parameter to see all info. A good example would be Get-DOConfig to won't show you memory or VPN settings unless you add the -verbose switch.

Understanding the Delivery Optimization Logs

Deliver Optimizati0n logs pretty much everything it does in the event log, and there are two main types of log files:

• domgmt.*.etl files
• dosvc.*.etl

The domgmt.*.etl files are pretty much useless since they just have generic info about internal DO events, but the dosvc.*.etl files contain all the good download and peering info. This info can be used to determine DO efficiency and can help you troubleshoot DO configurations. The dosvc.*.etl files have also been particular of interest for forensic analysts, and other, eh, creative individuals, since they contain info about what the machines connect to, both externally, and internally.

The DO log files are stored in the C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Microsoft\Windows\DeliveryOptimization\Logs folder

Increasing details in the Delivery Optimization Logs

Starting with Windows 10 2004, you can increase details in the DO logs by running Enable-DeliveryOptimizationVerboseLogs. With verbose logging you get more then double the number of functions with data in the logs, and with easier accessibility to individual data. For example, later in this post you learn how to extract the Country Code in the DO log details, from a JSON object embedded in a Message string in the event log. With verbose logging, Country Code is a separate function, making it a bit easier to parse.

For example, compare this:

Message : GEO response: {"ExternalIpAddress":"184.85.245.183","CountryCode":"US","KeyValue_EndpointFullUri":"https
://kv801.prod.do.dsp.mp.microsoft.com/all","Version":"5B36157A03CF0500DA3C2D8238E6005F3469E237734C20468
90202DB6F874840","CacheId":"7","CompactVersion":"10.0.19041.1266","ContentCert":false,"DownloadModeFail
Safe":""}

To this:

Message : Country code retrieval hr = 0, ccode = US

Another function I stumbled across with verbose logging enabled was CDeviceProfile::_IsOnVirtualMachine, which as you probably can guess, tells weather you are using a virtual machine or not. The message string from this one on a VM is: Message : VM detection using WMI, hr = 0, isVM? 1

Working with the Delivery Optimization logs in PowerShell

Retrieving information from the Delivery Optimization logs can be done with the built-in PowerShell cmdlet, Get-DeliveryOptimizationLog, which you can run either online on the system, or in offline mode by specifying the Path parameter. Below you find an example for reading log files that have been saved previously as well as reading from a live system:

# Get data from previously saved log files
$SavedLogsPath = "C:\Demo\DOLogs"
$ETLFiles = Get-ChildItem $SavedLogsPath -File -Filter "dosvc*.etl" | Sort-Object Name
$DOLogsOutPut = Get-DeliveryOptimizationLog -Path $ETLFiles.FullName
 
# Get data from the Get-DeliveryOptimizationLog cmdlet
$DOLogsOutPut = Get-DeliveryOptimizationLog

Once you have the log output, you can now start to poke around in the various Delivery Optimization functions and metrics. Below is how you can list the DO functions in your log data:

# Get all unique functions, and save to a text file
$ListFunctions = $DOLogsOutPut | Select Function -Unique | Sort-Object Function
$ListFunctions | Out-File C:\Demo\DOFunctions.txt

If you just want to dump all of the data in a text file, you can run this little command:

# Save log output to a text file
$DOLogsOutPut | Out-File C:\Demo\DOLogOutPut.txt

Getting details from the Delivery Optimization logs

Now, the 100 or so DO functions (200 with verbose logging) you will find in the logs are not documented, at least not anywhere I have seen, but by looking at the data you can kind of figure out what they are doing. For example, one of the more interesting functions is the CAnnounceSequencer::_InternalAnnounce function, showing network information about the client itself, together with information about the download. For this function the Message data string from the event log is stored in JSON format, which after some cleanup, is making it at least somewhat easy to extract with a bit of PowerShell. Other functions just have the string of data, requiring some parsing to extract it.

Another function of interest is the CServiceConfigProvider::_CallService that shows the external IP address of the client together with what country it's in.

You will also find some interesting info in the CServiceConfigProvider::_CallService function, for example which strings DO is using to figure out if a client is on VPN or not. When writing this blog post those strings were: VPN, Secure, Virtual Private Network, Juniper, and PANGP. Interesting enough, the info from the DO logs shows more VPN strings, then the VpnKeywords data from Get-DOConfig -Verbose displays. Wonder which one that is true :)

And speaking of network connectivity, the CSettingsMonitor::Start function will give you an idea. Here is a sample message from that function: Message : Settings mon init: connected? 1, vpn_connected? 0, cellular? 0, battery? 0, downloadMode: 1, costState: 2

Below you find a few PowerShell samples that exports selected data from some of the functions into CSV files you can easily open in Excel for further analysis:

# Show info about download and local network info
# This assumes you have gathered the logs into the $DOLogsOutPut array per the preceding examples
$InternalAnnounceExportFile = "C:\Demo\InternalAnnounce.csv"
$InternalAnnounce = $DOLogsOutPut | Where-Object {($_.Function -eq "CAnnounceSequencer::_InternalAnnounce")}
$InternalAnnounce = $InternalAnnounce | Select-Object @{N="Message";E={$_.Message -replace "Swarm.*announce request:",""}},TimeCreated,Level,LevelName,Function,ErrorCode 
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name ContentId -Value (($_.Message | ConvertFrom-Json).ContentId) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name AltCatalogId -Value (($_.Message | ConvertFrom-Json).AltCatalogId) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name PeerId -Value (($_.Message | ConvertFrom-Json).PeerId) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name ReportedIp -Value (($_.Message | ConvertFrom-Json).ReportedIp) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name SubnetMask -Value (($_.Message | ConvertFrom-Json).SubnetMask) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Ipv6 -Value (($_.Message | ConvertFrom-Json).Ipv6) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name IsBackground -Value (($_.Message | ConvertFrom-Json).IsBackground) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name ClientCompactVersion -Value (($_.Message | ConvertFrom-Json).ClientCompactVersion) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Uploaded -Value (($_.Message | ConvertFrom-Json).Uploaded) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Downloaded -Value (($_.Message | ConvertFrom-Json).Downloaded) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name DownloadedCdn -Value (($_.Message | ConvertFrom-Json).DownloadedCdn) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name DownloadedDoinc -Value (($_.Message | ConvertFrom-Json).DownloadedDoinc) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Left -Value (($_.Message | ConvertFrom-Json).Left) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name JoinRequestEvent -Value (($_.Message | ConvertFrom-Json).JoinRequestEvent) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name RestrictedUpload -Value (($_.Message | ConvertFrom-Json).RestrictedUpload) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name PeersWanted -Value (($_.Message | ConvertFrom-Json).PeersWanted) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name GroupId -Value (($_.Message | ConvertFrom-Json).GroupId) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Scope -Value (($_.Message | ConvertFrom-Json).Scope) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name UploadedBPS -Value (($_.Message | ConvertFrom-Json).UploadedBPS) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name DownloadedBPS -Value (($_.Message | ConvertFrom-Json).DownloadedBPS) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name HttpDelayLeftSecs -Value (($_.Message | ConvertFrom-Json).HttpDelayLeftSecs) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Profile -Value (($_.Message | ConvertFrom-Json).Profile) -PassThru}
$InternalAnnounce = $InternalAnnounce | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name Seq -Value (($_.Message | ConvertFrom-Json).Seq) -PassThru}
$InternalAnnounce | Export-csv -Path $InternalAnnounceExportFile -NoTypeInformation
 
# Show IP address of DO peers the machine connected to
$ConnectionCompleteExportFile = "C:\Demo\DOPeers.csv"
$ConnectionComplete = $DOLogsOutPut | Where-Object {($_.Function -eq "CConnMan::ConnectionComplete")}
$ConnectionComplete = $ConnectionComplete | Select-Object @{N="PeerIP";E={($_.Message | Select-String -Pattern "\d{1,3}(\.\d{1,3}){3}" -AllMatches).Matches.Value}},TimeCreated,Level,LevelName,Function,ErrorCode
$ConnectionComplete | Export-csv -Path $ConnectionCompleteExportFile -NoTypeInformation
 
# Show External IP address and Country Code
$CallServiceExportFile = "C:\Demo\ExternalIPsAndCountry.csv"
$CallService = $DOLogsOutPut | Where-Object {($_.Function -eq "CServiceConfigProvider::_CallService") -and ($_.Message -match "GEO(:)? response:")}
$CallService = $CallService | Select-Object @{N="Message";E={$_.Message -replace "GEO response: ",""}},TimeCreated,Level,LevelName,Function,ErrorCode 
$CallService = $CallService | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name ExternalIpAddress -Value (($_.Message | ConvertFrom-Json).ExternalIpAddress) -PassThru}
$CallService = $CallService | ForEach-Object {$_ | Add-Member -MemberType NoteProperty -Name CountryCode -Value (($_.Message | ConvertFrom-Json).CountryCode) -PassThru}
$CallService | Export-csv -Path $CallServiceExportFile -NoTypeInformation

Below is a screenshot of data gathered from one of my Windows 10 clients.

Happy researching / Johan