Rethinking Logging for Critical Assets



Going beyond log files, accepting memory as its own format.

image

Logging is a common practice for IT and Security purposes. Mature organizations tend to have extensive and in-depth logging capabilities using either commercial or free solutions.

Although, logging is a powerful way to troubleshoot and investigate events it’s often limited by the initial input format of the logs during the collection process.

As the complexity of attacks increase, it’s almost natural for defensive capabilities to also evolve — particularly in the logging capabilities area. Seasoned attackers focus their operational efforts on advanced infiltration techniques but also on anti-forensics techniques to erase evidences of their visits as we have seen in the some of the files leaked by TheShadowBrokers last year.

The evolution of backdoors, such as in-memory kernel-mode backdoors like DOUBLEPULSAR, makes it even harder for traditional log capture and analysis softwares to understand when an attacker was in your network, and for how long. > “Your perimeter is not the boundary of your network it’s the boundary of your telemetry” — the grugq

Workflow

As you can see above, we designed a four-step compressive process for organizations willing to increase the health-check process of their critical assets by integrating memory logging/forensics to their capabilities and processes.

Identify

Lately, more and more incidents against financial institutions have been publicly reported in the media. As part of defining your perimeter of defense, you’ll identify what “_critical assets_” such as servers you want to increase your visibility on. Those critical assets can be production servers or even honeypots, that you’d tightly control.

Capture

Memory acquisition of a running system is a very simple & straightforward process which can be achieved quickly through our standalone utility, Comae DumpIt, downloadable for free on our platform at https://my.comae.com.

Comae DumpIt generates full memory Microsoft crash dumps, rather that legacy raw memory dumps, on the fly for interoperability and efficiency purposes. Furthermore, Comae DumpIt is standalone allowing the user more flexibility (PsExec, PsRemote etc.) in how to deploy the utility without having to deal with the friction of installing an agent or endpoint.

There are three main scenarios that we identified where memory acquisition is valuable:

  • Manually, after an incident. This scenario is the most common for memory forensics.
  • As part of orchestration. Many of our users leverage memory forensics to have a more comprehensive understanding of the alerts returned by their endpoints. This could be automated through playbooks to automatically perform a memory acquisition after a suspicious activity has been detected by a third-party endpoint.

image

Configuring a scheduled task for DumpIt.

  • Periodically through scheduled jobs. This scenario is the most interesting for increasing logging capabilities, as acquisition is triggered by periodical event (once a month, once a week, etc…) before archiving the memory copies locally (e.g. a file share) as you’d normally do with logs. Configurable directly through PowerShell or Windows’ Task Scheduler, this allows the user to keep periodical copies of their machines states to later enable possibilities such as retro-threat-hunting.$Action = New-ScheduledTaskAction -Execute C:\Comae-Toolkit-3.0.20180207.1\x64\DumpIt.exe -Argument ‘/Q /R’ -WorkingDirectory ‘C:\Logs’ $Trigger = New-ScheduledTaskTrigger -Weekly -At ‘9am’ -DaysOfWeek ‘Monday’ $Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -StartWhenAvailable -Hidden Register-ScheduledTask -Action $Action -Trigger $Trigger -Settings $Settings -TaskName ‘DumpIt’ -User System -RunLevel Highest

Performance and scalability are vital metrics when it comes to memory acquisition of large servers.

image

Acquisition benchmark.

One of the main pain point reported from our users, before they adopting Comae DumpIt, was that several vendors have issues during memory acquisition for servers with more than 64GBs of RAM which prevented them to investigate large servers.

Attached are the results of our acquisition benchmark on traditional desktops and servers with up to half a terabyte of memory. As you can see, the acquisition time is significantly faster (less than 10 minutes) if our built-in compression flag is enabled on such servers.

Archive

As large organizations with several terabytes of memory images would usually opt to store and keep memory images locally in a corporate file share. Those locally stored images can be sent later to our analysis platform Comae Stardust, or be preprocessed to only send the extracted metadata returned by our preprocessing utility Comae Dmp2Json.

During the acquisition, users can also chose to compress the output file on the fly using the /R or /COMPRESS options in order to create a .zdmp (Comae compressed crash dump) file. In addition of file size optimization, this also allows the acquisition to be significantly faster by reducing the number of I/O operations to disk — especially when acquiring the memory from servers with a gargantuan amount of physical memory.

Analyze

image

Machine view from Comae Stardust Platform.

Memory images management can easily be done locally using our PowerShell command line interface available on GitHub. The below command is a sample command that allows the users to pre-process several acquired images from a given folder into their Comae metadata snapshot format before uploading them to our analysis platform.

Get-ChildItem -Path C:\Logs -File | ForEach-Object { Convert-DumpFileToSnapshot -FilePath $_.FullName -Directory "C:\Snapshots" }

Get-ChildItem -Path C:\Snapshots -File | ForEach-Object { Send-ComaeSnapshot -Key "<APIKEY>" -Path $_.FullName -ItemType "File" }

image

Machines View

image

Infected machine

Rethinking Logging

Don’t hesitate to share your comments or feedback on our platform and use cases you encountered where memory forensics was a very valuable asset for your company.

You can already create an account on https://my.comae.com and get your copy of DumpIt.

Please also refer to our user guide to learn more about our platform:
https://www.comae.com/res/en-comae-stardust-user-guide-v1.0.pdf