Beginner Malware Analysis: Setting Up Your Lab

18 min readMay 9, 2023

Welcome everyone to my brand new blog series on Malware Analysis. Let’s make some Malware Devs Cry! In this post, I will walk you through how to setup your first lab with all the tools you need to do Malware Analysis properly. This blog series assumes you have the basic understandings of networking, troubleshooting, command-line, PowerShell, and security concepts. I will go into Computer Science subjects such as: Assembly concepts, some C/C++ Programming concepts, Operating System internals concepts, and memory concepts, but I also highly recommend taking courses on these concepts. I will include links at the end of the article.

What is Malware Analysis?

Malware Analysis is the process of understanding what, where, when, why and how a malware sample works in order to understand and prevent further attacks from threat actors. Cyber Defenders use intelligence from these samples to understand the impact and behaviors malware can have on potential victims as well as to help prevent or mitigate future attacks.

There are two types of malware analysis. Dynamic and Static.

Dynamic Analysis is the process of running the malware and capturing data as it runs. This is typically done in a secured, sand-boxed environment. You can run samples in your own lab, or submit samples to an automated lab such as “Any.Run” or “Cuckoo Sandbox”. Putting a malware sample inside a debugger and running it while stepping through the code is also dynamic analysis.
Static Analysis is the process of learning about the malware sample without running it. This includes disassembly, reverse-engineering, and manual code analysis.
Hybrid Analysis combines both aspects of Dynamic and Static Analysis.

My methodology or steps to Malware Analysis is this:

File Properties Analysis
Dynamic Analysis
Static Analysis
Report and Signature Creation

As a Cyber Security Analyst, I use the intelligence gained from Malware Analysis to better enhance the defenses of the organizations I’m protecting.

Requirements:

A Windows PC with at least 16 GB of RAM and an AMD Processor or Intel Processor that supports virtualization. Most processors come with this feature today. Personally, I recommend you at least have an AMD Ryzen 5, or an Intel i5. If you are a Linux user, it’s about the same. MAC users, from my understanding, they come with intel processors already. Check the amount of RAM you have. If you have any issues during setup, please let me know. Depending on your processor, make sure to enable virtualization in the BIOS.

If you are completely new to virtualization, here’s some resources:

Beginner’s tutorial on VMware Workstation: https://www.youtube.com/watch?v=2KrzgeB6qEI
VirtualBox: https://www.youtube.com/watch?v=VZJ6KZUc25M&list=PLS1QulWo1RIZy4TwTlF3lgxvUs5kDcOgE
Enabling Virtualization in the BIOS: https://support.microsoft.com/en-us/windows/enable-virtualization-on-windows-11-pcs-c5578302-6e43-4b4b-a449-8ced115f58e1

Drive Space with at least 500GB or more. I typically make the VMs at least 80GBs.(Personal preference).

Any of the following virtualization software:

VMware Workstation, VMware Workstation Pro, VMware ESXI, VMware VCenter Server, Oracle VirtualBox, or your preferred VM software.

For this tutorial, I will be using VMware Workstation Pro 16. Let’s get started!

First Steps:

Let’s start by downloading a Windows 10 “.iso” file. I have a way of getting around downloading the iso file from Microsoft’s website. I recommend doing this within Google Chrome.

First, visit Microsoft’s website through this link: https://www.microsoft.com/en-us/software-download/windows10

You should be at this screen in Chrome:

Microsoft’s site for downloading Windows 10.

In Chrome, click on the “three dots” in the right hand corner. Then go to “More tools”, and then, “Developer tools”:

Once clicked, you should be here:

Next, in the “Developer console” there is another three dots in the top right. Click on those dots, and go to “More tools”, and then “Network ”:

You should see a “Network conditions” tab popup at the bottom of the console here:

Now you see where the “User agent” is checked? You’re going to uncheck that and choose a new one. As a note, a “User agent” is an HTTP/HTTPS header that contains a string. This tells other hosts, the type of browser, operating system, version, and vendor that the requesting host is using. If you ever do any network packet analysis(Which we will do in this series), you’ll often see this string identified when packets are being sent over a network. Cyber Security Analysts often use this to help us identify potential malicious traffic, such as traffic coming from a C2. In this case, we are changing the agent to tell the Microsoft website that we are not Chrome, but iOS Safari, Blackberry, Android, etc…

Go ahead and uncheck “Use browser default” and choose a user agent. I’m going with a Blackberry user agent because I’ve had better success with it:

Once chosen, refresh the page and you should get a different page from Microsoft:

Once you get to that page, you’re going to select the iso file that we need:

Then click “Confirm”. Next you will be asked to choose a language. I will be using “English(United States)”:

Now click “Confirm” again, and you will be brought here:

Choose the architecture version of the iso.

For the architecture version, we will go with 64-bit (x64). Once you click on the download, you will now be on your way to downloading the iso. IT may take awhile depending on your internet speed:

Once you have the iso file downloaded, we will need to download a script that will install our malware lab for us. We are going to use Mandiant’s “Flare VM” script from Github: https://github.com/mandiant/flare-vm

It will be the file that says “install.ps1”. This is a Windows Powershell script:

Go ahead and download the Zip file to your computer:

Once we create our VM and install Windows 10, we will copy the Powershell script to it. Yes, for this we will have VMware tools installed. Some malware samples are configured to look for VMware tools, but there are ways we can configure the VM to make us look like a legitimate host machine as well as run some tests. There are also other tools that can help hide us as well. More on this later.

Setting Up Your VM:

Now that we have the files we need, let’s go through VMware Workstation’s VM setup wizard. Click on “Create a New Virtual Machine”:

Once you click that, you will be greeted by the VM setup wizard:

Leave the option as “Typical” for now. Then hit “Next”. Now we are going to choose our iso file that we downloaded. Hit “browse”:

VMware will automatically detect the operating system:

Click “Next”. and then you will be greeted by this screen:

I will enter a product key here(I will not be sharing it). For the version of Windows 10, I typically go with Pro. Then set a full name and password.

Click “Next”, and then enter the name of the VM and where you want to save to. I will be saving mine to a 4TB drive on D:

Click “Next”, and you will now be asked how large you want the VM’s storage space to be. I’m going to set it to 80GBs, and I will split the VM disk(VHD) into multiple files:

Click “Next” and you will be brought here:

Click on “Customize Hardware”. We are going to change the amount of RAM to 8GB:

After that hit “Close”. We need “NAT” to stay enabled for now until we have all our tools installed. Then I will show you how to turn it off. During analysis, our network adapter will always be turned off so samples can’t communicate with your production host and network. Instead, we are going to have a tool that acts as a fake DNS server that responds to DNS requests. As an optional feature, we will also setup a third VM that acts as a DNS server strictly for installing Wireshark so we can capture network traffic. The reason for this is that some samples will detect Wireshark and stop it from running in the lab. So a way around this is to set the fake DNS VM with a static IP, and then set the DNS settings on your lab’s network adapter to that static IP. That way, we can capture traffic and not worry about our sample seeing Wireshark or any other packet capture software that you use. Again this will be optional, but I recommend it. It won’t always be used, we will mostly be using the FakeNET-NG tool in the lab.

Now that our VM is configured, hit “Finish”, and wait for VMware to create the disk.

Disabling and Enabling Settings in Windows

Once the disk is created, the VM should start automatically as long as you left “Power on after disk creation” checked. Windows will install itself.

Once Windows is finished installing, we will need to enable and disable a few things. First, lets disable Windows Defender. Go to the start menu and type “virus”, and you should see “Virus & Threat Protection” pop-up:

Open that, and then you’ll be greeted by this screen:

Virus and Threat Protection(Windows Defender)

Under Virus & threat protection, go to “Manage settings”. In the settings, turn off everything. User Account Control will ask you if you want to make that change. Allow it to do so:

Next, we will turn off other Defender settings in Local Group Policy. Press Windows Key + R on your keyboard to open the run box. Then, in the run box, type “gpedit.msc”:

Hit “Ok” and you will be brought to Group Policy Edit. In the left-pane, Click on “Computer Configuration”. In the Right-pane, click on “Administrative Templates”, then “Windows Components”, then “Microsoft Defender Antivirus”. Next, click on the folder that says “Real-time Protection”:

Then double-click the setting that says “Turn off real-time protection”:

In the setting where it says “Not Configured”, change that to “Enabled”. Then click “Apply” and “Ok”. Back out of the “Real-time Protection” folder, and then double-click on the setting that says “Turn off Microsoft Defender Antivirus”:

Then, switch “Not configured” to “Enabled”. Hit “Apply” and “Ok”.

You can close out of the Group Policy Editor. Next, go back to the start menu. Type in “powershell”:

Right-click on “Windows PowerShell”, and hit “Run as Administrator”.

In the PowerShell console, type in “set-executionpolicy remotesigned":

Type in “A” and press Enter.

Lastly, we want to disable Windows Updates completely. There’s multiple ways to do this, but we will just disable the services for now.

Press “Windows key + R” once again and type in “services.msc”:

Next, scroll down until you find “Windows Update”:

Double-click on “Windows Update”, and where it says “Startup type”, change this to “Disabled”. After that, make sure to stop the service.

Lastly, create a snapshot of the VM:

At this point, we will now install Flare VM.

Installing Flare VM

By this point, you should have FlareVM downloaded from Github. Place the Zip archive on the desktop and unzip it:

Once unzipped, you should be greeted by the explorer window with a new folder:

Double-click on the folder, and in there you should see a PowerShell script named “install”:

In the address bar, copy the location of this script, and then open PowerShell as administrator:

Next in the PowerShell console, type “cd” to change directory, and paste the path like in the picture above. Then press “Enter”:

Next type “./install.ps1”. You may be asked to change the execution policy again. If so, press any key to exit and then type in “Set-ExecutionPolicy Unrestricted”. Lastly type “A”:

Then change the directory again. You can press the up arrow key to re-visit a previous command:

Now re-run the script. Note, if you are still having trouble with Windows Defender restart. If that doesn’t work, visit this link: https://www.windowscentral.com/how-permanently-disable-windows-defender-windows-10 OR: https://github.com/jeremybeaume/tools/blob/master/disable-defender.ps1(Must be in Safe Mode for this script to work)

Once this is done, re-run the script and you should pass each check at the beginning as long as everything was disabled:

When prompted during the install, always type “Y”.

If you have a password on the machine, make sure to enter it, otherwise press “Enter” and then let the script run. This may take several minutes to an hour. The max I’ve had to wait is up to two hours. This also depends on your internet connection:

FlareVM now has a GUI where you can choose what tools to install. Depending on how much space you have, you can choose to install as little or as much as you want. I just install everything because I love having different tools to play with:

Hit “Ok”, and let the install run. You can sit back, grab a cup of coffee and wait for the install to complete.

Once finished, you will get a log with any errors that may have occurred during the install. In my case, “PEStudio” and “Python3” did not install correctly so I will install them manually. Your lab should now look like this:

If you also look in “C:\Tools” you should have many tools at your disposal:

Next, I like to clean up the desktop and make shortcuts for all my tools:

You will need “oletools” for this series as we will be looking at macro embedded files such as “.docx”, “.xlsx”, “.pptx”, etc.. instructions to install these tools can be found here:

GitHub - decalage2/oletools: oletools - python tools to analyze MS OLE2 files (Structured Storage…

oletools - python tools to analyze MS OLE2 files (Structured Storage, Compound File Binary Format) and MS Office…

github.com

Additionally I installed PEStudio(Free): https://www.winitor.com/download

Resource Hacker: http://www.angusj.com/resourcehacker/#download

Unpacking tools: https://github.com/topics/unpacker

Visual Studio Code: https://code.visualstudio.com/

Lastly, we are going to go through how to setup the tool “ProcDOT” together. It is a tool that is used with Process Monitor, to give us an in-depth visual diagram of what malware samples are doing to the host.

Let’s start by going to: https://www.procdot.com/

Go to the “Downloads” section and download the Windows binaries:

When the Zip file downloads, unzip it. The password to unzip is “procdot”. Then, move the folder to your tools folder in “C:\Tools”:

The developer says to use the “readme” to install “ProcDOT” but we are going to do this together.

First let’s install GraphViz. ProcDOT needs this tool to create the visuals.

Go to: https://www.graphviz.org/download/

In the downloads section, download both Windows 32-bit and 64-bit installers:

Once downloaded, install both. If Windows “Smartscreen” comes up, Run Anyway.

Next, install WinPcap: https://www.winpcap.org/install/default.htm

Where it says “Version 4.1.3 installer for Windows”, download there and install.

If you get a notice that says “WinPcap” is already installed, you can skip this.

Next step, on the same page, click on “WinDump” and download that:

“WinDump” does not have an install file. To run it, it is done through command line. “ProcDOT” will handle that. What we need to do is store the binary in our tools.

Once you download “WinDump”, go back to “C:\Tools” and create a folder that says “WinDump”. Then, in the newly created folder, create another folder that says “bin” like this:

Drag the executable into the “bin” folder here.

Now we will need to make some changes in Process Monitor. First, open Process Monitor:

Next, we want to stop the capture. Press “CTRL +E” or click on the box icon next to the save icon to stop it:

Then click on the trashcan icon and we should be left with nothing:

Now, go to “options” and uncheck “Show Resolved Network Addresses”. ProcDOT will not work if this is on:

Next, check that “Enable Advanced Options” is unchecked in “filter”:

Next go to “Options” and “Select Columns”. Uncheck “Sequence Number” and check “Thread ID”:

We are now done in Process Monitor and you can close it out. The last thing we need to do here is go into ProcDOT.

Go back into your tools folder in C:\Tools and navigate to your ProcDOT folder. Open “procdot.exe”:

Upon opening, you will be prompted to set up ProcDOT for the first time:

Here we need to setup both paths. One path to WinDump and one to GraphViz. Click on the three dots next to “Path to windump/tcpdump” first, and choose the executable from “C:\Tools\WinDump\bin\”:

Next for GraphViz, I installed both the 32-bit version and the 64-bit version. I am currently running ProcDOT64 so we will point to the 64-bit path in “C:\Program Files\Graphviz\bin\dot.exe”:

If you setup ProcDOT32, you will need to find the bin in “C:\Program Files(x86)”.

After adding both paths, click “Ok”, and the setup process for ProcDOT is complete.

The next thing we want to do is create a snapshot of the VM like we did earlier:

Then wait for the snapshot to save before continuing. Once this is done I like to disable the virtual NIC (the network adapter) so that my VM stops all communications with the internet. In this tutorial, I will show you where to edit the settings to change this.

First turn off the VM completely.

Then, under “Devices”, click on where it says “NAT”:

This will open the VM’s settings. Under “Network Connection” change “NAT” to “Host-only” and click “Ok”:

This concludes the first blog post in this series. In the next posts we will be exploring different malware samples, tools, etc…

(Optional) Setup a VM for DNS and Wireshark

I mentioned earlier in this article that I like to setup an additional VM for DNS in the case that some malware samples will look for analysis tools like Wireshark and disable them or stop them from running. The way I found around this was to create a separate VM hosting Wireshark. This VM will have no connection to the internet. It will simply catch traffic coming from the lab. For this VM, I just install Windows 10, then I set the Fake DNS VM’s IP to static on the network adapter. Once I do that, I then take the static IP and set that as the DNS on the lab’s virtual NIC(network adapter). I’ll show you what I mean.

Go through the previous steps for creating a new VM in VMware Workstation. When you get to the summary screen, go to “Customize Hardware”, and change to “Host-only”. The VM will assign itself a “non-routable” IP address.

Start the VM and let Windows install. After that you will be brought to the desktop.

Next, open CMD and run “ipconfig”:

Whatever IP address is shown in the config is what you will use as the static IP. Make a note of this IP.

We will also make the Malware Lab static as well. In the Malware Lab VM, open the run box and type “control panel”:

Then, go to the “Network and Sharing” settings under “Network and Internet, and on the left-side click on “Change adapter settings”:

Next, double-click on the adapter that says “Ethernet0”:

You will see this screen:

Click on “Properties” and highlight “Internet Protocol version 4 (TCP/IP)”. Then hit “Properties” on the lower right:

Both VMs are in host-only mode sharing the same adapter (VMnet1) so they will be on the same subnet. The settings here reflect this. Yours will probably be different:

In this screenshot, the IP address ending in 129 is the malware lab. Where it says “Use the following DNS server addresses”, that is the IP address of the fake DNS server VM. If all is configured properly, they should be able to ping each other:

Then, I will transfer the Wireshark installer to the VM and install:

Lastly I just felt like setting a wallpaper and then created a snapshot:

Thanks for taking the time to read this blog. If you enjoyed it please stick around for the next blog in the series. Share with others who may be interested. As promised, here are some additional resources:

Lenny Zeltser’s blog:

Books:

Courses:

C Programming for Beginners — Master the C Language: https://www.udemy.com/course/c-programming-for-beginners-/
C++ Programming for Beginners: https://www.udemy.com/course/beginning-c-plus-plus-programming/learn/lecture/9535214#overview
Adventures in Assembly: https://www.udemy.com/course/x86-asm-foundations/learn/lecture/1366110#overview
x86 Assembly from the Ground Up: https://www.udemy.com/course/x86-assembly-programming-from-ground-uptm/learn/
Operating Systems from Scratch: https://www.udemy.com/user/vignesh-sekar-3/
Security+ by Jason Dion: https://www.udemy.com/course/securityplus/learn/lecture/12823205#overview
Network+ by Jason Dion: https://www.udemy.com/course/comptia-network-n10-008/learn/lecture/27856048#overview