Key highlight: Investigative forensic abilities and broad theoretical knowledge on computing and security are critical in securing online resources such as websites. Strategic, methodical progress in the investigation works best with clearly articulated, hypothesized security flaws and small calculated experiments to verify progress.

Introduction

Recently, my primary blog, https://sapalo.dev had been attacked by malware. This blog post covers my discovery and investigation process using WordFence, and the actions taken to remediate and prevent future security vulnerabilities.

The objective of this blog is to highlight the key skills that a software engineer needs, such as asking critical questions to analyze problems and defining mutually exclusive and completely exhaustive solutions to fix and secure their systems. These are critical problem skills that we teach at Tome.gg, a coaching and mentoring platform for mid- to senior-level software engineers.

Discovery

Last May 2023, I was notified by JetPack via email that my website was down and unavailable. I inspected it and found that accessing my blog was repeatedly redirected to different websites which was unexpected behavior. This immediately put me on urgent alert to investigate this hacking incident.

At this time, my current system reliability measure was backups provided by JetPack, that I could easily revert to. I also use All-in-One WordPress Migration plugin to download the whole website and back it up as a plan B, which I did manually at a certain schedule. Security-wise, my website could have had more measures in place. System observability was also lacking, which prevented me from deep investigation on what was occurring with my self-hosted WordPress site.

As I haven’t had many new content or blog posts lately, I decided to revert my website to an older version hoping to resolve the issue. This addressed the malicious redirects. In addition to this, I changed my admin account’s credentials as an added security measure.

I thought that the malware issues ended then.

A few days ago (June 2023), I received a notification from Google Search Console informing me that sapalo.dev was flagged as malicious and tagged as a phishing website. I visited my website and found that indeed, my Chrome browser did flag my website with bright red, informing me (a potential audience or viewer of the blog) of the security issues on the website.

Objectives

This time, I decided to take a deeper look into what was going on with my website. I decided to identify the following:

1. Google Search Console flagged phishing. Does this really occur, and if so, how?
2. Are there other potentially malicious behavior occurring that GSC did not detect?
3. How can I distinguish unauthorized access from authorized access?
   • How can I review the network traffic occurring on my website?
   • How can I prevent unauthorized access?
4. Up to what degree is my system infected? Is it at the application (WordPress) level only or does it reach the virtual machine (server host) level?
5. How can I better monitor my system so that I can proactively prevent these kinds of issues from occurring in the future?

Investigation

I consulted my security analyst and they pointed out to me using urlscan.io that my website was indeed going through a redirect journey which ends up at an adult pornography site. This redirect (and possibly to other portals) might be where the phishing occurs.

Investigating application source code

The first thing I did was connect to my remote instance so that I can investigate the application source code. I used FileZilla‘s SFTP option to connect to my host securely, and copied the application source code (Bitnami WordPress) into my local machine. At this point, Windows Defender detected malicious PHP files and immediately quarantined some of them. I decided to investigate that later. Then, I opened the directory using Visual Studio Code in restricted mode, which prevents remote code execution.

I focused on the wp-content directory to analyze any unusual code. Last May 2023, I found that there were some remote code execution occurring on my index.php files. I noticed them because of the obfuscated nature of the code. For an example, it looked somewhat similar to these obfuscated code.

From my observations, I noted that there were two different kinds of malicious code: (a) legitimate code with infection such as wp-content/index.php and (b) complete malicious files such as wp-content/media/2023/05/abcdef123.php.

After seeing examples of obfuscated PHP code, I copied a portion of it so that I can easily pattern match and search it across the application. I found more instances of them replicated on a variety of different PHP files. The wp-content/uploads/ directory as well are expected to contain media files (PNG, JPG, etc.) and not PHP files, so that was also an area that I investigated and deleted malicious files in.

Action taken:

• I decided to fix the legitimate code that was infected (removing unneeded, illogical, obfuscated PHP code and retaining actual WordPress core application code);
• Took note of the names of the malicious files and deleted them.
• Deleted malicious media files in wp-content/uploads that were PHP files, not media files.

Curiously, in one of the uploads directory, I found a gzipped file of xmrig, which is an open-source crypto mining application. I deleted it and moved on.

Analyzing the obfucated code

Because of my curiosity on the obfuscated PHP code, I went to unphp.net to decode it and analyze it. After the first pass of decoding, it resulted into four PHP lines of code:

1. A large PHP string definition.
2. Decoding (base64) that PHP string definition.
3. Inflating that decoded string.
4. Running eval() which is a PHP function that allows you to run arbitrary PHP code.

I used onlinephp.io to decode the large PHP string definition, and found the malware source code, and found that it did the following (non-exhaustive list):

1. Searching for files on the remote host.
2. Constructing phishing HTML forms (login inputs, etc).
3. Replicating the malicious code elsewhere in the application.

Very scary stuff. Anyway, the call-to-action for these malicious files is to delete them. Let’s get back to the investigation to secure our system.

Ephemeral malicious code execution

Connected to FileZilla, I found that refreshing the folder directory sometimes shows that there are some folders being created and immediately deleted. Their folder directory names were a short 8-letter hexadecimal string.

I wondered what process could be running to create and delete it. I hypothesized that this could possibly be a malicious process that is running itself, replicating itself to a different file directory, deleting its old file version (to hide itself), and running the new copy again.

Action taken:

• I decided to turn off my host instance completely (thus resetting the running processes);
• I deleted the malicious directories that remained.

Refreshing the folder directory using FileZilla afterwards showed that it no longer created folder directories every few seconds.

Using WordFence for observability

I read a little bit more on WordPress security and found that a recommended plugin was WordFence. I installed it (free version) and began to study how I could use its tools and configure it to prevent the malware.

Interestingly, WordFence had very useful features:

• It had a scanner that allowed you to inspect your application’s source code, performing pattern-matching to find any potentially malicious code. This was super useful in helping me exhaustively inspect my source code, rather than my initial, manual efforts.
• It had a logging system that allowed you to review network requests to your system. This helped me understand more the malicious hosts attempting to connect to my website, trying to access the PHP files I had already deleted. This let me analyze the patterns of their requests so that I can block them later.
• It also had 2FA support, which allowed you to require 2FA configuration for all of your users. For me, as this was a personal blog that I managed alone, I simply configured 2FA for myself and enabled 2FA for all users.
• It also has a blocking system, which allows you to block by IP or by IP ranges, user agent, and many other options. The paid version of WordFence also allows you to block access by country.

WordFence gave a lot of insights into how my system was being interacted with by the malicious actors, and helped me determine patterns that I could use to distinguish bad actors from my real audiences and customers.

Using CloudFlare for WAF rules

I use CloudFlare to manage my DNS and other security related capabilities for my websites. I reviewed the features that CloudFlare provided and found that they have Web Application Firewall (WAF) which allows you to define rules that allow or block requests right away from the DNS level, before it even reaches my host machine.

Based on the investigation I did above, I defined rules (e.g. accessing the malicious files, which were statically named) which would result to a blocked request.

Summary

Overall, it was a troublesome and tedious process to investigate, fix, and secure the system. But after applying all of the changes, I could verify through WordFence that the malicious requests no longer were being detected (i.e. CloudFlare is working) and I had greater confidence that my application was secure.