The 2 AM Wake-Up Call: A Tale of Two Vulnerabilities
PagerDuty doesn’t care about your sleep schedule. At 2:14 AM, my phone’s vibration nearly rattled it off the nightstand, signaling a 400% spike in 500 errors on our production API.
By the time I logged in, our SIEM was already screaming about suspicious outbound traffic. A web server was trying to talk to a rogue IP in a data center halfway across the world. Someone wasn’t just poking around; they were exfiltrating data using two of the most dangerous vulnerabilities in modern stacks: XML External Entity (XXE) and Server-Side Template Injection (SSTI).
Sifting through the access logs revealed a coordinated attack. One actor was bombarding an old SOAP endpoint with malformed XML payloads. Simultaneously, another was hitting our dynamic reporting engine with strange curly-brace syntax. This wasn’t a random bot. It was a targeted attempt to bypass our WAF by weaponizing the very parsers we trusted to handle our data.
Root Cause 1: When XML Parsers Talk to Strangers (XXE)
The first fire I had to put out was the XXE vulnerability. While XML feels like a relic, it’s still the backbone of SAML tokens and legacy APIs. The danger isn’t the format itself. It’s the Document Type Definition (DTD) feature that many libraries leave enabled by default.
The Exploit Mechanism
An attacker sends an XML document that defines an “external entity.” If the parser is misconfigured, it will fetch files from the local disk or make network requests on the attacker’s behalf. During the incident, I found this payload hitting our `/api/v1/shipping` endpoint:
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE foo [
<!ELEMENT foo ANY >
<!ENTITY xxe SYSTEM "file:///etc/passwd" >]>
<foo>&xxe;</foo>The parser sees &xxe; and obediently replaces it with the contents of /etc/passwd. If that result is reflected in the API response, the attacker can read any file the web service can access. In our case, they were also using http:// schemes to scan our internal Redis instance on port 6379, effectively turning our server into a proxy for their reconnaissance.
Why it happened
Our backend was running an unhardened version of lxml 3.7.1. We assumed the WAF would block these attempts, but the attacker used UTF-16 encoding to slip past the regex filters. It was a classic mistake: relying on perimeter defense instead of securing the code.
Root Cause 2: Logic Execution via SSTI
While I was patching the XML parser, a second alert triggered. Our custom dashboard tool, which lets users customize email templates, was being abused. We were using the Jinja2 engine and thought we were safe because we weren’t using eval(). We were wrong.
The Exploit Mechanism
Server-Side Template Injection (SSTI) occurs when user input is concatenated directly into a template string. The attacker realized that entering {{ 7*7 }} in a “Company Name” field resulted in the preview showing 49. That’s the smoking gun for code execution.
They didn’t stop at math. They used Python’s Method Resolution Order (MRO) to climb the object tree and grab the os module. This was the exact payload that gave them a shell:
{{ self.__init__.__globals__['__builtins__']['__import__']('os').popen('id').read() }}This gave them Remote Code Execution (RCE). They weren’t just reading files; they were running system commands with the privileges of our web worker.
Comparing the Fixes: Band-aids vs. Real Security
When production is melting down, it’s tempting to write a quick regex to block ENTITY or {{. Don’t do that. Attackers can bypass these filters using hex encoding, whitespace variations, or alternative tags like {%. Filters are just speed bumps.
The Failed Approach: Blacklisting
I’ve seen teams try to block keywords like SYSTEM or __globals__. This is a losing game. In XML, you can change the character encoding to bypass string matching. In SSTI, you can use dictionary access like request['__class__'] to avoid dot-notation filters. You cannot out-regex a determined attacker.
The Definitive Fix: Hardening the Library
The only real way to stop XXE is to tell the parser to ignore DTDs entirely. For SSTI, you must strictly separate code from data. No exceptions.
Implementing Practical Protections
After neutralizing the immediate threat, I refactored our utility libraries. Here is how you should handle these issues in a production environment.
1. Securing XML Parsers (Python Example)
If you are using lxml, you must explicitly disable external entities and DTD loading. I updated our shared configuration to use this safe setup:
from lxml import etree
# Secure parser configuration
parser = etree.XMLParser(
resolve_entities=False,
no_network=True,
dtd_validation=False,
load_dtd=False
)
def safe_parse_xml(xml_string):
# This raises an error if DTDs or external entities are detected
return etree.fromstring(xml_string, parser=parser)For Java environments using Jackson or DocumentBuilderFactory, you must set the disallow-doctype-decl feature to true. Always verify your library’s defaults against the OWASP prevention cheat sheet.
2. Defeating SSTI with Logic Separation
Fixing SSTI requires discipline. Never use string formatting to build your templates.
Vulnerable:
template = "Hello " + user_input
render_template_string(template)
Secure:
render_template_string("Hello {{ name }}", name=user_input)
By passing the input as a variable, the engine treats it as a literal string. It won’t execute anything inside it. If you must allow users to provide their own templates, use a sandboxed environment with a strict whitelist of allowed functions.
The Best Approach: Defense in Depth
Security isn’t a single line of code; it’s a stack. After patching the vulnerabilities, I took three extra steps to harden our infrastructure.
- Least Privilege: We moved the web worker to a restricted user profile. Even with RCE, the attacker couldn’t read
/root/or install tools likenmap. - Egress Filtering: We configured the firewall to block all outbound connections from the web tier unless they were to a whitelisted internal service. This kills XXE-based SSRF and prevents reverse shells from connecting back to the attacker.
- Secure Secret Generation: During the cleanup, I had to rotate several admin passwords. I used the browser-based generator at toolcraft.app/en/tools/security/password-generator. Because it runs entirely on the client side, I didn’t have to worry about the new secrets being intercepted over the network.
Summary Checklist
- For XXE: Disable DTDs and External Entity resolution in every parser. Switch to JSON for new endpoints.
- For SSTI: Use static templates. Never use string concatenation for user-provided data.
- Monitoring: Set up alerts for
{{,<!ENTITY, and/etc/passwdin your application logs.
By sunrise, the patches were live and the logs were clean. These vulnerabilities are old, but they remain effective because we often trust our libraries to be secure by default. They aren’t. As engineers, it’s our job to verify those defaults before the 2 AM pager goes off.
