Challenge Description
We found this on an old machine. Can you decrypt it?
Files provided:
cryptpad.exe - Windows PE32 executable (encryption tool)flag.enc - Encrypted flag file (64 bytes)
Analysis
Initial Reconnaissance
The challenge provides a Windows executable and an encrypted file. Running file on the binary:
cryptpad.exe: PE32 executable (GUI) Intel 80386, for MS Windows
Strings in the binary reveal:
- “CryptPad 1.0 is an encrypted notepad that uses a custom encryption algorithm”
- References to “Crackmes.One” (confirming this is a reversing challenge)
Reverse Engineering with IDA
Using IDA Pro to decompile the binary, the main encryption function at 0x4014EB reveals the algorithm:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
// Decryption mode (a3 == 0)
if ( a3 == 0 ) {
// Extract key from end of file
v3 = *(_DWORD *)&a1[a2 - 1]; // Key size (last byte)
v4 = &a1[a2 - 1 - v3]; // Key location
qmemcpy(byte_4024C5, v4, v3); // Copy key
v5 = *((_DWORD *)v4 - 1); // Original size
}
// Step 1: XOR with key
do {
*v7++ ^= *v8++;
if ( ++v10 == 8 ) goto LABEL_6; // Cycle 8-byte key
} while ( v9 );
// Step 2: RC4 Key Scheduling Algorithm (KSA)
// Initialize S-box and mix with key...
// Step 3: RC4 PRGA - XOR data with keystream
do {
// Standard RC4 PRGA
v24 = (v21 + 1) % 256;
v22 = (v22 + S[v24]) % 256;
swap(S[v24], S[v22]);
data[v21] ^= S[(S[v24] + S[v22]) % 256];
} while (--length);
// Step 4: XOR with key again
do {
*v27++ ^= *v28++;
if ( ++v30 == 8 ) goto LABEL_20;
} while ( v29 );
|
Encryption Scheme
The “custom encryption” is:
- XOR data with 8-byte key (cycling)
- RC4 encrypt with same key
- XOR with key again
File Structure
The encrypted file format stores the key at the end:
[Encrypted Data (N bytes)]
[Original Size (4 bytes, little-endian)]
[Key (8 bytes)]
[Key Length (1 byte)]
For flag.enc (64 bytes):
- Key length (byte 63):
0x08
- Key (bytes 55-62):
e8 17 1b f4 50 3f 3d 70
- Original size (bytes 51-54):
0x1c (28 bytes)
- Encrypted data: first 51 bytes
Solution
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
#!/usr/bin/env python3
def xor_key(data, key):
"""XOR data with repeating key"""
return bytearray(data[i] ^ key[i % len(key)] for i in range(len(data)))
def rc4(key, data):
"""RC4 stream cipher"""
# Key Scheduling Algorithm (KSA)
S = list(range(256))
j = 0
for i in range(256):
j = (j + S[i] + key[i % len(key)]) % 256
S[i], S[j] = S[j], S[i]
# Pseudo-Random Generation Algorithm (PRGA)
i = j = 0
result = bytearray()
for byte in data:
i = (i + 1) % 256
j = (j + S[i]) % 256
S[i], S[j] = S[j], S[i]
result.append(byte ^ S[(S[i] + S[j]) % 256])
return result
def decrypt(filepath):
with open(filepath, 'rb') as f:
enc = bytearray(f.read())
# Extract key from end of file
key_size = enc[-1]
key = bytes(enc[-1-key_size:-1])
# Original size is 4 bytes before the key
orig_size_pos = len(enc) - 1 - key_size - 4
orig_size = int.from_bytes(enc[orig_size_pos:orig_size_pos+4], 'little')
# Data to decrypt
data = bytearray(enc[:orig_size_pos])
# Decrypt: XOR -> RC4 -> XOR
step1 = xor_key(data, key)
step2 = rc4(key, step1)
step3 = xor_key(step2, key)
return step3[:orig_size].decode('utf-8', errors='ignore')
if __name__ == '__main__':
flag = decrypt('flag.enc')
print(f'Flag: {flag}')
|
Flag
CMO{r0ll_y0ur_0wn_b4d_c0d3}
Lessons Learned
The flag “roll your own bad code” is a reference to the security principle “Don’t roll your own crypto”. This challenge demonstrates why:
- The encryption key is stored in plaintext at the end of the encrypted file
- The XOR-RC4-XOR scheme provides no additional security over plain RC4
- Custom cryptographic implementations often have subtle flaws that make them trivially breakable
Always use well-vetted, standard cryptographic libraries instead of implementing your own.