CTF Reverse Engineering
Quick reference for RE challenges. For detailed techniques, see supporting files.
Additional Resources
- tools.md - Static analysis tools (GDB, Ghidra, radare2, IDA, Binary Ninja, dogbolt.org, RISC-V with Capstone, Unicorn emulation, Python bytecode, WASM, Android APK, .NET, packed binaries)
- tools-dynamic.md (includes Intel Pin instruction-counting side channel for movfuscated binaries) - Dynamic analysis tools: Frida (hooking, anti-debug bypass, memory scanning, Android/iOS), angr symbolic execution (path exploration, constraints, CFG), lldb (macOS/LLVM debugger), x64dbg (Windows), Qiling (cross-platform emulation with OS support), Triton (dynamic symbolic execution)
- tools-advanced.md - Advanced tools: VMProtect/Themida analysis, binary diffing (BinDiff, Diaphora), deobfuscation frameworks (D-810, GOOMBA, Miasm), Rizin/Cutter, RetDec, advanced GDB (Python scripting, conditional breakpoints, watchpoints, reverse debugging with rr, pwndbg/GEF), advanced Ghidra scripting, patching (Binary Ninja API, LIEF)
- ebpf-analysis.md - eBPF program analysis: structure/opcodes, identifying eBPF in challenges, bpftool disassembly, map data extraction, verifier constraints, extracting eBPF from binaries/PCAP, bpftrace dynamic analysis, CTF patterns (flag in map, eBPF as VM, seccomp filter bypass)
- js-engine-re.md - JS engine & LLVM obfuscation: V8 object layout/tagged pointers/bytecode disassembly, SpiderMonkey shapes, JIT spray recognition, LLVM control flow flattening (D-810/angr deobfuscation), bogus control flow (Miasm), instruction substitution (sympy), JS AST deobfuscation (babel), eval/Function hook, V8 snapshot flag extraction, Wasm+JS combined challenges
- anti-analysis.md - Comprehensive anti-analysis: Linux anti-debug (ptrace, /proc, timing, signals, direct syscalls), Windows anti-debug (PEB, NtQueryInformationProcess, heap flags, TLS callbacks, HW/SW breakpoint detection, exception-based, thread hiding), anti-VM/sandbox (CPUID, MAC, timing, artifacts, resources), anti-DBI (Frida detection/bypass), code integrity/self-hashing, anti-disassembly (opaque predicates, junk bytes), MBA identification/simplification, bypass strategies
- patterns.md - Foundational binary patterns: custom VMs, anti-debugging, nanomites, self-modifying code, XOR ciphers, mixed-mode stagers, LLVM obfuscation, S-box/keystream, SECCOMP/BPF, exception handlers, memory dumps, byte-wise transforms, x86-64 gotchas, signal-based exploration, malware anti-analysis, multi-stage shellcode, timing side-channel, multi-thread anti-debug with decoy + signal handler MBA
- patterns-ctf.md - Competition-specific patterns (Part 1): hidden emulator opcodes, LD_PRELOAD key extraction, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Windows PE XOR bitmap OCR, two-stage RC4+VM loaders, GBA ROM meet-in-the-middle, Sprague-Grundy game theory, kernel module maze solving, multi-threaded VM channels, backdoored shared library detection via string diffing
- patterns-ctf-2.md - Competition-specific patterns (Part 2): multi-layer self-decrypting brute-force, embedded ZIP+XOR license, stack string deobfuscation, prefix hash brute-force, CVP/LLL lattice for integer validation, decision tree function obfuscation, GLSL shader VM, GF(2^8) Gaussian elimination, Z3 single-line Python circuit, sliding window popcount, keyboard LED Morse code via ioctl
- languages.md - Language/platform-specific: Python bytecode & opcode remapping, Python version-specific bytecode, Pyarmor static unpack, DOS stubs, Unity IL2CPP, HarmonyOS HAP/ABC, Brainfuck/esolangs, UEFI, transpilation to C, code coverage side-channel, OPAL functional reversing, non-bijective substitution, Roblox place file analysis, Godot game asset extraction, Rust serde_json schema recovery, Verilog/hardware RE, Android JNI RegisterNatives, Ruby/Perl polyglot, Electron ASAR extraction + native binary analysis, Node.js npm runtime introspection
- languages-compiled.md - Go binary reversing (GoReSym, goroutines, memory layout, channel ops, embed.FS), Rust binary reversing (demangling, Option/Result, Vec, panic strings), Swift binary reversing (demangling, protocol witness tables), Kotlin/JVM (coroutine state machines), C++ (vtable reconstruction, RTTI, STL patterns)
- platforms.md - Platform-specific RE: macOS/iOS (Mach-O, code signing, Objective-C runtime, Swift, dyld, jailbreak bypass), embedded/IoT firmware (binwalk, UART/JTAG/SPI extraction, ARM/MIPS, RTOS), kernel drivers (Linux .ko, eBPF, Windows .sys), game engines (Unreal Engine, Unity, anti-cheat, Lua), automotive CAN bus, RISC-V advanced
Output Size Rules
CRITICAL: Never print large data directly to the conversation. Always write to a file first.
bash
# BAD — causes token limit errors:
strings binary | grep -A2 base64 # may output megabytes
cat large_output.txt # may exceed 32k tokens
# GOOD — write to file, then read selectively:
strings binary > /tmp/strings_out.txt
wc -l /tmp/strings_out.txt # check size first
grep -i 'base64\|flag\|key\|pass' /tmp/strings_out.txt | head -50
# Extracting embedded base64 — always write decoded result to file:
strings binary | grep -E '^[A-Za-z0-9+/]{40,}={0,2}$' > /tmp/b64_candidates.txt
wc -l /tmp/b64_candidates.txt
# Then decode each candidate to file:
head -1 /tmp/b64_candidates.txt | base64 -d > /tmp/decoded_1.bin
file /tmp/decoded_1.bin # identify what it is
# For Python bytecode / large blobs:
base64 -d /tmp/b64_candidates.txt > /tmp/payload.bin
file /tmp/payload.bin # check type before reading
python3 -c "import dis,marshal; dis.dis(marshal.loads(open('/tmp/payload.bin','rb').read()[16:]))" \
> /tmp/disasm.txt 2>&1
wc -l /tmp/disasm.txt
grep -i 'flag\|key\|pass\|LOAD_CONST' /tmp/disasm.txt | head -30
Rule: if output might exceed 200 lines, pipe to a file. Read with grep/head/tail.
Problem-Solving Workflow
- Start with strings extraction - many easy challenges have plaintext flags
- Try ltrace/strace - dynamic analysis often reveals flags without reversing
- Try Frida hooking - hook strcmp/memcmp to capture expected values without reversing
- Try angr - symbolic execution solves many flag-checkers automatically
- Try Qiling - emulate foreign-arch binaries or bypass heavy anti-debug without artifacts
- Map control flow before modifying execution
- Automate manual processes via scripting (r2pipe, Frida, angr, Python)
- Validate assumptions by comparing decompiler outputs (dogbolt.org for side-by-side)
Quick Wins (Try First!)
bash
# Plaintext flag extraction
strings binary | grep -E "flag\{|CTF\{|pico"
strings binary | grep -iE "flag|secret|password"
rabin2 -z binary | grep -i "flag"
# Dynamic analysis - often captures flag directly
ltrace ./binary
strace -f -s 500 ./binary
# Hex dump search
xxd binary | grep -i flag
# Run with test inputs
./binary AAAA
echo "test" | ./binary
Initial Analysis
bash
file binary # Type, architecture
checksec --file=binary # Security features (for pwn)
chmod +x binary # Make executable
Memory Dumping Strategy
Key insight: Let the program compute the answer, then dump it. Break at final comparison (
), enter any input of correct length, then
to dump computed flag.
Decoy Flag Detection
Pattern: Multiple fake targets before real check.
Identification:
- Look for multiple comparison targets in sequence
- Check for different success messages
- Trace which comparison is checked LAST
Solution: Set breakpoint at FINAL comparison, not earlier ones.
GDB PIE Debugging
PIE binaries randomize base address. Use relative breakpoints:
bash
gdb ./binary
start # Forces PIE base resolution
b *main+0xca # Relative to main
run
Comparison Direction (Critical!)
Two patterns:
transform(flag) == stored_target
- Reverse the transform
transform(stored_target) == flag
- Flag IS the transformed data!
Pattern 2 solution: Don't reverse - just apply transform to stored target.
Common Encryption Patterns
- XOR with single byte - try all 256 values
- XOR with known plaintext (, )
- RC4 with hardcoded key
- Custom permutation + XOR
- XOR with position index ( or ) layered with a repeating key
Quick Tool Reference
bash
# Radare2
r2 -d ./binary # Debug mode
aaa # Analyze
afl # List functions
pdf @ main # Disassemble main
# Ghidra (headless)
analyzeHeadless project/ tmp -import binary -postScript script.py
# IDA
ida64 binary # Open in IDA64
Binary Types
Python .pyc
Disassemble with
+
. Header: 8 bytes (2.x), 12 (3.0-3.6), 16 (3.7+). See
languages.md.
WASM
bash
wasm2c checker.wasm -o checker.c
gcc -O3 checker.c wasm-rt-impl.c -o checker
# WASM patching (game challenges):
wasm2wat main.wasm -o main.wat # Binary → text
# Edit WAT: flip comparisons, change constants
wat2wasm main.wat -o patched.wasm # Text → binary
WASM game patching (Tac Tic Toe, Pragyan 2026): If proof generation is independent of move quality, patch minimax (flip
→
, change bestScore sign) to make AI play badly while proofs remain valid. Invoke
for full game patching patterns (games-and-vms).
Android APK
apktool d app.apk -o decoded/
for resources;
for Java decompilation. Check
decoded/res/values/strings.xml
for flags. See
tools.md.
Flutter APK (Dart AOT)
If
+
present, use
Blutter:
python3 blutter.py path/to/app/lib/arm64-v8a out_dir
. Outputs reconstructed Dart symbols + Frida script. See
tools.md.
.NET
- dnSpy - debugging + decompilation
- ILSpy - decompiler
Packed (UPX)
bash
upx -d packed -o unpacked
If unpacking fails, inspect UPX metadata first: verify UPX section names, header fields, and version markers are intact. If metadata looks tampered or uncertain, review UPX source on GitHub to identify likely modification points.
Tauri Packed Desktop Apps
Tauri embeds Brotli-compressed frontend assets in the executable. Find
xrefs to locate asset index table, dump blobs, Brotli decompress. Reference:
tauri-codegen/src/embedded_assets.rs
.
Anti-Debugging Bypass
Common checks:
- / PEB.BeingDebugged / NtQueryInformationProcess (Windows)
- / TracerPid (Linux)
- TLS callbacks (run before main — check PE TLS Directory)
- Timing checks (, , )
- Hardware breakpoint detection (DR0-DR3 via GetThreadContext)
- INT3 scanning / code self-hashing (CRC over .text section)
- Signal-based: SIGTRAP handler, SIGALRM timeout, SIGSEGV for real logic
- Frida/DBI detection: scan, port 27042, inline hook checks
Bypass: Set breakpoint at check, modify register to bypass conditional.
pwntools patch:
elf.asm(elf.symbols.ptrace, 'ret')
to replace function with immediate return. See
patterns.md.
For comprehensive anti-analysis techniques and bypasses (30+ methods with code), see anti-analysis.md.
S-Box / Keystream Patterns
Xorshift32: Shifts 13, 17, 5
Xorshift64: Shifts 12, 25, 27
Magic constants: ,
Custom VM Analysis
- Identify structure: registers, memory, IP
- Reverse for opcode meanings
- Write disassembler mapping opcodes to mnemonics
- Often easier to bruteforce than fully reverse
- Look for the bytecode file loaded via command-line arg
See patterns.md for VM workflow, opcode tables, and state machine BFS.
Python Bytecode Reversing
XOR flag checkers with interleaved even/odd tables are common. See languages.md for bytecode analysis tips and reversing patterns.
Signal-Based Binary Exploration
Binary uses UNIX signals as binary tree navigation; hook
via
, DFS by sending signals. See
patterns.md.
Malware Anti-Analysis Bypass via Patching
Flip
/
(0x75/0x74), change sleep values, patch environment checks in Ghidra (
). See
patterns.md.
Expected Values Tables
Locating:
bash
objdump -s -j .rodata binary | less
# Look near comparison instructions
# Size matches flag length
x86-64 Gotchas
Sign extension and 32-bit truncation pitfalls. See patterns.md for details and code examples.
Iterative Solver Pattern
Try each byte (0-255) per position, match against expected output. Uniform transform shortcut: if one input byte only changes one output byte, build 0..255 mapping then invert. See patterns.md for full implementation.
Unicorn Emulation (Complex State)
-- map segments, set up stack, hook to trace.
Mixed-mode pitfall: 64-bit stub jumping to 32-bit via
requires switching to UC_MODE_32 and copying GPRs + EFLAGS + XMM regs. See
tools.md.
Multi-Stage Shellcode Loaders
Nested shellcode with XOR decode loops; break at
, bypass ptrace with
, extract flag from
instructions. See
patterns.md.
Timing Side-Channel Attack
Validation time varies per correct character; measure elapsed time per candidate to recover flag byte-by-byte. See patterns.md.
Godot Game Asset Extraction
Use KeyDot to extract encryption key from executable, then gdsdecomp to extract .pck package. See languages.md.
Roblox Place File Analysis
Query Asset Delivery API for version history; parse
chunks (INST/PROP/PRNT) to diff script sources across versions. See
languages.md.
Unstripped Binary Information Leaks
Pattern (Bad Opsec): Debug info and file paths leak author identity.
Quick checks:
bash
strings binary | grep "/home/" # Home directory paths
strings binary | grep "/Users/" # macOS paths
file binary # Check if stripped
readelf -S binary | grep debug # Debug sections present?
Custom Mangle Function Reversing
Binary mangles input 2 bytes at a time with running state; extract target from
, write inverse function. See
patterns.md.
Rust serde_json Schema Recovery
Disassemble serde
implementations to recover expected JSON schema; field names in order reveal flag. See
languages.md.
Position-Based Transformation Reversing
Binary adds/subtracts position index; reverse by undoing per-index offset. See patterns.md.
Hex-Encoded String Comparison
Input converted to hex, compared against constant. Decode with
. See
patterns.md.
Embedded ZIP + XOR License Decryption
Binary with named symbols (
,
) in
→ extract ZIP containing license, XOR encrypted message with license bytes to recover flag. No execution needed. See
patterns-ctf-2.md.
Stack String Deobfuscation (.rodata XOR Blob)
Binary mmaps
blob, XOR-deobfuscates, uses it to validate input. Reimplement verification loop with pyelftools to extract blob. Look for
,
constants and
. See
patterns-ctf-2.md.
Prefix Hash Brute-Force
Binary hashes every prefix independently. Recover one character at a time by matching prefix hashes. See patterns-ctf-2.md.
Mathematical Convergence Bitmap
Pattern: Binary classifies coordinate pairs by Newton's method convergence (e.g., z^3-1=0). Grid of pass/fail results renders ASCII art flag. Key: the binary is a classifier, not a checker — reverse the math and visualize. See patterns-ctf.md.
RISC-V Binary Analysis
Statically linked, stripped RISC-V ELF. Use Capstone with
CS_MODE_RISCVC | CS_MODE_RISCV64
for mixed compressed instructions. Emulate with
. Watch for fake flags and XOR decryption with incremental keys. See
tools.md.
Sprague-Grundy Game Theory Binary
Game binary plays bounded Nim with PRNG for losing-position moves. Identify game framework (Grundy values = pile % (k+1), XOR determines position), track PRNG state evolution through user input feedback. See patterns-ctf.md.
Kernel Module Maze Solving
Rust kernel module implements maze via device ioctls. Enumerate commands dynamically, build DFS solver with decoy avoidance, deploy as minimal static binary (raw syscalls, no libc). See patterns-ctf.md.
Multi-Threaded VM with Channels
Custom VM with 16+ threads communicating via futex channels. Trace data flow across thread boundaries, extract constants from GDB, watch for inverted validity logic, solve via BFS state space search. See patterns-ctf.md.
CVP/LLL Lattice for Constrained Integer Validation (HTB ShadowLabyrinth)
Binary validates flag via matrix multiplication with 64-bit coefficients; solutions must be printable ASCII. Use LLL reduction + CVP in SageMath to find nearest lattice point in the constrained range. Two-phase pattern: Phase 1 recovers AES key, Phase 2 decrypts custom VM bytecode with another linear system (mod 2^32). See patterns-ctf-2.md.
Decision Tree Function Obfuscation (HTB WonderSMS)
~200+ auto-generated functions routing input through polynomial comparisons. Script extraction via Ghidra headless rather than reversing each function manually. Constraint propagation from known output format cascades through arithmetic constraints. See patterns-ctf-2.md.
Android JNI RegisterNatives Obfuscation (HTB WonderSMS)
in
hides which C++ function handles each Java native method (no standard
Java_com_pkg_Class_method
symbol). Find the real handler by tracing
→
→
. Use x86_64
from APK for best Ghidra decompilation. See
languages.md.
Multi-Layer Self-Decrypting Binary
N-layer binary where each layer decrypts the next using user-provided key bytes + SHA-NI. Use oracle (correct key → valid code with expected pattern). JIT execution with fork-per-candidate COW isolation for speed. See patterns-ctf-2.md.
GLSL Shader VM with Self-Modifying Code
Pattern: WebGL2 fragment shader implements Turing-complete VM on a 256x256 RGBA texture (program memory + VRAM). Self-modifying code (STORE opcode) patches drawing instructions. GPU parallelism causes write conflicts — emulate sequentially in Python to recover full output. See patterns-ctf-2.md.
GF(2^8) Gaussian Elimination for Flag Recovery
Pattern: Binary performs Gaussian elimination over GF(2^8) with the AES polynomial (0x11b). Matrix + augmentation vector in
; solution vector is the flag. Look for constant
in disassembly. Addition is XOR, multiplication uses polynomial reduction. See
patterns-ctf-2.md.
Z3 for Single-Line Python Boolean Circuit
Pattern: Single-line Python (2000+ semicolons) with walrus operator chains validates flag as big-endian integer via boolean circuit. Obfuscated XOR
. Split on semicolons, translate to Z3 symbolically, solve in under a second. See
patterns-ctf-2.md.
Sliding Window Popcount Differential Propagation
Pattern: Binary validates input via expected popcount for each position of a 16-bit sliding window. Popcount differences create a recurrence:
bit[i+16] = bit[i] + (data[i+1] - data[i])
. Brute-force ~4000-8000 valid initial 16-bit windows; each determines the entire bit sequence. See
patterns-ctf-2.md.
Ruby/Perl Polyglot Constraint Satisfaction
Pattern: Single file valid in both Ruby and Perl, each imposing different constraints on a key. Exploits
/
(Ruby block comment) vs
/
(Perl POD) to run different code per interpreter. Intersect constraints from both languages to recover the unique key. See
languages.md.
Verilog/Hardware RE
Pattern: Verilog HDL source for state machines with hidden conditions gated on shift register history. Analyze
blocks and
statements to find correct input sequences. See
languages.md.
Backdoored Shared Library Detection
Binary works in GDB but fails when run normally (suid)? Check
for non-standard libc paths, then
the suspicious vs. system library to find injected code/passwords. See
patterns-ctf.md.
Go Binary Reversing
Large static binary with
? Use GoReSym to recover function names (works even on stripped binaries). Go strings are
pairs — not null-terminated. Look for
,
, channel ops (
/
). Use Ghidra golang-loader plugin for best results. See
languages-compiled.md.
Rust Binary Reversing
Binary with
strings and
mangled symbols? Use
for demangling. Panic messages contain source paths and line numbers —
strings binary | grep "panicked"
is the fastest approach. Option/Result enums use discriminant byte (0=None/Err, 1=Some/Ok). See
languages-compiled.md.
Frida Dynamic Instrumentation
Hook runtime functions without modifying binary.
frida -f ./binary -l hook.js
to spawn with instrumentation. Hook
/
to capture expected values, bypass anti-debug by replacing
return value, scan memory for flag patterns, replace validation functions. See
tools-dynamic.md.
angr Symbolic Execution
Automatic path exploration to find inputs satisfying constraints. Load binary with
, set find/avoid addresses, call
. Constrain input to printable ASCII and known prefix for faster solving. Hook expensive functions (crypto, I/O) to prevent path explosion. See
tools-dynamic.md.
Qiling Emulation
Cross-platform binary emulation with OS-level support (syscalls, filesystem). Emulate Linux/Windows/ARM/MIPS binaries on any host. No debugger artifacts — bypasses all anti-debug by default. Hook syscalls and addresses with Python API. See tools-dynamic.md.
VMProtect / Themida Analysis
VMProtect virtualizes code into custom bytecode. Identify VM entry (pushad-like), find handler table (large indirect jump), trace handlers dynamically. For CTF, focus on tracing operations on input rather than full devirtualization. Themida: dump at OEP with ScyllaHide + Scylla. See tools-advanced.md.
Binary Diffing
BinDiff and Diaphora compare two binaries to highlight changes. Essential when challenge provides patched/original versions. Export from IDA/Ghidra, diff to find vulnerability or hidden functionality. See tools-advanced.md.
Advanced GDB (pwndbg, rr)
pwndbg:
,
,
,
. GEF alternative. Reverse debugging with
/
— step backward through execution. Python scripting for brute-force and automated tracing. See
tools-advanced.md.
macOS / iOS Reversing
Mach-O binaries:
for load commands,
for Objective-C headers. Swift:
for symbols. iOS apps: decrypt FairPlay DRM with frida-ios-dump, bypass jailbreak detection with Frida hooks. Re-sign patched binaries with
. See
platforms.md.
Embedded / IoT Firmware RE
for recursive extraction. Hardware: UART/JTAG/SPI flash for firmware dumps. Filesystems: SquashFS (
), JFFS2, UBI. Emulate with QEMU:
qemu-arm -L /usr/arm-linux-gnueabihf/ ./binary
. See
platforms.md.
Kernel Driver Reversing
Linux
: find ioctl handler via
struct, trace
/
. Debug with QEMU+GDB (
). eBPF:
. Windows
: find
→
→ IRP handlers. See
platforms.md.
Game Engine Reversing
Unreal: extract .pak with UnrealPakTool, reverse Blueprint bytecode with FModel. Unity Mono: decompile Assembly-CSharp.dll with dnSpy. Anti-cheat (EAC, BattlEye, VAC): identify system, bypass specific check. Lua games:
/
for bytecode. See
platforms.md.
Swift / Kotlin Binary Reversing
Swift:
symbols, protocol witness tables for dispatch,
sections. Kotlin/JVM: coroutines compile to state machines in
,
with Kotlin mode for best decompilation. Kotlin/Native: LLVM backend, looks like C++ in disassembly. See
languages-compiled.md.