Skip to content

x64 Architecture

x64 Architecture and Modern Techniques

Modern malware leverages advanced x64 architecture features to evade detection, obscure execution logic, and execute malicious payloads. This section covers critical concepts like x64 calling conventions, RIP-relative addressing, and advanced control flow techniques commonly used in malware analysis.

x64 Calling Conventions

The x64 calling convention defines how functions pass arguments, return values, and manage the stack. Key aspects include:

  • Register Usage: The first four integer/pointer arguments are passed via RCX, RDX, R8, and R9. Floating-point arguments use XMM0–XMM3.
  • Return Value: The result of a function is stored in RAX (integer/pointer) or XMM0 (floating-point).
  • Stack Alignment: The caller ensures the stack is 16-byte aligned before calling a function.
  • Callee-Saved Registers: Registers like RBX, RBP, R12–R15 must be preserved by the callee.

Example:

; Function call in x64
mov rcx, 0x42        ; First argument
call my_function     ; my_function receives RCX as its first parameter

Analysis Tip: Use Ghidra’s "Function" view to identify calling conventions and track register usage during reverse engineering.

RIP-Relative Addressing

RIP-relative addressing allows position-independent code (PIC), which is critical for malware to avoid detection. Instead of absolute addresses, offsets are calculated relative to the RIP (Instruction Pointer) register.

  • Mechanism: Instructions like LEA or JMP use RIP + displacement to compute addresses.
  • Malware Use Cases:
  • Code Injection: Malware can inject code into memory without relying on fixed addresses.
  • Polymorphism: Encrypted payloads use RIP-relative jumps to decode themselves at runtime.

Example:

; RIP-relative jump (e.g., in a polymorphic payload)
lea rax, [rip + 0x10]  ; Load address of instruction 0x10 bytes ahead
jmp rax               ; Jump to the computed address

Analysis Tip: Look for RIP-based offsets in Ghidra’s disassembly to identify obfuscated control flow or encrypted payloads.

Advanced Control Flow Techniques

Malware often employs sophisticated control flow techniques to evade static analysis and sandbox detection:

  1. Indirect Jumps:
  2. Use registers or memory to determine the target address (e.g., jmp [rax]).

  3. Example:

    jmp [rax]  ; Target address stored in RAX

  4. Jump Tables:

  5. Arrays of addresses used to route execution dynamically.

  6. Example:

    jmp qword ptr [rax + 0x10]  ; Jump to a table entry

  7. Control Flow Flattening:

  8. Merge all branches into a single structure, using a dispatcher to select the next step.

  9. Example:

    mov rax, [rbp + 0x20]  ; Select next branch
jmp rax               ; Dispatch to the chosen path

  10. Return-Oriented Programming (ROP):

  11. Chain together "gadgets" (short instruction sequences) to execute arbitrary code without direct CALL/JMP.
  12. Example:
    ; ROP gadget: mov rax, 0x1234; ret
mov rax, 0x1234
ret

Analysis Tip: Ghidra’s "Graph View" can help visualize control flow flattening, while ROP chains often require manual reconstruction of gadget sequences.

Key takeaways

  • x64 calling conventions use registers for arguments and RAX for return values, requiring careful analysis of register usage.
  • RIP-relative addressing enables position-independent code, critical for malware evasion and polymorphism.
  • Advanced control flow techniques like indirect jumps, jump tables, and ROP are common in malware to obfuscate execution paths.
  • Tools like Ghidra are essential for analyzing these techniques, especially when combined with dynamic analysis and code reconstruction.