Segmentation fault in gfx_texture_cache_lookup() due to invalid rdp.loaded_texture address

[matheuspaulo93]

Description

I'm experiencing a reproducible segmentation fault in Render96ex while running the Windows PC build.

I'm currently working on the tester branch.

The game runs normally, but at a specific point during gameplay it crashes with:

Segmentation fault

Windows Event Viewer reports:

Exception code: 0xc0000005
Fault offset: 0x00000000000dd660

The faulting module is the game executable itself:

sm64.us.f3dex2e.exe

GDB investigation

I used GDB with:

catch signal SIGSEGV

This allowed me to catch the crash at the exact moment it occurs without using breakpoints that would interrupt normal gameplay.

The backtrace is:

#0  0x00000001400dd660 in gfx_texture_cache_lookup ()
#1  0x00000001400ed23c in import_texture ()
#2  0x00000001400ee21c in gfx_sp_tri1 ()
#3  0x00000001400ee47c in gfx_draw_rectangle ()
#4  0x00000001400ee6e9 in gfx_run_dl ()
#5  0x00000001400f1253 in gfx_run ()
#6  0x0000000140048cca in display_and_vsync ()
#7  0x0000000140049361 in game_loop_one_iteration ()
#8  0x00000001400d79a0 in produce_one_frame ()
#9  0x00000001400d7da1 in main_func ()
#10 0x00000001400d7e65 in SDL_main ()
#11 0x00000001401f05dc in main_getcmdline ()
#12 0x0000000140001157 in __tmainCRTStartup ()
#13 0x00007ffbacaaccb7 in KERNEL32!BaseThreadInitThunk ()
#14 0x00007ffbad7cad6c in ntdll!RtlUserThreadStart ()

So the execution path leading to the crash is:

gfx_run_dl
    -> gfx_sp_tri1
        -> import_texture
            -> gfx_texture_cache_lookup
                -> SIGSEGV

The crash occurs at:

0x1400dd660

The corresponding instruction is:

1400dd660: movzbl (%r8),%eax

This instruction attempts to read one byte from the memory address contained in R8.

At the exact moment of the crash, GDB reported:

rcx = 0x000000000
rdx = 0x000000014203ba40
r8  = 0x746361726168632f
r9  = 0x000000000

Checking R8:

(gdb) x/s $r8
0x746361726168632f: <error: Cannot access memory at address 0x746361726168632f>

Therefore, R8 is clearly an invalid pointer at the time of the crash.

Relevant source code

The function where the crash occurs is:

static bool gfx_texture_cache_lookup(int tile, struct TextureHashmapNode **n, const uint8_t *orig_addr, uint32_t fmt, uint32_t siz) {
    size_t hash = string_hash(orig_addr);
    ...
}

The call from import_texture() is:

gfx_texture_cache_lookup(tile, &rendering_state.textures[tile], rdp.loaded_texture[tile].addr, fmt, siz);

Based on the x64 calling convention and the disassembly, R8 contains the third argument, orig_addr.

The crash happens when gfx_texture_cache_lookup() attempts to access orig_addr, which appears to contain an invalid address.

Question

Has anyone encountered this issue before?

In particular, I'm interested in knowing under what circumstances:

rdp.loaded_texture[tile].addr

could contain an invalid pointer before being passed to:

gfx_texture_cache_lookup()

Could this be related to the texture cache, rdp.loaded_texture, or some specific condition in display list processing?

Any pointers on where I should investigate next would be greatly appreciated.

Thanks!