SIGABRT/SIGSEGV in jni_GetArrayLength during pyembed_startup when creating SubInterpreter instances since 4.3.0

[rackd]

Describe the bug
The goal is to create several SubInterpreter instances in succession; however, I observed that the JVM crashes on the first instantiation in JEP 4.3.0 and 4.3.1. The crash occurs before any Python code executes, during JEP’s startup phase (pyembed_startup), and it prevents any additional interpreters from being created.

Thread inspection confirms that only one thread is involved in JEP initialization at the time of the failure.

Downgrading to JEP 4.2.2 avoids the crash entirely. The application runs reliably under 4.2.2, and the issue only appears in 4.3.x. The second SubInterpreter is never created because the first one fails during startup.

Hopefully this is enough (or at least something) to work with devs. Quite a bit of business logic was redacted. I can provide more information privately, if needed.

To Reproduce

public class Main {
    public static void main(String[] args) throws Exception {
        System.load("/path/to/jep_lib/jep/libjep.so");

        for (int i = 0; i < 100; i++) {
            String result = PythonUtil.foo();
            System.out.println("Iteration " + i + ": " + result);
        }
    }
}

This issue happens whether JepConfig is instantiated inside or outside the foo() method. Both are included in the following code to convey this:

public class PythonUtil {

    private static final JepConfig STATIC_CONFIG =
        new JepConfig().addIncludePaths("/path/to/jep_lib");

    public static String foo() {
        JepConfig config = new JepConfig()
            .addIncludePaths("/path/to/jep_lib");

        try (Interpreter interp = new SubInterpreter(config)) {
            interp.exec("import uuid");
            interp.exec("test = str(uuid.uuid4())");
            return (String) interp.getValue("test");
        }
    }
}

Expected behavior
The interpreter to execute without crashing.

Environment

• OS: Linux 6.1
• Python 3.11
• Java 23.0.2
• Jep 4.3.1
• Python packages used: None

Additional context
Backtrace:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007f6c0533df4f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007f6c052eefb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f6c052d9472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007f6c0409528d in os::abort(bool, void*, void const*) [clone .cold] () from /usr/lib/jvm/jdk-23.0.2-oracle-x64/lib/server/libjvm.so
#5  0x00007f6c04d3685a in VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long) () from /usr/lib/jvm/jdk-23.0.2-oracle-x64/lib/server/libjvm.so
#6  0x00007f6c04d36f8b in VMError::report_and_die(Thread*, unsigned int, unsigned char*, void*, void*, char const*, ...) () from /usr/lib/jvm/jdk-23.0.2-oracle-x64/lib/server/libjvm.so
#7  0x00007f6c04d36fae in VMError::report_and_die(Thread*, unsigned int, unsigned char*, void*, void*) () from /usr/lib/jvm/jdk-23.0.2-oracle-x64/lib/server/libjvm.so
#8  0x00007f6c04ba1b4b in JVM_handle_linux_signal () from /usr/lib/jvm/jdk-23.0.2-oracle-x64/lib/server/libjvm.so
#9  <signal handler called>
#10 0x00007f6c04771887 in jni_GetArrayLength () from /usr/lib/jvm/jdk-23.0.2-oracle-x64/lib/server/libjvm.so
#11 0x00007f6bdc0a315b in pyembed_startup (env=0x7f6bfd75d290, isolated=<optimized out>, argv=0x3, hashSeed=4, useHashSeed=-339206583, home=0x0, optimizationLevel=-339297509, parseArgv=-9, programName=0x7f6ad1aad260, siteImport=4,
    useEnvironment=-777329936, userSiteDirectory=0, verbose=401575584, writeByteCode=-777334176) at src/main/c/Jep/pyembed.c:456
#12 0x00007f6bebc6be7c in ?? ()
#13 0x00000000ebc6bb1b in ?? ()
#14 0x00007f6bfffffff7 in ?? ()
#15 0x00007f6ad1aad260 in ?? ()
#16 0x00007f6a00000004 in ?? ()
#17 0x00000000d1aae2f0 in ?? ()
#18 0x00007f6a00000000 in ?? ()
#19 0x0000000017ef8ea0 in ?? ()
#20 0x00000006d1aad260 in ?? ()
#21 0x00007f6ad1aad260 in ?? ()
#22 0x0000000000000000 in ?? ()

[bsteffensmeier]

Is it possible that you are using a Jep 4.3.1 native library with Jep 4.2.2 jar?

[rackd]

I do not believe the crash is caused by using the Jep 4.3.1 native library with the Jep 4.2.2 jar.

The /path/to/jep_lib directory is deleted during every deployment of the application using rm -rf /path/to/jep_lib. A fresh version of Jep is installed via commands below:

JEP_DIR=/path/to/jep_lib
"$PIP_PATH" install --target="${JEP_DIR}" jep

[bsteffensmeier]

I have been unable to reproduce the problem using your example code. I haven't quite matched your environment but I am not confident that would help.

#11 0x00007f6bdc0a315b in pyembed_startup (env=0x7f6bfd75d290, isolated=, argv=0x3, hashSeed=4, useHashSeed=-339206583, home=0x0, optimizationLevel=-339297509, parseArgv=-9, programName=0x7f6ad1aad260, siteImport=4,
useEnvironment=-777329936, userSiteDirectory=0, verbose=401575584, writeByteCode=-777334176) at src/main/c/Jep/pyembed.c:456

This is the part that confuses me. Most of those arguments are only ever set to -1, 0, or 1 so the fact they are set differently should be impossible. A few of them should be pointers to java objects such as argv, home, and programName. It is interesting that home and programName are plausiblely pointers but argv is set to 0x3 which doesn't seem valid and that is the array it is trying to get the length of when it crashes.

In Jep 4.2.2 the java code that calls that startup function is only passing 2 arguments and there is no mechanism to guarantee the c code is expecting the right number of arguments so if you were running a 4.2.2 java class it would put those two args in memory and the c code would read the remaining args with whatever happened to be in memory which would probably be meaningless values, which is what you are seeing.

When I deliberatly run with a 4.2.2 jar and a 4.3.1 native library on my system I am also seeing a crash in pyembed_startup, although mine crashes in jni_GetStringUTFChars rather than jni_GetArrayLength but when reading unset args I don't find that inconsistent behavior too surprising.

[ndjensen]

I do not believe the crash is caused by using the Jep 4.3.1 native library with the Jep 4.2.2 jar.

The /path/to/jep_lib directory is deleted during every deployment of the application using rm -rf /path/to/jep_lib. A fresh version of Jep is installed via commands below:

JEP_DIR=/path/to/jep_lib "$PIP_PATH" install --target="${JEP_DIR}" jep

Just a thought but for your application are you using a jep jar that is in a different location so it's included on the classpath? The pip build does install a jar to your python's site-packages/jep directory, but I suspect most Java applications using jep have a copy of the jar in a lib directory or something similar and add all the jars in that directory on the classpath. Or maybe your application is using maven to pull the jar and place it in an appropriate location to end up on the classpath. If you're not adding ${JEP_DIR}/jep-4.3.1.jar to your classpath, maybe you're using a jar located somewhere else?