Work around clang hang compiling interp.c with -O1 and higher

[guillerodriguez]

Clang's LLVM backend hangs indefinitely when compiling src/interp/engine/interp.c with optimization level -O1 or higher. The hang is caused by LLVM's TailDuplication pass encountering exponential time complexity with the computed goto dispatch pattern used in the interpreter. The problem can be reproduced on Linux and macOS, and the actual cause was confirmed via profiling.

Work around by adding -mllvm -tail-dup-limit=0 to the interp CFLAGS when compiling with clang. This disables the problematic pass (for interp.c only) while keeping all other LLVM optimizations intact.

[guillerodriguez]

Quick instructions to reproduce the problem using Docker:

docker run --rm -it -e DEBIAN_FRONTEND=noninteractive ubuntu:24.04 bash 

Inside container:

apt-get update
apt-get install -y clang-14 make automake autoconf libtool zlib1g-dev zip \
  gettext texinfo openjdk-8-jdk-headless git

cd /tmp
git clone --depth 1 -q https://github.com/ingelabs/classpath.git
cd classpath && ./autogen.sh
./configure JAVAC=javac --disable-gconf-peer \
  --enable-default-preferences-peer=file --disable-gtk-peer \
  --disable-qt-peer --disable-alsa --disable-dssi --disable-plugin \
  --disable-examples --disable-tools --disable-gjdoc
make -j4 && make install

cd /tmp && git clone -q https://github.com/ingelabs/jamvm.git
cd jamvm
NOCONFIGURE=1 ./autogen.sh
./configure CC=clang JAVAC=javac
make  # Hangs without the fix

[guillerodriguez]

Profiling results, using macOS sample command on the hanging clang process:

Call graph:
    Thread_78965266   DispatchQueue_1: com.apple.main-thread  (serial)
      1000 (100.0%) clang  (3556 + 32)
        990 (99.0%) clang::CompilerInvocation::ExecuteCompiler(...)
          989 (98.9%) llvm::TailDuplicatePass::processBB(...)
            988 (98.8%) llvm::TailDuplicatePass::tailDuplicate(...)

TailDuplicatePass::processBB consumes 99% of CPU time and never completes. Memory usage keeps growing.