Segmentation fault(core dumped) on tissue_extraction_to_bgef

[OswaldZhang]

Hi, this is a transfer issue from SAW to Stereopy. STOmics/SAW#81, as suggested by @Clouate STOmics/SAW#81 (comment)

As you can see here, when performing segmentation on my own converted GEF file, there is a trouble for me:

STOmics/SAW#81 (comment)

My question is, can I convert my combined gem file to the proper gef file as input to tissue_extraction_to_bgef? The reason behind this is we have 2 different separate stereo-seq SN.tissue.gem files measured in one tissue section (one is more dense and the other is sparse). We want to do tissue segmentation in an unbiased manner( this is to say, the transcripts located in the more dense one may not contain the transcripts located in the sparse one).

[tanliwei-coder]

Sorry for replying so late, could you tell me how you combined those gem files to one and converted it to a gef file?
Also, It would be best to provide the gem files to me if you can so that I can try to reproduce the error to find out the reson.

[OswaldZhang]

Thanks for your kind reply. The combination of my gem file is done by simply adding the sparse one to the end of the dense one, and converting it to gef either by stereopy STOmics/SAW#81 (comment) or geftools BGIResearch/geftools#1 (comment).

When running SAW tissueCut or done this by stereopy, it always returns

/usr/local/bin/tissueCut: line 1: 139646 Segmentation fault (core dumped) python3 /opt/saw_v6.0.0_software/pipeline/tissueCut/Tissuecut.pyc $*

As suggested by @clouat, it seems there is sth wrong with the conversion from GEM to GEF as input for tissue cut:

STOmics/SAW#81 (comment)

However, I do find there exists a mask.tif file on the dest. dir, so currently I use this file to manually filter my combined raw gem file.

Here is a file I think is minimally required to reproduce the bug

geneID	x	y	MIDCount	ExonCount

0 Gm1992 15244 8385 1 0
1 Gm1992 13665 24443 1 0
2 Gm1992 20889 18202 1 0
3 Gm37381 18336 12669 1 0
4 Gm37381 21477 16541 1 0
5 Rp1 21177 13510 2 0
6 Rp1 17217 14497 1 0
7 Rp1 10429 16570 1 0
8 Rp1 21462 16391 1 0
9 csRNA 22192 12321 1 1
10 cdRNA 22213 12133 1 1

Is it because of the index?

[wanruiwen-genomics-cn]

Dear @OswaldZhang,

Thank you for reporting this issue and for the cross-reference to the SAW repository!

Short answer: The segmentation fault you are experiencing most likely originates in the native C/C++ GEF processing library (gefpy), not in Stereopy's Python code. To help investigate further, we would need more details about your environment, the exact commands used, and the GEF file format.

---

Assessment

• Type: needs-info
• Severity: medium
• Affected module: stereo/tools/tissue_extraction.py, gefpy.bgef_creater_cy.BgefCreater

What Is Happening

The tissue_extraction_to_bgef function in Stereopy performs two major steps:

1. Tissue segmentation — using either RNATissueCut (for RNA staining type) or SingleStrandDNATissueCut (for ssDNA/dapi/HE/mIF), which generates a mask TIFF file.
2. GEF extraction — using gefpy.bgef_creater_cy.BgefCreater.create_bgef(), which is a native C/C++ extension that reads the source GEF, applies the mask, and writes the destination bGEF.

A “Segmentation fault (core dumped)” error typically occurs at the native code level (step 2), and can be caused by:

• An invalid or corrupted GEF file (e.g., one manually assembled from multiple GEM files)
• A mismatch between the GEF file’s spatial dimensions and the mask dimensions
• An incompatible version of the gefpy library

Since you mentioned combining two separate SN.tissue.gem files into one GEF, the resulting GEF may not have the expected internal structure that gefpy requires, leading to the crash.

Is This Expected?

Not exactly. While a segfault should ideally be caught and reported as a user-friendly error, the root cause appears to be in the native gefpy library rather than in Stereopy’s Python layer.

Recommended Workflow

Step 1: Please provide the following information so we can investigate further:

• Stereopy version (import stereo; print(stereo.__version__))
• gefpy version (import gefpy; print(gefpy.__version__))
• Python version
• OS and architecture
• The exact code or commands you used to create the combined GEF file
• The exact code you used to call tissue_extraction_to_bgef

Step 2: Verify your GEF file is valid by trying to read it with Stereopy first:

import stereo as st
data = st.io.read_gef("/path/to/your/combined.gef")
print(data)

If this step fails, the GEF file itself may be malformed.

Step 3: As an alternative approach, consider running tissue segmentation on each GEM file separately before merging results, rather than combining GEM files into a single GEF first.

Notes

• The tissue_extraction_to_bgef function relies on gefpy, which is a separate native library. Segmentation faults in native code cannot be caught by Python’s exception handling.
• If you are combining GEM files, the resulting GEF must preserve the correct spatial coordinate system and bin structure expected by gefpy.
• For the RNA staining pathway, the rna_tissue_cut_bin_size parameter defaults to 1 (bin1). Make sure this matches your data.

Please let us know if you have further questions!

Best regards,
Stereopy Maintainer