Fix slow basic block export and resume handling

[Raincarnator]

Summary

This PR fixes #305 and several related export-time and crash-resume issues found while investigating very large Diaphora exports.

The work was completed jointly by Raincarnator and Codex.

Changes

1. Remove the per-basic-block get_bb_id() query

insert_basic_blocks_to_database() used to resolve every basic block id by querying basic_blocks with its address before inserting the bb_instructions relation.

The historical behavior appears to have made sense in older schema versions, where a basic block address uniquely identified a row in basic_blocks. In Diaphora 2.1.0, for example, the table was effectively modeled like this:

basic_blocks (
  id integer primary key,
  num integer,
  address text unique
)

With that schema, get_bb_id(address) was a reasonable safeguard: if a basic block row for that address had already been inserted, the exporter could reuse it; if not, it inserted a new row. Because address was unique, the lookup had unambiguous semantics, and because the column was unique/indexed, the lookup was cheap.

That assumption no longer holds in the current schema. basic_blocks now also stores asm_type:

basic_blocks (
  id integer primary key,
  num integer,
  address text,
  asm_type text
)

The same address can now legitimately appear more than once, for example once as native basic block data and once as microcode basic block data. The address uniqueness assumption is therefore gone, and the old address lookup is no longer a reliable global identity lookup. More importantly, the current get_bb_id() query only filters by address:

select id from basic_blocks where address = ?

It does not filter by asm_type, and it does not restrict the row to the current function. In the current schema, such a query may return a row from another representation, or another context that happens to share the same address. So the old logic is not only expensive on large exports after the address uniqueness/index assumption disappeared; it is also less precise than using the row that was just inserted.

For the native basic-block export path fixed here, re-querying by address is not necessary. The exporter is already executing this sequence locally:

1. insert one native row into basic_blocks;
2. read the inserted row id;
3. insert bb_instructions, bb_relations, and function_bblocks rows that should point to that newly inserted native basic block.

In this sequence, cursor.lastrowid is exactly the id that the exporter needs. It identifies the row inserted by the immediately preceding insert into basic_blocks statement on the same cursor. Re-querying by address is therefore not adding correctness for this relation; it tries to rediscover the same local row through a slower and now ambiguous key.

This does mean that basic_blocks may contain multiple rows with the same address. In the current schema that is already allowed and expected, because asm_type makes address alone insufficient as a row identity. The exported graph queries already go through function_bblocks and basic_block_id, and also filter by asm_type, so each function should use the specific basic block rows linked to it rather than a globally shared row found by address alone.

Experiment result:

• on the provided small sample, this change reduced the native basic-block insertion hot path from repeatedly querying basic_blocks by address to directly using the inserted row id;
• this specifically removes the per-basic-block SQLite lookup identified as the main source of the Diaphora 3.2.1 is much slower than diaphora 2.1.0 when exporting large binaries #305 export-time regression;
• adding an index to the address lookup alone was not enough to resolve the slowdown reported in Diaphora 3.2.1 is much slower than diaphora 2.1.0 when exporting large binaries #305, because the exporter still performed the extra lookup for every basic block.

Expected effect:

• removes one hot-loop SQLite query per native basic block;
• avoids depending on an address lookup that is no longer uniquely identifying;
• preserves the intended relation between the newly inserted basic block and its instructions;
• improves export speed for binaries with many basic blocks.

2. Fix typed DIAPHORA_* option parsing

get_value_for() previously checked isinstance(value, type(default)), but environment variables are strings. This meant bool/int-like settings were not reliably converted.

This was especially problematic for boolean values. For example, an environment value such as "0" was still a non-empty string, so code paths that treated it as a boolean could behave as enabled instead of disabled.

This PR:

• parses boolean defaults explicitly (1/true/yes/on become true; values such as 0/false/no/off do not);
• converts non-string defaults such as integers using the default’s type.

Expected effect:

• automated export options now match the requested configuration;
• DIAPHORA_EXPORT_MICROCODE=0 can correctly disable microcode.

3. Fix automatic export microcode option name

Automatic export read "self.export_microcode" instead of "export_microcode", so DIAPHORA_EXPORT_MICROCODE was ignored.

Expected effect:

• automatic exports now honor the microcode option;
• disabling microcode reduces unexpected export work and DB size.

Experiment result:

• on the provided small sample, fixing this option reduced the decompiler + summary-only + microcode-off output from the microcode-on-sized DB to about 76.5MB instead of about 138MB.

4. Avoid duplicate microcode generation

Microcode was generated once inside decompile_and_get() and then again in extract_all_features().

This PR removes the earlier generation and keeps the later one, so microcode is produced once and then immediately consumed by extract_microcode().

Expected effect:

• avoids duplicated Hex-Rays microcode generation;
• keeps the exported microcode data and schema unchanged.

Experiment result:

• on the provided small sample with microcode enabled, export time improved from about 213s to about 182s;
• output DB size stayed in the expected range because the same microcode data is still exported, just not generated twice.

5. Make crash resume safer

do_export() now returns success explicitly, and export() only finalizes the DB after a successful export.

Expected effect:

• failed exports keep the crash marker;
• incomplete exports are not finalized as successful ones.

6. Preserve crashed export DBs in automatic mode

Automatic export now keeps an existing output DB when the matching output.sqlite-crash marker exists.

Expected effect:

• automatic exports can resume from the existing partial DB instead of deleting it first.

7. Improve progress dialog refresh cadence

The progress dialog still updates by percentage, but now also refreshes about once every 60 seconds.

Expected effect:

• long exports show visible progress more regularly;
• export data and DB output are unchanged.

Verification

• python -m py_compile diaphora.py diaphora_ida.py
• git diff --check -- diaphora.py diaphora_ida.py
• Small-sample automatic export experiments confirmed:
  • microcode-off is now honored;
  • duplicate microcode generation is removed;
  • microcode-enabled export time improved from about 213s to about 182s in the measured run.

[joxeankoret]

AI generated explanations are terrible to read and actually magnitudes worse to understand than human made explanations. Please don't again. Thanks.

[Raincarnator]

I'm really sorry. English isn't my first language, and in such long and specialized explanations, using a translator directly would lose the context, so I had to use Codex to generate the explanation and then translate it back into Chinese to check the explanation.

Sorry for adding extra work for you, feel free to ask me anything if needed!

[joxeankoret]

I understand you, don't worry. I'm not a native English speaker either and I understand how hard can it be to explain deep technical problems in something that is not your main language.