Skip to content

Commit f6f7cea

Browse files
adi-kumoadi-kumo
committed
T1.32e tests: P1B-40 option (b), null-key rows left out with one warning
The owner's decision of 2026-10-13 (relayed): P1B-40 becomes option (b). Where sampling reaches a table whose key mixes nulls with real text or decimal values (the reference refuses INVALID_REQUEST or fails INTERNAL_ERROR), the core leaves the null-key rows out of the walk and succeeds, with one warning per table saying how many rows it left out; the singleton and all-null cases stay the reference's request. Sampled vectors (contract/tests/test_t132e_sampled_null_keys.py): mixed-null-text and mixed-null-decimal must succeed on python-core with the keys the reference gave, a request holding no row of the null key (the payments reached for account 3 are rows 1 and 2, never row 0) and equal to its twin's, the request the reference sends for the same tables with the null row not reached (null-text-not-sampled, and a new recorded twin null-decimal-not-sampled); and the twin's warnings plus exactly one, whose text names payments and the count (1). The recorder now keeps each warning's text beside the answer, never compared, only to find which warning names a table (the recorded answers are unchanged). Key vectors (test_t132e_key_inference.py, t132e_key_inference.rs): nulls-1-decimal.200 (INTERNAL_ERROR in the reference) and nulls-1-text.200 (INVALID_REQUEST: the same mixed text key reached by sampling, which the decision's rule covers though the relayed list did not name it) must materialize on both hosts, describe the tables as the reference did, and give the twin's (nulls-1.200) warnings and exactly one more: on rust-host a code the twin's run never gives, its fields naming payments. The per-vector tests pass the divergence only under the approved entry's binding refs (refusal INVALID_REQUEST -> success for mixed-null-text and nulls-1-text.200; refusal INTERNAL_ERROR -> success for mixed-null-decimal and nulls-1-decimal.200), which the implementer adds to P1B-40. The six new checks fail at a1e7a49 (the core refuses) and are added to pending["exits"]. Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
1 parent 1934a39 commit f6f7cea

6 files changed

Lines changed: 613 additions & 2 deletions

File tree

‎contract/sampled-null-key-vectors.json‎

Lines changed: 317 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,11 @@
285285
"SDMWarning"
286286
]
287287
},
288+
"reference_messages": [
289+
"payments.payment_id identifies a row, but 1 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
290+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`.",
291+
"every context row has the same accounts.amount, so nothing in the context distinguishes one answer from another. The endpoint will answer, but the answer cannot say much. Gather a wider context with num_context_samples, or ask about entities whose answers differ."
292+
],
288293
"register": {
289294
"required": false,
290295
"why": []
@@ -313,6 +318,10 @@
313318
"UserWarning"
314319
]
315320
},
321+
"reference_messages": [
322+
"payments.payment_id identifies a row, but 1 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
323+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`."
324+
],
316325
"register": {
317326
"required": false,
318327
"why": []
@@ -597,6 +606,11 @@
597606
"SDMWarning"
598607
]
599608
},
609+
"reference_messages": [
610+
"payments.payment_id identifies a row, but 1 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
611+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`.",
612+
"every context row has the same accounts.amount, so nothing in the context distinguishes one answer from another. The endpoint will answer, but the answer cannot say much. Gather a wider context with num_context_samples, or ask about entities whose answers differ."
613+
],
600614
"register": {
601615
"required": false,
602616
"why": []
@@ -867,6 +881,300 @@
867881
"SDMWarning"
868882
]
869883
},
884+
"reference_messages": [
885+
"payments.payment_id identifies a row, but 1 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
886+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`.",
887+
"every context row has the same accounts.amount, so nothing in the context distinguishes one answer from another. The endpoint will answer, but the answer cannot say much. Gather a wider context with num_context_samples, or ask about entities whose answers differ."
888+
],
889+
"register": {
890+
"required": false,
891+
"why": []
892+
}
893+
},
894+
{
895+
"id": "null-decimal-not-sampled",
896+
"setup": [
897+
"CREATE TABLE accounts AS SELECT i AS account_id, i::DOUBLE AS amount FROM range(200) t(i)",
898+
"CREATE TABLE payments AS SELECT CASE WHEN i=0 THEN NULL ELSE i::DECIMAL(18, 0) END AS payment_id, CASE WHEN i IN (1, 2) THEN 3 ELSE 150 END AS account_id, i::DOUBLE AS amount FROM range(200) t(i)"
899+
],
900+
"reference": {
901+
"outcome": {
902+
"result": "success",
903+
"stage": "predict",
904+
"error": null,
905+
"code": null
906+
},
907+
"keys": {
908+
"accounts": "account_id",
909+
"payments": "payment_id"
910+
},
911+
"request": {
912+
"model": "kumo-relational",
913+
"task": {
914+
"kind": "regression",
915+
"target": {
916+
"column_name": "TARGET",
917+
"dtype": "float32"
918+
},
919+
"entity_table_names": [
920+
"accounts"
921+
]
922+
},
923+
"schema": {
924+
"instance_table": {
925+
"columns": {
926+
"instance_id": {
927+
"dtype": "int64",
928+
"stype": "ID",
929+
"nullable": false
930+
},
931+
"__kumo_entity_ref_0": {
932+
"dtype": "int64",
933+
"stype": "ID"
934+
},
935+
"TARGET": {
936+
"dtype": "float32",
937+
"stype": "numerical"
938+
}
939+
},
940+
"primary_key": "instance_id"
941+
},
942+
"related_tables": {
943+
"accounts": {
944+
"columns": {
945+
"instance_id": {
946+
"dtype": "int64",
947+
"stype": "ID",
948+
"nullable": false
949+
},
950+
"account_id": {
951+
"dtype": "int64",
952+
"nullable": false
953+
}
954+
},
955+
"primary_key": [
956+
"instance_id",
957+
"account_id"
958+
]
959+
},
960+
"payments": {
961+
"columns": {
962+
"instance_id": {
963+
"dtype": "int64",
964+
"stype": "ID",
965+
"nullable": false
966+
},
967+
"__node_id": {
968+
"dtype": "int64",
969+
"stype": "ID",
970+
"nullable": false
971+
},
972+
"amount": {
973+
"dtype": "float64",
974+
"stype": "numerical"
975+
},
976+
"account_id": {
977+
"dtype": "int64",
978+
"stype": "ID"
979+
}
980+
},
981+
"primary_key": [
982+
"instance_id",
983+
"__node_id"
984+
]
985+
}
986+
},
987+
"relationships": [
988+
{
989+
"source_columns": [
990+
"__kumo_entity_ref_0"
991+
],
992+
"target_table": "accounts",
993+
"target_columns": [
994+
"account_id"
995+
]
996+
},
997+
{
998+
"source_table": "payments",
999+
"source_columns": [
1000+
"account_id"
1001+
],
1002+
"target_table": "accounts",
1003+
"target_columns": [
1004+
"account_id"
1005+
]
1006+
}
1007+
]
1008+
},
1009+
"context": {
1010+
"instance_table": {
1011+
"format": "arrays",
1012+
"columns": [
1013+
"instance_id",
1014+
"__kumo_entity_ref_0",
1015+
"TARGET"
1016+
],
1017+
"rows": [
1018+
[
1019+
{
1020+
"number": "0"
1021+
},
1022+
{
1023+
"number": "40"
1024+
},
1025+
{
1026+
"number": "40.0"
1027+
}
1028+
]
1029+
]
1030+
},
1031+
"related_tables": {
1032+
"accounts": {
1033+
"format": "arrays",
1034+
"columns": [
1035+
"instance_id",
1036+
"account_id"
1037+
],
1038+
"rows": [
1039+
[
1040+
{
1041+
"number": "0"
1042+
},
1043+
{
1044+
"number": "40"
1045+
}
1046+
]
1047+
]
1048+
},
1049+
"payments": {
1050+
"format": "arrays",
1051+
"columns": [
1052+
"instance_id",
1053+
"__node_id",
1054+
"amount",
1055+
"account_id"
1056+
],
1057+
"rows": []
1058+
}
1059+
}
1060+
},
1061+
"predict": {
1062+
"instance_table": {
1063+
"format": "arrays",
1064+
"columns": [
1065+
"instance_id",
1066+
"__kumo_entity_ref_0"
1067+
],
1068+
"rows": [
1069+
[
1070+
{
1071+
"number": "1"
1072+
},
1073+
{
1074+
"number": "3"
1075+
}
1076+
]
1077+
]
1078+
},
1079+
"related_tables": {
1080+
"accounts": {
1081+
"format": "arrays",
1082+
"columns": [
1083+
"instance_id",
1084+
"account_id"
1085+
],
1086+
"rows": [
1087+
[
1088+
{
1089+
"number": "1"
1090+
},
1091+
{
1092+
"number": "3"
1093+
}
1094+
]
1095+
]
1096+
},
1097+
"payments": {
1098+
"format": "arrays",
1099+
"columns": [
1100+
"instance_id",
1101+
"__node_id",
1102+
"amount",
1103+
"account_id"
1104+
],
1105+
"rows": [
1106+
[
1107+
{
1108+
"number": "1"
1109+
},
1110+
{
1111+
"number": "0"
1112+
},
1113+
{
1114+
"number": "1.0"
1115+
},
1116+
{
1117+
"number": "3"
1118+
}
1119+
],
1120+
[
1121+
{
1122+
"number": "1"
1123+
},
1124+
{
1125+
"number": "1"
1126+
},
1127+
{
1128+
"number": "2.0"
1129+
},
1130+
{
1131+
"number": "3"
1132+
}
1133+
]
1134+
]
1135+
}
1136+
}
1137+
},
1138+
"output": {
1139+
"fields": [
1140+
"prediction"
1141+
],
1142+
"prediction_statistic": "median"
1143+
},
1144+
"inference": {
1145+
"run_mode": "fast",
1146+
"use_prediction_time": false,
1147+
"inference_config": {
1148+
"kind": "regression",
1149+
"num_estimators": {
1150+
"number": "1"
1151+
},
1152+
"column_shuffle": false,
1153+
"category_shuffle": false,
1154+
"hop_shuffle": false
1155+
}
1156+
},
1157+
"metadata": {
1158+
"source_query": "PREDICT accounts.amount FOR EACH accounts.account_id",
1159+
"num_context_examples": {
1160+
"number": "1"
1161+
},
1162+
"num_prediction_examples": {
1163+
"number": "1"
1164+
}
1165+
}
1166+
},
1167+
"warnings": [
1168+
"SDMWarning",
1169+
"UserWarning",
1170+
"SDMWarning"
1171+
]
1172+
},
1173+
"reference_messages": [
1174+
"payments.payment_id identifies a row, but 1 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
1175+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`.",
1176+
"every context row has the same accounts.amount, so nothing in the context distinguishes one answer from another. The endpoint will answer, but the answer cannot say much. Gather a wider context with num_context_samples, or ask about entities whose answers differ."
1177+
],
8701178
"register": {
8711179
"required": false,
8721180
"why": []
@@ -895,6 +1203,10 @@
8951203
"UserWarning"
8961204
]
8971205
},
1206+
"reference_messages": [
1207+
"payments.payment_id identifies a row, but 1 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
1208+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`."
1209+
],
8981210
"register": {
8991211
"required": true,
9001212
"why": [
@@ -1167,6 +1479,11 @@
11671479
"SDMWarning"
11681480
]
11691481
},
1482+
"reference_messages": [
1483+
"payments.payment_id identifies a row, but 2 row(s) have no value for it. Those rows cannot be scored and the model will not see them; fill or drop them, or name a different primary key.",
1484+
"Missing 3 indices in 2 tables for optimal database querying. For improving runtime, we strongly suggest to create indices for primary and foreign keys, e.g., automatically by instantiating Kumo Relational via `KumoRelational(graph, optimize=True)`.",
1485+
"every context row has the same accounts.amount, so nothing in the context distinguishes one answer from another. The endpoint will answer, but the answer cannot say much. Gather a wider context with num_context_samples, or ask about entities whose answers differ."
1486+
],
11701487
"register": {
11711488
"required": false,
11721489
"why": []

‎contract/tests/t132e-pending.json‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"about": "T1.32e part 1's acceptance tests that failed when the independent test author wrote them, before the implementation, at a4ed935f (contract/tests/test_t132e_time_inference.py, test_t132e_key_inference.py, test_t132e_benchmark_seeds.py, test_t132e_sampled_null_keys.py, sdm-testkit's tests/t132e_time_inference.rs and t132e_key_inference.rs). 'python-core' and 'rust-host' are time-inference vector ids, 'key-python-core' and 'key-rust-host' key vector ids, 'sampled-python-core' sampled null key vector ids (measured at 18bd89d6); 'exits' holds the exits' tests ('py:' the Python test ids, 'rs:' the vectors of rust_host_honours_a_named_time_column_on_a_large_table, 'rs-key:' those of rust_host_refuses_no_table_for_a_null_in_its_key) and the benchmark's seeds. Each listed test must still fail (strict); the implementer takes an id out of 'pending' once it passes, and changes nothing else: 'pending' stays within 'authored', which is the test author's.",
2+
"about": "T1.32e part 1's acceptance tests that failed when the independent test author wrote them, before the implementation, at a4ed935f (contract/tests/test_t132e_time_inference.py, test_t132e_key_inference.py, test_t132e_benchmark_seeds.py, test_t132e_sampled_null_keys.py, sdm-testkit's tests/t132e_time_inference.rs and t132e_key_inference.rs). 'python-core' and 'rust-host' are time-inference vector ids, 'key-python-core' and 'key-rust-host' key vector ids, 'sampled-python-core' sampled null key vector ids (measured at 18bd89d6); 'exits' holds the exits' tests ('py:' the Python test ids, 'rs:' the vectors of rust_host_honours_a_named_time_column_on_a_large_table, 'rs-key:' those of rust_host_refuses_no_table_for_a_null_in_its_key, 'rs-key-b:' those of rust_host_leaves_null_key_rows_out_with_one_warning) and the benchmark's seeds. Each listed test must still fail (strict); the implementer takes an id out of 'pending' once it passes, and changes nothing else: 'pending' stays within 'authored', which is the test author's.",
33
"pending": {
44
"python-core": [
55
"fallback-doubted-beside-plain.200000",
@@ -17,6 +17,12 @@
1717
],
1818
"exits": [
1919
"py:test_a_named_time_column_is_honoured_on_a_large_table[fallback-unconfirmed-named.200000]",
20+
"py:test_option_b_leaves_the_null_key_rows_out_with_one_warning[mixed-null-decimal]",
21+
"py:test_option_b_leaves_the_null_key_rows_out_with_one_warning[mixed-null-text]",
22+
"py:test_option_b_materializes_with_one_more_warning[nulls-1-decimal.200]",
23+
"py:test_option_b_materializes_with_one_more_warning[nulls-1-text.200]",
24+
"rs-key-b:nulls-1-decimal.200",
25+
"rs-key-b:nulls-1-text.200",
2026
"rs:fallback-unconfirmed-named.200000"
2127
],
2228
"key-python-core": [
@@ -282,6 +288,10 @@
282288
"py:test_an_old_route_refusal_is_inconclusive_and_operational",
283289
"py:test_no_host_appears_in_the_summary",
284290
"py:test_no_request_passes_the_plan",
291+
"py:test_option_b_leaves_the_null_key_rows_out_with_one_warning[mixed-null-decimal]",
292+
"py:test_option_b_leaves_the_null_key_rows_out_with_one_warning[mixed-null-text]",
293+
"py:test_option_b_materializes_with_one_more_warning[nulls-1-decimal.200]",
294+
"py:test_option_b_materializes_with_one_more_warning[nulls-1-text.200]",
285295
"py:test_run_keeps_the_floor_when_the_core_refuses",
286296
"py:test_the_budget_is_500_predictions[-1]",
287297
"py:test_the_budget_is_500_predictions[0]",
@@ -290,6 +300,8 @@
290300
"py:test_the_datasets_are_the_real_ones",
291301
"py:test_the_default_plan_fits_the_budget",
292302
"py:test_the_floor_reads_the_old_route_averaged_over_the_seeds",
303+
"rs-key-b:nulls-1-decimal.200",
304+
"rs-key-b:nulls-1-text.200",
293305
"rs-key:all-null.200",
294306
"rs-key:all-null.5000",
295307
"rs-key:entity-all-null.200",

0 commit comments

Comments
 (0)