Skip to content

[#14049] improvement(test): Run Hive SQL over JDBC and check Hive readiness without the CLI - #14052

Open
nevzheng wants to merge 7 commits into
apache:mainfrom
nevzheng:claude/13900-tc-10-hive-jdbc
Open

nevzheng wants to merge 7 commits into
apache:mainfrom
nevzheng:claude/13900-tc-10-hive-jdbc

Conversation

@nevzheng

@nevzheng nevzheng commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

  • HiveServer2Sql runs HiveQL over JDBC against the container's HiveServer2; CatalogHiveViewIT and CatalogHive3IT use it instead of docker exec hive -S -e. Adds hive-jdbc as a catalog-hive test dependency.
  • HiveContainer checks readiness without the Hive CLI: a live DataNode through the NameNode's JMX, the warehouse directory through WebHDFS, then the metastore and HiveServer2 ports. Kerberos Hive keeps the CLI path.
  • HiveContainer.getHadoopUser(): clients act as the user the container's processes run as (HADOOP_USER_NAME), as the CLI calls did.
  • Before Hadoop starts, the container lets that user impersonate users, which HiveServer2 needs for its sessions. The image change that makes this unnecessary is tracked in a follow-up issue.

This PR includes the commits of the PRs below it. Review only its own commit: nevzheng/gravitino@claude/13900-tc-09-durability...claude/13900-tc-10-hive-jdbc

Why are the changes needed?

Each hive -e call starts a JVM with its own embedded metastore: seconds per statement and per readiness poll. Locally (3 runs, medians): catalog-hive 380 s → 235 s (−38 %), CatalogHiveViewIT 122 s → 37 s (−70 %), hive-metastore-common 164 s → 147 s (−10 %); Hive readiness 8 s → 0.5 s per start.

Fix: #14049

Does this PR introduce any user-facing change?

No. Test code only.

How was this patch tested?

catalog-hive, hive-metastore-common, catalog-fileset, filesystem-hadoop3, authorization-ranger and authorization-chain integration tests.

Stack: #14103 → #14083 → #14084 → #14085 → #14005 → #14006 → #14052 → #14076 → #14007 → #14008 → #14009 → #14010 → #14011 → #14012 → #14013 → #14079 → #14077 → #14078 → #14086 → #14087 → #14088 → #14089 → #14090 → #14091 → #14094

@nevzheng nevzheng self-assigned this Oct 21, 2026
@nevzheng
nevzheng force-pushed the claude/13900-tc-10-hive-jdbc branch from 710c8ea to 9984fb4 Compare October 21, 2026 07:09
@nevzheng
nevzheng force-pushed the claude/13900-tc-10-hive-jdbc branch from 9984fb4 to 4412886 Compare October 21, 2026 17:20
@nevzheng
nevzheng force-pushed the claude/13900-tc-10-hive-jdbc branch 4 times, most recently from 0bb3413 to 17b5668 Compare October 21, 2026 18:40
Design doc for apache#13900 and a rumdl config for design-docs/testing.
…e ContainerSuite lifecycle

- Log every container start with pull, start and readiness times; add test
  log4j configs so the lines show.
- Reset all ContainerSuite state in close(); key Hive by version; make the
  PostgreSQL retry wait; pin StarRocks; drop a dead start method.
- Poll readiness with no initial delay; run cleanDisk() only on real starts.
- idp-basic starts its containers once per run; Ryuk back on, unprivileged.
- Drop the suite-wide closes in Hive2, Flink and Spark ITs.
@nevzheng
nevzheng force-pushed the claude/13900-tc-10-hive-jdbc branch from b6565cc to c35303d Compare October 21, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask] Run test SQL and readiness checks without CLIs in containers

1 participant