issue #7547: a Hop marketplace#7577
Open
mattcasters wants to merge 28 commits into
Open
Conversation
…lugins Split heavy plugins out of the default client so ASF Artifactory stays under the ~850MB package limit. Add a bundled marketplace plugin with CLI install/list/uninstall, keep lib/beam in core while making the Beam engine plugin optional, and add local Artifactory tooling plus a CI assembly size check.
Add Tools → Marketplace dialog for install/uninstall of optional plugins, a Wave 1 catalog, and install-wave1-plugins.sh wired into Jenkins and run-tests-docker so integration tests still get spark, beam, cassandra, and other marketplace-only plugins.
Add declarative environment files for marketplace plugins and JDBC dependencies: apply (--prune), validate, sample hop-env.yaml, and an optional HopRunStart enforcement when enforceOnRun is set.
…ce stack Generate and mount keys/master.key (openssl rand -hex 32, mode 600, owner 1030:1030) so JFrog Artifactory OSS no longer hangs at startup.
Single-node OSS still requires a join key; generate and mount both master.key and join.key (64-char hex, 600, 1030:1030) and point JF_SHARED_SECURITY_JOIN_KEY_FILE at the mount.
Modern Artifactory 7.x requires Postgres (no embedded DB). Add a postgres service, pass master/join keys via .env (not RO file mounts), fix master key size to AES-128 (hex-16), and add start.sh --reset.
…401) Send HTTP Basic credentials when installing plugins so private Maven repos work. Prefer HOP_MARKETPLACE_USERNAME/PASSWORD (or ARTIFACTORY_*) over storing secrets in hop-config.json.
…ce installs Add configure-anonymous-read.sh and document the intended security model: admin only for publish; anonymous read-only on hop-plugins-local for Hop downloads (no admin credentials in hop-config).
Repository create REST is Pro-only on Artifactory OSS. Rewrite configure-anonymous-read.sh to document UI steps and verify anonymous read instead of calling Pro APIs.
Clarify that create-repository REST is Pro-only and point users at the UI steps; keep configure-anonymous-read.sh as a verifier.
Document Administration module switching and deep links for repositories, anonymous access (Security → General), and permissions. Stop implying old "User Management → Settings" paths as primary.
Add docker/marketplace-nexus with scripted bootstrap (anonymous read, hosted hop-plugins repo), publish Wave 1 zips, and SNAPSHOT resolution via maven-metadata.xml. Remove docker/marketplace-artifactory entirely.
Relative HOP_HOME values were resolved against the current directory, doubling paths like assemblies/client/target/hop when already inside the install. Prefer any candidate that contains plugins/, including cwd.
…oke test Resolve the client install from process cwd (hop launchers already cd there), drop HOP_HOME/hop.home indirection, clarify hop-config is always under HOP_CONFIG_FOLDER, improve 401 auth errors, and add an anonymous Nexus CLI smoke script for install/list/validate/apply/uninstall.
…ll fallback Pre-configure repository.apache.org as primary with Maven Central fallback, add hop marketplace repo commands and Manage Repositories dialog, try enabled repos in order on install, and report disabled IT projects as skipped instead of a false docker environment failure.
…e gate Add user-manual marketplace page, release-manager notes for deploying optional plugin zips to ASF Nexus, and packaging/size guidance for the lean hop-client.
…v.yaml Expand the user-manual marketplace page with apply and validate options, file discovery, hop-run enforcement, and a full hop-env.yaml field reference with examples.
…gistry Add optional-plugins.yaml as the source of truth for marketplace-optional plugins, generate full-client-env.yaml at build time, package it at the Hop install root (and samples), load the GUI catalog from the registry, and document one-command restore of the previous full plugin set.
Add environment file path, Browse, Validate, and Apply to the Marketplace dialog (with Prune and Strict options matching the CLI), pre-fill full-client-env.yaml when present, and document the GUI flow.
Package Beam runners/libraries with the standard Beam marketplace plugin under lib-beam instead of lib/beam. Slim client libs keep spark-client and shared lib/core extras. Launch scripts add lib-beam only when present. Maven profile -Pbeam bakes the full Beam plugin into hop-client for IT/dev.
Add CLI query FILTER to search the local optional-plugins registry by case-insensitive substring across artifact id, name, category, description, and install path, with optional [installed] markers and docs.
mattcasters
marked this pull request as ready for review
July 19, 2026 23:16
… Nexus Add a developer-manual guide for packaging marketplace-compatible plugin zips, declaring Hop version and plugin dependencies, configuring Nexus, and automating releases with a template script for a specific plugin version.
Contributor
|
We will have to take a look at our release process too; currently, we are not publishing zips to Maven Central. |
…docs Improve the marketplace GUI with Plugins/Environment/Repositories tabs, inline repository and hop-env editors, search, and TableView lists. Add AttributesContext and IAttributes-based lifecycle environment hooks so marketplace can validate or enforce hop-env on project enable without a hard dependency between plugins. Include GUI screenshots, user manual updates, 13 locale translations, and related tooling.
Integrate upstream main (friction reduction, Databricks VFS, editor improvements, etc.) with marketplace work. Resolve conflicts in LifecycleEnvironmentDialog by keeping tabbed layout and marketplace XP tabs while retaining main's canvas text, import variables, and name suggestions.
mattcasters
changed the title
Bring in ExecProcess single-command tokenization fix (apache#7581/apache#7584) and other main updates to unblock PR build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements apache/hop#7547: optional Wave 1 plugins out of the fat client, a bundled marketplace (CLI + GUI) that installs Maven GAV plugin zips into the client
plugins/tree, declarativehop-env.yaml, and local Nexus tooling for SNAPSHOT development.lib/beamstays in the core client.repository.apache.orgprimary + Maven Central fallback chain), GUI Tools → Marketplace with Manage repositories.hop-env.yamlapply/validate and optional hop-run drift enforcement.docker/marketplace-nexus/) + smoke test; IT path re-injects Wave 1 viatools/install-wave1-plugins.sh.Done in this PR
repo *plugins/(launcher cwd; noHOP_HOMErequired)enforceOnRuntools/check-assembly-size.sh) and packaging docsIntentionally out of scope / follow-up
searchApiUrlreserved)disabled.txtleft in place)Test plan
./mvnw -pl plugins/misc/marketplace test(28 tests)./tools/check-assembly-size.sh— hop-client ~590 MBsmoke-test.shand/or./hop marketplace install hop-tech-parquet./hop marketplace repo list, install a Wave 1 plugin, restart, confirm plugin loads./hop marketplace apply -fsample hop-env against local or ASF repo when available