[Feature Request]: add parent project support to basic image

[hansva]

What would you like to happen?

Migration: https://issues.apache.org/jira/browse/HOP-3470

if you have a project structure like

projects

project A
project B
where project B inherits from project A (so not by the default project, see project-config.json > parentProjectName) the configured metadata path "metadataBaseFolder" from project A is not resolved and not available (e.g. merged into project B's metatdata folder) for project B executions

docker call for example by (/files/projects/ in that case will include both projects "project A" and "project B")

docker run --rm
--env HOP_LOG_LEVEL=Basic
--env HOP_FILE_PATH="jobs/load_projectB_data.hwf"
--env HOP_PROJECT_DIRECTORY=/files/projects/projectB
--env HOP_PROJECT_NAME=projectB
--env HOP_ENVIRONMENT_NAME=prod_docker_exec
--env HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS="/files/projects/projectB/prod_docker_exec-config.json"
--env HOP_RUN_CONFIG=docker_run_config
--name "docker-exec-projectB"
apache/incubator-hop:latest

Issue Priority

Priority: 3

Issue Component

Component: Containers, Component: Hop Config

[dsanderbi]

Hi @hansva,

i have a question regarding the feature request.

What behavior exactly did you have in mind?

If we introduce the variables

ENV HOP_PARENT_PROJECT_NAME=
ENV HOP_PARENT_PROJECT_CONFIG_FILE_NAME=parent-project-config.json
ENV HOP_PARENT_PROJECT_FOLDER=

in the Dockerfile, we could add a check in load-and-execute.sh right after the HOP_PROJECT_FOLDER (since it doesn't make sense to add a parent without the main project).

---

The downside is that only one hierarchy level can be passed.

Compared to the GUI if you have for example three projects like
project-super, project-main and project-parent where

• project-parent is the parent of project-main
• project-main is the parent of project-super
  the project-super also can access metadata from project-parent

This would not be possible with implementation using additional Docker variables.

Other approaches, like creating a project for each folder mounted under /files, are difficult because we don’t know which of the contained projects belongs to which parent.

---

Would it be sufficient for you if only one level of inheritance could be specified for the Container?

I tested this with a short-lived container,
and there the metadata passing works (e.g., a database connection defined in the parent and then used in the main project).

When testing with a long-lived container, for example with a service defined in the parent,
services coming from the parent cannot be accessed,
unless you adjust the path in the main project but then it also gets changed in the parent. The same behavior occurs in the GUI.

For this to work, there would need to be a PARENT_PROJECT_HOMEas requested here #2331
which also supports the idea that this could be solved using the additional variables.

[mattcasters]

Parent project support in the Docker basic image

Implemented one-level parent project registration for the official Hop Docker / Hop Web images so a child project can inherit parent metadata when both homes are mounted.

Root cause

The entrypoint only registered HOP_PROJECT_NAME / HOP_PROJECT_FOLDER. Parent inheritance in Project.modifyVariables looks up the parent in hop-config; if the parent is not registered, its metadataBaseFolder is never merged into HOP_METADATA_FOLDER.

Changes

• load-and-execute.sh / run-web.sh: register the parent project first (when configured), then the main project with --project-parent and --project-keep-config-file.
• New environment variables:
  • HOP_PARENT_PROJECT_NAME
  • HOP_PARENT_PROJECT_FOLDER
  • HOP_PARENT_PROJECT_CONFIG_FILE_NAME (default: project-config.json)
• hop-conf project-create: if an existing project-config.json already defines parentProjectName, that value is kept instead of always forcing the standard parent (unless --project-parent is set).
• Docs: docker container tech manual + hop-web env table, including a parent-project example.

Scope

Only one parent level is supported via these env vars (as discussed). Deeper chains would need a follow-up.

Related

Works together with #2331 (PARENT_PROJECT_HOME / PARENT_PROJECT_NAME) so paths and long-lived setups can reference the parent home after the parent is registered.

Example

docker run --rm \
  -v /host/projects:/files/projects \
  --env HOP_FILE_PATH='jobs/load_projectB_data.hwf' \
  --env HOP_PROJECT_FOLDER=/files/projects/projectB \
  --env HOP_PROJECT_NAME=projectB \
  --env HOP_PARENT_PROJECT_FOLDER=/files/projects/projectA \
  --env HOP_PARENT_PROJECT_NAME=projectA \
  --env HOP_ENVIRONMENT_NAME=prod_docker_exec \
  --env HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/files/projects/projectB/prod_docker_exec-config.json \
  --env HOP_RUN_CONFIG=docker_run_config \
  apache/hop:<tag>

HOP_PARENT_PROJECT_NAME should match parentProjectName in the child project’s project-config.json.

Commit on branch eil-friction: 3830b5f8af (Issue #2596 : add parent project support to Docker basic image).