tools: introduce git-change-exec tool by christoph-zededa · Pull Request #4202 · lf-edge/eve

christoph-zededa · 2024-09-03T17:12:01Z

This new tool detects if in your git tree changed files (compared to master branch and local-only changed files) and uses this information to run only the specified actions.

Here it is used to run pillar's go-tests only if something changed there, same for this tool itself and the get-deps tool.

christoph-zededa · 2024-09-03T17:53:20Z

No, this is not possible with what gh checks out here, so closing ...

christoph-zededa · 2024-09-03T18:19:43Z

No, this is not possible with what gh checks out here, so closing ...

hmm, seems it is possible, but I don't know what I am doing with those gh actions ...

OhmSpectator · 2024-09-04T09:31:46Z

Do you know how you're going to use it? Will developers use it for local runs, or will you make it part of CI?

christoph-zededa · 2024-09-04T09:37:18Z

Do you know how you're going to use it? Will developers use it for local runs, or will you make it part of CI?

It is used by make test which is also used by the CI.
So the answer is: both

deitch · 2024-09-04T09:43:49Z

I am a little bit confused. Is this tool's purpose to check which directories have changed files, and thus determine where we should run go test? For example, if pkg/a had changes and pkg/b did not, then run go test ./pkg/a as opposed to go test ./...?

Does it actually execute the go test, or just give the relevant paths, sort of like:

$ gce ./...
pkg/a
pkg/c
pkg/q

thus telling us which dirs changed? And we always can choose to ignore it and run all tests? And with it, I would run go test $(gce ./...)? Something like that?

A README would help, along with some sample of what it would look like when run.

OhmSpectator · 2024-09-04T09:46:29Z

Aha! I've got the idea. I like it. Good luck with the implementation =) I find it easier to write such tools in interpreted languages, but okay, let it be =)

christoph-zededa · 2024-09-04T09:47:35Z

I am a little bit confused. Is this tool's purpose to check which directories have changed files, and thus determine where we should run go test? For example, if pkg/a had changes and pkg/b did not, then run go test ./pkg/a as opposed to go test ./...?

Does it actually execute the go test, or just give the relevant paths, sort of like:
$ gce ./...
pkg/a
pkg/c
pkg/q
thus telling us which dirs changed? And we always can choose to ignore it and run all tests? And with it, I would run go test $(gce ./...)? Something like that?

A README would help, along with some sample of what it would look like when run.

It does not do magic, it just looks in actions.go what to do exactly, f.e.: https://github.com/lf-edge/eve/pull/4202/files#diff-5bf19742ee6060707f406c1625c23226124e161d2049d12a80a8a61ef076ace8R47

Let me just be lazy with the example: https://github.com/lf-edge/eve/actions/runs/10699251983/job/29660387351?pr=4202#step:3:120 :-)

OhmSpectator · 2024-09-04T12:15:59Z

+			return dir
+		}
+
+		dir = filepath.Join(dir, "..")


I remember the case when Eden removed my home dir because it accessed something in .., and I'm afraid of such patterns now. Could you consider using git rev-parse --show-toplevel instead?

https://github.com/go-git/go-git ?

Doesn't git rev-parse --show-toplevel just do the same?

45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog", {st_mode=S_IFDIR|0755, st_size=226, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog/.git", 0x7ffe67413ec0, 0) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog/.git/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar", {st_mode=S_IFDIR|0755, st_size=1204, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/.git", 0x7ffe67413ec0, 0) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/.git/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg", {st_mode=S_IFDIR|0755, st_size=804, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/.git", 0x7ffe67413ec0, 0) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/.git/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve", {st_mode=S_IFDIR|0755, st_size=824, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/.git", {st_mode=S_IFDIR|0755, st_size=240, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/.git/HEAD", {st_mode=S_IFREG|0644, st_size=49, ...}, AT_SYMLINK_NOFOLLOW) = 0

https://github.com/go-git/go-git ?

I am not sure what you mean.

I'm afraid of manually adding logic to reverse directories. If there is a tool that does it for us, I would prefer it.

Now it is using https://github.com/lf-edge/eve/pull/4202/files#diff-711d61f4f130257efb2bca48eebbdc4d829cf9f3f687f00014abe0c9aafc2ccfR49

Yay, thanks! Will take a look at the rest of the script later...

christoph-zededa · 2024-11-11T13:35:08Z

I didn't have time lately to work on it, so I move it into draft state.

christoph-zededa · 2025-10-05T10:23:04Z

Some updates:

Now gce (git-change-exec) does not only provide the path to the changed file, but also:

the diff
tree-sitter annotations

Actions can now be written in go or in lua
The two steps (match and do) are now less entangled and one can export a json from the first step and load it into the second

Still there are some things I have to make nicer:

rethink if we really need 3 binaries
the cli parameters (f.e. introduce a debug subcommand)
Unify MatchDiff and MatchPath
Your input
etc.

christoph-zededa · 2025-10-05T10:42:25Z

Can we make this more modular by splitting it into a git-change tool and a change-exec tool, with a list of files being the output from the first and the input to the second?

Yes, for example you can run:

$ ./cmd/gc/gc actions/tests/debug/ssh.gce.lua
2025/10/05 12:36:53 Loading /home/christoph/projects/eve-3/tools/git-change-exec/actions/tests/debug/ssh.gce.lua ...
{
	"Actions": {
		"lua:/home/christoph/projects/eve-3/tools/git-change-exec/actions/tests/debug/ssh.gce.lua": [
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 0,
					"Line": "RUN ln -s /usr/src/openssh-${OPENSSH_VERSION} /usr/src/openssh",
					"LineNumber": 79,
					"TypeOfLine": {
						"run_instruction": [
							{
								"ColFrom": 0,
								"ColTo": 62
							}
						],
						"shell_command": [
							{
								"ColFrom": 4,
								"ColTo": 62
							}
						],
						"shell_fragment": [
							{
								"ColFrom": 4,
								"ColTo": 62
							}
						],
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 1,
					"Line": "",
					"LineNumber": 129,
					"TypeOfLine": {
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 1,
					"Line": "FROM build AS test",
					"LineNumber": 130,
					"TypeOfLine": {
						"from_instruction": [
							{
								"ColFrom": 0,
								"ColTo": 18
							}
						],
						"image_alias": [
							{
								"ColFrom": 14,
								"ColTo": 18
							}
						],
						"image_name": [
							{
								"ColFrom": 5,
								"ColTo": 10
							}
						],
						"image_spec": [
							{
								"ColFrom": 5,
								"ColTo": 10
							}
						],
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 1,
					"Line": "COPY collect-info-test /collect-info-test",
					"LineNumber": 132,
					"TypeOfLine": {
						"copy_instruction": [
							{
								"ColFrom": 0,
								"ColTo": 41
							}
						],
						"path": [
							{
								"ColFrom": 5,
								"ColTo": 22
							},
							{
								"ColFrom": 23,
								"ColTo": 41
							}
						],
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 1,
					"Line": "WORKDIR /collect-info-test",
					"LineNumber": 133,
					"TypeOfLine": {
						"path": [
							{
								"ColFrom": 8,
								"ColTo": 26
							}
						],
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						],
						"workdir_instruction": [
							{
								"ColFrom": 0,
								"ColTo": 26
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 1,
					"Line": "RUN go test",
					"LineNumber": 134,
					"TypeOfLine": {
						"run_instruction": [
							{
								"ColFrom": 0,
								"ColTo": 11
							}
						],
						"shell_command": [
							{
								"ColFrom": 4,
								"ColTo": 11
							}
						],
						"shell_fragment": [
							{
								"ColFrom": 4,
								"ColTo": 11
							}
						],
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Dockerfile",
				"Ld": {
					"Operation": 1,
					"Line": "",
					"LineNumber": 135,
					"TypeOfLine": {
						"source_file": [
							{
								"ColFrom": 0,
								"ColTo": 4294967295
							}
						]
					}
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 1,
					"Line": ".PHONY: test",
					"LineNumber": 3,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 1,
					"Line": "test:",
					"LineNumber": 4,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": ".PHONY: test-collect-info test-openssh debug-docker-image-build",
					"LineNumber": 3,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "",
					"LineNumber": 4,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "debug-docker-image-build: Dockerfile",
					"LineNumber": 5,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 1,
					"Line": "\tdocker build --rm --target test .",
					"LineNumber": 6,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "\tdocker build --target build .",
					"LineNumber": 7,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "",
					"LineNumber": 8,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "test-collect-info: debug-docker-image-build",
					"LineNumber": 9,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "\tdocker run -w /collect-info-test -v $(realpath collect-info-test):/collect-info-test  -it $(shell docker build --target build -q .) go test",
					"LineNumber": 10,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "test-ssh: debug-docker-image-build",
					"LineNumber": 11,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "\tdocker run -w /usr/src/openssh -it $(shell docker build --target build -q .) make tests",
					"LineNumber": 12,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "",
					"LineNumber": 13,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "",
					"LineNumber": 14,
					"TypeOfLine": null
				}
			},
			{
				"Path": "pkg/debug/Makefile",
				"Ld": {
					"Operation": 0,
					"Line": "test: test-collect-info test-ssh",
					"LineNumber": 15,
					"TypeOfLine": null
				}
			}
		]
	}
}

This output can be imported into another tool, f.e. this way:

$ ./cmd/gc/gc actions/tests/debug/ssh.gce.lua
2025/10/05 12:40:18 Loading /home/christoph/projects/eve-3/tools/git-change-exec/actions/tests/debug/ssh.gce.lua ...
2025/10/05 12:40:26 Loading /home/christoph/projects/eve-3/tools/git-change-exec/actions/tests/debug/ssh.gce.lua ...
2025/10/05 12:40:26 Running ...
2025/10/05 12:40:33 would run lua:/home/christoph/projects/eve-3/tools/git-change-exec/actions/tests/debug/ssh.gce.lua, but running dry ...

BUT: the actions have to be written in a way so that no state is saved in the first step and will be accessed in the second step.

also allow to use this tool directly with a path as a parameter Signed-off-by: Christoph Ostarek <christoph@zededa.com>

The option is called `ignore-words` according to https://github.com/codespell-project/codespell?tab=readme-ov-file#using-a-config-file Also add `OptionAll` as is not a misspelled `optionally` Signed-off-by: Christoph Ostarek <christoph@zededa.com>

add this to have a somehow similar interface as other parts in the repo to run the tests Signed-off-by: Christoph Ostarek <christoph@zededa.com>

This new tool detects if in your git tree changed files (compared to master branch and local-only changed files) and uses this information to run only the specified actions. Here it is used to run pillar's go-tests only if something changed there, same for this tool itself and the get-deps tool. Signed-off-by: Christoph Ostarek <christoph@zededa.com>

necessary after the recent changes Signed-off-by: Christoph Ostarek <christoph@zededa.com>

actions/checkout takes the PR head from the contributor's fork, leaving origin/master stale, fix it Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com>

christoph-zededa requested review from deitch and eriknordmark as code owners September 3, 2024 17:12

github-actions Bot requested a review from jsfakian September 3, 2024 17:12

christoph-zededa marked this pull request as draft September 3, 2024 17:13

christoph-zededa force-pushed the git_change_detect branch from 97865b3 to 51a141e Compare September 3, 2024 17:21

christoph-zededa closed this Sep 3, 2024

christoph-zededa reopened this Sep 3, 2024

christoph-zededa force-pushed the git_change_detect branch 4 times, most recently from a9877ab to 575f3f0 Compare September 3, 2024 18:19

christoph-zededa force-pushed the git_change_detect branch from 575f3f0 to 2dd51fe Compare September 3, 2024 18:25

christoph-zededa force-pushed the git_change_detect branch 5 times, most recently from 6a6c964 to ea1bf18 Compare September 4, 2024 10:48

christoph-zededa marked this pull request as ready for review September 4, 2024 11:35

OhmSpectator reviewed Sep 4, 2024

View reviewed changes

christoph-zededa force-pushed the git_change_detect branch from ea1bf18 to d55a98b Compare September 4, 2024 13:38

christoph-zededa requested a review from yash-zededa as a code owner September 4, 2024 13:38

github-actions Bot requested a review from uncleDecart September 4, 2024 13:39

christoph-zededa force-pushed the git_change_detect branch from d55a98b to 112ed2d Compare September 4, 2024 13:40

christoph-zededa marked this pull request as draft November 11, 2024 13:34

christoph-zededa mentioned this pull request Jul 29, 2025

vcomLink communication refactoring #4502

Merged

9 tasks

christoph-zededa force-pushed the git_change_detect branch from d6a6fec to a481ba0 Compare October 5, 2025 10:14

github-actions Bot requested review from eriknordmark and rouming October 5, 2025 10:15

christoph-zededa force-pushed the git_change_detect branch from a481ba0 to b5ac4b0 Compare October 5, 2025 10:24

christoph-zededa force-pushed the git_change_detect branch from b5ac4b0 to 01d4402 Compare October 5, 2025 10:48

christoph-zededa force-pushed the git_change_detect branch from 01d4402 to 7a8dd82 Compare June 17, 2026 17:21

github-actions Bot requested a review from shjala June 17, 2026 17:22

christoph-zededa force-pushed the git_change_detect branch from 7a8dd82 to d391765 Compare June 17, 2026 17:31

github-actions Bot requested review from OhmSpectator, andrewd-zededa, europaul, milan-zededa, naiming-zededa, rene, rucoder and zedi-pramodh June 17, 2026 17:32

christoph-zededa force-pushed the git_change_detect branch from d391765 to 637449a Compare June 18, 2026 16:38

christoph-zededa and others added 7 commits July 31, 2026 14:54

tools: spdx-check file parameter

70d3fda

also allow to use this tool directly with a path as a parameter Signed-off-by: Christoph Ostarek <christoph@zededa.com>

codespellrc: add OptionAll and fix

769e2ce

The option is called `ignore-words` according to https://github.com/codespell-project/codespell?tab=readme-ov-file#using-a-config-file Also add `OptionAll` as is not a misspelled `optionally` Signed-off-by: Christoph Ostarek <christoph@zededa.com>

tools/get-deps: add 'test' to Makefile

6ed92a5

add this to have a somehow similar interface as other parts in the repo to run the tests Signed-off-by: Christoph Ostarek <christoph@zededa.com>

pkgs: bump Dockerfiles

d5fecd0

necessary after the recent changes Signed-off-by: Christoph Ostarek <christoph@zededa.com>

wip

b500464

.github: point origin at base repo in go-tests

43a6294

actions/checkout takes the PR head from the contributor's fork, leaving origin/master stale, fix it Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com>

christoph-zededa force-pushed the git_change_detect branch from 637449a to 43a6294 Compare July 31, 2026 15:53

Uh oh!

Conversation

christoph-zededa commented Sep 3, 2024

Uh oh!

christoph-zededa commented Sep 3, 2024

Uh oh!

christoph-zededa commented Sep 3, 2024

Uh oh!

OhmSpectator commented Sep 4, 2024

Uh oh!

christoph-zededa commented Sep 4, 2024

Uh oh!

deitch commented Sep 4, 2024

Uh oh!

OhmSpectator commented Sep 4, 2024

Uh oh!

christoph-zededa commented Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

OhmSpectator Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

OhmSpectator Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

christoph-zededa Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

OhmSpectator Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

christoph-zededa Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

OhmSpectator Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

christoph-zededa commented Nov 11, 2024

Uh oh!

christoph-zededa commented Oct 5, 2025

Uh oh!

christoph-zededa commented Oct 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

christoph-zededa commented Sep 4, 2024 •

edited

Loading