Interfaces for Selecting Agents

[CharliePoole]

@oznetmaster @simoneb @rowo360

For beta9 we plan to add support for Remote and InProcess agents in addition to LocalProcess agents. This will, of course, require support from NUnit for those agent types, which is not yet available. In this discussion we need to talk about "interfaces" in two different contexts:

1. The GUI interface seen and used by a user to select an agent.
2. The programming interfaces to be implemented in NUnit and used by TestCentric to carry out this functionality.

We should qualify the word "interface" whenever the context is not clear.

This discussion is publicly visible and anyone interested is free to join. I've mentioned three people above. One of them is part of the NUnit engine team and two are part of the TestCentric GUI team. I'm involved in both.

I'll next seed this discussion with comments I already received by email. Hang on... :-)

[CharliePoole]

I asked...

Think about how to integrate remote and in-process execution into the GUI interface.

Simone responded...

Some first thoughts after looking at how the GUI works today. The Select Agent menu is built from ITestModel.AvailableAgents, which is just a list of names, and the chosen one goes into the RequestedAgentName setting. That’s fine for a handful of local agents, but not for remote ones.

1. Group by type. In-process, local process and remote as three groups, with “automatic” as the default. If the engine lets us request a type instead of a name, that’s also what we’d pass when the user doesn’t pick a specific agent.
2. Remote agents need more than a name. The list could be long and change over time, so I’d move them out of the menu into a small dialog showing the description, runtime and machine for each agent. That’s where the Description field you mentioned, and probably a location or host field, would come in.
3. Maybe the user shouldn’t see every remote agent at all, but pick a server and then the agents it offers.

[CharliePoole]

I asked...

Think about how to integrate remote and in-process execution into the GUI interface.

Roald (@rowo360) responded...

Simone's thoughts are already spot on—I can only add:

• We can extend our settings so that a user can manage a list of servers there. That would involve an extension to the setting UI so that he can add and remove servers (or maybe even move servers up or down in that list). But the list of servers would also be saved in our settings file, so that his preferred servers would be restored upon restart.
• I wouldn’t store the identifiers of the available agents. As you said, they will change over time. But instead, I would query for the available agents on demand.
• Perhaps we could keep the “Select agent” menu as well. All local agents remain menu items as it is. And for each server, we add a submenu. When you open it, all of the server's agents are displayed. We might display the agents name only, or any additional info. Or use a tooltip to help identifying the right one.
• But opening a dialog to select a ‘remote agent’ sounds also fine.

[CharliePoole]

@oznetmaster
I had planned to post excerpts from our email exchanges here, but they are extensive and primarily concerned with internal architecture. Of course, we'll get to internal stuff soon enough but I think it's easier if you respond yourself about the user interface. I know you're familiar with TestCentric and also have your own GUI. How do you see users interacting to get a particular agent running on a remote server?

As to programming interfaces, I believe the GUI will not see anything below the level of info provided by TestAgentServer, so our discussions of how RemoteProcessTestAgency gets an agent may be out of scope here, even though it will be driven in large part by our needs. Would you agree?

[oznetmaster]

@CharliePoole , @simoneb and @rowo360 ,

I agree with the direction Simone and Roald have outlined. My preference would be to keep the common local case simple and provide a small dialog for remote selection.

I would envisage this workflow:

1. Leave Default behaving as local execution does today. Offer explicit choices for In process, Local process and Remote.

2. Selecting Remote opens a dialog listing the user's configured targets, with friendly names and descriptions. Users can add a target explicitly; automatic server discovery need not be part of the initial implementation.

3. After choosing a target, the user normally leaves agent selection at Automatic for this target. Someone needing a particular runtime or agent can expand the available choices, showing runtime, architecture and description.

4. Keep the selected destination visible beside the execution controls. When results arrive, retain the actual destination and agent information so users can tell where the tests ran.

I would support Roald's suggestion to persist configured servers and query their capabilities when needed. What we persist for a project should describe the requested destination and selection policy, rather than identify a particular running agent process. An explicitly requested agent implementation could also be remembered, provided its absence produces a clear explanation.

One important detail is that destination selection must be possible before test discovery, including when the package cannot execute on the GUI machine. Changing the destination after discovery would require an appropriate unload/reload and fresh discovery.

I would also distinguish a configured target from an already-running server. Our deployment approach allows the server to be deployed with the test package. The user should therefore be able to select that target even when its server is not currently running. Opening the selection dialog should not itself deploy software, acquire a reservation or start a test agent.

Once a remote destination is explicitly selected, inability to use it should produce a clear failure and an opportunity to retry or change the selection. It should never silently execute the tests locally.

My older Crestron GUI reinforces the usefulness of remembering a destination and making it visible. Its processor-specific controls would not belong in TestCentric, but the general destination-selection workflow does.

[CharliePoole]

I'm seeing more agreement than disagreement here. I'm in agreement as well. Here are the details as I see them.

1. A menu like our existing one, with additions for "In Process" and "Remote".
2. I'd put "In Process" and "Remote" below the existing list with a separator line above them.
3. "In Process" is actually an agent, like the others. However, I think it should be handled differently. The text will show "In Process" rather than the agent name and will be disabled if the in process extension is not installed.
4. "Remote" will lead to a submenu, still to be designed.

This is still pretty high level. @simoneb, can you take what we have all said and come up with a mockup for us?

[CharliePoole]

@oznetmaster I have a bunch of questions about your comment, which I'll ask inline...

I agree with the direction Simone and Roald have outlined. My preference would be to keep the common local case simple and provide a small dialog for remote selection.

Agreed!

I would envisage this workflow:

1. Leave Default behaving as local execution does today. Offer explicit choices for In process, Local process and Remote.

Yes, Default implies a separate local process for each assembly, targeting the runtime for which the assembly was built or the next higher compatible runtime for which an agent is available.

2. Selecting Remote opens a dialog listing the user's configured targets, with friendly names and descriptions. Users can add a target explicitly; automatic server discovery need not be part of the initial implementation.

What is meant by "configured targets in this context?" Is it a server with one or more agents available potentially available?

3. After choosing a target, the user normally leaves agent selection at Automatic for this target. Someone needing a particular runtime or agent can expand the available choices, showing runtime, architecture and description.

So this would be similar to our own agent selection minus "In Process" and "Remote" but also showing more information about the agent? Maybe our own menu should provide more information as well for local process agents, perhaps on hovering.

4. Keep the selected destination visible beside the execution controls. When results arrive, retain the actual destination and agent information so users can tell where the tests ran.

Do you mean the buttons in the GUI? If so, perhaps we should move agent selection from the File menu to the bar using a dropdown button.

I would support Roald's suggestion to persist configured servers and query their capabilities when needed. What we persist for a project should describe the requested destination and selection policy, rather than identify a particular running agent process. An explicitly requested agent implementation could also be remembered, provided its absence produces a clear explanation.

I agree. Our settings are mainly strings, but that could be json or xml if we like.

One important detail is that destination selection must be possible before test discovery, including when the package cannot execute on the GUI machine. Changing the destination after discovery would require an appropriate unload/reload and fresh discovery.

This one confused me. Is this a problem with respect to the format of the test assembly? Or is the assembly simply not visible to the machine running the GUI? I think I'm missing something in your use case. Of course, this would not be a problem in the console runner, which doesn't do a separate discovery step before execution.

I would also distinguish a configured target from an already-running server. Our deployment approach allows the server to be deployed with the test package. The user should therefore be able to select that target even when its server is not currently running. Opening the selection dialog should not itself deploy software, acquire a reservation or start a test agent.

This brings us back to "configured target" which I was assuming was a server and was likely already running. However, I recognize this as a back-end discussion and most likely can be postponed for a bit. :-) Neither opening the dialog nor even selecting an agent to use should cause anything to happen. Although older agents needed to be loaded in memory in order to ask them if they could run a particular package, our latest versions no longer require being loaded until the actual discovery or execution begins.

Once a remote destination is explicitly selected, inability to use it should produce a clear failure and an opportunity to retry or change the selection. It should never silently execute the tests locally.

Agreed.

My older Crestron GUI reinforces the usefulness of remembering a destination and making it visible. Its processor-specific controls would not belong in TestCentric, but the general destination-selection workflow does.

It may be helpful to see an image of your Crestron GUI if that would be acceptable to the client. Otherwise, perhaps a rough drawing?

In spite of the questions, I think we are making progress.

[simoneb]

@CharliePoole here's a first mockup putting together what you, Roald and Neil said, including this morning's points about the toolbar drop-down and the info on hover. Names, menus and toolbar buttons come from the current code; everything else is a proposal.

A few things I'm not sure about yet:

1. Each remote target has its own submenu, and there's also a "Choose agent on target..." dialog. I'd keep only the dialog, so the menu stays short.
2. A target is shown as host and port. That doesn't cover Neil's case where the server is deployed with the test package, so the target details depend on how the engine ends up describing a target.
3. The dialog shows "Server reachable. Agents listed at...". To match "nothing happens until discovery or execution", it should only contact the server on Refresh or when you expand the agent list.
4. The status dots next to each target suggest we keep checking the servers. I'd leave them out of a first version.
5. Should File › Select Agent stay as a mirror of the drop-down, or go away?

[CharliePoole]

Thanks for doing this. It's late so I won't comment on the details now except...

1. I assume the "Remote Targets" page would appear by itself when clicking "Manage Remote Targets" in the dropdown.
2. Yes, I'd remove SelectAgent from the menu.

[oznetmaster]

@CharliePoole , @simoneb and @rowo360

The toolbar drop-down in Simone's mockup captures what I meant by keeping the destination visible beside the execution controls. I support that direction, preserving the existing local choices and adding In Process and Remote below the separator as Charlie suggested.

To answer Charlie's questions, a configured target is a saved description of where we intend to run tests and how the execution infrastructure can reach that environment. For a resident server it can identify an existing endpoint. For deployment on demand it identifies the machine or processor and the deployment/startup configuration; the NUnit server endpoint may only become known when preparation occurs. In either case, the target may offer one or more compatible agents. It is not the identity of an already-running agent process.

Yes, Automatic on a selected remote target is analogous to the existing local automatic choice, constrained to that target. Additional information on hover would be useful for local agents as well. An explicit remote selection should remain remote if acquisition fails, as we have agreed.

My discovery point needs a clearer example. The test assembly can be present on the GUI machine, ready to deploy, but require a runtime environment or platform/vendor dependencies available only on the remote target. Reading the assembly's metadata locally is different from loading it and discovering its NUnit fixtures there. The GUI therefore needs to let the user select the remote destination before the initial engine discovery attempt, or recover from an unsuccessful local attempt by changing the destination. I was not proposing that the initial implementation must open assemblies that exist only on the server. The Console can receive the destination in its initial invocation; the GUI needs an equivalent path before automatic discovery.

My earlier statement about changing destinations was also broader than necessary: when the change means discovery will use a different execution environment, we should unload and rediscover rather than assume that the old tree and test identities remain valid. We can decide the exact reload behavior as the GUI implementation develops.

On Simone's specific questions:

1. I favor one remote-selection dialog for the initial version, rather than maintaining both a detailed hierarchy of target/agent submenus and the dialog. A short Remote entry leading to that dialog is sufficient.

2. Use a friendly target name and description as the primary identity. Host/port can be details for a resident server, but should not be required to identify every target. The engine-side target definition can supply the connection details when needed. Charlie has now confirmed in Refactor to support all three AgentTypes nunit/nunit-console#1919 that both resident and deployed-on-demand servers are intended to be supported.

3. I agree that opening the menu/dialog or choosing a destination should not deploy or start anything. An explicit Refresh could query an already-running server for current capabilities. If an on-demand server has not been started, show that its agent information has not been queried and leave Automatic selectable. An inability to contact an unstarted server is not proof that the configured target cannot be used.

4. I agree with omitting the status dots initially. We should not imply background polling or current availability from an old observation. If cached agent information is displayed, label it as such.

5. I would keep File > Select Agent initially as a mirror of the toolbar control, sharing the same action and selection state. That preserves the established menu route without requiring separate behavior.

My older GUI's relevant feature was remembering the processor/program destination and making it clear while selecting tests and viewing results; deployment was handled by a separate program. Simone's mockup already conveys the general idea, so I do not think we need to reproduce those processor-specific controls. One small accuracy note: the CP4 example in the mockup is a possible future backend for us; our implemented processor backend is for Crestron Home.

I agree that the deployment and provider internals belong in the NUnit engine discussion. The GUI needs the target description, selection, useful progress and failure information, and the ordinary discovery/event/result interfaces.

[oznetmaster]

@CharliePoole , we crossed posts. I'm happy to remove File > Select Agent and use the toolbar drop-down as the single entry point; keeping the menu was only a preference, not a requirement.

Having Manage Remote Targets open the target-management page directly also makes sense to me.

[simoneb]

Here's v5 of the mockup, updated with the latest feedback:

• Run on is the only entry point, and File ▸ Select Agent is gone. Remote... is a single item that opens the dialog, with no per-target submenus.
• Targets are identified by name and description only. Host and port are optional connection details, shown only on the Remote Targets page, together with the target type ("Already running server" or "Started on demand").
• The dialog no longer shows any reachability text or status dots. There's a Refresh button that only queries servers that are already running, the selected target shows "agents not queried yet", and Automatic can always be selected.
• Manage remote targets... opens the Remote Targets page directly.
• The examples are now generic: "crestron-home" and "lab-device".
• When a load fails, the default button is Change Destination..., which lets you pick a remote target and then unload and rediscover. The same path is offered when a run fails, and tests are never silently run locally instead.

Feedback welcome!

[oznetmaster]

@simoneb, v5 captures the workflow I had in mind. In particular, the single Remote dialog, named targets with optional connection details, support for servers started on demand, and Automatic remaining selectable before capabilities have been queried all look right to me. The toolbar destination and actual destination retained with the result are also useful.

One small clarification for the failure flow: I would distinguish failure to prepare/start the selected destination from an ordinary NUnit test failure or an interrupted run. A failed assertion should remain a test result rather than prompt for a different destination. If execution has started and the connection is lost, changing the selection must not imply that the previous execution has stopped or its cleanup has completed; the previous result and any unresolved lifecycle state should remain visible.

Change Destination can select the new environment and trigger the necessary unload/discovery. Running the tests again should remain an explicit user action, rather than happen automatically as a consequence of changing destinations.

Subject to that distinction, I think this is a good basis for the initial implementation. I have no further menu-layout changes to suggest.

[simoneb]

@oznetmaster thanks, that's a useful distinction. Here's v6 of the mockup with it folded in:

• Change Destination... is only offered when the selected destination couldn't be prepared, started or reached (4b, 4c). A failed assertion is just a test result (4d), with no prompt to switch destinations.
• If the connection is lost mid-run (4e), the partial results stay visible and the run is shown as interrupted, with its remote state unknown and cleanup not confirmed. You can still change destination, but nothing suggests the previous execution has stopped or been cleaned up.
• Change Destination... only selects the new environment and triggers unload and rediscovery. The tests stay loaded and idle, and running them again is always an explicit action.

I also added a Reconnect button in 4e, which only asks the target for the state of that run. Happy to drop it if it doesn't fit.

With that settled, I agree this is a good basis for the initial implementation.

[oznetmaster]

@simoneb , v6 addresses the distinctions I raised, and I support proceeding on that basis.

The read-only query behind Reconnect could be useful. I would consider calling it Check Run Status, because Reconnect can suggest that the GUI will resume the event stream or recover missing results, which is more than querying the state of a run.

The query needs to identify the original run and server instance, rather than merely contact something now answering at the same address. If the backend cannot identify that run, or does not support querying it, the state should remain unknown. The action should not deploy a replacement server, acquire a new agent or rerun tests.

I would make this conditional on backend support, so it need not hold up the initial implementation. Restoring a live event stream or retrieving a completed result can be considered separately when the relevant backend supports it.

[simoneb]

Agreed, Check Run Status is clearer. I'll treat it as optional, enabled only when the backend can identify that run, and leave it out of the initial implementation.

[rowo360]

The discussion has already moved forward, with a great mock-up, lots of details, and good results. I agree with all of them. 👍
I can only add another aspect: while we all focus on Neil's use case, I believe that the majority of users will neither use remote agents nor deal with agents at all. I think they're actually confused or overwhelmed when they have the option to select an agent.
With these users in mind, I would keep the user interface simple by default. So, they can continue to work in 'automatic' mode and the toolbar keeps the focus on the main actions 'Run', 'Stop', 'Debug',... tests.
A simple solution would be to display the drop-down box in the toolbar only if any remote servers have been defined. Okay, just a minor point in this discussion. 😄

[oznetmaster]

@rowo360, I agree that the ordinary automatic/local workflow should remain simple and keep Run, Stop and Debug prominent.

My only reservation is tying the selector’s visibility exclusively to configured remote servers. Someone may want a particular local agent or In Process without having any remote targets, especially once File → Select Agent is removed.

Could we make the selector available through an explicit setting or command, and show it whenever a non-default execution choice is active? Configuring a remote target could also reveal it automatically. That would keep the default interface simple without making local agent selection dependent on remote configuration.

[CharliePoole]

@rowo360
I'm glad you said that. Let's try to keep our base users in mind.

I imagine three tiers of users...

1. The typical user who will use local processes and generally not change agents at all except possibly to use the InProcess agent when we have it. Some of them will use multiple agents because they develop for various platforms.
2. Users working who have separate servers to run tests. In fact, I'm one of those and many corporate users would probably fall into this category.
3. Users like Neil working in embedded environments, whose servers have unique features not generally needed. I used to be one of those although it's not something I do these days.

Normally, I'd focus entirely on the first two. However, since Neil is contributing his efforts on the NUnit team as well as here, we're paying a bit more attention to his needs. This seems like a fair tradeoff to me. It's beneficial to this project, since we'll get to use the features for tier 2 out of NUnit, which would otherwise not be included in the 4.0 engine.

It's not yet decided what interface we will eventually implement, but the design has another purpose as well. It's giving a picture of what info any client needs to get about a servers capabilities and what actions the client can perform. I now have a better understanding I had than I did when there were only verbal descriptions. I'm applying that in the NUnit interface design and we'll be able to use that new interface in TestCentric. You can see some of that in the discussion around nunit/nunit-console#1918.

On a more immediate note... if we don't show the dropdown, should we keep the file menu? I was thinking that the dropdown could be an easier thing for users, just as we use a dropdown for display strategies.

[oznetmaster]

@CharliePoole, I agree that the ordinary local workflow and general remote-server use should drive the initial GUI. The mockup has been useful for identifying what clients need from the engine, without committing us to every proposed control.

On the immediate question, I would be comfortable keeping a compact Run on drop-down visible, initially showing Default, with detailed choices appearing only when opened. That would keep local agent and In Process selection discoverable as well. If we choose to hide the drop-down, I agree that File > Select Agent, or an equally discoverable command, should remain available even when no remote targets have been configured.

Our general remote execution and deployment work is also intended for ordinary Windows, Linux and macOS test machines. Deploying the server on demand can be useful there as well as on embedded targets. Processor-specific configuration, reservations and restoration should remain backend responsibilities; the ordinary GUI user should not have to deal with them.

I am happy for the first implementation to stay small and for additional backend capabilities to appear only where supported and needed.

[simoneb]

Thanks everyone. I'd go with Neil's suggestion: keep a compact Run on drop-down always visible, showing Default. The typical user never needs to open it, but local agents (and In Process, once we have it) stay one click away, so we don't need File > Select Agent anymore. That's how #1629 works today: it lists only the local agents, shows "Run on: Default (local process)" and is disabled during a run. Remote targets would later just add entries to the same list. @rowo360, if it still feels like too much once you've tried it, we can revisit hiding it behind a setting.