iosxrd nodes initital configurations: no route to host

[stsfred]

Hi Ivan, Team,
I am testing mostly iosxrd (provider clab) but initial configurations do not get loaded to the nodes. Messages during nodes booting:

TASK [iosxr_config: deploying initial from /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/templates/initial/iosxr.j2] *** fatal: [PE5]: FAILED! => changed=false msg: 'ssh connection failed: ssh connect failed: no route to host' fatal: [PE6]: FAILED! => changed=false msg: 'ssh connection failed: ssh connect failed: no route to host'
etc.
Strange, because if I try to ssh like this to one of the nodes:

(mylab) fred@fred-lnx:~$ ssh clab@clab-ContainerLab-PE1
Warning: Permanently added 'clab-containerlab-pe1' (ED25519) to the list of known hosts.
(clab@clab-containerlab-pe1) Password: 

there is IP connectivity and the DNS name of the host is resolved.
Does anyone have an idea how to get rid of this?
Thanks.

[ipspace]

I don't think anyone ever properly tested the IOS XRd container (so: welcome to the club ;). Someone submitted the changes that worked for him, I merged them, and we moved on. Unfortunately, the vendors don't find it convenient to help independent software developers 🤷‍♂️

With that out of the way (sorry, I had to get it out):

• You're using device: iosxr with provider: clab, right?
• What version of netlab are you using?
• Just to be on the safe side, does the same thing happen with the simplest possible topology (one node, one link)?
• What happens if you do netlab connect PE1?
• Does netlab initial work after a while? Maybe it's just that we don't wait for IOS XR to boot?

[stsfred]

thanks Ivan for your quick reply.

• Yes, device: iosxr with provider: clab
• netlab latest version, clab latest version
• simple topology with no ospf/isis/bgp modules seems fine
• netlab connect PE1: did not try yet, but:
  I waited 2 mins after starting the nodes, then issued netlab initial , then all configs successfully loaded on all nodes. TBH, I just started with netlab I was not familiar with this command.
  Waiting for ~1 min. was not enough.
  My lab which I had problem with, includes 8x iosxrd nodes, custom addressing (ipv4+ipv6, custom loopbacks, custom link addresses), ospf, bgp, isis.
  Thanks again, sorry for the noob question, it seems like I was not patient enough.

[ipspace]

simple topology with no ospf/isis/bgp modules seems fine

Because a single XRd node starts soon enough

I waited 2 mins after starting the nodes, then issued netlab initial , then all configs successfully loaded on all nodes.

So it's just a matter of waiting for SSH to be ready.

TBH, I just started with netlab I was not familiar with this command.

No worries, we should have handled this internally.

Thanks again, sorry for the noob question, it seems like I was not patient enough.

Hey, it wasn't a noob question but a genuine bug. Thanks a million for reporting it.

[ipspace]

Please try to change the following default setting and tell me if it works:

netlab defaults devices.iosxr.clab.group_vars.netlab_ready=[ssh]

If it doesn't work, you can always delete it with

netlab defaults --delete --user devices.iosxr.clab.group_vars.netlab_ready

[stsfred]

Tried it, it helps a lot!

`mylab) fred@fred-lnx:~$ netlab up ContainerLabs/2as-v1.yml

┌──────────────────────────────────────────────────────────────────────────────────┐
│ CREATING configuration files │
└──────────────────────────────────────────────────────────────────────────────────┘
[CREATED] provider configuration file: clab.yml
[CREATED] transformed topology dump in YAML format in netlab.snapshot.yml
[CREATED] pickled transformed topology data into netlab.snapshot.pickle
[GROUPS] group_vars for all
[GROUPS] group_vars for modules
[GROUPS] group_vars for iol
[HOSTS] host_vars for CE1
[GROUPS] group_vars for iosxr
[HOSTS] host_vars for PE1
[HOSTS] host_vars for PE2
[HOSTS] host_vars for PE3
[HOSTS] host_vars for PE4
[HOSTS] host_vars for PE5
[HOSTS] host_vars for PE6
[HOSTS] host_vars for RR1
[CREATED] minimized Ansible inventory hosts.yml
[CREATED] Ansible configuration file: ansible.cfg

┌──────────────────────────────────────────────────────────────────────────────────┐
│ CHECKING virtualization provider installation │
└──────────────────────────────────────────────────────────────────────────────────┘
[SUCCESS] clab installed and working correctly

┌──────────────────────────────────────────────────────────────────────────────────┐
│ STARTING clab nodes │
└──────────────────────────────────────────────────────────────────────────────────┘
provider clab: executing sudo -E containerlab deploy --reconfigure -t clab.yml
[sudo] fred jelszava:
17:10:43 INFO Containerlab started version=0.72.0
17:10:43 INFO Parsing & checking topology file=clab.yml
17:10:43 INFO Removing directory path=/home/fred/clab-ContainerLab
17:10:43 INFO Creating docker network name=netlab_mgmt IPv4 subnet=192.168.121.0/24 IPv6 subnet="" MTU=0
17:10:43 INFO Creating lab directory path=/home/fred/clab-ContainerLab
17:10:43 INFO node "PE4" is being delayed for 7 seconds
17:10:43 INFO node "RR1" is being delayed for 7 seconds
17:10:43 INFO node "PE5" is being delayed for 7 seconds
17:10:43 INFO node "PE6" is being delayed for 7 seconds
17:10:43 INFO Creating container name=CE1
17:10:43 INFO Creating container name=PE3
17:10:43 INFO Creating container name=PE1
17:10:43 INFO Creating container name=PE2
17:10:44 INFO Created link: CE1:eth1 (Ethernet0/1) ▪┄┄▪ PE1:Gi0-0-0-0
17:10:44 INFO Created link: CE1:eth2 (Ethernet0/2) ▪┄┄▪ PE2:Gi0-0-0-0
17:10:44 INFO Running postdeploy actions for Cisco IOL 'CE1' node
17:10:44 INFO Running postdeploy actions for Cisco XRd 'PE3' node
17:10:44 INFO Created link: PE2:Gi0-0-0-3 ▪┄┄▪ PE3:Gi0-0-0-1
17:10:44 INFO Running postdeploy actions for Cisco XRd 'PE2' node
17:10:44 INFO Created link: PE1:Gi0-0-0-1 ▪┄┄▪ PE2:Gi0-0-0-1
17:10:44 INFO Created link: PE1:Gi0-0-0-2 ▪┄┄▪ PE3:Gi0-0-0-0
17:10:44 INFO Running postdeploy actions for Cisco XRd 'PE1' node
17:10:50 INFO Creating container name=PE4
17:10:50 INFO Creating container name=RR1
17:10:50 INFO Creating container name=PE6
17:10:50 INFO Creating container name=PE5
17:10:51 INFO Created link: PE2:Gi0-0-0-2 ▪┄┄▪ PE4:Gi0-0-0-0
17:10:51 INFO Created link: PE1:Gi0-0-0-3 ▪┄┄▪ PE4:Gi0-0-0-1
17:10:51 INFO Running postdeploy actions for Cisco XRd 'RR1' node
17:10:51 INFO Created link: PE3:Gi0-0-0-2 ▪┄┄▪ PE5:Gi0-0-0-0
17:10:51 INFO Running postdeploy actions for Cisco XRd 'PE5' node
17:10:51 INFO Created link: PE4:Gi0-0-0-3 ▪┄┄▪ RR1:Gi0-0-0-0
17:10:51 INFO Running postdeploy actions for Cisco XRd 'PE4' node
17:10:51 INFO Created link: PE4:Gi0-0-0-2 ▪┄┄▪ PE6:Gi0-0-0-0
17:10:51 INFO Created link: PE5:Gi0-0-0-1 ▪┄┄▪ PE6:Gi0-0-0-1
17:10:51 INFO Running postdeploy actions for Cisco XRd 'PE6' node
17:10:51 INFO Adding host entries path=/etc/hosts
17:10:51 INFO Adding SSH config for nodes path=/etc/ssh/ssh_config.d/clab-ContainerLab.conf
╭───────────────────────┬─────────────────────────────────┬─────────┬─────────────────╮
│ Name │ Kind/Image │ State │ IPv4/6 Address │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-CE1 │ cisco_iol │ running │ 192.168.121.101 │
│ │ vrnetlab/cisco_iol:17.12.01 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-PE1 │ cisco_xrd │ running │ 192.168.121.102 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-PE2 │ cisco_xrd │ running │ 192.168.121.103 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-PE3 │ cisco_xrd │ running │ 192.168.121.104 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-PE4 │ cisco_xrd │ running │ 192.168.121.105 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-PE5 │ cisco_xrd │ running │ 192.168.121.106 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-PE6 │ cisco_xrd │ running │ 192.168.121.107 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
├───────────────────────┼─────────────────────────────────┼─────────┼─────────────────┤
│ clab-ContainerLab-RR1 │ cisco_xrd │ running │ 192.168.121.108 │
│ │ ios-xr/xrd-control-plane:7.11.2 │ │ N/A │
╰───────────────────────┴─────────────────────────────────┴─────────┴─────────────────╯

┌──────────────────────────────────────────────────────────────────────────────────┐
│ DEPLOYING initial device configurations │
└──────────────────────────────────────────────────────────────────────────────────┘
[WARNING]: Could not match supplied host pattern, ignoring: unprovisioned

PLAY [Deploy initial device configuration] *****************************************************************************
[WARNING]: Found variable using reserved name: hosts

TASK [Set variables that cannot be set with VARS] **********************************************************************
ok: [CE1]
ok: [PE3]
ok: [PE2]
ok: [PE1]
ok: [PE4]
ok: [PE5]
ok: [PE6]
ok: [RR1]

TASK [Generic readiness tests] *****************************************************************************************
included: /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/tasks/readiness-check/ssh.yml for CE1, PE1, PE2, PE3, PE4, PE5, PE6, RR1 => (item=ssh)

TASK [Check if 'sshpass' is installed] *********************************************************************************
ok: [CE1 -> localhost]

TASK [Check for 'timeout' command] *************************************************************************************
ok: [CE1 -> localhost]
ok: [PE3 -> localhost]
ok: [PE2 -> localhost]
ok: [PE1 -> localhost]
ok: [PE5 -> localhost]
ok: [PE6 -> localhost]
ok: [PE4 -> localhost]
ok: [RR1 -> localhost]

TASK [Execute local ssh command to check ssh readiness] ****************************************************************
FAILED - RETRYING: [PE4 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [PE6 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [PE5 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [RR1 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [PE1 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [PE3 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [PE2 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [CE1 -> localhost]: Execute local ssh command to check ssh readiness (20 retries left).
FAILED - RETRYING: [PE3 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
FAILED - RETRYING: [PE2 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
FAILED - RETRYING: [PE1 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
FAILED - RETRYING: [PE4 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
FAILED - RETRYING: [PE6 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
FAILED - RETRYING: [PE5 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
FAILED - RETRYING: [RR1 -> localhost]: Execute local ssh command to check ssh readiness (19 retries left).
ok: [CE1 -> localhost]
FAILED - RETRYING: [PE3 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [PE1 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [PE2 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [PE1 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
FAILED - RETRYING: [PE2 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
FAILED - RETRYING: [PE3 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
FAILED - RETRYING: [PE4 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [PE6 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [PE5 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [RR1 -> localhost]: Execute local ssh command to check ssh readiness (18 retries left).
FAILED - RETRYING: [PE1 -> localhost]: Execute local ssh command to check ssh readiness (16 retries left).
FAILED - RETRYING: [PE2 -> localhost]: Execute local ssh command to check ssh readiness (16 retries left).
FAILED - RETRYING: [PE3 -> localhost]: Execute local ssh command to check ssh readiness (16 retries left).
ok: [PE3 -> localhost]
ok: [PE1 -> localhost]
ok: [PE2 -> localhost]
FAILED - RETRYING: [PE4 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
FAILED - RETRYING: [PE6 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
FAILED - RETRYING: [PE5 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
FAILED - RETRYING: [RR1 -> localhost]: Execute local ssh command to check ssh readiness (17 retries left).
ok: [PE4 -> localhost]
ok: [PE6 -> localhost]
ok: [PE5 -> localhost]
ok: [RR1 -> localhost]

TASK [Confirm CE1 SSH server works] ************************************************************************************
ok: [CE1] =>
msg: Node CE1 is ready.
ok: [PE1] =>
msg: Node PE1 is ready.
ok: [PE2] =>
msg: Node PE2 is ready.
ok: [PE3] =>
msg: Node PE3 is ready.
ok: [PE4] =>
msg: Node PE4 is ready.
ok: [PE5] =>
msg: Node PE5 is ready.
ok: [PE6] =>
msg: Node PE6 is ready.
ok: [RR1] =>
msg: Node RR1 is ready.

TASK [Find device readiness script] ************************************************************************************
ok: [CE1]
ok: [PE2]
ok: [PE1]
ok: [PE3]
ok: [PE4]
ok: [PE5]
ok: [PE6]
ok: [RR1]
`

After that there was a config conflict on PE1:

`TASK [ios_config: deploying initial from /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/templates/initial/ios.j2] ***
[WARNING]: To ensure idempotency and correct diff the input configuration lines should be similar to how they appear if
present in the running configuration on device including the indentation
changed: [CE1]

TASK [iosxr_config: deploying initial from /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/templates/initial/iosxr.j2] ***
changed: [RR1]
changed: [PE6]
changed: [PE5]
changed: [PE3]
fatal: [PE1]: FAILED! => changed=false
msg: |-
commit comment configured by iosxr_config

% Failed to commit .. Another configuration session
had a lock on the running configuration.
If you still want to commit your changes,
you may try the 'commit' command again.
RP/0/RP0/CPU0:PE1(config-if)#

changed: [PE2]
changed: [PE4]
`
...

PLAY RECAP *************************************************************************************************************
CE1                        : ok=26   changed=2    unreachable=0    failed=0    skipped=13   rescued=0    ignored=0   
PE1                        : ok=15   changed=0    unreachable=0    failed=1    skipped=6    rescued=0    ignored=0   
PE2                        : ok=29   changed=3    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
PE3                        : ok=29   changed=3    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
PE4                        : ok=29   changed=3    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
PE5                        : ok=29   changed=3    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
PE6                        : ok=29   changed=3    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
RR1                        : ok=29   changed=3    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
Error executing ansible-playbook /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/initial-config.ansible:
  Command '['ansible-playbook', '/home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/initial-config.ansible']' returned non-zero exit status 2.
[FATAL]   Executing Ansible playbook /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/initial-config.ansible failed
Error executing netlab initial --no-message:
  Command '['netlab', 'initial', '--no-message']' returned non-zero exit status 1.
[FATAL]   netlab up: netlab initial failed, aborting...

after issuing netlab initiate, it was corrected:
`TASK [iosxr_config: deploying ospf from /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/templates/ospf/iosxr.j2] ***
ok: [PE2]
ok: [PE4]
ok: [PE3]
ok: [PE6]
ok: [RR1]
ok: [PE5]
changed: [PE1]

...

TASK [Deploy bgp configuration] ****************************************************************************************
included: /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/tasks/deploy-config/ios.yml for CE1
included: /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/tasks/deploy-config/iosxr.yml for PE1, PE2, PE3, PE4, PE5, PE6, RR1

TASK [ios_config: deploying bgp from /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/templates/bgp/ios.j2] ***
changed: [CE1]

TASK [iosxr_config: deploying bgp from /home/fred/mylab/lib/python3.12/site-packages/netsim/ansible/templates/bgp/iosxr.j2] ***
ok: [PE2]
ok: [PE5]
ok: [RR1]
ok: [PE6]
ok: [PE3]
ok: [PE4]
changed: [PE1]

PLAY [Deploy custom deployment templates] ******************************************************************************
skipping: no hosts matched

PLAY RECAP *************************************************************************************************************
CE1 : ok=26 changed=2 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
PE1 : ok=29 changed=3 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
PE2 : ok=29 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
PE3 : ok=29 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
PE4 : ok=29 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
PE5 : ok=29 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
PE6 : ok=29 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
RR1 : ok=29 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 `

so yes, in summary netlab defaults devices.iosxr.clab.group_vars.netlab_ready=[ssh] helped a lot.
Thanks!

[ipspace]

Tried it, it helps a lot!

Thank you! Added in 928e236, will be part of the January release

After that there was a config conflict on PE1:

No idea what that could be. Ansible playbook is definitely serializing access to a single node. Maybe it's an internal initialization process that hasn't completed yet?