Skip to content

Conversation

@donald
Copy link
Collaborator

@donald Donald Buczek (donald) commented Mar 14, 2018 •

After the upgrade of systemd from version 27 to version 238, we had the
problem, that mxnetctl start from network.service ("Network
Connectivity") was run before all network interfaces were available and
thus didn't rename them.

network.service is configured with DefaultDependencies=no to prevent
problems with nfs umounts during shutdown. But even when
network.services was returned to DefaultDependencies=yes and waited
for basic.target ("Basic System") the problem persisted:

root@nsa:/etc/systemd/system (master)#  journalctl --system --boot=-0|egrep 'mxnetctl|Gigabit|renamed|Basic System|Network Connectivity'
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10)
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
Mar 14 08:36:19 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:19 nsa.molgen.mpg.de systemd[1]: Reached target Basic System.
Mar 14 08:36:19 nsa.molgen.mpg.de systemd[1]: Starting Network Connectivity...
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth0 to net00
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: igb 0000:05:00.0 net00: renamed from eth0
Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth1 to net01
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: igb 0000:06:00.0 net01: renamed from eth1
Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth2 to net02
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.0 net02: renamed from eth2
Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth3 to net03
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.1 net03: renamed from eth3
Mar 14 08:36:20 nsa.molgen.mpg.de systemd[1]: Started Network Connectivity.
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:84:00.0: Intel(R) 10 Gigabit Network Connection
Mar 14 08:36:21 nsa.molgen.mpg.de kernel: ixgbe 0000:84:00.1: Intel(R) 10 Gigabit Network Connection

Similar problems are possible with other startup services (e.g.
mxmount) as they are expecting the local devices to be available.

So we want to start and wait for systemd-udev-settle.service ("udev Wait for
Complete Device Initialization") before our services. We do this by
adding our own target with Before=basic.target and
After=systemd-udev-settle.service. The normal services will wait for
basic.target per default. Because network.service has
DefaultDependencies=no, we add the order dependency explicitly. Because
the unit file of network.service is system specific and not distributed
by the distmaster, we add this order dependency in the unit file of the
new target.

Add new target unit udev-settled.target.

@pmenzel
Copy link
Contributor

In the commit and merge/pull request message the version should be changed from 229 to 238.

The rest looks fine.

Somebody also added systemd-network-after-root-writable-229-0.bee in the past by the way.

After the upgrade of systemd from version 27 to version 238, we had the
problem, that `mxnetctl start` from network.service ("Network
Connectivity") was run before all network interfaces were available and
thus didn't rename them.

network.service is configured with `DefaultDependencies=no` to prevent
problems with nfs umounts during shutdown. But even when
network.services was returned to `DefaultDependencies=yes` and waited
for basic.target ("Basic System") the problem persisted:

    root@nsa:/etc/systemd/system (master)#  journalctl --system --boot=-0|egrep 'mxnetctl|Gigabit|renamed|Basic System|Network Connectivity'
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10)
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
    Mar 14 08:36:19 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:19 nsa.molgen.mpg.de systemd[1]: Reached target Basic System.
    Mar 14 08:36:19 nsa.molgen.mpg.de systemd[1]: Starting Network Connectivity...
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth0 to net00
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: igb 0000:05:00.0 net00: renamed from eth0
    Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth1 to net01
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: igb 0000:06:00.0 net01: renamed from eth1
    Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth2 to net02
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.0 net02: renamed from eth2
    Mar 14 08:36:20 nsa.molgen.mpg.de mxnetctl[412]: rename eth3 to net03
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:02:00.1 net03: renamed from eth3
    Mar 14 08:36:20 nsa.molgen.mpg.de systemd[1]: Started Network Connectivity.
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:20 nsa.molgen.mpg.de kernel: ixgbe 0000:84:00.0: Intel(R) 10 Gigabit Network Connection
    Mar 14 08:36:21 nsa.molgen.mpg.de kernel: ixgbe 0000:84:00.1: Intel(R) 10 Gigabit Network Connection

Similar problems are possible with other startup services (e.g.
mxmount) as they are expecting the local devices to be available.

So we want to start and wait for systemd-udev-settle.service ("udev Wait for
Complete Device Initialization") before our services. We do this by
adding our own target with `Before=basic.target` and
`After=systemd-udev-settle.service`. The normal services will wait for
basic.target per default. Because network.service has
`DefaultDependencies=no`, we add the order dependency explicitly. Because
the unit file of network.service is system specific and not distributed
by the distmaster, we add this order dependency in the unit file of the
new target.

Add new target unit udev-settled.target.
@donald
Copy link
Collaborator Author

Donald Buczek (donald) commented Mar 14, 2018 •

I fixed the message.

Good you found systemd-network-after-root-writable-229-0.bee It did arrange network.service after udev-settle.service on systems with the old systemd, but it failed to do so on systems with the new systemd, because on systems with the new systemd, udev-settle.service is never started.

Only on the old systems, basic.target had an own 'Wants' dependency for udev-settle.service. This was provided by udev-164-2.x86_64 as a symlink /lib/systemd/system/basic.target.wants/udev-settle.service. So at least now we know where the difference originated.

I'll check whether remount-rootfs.service is still needed. If so, I might merge that into this unit, too.

@donald
Copy link
Collaborator Author

Hmm. remount-rootfs.service now is systemd-remount-fs.service. It seems to run long before basic.target is reached, although I can't figure out, what dependency path enforces this. Ignore for now until problem manifests.

Sign in to join this conversation on GitHub.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants