Skip to content

Update systemd from 241 to 242 #1072

Merged
merged 7 commits into from
Apr 18, 2019
Merged

Conversation

pmenzel
Copy link
Collaborator

@pmenzel pmenzel commented Apr 12, 2019

No description provided.

pmenzel and others added 6 commits April 3, 2019 13:48
Two issues concerning us are fixed with this update.

> * systemd-journald and systemd-journal-remote reject entries which
>   contain too many fields (CVE-2018-16865) and set limits on the
>   process' command line length (CVE-2018-16864).

In German:

> Wir haben scheinabr ein neues Problem mit ssh. Ab und zu failed eine ssh mal ohne Fehlermeldung.
> Im Log des Servers findet man "Failed to create session: Start job for unit user-0.slice failed with 'canceled'"
>
> Das ist für die Fehler beim archiver.pl verantwortlich (keine Fehlermeldung, aber .tmp file noch da) und auch für manche nachtwaechter-Fehlalarme und wer weiß, für was noch alles.
>
> Möglicherweise eine Kollision, wenn eine ssh-Verbindung aufgebaut wird, während eine andere gerade abgebaut wird. Vielleicht wird die neue Verbindung im tear-down der alten Session mit abgeräumt?
>
> ```
> 2019-04-01T11:03:20+02:00 tldr sshd[2236]: Accepted publickey for root from 141.14.28.170 port 57178 ssh2: RSA SHA256:FYp+0igZejHePYAljVxfzXIcV4I2cicm+Atk24b4cEg
> 2019-04-01T11:03:20+02:00 tldr sshd[2236]: pam_unix(sshd:session): session opened for user root by (uid=0)
> 2019-04-01T11:03:21+02:00 tldr sshd[2236]: Received disconnect from 141.14.28.170 port 57178:11: disconnected by user
> 2019-04-01T11:03:21+02:00 tldr sshd[2236]: Disconnected from user root 141.14.28.170 port 57178
> 2019-04-01T11:03:21+02:00 tldr sshd[2236]: pam_unix(sshd:session): session closed for user root
> ```
>
> ```
> 2019-04-01T11:03:21+02:00 tldr sshd[2248]: Accepted publickey for root from 141.14.28.170 port 57180 ssh2: RSA SHA256:FYp+0igZejHePYAljVxfzXIcV4I2cicm+Atk24b4cEg
> 2019-04-01T11:03:21+02:00 tldr sshd[2248]: pam_unix(sshd:session): session opened for user root by (uid=0)
> 2019-04-01T11:03:21+02:00 tldr sshd[2248]: pam_systemd(sshd:session): Failed to create session: Start job for unit user-0.slice failed with 'canceled'
> 2019-04-01T11:03:21+02:00 tldr sshd[2248]: error: PAM: pam_open_session(): System error
> 2019-04-01T11:03:21+02:00 tldr sshd[2248]: Received disconnect from 141.14.28.170 port 57180:11: disconnected by user
> 2019-04-01T11:03:21+02:00 tldr sshd[2248]: Disconnected from user root 141.14.28.170 port 57180
> ```

The change-log is available [online].

[1]: https://github.com/systemd/systemd/blob/master/NEWS
We dont feel safe with the new hard limit of 524288, so change that back
to 4096 as before.

Do this in system.conf, because to modify the compiletime default, we'd
need to patch the hardcoded value in meson.build, there is no configure
option:

    conf.set('HIGH_RLIMIT_NOFILE',          512*1024)
We dont't feel safe with the limits raised to the maximum, so restore
old limits.
[Change-log][1]:

> CHANGES WITH 242:
>
> * In .link files, MACAddressPolicy=persistent (the default) is changed
>   to cover more devices. For devices like bridges, tun, tap, bond, and
>   similar interfaces that do not have other identifying information,
>   the interface name is used as the basis for persistent seed for MAC
>   and IPv4LL addresses. The way that devices that were handled
>   previously is not changed, and this change is about covering more
>   devices then previously by the "persistent" policy.
>
>   MACAddressPolicy=random may be used to force randomized MACs and
>   IPv4LL addresses for a device if desired.
>
>   Hint: the log output from udev (at debug level) was enhanced to
>   clarify what policy is followed and which attributes are used.
>   `SYSTEMD_LOG_LEVEL=debug udevadm test-builtin net_setup_link /sys/class/net/<name>`
>   may be used to view this.
>
> * The .device units generated by systemd-fstab-generator and other
>   generators do not automatically pull in the corresponding .mount unit
>   as a Wants= dependency. This means that simply plugging in the device
>   will not cause the mount unit to be started automatically. But please
>   note that the mount unit may be started for other reasons, in
>   particular if it is part of local-fs.target, and any unit which
>   (transitively) depends on local-fs.target is started.
>
> * networkctl list/status/lldp now accept globbing wildcards for network
>   interface names to match against all existing interfaces.
>
> * The $PIDFILE environment variable is set to point the absolute path
>   configured with PIDFile= for processes of that service.
>
> * The fallback DNS server list was augmented with Cloudflare public DNS
>   servers. Use `-Ddns-servers=` to set a different fallback.
>
> * A new special target usb-gadget.target will be started automatically
>   when a USB Device Controller is detected (which means that the system
>   is a USB peripheral).
>
> * A new unit setting CPUQuotaPeriodSec= assigns the time period
>   relatively to which the CPU time quota specified by CPUQuota= is
>   measured.
>
> * A new unit setting ProtectHostname= may be used to prevent services
>   from modifying hostname information (even if they otherwise would
>   have privileges to do so).
>
> * A new unit setting NetworkNamespacePath= may be used to specify a
>   namespace for service or socket units through a path referring to a
>   Linux network namespace pseudo-file.
>
> * The PrivateNetwork= setting and JoinsNamespaceOf= dependencies now
>   have an effect on .socket units: when used the listening socket is
>   created within the configured network namespace instead of the host
>   namespace.
>
> * ExecStart= command lines in unit files may now be prefixed with ':'
>   in which case environment variable substitution is
>   disabled. (Supported for the other ExecXYZ= settings, too.)
>
> * .timer units gained two new boolean settings OnClockChange= and
>   OnTimezoneChange= which may be used to also trigger a unit when the
>   system clock is changed or the local timezone is
>   modified. systemd-run has been updated to make these options easily
>   accessible from the command line for transient timers.
>
> * Two new conditions for units have been added: ConditionMemory= may be
>   used to conditionalize a unit based on installed system
>   RAM. ConditionCPUs= may be used to conditionalize a unit based on
>   installed CPU cores.
>
> * The @default system call filter group understood by SystemCallFilter=
>   has been updated to include the new rseq() system call introduced in
>   kernel 4.15.
>
> * A new time-set.target has been added that indicates that the system
>   time has been set from a local source (possibly imprecise). The
>   existing time-sync.target is stronger and indicates that the time has
>   been synchronized with a precise external source. Services where
>   approximate time is sufficient should use the new target.
>
> * "systemctl start" (and related commands) learnt a new
>   --show-transaction option. If specified brief information about all
>   jobs queued because of the requested operation is shown.
>
> * systemd-networkd recognizes a new operation state 'enslaved', used
>   (instead of 'degraded' or 'carrier') for interfaces which form a
>   bridge, bond, or similar, and an new 'degraded-carrier' operational
>   state used for the bond or bridge master interface when one of the
>   enslaved devices is not operational.
>
> * .network files learnt the new IgnoreCarrierLoss= option for leaving
>   networks configured even if the carrier is lost.
>
> * The RequiredForOnline= setting in .network files may now specify a
>   minimum operational state required for the interface to be considered
>   "online" by systemd-networkd-wait-online. Related to this
>   systemd-networkd-wait-online gained a new option --operational-state=
>   to configure the same, and its --interface= option was updated to
>   optionally also take an operational state specific for an interface.
>
> * systemd-networkd-wait-online gained a new setting --any for waiting
>   for only one of the requested interfaces instead of all of them.
>
> * systemd-networkd now implements L2TP tunnels.
>
> * Two new .network settings UseAutonomousPrefix= and UseOnLinkPrefix=
>   may be used to cause autonomous and onlink prefixes received in IPv6
>   Router Advertisements to be ignored.
>
> * New MulticastFlood=, NeighborSuppression=, and Learning= .network
>   file settings may be used to tweak bridge behaviour.
>
> * The new TripleSampling= option in .network files may be used to
>   configure CAN triple sampling.
>
> * A new .netdev settings PrivateKeyFile= and PresharedKeyFile= may be
>   used to point to private or preshared key for a WireGuard interface.
>
> * /etc/crypttab now supports the same-cpu-crypt and
>   submit-from-crypt-cpus options to tweak encryption work scheduling
>   details.
>
> * systemd-tmpfiles will now take a BSD file lock before operating on a
>   contents of directory. This may be used to temporarily exclude
>   directories from aging by taking the same lock (useful for example
>   when extracting a tarball into /tmp or /var/tmp as a privileged user,
>   which might create files with really old timestamps, which
>   nevertheless should not be deleted). For further details, see:
>
>   https://systemd.io/TEMPORARY_DIRECTORIES
>
> * systemd-tmpfiles' h line type gained support for the
>   FS_PROJINHERIT_FL ('P') file attribute (introduced in kernel 4.5),
>   controlling project quota inheritance.
>
> * sd-boot and bootctl now implement support for an Extended Boot Loader
>   (XBOOTLDR) partition, that is intended to be mounted to /boot, in
>   addition to the ESP partition mounted to /efi or /boot/efi.
>   Configuration file fragments, kernels, initrds and other EFI images
>   to boot will be loaded from both the ESP and XBOOTLDR partitions.
>   The XBOOTLDR partition was previously described by the Boot Loader
>   Specification, but implementation was missing in sd-boot. Support for
>   this concept allows using the sd-boot boot loader in more
>   conservative scenarios where the boot loader itself is placed in the
>   ESP but the kernels to boot (and their metadata) in a separate
>   partition.
>
> * A system may now be booted with systemd.volatile=overlay on the
>   kernel command line, which causes the root file system to be set up
>   an overlayfs mount combining the root-only root directory with a
>   writable tmpfs. In this setup, the underlying root device is not
>   modified, and any changes are lost at reboot.
>
> * Similar, systemd-nspawn can now boot containers with a volatile
>   overlayfs root with the new --volatile=overlay switch.
>
> * systemd-nspawn can now consume OCI runtime bundles using a new
>   --oci-bundle= option. This implementation is fully usable, with most
>   features in the specification implemented, but since this a lot of
>   new code and functionality, this feature should most likely not
>   be used in production yet.
>
> * systemd-nspawn now supports various options described by the OCI
>   runtime specification on the command-line and in .nspawn files:
>   --inaccessible=/Inaccessible= may be used to mask parts of the file
>   system tree, --console=/--pipe may be used to configure how standard
>   input, output, and error are set up.
>
> * busctl learned the `emit` verb to generate D-Bus signals.
>
> * systemd-analyze cat-config may be used to gather and display
>   configuration spread over multiple files, for example system and user
>   presets, tmpfiles.d, sysusers.d, udev rules, etc.
>
> * systemd-analyze calendar now takes an optional new parameter
>   --iterations= which may be used to show a maximum number of iterations
>   the specified expression will elapse next.
>
> * The sd-bus C API gained support for naming method parameters in the
>   introspection data.
>
> * systemd-logind gained D-Bus APIs to specify the "reboot parameter"
>   the reboot() system call expects.
>
> * journalctl learnt a new --cursor-file= option that points to a file
>   from which a cursor should be loaded in the beginning and to which
>   the updated cursor should be stored at the end.
>
> * ACRN hypervisor and Windows Subsystem for Linux (WSL) are now
>   detected by systemd-detect-virt (and may also be used in
>   ConditionVirtualization=).
>
> * The behaviour of systemd-logind may now be modified with environment
>   variables $SYSTEMD_REBOOT_TO_FIRMWARE_SETUP,
>   $SYSTEMD_REBOOT_TO_BOOT_LOADER_MENU, and
>   $SYSTEMD_REBOOT_TO_BOOT_LOADER_ENTRY. They cause logind to either
>   skip the relevant operation completely (when set to false), or to
>   create a flag file in /run/systemd (when set to true), instead of
>   actually commencing the real operation when requested. The presence
>   of /run/systemd/reboot-to-firmware-setup,
>   /run/systemd/reboot-to-boot-loader-menu, and
>   /run/systemd/reboot-to-boot-loader-entry, may be used by alternative
>   boot loader implementations to replace some steps logind performs
>   during reboot with their own operations.
>
> * systemctl can be used to request a reboot into the boot loader menu
>   or a specific boot loader entry with the new --boot-load-menu= and
>   --boot-loader-entry= options to a reboot command. (This requires a
>   boot loader that supports this, for example sd-boot.)
>
> * kernel-install will no longer unconditionally create the output
>   directory (e.g. /efi/<machine-id>/<kernel-version>) for boot loader
>   snippets, but will do only if the machine-specific parent directory
>   (i.e. /efi/<machine-id>/) already exists. bootctl has been modified
>   to create this parent directory during sd-boot installation.
>
>   This makes it easier to use kernel-install with plugins which support
>   a different layout of the bootloader partitions (for example grub2).
>
> * During package installation (with `ninja install`), we would create
>   symlinks for systemd-networkd.service, systemd-networkd.socket,
>   systemd-resolved.service, remote-cryptsetup.target, remote-fs.target,
>   systemd-networkd-wait-online.service, and systemd-timesyncd.service
>   in /etc, as if `systemctl enable` was called for those units, to make
>   the system usable immediately after installation. Now this is not
>   done anymore, and instead calling `systemctl preset-all` is
>   recommended after the first installation of systemd.
>
> * A new boolean sandboxing option RestrictSUIDSGID= has been added that
>   is built on seccomp. When turned on creation of SUID/SGID files is
>   prohibited.
>
> * The NoNewPrivileges= and the new RestrictSUIDSGID= options are now
>   implied if DynamicUser= is turned on for a service. This hardens
>   these services, so that they neither can benefit from nor create
>   SUID/SGID executables. This is a minor compatibility breakage, given
>   that when DynamicUser= was first introduced SUID/SGID behaviour was
>   unaffected. However, the security benefit of these two options is
>   substantial, and the setting is still relatively new, hence we opted
>   to make it mandatory for services with dynamic users.

[1]: https://github.com/systemd/systemd/blob/master/NEWS
@donald
Copy link
Collaborator

donald commented Apr 15, 2019

bee update systemd-242-0
systemctl daemon-reexec
systemctl restart systemd-udevd
systemctl restart systemd-journald
systemctl restart systemd-resolved
systemctl restart systemd-timesyncd
systemctl restart systemd-logind
root@theinternet:~# bee update systemd-242-0
installing /src/mariux/beeroot/packages/systemd-242-0.x86_64.bee.tar.bz2 ..
Detected autofs mount point /home during canonicalization of /home.
Skipping /home
removing systemd-241-0.x86_64 ..
rmdir: failed to remove ‘/etc/systemd/system/sockets.target.wants’: Directory not empty

harmless but
root@theinternet:~# systemctl restart systemd-timesyncd
Job for systemd-timesyncd.service failed because the control process exited with error code.
See "systemctl status systemd-timesyncd.service" and "journalctl -xe" for details.

Apr 15 08:45:01 theinternet.molgen.mpg.de systemd[29166]: systemd-timesyncd.service: Failed to set up special execution directory in /var/lib: Not a directory
Apr 15 08:45:01 theinternet.molgen.mpg.de systemd[29166]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /lib/systemd/systemd-timesyncd: Not a directory



@donald
Copy link
Collaborator

donald commented Apr 15, 2019

systemd-timesync has been switch from DynamicUser=yes to DynamicUser=no. See http://0pointer.net/blog/dynamic-users-with-systemd.html for the dynamic user stuff. /var/lib/systemd/timesync is a symlink to /var/lib/private/systemd/timesync which is supposed to be owned by the dynamic user and /var/lib/private should prevent access by its restrictive mode. This is necessary, because the uid/gid might become obsolete when the service is no longer running. The service itself gets access by a separate mount namespace, so it doesn't need to access /var/lib/private.

So the correct upgrade procedure is to just remove the junk which resulted from DynamicUser=yes:

systemctl stop systemd-timesyncd
rm -r /var/lib/private/systemd/timesync /var/lib/systemd/timesync
systemctl start systemd-timesyncd

@donald
Copy link
Collaborator

donald commented Apr 15, 2019

After the update, theinternet can no longer be rebootet. Shutdown hangs after "Stopped NFS Daemon" or "Stopped Session c1 of user gmd".

@donald
Copy link
Collaborator

donald commented Apr 15, 2019

Update failed on sigusr2 : After bee update systemd-242-0 the following systemctl daemon-reexec gives "Access denied". Same error for any systemctl command.

@donald
Copy link
Collaborator

donald commented Apr 15, 2019

The upgrade from 240 to 241 doesn't work with systemctl daemon-reload:

root@theinternet:~# systemctl --version
systemd 239
+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
root@theinternet:~# bee update systemd-241-1
installing /src/mariux/beeroot/packages/systemd-241-1.x86_64.bee.tar.bz2 ..
Detected autofs mount point /home during canonicalization of /home.
Skipping /home
removing systemd-239-2.x86_64 ..
root@theinternet:~# systemctl daemon-reload
root@theinternet:~# systemctl --version
systemd 241 (241)
+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
root@theinternet:~# bee update systemd-242-0
installing /src/mariux/beeroot/packages/systemd-242-0.x86_64.bee.tar.bz2 ..
Detected autofs mount point /home during canonicalization of /home.
Skipping /home
removing systemd-241-1.x86_64 ..
rmdir: failed to remove ‘/etc/systemd/system/sockets.target.wants’: Directory not empty
root@theinternet:~# systemctl daemon-reload
Failed to reload daemon: Access denied

@donald
Copy link
Collaborator

donald commented Apr 15, 2019

New Update procedure (fixes the "Access denied" problem) :

#! /bin/sh

set -ve

cp /bin/systemctl /tmp/systemctl.239
cp /lib/systemd/libsystemd-shared-239.so /tmp/

bee update systemd-242-0
LD_LIBRARY_PATH=/tmp /tmp/systemctl.239 daemon-reexec
systemctl restart systemd-udevd
systemctl restart systemd-journald
systemctl restart systemd-resolved
systemctl stop systemd-timesyncd
rm -r /var/lib/private/systemd/timesync /var/lib/systemd/timesync
systemctl start systemd-timesyncd
systemctl restart systemd-logind

rm /tmp//systemctl.239 /tmp/libsystemd-shared-239.so

Reboot (with Ctrl-Alt-Del or shutdown -r now) still doesn't work after the upgrade. This is persistent - so even after after Sysctrl-forces reboot, the command don't work on the newly booted, updated system).

@donald
Copy link
Collaborator

donald commented Apr 17, 2019

Shutdown problem analyzed and reported upstream: https://github.com/systemd/systemd/issues/12335

Add patch to fix https://github.com/systemd/systemd/issues/12335
("systemd v242 may go into loop when ListenDatagram socket units are
stopped").
@donald donald merged commit 6bb50ee into master Apr 18, 2019
@donald
Copy link
Collaborator

donald commented Apr 18, 2019

I'll install that system by system and check the respoiveness of each machine before going to the next, so that the process can be aborted on the first failure.

For this reason, I've disabled the pdist cronjob on the distmaster. Will be re-enabled if the update has been done ( or undone ;-) ) on all systems.

@donald
Copy link
Collaborator

donald commented Apr 18, 2019

All done. dist re-enabled.

@pmenzel
Copy link
Collaborator Author

pmenzel commented Apr 18, 2019

Currently no getty is started on tty1. Reported upstream. We might need to activate some service, which is not activated during installation anymore.

@pmenzel
Copy link
Collaborator Author

pmenzel commented Apr 18, 2019

The old behavior was restored by manually enabling getty@tty1.service.

deinemuddah:~> sudo systemctl enable getty@tty1.service
Created symlink /etc/systemd/system/getty.target.wants/getty@tty1.service → /lib/systemd/system/getty@.service.
deinemuddah:~> sudo pdist push ALL --files /etc/systemd/system/getty.target.wants/getty@tty1.service --para 100

Sign in to join this conversation on GitHub.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants