We are proud to announce the 17.2 release of Topicus KeyHub. This release takes our high availability solution out of beta and into production. We also continued our efforts to further strengthen our application and implemented several new defensive measures. We therefore strongly recommend our users to upgrade to 17.2, after carefully reading these release notes. As usual, a number of smaller improvements have been made and several issues have been fixed.

Important notice: changes to SSH keys

TKH-1632 Topicus KeyHub 17.2 has a much stricter configuration of OpenSSH sshd. Many algorithms and key types that were deemed not strong enough are now disabled. Although most of these were unlikely to be used anymore and will therefore not cause any issues, the removal of support for ssh-rsa keys, both for the host key and for key-based authentication, will likely require some changes. The personal ssh keys that can be uploaded under your profile are not affected.
Be sure to check the authentication used for retrieval of the backups with the keyhub-backup user!

Host key verification

Because a different host key will now be used for verification, you are likely to receive the warning below when trying to connect to the Topicus KeyHub VM via SSH. This warning is to be expected (the host key was renewed due to the rsa algorithm being disabled) and you can trust the new key of type ED25519. Follow the instructions to remove the old key and accept the new one.
(Note that the key fingerprint shown by your installation will be different from the example below, as every installation has a unique key.)

user@ws:~$ ssh -p 50022 keyhub@keyhub.local
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:K2w9lyRywaA1r0Vi1pRlnpCkz0vBr9gev0aRXOj+8pc.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:291
  remove with:
  ssh-keygen -f "/home/user/.ssh/known_hosts" -R "[keyhub.local]:50022"
ED25519 host key for [keyhub.local]:50022 has changed and you have requested strict checking.
Host key verification failed.

Key-based authentication

If you previously setup key-based authentication for SSH on the KeyHub VM using a ssh-rsa key, this key will no longer be accepted. The use of ED25519 is advised for your new key. We recommend to generate a new key before updating to 17.2. Earlier versions already accept ED25519-keys. Replacing the key before the upgrade will prevent a possible lockout. Generate a new key with the following command:

user@ws:~$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/user/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/user/.ssh/id_ed25519
Your public key has been saved in /home/user/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:ziTj7P5b4moCKrlVfR+vAOZG4t3Sw+DgFHyuA6ykGJc user@ws
The key's randomart image is:
+--[ED25519 256]--+
|                 |
|   .             |
|    o .          |
| . . =           |
|..E = X S .      |
|++.* % & . o     |
|+o..= O X o .    |
|+.  .+.o = .     |
|o.   ++o+..      |
+----[SHA256]-----+

Important notice: Several security improvements

In our continuing quest for security improvements, we've had the security of our appliance tested in a combination penetration test/configuration review. The following findings were corrected in this release. Feel free to contact us for more information on this test.

  • TKH-1619 The keys for linking nodes in a cluster were generated at an incorrect moment. This could cause multiple machines to share the same keys. All previously generated keys will be invalidated and replaced by stronger ones. This issue only existed when running Topicus KeyHub in a cluster, which was still beta prior to this version and not recommended for production use.
  • TKH-1625 The docker container for the internal DNS-proxy could write in /etc on the host. As access to /etc is only needed when Topicus KeyHub uses DHCP for its primary NIC and it only needs read access, this access has been restricted accordingly.
  • TKH-1626 The full web session identifiers were written to the logs. This could allow someone with access to the logs to take over sessions if that attacker could also use the same IP address. Session identifiers are now truncated to prevent this.
  • TKH-1627 The tkh-scriptengine docker image was rebased to extend from a much smaller and more recent alpine:3.13 image. The tkh-dns container was updated to the same base.
  • TKH-1631 All docker containers are now started with  no-new-privileges preventing any privilege escalation in the case of breach of the software running in the container.
  • TKH-1633 The internal HTTP reverse proxy now handles Host headers more reliable, making it much harder to spoof the host in a HTTP request.

Important notice: Issues with installing OS updates

TKH-1638 Topicus KeyHub installations of version 17.1 and prior may face issues installing OS updates due to an unforeseen change in the way SaltStack handles the update to 3002.5. This issue manifests itself with a recovery after a timeout during the installation of OS updates. The upgrade process for SaltStack was fixed to account for this change in 17.2.

High availability

TKH-1580 TKH-1588 TKH-1589 TKH-1592 TKH-1594 With 17.0 we released the first iteration of high availability for Topicus KeyHub. This release lifts the beta status of this feature. Please refer to the documentation for the prerequisites and instructions on how to setup a cluster. If you configure your load balancer to regularly poll the readiness endpoint of the appliance, you can even upgrade with minimal downtime.

Note that running KeyHub in a cluster requires static ip addresses and therefore running a cluster on Azure is currently unsupported.

Small improvements

The following smaller improvements and bug fixes were made:

  • TKH-1563 Automated regression tests were added for performing full version upgrades.
  • TKH-1579 The best practice guides were updated, along with our guide for installing Topicus KeyHub from the Azure Marketplace.
  • TKH-1582 Some textual changes were made to the screen for a forced password change.
  • TKH-1585 A race condition was fixed in the background polling during 2FA that could cause warnings in the browser log.
  • TKH-1587 The application server was upgraded to WildFly 22.
  • TKH-1590 A race condition was fixed between logout and lazy loading of some components that could cause those components to keep reloading.
  • TKH-1591 We improved the usability of the dashboard for the visually impaired.
  • TKH-1595 OpenAPI documentation was added to resources related to provisioning.
  • TKH-1612 The token revocation endpoint now handles CORS requests correctly, allowing Javascript clients to use it.
  • TKH-1614 The logs of the docker containers are now restricted in size to prevent the docker volume from filling up when a container outputs massive amounts of logging in exceptional circumstances.
  • TKH-1615 The group property for a launchpad tile is now required, fixing an error when trying to save it without a group specified.
  • TKH-1616 The add button for launchpad tiles is now only shown to users that actually have permission to add tiles.
  • TKH-1617 The web application now correctly handles updates to deleted items, such as trying to save a vault record that was deleted by another user.
  • TKH-1618 A regression since 17.1 was fixed that prevented shared vault records from being automatically removed after their availability expired.
  • TKH-1620 A description was added to various properties of vault records in an effort to better explain their use.
  • TKH-1621 A regression since 17.1 was fixed that stopped Let's Encrypt certificates from being renewed.
  • TKH-1622 An error was fixed when a KeyHub administrator would try to recover a deleted vault to a vault that does not have a recovery key.
  • TKH-1637 A problem in Docker could sometimes cause backups to be corrupted. Backups are now created in a different way to circumvent this issue.