linuxmonk.ch

Testing oVirt 4.4-alpha on CentOS 8: Part 2 - Setup Self-Hosted Engine

Mon, 24 Feb 2020 02:02:34 +0100

This is the second part of my field report about installing the oVirt 4.4-alpha release on CentOS 8 in a hyperconverged setup. In the first part I was focusing on setting up the GlusterFS storage cluster and now I’m going to describe my experience of the self-hosted engine installation.

If you think about repeating this installation on your hardware please let me remind you: This software is currently in alpha status. This means there are likely still many bugs and rough edges and if you succeed to install it successfully there is no guarantee for updates not breaking everything again. Please don’t try this anywhere close to production systems or data. I won’t be able to assist you in any way if things turn out badly.

Cockpit Hosted-Engine Wizard

Before we can start installing the self-hosted engine, we need to install a few more packages:

# dnf install ovirt-engine-appliance vdsm-gluster

Similar to the GlusterFS setup, also the hosted engine setup can be done from the Cockpit Web interface:

The wizard is also here pretty self-explanatory. There are a few options missing in Web-UI compared to the commandline installer ( hosted-engine --deploy) e.g. you cannot customize the name of the libvirt domain which is called ‘HostedEngine’ by default. You must give the common details such as hostname, VM resources, some network settings, credentials for the VM and oVirt and that’s pretty much it:

Before you start deploying the VM there is also a quick summary of the settings and then an answer file will be generated. While the GlusterFS setup created a regular Ansible inventory the hosted engine setup has its own INI-format. It’s a useful feature that even when the deployment aborts, it can always be restarted from the Web interface without the need to fill in the form again and again. Indeed, I used this to my advantage a lot because it took me at least 20 attempts before the hosted engine VM was setup successfully.

Troubleshooting hosted-engine issues

Once VM deployment was running I found that the status output in the Cockpit Web interface heavily resembled Ansible output. It seems that a big part of the deployment code in the hosted-engine tool was re-implemented now using the ovirt-ansible-hosted-engine-setup Ansible roles in the background. If you’re familiar with Ansible this definitely simplifies troubleshooting and also allows a better understanding what is going on. Unfortunately there is still a layer of hosted-engine code above Ansible so that I couldn’t figure out, if it’s possible to run a playbook from the shell that would do the same setup.

Obviously it didn’t take long for an issue to pop-up:

The first error was that Ansible couldn’t connect to the hosted-engine VM that was freshly created from the oVirt appliance disk image. The error output in the Web interface is rather limited but also here a log file exists that can be found at a path like /var/log/ovirt-engine/setup/ovirt-engine-setup-20200218154608-rtt3b7.log. In the log file I found:

2020-02-18 15:32:45,938+0100 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Wait for the local VM kwargs
2020-02-18 15:35:52,816+0100 ERROR ansible failed {
    "ansible_host": "localhost",
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
    "ansible_result": {
        "_ansible_delegated_vars": {
            "ansible_host": "ovirt.oasis.home"
        },
        "_ansible_no_log": false,
        "changed": false,
        "elapsed": 185,
        "msg": "timed out waiting for ping module test success: Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this.  Please add this host's fingerprint to your know
n_hosts file to manage this host."
    },
    "ansible_task": "Wait for the local VM",
    "ansible_type": "task",
    "status": "FAILED",
    "task_duration": 187
}

A manual SSH login with the root account on the VM was possible after accepting the fingerprint. Maybe this is still a bug or I missed a setting somewhere, but the easiest way to solve this was to create a ~root/.ssh/config file on the hypervisor host with the following content. The hostname is the hosted-engine FQDN:

Host ovirt.oasis.home
    StrictHostKeyChecking accept-new

Each installation attempt will make sure that the previous host key is deleted from the known_hosts file so no need to worry about changing keys on multiple installation tries. The deployment could simply be restarted by pressing the “Prepare VM” button once again.

During the next run the connection to the hosted-engine VM succeeded and it nearly completed all of the setup task within the VM but then failed when trying to restart the ovirt-engine-dwhd service:

2020-02-18 15:48:45,963+0100 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.core.service service._closeup:52 Starting dwh service
2020-02-18 15:48:45,964+0100 DEBUG otopi.plugins.otopi.services.systemd systemd.state:170 starting service ovirt-engine-dwhd
2020-02-18 15:48:45,965+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'ovirt-engine-dwhd.service'), executable='None', cwd='None', env=None
2020-02-18 15:48:46,005+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'ovirt-engine-dwhd.service'), rc=1
2020-02-18 15:48:46,006+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-engine-dwhd.service') stdout:

2020-02-18 15:48:46,006+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-engine-dwhd.service') stderr:
Job for ovirt-engine-dwhd.service failed because the control process exited with error code. See "systemctl status ovirt-engine-dwhd.service" and "journalctl -xe" for details.

2020-02-18 15:48:46,007+0100 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-dwh/core/service.py", line 55, in _closeup
    state=True,
  File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 181, in state
    service=name,
RuntimeError: Failed to start service 'ovirt-engine-dwhd'
2020-02-18 15:48:46,008+0100 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed to start service 'ovirt-engine-dwhd'
2020-02-18 15:48:46,009+0100 DEBUG otopi.context context.dumpEnvironment:765 ENVIRONMENT DUMP - BEGIN
2020-02-18 15:48:46,010+0100 DEBUG otopi.context context.dumpEnvironment:775 ENV BASE/error=bool:'True'
2020-02-18 15:48:46,010+0100 DEBUG otopi.context context.dumpEnvironment:775 ENV BASE/exceptionInfo=list:'[(, RuntimeError("Failed to start service 'ovirt-engine-dwhd'",), )]'
2020-02-18 15:48:46,012+0100 DEBUG otopi.context context.dumpEnvironment:779 ENVIRONMENT DUMP - END

Fortunately I was able to login to the hosted-engine VM and found the following blunt error:

-- Unit ovirt-engine-dwhd.service has begun starting up.
Feb 18 15:48:46 ovirt.oasis.home systemd[30553]: Failed at step EXEC spawning /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.py: Permission denied
-- Subject: Process /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.py could not be executed

Indeed, the referenced script was not marked executable. Fixing it manually and restarting the service showed that this would succeed. But there is one problem. This change is not persisted. On the next deployment run, the hosted-engine VM will be deleted and re-created again. When searching for a nicer solution I found that this bug is actually already fixed in the latest release of ovirt-engine-dwh-4.4.0-1.el8.noarch.rpm but the appliance image ( ovirt-engine-appliance-4.4-20200212182535.1.el8.x86_64) was only including ovirt-engine-dwh-4.4.0-0.0.master.20200206083940.el7.noarch and there is no newer appliance image. That’s part of the experience when trying alpha releases but it’s not a blocker. Eventually I found that there is a directory where you can place an Ansible tasks file which will be executed in the hosted-engine VM before the setup is run. So I created the file hooks/enginevm_before_engine_setup/yum_update.yml in the /usr/share/ansible/roles/ovirt.hosted_engine_setup/ directory with the following content:

  ---
- name: Update all packages
  package:
  name: '*'
  state: latest

From then on each deployment attempt was first updating the packages including ‘ovirt-engine-dwh’ in the VM before the hosted-engine would continue to configure and restart the service.

The next issue was suddenly appearing when I tried to re-run the deployment. The Ansible code would fail early with an error that it cannot update the routing rules on the hypervisor:

2020-02-18 16:17:38,330+0100 DEBUG ansible on_any args  kwargs
2020-02-18 16:17:38,664+0100 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Add IPv4 outbo
und route rules'}
2020-02-18 16:17:38,664+0100 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Add IPv4 outbound route rules kwargs is_conditional:False
2020-02-18 16:17:38,665+0100 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Add IPv4 outbound route rules kwargs
2020-02-18 16:17:39,214+0100 DEBUG var changed: host "localhost" var "result" type "" value: "{
    "changed": true,
    "cmd": [
        "ip",
        "rule",
        "add",
        "from",
        "192.168.222.1/24",
        "priority",
        "101",
        "table",
        "main"
    ],
    "delta": "0:00:00.002805",
    "end": "2020-02-18 16:17:38.875350",
    "failed": true,
    "msg": "non-zero return code",
    "rc": 2,
    "start": "2020-02-18 16:17:38.872545",
    "stderr": "RTNETLINK answers: File exists",
    "stderr_lines": [
        "RTNETLINK answers: File exists"
    ],
    "stdout": "",
    "stdout_lines": []
}"

So I was checking the rules manually and yes, they were already there. I thought that’s an easy case, that must be a simple idempotency issue in the Ansible code. But when looking up the code there was already a condition in place that should prevent this case from happening. Even after multiple attempts to debug this code, I couldn’t find the reason why this check is failing. Eventually I found the GitHub pull request #96 where someone was already refactoring this code with a commit message “Hardening existing ruleset lookup”. So I forward-ported the patch to the release 1.0.35 which fixed the problem. The PR is already open for more than a year with no indication that it would be merged soon, so I still reported the issue in ovirt-ansible-hosted-engine-setup #289. I only found out about ovirt-hosted-engine-cleanup a few hours later, so with its help you can easily work-around this issue by cleaning up the installation before another retry.

Another though to debug but easy to fix issue popped up after the hosted-engine VM setup completed and the Ansible role was checking the oVirt events for errors:

2020-02-19 01:46:53,723+0100 ERROR ansible failed {
    "ansible_host": "localhost",
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
    "ansible_result": {
        "_ansible_no_log": false,
        "changed": false,
        "msg": "The host has been set in non_operational status, deployment errors:   code 4035: Gluster command [] failed on server .,    code 10802: VDSM loki.oasis.home command GlusterServersListVDS failed: The method does not exist or is not available: {'method': 'GlusterHost.list'},   fix accordingly and re-deploy."
    },
    "ansible_task": "Fail with error description",
    "ansible_type": "task",
    "status": "FAILED",
    "task_duration": 0
}

This error is not in the Ansible code anymore but the engine itself fails to query the GlusterFS status on the hypervisor. This is done via VDSM, a daemon that runs on each oVirt hypervisor and manages the hypervisor configuration and status. Maybe the VDSM log ( /var/log/vdsm/vdsm.log) reveals more insights:

2020-02-19 01:46:45,786+0100 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 3.33 seconds (__init__:312)
2020-02-19 01:46:45,981+0100 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call GlusterHost.list failed (error -32601) in 0.00 seconds (__init__:312)

Seems that regular RPC calls to VDSM are successful but only the GlusterFS query is failing. I tracked down the source code of this implementation and found that there is a CLI command that can be used to run this query:

# vdsm-client --gluster-enabled -h
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsmclient/client.py", line 276, in find_schema
    with_gluster=gluster_enabled)
  File "/usr/lib/python3.6/site-packages/vdsm/api/vdsmapi.py", line 156, in vdsm_api
    return Schema(schema_types, strict_mode, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/api/vdsmapi.py", line 142, in __init__
    with io.open(schema_type.path(), 'rb') as f:
  File "/usr/lib/python3.6/site-packages/vdsm/api/vdsmapi.py", line 95, in path
    ", ".join(potential_paths))
vdsm.api.vdsmapi.SchemaNotFound: Unable to find API schema file, tried: /usr/lib/python3.6/site-packages/vdsm/api/vdsm-api-gluster.pickle, /usr/lib/python3.6/site-packages/vdsm/api/../rpc/vdsm-api-gluster.pickle

Ah, that’s better. I love such error messages. Thanks to that it was not so hard to find, that I actually overlooked to install the vdsm-gluster package on the hypervisor.

That’s it after that the deployment completed successfully:

And finally a screenshot of the oVirt 4.4-alpha administration console. Yes, it works:

Conclusion

At the end the of the day most of the issues happened because I was not very familiar with the setup procedure and at the same time refused to follow any setup instructions for an older release. There were a minor bug with the ovirt-engine-dwh restart issue, that was already fixed upstream but didn’t made it yet into the hosted-engine appliance image. Something that is expected in an alpha release.

I also quickly setup some VMs to test the basic functionality of oVirt and couldn’t find any major issues so far. I guess most people using oVirt are much more experienced with it than me anyway, so there shouldn’t be any concerns in trying oVirt 4.4-alpha yourself. To me that was an interesting experience and I’m very happy about the Ansible integration that this project is pushing. It was also a nice experience to use Cockpit and I believe that’s definitely something that makes this product more appealing to setup and use for a wide range of IT professionals. As long as it can be done via command line too, I’ll be happy.

Testing oVirt 4.4-alpha on CentOS 8: Part 1 - Setup GlusterFS

Sun, 23 Feb 2020 20:03:17 +0100

For a while I had a oVirt server in a hyperconverged setup which means that the hypervisor was also running a GlusterFS storage server and that the oVirt management virtual machine was inside the oVirt cluster (self-hosted engine). On top of oVirt I was running a OKD cluster with a containerized GlusterFS cluster that could be used for persistent volumes by the container workload. All of this was running on a single hypervisor with a single SSD which unfortunately gave up on me after a few years in operation. Recently I stumbled upon the oVirt 4.4-alpha. Next to initial support for running it on CentOS 8 also a proper support for ignition that is used by (Fedora and Red Hat) CoreOS and therefore OpenShift/OKD 4 attracted my attention. Why not give it a try and see how far I come…? After a few hours of tinkering I succeeded the installation:

And now I’m going to describe what was necessary to do so. Not everything that I’ll mention is brand new. My past experience of setting up such a system is more or less based on the guide Up and Running with oVirt 4.0 and Gluster Storage that I was following a few years ago, so I’m also highlighting a few things that have changed since then.

If you think about repeating this installation on your hardware please let me remind you: This software is currently in alpha status. This means there are likely still many bugs and rough edges and if you succeed to install it successfully there is no guarantee for updates not breaking everything again. Please don’t try this anywhere close to production systems or data. I won’t be able to assist you in any way if things turn out badly.

I split this field report into two parts, the first one discussing the GlusterFS storage setup and the second one explaining my challenges when setting up the oVirt self-hosted-engine.

Hypervisor disk layout

I was using a minimal install of a CentOS 8 on a bare-metal server. Make sure you either have two disks or create a separate partition when installing CentOS so that the GlusterFS storage can life on its own block device. My disk layout looks something like this:

[root@loki ~]# lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                       8:0    0 931.5G  0 disk
├─sda1                    8:1    0   100M  0 part
├─sda2                    8:2    0   256M  0 part /boot
├─sda3                    8:3    0    45G  0 part
│ ├─vg_loki-slash       253:0    0    10G  0 lvm  /
│ ├─vg_loki-swap        253:1    0     4G  0 lvm  [SWAP]
│ ├─vg_loki-var         253:2    0     5G  0 lvm  /var
│ ├─vg_loki-home        253:3    0     2G  0 lvm  /home
│ └─vg_loki-log         253:4    0     2G  0 lvm  /var/log
└─sda4                    8:4    0   500G  0 part

The CentOS installation is placed on a LVM volume group ( vg_loki on /dev/sda3) with some individual volumes for dedicated mount points and then there is /dev/sda4 which is an empty disk partition that will be used by GlusterFS later. If you wonder how to do such a setup with the CentOS 8 installer… I don’t know. I tried for a moment to somehow configure this setup in the installer, but eventually I gave up, manually partitioned the disk and created the volume group and then used the pre-generated setup in the installer which perfectly detected what I was doing on the shell.

Install software requirements

First you need to enable the oVirt 4.4-alpha package repository by installing the corresponding release package:

# dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm

Recently a lot of effort was invested into incorporating the oVirt setup into the Cockpit Web interface and it’s now even the recommended installation method for the downstream Red Hat Virtualization (RHV). When I setup my previous hyperconverged oVirt 4.0 this wasn’t available back then so of course I’m going to try this. To setup Cockpit and the oVirt integration the following packages need to be installed:

# dnf install cockpit cockpit-ovirt-dashboard glusterfs-server

After logging into Cockpit that runs on the hypervisor host on port 9090 there is a dedicated oVirt tab with two entries:

If you continue with the hyperconverged setup, there now even is a dedicated option to install a single node only GlusterFS “cluster”!

This was a big positive surprise to me because the previously used gdeploy tool was insisting on a three node GlusterFS cluster years ago.

Running the GlusterFS wizard

After this revelation the GlusterFS setup is supposedly straight forward. Still I ran into some issues that I could probably have avoided by carefully reading the installation instructions for oVirt 4.3. Nonetheless I’m quickly going to mention a few points here in case other people are struggling with the same and search the Web for these error messages:

On the first screen I had an error that the setup cannot proceed because "gluster-ansible-roles is not installed on Host": However, the related package including the Ansible roles from gluster-ansible was clearly there:
```
# rpm -q gluster-ansible-roles
gluster-ansible-roles-1.0.5-7.el8.noarch
```
Eventually I found that my sudo rules for my unprivileged user account are not properly picked up by Cockpit so I restarted the setup by using the root account which then successfully detected the Ansible roles.

When selecting the brick setup the “Raid Type” must be changed to “JBOD” and the device name that was reserved for the GlusterFS storage must be entered: Eventually the wizard will create a dedicated LVM volume group for GlusterFS bricks and a LVM thin-pool volume if you wish so:

# lvs gluster_vg_sda4
  LV                               VG              Attr       LSize    Pool                             Origin Data%  Meta%  Move Log Cpy%Sync Convert
  gluster_lv_data                  gluster_vg_sda4 Vwi-aot---  125.00g gluster_thinpool_gluster_vg_sda4        3.35
  gluster_lv_engine                gluster_vg_sda4 -wi-ao----   75.00g
  gluster_lv_vmstore               gluster_vg_sda4 Vwi-aot---  300.00g gluster_thinpool_gluster_vg_sda4        0.05
  gluster_thinpool_gluster_vg_sda4 gluster_vg_sda4 twi-aot--- <421.00g                                         1.03   0.84

Before the Ansible playbook that will setup the storage is executed the generated inventory file will be displayed. Because I’m very familiar with Ansible anyway, I love this part! It also makes it easier to understand which playbook command to run when troubleshooting something on the command line where the Cockpit Web interface doesn’t make sense to be involved anymore: The “Enable Debug Logging” is definitely worth to be enabled especially if you run this the first time on your server. It gives you much more insight in what Ansible is actually doing on your hypervisor.

When finally running the “Deploy” step it didn’t take long to fail the playbook with the following error:

TASK [Check if provided hostnames are valid] ***********************************
task path: /usr/share/cockpit/ovirt-dashboard/ansible/hc_wizard.yml:29
fatal: [loki.oasis.home]: FAILED! => {"msg": "The conditional check 'result.results[0]['stdout_lines'] > 0' failed. The error was: Unexpected templating type error occurred on ({% if result.results[0]['stdout_lines'] > 0 %} True {% else %} False {% endif %}): '>' not supported between instances of 'list' and 'int'"}

The involved code is untouched since a long time. So I reported the issue at RHBZ #1806298. I’m not sure how this could ever work but the fix is also trivial. After changing the following lines this task was successfully passing:

  --- /usr/share/cockpit/ovirt-dashboard/ansible/hc_wizard.yml.orig       2020-02-18 14:48:33.678471259 +0100
  +++ /usr/share/cockpit/ovirt-dashboard/ansible/hc_wizard.yml    2020-02-18 14:48:55.810456470 +0100
  @@ -30,7 +30,7 @@
         assert:
           that:
           - "result.results[0]['rc'] == 0"
-          - "result.results[0]['stdout_lines'] > 0"
+          - "result.results[0]['stdout_lines'] | length > 0"
         fail_msg: "The given hostname is not valid FQDN"
       when: gluster_features_fqdn_check | default(true)

Btw. the error message can not only be seen in the Web UI but is also written to a log file: /var/log/cockpit/ovirt-dashboard/gluster-deployment.log

There was another issue that cost me more effort to track down. The deployment playbook was failing to add the firewalld rules for GlusterFS:

TASK [gluster.infra/roles/firewall_config : Add/Delete services to firewalld rules] ***
task path: /etc/ansible/roles/gluster.infra/roles/firewall_config/tasks/main.yml:24
failed: [loki.oasis.home] (item=glusterfs) => {"ansible_loop_var": "item", "changed": false, "item": "glusterfs", "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Permanent and Non-Permanent(immediate) operation, Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}

Indeed firewalld doesn’t know about a ‘glusterfs’ service:

# rpm -q firewalld
firewalld-0.7.0-5.el8.noarch
# firewall-cmd --get-services | grep glusterfs
#

Is this an old version? From where do I get the ‘glusterfs’ firewalld service definition? The solution to this is as simple as embarrassing. I found that the service definition is packaged as part of the glusterfs-server RPM which was still missing on my server. After installing it also this issue was solved.

Eventually the deployment succeeded and the CentOS 8 host was converted into a single node GlusterFS storage cluster:

# gluster volume status
Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick loki.oasis.home:/gluster_bricks/data/
data                                        49153     0          Y       23806

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

Status of volume: engine
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick loki.oasis.home:/gluster_bricks/engin
e/engine                                    49152     0          Y       23527

Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks

Status of volume: vmstore
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick loki.oasis.home:/gluster_bricks/vmsto
re/vmstore                                  49154     0          Y       24052

Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks

Conclusion

I’m super pleased with the installation experience so far. The new GlusterFS Ansible roles had no issues setting up the bricks and volumes. The Cockpit Web-GUI was easy to use and always clearly communicated what was going on. There is now a supported configuration of a one node hyperconverged oVirt setup which makes me happy too. Kudos to everyone involved with this, great work!

Now let’s continue with the oVirt self-hosted engine setup.

My free software activities (2019-10)

Wed, 06 Nov 2019 08:39:22 +0100

My activities in October were mostly related to updating my COPR repositories for CentOS 8 and cleaning up the old repositories:

I updated the ganto/jo COPR repository to support CentOS 8.
I updated the ganto/vcsh COPR repository to support CentOS 8 and added package builds for the alternative architectures (aarch64 and ppc64le).
Thanks to the help of jmontleon I was finally able to build LXD which is available in my ganto/lxc3 repository for CentOS 8. I also updated the RPM for the latest stable release LXD 3.18.
After years of development the distrobuilder tool which is meant to replace the shellscript-based LXC templates was tagged in a first 1.0 release that should now also be able to build CentOS 8 container images. Of course I updated the corresponding RPM in the ganto/lxc3 COPR repository accordingly. I’m not sure how they decide to do new releases therefore I might decide to go back building regular git snapshot releases of this tool in the future.
I updated the ganto/goaccess COPR repository to support CentOS 8 and also increased the built goaccess version to a git snapshot from May 2019 based on version 1.3. Unfortunately the official Fedora package is still only at version 1.2. I was first testing the latest git snapshot but then found that it is affected by a bug (GitHub issue 1575) which would fail to render the access graphs properly.
The last COPR repository pending an update for CentOS 8 is ganto/umoci which still fails because of go-md2man missing from EPEL 8.
I deleted some outdated COPR repositories (ganto/lxc, ganto/lxd, ganto/lxdock) and archived the related GitHub repositories holding the RPM spec files.

Then I was also experimenting with adding Debian machines to a CentOS FreeIPA identity management server via Ansible. Years ago I wrote an Ansible role freeipa-client which was able to do that but still required manual setup of the Kerberos keytab on the client machine. I plan to replace that with a collection of new roles trying to blend-in with DebOps as much as possible. But unfortunately nothing ready to show yet.

Finally, as always, I updated a lot of ebuilds in my linuxmonk-overlay Gentoo overlay.

My free software activities (2019-09)

Tue, 01 Oct 2019 00:21:56 +0200

I’m starting a new series of blog posts summarizing my various activities regarding free software projects. There might not be every month something worth mentioning, but this month I was quite busy what might be interesting for some of you.

Following I’ll list some of the activities I was involved regarding free software projects in September:

After the official release of CentOS 8, I started rebuilding the packages in my lxc3 COPR repository for CentOS 8. The lxd package is still missing and I’m planning to provide it for CentOS 8 together with the pending update to lxd-3.17. A rebuild of the packages in my various other COPR repositories can be expected in the coming weeks.
Being the package maintainer of the spectre-meltdown-checker package in Fedora and EPEL, I followed the instructions to request a package branch for epel-8. This was approved a few hours ago, so the packages is now available via Koji and awaits approval in Bodhi for inclusion into the EPEL testing and eventually stable repository. Please give some karma if you’d like to accelerate this.
I merged some pull-requests in the Gentoo go-overlay git repository where the original maintainer entrusted me with commit permissions. Because he didn’t participate since last December, I used the chance to cleanup the repository to pass the repoman checks again and eventually merged a PR for the latest traeffik 1.x (1.7.18) release.
I put some effort into packaging the Gnome 3.34 release in my personal Gentoo linuxmonk-overlay. Of course I’m running it on my main workstation on top of Wayland without any major issues so far. Give it a try if you can’t wait for the official ebuilds to be ready.
I released version 0.1.2 of my acme-tiny Ansible role which fixes an annoying bug. It could happen that if the certificate renewal was unsuccessful, a still valid certificate would have been overwritten with an empty file. Now the role will make a backup copy of the old certificate by default and validate the new certificate before replacing the old one.

A user story about an OKD update from 3.9 to 3.10

Mon, 17 Dec 2018 17:50:09 +0100

The OpenShift community produces a lot of interesting tutorials about how to try new solutions and configurations but unfortunately they are mostly based on a minimal setup such as MiniShift, which is definitely a cool gimmick, but badly resembles a real cluster setup. Often those posts only concentrate on the known good path about how something is supposed to function in the best case. They rarely mention how it could be debugged or fixed if it doesn’t work as expected. As all of us know, the more complex a system is, the more can go wrong and this technology is no exception especially when run in a real distributed setup. To give you some insight in how such procedures can go wrong, I’d like to share the experience I made when I tried to update my multi-master/multi-node OKD cluster. As an experienced Linux engineer or developer you might think that version updates are nothing special or exciting, but this experience will disabuse you. I hit many issues and here is how I did it.

IMPORTANT: This is not a guide how to upgrade OpenShift. It’s only a field report which is missing a lot of technical details for a successful upgrade. Please always investigate the official documentation.

Starting Position

At home I run a small OKD cluster consisting of three masters, each also hosting a etcd member, and four nodes, of which there are two infrastructure nodes, hosting the routers and registry, and two compute nodes, hosting the applications. I feel that’s the minimal setup required to resemble a production-like cluster. To make the setup a bit more interesting the persistent storage is served by a container-native storage (CNS) configuration were three GlusterFS pods are distributed on the masters. This definitely deviates from how a production cluster should be setup, but as I’m running this locally at home, I don’t have enough resources available for separate storage nodes.

My masters and nodes are running atop of CentOS Atomic Host which I updated to the latest 7.1811 release just a few days ago. As identity provider for OpenShift I’m using a FreeIPA server on a separate CentOS box. Since I installed this cluster with OpenShift Origin 3.9 five months ago, it was running stable and I had a lot of fun with it. After the recently published security advisories time has come to finally take a chance to upgrade OpenShift.

What will change?

The first thing, you should always do before starting an OpenShift update, is to carefully read the release notes. I explicitly linked the Red Hat OpenShift Container Platform (OCP) release notes here, because OKD unfortunately doesn’t nicely touch up theirs. For the initial release update they are mostly congruent. Make sure to study it carefully, as this might be the primary source of information once something starts going down. For the 3.10 update, an important information is the new handling of the containerized master controller and API services. Eventually we now have a basic idea about what to expect.

Updating the Ansible Inventory

It would be nice, if there was a fool-proof command to run the update and it seems that OpenShift 4 with its Cluster Version Operator is heading there. But until then we need to carefully study and follow the official OKD 3.10 Upgrade Guide. It’s important to get the documentation for the correct release because the involved adjustments to the Ansible inventory are different from release to release. For those not knowing how OpenShift 3.x release upgrades work, it’s done via Ansible playbooks which are using the same inventory (definition on how everything will be configured) as the initial OpenShift cluster installation.

In my inventory file, I first added the Node Group assignments. E.g. the infrastructure nodes are no longer defined via openshift_node_labels variable, but via a dedicated openshift_node_group_name variable which references a node group definition from the openshift_node_groups configuration. The same changes have to be done also for the master and the compute nodes:

OpenShift 3.9:

[infra-nodes:vars]
# Set region to be dedicated for infrastructure pods
openshift_node_labels={'region': 'infra', 'zone': 'default'}

OpenShift 3.10:

[infra-nodes:vars]
# Set infra node group
openshift_node_group_name='node-config-infra'

Note that although the openshift_node_labels variable is no longer effective, no labels will be removed during the upgrade. So if you don’t get the label definition right at the beginning you don’t have to worry that after an in-place upgrade some workload is suddenly not scheduled anymore.

I had some custom openshift_node_kubelet_args defined in my OpenShift 3.9 inventory but this variable is also not respected any longer. With 3.10 the correct way to customize the node configuration is to define a edits argument in the corresponding node group definition, which is then applied to a ConfigMap resource by the custom yedit Ansible module. While writing such a definition itself is already not super intuitive, it can only be done by re-defining the entire openshift_node_groups variable, possibly also blowing up every other node group definition if done wrong. For the moment, I chose to drop my custom node configuration entirely to make the inventory less error prone for now.

Before running the upgrade playbooks it’s also important that every manual configuration update done in the past (e.g. in the master-config.yml) has to be reflected somewhere in the Ansible inventory. Otherwise the change might be lost after the upgrade. In my setup I still had to add the LDAP authenticator to the openshift_master_identity_providers variable because I added it manually after the initial cluster installation.

The section about Special Considerations When Using Containerized GlusterFS gave me a bit of a bad feeling as my GlusterFS pods are running on the control-plane hosts. But it’s not an easy task to change that now, so I chose to still go on with the upgrade and hope for a work-around in case something should break.

Fixing a failed CNS Brick Process

Once I felt confident that my inventory was in good shape, I started the control-plane upgrade playbook placed at playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml. Only after a few minutes it already failed for the first time. The error message said, that my hosted registry presistent volume is not healthy. That are good news, because the playbook did properly detect my unconventional CNS setup and was even able to check the healthiness. Fortunately, this issue was already familiar to me and it was easily fixed. Here is how you do this:

Change to the glusterfs project (or any custom project where you are running the GlusterFS pods) as a project or cluster administrator and query the names of the GlusterFS pods:

$ oc project glusterfs
Now using project "glusterfs" on server "https://openshift.example.com:8443".
$ oc get pods -n glusterfs -o wide
NAME                      READY     STATUS    RESTARTS   AGE       IP            NODE
glusterfs-storage-dz8qj   1/1       Running   3          2d        10.0.0.10     master01.example.com
glusterfs-storage-jncsl   1/1       Running   3          1d        10.0.0.11     master02.example.com
glusterfs-storage-r24wg   1/1       Running   2          11h       10.0.0.12     master03.example.com
heketi-storage-1-w8c42    1/1       Running   1          6d        10.129.0.21   node02.example.com

Connect to one of the GlusterFS pods and list the volume status. E.g.:

$ oc rsh glusterfs-storage-dz8qj
sh-4.2# gluster volume status
Status of volume: glusterfs-registry-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.0.0.11:/var/lib/heketi/mounts/vg_61
bc5a26248cc6ea9fb7ffaae4edbe93/brick_128dfe
b5436dad25702e689d3d6f4b8a/brick            49152     0          Y       204
Brick 10.0.0.12:/var/lib/heketi/mounts/vg_5e
cf67ed85f71cf28090d7db1acc6433/brick_d36469
4c7034276c22e264eb2576413b/brick            49152     0          N
Brick 10.0.0.10:/var/lib/heketi/mounts/vg_49
e81b0f4c91942c7657e9b7ffff7834/brick_8717e0
992390a7c04431890ba56b7656/brick            49152     0          Y       224
Self-heal Daemon on localhost               N/A       N/A        Y       215
Self-heal Daemon on 10.0.0.11               N/A       N/A        Y       163
Self-heal Daemon on 10.0.0.12               N/A       N/A        Y       172

Task Status of Volume glusterfs-registry-volume
------------------------------------------------------------------------------
There are no active volume tasks
[...]

According to my experience it can happen that sometimes a brick displays a N in the Online column which means that the corresponding brick process wasn’t started successfully. If multiple bricks of the same volume are down, your entire volume is down and must be properly recovered. In such a case don’t continue with the steps below!

Via IP address of the brick, you can figure out which host is affected and then you can simply delete the corresponding pod:
```
$ oc delete pod glusterfs-storage-r24wg
```
The pod will be automatically restarted and the brick processes should come up this time.

Fixing the Hosted Registry Storage Definition

The second run of the control-plane playbook eventually attested that all GlusterFS volumes are healthy but again it failed only two tasks later with a rather cryptic error message, something like:

TASK [openshift_storage_glusterfs : Check for GlusterFS cluster health] **********************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks/cluster_health.yml:4
Using module file /usr/share/ansible/openshift-ansible/roles/lib_utils/library/glusterfs_check_containerized.py
FAILED - RETRYING: Check for GlusterFS cluster health (120 retries left).Result was: {
    "attempts": 1,
    "changed": false,
    "invocation": {
        "module_args": {
            "cluster_name": "registry",
            "exclude_node": "master01.example.com",
            "oc_bin": "/usr/local/bin/oc",
            "oc_conf": "/etc/origin/master/admin.kubeconfig",
            "oc_namespace": "default"
        }

Didn’t it just said, that all the GlusterFS volumes are healthy? What the heck is "cluster_name": "registry" and what is it doing with GlusterFS in the ‘default’ namespace anyway?

The solution for this, I found after digging deep in the openshift_storage_glusterfs Ansible role and reading the CNS installation instructions again and again. I became a victim of my “simplified” CNS setup. The reference installation is meant to have two separate CNS GlusterFS clusters. One exclusively for the hosted registry volume (hinted by the [glusterfs_registry] Ansible host group) and a second cluster for any other persistent volumes (hinted by the [glusterfs] Ansible host group). As mentioned before, I’m limited in available hosts so I added the master hosts to both host groups and set the glusterfs_devices variable to the same device when installing the CNS. That’s already everything that was needed to create the hosted registry volume with OpenShift 3.9 in the “regular” CNS cluster. However the 3.10 playbook expects the registry volume to be in a different project with a different naming. Fortunately all that was needed to fix this were some additional inventory variables in the [OSEv3:vars]:

# Adjust variables for registry storage to match default converged glusterfs storage setup
openshift_storage_glusterfs_registry_name=storage
openshift_storage_glusterfs_registry_namespace=glusterfs

With the updated inventory I started the control-plane upgrade playbook once more. This time it ran for quite a while and even started to do some real stuff. It replaced the docker run command in the ‘origin-node’ systemd service with a runc command using the 3.10 image. Finally some progress. But eventually another error aborted the playbook and again it was a totally unexpected one.

etcd Backup Failure

Before updating the etcd cluster, there is a task which would backup the etcd database and this failed miserably. It couldn’t run docker exec etcd_container etcdctl backup [...]. When executing the command manually on a master host, I received the same error message:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

My first suspicion was the Ansible role. Maybe the backup command is wrong? But I couldn’t find any radical changes in the commit history regarding the etcd backup and a blocking issue like this is unlikely to stay unnoticed for such a long time. Maybe something wrong with the image? The OpenShift Origin 3.9 setup was using a rather atypical image at that time, the only one from the Fedora image registry ( registry.fedoraproject.org/latest/etcd:latest). When I see the ’latest’ tag being used with containers I’m instantly suspicious that bugs might sneak in unnoticed as different users may get different images depending on when they are pulling them. Maybe they mistakenly pushed an image without a shell or without the etcdctl binary? So I tried to ask in the #openshift IRC channel on Freenode if someone experienced the same issue before but didn’t get any reply. Suddenly I had an idea: Only a few hours before I was using the etcdctl tool from the Atomic host to do my own etcd backup. I just need to find a way to make Ansible use the etcdctl from the host and everything would be fine. So I was digging a bit in the Ansible etcd role and a few minutes later I set r_etcd_common_etcdctl_command to "etcdctl" in my inventory, being confident that this would fix my issue. It won’t, but I won’t find out anytime soon…

The Master API cannot find the LDAP CA Certificate

In the next attempt, the playbook happily ran the etcd backup, upgraded the etcd images, converted them to be a static pod on all masters and did the same for the other two control plane services, the API service and the controller service, starting on the first master. Eventually the ‘origin-master-api’ and ‘origin-master-controller’ services were shutdown and the corresponding pods should be started, so the playbook was waiting for the API service to come up and waited and waited… The pod didn’t come up. Hmpf. Time has already come to use the new debugging command I read about in the release notes to see what’s going on:

# /usr/local/bin/master-logs api api

That is an alternative for the corresponding oc command that I’m also able to run from my client machine:

$ oc logs master-api-node01.example.com -n kube-system

But the latter one was behaving weird. Sometimes it hung although the API services of the two other nodes were still up. There is definitely something wrong.

When checking the logs locally, I saw an error that my FreeIPA CA certificate which should be used to validate the LDAPS connections cannot be found. That’s strange. I explicitly configured the ca key in the openshift_master_identity_providers variable pointing it to the correct CA certificate. I did this in other OpenShift cluster inventories before and there it was working… But those were not running OpenShift 3.10 or later. With 3.10 the playbook developers removed the possibility to custom-name the CA certificate so the ca key from the inventory was silently ignored. Only after checking the installation instructions regarding Configuring identity providers with Ansible, I found an inconspicuous comment that the CA certificate destination path now follows a given naming convention. When I was adding the identity provider configuration to the inventory before the update, I didn’t specify the openshift_master_openid_ca or openshift_master_openid_ca_file variables which will ensure that the CA certificate is copied to the correct place. After all the certificates were already on the master hosts and the identity provider was working, so I didn’t want the upgrade playbook to touch the certificate. Now that’s the result: My mistake. Still, I like issues that are clear and can be fixed so easily. A quick rename of the certificate on the master hosts made the API service successfully start again.

How a Docker Bug broke the etcd Cluster

All API servers are running again, although only the first one in the final configuration, but the oc command invocation still feel sluggish. Sometimes it even hangs completely. When checking the process list it attracted my attention that the etcd processes are only a few minutes old and sometimes they are not running at all. So I was checking the etcd cluster health and here it is: Two cluster members are down and one is in the state unhealthy. That is bad… Immediately, I started manually triggering etcd restarts. But only a few minutes later they shutdown again. I was checking the log files and there were errors, but I couldn’t figure out a single reason what might cause this mess. Then I found that the /etc/etcd/etcd.conf was updated during the playbook run, so I restored the backup, but again it wouldn’t fix the issue. Eventually I started to accept the thought that I might need to completely restore the etcd database from a backup because the database might already be so corrupt in the mean time that it is not able to find a stable state anymore.

The OKD documentation for Restoring etcd quorum would be the correct guide that you need to follow in this situation, but for a reason I landed at Restoring etcd. That confronted me with yet another issue: This guide was not yet properly updated for OpenShift 3.10. Some parts of the documentation still reference etcd as systemd service. But in my setup it’s a pod. Trying to pass the --force-new-cluster parameter to the etcd process via systemd override obviously doesn’t have any effect. Eventually I found out about the /etc/origin/node/pod/etcd.yaml file which contains the pod definition. And here we are able to correctly pass the parameter so that it is picked up by the pod startup command. But again, even with an empty database, a few minutes later my pod would die again. Something is badly broken here. In the YAML definition I also found the liveness probe. So once the pod was started once more, I tried to execute the liveness probe to see what it returns and the result looked familiar, in a bad way:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

Ouch! Now I understand why the pods keep restarting. It’s again the same error that also caused the etcd backup failure before. But now I’m using the new quay.io/coreos/etcd:v3.2.22 image. This disproved my theory that a buggy image might be the reason. For the moment, I ran out of ideas… Until I remembered that I recently read a post about a docker bug (#1655214) that affected CentOS 7. Thanks for that! After checking the docker version on Atomic host 7.1811 ( docker-1.13.1-84.git07f3374.el7.centos) it’s confirmed. That’s the root cause for so much trouble so far.

Updating Docker on Atomic Host

I didn’t need to dig too much into Atomic Host so far, as most of the stuff was simply working or was easily fixed with an update in the past. But this time it didn’t look that there is an imminent update. Release 7.1811 was only a few days old. I could roll back, but the previous version is 7.1808. That’s three months back and somehow defeats the purpose of my update, to get the latest security fixes. Fortunately CentOS already released new docker packages where this bug is fixed. Now I just need to find a way to update the docker packages independently from the ostree image? This time the documentation gods were on my side. I quickly found Dusty Mabe’s Atomic Host 101 Lab Part 4: Package Layering, Experimental Features.

Here my guide for quickly working around Bug #1655214 by updating the docker packages to release 1.13.1-88.git07f3374.el7.centos on CentOS Atomic Host:

Create a temporary directory and download the corresponding RPM packages from a mirror of your choice:

# mkdir /tmp/docker-1.13.1-88
# cd /tmp/docker-1.13.1-88
# for pkg in docker docker-client docker-common docker-lvm-plugin docker-novolume-plugin ; do \
    curl -O https://mirror.init7.net/centos/7/extras/x86_64/Packages/$pkg-1.13.1-88.git07f3374.el7.centos.x86_64.rpm ; \
  done

From within the directory run rpm-ostree override replace to replace the docker packages from the ostree layer with the new RPMs:

# rpm-ostree override replace docker*
Checking out tree ee5a6f2... done
Inactive requests:
  docker (already provided by docker-2:1.13.1-84.git07f3374.el7.centos.x86_64)
Enabled rpm-md repositories: base updates extras
Updating metadata for 'base': [=============] 100%
rpm-md repo 'base'; generated: 2018-11-25 16:00:34
Updating metadata for 'updates': [=============] 100%
rpm-md repo 'updates'; generated: 2018-12-10 15:34:27
Updating metadata for 'extras': [=============] 100%
rpm-md repo 'extras'; generated: 2018-12-10 16:00:03
Importing metadata [=============] 100%
Resolving dependencies... done
Relabeling (5/5) [=============] 100%
Applying 5 overrides
Processing packages (10/10) [=============] 100%
Running pre scripts... 1 done
Running post scripts... 5 done
Writing rpmdb... done
Writing OSTree commit... done
Copying /etc changes: 42 modified, 5 removed, 613 added
Transaction complete; bootconfig swap: no; deployment count change: 0
Freed: 580.5 kB (pkgcache branches: 0)
Upgraded:
  docker 2:1.13.1-84.git07f3374.el7.centos -> 2:1.13.1-88.git07f3374.el7.centos
  docker-client 2:1.13.1-84.git07f3374.el7.centos -> 2:1.13.1-88.git07f3374.el7.centos
  docker-common 2:1.13.1-84.git07f3374.el7.centos -> 2:1.13.1-88.git07f3374.el7.centos
  docker-lvm-plugin 2:1.13.1-84.git07f3374.el7.centos -> 2:1.13.1-88.git07f3374.el7.centos
  docker-novolume-plugin 2:1.13.1-84.git07f3374.el7.centos -> 2:1.13.1-88.git07f3374.el7.centos
Run "systemctl reboot" to start a reboot

Reboot the host.

I carefully did this on one master server after the other and surprisingly all the services (except etcd) were started normally. Even my GlusterFS pods came up again as nothing had happened. But still, the etcd cluster was offline and with it the Master API was inaccessible. No oc commands were possible.

Fixing the etcd Cluster

With the docker issue being fixed, I now had to bring up the etcd cluster again. The database was likely in a confused state because of all the failed attempts before, so I decided to restore a known good state. As briefly mentioned before, to do so, you actually create a new cluster with the database from a backup. Because the OpenShift documentation on how to do this cannot be followed easily, I list the exact steps below how I manged to do it:

Make sure that all the etcd processes are down and not coming up again automatically. On an OpenShift 3.10 cluster, you prevent automatic startup by moving the /etc/origin/node/pod/etcd.yaml definition to a backup location e.g. /etc/origin/node/pod/disabled/ on every etcd host.
First create a new one node etcd cluster on the first etcd host. To do so, we need some preparation:
- The /etc/etcd/etcd.conf configuration must not contain any previous configurations regarding the INITIAL_CLUSTER or INITIAL_CLUSTER_STATE. I was able to simply use the etcd.conf generated by the upgrade playbook which already set those two variables to the correct value:
```
ETCD_INITIAL_CLUSTER=
ETCD_INITIAL_CLUSTER_STATE=new
```
  Also make sure, the ETCD_INITIAL_ADVERTISE_PEER_URLS only contains the URL of the first host itself and no other peers:
```
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.0.0.10:2380
```
- Restore the etcd database from a backup. Fortunately the upgrade playbook automatically created a backup after the etcd upgrade, so I’m going to restore to that state:
```
# mv /var/lib/etcd/member /var/lib/etcd/member.orig
# cp -rP /var/lib/etcd/openshift-backup-post-3.0-20181214022846/member /var/lib/etcd/
```
- When starting the first etcd member for the first time, we need to pass the --force-new-cluster argument to the process. This will override the cluster definition from the database files. To do so, the etcd.yaml file has to be adjusted. Here the important snippet (everything else should be kept as it is):
```
      spec:
        containers:
  - args:
    - '#!/bin/sh

      set -o allexport

      source /etc/etcd/etcd.conf

      exec etcd --force-new-cluster

      '
```
- If everything is ready to start the etcd process, move the altered etcd.yaml file back to the /etc/origin/node/pod directory. Within a few moments, the pod should startup and create a new cluster.

Check the initial cluster state via:

# etcdctl2 cluster-health
member 67aa8b8cc701 is healthy: got healthy result from https://10.0.0.10:2379

If something went wrong, you might want to check the logs via:

# /usr/local/bin/master-logs etcd etcd

Initially the first member still advertises a PeerURL pointing to ’localhost’:

# etcdctl2 member list
67aa8b8cc701: name=master01.example.com peerURLs=http://localhost:2380 clientURLs=https://10.0.0.10:2379 isLeader=true

This must be updated by the correct host URL pointing to itself:

# etcdctl2 member update 67aa8b8cc701 https://10.0.0.10:2380
Updated member with ID 67aa8b8cc701 in cluster

Then it correctly shows:

# etcdctl2 member list
67aa8b8cc701: name=master01.example.com peerURLs=https://10.0.0.10:2380 clientURLs=https://10.0.0.10:2379 isLeader=true

This configuration was automatically saved in the database. So the --force-new-cluster argument can be removed again. Edit the etcd.yaml in-place to restore the original configuration. After doing so, restart the etcd process with:
```
# /usr/local/bin/master-restart etcd
```
If it comes up again and shows healthy, we can continue the add the other two cluster members.
The following steps to add another cluster member obviously have to be done on for both other etcd hosts:
1. Add the new host to the cluster by executing the following command on the first etcd host:
```
# etcdctl2 member add master02.example.com https://10.0.0.11:2380
Added member name master02.example.com with ID a6b2e8d0d392083b to cluster

ETCD_NAME="member02.example.com"
ETCD_INITIAL_CLUSTER="member01.example.com=https://10.0.0.10:2380,member02.example.com=https://10.0.0.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
```
  The new member will then be displayed as ‘unstarted’ in the member list.
2. Prepare the /etc/etcd/etcd.conf file on the new etcd host by defining the variables as shown in the output of the etcdctl2 member add command above. The ETCD_INITIAL_CLUSTER value will automatically be extended with each new member added to the cluster.
3. Delete the old database on the new etcd host. It will automatically be synced from the other cluster members once the new node has joined:
```
# mv /var/lib/etcd/member /var/lib/etcd/member.orig
```
4. Enable the etcd pod by moving the etcd.yaml back to /etc/origin/node/pod. Within a few minutes the etcd process should be started and eventually join the etcd cluster.

Once the etcd cluster was restored, the oc command was finally working again and I could check the state of the etcd pods also via OpenShift client:

$ oc get pods -n kube-system | grep etcd
master-etcd-master01.example.com          1/1       Running   5          1h
master-etcd-master02.example.com          1/1       Running   0          47m
master-etcd-master03.example.com          1/1       Running   0          2m

During the entire time the etcd cluster was down the OpenShift cluster continued running. The registry, routers and applications such as my Gitea setup were online all the time and even the CNS cluster running on the master hosts handled the debugging and restart session with bravery. Fortunately I had a super static setup during that time and so no deployments or replica count enforcement needed to be executed which would have been impossible anyway. Still I feel it’s a positive fact that shows the resiliency the platform has gained over time.

Finishing the Control Plane Upgrade

After a longer detour, I was finally back at the point were I could start another run of the control plane upgrade playbook. Remember, when the playbook aborted before it did so after upgrading the control plane services on the first master node, there are still two to go. So I started the playbook once again.

By now I have a really good feeling about the state of the playbook in this release. As you can see above, it failed on me many times in all different stages of the update, but it always had a good reason and it was always able to pick up where it left. My experience with initial upgrade attempts of earlier OpenShift releases was unfortunately not always that good. For example it happened to me that I had to restore a master host from a snapshot, because the playbook failed to correctly detect the upgrade state in the second run, after it aborted the first run due to a syntax error in a post-upgrade task.

This time the playbook finished successfully and my control plane was finally at release 3.10:

# /usr/local/bin/oc version
oc v3.10.0+c99b16a-90
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift.example.com:8443
openshift v3.10.0+c99b16a-90
kubernetes v1.10.0+b81c8f8

Running the Node Upgrade Playbook

After the control-plane was done, I had to upgrade the infrastructure and compute nodes. A separate playbook placed at playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_nodes.yml is available. Initially I only wanted to run it on a single node to make sure everything works as expected. This can be done by passing the -e openshift_upgrade_nodes_label=kubernetes.io/hostname=node03.example.com argument, where the given host name is obviously the node that should be upgraded, to the playbook execution command. The playbook completed without error already on the first attempt. So I continued with the other nodes.

One fact is super important when upgrading the nodes to OpenShift 3.10. The /etc/origin/node/node-config.yaml is completely regenerated based on the settings in the corresponding node group (and/or the defaults) and so any prior adjustment not reflected in the inventory is lost. Therefore make sure that you perfectly understand the Node Group concept and how it affects your node layout and configuration.

To give you an example how to customize the upgrade behavior on the infrastructure nodes, I added the following arguments to the playbook execution: -e openshift_upgrade_nodes_label=region=infra -e openshift_upgrade_nodes_serial=50%.

Fixing the Infrastructure Node Selector

It confused me that the NodeSelector of the infrastructure components such as the registry and routers were not updated to the new defaults. In the inventory I explicitly defined the new node selector:

openshift_hosted_registry_selector='node-role.kubernetes.io/infra=true'

But when checking the DeploymentConfig of the registry, I can still find the old NodeSelector:

$ oc get dc docker-registry -n default -o json | jq .spec.template.spec.nodeSelector
{
  "region": "infra"
}

So I manually triggered an update of the NodeSelector property in all the DeploymentConfig s using it. E.g.:

$ oc patch dc docker-registry -n default --type json --patch '[{"op":"replace","path":"/spec/template/spec/nodeSelector","value":{"node-role.kubernetes.io/infra":"true"}}]'
deploymentconfig "docker-registry" patched

NodeSelector s can also be set in DaemonSet s, as annotations in projects or even globally via master-config.yaml. Therefore make sure to update them all, when required, before removing any labels from the nodes.

After checking that all the pods are up and running again, I was finally able to remove the old infrastructure labels from the nodes:

$ oc label node node01.example.com region- zone-

Summarizing

This was not my first OpenShift update ever, but my first update from 3.9 to 3.10. This obviously means that I made some mistakes and had wrong assumptions from which I did learn a lot. I hope I could share some insights and useful hints for those of you that haven’t done this before. Otherwise it will at least help me in the future to run this update an other cluster much smoother.

At the end some advice for those of you who also need to do such an upgrade:

You need to have a test cluster where you can practice such updates. It doesn’t need to be big but the Ansible inventory variables should be structurally as similar as possible to those of the production cluster. As you saw above, a lot of errors just happened due to wrong inventory variables. Ideally the test cluster should have some workload so that you experience how the applications behave during the update and and so that you can test if everything still works after an upgrade.
Emphasis your Ansible inventory. Everything of your configuration that can fit into the Ansible inventory must be defined there and must be maintained there. It can cost you a lot of time debugging or even result in application downtime during an upgrade if you manually updated the cluster configuration without adjusting the configuration in the inventory. Even when it sometimes feels like it’s more work than benefit it’s always worth it.
Preparation is key. Carefully read through the upstream documentation available. Most likely you also have some internal documentation where your infrastructure specifics are written down. Run the upgrade on a test cluster before you do it in production. If it doesn’t work on the first attempt, update your notes and try it again. Try to gain as much experience as possible on the test infrastructure so that you already know what to do if something goes wrong in production.
Plan a lot of time. Doing such an upgrade is a lot of work! Give yourself enough time to do a proper preparation and also the actual upgrade window itself should give you enough time to fix issues when they arise. Plan in the scale of hours or better days. Ansible is slow. If you have to restart the playbook because of an error after 15 minutes this will eat up your time fast.

Thanks for reading. As always I’d welcome some feedback or critics in the comments.

Gitea on OpenShift with NodePort SSH access

Wed, 12 Dec 2018 03:48:34 +0100

The first thing that someone would need when operating or playing around with OKD (better known as OpenShift) is a git version control service. Personally I’m a fan of Gitea and that’s why I’d like to show a way how to run Gitea in a OpenShift environment. Gitea upstream already provides a great container image which I’m are going to use. But as some of you may have already experienced, running an image on docker and running it in OpenShift is two different pairs of shoes. The fact that the Gitea image runs an integrated SSH server means that it doesn’t simply match the widely discussed Web application pattern. Therefore I’m trying to explain some of the difficulties that one might encounter when moving such an application to OpenShift.

My environment consists of a multi-node OpenShift cluster. Obviously Gitea should be high available so that if a node goes down, one would still be able to access the git repositories. One pod is no pod, so Gitea must be deployed with a replica count of at least two. Accessing the pods over HTTP is already solved by the OpenShift default infrastructure via redundant HAProxy routers. I’d probably explain how to achieve a redundant router setup in one of my next blog posts but this time I’d like to emphasis on the Gitea SSH access via NodePort service feature. The following graphic shows a communication overview of such a setup:

NodePort Service

In OpenShift the Kubernetes Service resource is responsible for directing the traffic (TCP, UDP or SCTP) to the individual application pods. It maps the service name (e.g. ‘gitea’) via SkyDNS to a so-called ClusterIP. This is a virtual IP address that is not assigned to any host or container network interface but still used as packet destination within the cluster SDN (software defined network). After receiving a packet to this so-called ClusterIP the Linux kernel of an OpenShift node rewrites the packet destination to an IP address of an actual application pod and acts as a virtual network load-balancer.

In our example there is a ‘gitea’ service managing the HTTP traffic to port 3000 of the Gitea pod and a ‘gitea-ssh’ service managing the SSH traffic to port 22 of the Gitea pod. Because we can’t use the OpenShift Router as ingress for SSH, the ‘gitea-ssh’ service defines a special type called NodePort. This means that a packet sent to this port (e.g. 30022) on any OpenShift node will be received by the corresponding service and therefore forwarded to a Gitea pod. This is the simplest way how to direct non-HTTP traffic from outside of OpenShift to an application pod and can also be used for e.g. database protocols or Java RMI. Here the corresponding resource definition for the Gitea SSH service:

apiVersion: v1
kind: Service
metadata:
  name: gitea-ssh
spec:
  ports:
    - name: ssh
      nodePort: 30022
      port: 22
      protocol: TCP
      targetPort: 22
  selector:
    app: gitea
    deploymentconfig: gitea
  sessionAffinity: ClientIP
  type: NodePort

The sessionAffinity: ClientIP setting defines “sticky sessions” to avoid distributing multiple requests of the same client to different pods. I didn’t test yet, how SSH would behave without it, but I think it generally makes sense. In a running setup the service additionally shows the discussed ClusterIP which is statically assigned and the endpoints (pod IPs) which may change when pods are started and stopped:

$ oc describe service gitea-ssh
Name:                     gitea-ssh
Namespace:                vcs
Labels:                   app=gitea
                          template=gitea-persistent-template
Annotations:
Selector:                 app=gitea,deploymentconfig=gitea
Type:                     NodePort
IP:                       172.30.8.9
Port:                     ssh  22/TCP
TargetPort:               22/TCP
NodePort:                 ssh  30022/TCP
Endpoints:                10.129.2.44:22,10.130.3.71:22
Session Affinity:         ClientIP
External Traffic Policy:  Cluster
Events:

From within the cluster, the Gitea SSH service can be reached via service name (extended with OpenShift project name, here ‘vcs’) DNS entry or directly via ClusterIP:

$ host gitea-ssh.vcs.svc
gitea-ssh.vcs.svc has address 172.30.8.9

$ ssh git@gitea-ssh.vcs.svc
PTY allocation request failed on channel 0
Hi there, You've successfully authenticated, but Gitea does not provide shell access.
If this is unexpected, please log in with password and setup Gitea under another user.
Connection to gitea-ssh.vcs.svc closed.

From outside the cluster, the Gitea SSH service can be reached via NodePort on any OpenShift node. To avoid a dependency on a single node in the git repository URL you can define multiple DNS entries using the same name (e.g. services.example.com) to all OpenShift node addresses:

$ host services.example.com
services.example.com has address 10.0.0.2
services.example.com has address 10.0.0.3

$ ssh -p 30022 git@services.example.com
PTY allocation request failed on channel 0
Hi there, You've successfully authenticated, but Gitea does not provide shell access.
If this is unexpected, please log in with password and setup Gitea under another user.
Connection to services.example.com closed.

Issues with NodePort

Port Assignment The NodePort mechanism is allocating the corresponding port on each OpenShift node. To avoid a clash with node services such as the DNS resolver or the OpenShift node service the port range is restricted. It can be configured in the /etc/origin/master/master-config.yml with the option servicesNodePortRange and defaults to 30000-32767. Obviously multiple applications in the same cluster cannot use the same port and traffic to the chosen port must be allowed by the host firewall on the OpenShift nodes.

Node Groups NodePort s are always allocated on every OpenShift cluster host running the node service which also includes the OpenShift master servers. OpenShift doesn’t provide a way to restrict the involved hosts to a subset. In my example I choose to restrict the hosts receiving traffic by only adding a limited number of nodes to the service DNS entry and block access on the others via iptables. If you don’t use an application load-balancer in front of the OpenShift routers you could also re-use the wildcard DNS entry defined for the HTTP traffic. The NodePort traffic would then follow the same path as the normal Web traffic.

Node Failure If one OpenShift node goes down a client trying to access the Gitea SSH service might still try to connect to the unreachable host. Fortunately, the default SSH implementation used by the git command line client is quite tolerant and simply retries with another IP address. When testing this case I therefore haven’t experienced a major issue except a slight connection delay. The failure behavior might be different for other git clients or other application protocols altogether and is definitely not ideal but simple instead.

One way to improve this failure scenario would be to add a real TCP load-balancer in front of the NodePort but then there would be another piece of infrastructure that must be managed synchronously with the OpenShift infrastructure and which might be a new single point of failure.

Container with root Permissions

When starting the upstream Gitea container image in OpenShift you might likely encounter a startup failure with the following error message in the log:

s6-svscan: fatal: unable to mkfifo .s6-svscan/control: Permission denied

The Gitea image, and many other docker images not optimized for the pod concept introduced by Kubernetes, doesn’t start a single application process but a supervisor process (in this case s6) which then spawns multiple different application processes defined in /etc/s6. To do so it wants to create a FIFO in the /etc/s6/.s6-svscan directory which is only writable by the root user which fails as by default processes are started with a random unprivileged account.

Security Context Constraints

Unlike docker, OpenShift controls the actions a pod can do and access with a tight set of rules called Security Context Constraints (SCC). By default the ‘default’ ServiceAccount used to run the application pods is a member of the ‘restricted’ SCC which among other things defines the previously mentioned randomized UID. As Gitea won’t work like this, a less restrictive SCC must be used. After reading the documentation we find, that there is already a predefined SCC which grants us just enough permissions to start our container process as root user without weakening too many other restrictions. The SCC we are heading for is ‘anyuid’. Below I’ll present different approaches how this SCC can now be assigned to the Gitea deployment:

The OpenShift cluster administrator can add the ‘default’ ServiceAccount of a project to the list of users in the SCC definition. This doesn’t need any special configuration in the DeploymentConfig of the application but also grants every deployment in the corresponding project ‘anyuid’ privileges. In our setup this would be done with the following command, when assuming Gitea should be deployed in the ‘vcs’ project:
```
$ oc adm policy add-scc-to-user anyuid system:serviceaccount:vcs:default
```
I’m not in favor of this approach as it “hides” the additional permissions in the default ServiceAccount and prone to break the principle of least privilege by assigning the SCC to potentially more applications than necessary.
Another approach is using a dedicated ServiceAccount for the Gitea deployment and only adding that to the ‘anyuid’ SCC. The project owner can create a ServiceAccount with:
```
$ oc create serviceaccount gitea
```
The cluster administrator then has to add it to the SCC as before:
```
$ oc adm policy add-scc-to-user anyuid system:serviceaccount:vcs:gitea
```
In the DeploymentConfig the ServiceAccount must be referenced with a entry under the spec.template.spec key:
```
$ oc patch dc/gitea --patch '{"spec":{"template":{"spec":{"serviceAccountName": "gitea"}}}}'
```
The dedicated ServiceAccount used in this approach already hints that there might be special privileges connected to it and is in my opinion easier to audit. The disadvantage however is the more complex configuration.
Instead of adding every user account individually to the SCC a dedicated user group could be created having the SCC assigned to this group. Individual ServiceAcccounts s would then be added to the group and therefore inherit the SCC. This would follow common practice in identity management to assign permissions to users via privilege groups. Additionally a group management role could be created which then would permit dedicated users not having the ‘cluster-admin’ privilege to manage the group membership.
Unfortunately I couldn’t figure out a true self-service model where a responsible project admin could expand the necessary permissions without the possibility to interfere with other projects. In the documentation of OpenShift (<=3.7) I found a hint that it is/was(?) possible to extend the default ServiceAccount s available after creating a new project by adding the account name (e.g. ‘anyuid-service-account’) to the serviceAccountConfig.managedNames list in /etc/origin/master/master-config.yml. While this configuration is still present in newer master-config.yml, the documentation is gone and I also didn’t find a way how to automatically add a user created like this to the ‘anyuid’ SCC. Maybe it’s possible by somehow modifying the project template. If you have done this before or at least have an idea how this could be done, please drop me a line.

At the end, the way how the ‘anyuid’ SCC is assigned to the Gitea application is unimportant as long as the application pod is allowed to start the s6 supervisor process with root permissions.

Gitea Application Template

The way how OpenShift administrators can provide an application setup ready for instantiation by OpenShift project owners is through Templates. Inspired by the My journey through Openshift blog post, I wanted to create my own Gitea template fixing some issues found in the original template and extending it with the opinionated configuration presented above. You can download it from here.

The template is able to automatically setup Gitea with exception of the ‘anyuid’ SCC configuration. It requires a persistent volume (PV) for storing the git repositories and some static configurations such as the SSH authorized_keys file. By default it will use a SQLite database backend which is also stored in the PV. Optionally you can also give the connection string and credentials of a PostgreSQL or MariaDB backend which can run on OpenShift or externally.

If you want the template to be available in the Service Catalog the YAML file has to be applied to the ‘openshift’ project by a cluster administrator:

$ oc create -f gitea-persistent-template.yaml -n openshift

Afterwards it can be instantiated by any project admin via Service Catalog Web-UI or from the command line via:

$ oc new-project vcs
$ oc new-app --template=gitea-persistent -p HTTP_DOMAIN=git.example.com -p SSH_DOMAIN=services.example.com

Alternatively, if no Service Catalog is available, or the template shouldn’t be loaded to OpenShift, the application can also be created directly from the YAML file via:

$ oc new-app -f gitea-persistent-template.yaml -p HTTP_DOMAIN=git.example.com -p SSH_DOMAIN=services.example.com

IMPORTANT: The template will configure Gitea to use a ServiceAccount named according to the parameter APPLICATION_NAME (defaults to ‘gitea’). It must be added to the ‘anyuid’ SCC as described above. E.g.:

$ oc adm policy add-scc-to-user anyuid system:serviceaccount:vcs:gitea

If you have some feedback regarding the template or troubles using it, please open a Github issue. Comments, corrections or general feedback to my article can be posted below. Thanks for reading.

spectre-meltdown-checker in Fedora and EPEL repositories

Thu, 15 Feb 2018 22:14:27 +0100

The recently disclosed Spectre and Meltdown CPU vulnerabilities are some of the most dramatic security issues in the recent computer history. Fortunately even six weeks after public disclosure sophisticated attacks exploiting these vulnerabilities are not yet common to observe. Fortunately, because the hard- and software vendors are still stuggling to provide appropriate fixes.

If you happen to run a Linux system, an excellent tool for tracking your vulnerability as well as the already active mitigation strategies is the spectre-meltdown-checker script originally written and maintained by Stéphane Lesimple.

Within the last month I set myself the target to bring this script to Fedora and EPEL so it can be easily consumed by the Fedora, CentOS and RHEL users. Today it finally happend that the spectre-meltdown-checker package was added to the EPEL repositories after it is already available in the Fedora stable repositories since one week.

On Fedora, all you need to do is:

dnf install spectre-meltdown-checker

After enabling the EPEL repository on CentOS this would be:

yum install spectre-meltdown-checker

The script, which should be run by the root user, will report:

If your processor is affected by the different variants of the Spectre and Meltdown vulnerabilities.
If your processor microcode tries to mitigate the Spectre vulnerability or if you run a microcode which is known to cause stability issues.
If your kernel implements the currently known mitigation strategies and if it was compiled with a compiler which is hardening it even more.
And eventually if you’re (still) affected by some of the vulnerability variants.

On my laptop this currently looks like this (Note, that I’m not running the latest stable Fedora kernel yet):

# spectre-meltdown-checker
Spectre and Meltdown mitigation detection tool v0.33

Checking for vulnerabilities on current system
Kernel is Linux 4.14.14-200.fc26.x86_64 #1 SMP Fri Jan 19 13:27:06 UTC 2018 x86_64
CPU is Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):  UNKNOWN
  * CPU microcode is known to cause stability problems:  YES  (Intel CPU Family 6 Model 61 Stepping 4 with microcode 0x28)

The microcode your CPU is running on is known to cause instability problems,
such as intempestive reboots or random crashes.
You are advised to either revert to a previous microcode version (that might not have
the mitigations for Spectre), or upgrade to a newer one if available.

* CPU vulnerability to the three speculative execution attacks variants
  * Vulnerable to Variant 1:  YES
  * Vulnerable to Variant 2:  YES
  * Vulnerable to Variant 3:  YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:  NO  (kernel confirms your system is vulnerable)
> STATUS:  VULNERABLE  (Vulnerable)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  NO
  * Currently enabled features
    * IBRS enabled for Kernel space:  NO
    * IBRS enabled for User space:  NO
    * IBPB enabled:  NO
* Mitigation 2
  * Kernel compiled with retpoline option:  YES
  * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
  * Retpoline enabled:  YES
> STATUS:  NOT VULNERABLE  (Mitigation: Full generic retpoline)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

A false sense of security is worse than no security at all, see --disclaimer

The script also supports a mode which outputs the result as JSON, so that it can easily be parsed by any compliance or monitoring tool:

# spectre-meltdown-checker --batch json 2>/dev/null | jq
[
  {
    "NAME": "SPECTRE VARIANT 1",
    "CVE": "CVE-2017-5753",
    "VULNERABLE": true,
    "INFOS": "Vulnerable"
  },
  {
    "NAME": "SPECTRE VARIANT 2",
    "CVE": "CVE-2017-5715",
    "VULNERABLE": false,
    "INFOS": "Mitigation: Full generic retpoline"
  },
  {
    "NAME": "MELTDOWN",
    "CVE": "CVE-2017-5754",
    "VULNERABLE": false,
    "INFOS": "Mitigation: PTI"
  }
]

For those who are (still) using a Nagios-compatible monitoring system, spectre-meltdown-checker also supports to be run as NRPE check:

# spectre-meltdown-checker --batch nrpe 2>/dev/null ; echo $?
Vulnerable: CVE-2017-5753
2

I just mailed to Stéphane and he will soon release version 0.35 with many new features and fixes. As soon as it will be released I’ll submit a package update, so that you’re always up to date with the latest developments.

Getting started with LXD on Fedora

Tue, 20 Dec 2016 18:30:01 +0100

Since a long time I’m using and following the development of the LXC (Linux Container) project. I feel that it unfortunately never really had the success it deserved and in the recent years new technologies such as Docker and rkt pretty much redefined the common understanding of a container according to their own terms. Nonetheless LXC still claims its niche as full Linux operating system container solution especially suited for persistent pet containers, an area where the new players on the market are still in the stage of figuring out how to implement this properly according to their concept. LXC development hasn’t stalled, quite the contrary, they extended the API with a HTTP REST interface (served via Linux Container Daemon, LXD), implemented support for container live-migration, added container image management and much more. This means that there are a lot of reasons why someone, including me, would want to use Linux containers and LXD.

Enable LXD COPR repository LXD is not officially packaged for Fedora. Therefore I spent the last few weeks by creating some community packages via their COPR build system and repository service. Similar to the better known Ubuntu PPA (Personal Package Archive) system, COPR provides a RPM package repository which can easily be consumed by Fedora users. To use the LXD repository, all you need to do is enabling it via dnf:

# dnf copr enable ganto/lxd

Please note that COPR packages are not reviewed by the Fedora package maintainers therefore you should only install packages where you trust the author. For this reason I also provide a Github repository with the RPM spec files, so that everyone could also build the RPMs on their own if they feel uncomfortable using the pre-built RPMs from the repository.

Install and start LXD LXD is split into multiple packages. The important ones are lxd, the Linux Container Daemon and lxd-client, the LXD client binary called lxc. Install them with:

# dnf install lxd lxd-client

Unfortunately I didn’t had time to figure out the correct SELinux labels for LXD yet, therefore you need to disable SELinux prior to starting the daemon. LXD supports user namespaces to map the root user in a container to an unprivileged user ID on the container host. For this you need to assign an UID range on the host:

# echo "root:1000000:65536" >> /etc/subuid
# echo "root:1000000:65536" >> /etc/subgid

If you don’t do this, user namespaces won’t be used which is indicated by a message such as:

lvl=warn msg="Error reading idmap" err="User \"root\" has no subuids."
lvl=warn msg="Only privileged containers will be able to run"

Eventually start LXD with:

# systemctl start lxd.service

LXD configuration LXD doesn’t have a configuration file. Configuration properties must be set and retrieved via client commands. Here you can find a list of all supported configuration properties. Most tutorials will suggest to initially run lxd init which would generate a basic configuration. However there is only a limited set of configuration options available via this command and therefore I prefer to set the properties via LXD client. A normal user account can be used to manage LXD via client when it’s a member of the lxd POSIX group:

# usermod --append --groups lxd myuser

By default LXD will store its images and containers in directories under /var/lib/lxd. Alternatives storage back-ends such as LVM, Btrfs or ZFS are available. Here I will show an example how to use LVM. Similar to the recommended Docker setup on Fedora it will use LVM thin volumes to store images and containers. First create a LVM thin pool. For this we still need some space available on the default volume group. Alternatively you can use a second disk with a dedicated volume group. Replace vg00 with the volume group name you want to use:

# lvcreate --size 20G --type thin-pool --name lxd-pool vg00

Now we set this thin pool as storage back-end in LXD:

$ lxc config set storage.lvm_vg_name vg00
$ lxc config set storage.lvm_thinpool_name lxd-pool

For each image which is downloaded LXD will create a thin volume storing the image. If a new container is instantiated a new writeable snapshot will be created from which you can create an image again or make further snapshots for fast roll-back. By default the container file system will be ext4. If you prefer XFS, it can be set with the following command:

$ lxc config set storage.lvm_fstype xfs

Also for networking various options are available. If you ran lxd init, you may have already created a lxdbr0 network bridge. Otherwise I will show you how to manually create one in case you want a dedicated container bridge or attach LXD to an already existing bridge which would be configured through an external DHCP server.

To create a dedicated network bridge where the traffic will be NAT’ed to the outside, run:

$ lxc network create lxdbr0

This will create a bridge device with the given name and also start-up a dedicated instance of dnsmasq which will act as DNS and DHCP server for the container network.

A big advantage of LXD in comparison to plain LXC is a feature called container profiles. There you can define settings which should be applied to a new container instance. In our case, we now want containers to use the network bridge created before or any other network bridge which was created independently. For this it will be added to the “default” profile which is applied by default when creating a new container:

$ lxc network attach-profile lxdbr0 default eth0

The eth0 is the network device name which will be used inside the container. We could also add multiple network bridges or create multiple profiles ( lxc profile create newprofile) with different network settings.

Create a container Finally we have the most important pieces together to launch a container. A container is always instantiated from an image. The LXC projects provides an image repository with a big number of prebuilt container images pre-configured under the remote name images:. The images are regular LXC containers created via upstream lxc-create script using the various distribution templates. To list the available images run:

$ lxc image list images:

If you found an image you want to run, it can be started as following. Of course in my example I will use a Fedora 24 container (unfortunately there are no Fedora 25 containers available yet, but I’m also working on that):

$ lxc launch images:fedora/24 my-fedora-container

With the following command you can create a console session into the container:

$ lxc exec my-fedora-container /bin/bash

I hope this short guide made you curious to try LXD on Fedora. I’m glad to hear some feedback via comments or Email if you find this guide or the my COPR repository useful or if you have some corrections or found some issues.

Further reading If you want to know more about how to use the individual features of LXD, I can recommend the how-to series of Stéphane Graber, one of the core developers of LXC/LXD:

Managing DNS zones with Ansible

Fri, 23 Sep 2016 16:44:04 +0200

Currently, I’m working on automating the setup of a authoritative DNS server, namely gdnsd. There are many nice features in gdnsd, but what might be interesting for you, that it requires the zone data to be in the regular RFC1035 compliant format. This is also true for bind, probably the most widely used DNS server, therefore the approach explained here, could be also used for bind. Again I wanted to use Ansible as automation framework, not only to setup and configure the service, but also to generate the DNS zone file. One reason for this is, because gdnsd doesn’t support zone transfers, therefore Ansible should be used as synchronization mechanism and because in my opinion the JSON-based inventory format is a simple, generic but very powerful data interface. Especially when considering the dynamic inventory feature of Ansible, one is completely free where and how to actually store the configuration data.

There are already a number of Ansible bind roles available, however they mostly use a very simple approach when it’s about configuring the zone file and its serial. When generating zone files with an automation tool the trickiest part is the handling of the serial number, which has to be increased on every zone update. I’d like to explain what solutions I implemented to solve this challenge.

Zone data generation must be idempotent One strength of Ansible is, that it can be run over and over again and only ever changes something in the system if the current state is not as desired. In my context this means, that the zone file only needs to be updated if the zone data from the inventory has changed. Therefore, also the serial number only ever has to be updated in that case. But how to know if the data has changed?

Using the powerful Jinja2 templating engine, I defined dictionary and assigned every value which would later go into the zone file. Then, I’ll create a checksum over the dictionary content and save it as comment into the zone file. If the checksum changed, the serial has to be updated. Otherwise the zone file includes the old serial and nothing changed. In practice this would look like this:

Read the hash and serial which are saved as comment in the existing zone file and register a temporary variable. It will be empty if the zone file doesn’t exist yet:
```
- name: Read zone hash and serial
  shell: 'grep "^; Hash:" /etc/gdnsd/zones/example.com || true'
  register: gdnsd__register_hash_and_serial
  [...]
```

Define a task which will update the zone file:

- name: Generate forward zones
  template:
    src: 'etc/gdnsd/zones/forward_zone.j2'
    dest: '/etc/gdnsd/zones/example.com'
    [...]

In the template, create a dictionary holding the zone data:

{% set _zone_data = {} %}
{% set _ = _zone_data.update({'ttl': item.ttl}) %}
{% set _ = _zone_data.update({'domain': 'example.com'}) %}
[...]

Create an intermediate variable _hash_and_serial holding the hash and serial read from the zone file before:
```
{% set _hash_and_serial = gdnsd__register_hash_and_serial.stdout.split(' ')[2:] %}
```
Create a hash from the final _zone_data dictionary, compare it with the hash (first element) in _hash_and_serial. If the hashes are equal set the serial as read before (second element) in _hash_and_serial. Otherwise set a new serial which was previously saved in gdnsd__fact_zone_serial (see following section):
```
{% set _zone = {'hash': _zone_data | string | hash('md5')} %}
{% if _hash_and_serial and _hash_and_serial[0] == _zone['hash'] %}
{%   set _ = _zone.update({'serial': _hash_and_serial[1]}) %}
{% else %}
{%   set _ = _zone.update({'serial': gdnsd__fact_zone_serial}) %}
{% endif %}
```
Safe the final hash and serial as comment to the zone file:
```
; Hash: {{ _zone['hash'] }} {{ _zone['serial'] }}
```

Identical zone serial on distributed servers I didn’t explain yet, how gdnsd__fact_zone_serial is defined. Initially, I simply had ansible_date_time.epoch, which corresponds the Unix time, assigned to the serial. This is the simplest way to make sure the serial is numerical and each zone update results in an increased value. However, in the introduction I also mentioned the issue of distributing the zone files between a set of DNS servers. Obviously, if they have the same zone data, they must also have the same serial.

To make sure multiple servers are using the same serial for a zone update, the serial is not computed individually in each template task execution, but once for each playbook run. In Ansible, one can specify that a task must only run once, even the playbook is executed on multiple servers. Therefore I defined such a task to store the Unix time in the temporary fact gdnsd__fact_zone_serial which is used in the zone template on all servers:

- name: Generate serial
  set_fact:
    gdnsd__fact_zone_serial: '{{ ansible_date_time.epoch }}'
  run_once: True

This approach is still not perfect. It won’t compare the two generated zone files between a set of servers. So you have to make sure that the zone data in the inventory is the same for all servers. Also, if you update the servers individually, the serial is generated twice and therefore are different, even when the zone data is identical. At the moment I can’t see any elegant approach to solve those issues. If you have some ideas, please let me know…

The example code listed above is a simplified version of my real code. If you are interested in the entire role, have a look at github.com: ganto/ansible-gdnsd. I hope this could give you some useful examples for using some of the more advanced Ansible features in a real-world scenario.

"Let's encrypt" with acme-tiny

Mon, 05 Sep 2016 18:47:29 +0200

Most of my readers must have heard about the “Let’s encrypt” public certificate authority (CA) by now. For those who haven’t: About two years ago, the Internet Security Research Group (ISRG), a public benefit group, supported by the Electronic Frontier Foundation (EFF), the Mozilla Foundation, Cisco, Akamai, the Linux Foundation and many more started the challenge to create a fully trusted public key infrastructure which can be used for free by everyone. Until then, the big commercial certificate authorities such as Comodo, Symantec, GlobalSign or GoDaddy dominated the market of SSL certificates which prevented a wide use of trusted encryption. The major goal of the ISRG is to increase the use of HTTPS for Web sites from then less than 40 percent two years ago to a 100 percent. One step to achieve this, is by providing certificates to everyone for free and the other step, to do this in a fully automated way. For this reason a new protocol called Advanced Certificate Management Environment (ACME) was designed and implemented. Going forward to today: The “Let’s encrypt” CA issued already more than five million certificates and the use of HTTPS is increasing to around 45 percent in June 2016.

acme-tiny is a small Python script which can be used to submit the certificate request to the “Let’s encrypt” CA. If you’re eligible to request a certificate for this domain you instantly get the certificate back. As such a certificate is only valid for 90 days and the renewal process doesn’t need any user interaction it’s a perfect opportunity for a fully automated setup.

Since a while I prefer Ansible for all kind of automation tasks. “Let’s encrypt” finally allows me to secure new services, which I spontaneously decide to host on my server via sub-domains. To ease the initial setup and fully automate the renewal process, I wrote an Ansible role ganto.acme_tiny. It will run the following tasks:

Generate a new RSA key if none is found for this domain
Create a certificate signing request
Submit the certificate signing request with help of acme-tiny to the “Let’s encrypt” CA
Merge the received certificate with the issuing CA certificate to a certificate chain which then can be configured for various services
Restart the affected service to load the new certificate

In practice, this would look like this:

Create a role variable file /etc/ansible/vars/mail.linuxmonk.ch.yml: acme_tiny__domain: [ 'mail.linuxmonk.ch', 'smtp.linuxmonk.ch' ] acme_tiny__cert_type: [ 'postfix', 'dovecot' ]
Make sure the involved service configurations load the certificate and key from the correct location (see ganto.acme_tiny: Service Configuration).
Run the playbook with the root user to do the initial setup: $ sudo ansible-playbook \ -e @/etc/ansible/vars/mail.linuxmonk.ch.yml \ /etc/ansible/playbooks/acme_tiny.yml

That’s it. Both SMTP and IMAP are now secured with help of a “Let’s encrypt” certificate. To setup automated certificate renewal I only have to add the executed command in a task scheduler such as cron from where it will be executed as unprivileged user acmetiny which was created during the initial playbook run. E.g. in /etc/cron.d/acme_tiny:

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

@monthly acmetiny /usr/bin/ansible-playbook -e @/etc/ansible/vars/mail.linuxmonk.ch.yml /etc/ansible/playbooks/acme_tiny.yml >/dev/null

If you became curious and want to have a setup like this yourself, checkout the extensive documentation about the Ansible role at Read the Docs: ganto.acme_tiny.

This small project was also a good opportunity for me, to integrate all the nice free software-as-a-service offers the Internet is providing for a (Ansible role) developer nowadays:

The code “project” is hosted and managed on Github.
Every release and pull request is tested via the Travis-CI continuous integration platform. It makes use of the rolespec Ansible role testing framework for which a small test suite has been written.
Ansible Galaxy is used as a repository for software distribution.
The documentation is written in a pimped version of Markdown, rendered via Sphinx and hosted on Read the Docs from where it can be accessed and downloaded in various formats.

That’s convenient!

Policy Routing on Linux based on Sender MAC Address

Thu, 21 Apr 2016 07:27:52 +0200

I recently had the task to setup and test a new Linux Internet gateway host as a replacement to an existing router. The setup is classical with some individual ports such as HTTP being forwarded with DNAT to some backend systems.

Redundant router setup requiring policy routing

The new router, from now on I will call it router #2, should be tested and put into operation without downtime. Obviously this means that I had to run the two routers in parallel for a while. The backend systems however, only know one default gateway. Accessing a service through a forwarded port on router #2 resulted in a timeout, as the backend system sent the replies to the wrong gateway where they were dropped.

Fortunately iptables and iproute2 came to my rescue. They enable you to implement policy routing on Linux. This means that the routing decision is not (only) made based on the destination address of a packet as in regular routing, but additional rules are evaluated. In my case: Every connection opened through router #2 has to be replied via router #2.

Using iptables/iproute2 for this task this means: Incoming packets with the source MAC address from router #2 are marked with help of the iptables ‘mark’ extension. The iptables ‘connmark’ extension will then help to associate outgoing packets to the previously marked connection. Based on the mark of the outgoing packet a custom routing policy will set the default gateway to router #2. Easy, eh?

Configuration Now I’ll show some commands how this can be accomplished. The following commands assume that the iptables rule set is still empty and are for demonstration purpose only. Likely they have to be adjusted slightly in a real configuration.

First the routing policy will be setup:

Define a custom routing table. There exist some default tables, so the custom entry shouldn’t overlap with those. For better understanding I will call it ‘router2’:

# echo "200 router2" >> /etc/iproute2/rt_tables
Add a rule to define the condition which packets should lookup the routing in the previously created table ‘router2’:

# ip rule add fwmark 0x2 lookup router2

This means that IP packets with the mark ‘2’ will be routed according to the table ‘router2’.
Set the default gateway in the ‘router2’ table to the IP address of router #2 (e.g. 10.0.0.2):

# ip route add default via 10.0.0.2 table router2
To make sure the routing cache is rebuilt, it need to be flushed after changes: # ip route flush cache

Afterwards I had to make sure that the involved connections coming from router #2 are marked appropriately (above the mark ‘2’ was used). The ‘mangle’ table is a part of the Linux iptables packet filter and meant for modifying network packets. This is the place where the packet markings will be set.

The first iptables rule will match all packets belonging to a new connection coming from router #2 and sets the previously defined mark ‘2’:

# iptables --table mangle --append INPUT \ --protocol tcp --dport 80 \ --match state --state NEW \ --match mac --mac-source 52:54:00:c2:a5:43 \ ! --source 10.0.0.0/24 \ --jump MARK --set-mark 0x2

The packets being marked are restricted to meet the following requirements:
- being sent by the network adapter of router #2 ( --mac-source)
- don’t originate in the local network ( ! --source)
- target destination port 80 ( --dport: example for the HTTP port being forwarded by router #2)
- belong to a new connection ( --state NEW) Of course additional (or less) extensions can be used to filter the packets according to individual requirements.
Next, the incoming packets are given to the ‘connmark’ extension which will do the connection tracking of the marked connections:

# iptables --table mangle --append INPUT \ --jump CONNMARK --save-mark
The packets which can be associated with an existing connection are also marked accordingly:

# iptables --table mangle --append INPUT \ --match state --state ESTABLISHED,RELATED \ --jump CONNMARK --restore-mark
All the previous rules where required that the outgoing packets can finally be marked too:

# iptables --table mangle --append OUTPUT \ --jump CONNMARK --restore-mark

Debugging The following commands and iptables rules should help when setting up and/or debugging policy routing with marked packets:

List policy of routing table ’table2’:

# ip route show table router2
List defined routing tables:

# cat /etc/iproute2/rt_tables
Log marked incoming/outgoing packets to syslog:

`# iptables -A INPUT -m mark –mark 0x2 -j LOG

iptables -A OUTPUT -m mark –mark 0x2 -j LOG`

Setup remote server management of a SuperMicro BMC with FreeIPMI on Linux

Thu, 03 Dec 2015 21:46:00 +0100

SuperMicro server mainboards often include a dedicated Baseband Management Controller (BMC) which offers out-of-band management of the server system. This is a dedicated embedded chip (Specifications) which allow to power cycle the machine, monitor hardware variables, update firmware, access the operating system console and much more. If you are used to big brand server systems, then you might be more familiar with the term iLO, which is the HP Integrated Lights-Out, or IMM, which is the IBM Integrated Management Module. The basic functionality is always comparable.

FreeIPMI Management Utility

A nice fact is, that most of these out-of-band management solutions support the Intelligent Platform Management Interface (IPMI) specification, which defines an interface for accessing the various functionalities from an operating system (OS) or the network (e.g. for monitoring or OS recovery). For Linux there exists GNU FreeIPMI which is a collection of powerful tools for accessing IPMI-compatible BMCs. It is available via default package manager on all major Linux distributions.

Linux Kernel Configuration As mentioned before, on Linux there are two different ways to access the BMC via IPMI. The first way is directly from the host running on the board connected to the BMC. The connection is done via kernel modules. Make sure you have at least the ipmi_defintf and ipmi_si modules loaded. If your kernel doesn’t include this modules, you might need to add them to your kernel configuration. The corresponding settings can be found under:

Device Drivers --->
   Character Devices --->
      <M> IPMI top-level message handler --->
          [ ]   Generate a panic event to all BMCs on a panic (NEW)
          <M>   Device interface for IPMI (NEW)
          <M>   IPMI System Interface handler (NEW)
          <M>   IPMI SMBus handler (SSIF) (NEW)
          <M>   IPMI Watchdog Timer (NEW)
          <M>   IPMI Poweroff (NEW)

You can check if it’s working by invoking ipmi-detect which should then output something like this:

# ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

[...]

There was an IPMI 2.0 compatible chip found, so we are good to go.

Network Configuration The second way to connect via IPMI to the BMC is over the network. For this, there often is a dedicated network port on the mainboard that obviously needs to be connected to a network switch over which you can access it. By default the SuperMicro BMC is configured to get an IP address via DHCP. If the local IPMI connection is working, you can get the IP address by executing:

# ipmi-config --checkout --section Lan_Conf

#
# Section Lan_Conf Comments
#
# In the Lan_Conf section, typical networking configuration is setup. Most users
# will choose to set "Static" for the "IP_Address_Source" and set the
# appropriate "IP_Address", "MAC_Address", "Subnet_Mask", etc. for the machine.
#
Section Lan_Conf
        ## Possible values: Unspecified/Static/Use_DHCP/Use_BIOS/Use_Others
        IP_Address_Source                             Use_DHCP
        ## Give valid IP address
        IP_Address                                    192.168.1.15
        ## Give valid MAC address
        MAC_Address                                   0C:C4:7A:73:18:4F
        ## Give valid Subnet Mask
        Subnet_Mask                                   255.255.255.0
        ## Give valid IP address
        Default_Gateway_IP_Address                    192.168.1.1
        ## Give valid MAC address
        Default_Gateway_MAC_Address                   00:0D:B9:22:15:A2
        ## Give valid IP address
        Backup_Gateway_IP_Address                     0.0.0.0
        ## Give valid MAC address
        Backup_Gateway_MAC_Address                    00:00:00:00:00:00
EndSection

Otherwise you might need to search the DHCP server log for an entry, which IP address was given to the BMC. I recommend to set the BMC IP address to a fixed value, to minimize the risk to loose connectivity in case of major data center issue. This can be done via Web interface at e.g. https://192.168.1.15 or via ipmi-config:

# ipmi-config --commit -e Lan_Conf:IP_Address_Source=Static
# ipmi-config --commit -e Lan_Conf:IP_Address=192.168.1.42

Alternatively you can write the configuration in file and commit it with the --filename argument. E.g. create a static-network.conf:

# Static network configuration for the BMC
Section Lan_Conf
        IP_Address_Source                             Static
        IP_Address                                    192.168.1.42
EndSection

Unfortunately the IP_Address_Source key cannot be changed concurrently with other Lan_Conf keys. So we had to run the commit command twice, when specifying the new values on the command line and the same is true with the configuration file option. The first time it will set the IP_Address_Source:

# ipmi-config --commit --filename=static-network.conf

And the second time, it will set the new IP_address:

# ipmi-config --commit --filename=static-network.conf

However, this is an exception and most other key/value pairs can be changed concurrently. See the ipmi-config(8) man-page for more details.

Eventually we can check if the IPMI interface is reachable via network. This can be done with the ipmi-ping tool. If the --verbose argument is added, it will even output some information about the configured authentication settings. E.g.:

# ipmi-ping --count 1 --verbose 192.168.1.42
ipmi-ping 192.168.1.42 (192.168.1.42)
response received from 192.168.1.42: rq_seq=23, auth: none=clear md2=set md5=set password=set oem=clear anon=clear null=clear non-null=set user=clear permsg=clear
--- ipmi-ping 192.168.1.42 statistics ---
1 requests transmitted, 1 responses received in time, 0.0% packet loss

Configure Serial-over-LAN (SoL) Console Access

A neat feature of a BMC is the ability to remotely access the Linux console even when the regular network connection to the server is not working any more. However, this needs some additional configuration in the operating system.

Check SoL Settings in BIOS First, we will check the BIOS settings to make sure the SoL feature is enabled. Note that accessing the BIOS usually already works in the default configuration, so if you are adventurous, you can already connect over remote IPMI (for demonstration purpose I’ll use the default user ADMIN):

$ ipmi-console -h 192.168.1.42 -u ADMIN -P

By default you won’t have access to the Linux system yet, so reboot the system via SSH so the BIOS can be setup accordingly. Unfortunately, the IPMI console won’t show the system’s POST output, so you have to keep pressing the DEL button in the terminal with the open IPMI console until you see the BIOS main screen. Then go to Advanced -> Serial Port Console Redirection and make sure, the SOL Console Redirection is enabled. Also remember how many additional serial ports you have available for configuration in this menu. This will become important when selecting the correct console in the Linux configuration. E.g. my board only has one other serial port (COM1) available and for this one console redirection one mustn’t be enabled for SoL to work correctly.

SuperMicro BIOS settings accessed via ipmi-console

If you want to see more verbose POST output, also disable Quiet Boot in the Advanced -> Boot Feature menu.

Leave the BIOS by pressing Save Changes and Reset. After a short while you should be greeted by the Grub menu. However, there won’t be any more output, as Linux doesn’t know yet, where to connect to the SoL console. To exit the IPMI console, you have to press &..

Setup Serial Console in Linux If you have a systemd-based distribution all you need to do is to modify the kernel command line and add a serial console. Edit /etc/default/grub and extend the GRUB_CMDLINE_LINUX variable as following:

GRUB_CMDLINE_LINUX="rd.lvm.lv=vg00/swap rd.lvm.lv=vg00/slash rhgb quiet console=tty0 console=ttyS1,115200n8"

Important: This has to be /dev/ttyS1 in case the BIOS knows only knows one additional serial port (which would correspond to /dev/ttyS0). If you have two additional serial ports in the BIOS (COM1/COM2), you must set /dev/ttyS2 here. Don’t configure Grub 2 to output itself to the serial console, as this is still handled by the default SoL configuration forwarding the initial system output.

Regenerate the grub.cfg with:

# grub2-mkconfig -o /boot/grub2/grub.cfg

You might need to adjust the command and grub.cfg path according to your installation. The next time when the system is booted, the Linux system output will be available on the serial terminal and there will also be a login prompt. If you don’t have a systemd-based system, you additionally need to enable the serial console login terminal in the /etc/inittab. E.g.:

T1:23:respawn:/sbin/getty -L ttyS1 115200 vt100

Remotely Manage the System over IPMI

Now everything we need is in place. We can independently from the Linux operating system or the local system console:

Configure the BMC via ipmi-config (see above)
Access the Linux console via ipmi-console
Query system sensor values (temperatures, voltages, …) via ipmi-sensors
Query system event log via ipmi-sel
Initiate system startup and power reset via ipmi-power

Even tough I have the latest available IPMI firmware installed, it sometimes happened to me, that the SoL wouldn’t properly connect to the Linux console any more after I disconnected when being in the BIOS. Or that I could navigate through the BIOS but it wouldn’t refresh the screen. Unfortunately the only way that I found to fix this, was to reboot the BMC. Of course also this is possible over IPMI:

$ bmc-device -h 192.168.1.42 -u ADMIN -P --cold-reset

There are many other things that you can do via IPMI. Also there are other tools which can access IPMI-enabled BMCs. Another famous one is ipmitool. You might want to checkout Adam Sweet’s wiki on IPMI on Linux for more details how to use ipmitool. Personally, I prefer FreeIPMI for being a GNU project, being very intuitive to use and so far it worked perfectly well for everything I needed.

If you have some hints or additions, please leave a comment below. Thanks for reading.

Migrate 32bit physical host on RAID 1 to KVM/Xen virtual machine

Tue, 01 Dec 2015 01:52:07 +0100

I recently had the challenge to migrate a physical host running Debian, installed on a ancient 32bit machine with a software RAID 1, to a virtual machine (VM). The migration should be as simple as possible so I decided to attach the original disks to the virtualization host and define a new VM which would boot from these disks. This doesn’t sound too complicated but it still included some steps which I was not so familiar with, so I write them down here for later reference. As an experiment I tried the migration on a KVM and a Xen host.

Prepare original host for virtualized environment

Enable virtualized disk drivers The host I wanted to migrate was running a Debian Jessie i686 installation on bare metal hardware. For booting it uses an initramfs which loads the required disk controller drivers. By default the initramfs only includes the drivers for the current hardware, so the paravirtualization drivers for KVM ( virtio-blk) or Xen ( xen-blkfront) are missing. This can be changed by adjusting the /etc/initramfs-tools/conf.d/driver-policy file to state that not the currently dependent ( dep) modules should be loaded, but most:

# Driver inclusion policy selected during installation
# Note: this setting overrides the value set in the file
# /etc/initramfs-tools/initramfs.conf
MODULES=most

Afterwards the initramfs must be rebuilt by running the following command (adjust the kernel version according to the kernel you are running):

# dpkg-reconfigure linux-image-3.16.0-4-686-pae

Now the initramfs contains all required modules to successfully boot the host from a KVM virtio or a Xen blockfront disk device. This can be checked with:

# lsinitramfs /boot/initrd.img-3.16.0-4-686-pae | egrep -i "(xen|virtio).*blk.*\.ko"
lib/modules/3.16.0-4-686-pae/kernel/drivers/block/virtio_blk.ko
lib/modules/3.16.0-4-686-pae/kernel/drivers/block/xen-blkback/xen-blkback.ko
lib/modules/3.16.0-4-686-pae/kernel/drivers/block/xen-blkfront.ko

Enable serial console To easily access the server console from the virtualization host, it’s required to setup a serial console in the guest, which must be defined in /etc/default/grub:

[...]
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
[...]
# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL="console serial"

After regenerating the Grub configuration, the boot loader and local console will then also be accessible via virsh console:

# grub-mkconfig -o /boot/grub/grub.cfg

Run the host on a KVM hypervisor

KVM is nowadays the de-facto default virtualization solution for running Linux on Linux and is very well integrated in all major distributions and a wide range of management tools. Therefore it’s really simple to boot the disks into a KVM VM:

Make sure the disks are not already assembled as md-device on the KVM host. I did this by uninstalling the mdadm utility.

Run the following virt-install command to define and start the VM:

# virt-install --connect qemu:///system --name myhost --memory 1024 --vcpus 2 --cpu host --os-variant debian7 --arch i686 --import --disk /dev/disk/by-id/ata-WDC_WD10EFRX-68PJCN0_WD-WCC4J4527214 --disk /dev/disk/by-id/ata-WDC_WD10EFRX-68PJCN0_WD-WCC4J4366412 --network bridge=virbr0 --graphics=none

To avoid confusion with the disk enumeration of the KVM host kernel, the disk devices are given as unique device paths instead of e.g. /dev/sdb and /dev/sdc. This clearly identifies the two disk devices which will then be assembled to a RAID 1 by the guest kernel.

Run the host on a Xen hypervisor

Xen has been a topic in this blog a few times before as I’m still using it since its early days. After a long time of major code refactorings it also made huge steps forward within the last few years, so that nowadays it’s nearly as easy to setup as KVM and again well supported in the large distributions. Because its architecture is a bit different and Linux is not the only supported platform some things work not quite the same as under KVM. Let’s find out…

Prepare Xen Grub bootloader Xen virtual machines can be booted in various different ways, also depending whether the guest disk is running on an emulated disk controller or as paravirtualized disk. In simple cases, this is done with help of PyGrub loading a Grub Legacy grub.conf or a Grub 2 grub.cfg from a guests /boot partition. As my grub.cfg and kernel is “hidden” in a RAID 1 a fully featured Grub 2 is required. It is perfectly able to load its configuration from advanced disk setups such as RAID arrays, LVM volumes and more. With Xen such a setup is only possible when using a dedicated boot loader disk image which must be in the target architecture of the virtual machine. As I want to run a 32bit virtual machine, a i386-xen Grub 2 flavor needs to be compiled as basis for the boot loader image:

Clone the Grub 2 source code:

$ git clone git://git.savannah.gnu.org/grub.git

Configure and build the source code. I was running the following commands on a Gentoo Xen server without issues. On other distributions you might need to install some development packages first:
```
$ cd grub
$ ./autogen.sh
$ ./configure --target=i386 --with-platform=xen
$ make -j3
```
Prepare a grub.cfg for the Grub image. The Grub 2 from the Debian guest host was installed on a mirrored boot disk called /dev/md0 which was mounted to /boot. Therefore the Grub image must be hinted to load the actual configuration from there:
```
# grub.cfg for Xen Grub image grub-md0-i386-xen.img
normal (md/0)/grub/grub.cfg
```

Create the Grub image using the previously compiled Grub modules:

# grub-mkimage -O i386-xen -c grub.cfg -o /var/lib/libvirt/images/grub-md0-i386-xen.img -d ~user/grub/grub-core ~user/grub/grub-core/*.mod

A more generic guide about generating Xen Grub images can be found at Using Grub 2 as a bootloader for Xen PV guests.

Define the virtual machine virt-install can also be used to import disk (images) as Xen virtual machines. This works perfectly fine for e.g. importing Atomic Host raw images, but unfortunately the setup now is a bit more complicated. It requires me to specify the custom Grub 2 image as guest kernel for which I couldn’t find a corresponding virt-install argument (using v1.3.0). Therefore I first created a plain Xen xl configuration file and then converted it to a libvirt XML file. The initial configuration /etc/xen/myhost.cfg looks as following:

name = "myhost"
kernel = "/var/lib/libvirt/images/grub-md0-i386-xen.img"
memory = 1024
vcpus = 2
vif = [ 'bridge=virbr0' ]
disk = [ '/dev/disk/by-id/ata-WDC_WD10EFRX-68PJCN0_WD-WCC4J4527214,raw,xvda,rw', '/dev/disk/by-id/ata-WDC_WD10EFRX-68PJCN0_WD-WCC4J4366412,raw,xvdb,rw' ]

Eventually this configuration can be converted to a libvirt domain XML with the following command:

# virsh domxml-from-native --format xen-xl --config /etc/xen/myhost.cfg > /etc/libvirt/libxl/myhost.xml
# virsh --connect xen:/// define /etc/libvirt/libxl/myhost.xml

Summary

At the end, I successfully converted the bare metal host to a KVM and/or Xen virtual machine. Once again Xen ended up being a bit more troublesome, but no nasty hacks were required and both configurations were quite straight forward. The boot loader setup for Xen could definitely be a bit simpler, but still it’s nice to see how well Grub 2 and Xen play together now. It’s also pleasing to see, that no virtualization specific configurations had to be made within the guest installation. The fact that the disk device names were changing from /dev/sd[ab] (bare metal) to /dev/vd[ab] (KVM) and /dev/xvd[ab] (Xen) was completely absorbed by the mdadm layer, but even without software RAID the /etc/fstab entries are nowadays generated by the installers in a way, that such migrations are easy possible.

Thanks for reading. If you have some comments, don’t hesitate to write a note below.

Kerberized HTTP Auth on Apache/mod_ssl with FreeIPA

Thu, 28 Aug 2014 00:20:03 +0200

Today I just found out, how super easy it is to setup a safe HTTP authentication via Kerberos with help of FreeIPA. Having the experience of managing a manually engineered MIT Kerberos/ OpenLDAP/ EasyRSA infrastructure, I’m once again blown away by the simplicity and usability of FreeIPA. I’ll describe with only a few commands which can be run within less than 10 minutes how it’s possible to setup a fully featured Kerberos-authenticated Web server configuration. Prerequisite is a FreeIPA server (a simple guide for installation can be found for example here) and a RedHat-based Web server host (RHEL, CentOS, Fedora).

Required Packages: First we are going to install the required RPM packages:

# yum install httpd mod_auth_kerb mod_ssl ipa-client

Register the Web server host at FreeIPA: Make sure the Web server host is managed by FreeIPA:

ipa-client-install --domain=example.com --server=ipaserver.example.com --realm=EXAMPLE.COM --mkhomedir --hostname=webserver.example.com --configure-ssh --configure-sshd

Create a HTTP Kerberos Principal and install the Keytab: The Web server is identified in a Kerberos setup through a keytab, which has to be generated and installed on the Web server host. First make sure that you have a valid Kerberos ticket of a FreeIPA account with enough permissions (e.g. ‘admin’):

# kinit admin
# ipa-getkeytab -s ipaserver.example.com -p HTTP/webserver.example.com -k /etc/httpd/conf/httpd.keytab

This will create a HTTP service principal in the KDC and install the corresponding keytab in the Apache httpd configuration directory. Just make sure that it can be read by the httpd server account:

# chown apache /etc/httpd/conf/httpd.keytab

Create a SSL certificate No need to fiddle around with OpenSSL. Requesting, signing and installing a SSL certificate with FreeIPA is one simple command:

# ipa-getcert request -k /etc/pki/tls/private/webserver.key -f /etc/pki/tls/certs/webserver.crt -K HTTP/webserver.example.com -g 3072

This will create a 3072 bit server key, generate a certificate request, send it to the FreeIPA Dogtag CA, sign it and install the resulting PEM certificate on the Web server host.

Configure Apache HTTPS The httpd setup is the only and last configuration which needs to be done manually. For HTTPS set the certificate paths in /etc/httpd/conf.d/ssl.conf:

[...]
SSLCertificateFile /etc/pki/tls/certs/webserver.crt
SSLCertificateKeyFile /etc/pki/tls/private/webserver.key
SSLCertificateChainFile /etc/ipa/ca.crt

Additionally do some SSL stack hardening (you may also want to read this):

[...]
SSLCompression off
SSLProtocol all -SSLv2 -SSLv3 -TLSv1.0
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4"

Kerberos HTTP Authentication: The final httpd authentication settings for ‘mod_auth_kerb’ are done in /etc/httpd/conf.d/auth_kerb.conf or any vhost you want:

<Location />
  SSLRequireSSL
  AuthType Kerberos
  AuthName "Kerberos Login"
  KrbMethodNegotiate On
  KrbMethodK5Passwd On
  KrbAuthRealms EXAMPLE.COM
  Krb5KeyTab /etc/httpd/conf/httpd.keytab
  require valid-user
</Location>

That’s it! After restarting the Web server you can login on https://webserver.example.com with your IPA accounts. If you don’t already have a valid Kerberos ticket in the Web client, KrbMethodNegotiate On enables interactive password authentication.

Troubleshooting In case you get the following error message in the httpd error log, make sure the keytab exists and is readable by the httpd account (e.g. ‘apache’):

[Wed Aug 27 07:23:04 2014] [debug] src/mod_auth_kerb.c(646): [client 192.168.122.1] Trying to verify authenticity of KDC using principal HTTP/webserver.example.com@EXAMPLE.COM
[Wed Aug 27 07:23:04 2014] [debug] src/mod_auth_kerb.c(689): [client 192.168.122.1] krb5_rd_req() failed when verifying KDC

Quickly setup a Fedora 19 boot disk for serial-only console access

Fri, 16 May 2014 00:01:11 +0200

I recently bought a PC Engines APU1C4 x86 embedded board which is meant to be the board for my future custom NAS box. In comparison to the various ARM boards it promises to be powerful and I/O friendly (3x Gbit LAN, SATA, 3x mini PCIe) and doesn’t include redundant graphics and sound circuits. On the other hand the only way to locally access it is via a serial terminal. Before installing the final system, hopefully more about this in a later article, I wanted to have a quick glance at the system from a Linux point of view. I tried booting the device over an USB stick prepared with my favorite live system SystemRescueCD, which by the way is based on Gentoo, but somehow failed as the boot process didn’t support output on a serial device and never spawned a terminal on it either. Before loosing too much time in searching for another media which would support a serial console, I simply setup my own minimal boot system based on Fedora 19. Here will follow a quick summary on what was required to achive this, as I couldn’t find a good and recent how-to about such a setup either. Because this minimal system is meant for ad-hoc booting only, I will keep things as simple as possible.

Prepare the installation medium

The APU1C4 supports booting over all possible storage devices, so you need to have a spare USB stick, external USB disk, mSATA disk, SATA disk or a SD card for storing the minimal Linux installation. Create at least one partition with a Linux file system of your choice on it and mount it. This will be the root directory of the new system. The following example will show how the setup is done on a device /dev/sdb with one partition mounted to /mnt/usbdisk:

host # mount /dev/sdb1 /mnt/usbdisk

Bootstrap minimal Fedora system in alternative root directory

Redhat-based distributions have an easy way to install a new system to an alternative root directory. Namely, it can be done with the main package manager yum. To keep it easy I used a Fedora 19 host system to setup the boot disk. While being in the context of the host system (below indicated with ’ host #’), always be careful that your commands are actually modifying the content under /mnt/usbdisk. Otherwise you might have a bad surprise when you reboot your host system the next time.

1. Prepare RPM database:

host # mkdir -p /mnt/usbdisk/var/lib/rpm
host # rpm --root /mnt/usbdisk/var/lib/rpm --initdb

2. Install Fedora release package:

host # yumdownloader --destdir=/tmp fedora-release
host # rpm --root /mnt/usbdisk -ivh /tmp/fedora-release*rpm

3. Install a minimal set of packages (add whatever packages you’d like to have in the minimal system):

host # yum --installroot=/mnt/usbdisk install e2fsprogs kernel \
rpm yum grub2 openssh-client openssh-server passwd less rootfiles \
vim-minimal dhclient pciutils ethtool dmidecode

4. Copy DNS resolver configuration:

host # cp -p /etc/resolv.conf /mnt/usbdisk/etc

5. Mount pseudo file systems for chroot:

host # mount -t proc none /mnt/usbdisk/proc
host # mount -t sysfs none /mnt/usbdisk/sys
host # mount -o bind /dev /mnt/usbdisk/dev

5. chroot into the new system tree to finalize the installation:

host # chroot /mnt/usbdisk /bin/bash

6. Set root password

chroot # passwd

7. Prepare system configurations:

chroot # echo "NETWORKING=yes" > /etc/sysconfig/network

8. If you only have one partition with the entire system, a fstab is not needed anymore as dracut and Systemd will already know how to mount it. Otherwise create the fstab (use the UUID if you’re not sure how the disk will be called on the target system):

chroot # dumpe2fs -h /dev/sdb1 | grep UUID
dumpe2fs 1.42.7 (21-Jan-2013)
Filesystem UUID: bfb2fba1-774d-4cfc-a978-5f98701fe58a
chroot # cat << EOF >> /etc/fstab
UUID=bfb2fba1-774d-4cfc-a978-5f98701fe58a / ext4 defaults 0 1
EOF

9. Setup Grub 2 for serial console:

chroot # cat << EOF >> /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 rd.lvm=0 rd.md=0 rd.dm=0 rd.luks=0 LANG=en_US.UTF-8 KEYTABLE=us"
GRUB_TERMINAL="serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
GRUB_DISABLE_OS_PROBER=true
EOF

chroot # grub2-install /dev/sdb
chroot # grub2-mkconfig -o /boot/grub2/grub.cfg

10. We’re done. Exit the chroot and unmount it:

chroot # exit
host # umount /mnt/usbdisk/dev
host # umount /mnt/usbdisk/proc
host # umount /mnt/usbdisk/sys
host # umount /mnt/usbdisk

Now you can remove the disk from the host and connect it with the embedded board you want to boot.

Connect to the serial console and start the system

For connecting to the embedded board a USB to serial adapter and a null modem cable is required. There exist a number of tools to connect to a serial console on Linux which you probably already know (e.g. screen or minicom). However, I always found them painful to use. The tool of my choice is called CuteCom and is a graphical serial terminal. After selecting the correct serial device ( /dev/ttyUSB0 in my case) and baud rate, you can power on your device and you’ll hopefully be greeted by the boot messages of your board and the freshly installed Linux system:

If there is no output in the terminal make sure, you use null-modem cable or adapter and not a simple serial extension cable. Further check for the correct serial port device in your serial terminal configuration and play around with the baud rate.

Good luck and have fun with your embedded device. :-)

Make Firefox subscribe RSS feeds with TinyTinyRSS

Sun, 04 May 2014 02:00:23 +0200

Since Google Reader shutdown its service in the middle of last year, Tiny Tiny RSS is one of the better alternatives that one could use for aggregating and reading RSS feeds. Next to the official Android app which unfortunately isn’t free of charge, there is an alternative called TTRSS-Reader which even supports offline synchronization. This combination makes me very happy. But one thing that bothered me since a while, was that subscribing to new feeds was very easy during the days of Google Reader. Today I finally figured out how to setup Firefox to automatically subscribe to a Tiny Tiny RSS instance.

First go to the about:config page in your Firefox and search for the keys starting with browser.contentHandlers. Here we can configure a title and a URI which is later given as choice when subscribing to a feed. E.g:

browser.contentHandlers.types.1.title: My Tiny Tiny RSS
browser.contentHandlers.types.1.type: application/vnd.mozilla.maybe.feed
browser.contentHandlers.types.1.uri: URL of your TT-RSS instance

The subscriber URL for TT-RSS looks like the following. Of course you have to substitute the domain name with your domain:

https://example.com/ttrss/public.php?op=subscribe&feed_url=%s

At the end, the Firefox configuration should look similar to this screenshot:

After the configuration is done, you have to restart Firefox. If you then browse to a Web site with a RSS feed and you click on it, the Firefox feed subscribtion page will appear. There you eventually can select the newly created entry for Tiny Tiny RSS:

After selecting the entry, you’ll be redirected to your Tiny Tiny RSS instance, where you can configure the feed settings such as category and more.

With this last piece of configuration, there is definitely nothing left anymore that makes me miss the time with Google Reader. :-)

10 years ago I switched to Gentoo Linux as my main operating system

Sun, 20 Apr 2014 13:11:50 +0200

Finally the day arrived, that I can say, I’m a Linux user since 10 years. On April 19 2004, I started my stage1 Gentoo Linux installation on my main workstation box. Since then, I changed my main desktop to Gnome, I adapted my work flow according to what’s possible with open source tools and I never looked back again to my old time with proprietary Windows tools.

However, I already made my first Linux experience a few years earlier than 2004. As far as I still remember, my first Linux distribution ever was SUSE Linux 7.0, which I was able to install without problems on an old computer of mine. I was kind of lost with this new concept that every small functionality is a program by itself, and why are these applications called so weird anyway? A bit later, I wanted to build a dedicated game and file server for LAN parties and it should run with an easy to maintain and lightweight alternative to Windows. The component that I was proud of the most in this server was a Promise SX4000 RAID controller which supported RAID5. I made sure that it was supporting Linux before buying it. At that time I learned the hard way, that Linux support doesn’t magically mean that the driver code is integrated into the upstream Linux kernel and therefore is supported by every Linux distribution. First, there were only some binary modules available, so my distribution for the server was Red Hat Linux 7.2. At that time RPMs were still some kind of annoying black magic to me, as there was no automated dependency resolving available by default. I didn’t know about yum which was optionally available already. Eventually, I never really could make the server work properly in the way how I imagined. Also, I learned from my geek friends, that there is Gentoo, which would be the best Linux distribution anyway. With Gentoo Linux I finally succeeded to install my RAID adapter and I started learning a basic principle, which I still think it’s true today: “If some open source software doesn’t work how you want, you simply don’t try hard enough”. I then also started using Gentoo on my Apple iBook G3, which released me from some Apple Java 1.4 Swing weirdness I experienced during the University programming exercises. These achievements showed me… Linux, ehmn Gentoo Linux, is the way to go. :-)

You may wonder, why I still remember exactly when I originally setup my Gentoo workstation. Simply, because thanks of the rolling release model of Gentoo, I’m still running the same installation since then. I still have my emerge.log around with the entire update history since day one. You want some goodies?

First emerged package:

     Mon Apr 19 14:47:13 2004 >>> sys-apps/portage-2.0.50-r6
       merge time: 28 seconds.

Original toolchain:

     Mon Apr 19 15:00:50 2004 >>> sys-devel/binutils-2.14.90.0.7-r4
       merge time: 4 minutes and 45 seconds.

     Mon Apr 19 15:41:29 2004 >>> sys-devel/gcc-3.3.2-r5
       merge time: 39 minutes and 34 seconds.

     Mon Apr 19 16:09:44 2004 >>> sys-libs/glibc-2.3.2-r9
       merge time: 28 minutes.

Original desktop environment and browser:

     Tue Apr 20 14:51:33 2004 >>> x11-base/xfree-4.3.0-r5
       merge time: 17 minutes and 26 seconds.

     Wed Apr 21 02:10:24 2004 >>> gnome-base/gnome-2.4.2
       merge time: 3 seconds.

     Wed Apr 21 11:18:36 2004 >>> net-www/mozilla-firefox-0.8-r2
       merge time: 42 minutes and 9 seconds.

It’s really interesting to dig around in this file. As you can see, it took a few days to compile the entire system, but at the end, I had a system which satisfied my expectations and still serves me well today.

With help of the emerge.log, I can also make some interesting comparisons on how PC hardware evolved. Building OpenOffice.org back then on a single core AMD Athlon XP and LibreOffice today on a six core AMD Phenom II, which is also rather antique already:

     Thu Sep 23 23:56:56 2004 >>> app-office/openoffice-1.1.2
       merge time: 6 hours, 18 minutes and 15 seconds.

     Sun Feb  9 11:35:57 2014 >>> app-office/libreoffice-4.1.4.2
       merge time: 1 hour, 2 minutes and 32 seconds.

If you are interested in more details of my Gentoo emerge history or if you know some tools to automatically analyze or graph the emerge.log, please leave me a comment below.

So how did I evolve during the last 10 years using Linux? I became an IT professional for Linux engineering, I was witnessing how open source software was gaining acceptance in the most conservative IT environments on one side and driving innovation and efficiency on the other side. This wouldn’t have been possible without Gentoo, which taught me to dig into documentation and community reports to solve the problems. A big thanks to Gentoo and the entire Linux and open source community for all their support and motivation! I had a great time with you for the last 10 years. At the beginning, I could have never imagined that today the majority of people are running a Linux-based mobile phone or that Linux is evolving so rapidly as gaming platform. I’m looking forward to the next 10 years with Linux and open source software…

DHCPv6: Stateful Address Autoconfiguration on Linux (Part 1: radvd)

Thu, 05 Dec 2013 23:33:39 +0100

There already exist many tutorials how to setup a basic IPv6 network environment on Linux. Most of the time they are limited to an example of a radvd.conf enabling Router Advertisement and Stateless Address Auto Configuration (SLAAC), an example how to configure BIND to serve DNS requests for AAAA records and reply to IPv6 reverse lookups. Sometimes even DHCPv6 is mentioned as an alternative way to assign IPv6 client addresses, but mostly with a very basic example configuration without fixed address assignments. Building on this basic knowledge, I want to summarize some further settings which might be interesting, when playing around with DHCPv6 in a Linux environment.

radvd and DHCPv6

As I’ve already mentioned earlier there is no way around radvd, at least if your router is based on Linux. Unlike with IPv4, in IPv6 the router can announce its presence with ICMPv6 Router Advertisement messages, triggered by radvd. By default these messages include the ‘Autonomous’ flag, which enables SLAAC and therefore ask a receiver to autonomously configure its address based on the given IPv6 prefix. In a DHCPv6 subnet you may want to disable this behaviour in the radvd.conf with the following prefix option:

AdvAutonomous off;

There are two more radvd.conf configuration directives which are important in a DHCPv6 setup:

AdvManagedFlag on;
AdvOtherConfigFlag on;

The ManagedFlag option (M flag) will hint the receiver to obtain a stateful address via DHCPv6. The OtherConfigFlag option (O flag) is used to inform the receiver that various other configuration information such as DNS, SIP or NTP server address lists can be requested via DHCPv6. The latter is often also used together with SLAAC, in case a client doesn’t understand RDNSS and DNSSL announcements.

Important: Not all IPv6 clients can handle address assignment if SLAAC is disabled! According to the RFC4294 (IPv6 Node Requirement), a IPv6 client must only support address auto-configuration via SLAAC. DHCPv6 may be supported optionally. Especially Windows XP but also Google Android (see Issue #32621) won’t be able to auto-configure a routable IPv6 address without SLAAC.

Another interesting radvd option not directly related to DHCPv6, but helpful if you like to analyse your network traffic without being distracted by all the Router Advertisement noise is the following interface option:

UnicastOnly on;

This will prevent radvd from broadcasting Router Advertisements and only reply with a unicast message if it receives a Router Solicitation message from a IPv6 host refreshing its routing table. Together with the possibility to only respond to a predefined list of host IPs in the radvd.conf, its even possible to run your router in complete stealth mode towards unknown IPv6 clients:

clients {
    fd41:3fb2:3196:b1bb:52b:4dc0:1631:6626;
    fd41:3fb2:3196:b1bb:9d4a:23c:bff:fe08;
};

radvd and ip6tables

To protect the router you may want to enable ip6tables. Additionally to the default ICMPv6 messages, such as Neighbor Solicitation/Advertisement and Destination Unreachable, which should be allowed on every IPv6 host anyway, the following rules must be configured to whitelist the radvd communication channels:

Allow incoming Router Solicitation (ICMPv6 Type 133) messages:

ip6tables -A INPUT -i <netdevice> -p ipv6-icmp -m icmp6 \
    --icmpv6-type router-solicitation -j ACCEPT

Allow outgoing Router Advertisement (ICMPv6 Type 134) messages:

ip6tables -A OUTPUT -o <netdevice> -p ipv6-icmp -m icmp6 \
    --icmpv6-type router-advertisement -j ACCEPT

That’s it for the moment. In part 2 of my DHCPv6 series, I’ll show you some interesting Linux DHCP server configuration directives. Stay tuned and don’t hesitate to leave a comment in case this article was helpful for you or if I got it all wrong… ;-)

Building a RPM for Oracle JRockit JDK (with mock)

Wed, 01 May 2013 13:07:10 +0200

Since a while, I always wanted to dig into RPM packaging as it would be very useful in my daily work with several hundreds of Red Hat machines. But I didn’t find a challenging software to package since it’s hard to find popular tools not available as RPM or at least SRPM already. This lasted until recently when I had to update Oracle JRockit Java, an enterprise JDK used with the Oracle Weblogic server, on multiple dozens of machines. Accurately defined the default installation of the JDK consists of only one folder which could be tar’ed and copied over, but a real Linux admin knows, this is not the way to install software. After several days of try and error and researching JVM packaging, the result is now available on my GitHub profile.

Download Spec File and Oracle JRockit Installer

The easiest way to get the .spec file is to clone the oracle-jrockit-rpm repository:

[user@host ~]$ git clone https://github.com/ganto/oracle-jrockit-rpm.git

The following files from the repository are then required to build the RPM:

oracle-jrockit-rpm/SOURCES/jrockit-silent.xml oracle-jrockit-rpm/SPECS/java-1.6.0-jrockit.spec

Also download the Oracle JRockit installer, the x64 and ia32 version is supported by the spec file, and place it into the oracle-jrockit-rpm/SOURCES directory.

Use mock to build the RPMs Mock is a useful tool to build RPMs for various target platforms. Even for the Gentoo friends it is available in the portage.

In the first step a chroot environment for the target distribution has to be setup. Mock already comes with a fair number of different definition files for different distributions, which can be found in /etc/mock. They can be adapted according to different requirements, e.g. when a local mirror or a different base set of packages should be used. When building RPMs for RHEL/CentOS 6, I had to modify the epel-6-x86_64.cfg to use the following setup command:

config_opts['chroot_setup_cmd'] = 'install bash bzip2 coreutils cpio diffutils findutils gawk gcc grep sed gcc-c++ gzip info patch redhat-rpm-config rpm-build shadow-utils tar unzip util-linux-ng which make'

After adding the unprivileged build user to the ‘mock’ group, the chroot can be initialized with the following command. In this example I want to build the RPMs for the already mentioned RHEL/CentOS 6 distributions:

[user@host ~]$ mock -r epel-6-x86_64 --init

Next, the SRPM needs to be packaged:

[user@host ~]$ mock -r epel-6-x86_64 --buildsrpm --spec oracle-jrockit-rpm/SPECS/java-1.6.0-jrockit.spec --sources oracle-jrockit-rpm/SOURCES

Eventually, the final RPMs can be compiled:

[user@host ~]$ mock -r epel-6-x86_64 --rebuild /var/lib/mock/epel-6-x86_64/root/builddir/build/SRPMS/java-1.6.0-jrockit-1.6.0.37_R28.2.5_4.1.0-1.el6.src.rpm

If everything went well, the final RPMs can be found under /var/lib/mock/epel-6-x86_64/root/builddir/build/RPMS:

[user@host ~]$ ls -1 /var/lib/mock/epel-6-x86_64/root/builddir/build/RPMS java-1.6.0-jrockit-1.6.0.37_R28.2.5_4.1.0-1.el6.x86_64.rpm java-1.6.0-jrockit-demo-1.6.0.37_R28.2.5_4.1.0-1.el6.noarch.rpm java-1.6.0-jrockit-devel-1.6.0.37_R28.2.5_4.1.0-1.el6.x86_64.rpm java-1.6.0-jrockit-jdbc-1.6.0.37_R28.2.5_4.1.0-1.el6.x86_64.rpm java-1.6.0-jrockit-missioncontrol-1.6.0.37_R28.2.5_4.1.0-1.el6.x86_64.rpm java-1.6.0-jrockit-src-1.6.0.37_R28.2.5_4.1.0-1.el6.x86_64.rpm Final Thoughts

Of course this guide can be used to build any RPM also from other spec files and for other distributions. With these notes, I hope to be more productive when an RPM quickly has to be compiled in the future.

If you find a bug in the spec file, feel free to open an issue on GitHub, so I can fix and learn from it. Otherwise just leave a comment below if you think this guide or the spec file was useful.

Gentoo Linux client for FreeIPA

Sun, 02 Dec 2012 20:31:53 +0100

FreeIPA is an integrated user, host and service identity management solution combining 389 Directory Server (LDAP), MIT Kerberos, a BIND DNS server and the Dogtag Certificate Authority service with a simple but powerful Web GUI and an extensive command line interface for easy administration. It claims to become something like an Active Directory for Linux and Unix environments and is heavily pushed by Red Hat, which also integrates it as IPA server in their Red Hat Enterprise Linux distribution. A nice overview can be found in this presentation.

After having the pleasure of playing around with the Red Hat IPA server on RHEL and CentOS for the past few weeks, I also wanted to use this excellent identity management platform with my Gentoo Linux boxes. Some years ago, a bug report was opened in the Gentoo bugzilla (#297665), to coordinate the inclusion of FreeIPA in Gentoo. Andreis Vinogradovs, another Gentoo user, then started an effort, to write some of the necessary ebuilds for building FreeIPA, however they are still far from complete and therefore haven’t made it into the official Gentoo repository yet. This means that FreeIPA is unfortunately still not fully available for Gentoo.

Based on Andreis’ work, I started another effort, to update and polish the FreeIPA ebuild and its dependencies, so that they can be used on a Gentoo Linux box. The server part has dozens of dependencies not yet officially integrated in Gentoo, and the available ebuilds are mostly outdated, so I haven’t put too much effort yet into integrating the server parts on Gentoo. Especially the entire PKI infrastructure is still missing.

However, I succeeded to configure a Gentoo box as full-featured FreeIPA client, including OpenRC support for `authconfig` and `ipa-client-install`. I also found and reported some bugs in official Gentoo ebuilds (#445394, #445478), where you have to work-around in case you try out the setup yourself.

Of course you are curious now, where you can find the ebuilds. Because the work on them and especially the testing is still ongoing, I created a repository on Github so that everybody who is interested can have a look at ebuilds and provide constructive feedback in terms of pull requests.

I’m especially looking for people who would like to try the FreeIPA client with a Gentoo systemd or/and a hardened SELinux system.

How can you test the FreeIPA client on your Gentoo box?

You have to begin with setting up a (Free)IPA server, which is currently only possible on a Red Hat-based distribution. The easiest way is to setup a CentOS 6 VM, then run:

[root@centos6 ~]# yum install ipa-server [root@centos6 ~]# ipa-server-install

More information can be found in the upstream installation guide.

Then add the ‘freeipa-overlay’ to the layman configuration of your Gentoo client. How you do this is described here.

ATTENTION: This guide is meant to be for experimental testing only. Don’t do this on your workstation if you are not familiar with FreeIPA and its technologies. I don’t take any responsibility if you blow up your machine. You have been warned!

Finally you are ready to emerge FreeIPA. Make sure tho have a look at the various USE flags. They don’t have too much influence on build-time functionality but on run-time dependencies. So you can slim down your installation in case you already know, that you don’t need another DNS server or winbind support for example. Set the ‘minimal’ USE flag for only building the IPA client
( Update 07.12.2012: This USE flag was replaces with ‘server’, so the client will be installed by default): gentoo ~ # emerge -av freeipa

Some keyword unmasking may be required when you run a stable Gentoo installation.

Before you can start your IPA client installation, you have to make sure, that an empty NSS certificate database exists. This is expected to be under /etc/pki/nssdb. Gentoo however puts all the SSL stuff under /etc/ssl. I solved this by creating a symlink: gentoo ~ # ln -s ssl /etc/pki gentoo ~ # certutil -N -d /etc/pki/nssdb Eventually the IPA client can be configured. E.g.: gentoo ~ # ipa-client-install --mkhomedir --no-dns-sshfp Discovery was successful! Hostname: gentoo.example.com Realm: EXAMPLE.COM DNS Domain: example.com IPA Server: centos6.example.com BaseDN: dc=example,dc=com Continue to configure the system with these values? [no]: yes User authorized to enroll computers: admin Synchronizing time with KDC... Password for admin@EXAMPLE.COM: Enrolled in IPA realm EXAMPLE.COM Created /etc/ipa/default.conf Domain example.com is already configured in existing SSSD config, creating a new one. The old /etc/sssd/sssd.conf is backed up and will be restored during uninstall. Configured /etc/sssd/sssd.conf Configured /etc/krb5.conf for IPA realm EXAMPLE.COM SSSD enabled Configured /etc/openldap/ldap.conf NTP enabled Configured /etc/ssh/ssh_config Warning: Installed OpenSSH server does not support dynamically loading authorized user keys. Public key authentication of IPA users will not be available. Configured /etc/ssh/sshd_config Client configuration complete.

That’s it! Your system is now able to use user accounts created on the IPA server. Check it with: gentoo ~ # id admin uid=155960000(admin) gid=155960000(admins) groups=155960000(admins)

As you can see in the generated /etc/pam.d/system-auth, pam_unix will be checked before pam_sssd. This means your local user accounts still have precedence towards the IPA accounts.

Happy testing… :-)

LVM boot with dracut and OpenRC on Gentoo

Wed, 31 Oct 2012 00:24:57 +0100

As a Linux enthusiast and Gentoo user I was always looking for the perfect boot experience. While I managed to boot my kernel with EFI and grub 2 (as described in my wiki), I still had some troubles with OpenRC playing nice with my LVM-only setup initialized by dracut. Tonight I finally figured out the missing configuration pieces to shut up all warnings on system init.

Initial situation All my Linux partitions are stored in a single LVM volume group, to stay as flexible as possible:

merkur ~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT [...] └─sda5 8:5 0 49.5G 0 part ├─vg_merkur-slash (dm-0) 253:0 0 2.5G 0 lvm / ├─vg_merkur-boot (dm-1) 253:1 0 200M 0 lvm /boot ├─vg_merkur-tmp (dm-2) 253:2 0 6G 0 lvm /tmp ├─vg_merkur-swap (dm-3) 253:3 0 4G 0 lvm [SWAP] ├─vg_merkur-var (dm-4) 253:4 0 4G 0 lvm /var ├─vg_merkur-usr (dm-5) 253:5 0 12.8G 0 lvm /usr └─vg_merkur-opt (dm-6) 253:6 0 8G 0 lvm /opt

My boot toolset currently consists of grub-2.00-r1, kernel-3.6.4, dracut-024, lvm-2.02.95-r4 and openrc-0.11.2

Kernel Configuration Before compiling the kernel, make sure to include all the required configurations. For this setup, the most important ones are:

CONFIG_BLK_DEV_INITRD CONFIG_DEVTMPFS CONFIG_MODULES CONFIG_SYSVIPC Dracut Configuration Before installing dracut, the desired modules have to be configured in /etc/make.conf. :

DRACUT_MODULES="caps lvm mdraid syslog"

For this setup at least the “lvm” module is mandatory. Further dracut was built with the “device-mapper” USE flag enabled.

Altough some Linux developers (especially from Red Hat/Fedora) advice against a separate /usr partition because of many boot time dependencies on this system path, I didn’t bother much to change my years old setup. Since version 014, dracut includes a module to fill this gap (/usr/lib/dracut/modules.d/98usrmount/mount-usr.sh). It simply mounts the /usr partition right after the root file system early in the boot process. Therefore we have to make sure that the dracut modules “usrmount” and “lvm” are included in the initramfs, which was possible without any manual modification of /etc/dracut.conf, when generating the boot image with:

dracut -H Kernel Command Line Configuration Dracut runtime parameters are given on the kernel command line in the Grub configuration. To automatically enable the LVM Volume Group and spawning a debug shell in case the boot should fail, I added the following parameters in grub:

root=/dev/vg_merkur/slash rd.lvm.vg=vg_merkur rd.shell LVM Configuration Since dracut is now responsible to enable our volume group, the corresponding init script has to be disabled:

rc-update del lvm boot Fsck and Fstab When booting the system now, the /etc/init.d/fsck script will complain that it cannot check the file systems which are already mounted. Fortunately, the init script allows us to define that fsck should be only run when specific “fs_passno” values are set. I therefore this value to “1” for the file systems which are mounted by dracut and to “2” for all the file systems which should be checked by OpenRC. Take care, when specifying a value of “0”, the file system will be never checked for consistency:

# [fs] [mountpoint] [type] [opts] [dump/pass] /dev/vg_merkur/boot /boot ext2 noatime,nosuid,nodev 0 2 /dev/vg_merkur/slash / ext4 noatime,discard 0 1 /dev/vg_merkur/usr /usr ext4 noatime,discard,nodev 0 1 /dev/vg_merkur/var /var ext4 noatime,discard,nosuid,nodev 0 2 /dev/vg_merkur/opt /opt ext4 noatime,discard,nosuid,nodev 0 2 /dev/vg_merkur/tmp /tmp ext4 noatime,discard,nosuid,nodev 0 2 /dev/vg_merkur/swap none swap sw 0 0

In /etc/conf.d/fsck we then can define, that the fsck init script should only care about file systems with a “fs_passno” larger than “1”:

fsck_passno=">1"

That’s it… If you have some questions or hints, please leave a comment.

Webtip - 1. November 2009

Sun, 01 Nov 2009 05:25:00 +0100

Ever heard about the “Busy Beaver Problem”? Interesting introduction and visualization with lots of source code (Python, C++, Perl)

catonmat.net: The Busy Beaver Problem

Migrate trac-0.11.x from Debian Etch to Lenny

Sun, 18 Oct 2009 23:59:35 +0200

Since I was using the nice FineGradePermissions feature of the trac 0.11 release, and Debian was only providing trac-0.10.3 in Etch, I had a custom trac installation running on my Etch server. For migrating to Lenny you would normally think that it’s enough to just copy your project directory to the new installation. Unfortunately this results in a nasty error message:

DatabaseError: file is encrypted or is not a database

Hmn, so let’s check the trac migration guide which advises you to first export the sqlite database with sqlite3 in a plain SQL file. Not much luck here either, the result is an empty database: `

sqlite3 trac.db .dump

BEGIN TRANSACTION; COMMIT; `

The reason is the trac installation in Etch was using the python-sqlite-1.0.1 back-end which uses the SQLite 2 format while in Lenny there is python-pysqlite2-2.4.1 which only knows about SQLite 3.

The conversion from SQLite 2 to 3 can be done by first exporting the database with the sqlite tool and then re-importing it with sqlite3: `

sqlite trac.db .dump | sqlite3 trac3.db

More infos about this can be found at the trac upgrade notes from 0.8.x to 0.9.

Finally your trac installation should work again as usual.

Using vanilla kernel to run as 'paravirt_ops' Xen domU

Thu, 16 Oct 2008 15:51:41 +0200

Xen is nowadays a widespread approach to run virtual hosts. Thanks to its flexible configuration it’s not only possible to run virtual Linux hosts but nearly any operating system that is compatible to the Xen hypervisor machine architecture will possibly run as guest host. The Xen paravirtualization feature, which provides a tighter association between the hypervisor and the guest kernel makes it especially interesting for running performance critical Linux guests.

For using Xen virtualization a separate patchset always had to be applied to the hypervisor and also the paravirtualized Linux kernel. Because of this it wasn’t always easy to migrate physical servers into virtual one’s, especially when requiring a special kernel version or another profound Linux patchset.

What is paravirt_ops? Paravirt_ops is a generic kernel structure that represents an interface to a virtualization hypervisor. It allows the kernel to run natively on the hardware but also virtual on a hypervisor. Since kernel 2.6.23 paravirt_ops support for Xen is integrated into the vanilla kernel and since then also support for VMI (VMware) and KVM was added. It is a first step to integrate full Xen compatibility into the mainline kernel tree. Today, with kernel 2.6.27, Xen paravirt_ops support exists at least for the x86 and x86_64 architecture.

How can I use the vanilla kernel as Xen domU? If you build the kernel by yourself, like every real man does, you have to activate the following features. Build the support directly into the kernel that you don’t have to fiddle around with an initramfs.

Processor type and features  --->
[*] Paravirtualized guest support  --->
--- Paravirtualized guest support
[*]   Xen guest support
...
[*] PAE (Physical Address Extension) Support

Device Drivers  --->
[*] Block devices  --->
<*>   Xen virtual block device support
...
[*] Network device support  --->
<*>   Xen network device frontend driver
...
Character devices  --->
[*] Xen Hypervisor Console support

Now we’re nearly done with it. After compiling the kernel we need to do some small changes concerning the console redirection in the baselayout of the Linux guest. In the future the distributions will likely prepare this for us.

Add hvc0 device node:

root # cd /dev
root # mknod hvc0 c 229 0
root # chown root:tty hvc0
root # chmod 600 hvc0

To change default console to hvc0 you have to edit the /etc/inittab. If not done already, disable all the other virtual consoles:

co:12345:respawn:/sbin/agetty 38400 hvc0 vt100-nav
#2:23:respawn:/sbin/agetty 38400 tty2
#3:23:respawn:/sbin/agetty 38400 tty3
...

Finally permit login on the hvc0 console by adding it to /etc/securetty.

Now we should be able to boot our paravirt_ops system on real hardware the same as on a Xen hypervisor. However compared with a Xen-patched paravirtualized kernel, the paravirt_ops kernel doesn’t support special features like framebuffer support or live migration.

Links: lwn.net - Connecting Linux to hypervisors linuxmonk.ch/trac - Building paravirt_ops domU kernel from vanilla sources

Gentoo: QEMU in 32bit Xen environment

Wed, 03 Oct 2007 18:13:13 +0200

Recently I wanted to set up a testing server for the different virtualization techniques for Linux. For this I have an Asus P5LD2 mainboard with an Intel dual core Pentium D 3,2 GHz which supports the Virtual Machine Extensions (VMX). Thanks to this I can compile Xen with the ‘hvm’USE-flag and run fully virtualized guest operating systems on my Xen supervisor. This means I could run nearly every i386 compatible operating system (even Windows ;-) ) in my Xen environment. Without such hardware every guest operating system has to have a Xen enabled kernel.

Another approach with the same result is the open source project QEMU. Its abstraction level is higher than with Xen and it can even emulate different target architectures from your current x86 host. So far x86_64, ARM, SPARC, PowerPC, MIPS and M68k target systems are supported. Its guest operating system does not need any single change to run on QEMU. This makes it very comfortable to test new live CDs or operating system images. But it is not so trivial to setup QEMU and Xen together on a Gentoo machine.

How to setup QEMU on 32bit Gentoo in Xen dom0?

If you compile Xen on a 32bit host you have to add ‘-mno-tls-direct-seg-refs’ to your CFLAGS. That is because the glibc TLS library is implemented in a way that will conflict with how Xen uses segment registers. For compiling the non-patched QEMU 0.9.0 you have to use a gcc version 3.x. The nowadays default gcc 4.x is not yet supported. After several compile failures I finally found to setup QEMU the following way:

1. For compiling gcc-3.x remove the ‘-mno-tls-direct-seg-refs’ from /etc/make.conf and set the ’nossp’ and ’nopie’ USE-flags. Otherwise gcc or later qemu will not compile.

2. Switch to gcc-3.x before compiling qemu-softmmu, qemu-user and qemu. In my case it’s: gcc-config i686-pc-linux-gnu-3.3.6 3. Check your CFLAGS again because the optimization flags for gcc 4.x are not always backwards compatible to gcc-3.x. In my case the make.conf looks like this:

# gcc-3.x
CFLAGS="-march=pentium4 -O2 -pipe -fomit-frame-pointer -mno-tls-direct-seg-refs"

# gcc-4.x for compiling gcc-3.3
#CFLAGS="-march=prescott -O2 -pipe -fomit-frame-pointer"

# gcc-4.x
#CFLAGS="-march=prescott -O2 -pipe -fomit-frame-pointer -mno-tls-direct-seg-refs"

4. Now you can compile QEMU. Do not forget to switch back to your original CFLAGS and gcc-4.x after successfully emerging QEMU. I recommend to you to also build the QEMU kernel accelerator module kqemu which has to be compiled with the same compiler as the kernel itself.

Now Xen and QEMU are able to run whatever operating system image you give them. Have fun with playing around…

Additional links:

DokuWiki - A small Wiki for everyday use

Wed, 26 Sep 2007 18:38:45 +0200

For organizing our move to the shared flat I was looking for a small, simple to use Wiki for collecting ideas and coordinating our flat inventory. After a little search I found DokuWiki. It can be easily installed by every Linux distribution’s package manager. Unfortunately the stable Debian package was not the newest version and annoyed me with banners that there are some upgrades available. So I tried again with the newest version from the developers site. After unpacking, the directory has to be made accessible through your Web server and after running the install.php where you actually only create the administrator user, the Wiki is already prepared to use. In the default configuration there is no database needed. But the strength of this Wiki is that it can be expanded by more sophisticated configurations using MySQL or an LDAP back-end for user administration. The syntax is quite simple and similar to other Wiki systems. Also my friends were surprised by the usability of this piece of Open Source software. So if you are planning to use a powerful but simple Wiki software, keep an eye on DokuWiki.

Finally I have my own public IP

Tue, 18 Sep 2007 16:05:59 +0200

This information is for all of you quite boring I know, but it is a happy thing for me. Finally my Xen domain, where I am hosting this Web site, has its own static public IP (193.138.215.129). Now I am ready for running all possible Internet services depending how I feel like. My first plan will be an own mail server with qmail. I am open about some tricks and hints about this. I eventually also found a domain name that suits for this site. When I do not change my mind again, the blog will be soon available under http://www.linuxmonk.ch.

There is a lot changing in my life at the moment so stay tuned for some more news later…

Open Source Instant Messaging with GoogleTalk

Thu, 16 Aug 2007 08:39:51 +0200

Most of you may have a similar problem. A lot of friends are present in the Internet but all of them use a different instant messenger. There are ICQ, MSN, Yahoo Messenger, Jabber and a lot more. With multi-protocol messenger like Pidgin (former Gaim) or Adium it is not that laborious to manage all your contacts anymore. But still you would like to reduce your number of accounts as much as possible.

Once more open source software gives an example how it could be. The protocol is called XMPP and originally used by Jabber. When Google introduced their own messenger GoogleTalk in 2005 they made the wise decision to also use XMPP instead of inventing a new protocol. The next big thing is that everybody with a Gmail account can also use this credentials for GoogleTalk. One account connects you to your friends of two networks. Why is not everything in information technology so handy?

Now you are maybe wondering that Gaim/Pidgin does not support GoogleTalk. But it does support Jabber/XMPP. So it is an easy thing to set up your GoogleTalk account. Enter your Gmail address as “Screen Name”; the server is not “jabber.org” but “gmail.com” and the connect server is “talk.google.com”. As resource you can leave “Home”. Finished and ready for chatting…

Arming against SSH Attacks

Tue, 24 Jul 2007 20:53:26 +0200

I think everybody that runs an own Linux server with the SSH daemon listening on port 22 is sooner or later annoyed by the amount of password attacks done by bots somewhere out in the Internet. What can you do against it?

Blocking via iptables ‘recent’ module How you can do this on a Gentoo system is described in the Gentoo Wiki here. Because it blocks the connection attempt only due to the number of tries within a certain time it is a very basic solution and needs quite a lot of testing to examine the good parameters for the ‘hitcount’ and ‘seconds’ arguments. You do not unintentionally want to block yourself when you only try to open several connections within a short time period. So not really the thing I recommend here.

Log parsing with sshguard sshguard uses another approach. It parses the SSH log messages and searches for login failures. For example when you try to connect with a non-existent user sshguard catches it and creates an iptables deny rule. But also sshguard has a small design mistake. It wants you to create a sshguard chain in iptables and redirect all the traffic to the chain assuming that your default INPUT policy is ACCEPT. When it wants to block a host it runs iptables -A sshguard -s host-to-block -j DROP. In case you have your policy set to DROP you cannot configure the iptables to accept the allowed SSH traffic because else the blocking rules will not work anymore. I made a small patch to change the blocking command to insert the rule in first place of the chain. After you applied the patch you have to make sure that you setup your iptables the following way:

iptables -N sshguard iptables -A sshguard -j ACCEPT iptables -A INPUT -p tcp --dport ssh -j sshguard

Further you have to edit your system logger configuration file. Please read the documentation.

For all the lazy people I even made a Ebuild that also adds a second patch where you can disable the IPv6 ability of sshguard. You can find it here.

Migration of Xen SMP domU

Mon, 09 Jul 2007 01:36:35 +0200

YES, YES, YES! Finally I made it.

Since a while I am playing with the migration functionality of a Xen domU between two Xen servers. It allows to move a virtual machine without reboot between two physical machines. But my problem was: It did not hold what it promised.

Trying all possible versions of Xen and the xen-sources in the Gentoo Portage I was able to migrate a domU but not when it was configured as SMP machine. After I found that the kernel has to include the CONFIG_HOTPLUG_CPU option, I was able to migrate the domU at least one time from one host to the other. When repeating this procedure the domU crashed with a kernel oops. By finally trying the latest Xen 3.1 version from the unofficial mescalito portage overlay with the xen-sources-2.6.18, repeated migration of a domU between the Xen servers does work now. Prerequisite is that both dom0 and domU run the 2.6.18 kernel. I am trying to run this in production now and will see how reliable it is.

I just remember that I did also remove the memory restriction from the dom0. The dom0_mem parameter in the grub configuration is optional and when omitting it the dom0 can use the unused memory for itself.

Parallel Port in Xen domU

Sat, 07 Jul 2007 19:00:20 +0200

For my home server I wanted to act a Xen domU as CUPS printing server. Therefore I needed access to my parallel printer connected to the Xen server. How are I/O-ports handled in Xen?

The I/O-addresses of the devices which can be found with cat /proc/ioports can be redirected from the Xen dom0. You have to enter the address range of the device you want to use in a domU in their configuration file ( /etc/xen/domain-config). For my parallel port it would be:

ioports = [ '0278-027a' ]

Further I make sure that my dom0 does not access the device by removing parallel port support in the kernel configuration. When you boot then a domU kernel with parallel port and printer support it can access the device natively. Here a cut out from dmesg:

parport0: PC-style at 0x278 [PCSPP,TRISTATE] parport0: Printer, Hewlett-Packard HP LaserJet 6P lp0: using parport0 (polling).

In CUPS the printer is now accessible under /dev/lp0.

Unfortunately the documentation about this feature of Xen is a little bit rare. Another interesting thing is the redirection of PCI devices to a domU. Unfortunately I did not test this at my setup yet. I hope to come back to this topic later. For your creativity here some more examples from the net: dulug.de - serielle Schnittstelle mit XEN wiki.xensource.com - Assign hardware to DomU with PCI Backend as module PCI Delegation in Xen

Powered by Xen and Debian Linux

Wed, 27 Jun 2007 00:58:41 +0200

Finally I’m back once more ;-) . While I was in military service my Zyxel modem suddenly refused to work with my fli4l router what caused me to switch the modem into router mode and this made it impossible to further access my Web server. That’s the reason for the downtime. It’s very embarrassing that I only write into my blog when there was a service failure before. But this time this should change!

I moved my Web site from my home server to a machine in a hosting center. Together with some friends we were sharing a physical machine with different Xen domains. Generally I did a lot of work with Xen recently therefore I hope to enjoy you with some tricks about it soon. In my Xen domain the new Debian 4.0 Etch is running. With help of debootstrap it is installed in a few minutes. By the way… does someone know the successor of the base-config program in the earlier versions?

Anyway I hope that the site can stay here for a while and you will come back when there are some more news.

Webtipps - 19. September 2006

Tue, 19 Sep 2006 01:26:50 +0200

damnsmalllinux.org: A guide to earning an income from a Free and Open Source project debianhelp.co.uk: Debian Network Utilities and Tools with Examples heise.de: Kommentar: Qualität durch Zufall

Neues vom Firefox

Sun, 10 Sep 2006 02:55:57 +0200

An den verschiedensten Enden gab es in letzter Zeit Neuigkeiten über den Open Source Internet Browser Firefox. Um die schlechte Nachricht schon einmal vorwegzunehmen, der Releasetermin von Firefox 2.0, welcher im August 2006 angesetzt war, konnte nicht eingehalten werden. Dafür hat sich die Oregon State University Linux User Group etwas einfallen lassen und einen Kornkreis in Form des Firefox Logos getrampelt.

Auch die Leute bei Mozilla selber sorgen dafür, dass das Warten auf die nächste Firefox Version nicht langweilig wird und verspricht jedem Firefox Benutzer, der noch bis am 15. September einen Neubenutzer gewinnen kann, dass deren Namen in der 2.0 Version des Programms verewigt wird. Damit sich nicht nur die Verbreitung des Browsers sondern auch dessen Kundennähe vergrössert, wurde im August eine Umfrage über die gewünschten Features anderer Browser durchgeführt. Die Auswertung ergab als wünschenswertesten Punkt eine integrierte PDF Engine wie im Apple Safari.

Dass sich das Projekt auf dem richtigen Weg befindet, zeigte im Juli der 200 Millionste Download von Firefox. Computerbase zeigt schliesslich, dass er nicht nur heruntergeladen, sondern vor allem von interessierten Computerbenutzern rege genutzt wird. Mit knapp 57% kann sich der Open Source Browser recht deutlich gegen den Microsoft Internet Explorer (35%) und Opera (6%) durchsetzen.

Und als richtiges Schmankerl zum Schluss gibt jetzt auch noch die Firefox 2.0 Beta 2. Erstmals sind jetzt auch Änderungen in der grafischen Oberfläche sichtbar. Für die neue Version wurde ein neues Theme erstellt. Neue glossy-like Icons, sowie Tabs, die schattiert werden, wenn sie im Hintergrund sind, soll die Übersicht verbessern. Hie und da sind aber nach meinem Geschmack noch einige Verbesserungen anzubringen. Wie schon früher erwähnt, besitzt jedes Tab einen eigenen Close-Button. Beim Starten wird zudem gefragt, ob man die Tabs der letzten Sitzung wieder herstellen soll. Die RSS Integration sowie das Extensions (neuerdings “Add-ons”) Menu wurden überarbeitet. Etwas sehr sinnvolles Neues ist die Rechtschreibeprüfung für Eingabefelder in Webseiten. Bisher hatte ich noch keine Probleme mit der Beta 2 und so kann ich allen, die schon jetzt die neuen Features ausprobieren wollen, empfehlen, diese Version zu installieren. Sonst bleibt nur das Warten auf den nächsten offiziellen Releasetermin, welcher nun auf den 24. Oktober lautet.

Neues von Apple

Mon, 07 Aug 2006 19:53:28 +0200

Heute hatte Steve Jobs seinen grossen Auftritt an der alljährlichen Apple Worldwide Developers Conference (WWDC 2006). Dabei wurde der Nachfolger des Apple Flaggschiffs PowerMac, der Mac Pro vorgestellt. Ab CHF 3599.- gibt es eine Dual Intel Xeon 2.66 GHz “Woodcrest” (neue Core Architektur!) Workstation mit 1GB DDR2-667MHz Arbeitsspeicher und einer Nvidia Geforce 7300 GT Grafikkarte. Ebenfalls vorgestellt wurden einige Details zum Mac OS X 10.5 Leopard, welches im Frühjahr 2007 ausgeliefert werden soll. So gibt es neu beispielsweise einen automatischen Backup-Mechanismus “Time Machine”, eine virtuelle Desktop-Verwaltung “Spaces” sowie diverse Updates zu bereits bekannten Apple Programmen wie Mail, iChat, Dashboard, Photo Booth, Front Row und Spotlight.

Ich bin ja eigentlich schon ein OS X Fan, seit ich mein erstes Apple iBook gekauft habe und mit jeder neuen Version von OS X, werde ich überzeugter. Microsoft versucht durch Biegen und Brechen ein völlig neues Betriebssystemgefühl zu kreieren. Doch leider haben sie sich zünftig übernommen (computerbase.de: Microsoft gibt Fehler bei Vista-Entwicklung zu). Apple dagegen schafft durch konstante Evaluation, Entwicklung und Verbesserungen das optimale System für alle normalen Computeranwender. Wann merken die das endlich?

Quelle: mactechnews.de: WWDC 2006 Live Ticker Update - 9. August 2006: Nun ist auch die Keynote der WWDC 2006 online. Anklicken und Zurücklehnen… Auch Apple macht sich übrigens recht ausdrücklich lustig über Windows Vista. Das darf man nicht verpassen!

Webtipps - 5. August 2006

Sat, 05 Aug 2006 01:43:58 +0200

heise.de: Wo bleibt der Linux Desktop? ibm.com: Linux threading models compared: LinuxThreads and NPTL linuxworld.com.au: Forget about open source at Apple

Debian Server durch Kernel-Schwachstelle komprimitiert

Thu, 13 Jul 2006 23:13:39 +0200

Die kürzlich gefundene Kernel-Schwachstelle CVE-2006-2451 der Kernel >2.6.13 und <2.6.16.24 und <2.6.17.4, wodurch ein entfernter Angreifer root-Privilegien erhalten kann, wurde am prominenten Beispiel des Debian Entwicklungsservers Gluck demonstriert. Dank der schnellen Reaktion der Administratoren wurde nur /bin/ping komprimitiert. Sonst wurden noch keine veränderten Daten/Packete oder Programme entdeckt. Das zeigt wieder einmal, dass man als Administrator gut dran tut seine Kernelversionen aktuell zu halten und die Maschinen genau zu überwachen.

Quelle: debian.org: Debian Server restored after Compromise Update - 7. August 2006: Für alle, welche noch einen der betroffenen Kernel auf einem produktiven System einsetzen, der Standardkernel (2.6.8) von Debian Sarge ist übrigens nicht betroffen, gibt es ein Workaround: Mitigating against recent GNU/Linux kernel bugs

Spam gibts nicht nur im Email

Wed, 28 Jun 2006 23:52:29 +0200

Vielleicht ist es schon jemandem aufgefallen. Ganz unten auf der Seite ist eine Statuszeile von Spam Karma 2 einem Programm, welches dieses WordPress-Blog von ungewollten Einträgen schützt. Seit ich dieses SK2 Plugin vor genau einer Woche installiert habe, sind jetzt sage und schreibe bereits 124 Spam Einträge gefiltert worden. Nein, ich habe definitiv keine Lust, dass jemand auf meiner Seite mit zwielichtigen Sachen wirbt. Die geposteten Links müssen aber nicht immer nur auf Werbeangebote führen. Im schlimmeren Fall haben sie auf ihrer Zielseite Code eingebaut, um durch eine Browserschwachstelle ungewollte Programme zu installieren. Ich möchte nicht unerwähnt lassen, dass der Internet Explorer immer noch Angriffsziel Nummer eins ist und daher nicht als bevorzugter Browser gewählt werden sollte. Diese Spameinträge werden dann schliesslich auch von solchen ungewollten Programmen gemacht, sogenannten Spam Bots. Das macht die Filterung relativ einfach. Beispielsweise wird auf die Zeit geschaut, welche verstreicht nach dem Aufrufen der Seite bis zum Abschicken des Kommentars. Nun kommen wir aber zum eigentlichen Problem, nämlich der Ursache dieser Bots. Und schon sind wir wieder einmal bei Windows angelangt. Leider ist dieses Betriebssystem für die meisten Benutzer so einfach zu bedienen, dass sie keine weiteren Ansprüche stellen sich tiefer mit der Materie auseinander zu setzen, was bei Windows leider eindeutig nötig wäre… hier haben wir jetzt den Mist. Merci

Windows Media Components auch unter Intel Mac

Sat, 24 Jun 2006 23:40:13 +0200

Vor einiger Zeit habe ich berichtet, wie man ohne Windows Media Player die neusten .wmv Codecs installiert (Apple OS X und Windows Media Files). Wenn man aber versucht die Windows Media Components auf einem Intel Mac einzurichten, meldet der Installer, dass diese nicht unter dem x86 OS X laufen. Eine kompatible Version ist zwar angekündigt, aber noch nicht erhältlich. Genau hier kommt jetzt Rosetta ins Spiel. Dabei handelt es sich um einen im Apple OS X integrierten Binary Translator, welcher ermöglicht, Programme welche für die PowerPC Architektur kompiliert wurden, auch auf Intel Macs laufen zu lassen. In unserem Fall wenden wir ihn auf den Installer an, welcher das “Flip4Mac WMV.mpkg” installiert. d.h. Rechtsklick auf /Applications/Utilities/Installer. Dann auf “Get Info” und dort “Open with Rosetta”. Danach können wir die Windows Media Components wie gewohnt installieren. Nun muss man natürlich auch noch den QuickTime Player unter Rosetta laufen lassen und kann dann sämtliche Windows Media Files auch auf seinem Intel Mac geniessen.

Quelle: OSNews.org: How to Get Your Intel Mac to Play WMV Files Update - 7. August 2006: Mittlerweile ist die Version 2.1 der Windows Media Components for Quicktime herausgekommen, welche auch nativ unter Intel Macs funktioniert: Flip4Mac WMV 2.1

Last Exit...

Wed, 14 Jun 2006 23:57:33 +0200

…ist nicht nur ein bekanntes Browsergame der UNHCR (Flüchtlingskommissariat der Vereinten Nationen), sondern auch ein GTK+ basierter Player für den Gnome-Desktop zum Empfang von last.fm Radio-Streams. Die Sourcen gibt es hier zum Download. Installiert wird Last Exit wie gewohnt mit ./configure && make && make install. Falls es noch zu Problemen mit GConf kommt, kann man im Ubuntu-Forum nachlesen, wie diese zu beheben sind. Im Gegensatz zum offiziellen QT-basierten Last.fm Player funktioniert Last Exit jedoch noch nicht so zuverlässig. Bei mir verabschiedet er sich ab und zu mit einem Segmentation Fault. Auch ist die persönliche Radiostation ausgegraut und kann nicht angewählt werden. Aber für alle, die ihre Gnome Installation nicht mit QT belasten wollen, ist Last Exit trotzdem noch eine brauchbare Alternative.

Gratis TV im Internet... alle WM Matchs Live!

Sun, 11 Jun 2006 14:43:58 +0200

Was macht man, wenn man WM schauen will, aber kein Fernseher in Sichtweite hat? Genau, man setzt sich an den Computer und schaut, was man denn dem Internet alles entlocken kann. Und wird auch fündig. In meinem Fall bei TVants. Das ganze ist ein P2P-Streaming-Netzwerk, an welches verschiedene Fernsehsender angeschlossen sind und welches sich vor für Sportübertragungen sehr gut eignet. Nach dem Installieren einer Clientapplikation (TvantsSetup.exe), welches leider nur für Windows erhältlich ist, wählt man aus den verfügbaren Sendern aus und nach einer längeren Bufferingphase, hat man auch schon das Livebild auf dem Computerschirm. Trotz meiner 2000 Kbit/s Leitung gibt es leider viele Bufferingphasen, was natürlich immer davon abhängt, wieviele Leute gleichzeitig auch diesen Sender schauen. Das ganze ist bisher wohl eher in Asien verbreitet, so gibt es vor allem asiatische TV Stationen, welche ihre Sendungen in das Netzwerk einspeisen. Aber unter anderem ist auch Eurosport France dabei.

Wichtige Links: tvants.com (übersetzt) - Übersicht der an TVants beteiligten Sender www.football4less.com/… - Installations-Tipps für TVants www.fussball-foren.net/… - Übersicht der WM-Spiele in Streaming Netzwerken (u.a. TVants) tvmix.net - Übersicht von Sportübertragungen auf TVants kompatiblen Stationen

Webtipps - 31. März 2006

Fri, 31 Mar 2006 00:16:36 +0200

www.geekpatrol.ca: MacBook Pro Running XP Geekbenched chabada.sk: 40+ Suggestions for Better Desktop

Apple OS X und Windows Media Files

Tue, 28 Mar 2006 18:25:09 +0200

Da Microsoft ja immer ihr eigenes Süppchen kochen, ist das auch bei den Videoformaten nicht anders. Teilweise bereitet bereits das Abspielen von wmv3 codierten Videos, welche mit dem Windows Media Player 9 eingeführt wurden, Probleme auf nicht Windows Plattformen. Für Apple OS X konnte bisher nur der Windows Media Player 9 abhilfe schaffen. Doch mit dem Windows Media Player 10 hat Microsoft wieder neue Formate geschaffen und dann auch noch beschlossen den Media Player für OS X nicht mehr weiterzuführen. Somit gibt es wieder ettliche .wmv und .asf Videos, die nicht mehr mit dem Windows Media Player 9 abspielbar sind. Auch bei freien Mediaplayern, wie MPlayer oder VLC für OS X, sucht man deren Unterstützung vergebens. Abhilfe schafft hier nur noch ein von Microsoft bereitgestelltes Plug-In für den Apple QuickTime Player. Windows Media Components for QuickTime nennt sich das ganze und ist hier erhältlich. Ein freuendes Auge, dass man endlich den Windows Media Player von seinem OS X deinstallieren kann, und ein weinendes, da offenbar immer noch die halbe Welt ihre Daten proprietären Microsoft Formaten anvertraut.

Mozilla Firefox Alpha 1 (Bon Echo) released!

Thu, 23 Mar 2006 11:33:14 +0100

Die von mir am meisten verwendete Applikation ist in einer neuen Alphaversion erschienen. Die implementierten Änderungen betreffen bisher vor allem das Back-end betreffend der History und den Bookmarks des Browsers. Die auffälligste Änderung für den Benutzer sind die separaten “Close”-Buttons an den einzelnen Tabs. Laut der Firefox 2.0 Roadmap will man die finale Version etwa mitte Sommer herausbringen. Bis dahin ist also noch viel Zeit. Wer sich schon jetzt über die zu erwartenden Neuerungen informieren will, kann das in der Firefox 2 Features-Liste machen. Leider wurde aber der verbesserte Downloadmanager bereits wieder von der Liste gestrichen. Wer sich nun bereits an das Preview wagen will (ich habe es natürlich schon längst getan), kann sich die passende Version vom Mozilla FTP herunterladen. Da dies die erste Version des 2.0 Zweigs ist, ist zu beachten, dass leider sämtliche Extensions noch nicht funktionieren.

Quelle: mozillazine.org: Bon Echo Alpha 1 Available

Webtipps - 2. März 2006

Thu, 02 Mar 2006 17:21:38 +0100

arstechnica.com: Review: Apple MacBook Pro

ganto.no-ip.org is online again :-)

Sun, 26 Feb 2006 03:03:43 +0100

Hallo zäme

Ah, endlich hanis gschafft. Öppe eis Jahr nach mim erschte Versuech, chum ig wider mit emene Blog-/News- und Knowledgebase-System online. Das ganze wird sech i ständiger Entwicklig befinde und hoffentlech ab und zue ou mau öppis interessants für dr eint oder ander vo öich parat ha.

Viu Spass wünscht dr

ganto

linuxmonk.ch

Testing oVirt 4.4-alpha on CentOS 8: Part 2 - Setup Self-Hosted Engine

Cockpit Hosted-Engine Wizard

Troubleshooting hosted-engine issues

Conclusion

Testing oVirt 4.4-alpha on CentOS 8: Part 1 - Setup GlusterFS

Hypervisor disk layout

Install software requirements

Running the GlusterFS wizard

Conclusion

My free software activities (2019-10)

My free software activities (2019-09)

A user story about an OKD update from 3.9 to 3.10

Starting Position

What will change?

Updating the Ansible Inventory

Fixing a failed CNS Brick Process

Fixing the Hosted Registry Storage Definition

etcd Backup Failure

The Master API cannot find the LDAP CA Certificate

How a Docker Bug broke the etcd Cluster

Updating Docker on Atomic Host

Fixing the etcd Cluster

Finishing the Control Plane Upgrade

Running the Node Upgrade Playbook

Fixing the Infrastructure Node Selector

Summarizing

Gitea on OpenShift with NodePort SSH access

NodePort Service

Issues with NodePort

Container with root Permissions

Security Context Constraints

Gitea Application Template

spectre-meltdown-checker in Fedora and EPEL repositories

Getting started with LXD on Fedora

Managing DNS zones with Ansible

"Let's encrypt" with acme-tiny

Policy Routing on Linux based on Sender MAC Address

iptables -A OUTPUT -m mark –mark 0x2 -j LOG`

Setup remote server management of a SuperMicro BMC with FreeIPMI on Linux

FreeIPMI Management Utility

Configure Serial-over-LAN (SoL) Console Access

Remotely Manage the System over IPMI

Migrate 32bit physical host on RAID 1 to KVM/Xen virtual machine

Prepare original host for virtualized environment

Run the host on a KVM hypervisor

Run the host on a Xen hypervisor

Summary

Kerberized HTTP Auth on Apache/mod_ssl with FreeIPA

Quickly setup a Fedora 19 boot disk for serial-only console access

Make Firefox subscribe RSS feeds with TinyTinyRSS

10 years ago I switched to Gentoo Linux as my main operating system

DHCPv6: Stateful Address Autoconfiguration on Linux (Part 1: radvd)

Building a RPM for Oracle JRockit JDK (with mock)

Gentoo Linux client for FreeIPA

LVM boot with dracut and OpenRC on Gentoo

Webtip - 1. November 2009

Migrate trac-0.11.x from Debian Etch to Lenny

sqlite3 trac.db .dump

sqlite trac.db .dump | sqlite3 trac3.db

Using vanilla kernel to run as 'paravirt_ops' Xen domU

Gentoo: QEMU in 32bit Xen environment

DokuWiki - A small Wiki for everyday use

Finally I have my own public IP

Open Source Instant Messaging with GoogleTalk

Arming against SSH Attacks

Migration of Xen SMP domU

Parallel Port in Xen domU

Powered by Xen and Debian Linux

Webtipps - 19. September 2006

Neues vom Firefox

Neues von Apple

Webtipps - 5. August 2006

Debian Server durch Kernel-Schwachstelle komprimitiert

Spam gibts nicht nur im Email

Windows Media Components auch unter Intel Mac

Last Exit...

Gratis TV im Internet... alle WM Matchs Live!

Webtipps - 31. März 2006

Apple OS X und Windows Media Files