bookmark_borderMaintaina Horde: Now on Leap 15.2

I changed the maintaina.com Horde Images to use openSUSE Leap 15.2 instead of openSUSE Tumbleweed as a base. You should not experience any issues but I have not yet tested much. The change was necessary as I had build failures since 2021-02-17 in the github actions CI builds. It’s something about the docker used in ubuntu-latest being too old or github’s security settings being too strict. I did invest some weekend hours, but could not really solve it. I intend to switch back to tumbleweed at some point.

Update July 2021: I can now build tumbleweed-based images again.

bookmark_borderDAVx5 CalDAV may break with Unicode symbols in Horde/Kronolith syncs.

If you get user complaints about broken CalDAV syncs with Horde, there’s many places to look at. In one particular instance, an event was created from travelling app Transportr into the stock android Calendar app. Through the DAVx5 sync app, the user wanted to push these events to Horde’s SabreDAV interface – and from there, also sync it to his desktop email solution, Mozilla Thunderbird.

5 programa efectivo de entrenamiento de culturismo – programa de entrenamiento de culturismo de 5 días estore de culturismo halotestin szkani hombre sin mangas con capucha chaleco de fitness culturismo largueros entrenamiento camisetas sin mangas.

However, his sync application told him about an error. The server administrator saw a 500 status code in the server log.

1.1.2.11 - user [24/Jan/2021:18:48:26 +0000] "PUT /horde/rpc/calendars/user/calendar~KL14jYhCMpbet4ecYzAg1bn/2bace303-f0d8-4df6-9652-baa5fb9e86c4.ics HTTP/1.1" 500 892 "-" "DAVx5/3.3.8-ose (2021/01/13; dav4jvm; okhttp/4.9.0) Android/11

The root cause was actually not in the software code but in the MariaDB database configuration. The calendar entry from Transportr included some Unicode icon characters like a fast train and some arrows. These characters are part of the standard unicode encoding, utf-8.

Now you might wonder: New installations of mysql and MariaDB default to a character set they call utf-8 since 2010 or so. This shouldn’t be an issue. However, what they call utf-8 is not what you would expect.

Some years ago, TV sets which did not support the full HD resolution were marketed as “HD ready”. In some sense, the default character set is “unicode ready” at best. The default data type saves on disk space by encoding a subset of utf-8 into up to three bytes. While this supports most natural language characters, it is only a fraction of what Unicode can offer. Database manufacturers are well aware that this is not something you should run nowadays that unicode icons like the hamburger are all over the place is user generated content. The mysql manual even says:

“Please use utf8mb4 instead. Although utf8 is currently an alias for utf8mb3, at some point utf8 is expected to become a reference to utf8mb4. To avoid ambiguity about the meaning of utf8, consider specifying utf8mb4 explicitly for character set references instead of utf8.”

Now that’s what I did. First I changed horde’s database encoding to utf8mb4 in conf.php:

$conf['sql']['protocol'] = 'tcp';
$conf['sql']['charset'] = 'utf8mb4';

Then I changed the mysql server’s and client’s default charset:

cat /etc/mysql/my.cnf
[server]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
[client]
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

I have obviously stripped a lot of file content not relevant to our story. After reloading the database, all new connections should use the real utf8 encoding and new tables should be created with the new standard. But what about existing content? We need to convert all tables and all their text-like columns, varchars, mediumtexts etc.

First, it’s backup time – better safe than sorry.

Then, let’s find all tables in our db server and feed them conversion commands.

mysql --database=horde -B -N -e "SHOW TABLES" | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"}' | mysql --database=horde

Do this in a downtime or ensure HA some way or other. It will take some time. Some sources suggest it might be sensible to also rebuild the tables with the optimize command. I am not very convinced, but it won’t harm.

mysqlcheck --auto-repair --optimize --user="hidden" -p --databases mysql horde

After this, repeat the sync test. It should work this time.

bookmark_borderHorde Development Review: October 2020

October was a very busy month in Horde development, even though a lot of things happened under the hood and cannot be accessed right now.

  • A decision was made that Horde RPC will default to json-rpc and deprecate xmlrpc in upcoming releases, maybe already dropping xmlrpc in Horde 6. PHP8 will remove xmlrpc from the core distribution and move it into PECL, giving it much less of an installation base.
  • B1 Systems GmbH released a typescript client for the Horde Ajax Framework https://github.com/b1-systems/horde-ajax-client
  • Horde Injector: I continued my work from September. The Maintaina version of Injector now supports PSR-4, PSR-11, PSR-12 specifications. Backward compatibility is good. The unit tests have been converted to namespaced versions – see https://github.com/maintaina-com/Injector/tree/refactor-psr4
  • Horde Components: The Components tool has been converted for PSR-4 autoloading in a composer setup. Also, some minor improvements on the composer file generator have been added
  • I worked on upgrading Diana Hille’s DNS library for the horde framework, giving it the same PSR-4 makeover, converted it from wrapping AWS cli to wrapping AWS Route53 SDK. I think we can opensource the component soon.
  • I created a little testbed application for the improved controller/routes handling in recent horde/core. horde/frogsie is the first Horde application stub to feature a RESTish interface. Frogsie writes iptables rules to implement a TCP/UDP Front Proxy in plain linux.
    It doesn’t sport web UI or CLI though as it’s both a case study and a stop-gap utility. However, it’s a good example to learn from.
  • Horde Installer Plugin v1.1.0 supports both composer 1.x and composer 2.0 – This is mostly based on work by Kieran Brahney of SupportPal. I also began working on converting horde-deployment to remove the last bits of PEAR and make it compatible with composer 2.0. It’s not released yet.
  • I am looking at the old vilma application at the moment. This will most likely not materialize before november. Vilma is a simple mailbox manager along the lines of postfixadmin. My primary interest is using it together with hordectl for a turnkey default setup. Vilma will get the PSR-4 treatment and likely I will want to change some of the architecture. Focus is on a minimum viable product that nicely integrates with my groupware bundle and fulfills very limited needs. I appreciate the work of Christian Bolz on PostfixAdmin and I do not intend to build a full scale replacement.

What’s next?

  • More library conversions to PSR-4, PHP7 features.
  • Most likely I will upgrade unit tests to a more recent PHPunit
  • Release Cleanup / Upstreaming efforts for existing non-released code.
  • Better theme support for horde-installer-plugin

Results first, more talk later.

bookmark_borderAutowiring Vfs in a Horde App

The Horde Vfs is an abstraction around storing and retrieving files. Calling code does not care about where the Vfs is actually stored, be it a remote filesystem, a dav resource, a database or a path in the local filesystem. Autowiring means the Injector knows how to create a class using some other class without having it explicitly defined

For this article, I want to get an instance of Horde\MyApp\ReportGenerator which uses the VFS to store reports.

First let’s create the ReportGenerator.

lib/ReportGenerator.php

<php
namespace Horde\MyApp;
class ReportGenerator
{
    private $driver;
     public function __construct(\Horde_Vfs_Base $driver
     {
         $this--->driver = $driver;
     }
     ...
}

When a controller in the app wants a report generator, it would require it through the dependency injector like this:


$reportGenerator = $injector->getInstance('Horde\MyApp\ReportGenerator');

Autowiring allows the injector to return an instance even if it has not been explicitly bound to the injector. For this to work, all required dependencies must be interfaces which have either been registered with the injector or can themselves be created from registered interfaces or have no dependencies themselves.

Now, Horde_Vfs_Base is the common base class of all Horde Vfs drivers.

However, Horde does not provide a default binding for Horde_Vfs_Base even though Horde_Core has a default factory Horde_Core_Factory_Vfs.

Thus, we need to define one. In Application.php, look for a method _bootstrap(). We will use it to setup the injector binding. Normally, we would just want to use the existing factory through $injector->bindFactory($interface, $factory, $method);

We cannot do this in this case. The Vfs Factory’s create method has two string parameters to allow creating multiple Vfs instances. For our app, we just want to get the default configured in horde base. But this doesn’t work. bindFactory would automatically pass a child injector to the create method. This would override the default first parameter “horde”. In the end, we would receive the factory’s exception saying we should configure a backend first – even if we did.

To work around this, we could create an own factory class in our app which uses Horde_Core_Factory_Vfs as a dependency. Horde_Injector would just know what to do. But let’s not do this as it’s quite some boilerplate just for creating a Vfs with default configuration.

Instead, let’s use the closure binder.


protected function _bootstrap()
{
     $injector = $GLOBALS['injector'];
     $vfsClosure = function(\Horde_Injector $injector) {
     return $injector->getInstance('Horde_Core_Factory_Vfs')->create();
};
$injector->bindClosure('Horde_Vfs_Base', $vfsClosure);

Now injector knows how to provide a Horde_Vfs_Base implementation which the ReportGenerator requires. We can ask injector to provide an instance. Injector will look if it already has an instance from a previous call, otherwise runs the closure to create the dependency, then create the actual ReportGenerator instance.

bookmark_borderDocker for Windows now runs on Windows Home

Docker for Windows used to require Hyper-V and Windows Professional. Now, you can do basic software development on Windows Home. Here’s the story.

I got hold of some older 4GB Windows convertible laptop. Unfortunately, it won’t run Linux in any useful way. In recent years, I did not really care for Windows a lot, apart from the usual “fix your parent’s gear for christmas” routine. Looking for uses case it would possibly cover, I found Windows has improved quite a lot.

Windows 10 May 2020 Update (also known as version 2004) introduces improvements to the “Windows Sub System For Linux” (WSL2). Recent versions of Docker Desktop CE for Windows allow to run based on WSL2. It’s even better than the original version based on Hyper-V.

To run Docker Desktop, you need to install the WSL and “Virtualization Platform” options in Windows. This requires a reboot. Docker Desktop will ask you to download a special Linux kernel from a Microsoft website. Docker and the Kernel will require an additional reboot. Microsoft promises that the manual kernel install is a one-time issue. Future kernel releases will be published via Windows Updates.

Now why bother? With the latest Windows improvements, I can keep most of my development workflows when switching to a tiny, low-powered device which I don’t especially care for – think of spare time while traveling by train or some free time project while laying in the hammock. There’s no way I would use my bigger, more expensive equipment in such situations.

The new optional Windows Terminal is available both from the Microsoft website and from their App Store. It’s free and as close as you can get to a native KDE Konsole or Gnome Terminal. It’s not as awkward as putty nor as ugly as the classic cmd or Power Shell. Paired with Windows’ posix-like ssh client suite, there’s little difference too remote sessions from Linux desktops .

Visual Studio Code feels exactly the same on both Linux and Windows platforms. It allows to edit code directly inside a runtime container. No need for rsync workarounds or push cycles. Keeping everything in the containers also greatly reduces the time to recover from a full reset.

In case you didn’t know, Docker Desktop has some builtin Kubernetes option. This allows development to be even closer to cloud/production scenarios. However, on a 4GB device, don’t expect too much.

The Windows Store now includes some FOSS software, even some KDE for Windows gadgets. Still no Firefox, Thunderbird, Chrome / Chromium … Microsoft has announced they will release some apt/zypper like commandline software installer later this year. Let’s hope that gets adopted more widely.

Windows Store offers some mainstream linux distributions like openSUSE Leap or ubuntu to run on top of WSL/WSL2. I have decided to postpone experimenting with these. Microsoft has announced they want to enable true, builtin capabilities for graphic applications on X/Wayland. I will give this a try when the time is right. It might also make sense to drop Docker Desktop for some podman or other runtime running inside such a linux platform. But for now, that’s out of scope.

While I’m still not ready to embrace Windows as a first-class development platform, I must concede it has become a viable option.

Update 2020/10: I just read an article on Medium which I totally agree with: https://levelup.gitconnected.com/this-is-why-developers-will-embrace-microsoft-windows-again-7437e494159d

They also point out what I am currently doing, using WSL2, docker desktop (with Kubernetes) and VS Code for an integrated environment.

bookmark_borderHeads Up: Cannot login as horde admin anymore?

Please be careful: Horde has introduced a new default setting $conf[‘auth’][‘lowercase’] which acts like the auth hook used to do.

This might lead to issues when your default admin account is named “Administrator”.

It is actually a very useful setting. It ensures that any prefs and other profiles also work with case-insensitive backends regardless of how you type the user name.

How to fix?

a) Lowercase all admin names in your $conf[‘auth’][‘admin’] settings in conf.php

b) disable $conf[‘auth’][‘lowercase’] by setting it to false value.

bookmark_borderThe state of horde composer deployments

In early 2019 I first wrote a few lines on migrating horde to composer based setups.

Even though I was too occupied with other business for several months, I have something to show and it is time to review what we have and what we don’t have.

The good news is, it really runs as far as I am concerned. For the mainstream apps, the Web UI works correctly as well as caldav/carddav access, handling of db schema migrations, some critical commandline scripts. There is still a lot of scripts to check and possibly fix.

The Horde Components helper has been modified to be run as a part of a horde deployment or standalone. It can also still be run from git-tools. I added some code to git-tools to improve the way it wraps horde-components. However, most things git-tools provides are no longer that interesting to the casual developer or to deploying horde master tree installations.

I also improved the horde Controller/Routes core. It is now more successful in detecting which app handles a request. Also, a new option for unauthenticated routes and for “API style” auth handling has been included. This enables REST interfaces.

I finally did a small proof of concept app which is completely based on the controller/routes core and does not have a single traditional “page” file. Maybe this should be provided as an alternative skeleton.

How to run your own composer based install

Running a new horde install for development or CI purpose is as easy as:

cd your/web/root/
git clone https://github.com/maintaina-com/horde-deployment.git .
composer install
cp web/horde/config/conf.php.dist web/horde/config/conf.php

For using a database, provide a config with a db backend instead.
You might also want to run web/horde/bin/horde-db-migrate.

I intend to improve the installer a little to detect if pre-made configs have been put into expected locations of the deployment and use them.

Why you should not use this in production

This deployment comes “as is”. It is still dependent on a satis server not controlled by the horde organisation, running a no-warranties fork of the unreleased git master code. If you run this in production, you need to do a lot of QA for your specific use case. Patches welcome. You may also encounter features not part of official horde releases because they are incomplete, lacking tests or else.

I am still trying to get this upstream. Dealing with a downsteam horde distribution of 150+ repos is tedious work. I’d rather invest time in actual software development. In case you really need to, contact me about commercial support options.

bookmark_borderPEAR down – Taking Horde to Composer

Since Horde 4, the Horde ecosystem heavily relied on the PEAR infrastructure. Sadly, this infrastructure is in bad health. It’s time to add alternatives.

Everybody has noticed the recent PEAR break-in.

A security breach has been found on the http://pear.php.net webserver, with a tainted go-pear.phar discovered. The PEAR website itself has been disabled until a known clean site can be rebuilt. A more detailed announcement will be on the PEAR Blog once it’s back online. If you have downloaded this go-pear.phar in the past six months, you should get a new copy of the same release version from GitHub (pear/pearweb_phars) and compare file hashes. If different, you may have the infected file.

While I am writing these lines, pear.php.net is down. Retrieval links for individual pear packages are down. Installation of pear packages is still possible from private mirrors or linux software distribution packages (openSUSE, Debian, Ubuntu). Separate pear servers like pear.horde.org are not directly affected. However, a lot of pear software relies on one or many libraries from pear.php.net – it’s a tough situation. A lot of software projects have moved on to composer, an alternative solution to dependency distribution. However, some composer projects have dependency on PEAR channels.

I am currently submitting some changes to Horde upstream to make Horde libs (both released and from git) more usable from composer projects.
Short-term goal is making use of some highlight libraries easier in other contexts. For example, Horde_ActiveSync and Horde_Mail, Horde_Smtp, Horde_Imap_Client are really shiny. I use Horde_Date so much I even introduced it in some non-horde software – even though most functionality is also somewhere in php native classes.

The ultimate goal however is to enable horde groupware installations out of composer. This requires more work to be done. There are several issues.

  • The db migration tool checks for some pear path settings during runtime https://github.com/horde/Core/pull/2 Most likely there are other code paths which need to be addressed.
  • Horde Libraries should not be web readable but horde apps should be in a web accessible structure. Traditionally, they are installed below the base application (“horde dir”) but they can also be installed to separate dirs.
  • Some libraries like Horde_Core contain files like javascript packages which need to be moved or linked to a location inside another package. Traditionally, this is handled either by the “git-tools” tool linking the code directory to a separate web directory or by pear placing various parts of the package to different root paths. Composer doesn’t have that out of the box.

Horde already has been generating composer manifest files for quite a while. Unfortunately, they were thin wrappers around the existing pear channel. The original generator even took all package information from the pear manifest file (package.xml) and converted it. Which means, it relied on a working pear installation. I wrote an alternative implementation which directly converts from .horde.yml to composer.json – Calling the packages by their composer-native names. As horde packages have not been released on packagist yet, the composer manifest also includes repository links to the relevant git repository. This should later be disabled for releases and only turned on in master/head scenarios. Releases should be pulled from packagist authority, which is much faster and less reliant on existing repository layouts. https://github.com/horde/components/pull/3

To address the open points, composer needs to be amended. I currently generate the manifests using package types “horde-library” and “horde-application” – I also added a package type “horde-theme” for which no precedent exists yet. Composer doesn’t understand these types unless one adds an installer plugin https://github.com/maintaina-com/installers. Once completed and accepted, this should be upstreamed into composer/installers. The plugin currently handles installing apps to appropriate places rather than /vendor/ – however, I think we should avoid having a super-special case “horde-base” and default to installing apps directly below the project dir. Horde base should also live on the same hierarchy. This needs some additional tools and autoconfiguration to make it convenient. Still much way to go.

That said, I don’t think pear support should be dropped anytime soon. It’s the most sensible way for distribution packaging php software. As long as we can bear the cost involved in keeping it up, we should try.

bookmark_borderCurrent (10/2018) Tumbleweed on Raspberry Pi 1

Hallo,

I just had a little struggle getting the current tumbleweed to run on the original Raspberry Pi (first generation, though the revision with larger RAM).

Just in case this helps anybody: I did not have any luck with a fresh openSUSE Tumbleweed image of one of the current arm6 builds. Don’t know why.

Here’s what I did:
– Download a pretty old known-good OpenSUSE 13.1 built by Bernhard Wiedemann

http://www.zq1.de/bernhard/linux/opensuse/raspberrypi-opensuse-latest.img.xz

Unzip, dump it to SD Card

xz -d raspberrypi-opensuse-latest.img.xz
dd if=raspberrypi-opensuse-latest.img of=/dev/mmcblk0 bs=8M

Boot up, change to text console (CTRL + ALT + F2)
Log In (root/linux)

Change Password (passwd)

nano /etc/zypp/repos.d/oss131.repo
Change baseurl line to
baseurl=http:/download.opensuse.org/ports/armv6hl/tumbleweed/repo/oss/
Save and get out (CTRL+X, Y)

#Resize partition and FS as this build won’t do:
# Adjust to more if your card is larger or to less if you need a more advanced partitioning scheme

parted resize 3 16G

resize2fs /dev/mmcblk0p3

zypper ref

zypper up gzip rpm

zypper dup –download in-advance

#(super conservative, get all needed packages first) – This is going to take quite a while

reboot, power cycle

Note that you will end up with a system booting into X11 login. You should probably change the default systemd target and maybe also get rid of some software. And you really don’t want a server with ssh password “linux”, so better don’t skip changing the PW

 

bookmark_borderhorde trustr – A new horde CA app step by step

Trustr is my current project to create a simple certificate management app.
I decided that it is just about the right scope to demonstrate a few things about application development in Horde 5.

I have not made any research if the name is already occupied by some other software. Should any problems arise, please contact me and we will find a good solution. I just wanted to start without losing much time on unrelated issues.

My goals as of now:

– Keep everything neat, testable, fairly decoupled and reusable. The core logic should be exportable to a separate library without much change. There won’t be any class of static shortcut methods pulling stuff out of nowhere. Config and registry are only accessed at select points, never in the deeper layers.
– Provide a CLI using Horde_Cli and Horde_Cli_Application (modeled after the backup tool in horde base git)
– Store to relational database using Horde_Db and Horde_Rdo for abstraction
– Use php openssl extension for certificate actions, but design with future options in mind
– Rely on magic openssl defaults as little as possible
– Use conf.xml / conf.php for any global defaults
– Show how to use the inter-app API (reusable for xml-rpc and json-rpc)
– Showcase an approach to REST danabol ds in Horde (experimental)

The app is intended as a resource provider. The UI is NOT a top priority. However, I am currently toying around with a Flux-like design in some unrelated larger project and I may or may not try some ideas later on.

Initial Steps: Creating the working environment

I set up a new horde development container using the horde tumbleweed image from Open Build Service and a docker compose file from my colleague Florian Frank. Please mind both are WIP and improve-as-needed projects.


git clone https://github.com/FrankFlorian/hordeOnTumbelweed.git
cd hordeOnTumbelweed
docker-compose -f docker-compose.yml up


This yields a running horde instance on localhost and a database container.
I needed to perform a little manual setup in the web admin ui to get the DB to run and create all default horde schemas.

Next I entered the developer container with a shell
docker exec -it hordeOnTumbelWeed_php_1 bash

There are other ways to work with a container but that’s what I did.

 

Creating a  skeleton app

The container comes with a fairly complete horde git checkout in /srv/git/horde and a clone of the horde git tools in /srv/git/git-tools

A new skeleton app can be created using

horde-git-tools dev new --author "Ralf Lang <lastname@b1-systems.de>" --app-name trustr

The new app needs to be linked to the web directory using

horde-git-tools dev install

Also, a registry entry needs to be created by putting a little file into /srv/git/horde/base/config/registry.d

cat trustr-registry.d.php

<?php
// Copy this example snipped to horde/registry.d
$this->applications['trustr'] = array(
'name' => _('Certificates'),
'provides' => array('certificates')
);

 

This makes the new app show up in the admin menu. To actually use it and make it appear in topbar, you also need to go to /admin/config and create the config file for this app. Even though the settings don’t actually mean anything by now, the file must be present.

I hope to follow up soon with articles on the architecture and sub systems of the little app.