Archives

bookmark_borderTurba Addressbook (II) – Architecture

Posted on by Ralf

Welcome back to our mini series on Turba.

Part I covered all the features and integrations provided by Turba.
Part II gives a dive into implementation, code structure etc.
Part III will consist of proposals for a changed architecture.

In the first chapter we looked at Turba’s features, APIs, Protocols. In the current installment, I want to present the concepts and structure of the code.

Turba is among the oldest horde applications. As such, it contains parts from various stages of Horde’s development.
Basically, it’s a layered architecture, but not fully fleshed-out or fully separated.

  • Presentation layer
  • Application Logic Layer
  • Storage/Backend layer

This is plugged together with some framework-provided integration points

  • with the sync services and inter-app API, RPC
  • with the portal/blocks service
  • with the Backup API
  • with the Content Tagger

Presentation Layer.

Turba provides both a desktop UI and a mobile UI. The following is mostly about the desktop UI. Blog de culturismo total fitness de lee hayward: debes probar estos entrenamientos musculares híbridos primobolan depot inicio – culturismo femenino y control de la natalidad, entrenamientos de culturismo femenino youtube – la casa de juegos del kama sutra negro.

Turba’s UI is organized into client pages rather than using a controller/route approach. This means, user visible URLs include files with a .php suffix.
The client pages build the UI, but are also API endpoints in a very traditional sense, catch interaction from forms or buttons as get variables and trigger actions in the backend
A representative example:

https://github.com/horde/turba/blob/master/search.php#L155
try {
$share = Turba::createShare(strval(new Horde_Support_Randomid()), $params);
$vid = $share->getName();
} catch (Horde_Share_Exception $e) {
$notification->push(sprintf(_("There was a problem creating the virtual address book: %s"), $e->getMessage()), 'horde.error');
Horde::url('search.php', true)->redirect();
}

Typically the top part of each client page is initializing the application, catching request and environment/session variables, setting up the business objects.
The middle part usually orchestrates actions depending on present or missing parameter scenarios.

The lower part will actually output the Horde Topbar, utility javascript, and the actual page content.

A rather extreme example is the data import/export part: https://github.com/horde/turba/blob/master/data.php
It contains mappings, attribute filtering, a longer cascade of if’s and switch statements and after about 360 lines, the actual UI logic starts.

This might sound messier than it really is. Actual functionality is mostly factored out into separate classes and the UI uses both some View classes and a form library.
Turba’s UI is heavy in forms and tables for the very reason that editing and displaying a highly configurable addressbook with hundreds of fields of data is
very crud-like by nature.

The Turba UI uses three types of helpers to compose the UI:

  • View classes
  • The Forms library
  • HTML templates with PHP snippets

View Classes

The View classes are very similar to the horde/view library and replicate some of its functionality, but are not using or inheriting from it.

Forms Library

Turba is a prominent user of the horde/forms library.
This utility allows to dynamically compose forms or multiple fields, check for internal/formal validity of entered values, missing mandatory values, etc.
It couples both a readonly and editable representation with a lot of processing logic.
Forms relies mostly on server-generated HTML with small parts of javascript injected for usability improvements.
The JavaScript snippets may utilize PrototypeJs and Scriptaculous, two formerly popular mainstream libraries.

Templates

The HTML templates provide most of the actual presentation apart from the forms – though they also provide some HTML forms to drive interaction.

Mobile View

Turba provides a read-only, touch friendly mobile phone presentation based on jQuery Mobile. This is completely separate from the rest of the turba UI.

Application Logic Layer and Backend Layer.

These two layers are closely tied together so it makes sense to discuss them as one.
From a problem domain perspective, it would make sense to expect these items:

  • Multiple addressbook sources or backends. These are individual configurations using drivers. Multiple sources on the same ldap driver can represent different directories or different views on one.
  • Addressbooks
  • Addressbook entries or contacts
  • Groups which are both entries and contain entries.

Turba’s logic layer consists of

  • Representations of actual addressbook entries via the Object and Object_Group classes.
  • A collection of reusable static functions in the Turba class.
  • Parts of the base backend driver.
  • Exporters/Importers for formats like LDIF and vCard.
  • specialised forms which deal both with presentation and state transformations.

Turba Objects and Groups are the common ground here. The objects glue together the actual data from the driver, files from VFS, permissions managed by the driver, the object’s change history and tags in the tagger app.
Groups are the only subtypes of Objects or Entries. Other subtypes suggested by the vcard standard like organisations or locations get no special treatment. Groups act as virtual addressbooks or views on addressbook data.
Turba’s groups only work with turba-accessible contacts. They cannot reference external contacts from other sources.

The addressbooks don’t really show up as entity objects. They are arrays of passive data managed by different parties.
This makes part of the logic a little hard to reason about and to setup unit tests.

A lot of Turba’s logic is data transformation. Backends have a native representation of data as SQL columns, LDAP objects, etc… as well as possibly native key names.
A person’s name may be a column “object_lastname” in one backend, but an attribute “givenName” in another one.

This is complicated by a highly configurable list of fields each backend can hold.
A driver transforms these native formats into a uniform format, transform date strings into date objects, handle blobs etc…
Both the driver dependent format and the “turba” format are hashes. It is up to the driver to actually generate a list object containing individual addressbook entry objects.

Another functionality delegated to the driver is deriving TimeObjects from addressbook data. TimeObjects are really not objects but hashes, representing anniversaries or birthdays.

Permission management in turba is using three distinct approaches:

  • The permission system allows restricting who can view or edit a certain addressbook
  • One addressbook source, usually the SQL db, can be configured as using the Shares system. In this case, the user can delegate access to addressbooks to other users or groups or make them world readable
  • The backing technology can restrict the user further. For example, the LDAP driver can be used to either bind using a service credential or using the user’s credential. Different users could have completely different data presented based on LDAP ACLs.

In the next article of this series, I am going to propose some modernisation approaches for Turba and discuss how they bring benefits in maintaining or extending the software vs being a tedious refactoring exercise.

bookmark_borderTurba Addressbook (I) – Features

Posted on by Ralf

I Turba Feature Overview

This will be the first part of a short series of articles exploring the Turba application and its architecture.
Part I covers all the features and integrations provided by Turba.
Part II will look into implementation, code structure etc.
Part III will consist of proposals for a changed architecture.

Turba is the addressbook application of the Horde Groupware Suite. It offers access to addressbook information via various means and protocols.

These addressbooks can be of quite different nature:

  • Readonly or Read/Write addressbooks stored in a backing system (LDAP, SQL databases, Kolab)
  • Virtual addressbooks derived from Horde Groups or from favourite addresses data out of the IMP mail program or any other provider implementing the appropriate API.
  • Addressbooks representing contact groups stored in addressbooks
  • There is also a pseudo backing store through the Preferences API, which in turn needs some backing store.

Depending on the type of addressbook and backing system, different features are available

  • browsing
  • searching for entries matching criteria
  • user-controlled sharing addressbooks with others
  • administrative reading or writing of data without the user being logged in

Administrators are allowed to customize the fields and presentation of the addressbooks presented to the user.
They may add custom fields relevant for their site like a student ID number or an ssh public key.
They may remove almost any default field if their LDAP does not expose it or if it’s simply unwanted.

While the primary work model used to be tp browse/search the addressbook or make the addressbook data available in the web interfaces of the calendar and webmail program,
modern users often consume the addressbooks by other methods.

  • Turba allows for syncing addressbooks via the CardDAV protocol into mobile devices or desktop clients like Mozilla Thunderbird.
  • Addressbooks can also be transmitted via Exchange ActiveSync (EAS) or used as a Global Address List (GAL). This allows limited integration with Microsoft Outlook.
  • An older protocol SyncML offers similar, but limited capabilities. It has fallen out of wide spread use since. Turba is still a valid SyncML Server.
  • Addressbooks and contacts are also served via WebDAV as folders and files.
  • Imports/exports to CSV/TSV and LDIF formats, vCard format and some proprietary formats.
  • Addressbook and Contact Reading/Writing is exposed via the Horde Inter-App API and – through this layer – via JSON-RPC and XML-RPC. The RPC layer also has a SOAP interface and an integration driver for phpGroupware / eGroupware though I have no idea if this works with current versions.

The next article in this series will give an overview of the current architecture.

bookmark_borderWhat’s new in Maintaina Horde: Status 3/2021

Posted on by Ralf
  • CalDAV and CardDAV now run off SabreDAV 4 rather than SabreDAV 2
  • We now support both the Composer installer versions 1 and 2.
  • Nothing still depends on the PEAR protocol.
  • The Horde Icalendar Library now supports vCard 4. Still, importing/exporting vCard 4 or using it in CardDAV in the addressbook App Turba is not yet done. This requires good test coverage, syncing is not something I’d like to break
  • PHPUnit Tests are being upgraded from PHPUnit 4 to PHPUnit 9.
  • All libraries and groupware-related libraries are now packaged as “alpha” versions from the FRAMEWORK_6_0 branch

bookmark_borderMaintaina Horde: Q&A

Posted on by Ralf

There’s sometimes a little confusion about that. Let’s answer common questions.

What is Maintaina Horde?

Horde is a long-living email and groupware solution. It’s free & open source and owned by Horde LLC.
Maintaina Horde is a fork from the official horde repository adding / improving code. It started out as a proof of concept but it went on to become stable and persistent.

Maintaina is not a company. Everything written for this fork is intended to be upstreamed into proper horde as time and quality permit. Maintaina includes both results of paid work hours and free time volunteering, depending on what’s the current job of the persons involved and how Horde belongs into the picture. There’s really little time for any marketing, sorry.

Who runs this for production?

Yes, this is in daily production use. B1 Systems GmbH, the company I work with, is using both the Horde Groupware and the Horde Framework for internal and external customers. They also offer paid development, operations and consulting services. Get it hosted externally, get it set up on premise/in your DC or cloud, consume security upgrades… yeah. You get it.

Can I run your fork for personal or commercial use?
Yes. All the needed parts are released on public github, including ready-to-run daily container builds and a turnkey docker-compose solution. Everything’s free, go ahead.

If you need something stable and reliable on the other hand, there’s more work to do. It’s not advisable to run production setups off bleading edge nightly code. You need to pick and review versions, run quality checks, decide if an update is good for you. That’s work you could do yourself or delegate to some company. Did I mention B1 Systems GmbH?

Will it run my custom app?

We’ve gone through quite some pains to keep everything as backward compatible as possible. If your custom app runs off horde 5.2 or horde master branch, it’s very likely it will run without any code level modification. You may need to craft a composer.json file to get the autoloading right. Another issue may be with code trespassing into other packages’ directory layout and unconditionally include/require’ing any files. These are easy to fix.

Will this code run from a traditional pear or git-tools setup or a distribution package?

Most likely. If you submit PRs addressing problems with that, they’re welcome. It’s just not our priority.

When will the proper Horde 6 be released? Will it contain all the Maintaina enhancements?
I have little say in that. FOSS projects are always short on time and developers. The owners of horde might disagree with some changes. Most likely delays happen because they lack time to review changes for quality. There’s a paypal button on the Horde Website.

I need that feature X …
Open an enhancement request at https://bugs.horde.org if you want to suggest an enhancement.
Or open a pull request on github.com to directly supply a change you developed.
If you need something quickly, ask the companies mentioned for a paid solution, be it a private customisation or a public feature enhancement. However, only Horde LLC has the last word on what goes into their upstream product.

Can I hire you directly?

No. I don’t do freelance work.

Links and Resources

Horde Upstream: https://github.com/horde
Maintaina Horde Fork: https://github.com/maintaina-com
Maintaina images and deployments: https://github.com/maintaina

2017 herfst nieuwe mannen katoen sweatshirt sportscholen fitness bodybuilding workout hoodies casualdresskily sustanon ervaring yemeke 2018 zomer casual shorts voor heren is een trainingsbroek voor heren, fitness, bodybuilding, workout, fashion heren, shorts.

bookmark_borderMaintaina Horde: Now on Leap 15.2

Posted on by Ralf

I changed the maintaina.com Horde Images to use openSUSE Leap 15.2 instead of openSUSE Tumbleweed as a base. You should not experience any issues but I have not yet tested much. The change was necessary as I had build failures since 2021-02-17 in the github actions CI builds. It’s something about the docker used in ubuntu-latest being too old or github’s security settings being too strict. I did invest some weekend hours, but could not really solve it. I intend to switch back to tumbleweed at some point.

Update July 2021: I can now build tumbleweed-based images again.

bookmark_borderDAVx5 CalDAV may break with Unicode symbols in Horde/Kronolith syncs.

Posted on by Ralf

If you get user complaints about broken CalDAV syncs with Horde, there’s many places to look at. In one particular instance, an event was created from travelling app Transportr into the stock android Calendar app. Through the DAVx5 sync app, the user wanted to push these events to Horde’s SabreDAV interface – and from there, also sync it to his desktop email solution, Mozilla Thunderbird.

5 programa efectivo de entrenamiento de culturismo – programa de entrenamiento de culturismo de 5 días estore de culturismo halotestin szkani hombre sin mangas con capucha chaleco de fitness culturismo largueros entrenamiento camisetas sin mangas.

However, his sync application told him about an error. The server administrator saw a 500 status code in the server log.

1.1.2.11 - user [24/Jan/2021:18:48:26 +0000] "PUT /horde/rpc/calendars/user/calendar~KL14jYhCMpbet4ecYzAg1bn/2bace303-f0d8-4df6-9652-baa5fb9e86c4.ics HTTP/1.1" 500 892 "-" "DAVx5/3.3.8-ose (2021/01/13; dav4jvm; okhttp/4.9.0) Android/11

The root cause was actually not in the software code but in the MariaDB database configuration. The calendar entry from Transportr included some Unicode icon characters like a fast train and some arrows. These characters are part of the standard unicode encoding, utf-8.

Now you might wonder: New installations of mysql and MariaDB default to a character set they call utf-8 since 2010 or so. This shouldn’t be an issue. However, what they call utf-8 is not what you would expect.

Some years ago, TV sets which did not support the full HD resolution were marketed as “HD ready”. In some sense, the default character set is “unicode ready” at best. The default data type saves on disk space by encoding a subset of utf-8 into up to three bytes. While this supports most natural language characters, it is only a fraction of what Unicode can offer. Database manufacturers are well aware that this is not something you should run nowadays that unicode icons like the hamburger are all over the place is user generated content. The mysql manual even says:

“Please use utf8mb4 instead. Although utf8 is currently an alias for utf8mb3, at some point utf8 is expected to become a reference to utf8mb4. To avoid ambiguity about the meaning of utf8, consider specifying utf8mb4 explicitly for character set references instead of utf8.”

Now that’s what I did. First I changed horde’s database encoding to utf8mb4 in conf.php:

$conf['sql']['protocol'] = 'tcp';
$conf['sql']['charset'] = 'utf8mb4';

Then I changed the mysql server’s and client’s default charset:

cat /etc/mysql/my.cnf
[server]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
[client]
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

I have obviously stripped a lot of file content not relevant to our story. After reloading the database, all new connections should use the real utf8 encoding and new tables should be created with the new standard. But what about existing content? We need to convert all tables and all their text-like columns, varchars, mediumtexts etc.

First, it’s backup time – better safe than sorry.

Then, let’s find all tables in our db server and feed them conversion commands.

mysql --database=horde -B -N -e "SHOW TABLES" | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"}' | mysql --database=horde

Do this in a downtime or ensure HA some way or other. It will take some time. Some sources suggest it might be sensible to also rebuild the tables with the optimize command. I am not very convinced, but it won’t harm.

mysqlcheck --auto-repair --optimize --user="hidden" -p --databases mysql horde

After this, repeat the sync test. It should work this time.

bookmark_borderHorde Development Review: October 2020

Posted on by Ralf

October was a very busy month in Horde development, even though a lot of things happened under the hood and cannot be accessed right now.

  • A decision was made that Horde RPC will default to json-rpc and deprecate xmlrpc in upcoming releases, maybe already dropping xmlrpc in Horde 6. PHP8 will remove xmlrpc from the core distribution and move it into PECL, giving it much less of an installation base.
  • B1 Systems GmbH released a typescript client for the Horde Ajax Framework https://github.com/b1-systems/horde-ajax-client
  • Horde Injector: I continued my work from September. The Maintaina version of Injector now supports PSR-4, PSR-11, PSR-12 specifications. Backward compatibility is good. The unit tests have been converted to namespaced versions – see https://github.com/maintaina-com/Injector/tree/refactor-psr4
  • Horde Components: The Components tool has been converted for PSR-4 autoloading in a composer setup. Also, some minor improvements on the composer file generator have been added
  • I worked on upgrading Diana Hille’s DNS library for the horde framework, giving it the same PSR-4 makeover, converted it from wrapping AWS cli to wrapping AWS Route53 SDK. I think we can opensource the component soon.
  • I created a little testbed application for the improved controller/routes handling in recent horde/core. horde/frogsie is the first Horde application stub to feature a RESTish interface. Frogsie writes iptables rules to implement a TCP/UDP Front Proxy in plain linux.
    It doesn’t sport web UI or CLI though as it’s both a case study and a stop-gap utility. However, it’s a good example to learn from.
  • Horde Installer Plugin v1.1.0 supports both composer 1.x and composer 2.0 – This is mostly based on work by Kieran Brahney of SupportPal. I also began working on converting horde-deployment to remove the last bits of PEAR and make it compatible with composer 2.0. It’s not released yet.
  • I am looking at the old vilma application at the moment. This will most likely not materialize before november. Vilma is a simple mailbox manager along the lines of postfixadmin. My primary interest is using it together with hordectl for a turnkey default setup. Vilma will get the PSR-4 treatment and likely I will want to change some of the architecture. Focus is on a minimum viable product that nicely integrates with my groupware bundle and fulfills very limited needs. I appreciate the work of Christian Bolz on PostfixAdmin and I do not intend to build a full scale replacement.

What’s next?

  • More library conversions to PSR-4, PHP7 features.
  • Most likely I will upgrade unit tests to a more recent PHPunit
  • Release Cleanup / Upstreaming efforts for existing non-released code.
  • Better theme support for horde-installer-plugin

Results first, more talk later.

bookmark_borderAutowiring Vfs in a Horde App

Posted on by Ralf

The Horde Vfs is an abstraction around storing and retrieving files. Calling code does not care about where the Vfs is actually stored, be it a remote filesystem, a dav resource, a database or a path in the local filesystem. Autowiring means the Injector knows how to create a class using some other class without having it explicitly defined

For this article, I want to get an instance of Horde\MyApp\ReportGenerator which uses the VFS to store reports.

First let’s create the ReportGenerator.

lib/ReportGenerator.php

<php
namespace Horde\MyApp;
class ReportGenerator
{
    private $driver;
     public function __construct(\Horde_Vfs_Base $driver
     {
         $this--->driver = $driver;
     }
     ...
}

When a controller in the app wants a report generator, it would require it through the dependency injector like this:


$reportGenerator = $injector->getInstance('Horde\MyApp\ReportGenerator');

Autowiring allows the injector to return an instance even if it has not been explicitly bound to the injector. For this to work, all required dependencies must be interfaces which have either been registered with the injector or can themselves be created from registered interfaces or have no dependencies themselves.

Now, Horde_Vfs_Base is the common base class of all Horde Vfs drivers.

However, Horde does not provide a default binding for Horde_Vfs_Base even though Horde_Core has a default factory Horde_Core_Factory_Vfs.

Thus, we need to define one. In Application.php, look for a method _bootstrap(). We will use it to setup the injector binding. Normally, we would just want to use the existing factory through $injector->bindFactory($interface, $factory, $method);

We cannot do this in this case. The Vfs Factory’s create method has two string parameters to allow creating multiple Vfs instances. For our app, we just want to get the default configured in horde base. But this doesn’t work. bindFactory would automatically pass a child injector to the create method. This would override the default first parameter “horde”. In the end, we would receive the factory’s exception saying we should configure a backend first – even if we did.

To work around this, we could create an own factory class in our app which uses Horde_Core_Factory_Vfs as a dependency. Horde_Injector would just know what to do. But let’s not do this as it’s quite some boilerplate just for creating a Vfs with default configuration.

Instead, let’s use the closure binder.


protected function _bootstrap()
{
     $injector = $GLOBALS['injector'];
     $vfsClosure = function(\Horde_Injector $injector) {
     return $injector->getInstance('Horde_Core_Factory_Vfs')->create();
};
$injector->bindClosure('Horde_Vfs_Base', $vfsClosure);

Now injector knows how to provide a Horde_Vfs_Base implementation which the ReportGenerator requires. We can ask injector to provide an instance. Injector will look if it already has an instance from a previous call, otherwise runs the closure to create the dependency, then create the actual ReportGenerator instance.

bookmark_borderDocker for Windows now runs on Windows Home

Posted on by Ralf

Docker for Windows used to require Hyper-V and Windows Professional. Now, you can do basic software development on Windows Home. Here’s the story.

I got hold of some older 4GB Windows convertible laptop. Unfortunately, it won’t run Linux in any useful way. In recent years, I did not really care for Windows a lot, apart from the usual “fix your parent’s gear for christmas” routine. Looking for uses case it would possibly cover, I found Windows has improved quite a lot.

Windows 10 May 2020 Update (also known as version 2004) introduces improvements to the “Windows Sub System For Linux” (WSL2). Recent versions of Docker Desktop CE for Windows allow to run based on WSL2. It’s even better than the original version based on Hyper-V.

To run Docker Desktop, you need to install the WSL and “Virtualization Platform” options in Windows. This requires a reboot. Docker Desktop will ask you to download a special Linux kernel from a Microsoft website. Docker and the Kernel will require an additional reboot. Microsoft promises that the manual kernel install is a one-time issue. Future kernel releases will be published via Windows Updates.

Now why bother? With the latest Windows improvements, I can keep most of my development workflows when switching to a tiny, low-powered device which I don’t especially care for – think of spare time while traveling by train or some free time project while laying in the hammock. There’s no way I would use my bigger, more expensive equipment in such situations.

The new optional Windows Terminal is available both from the Microsoft website and from their App Store. It’s free and as close as you can get to a native KDE Konsole or Gnome Terminal. It’s not as awkward as putty nor as ugly as the classic cmd or Power Shell. Paired with Windows’ posix-like ssh client suite, there’s little difference too remote sessions from Linux desktops .

Visual Studio Code feels exactly the same on both Linux and Windows platforms. It allows to edit code directly inside a runtime container. No need for rsync workarounds or push cycles. Keeping everything in the containers also greatly reduces the time to recover from a full reset.

In case you didn’t know, Docker Desktop has some builtin Kubernetes option. This allows development to be even closer to cloud/production scenarios. However, on a 4GB device, don’t expect too much.

The Windows Store now includes some FOSS software, even some KDE for Windows gadgets. Still no Firefox, Thunderbird, Chrome / Chromium … Microsoft has announced they will release some apt/zypper like commandline software installer later this year. Let’s hope that gets adopted more widely.

Windows Store offers some mainstream linux distributions like openSUSE Leap or ubuntu to run on top of WSL/WSL2. I have decided to postpone experimenting with these. Microsoft has announced they want to enable true, builtin capabilities for graphic applications on X/Wayland. I will give this a try when the time is right. It might also make sense to drop Docker Desktop for some podman or other runtime running inside such a linux platform. But for now, that’s out of scope.

While I’m still not ready to embrace Windows as a first-class development platform, I must concede it has become a viable option.

Update 2020/10: I just read an article on Medium which I totally agree with: https://levelup.gitconnected.com/this-is-why-developers-will-embrace-microsoft-windows-again-7437e494159d

They also point out what I am currently doing, using WSL2, docker desktop (with Kubernetes) and VS Code for an integrated environment.

bookmark_borderHeads Up: Cannot login as horde admin anymore?

Posted on by Ralf

Please be careful: Horde has introduced a new default setting $conf[‘auth’][‘lowercase’] which acts like the auth hook used to do.

This might lead to issues when your default admin account is named “Administrator”.

It is actually a very useful setting. It ensures that any prefs and other profiles also work with case-insensitive backends regardless of how you type the user name.

How to fix?

a) Lowercase all admin names in your $conf[‘auth’][‘admin’] settings in conf.php

b) disable $conf[‘auth’][‘lowercase’] by setting it to false value.