Author: Ralf

bookmark_borderHorde/Yaml: Graceful degradation

Posted on by Ralf

Tonight’s work was polishing maintaina’s version of horde/yaml. The test suite and the CI jobs now run successfully both on PHP 7.4 and PHP 8.1. The usual bits of upgrading, you might say.

However, I had my eye on horde/yaml for a reason. I wanted to use it as part of my improvements to the horde composer plugin. Composer famously has been rejecting reading any yaml files for roughly a decade so I need to roll my own yaml reader if I want to deal with horde’s changelog, project definition and a few other files. I wanted to keep the footprint small though, not install half a framework along with the installer utility.

You never walk alone – There is no singular in horde

The library used to pull in quite a zoo of horde friends and I wondered why exactly. The answer was quite surprising. There is no singular in horde. Almost none of the packages can be installed without at least one dependency. In detail, horde/yaml pulled in horde/util because it used exactly one static method in exactly one place. It turned out while that method is powerful and has many use cases, it was used in a way that resulted in a very simple call to a PHP builtin function. I decided whenever the library is not around I will directly call that function and lose whatever benefits the other library might grant over this. This pattern is called graceful degradation. If a feature is missing, deliver the next best available alternative rather than just give up and fail. The util library kept installing although the yaml parser no longer needed it. The parser still depended on the horde/exception package which in turn depended on horde/translation and a few other helpers. Finally horde/test also depended on horde/util. It was time to allow a way out. While all of these are installed in any horde centric use case, anybody who wants only a neat little yaml parser would be very unhappy about that dependency crowd.

Alternative exceptions

The library already used native PHP exceptions in many places but wrapped Horde exceptions for some more intricate cases. While this is all desirable, we can also do without it. If the horde/exception package is available, it will be used. Otherwise one of the builtin exceptions is raised instead. This required to update the test suite to make it run correctly either way. But what is the point if the test suite will install horde/util anyway?

Running tests without horde/test unless it is available

I noticed none of the tests really depended on horde/test functionality. Only some glue code for utilities like the horde/test runner or horde/components really did anything useful. I decided to change the bootstrap code so that it would not outright fail if horde/test was not around. Now the library can be tested by an external phpunit installation, phar or whatever. It does not even need a “composer install” run, only a “composer dump-autoload --dev” to build the autoloader file.

A standalone yaml parser

The final result is a horde/yaml that still provides all integrations when run together with its peer libraries but can be used as a standalone yaml parser if that is desirable. I hope this helps make the package more popular outside the horde context.

Lessons learned

Sometimes less is more. Factoring out aspects for reuse is good. Factoring out aspects into all-powerful utility libraries like “util”, “support” and the likes can glue an otherwise self contained piece of software together with too many other things. That makes them less attractive and harder to work with. Gracefully running nevertheless is one part. The other is redesigning said packages which cover too many aspects at once. This is a topic for another article in another night though.

bookmark_borderPHP: Tentative Return Types

Posted on by Ralf

PHP 8.1 has introduced tentative return types. This can make older code spit out warnings like mad.
Let’s examine what it means and how to deal with it.

PHP 8.1 Warnings that will become syntax errors by PHP 9

PHP 7.4 to PHP 8.1 have introduced a lot of parameter types and return types to builtin classes that previously did not have types in their signatures. This would make any class extending builtin classes or implementing builtin interface break for the new PHP versions if they did not have the return type specified and would create interesting breaks on older PHP versions.

Remember the Liskov Substitution Principle (LSP): Objects of a parent class can be replaced by objects of the child class. For this to work, several conditions must be met:

  • Return types must be covariant, meaning the same as the parent’s return type or a more specific sub type. If the parent class guarantees to return an iterable then the child class must guarantee an iterable or something more specific, i.e. an ArrayObject or a MyFooList (implements an iterable type).
  • Parameter types must be contravariant, meaning they must allow all parameters the parent would allow, and can possibly allow a wider set of inputs. The child class cannot un-allow anything the parent would accept.
  • Exceptions are often forgotten: Barbara Liskov‘s work implies that Exceptions thrown by a subtype must be the same type as exceptions of the parent type. This allows for child exceptions or wrapping unrelated exceptions into related types.
  • There are some more expectations on the behaviour and semantics of derived classes which usually are ignored by many novice and intermediate programmers and sadly also some senior architects.

Historically, PHP was very lax about any of these requirements. PHP 4 brought classes and some limited inheritance, PHP 5 brought private and protected methods and properties, a new type of constructor and some very limited type system for arrays and classes. PHP 7 and 8 brought union types, intersection types, return type declaration and primitive types (int, string) along with the strict mode. Each version introduced some more constraints on inheritance in the spirit of LSP and gave us the traits feature to keep us from abusing inheritance for language assisted copy/paste. Each version also came with some subtle exceptions from LSP rules to allow backward compatibility, at least for the time being.

In parallel to return types, a lot of internal classes have changed from returning bare PHP resources to actual classes. Library code usually hides these differences and can be upgraded to work with either, depending on which PHP version they run. However, libraries that extend internal classes rather than wrapping them are facing some issues.

PHP’s solution was to make the return type tentative. Extending classes are supposed to declare compatible return types. Incompatible return types are a syntax error just like in a normal user class. Missing return types, no declaration at all, however, are handled more gracefully. Before PHP 8.1, they were silently ignored. Starting in PHP 8.1 they still work as before, but emit a deprecation notice to PHP’s error output, usually a logfile or the systemd journal. Starting in PHP 9 they will be turned into regular syntax errors.

Why is this good?

Adding types to internal classes helps developers use return values more correctly. Modern editors and IDEs like Visual Studio Code or PhpStorm are aware of class signatures and can inform the users about the intended types just as they write the code. Static analysis tools recognize types and signatures as well as some special comments (phpdoc) and can give insight into more subtle edge cases. One such utility is PHPStan. All together they allow us to be more productive, write more robust code with less bugs of the trivial and not so trivial types. This frees us from being super smart on the technical level or hunting down inexplicable, hard to reproduce issues. We can use this saved time and effort to be smarter on the conceptual level: This is where features grow, this is where most performance is usually won and lost.

Why is this bad?

Change is inevitable. Change is usually for the better, even if we don’t see it at first. However, change brings maintenance burden. In the past, Linux distributions often shipped well-tested but old PHP versions to begin with and release cycles, especially in the enterprise environment, were quite long. Developers would have had to write code that would run on the most recent PHP as well as versions released many years ago. Administrators would frown upon developers who always wanted the latest, greatest versions for their silly PHP toys. Real men use Perl anyway. But this has changed a lot. Developers and administrators now coexist peacefully in DevOps teams, CI pipelines bundle OS components, PHP and the latest application code into container images. Containers are bundled into deployments and somebody out there on the internet consumes these bundles with a shell oneliner or a click in some UI and expects a whole zoo of software to start up and cooperate. Things are moving much faster now. The larger the code base you own, the more time you spend on technically boring conversion work. You can be lucky and leverage a lot of external code. The downside is you are now caught in the intersection between PHP’s release cycle and the external code developer’s release cycles – the more vendors the more components that must be kept in sync. PHP 9 is far away but the time window for these technical changes can be more narrow than you think. After all, you have to deliver features and keep up with subtle changes in the behaviour and API of databases, consumed external services, key/value stores and so on. Just keeping a larger piece of software running in a changing and diverse environment is actually hard work. Let’s look at the available options.

How to silence it – Without breaking PHP 5

You can leverage a new attribute introduced in PHP 8.1 – just add it to your code base right above the method. It signals to PHP that it should not emit a notice about the mismatch.

<?php
class Horde_Ancient_ArrayType implements ArrayAccess {
    /**
     * @return bool PHP 8.1 would require a bool return time 
     */
    #[\ReturnTypeWillChange]
    public function offsetExists(mixed $offset) {
        // Implementation here
    }
...
}

Older PHP that does not know this attribute would just read it as a comment. Hash style comments have been around for long and while most style guides avoid them, they are enabled in all modern PHP versions. This approach will work fine until PHP 9.

How to fix it properly – Be safe for upcoming PHP 9

The obvious way forward is to just change the signature of your extending class.

<?php
class Horde_Ancient_ArrayType implements ArrayAccess {
    public function offsetExists(mixed $offset): bool {
        // Implementation here
    }
...
}

The change itself is simple enough. If your class is part of a wider type hierarchy, you will need to update all downstream inheriting classes as well. If you like to, you can also reduce checking code on the receiving side that previously guarded against unexpected input or just satisfied your static analyzer.
Tools like rector can help you mastering such tedious upgrade work over a large code base though they require non-trivial time to properly configure them for your specific needs. There are experts out there who can do this for you if you like to hire professional services – but don’t ask me please.

<?php
...
$exists = isset($ancient['element1']);
// No longer necessary - never mind the silly example
if (!is_bool($exists)) {
    throw new Horde_Exception("Some issue or other");
}

Doing nothing is OK – For now

In many situations, reacting at all is a choice and not doing anything is a sane alternative. As always, it depends. You are planning a major refactoring, replace larger parts of code with a new library or major revision? Your customer has signaled he might move away from the code base? Don’t invest.

My approach for the maintaina-com code base

The maintaina-com github organization holds a fork of the Horde groupware and framework. With over 100 libraries and applications to maintain, it is a good example. While end users likely won’t see the difference, the code base is adapted for modern PHP versions, more recent major versions of external libraries, databases, composer as an installer and autoloader. Newer bits of code support the PHP-FIG standards from PSR-3 Logging to PSR-18 HTTP Client. Older pieces show their age in design and implementation. Exactly the amount of change described above makes it hard to merge back changes into the official horde builds – this is an ongoing effort. Changes from upstream horde are integrated as soon as possible.

I approach signature upgrades and other such tasks by grouping code in three categories:

  • Traditional code lives in /lib and follows a coding convention largely founded on PHP 5.x idioms, PSR-0 autoloading, PSR-1/PSR-2 guidelines with some exceptions. This code is mostly unnamespaced, some of it traces back into PHP 4 times. Coverage with unit tests is mostly good for libraries and lacking for applications. Some of this is just wrapping more modern implementations for consumption by older code, hiding incompatible improvements. This is where I adopt attributes when upstream does or when I happen to touch code but I make no active effort.
  • More modern code in /src follows PSR-4 autoloading, namespaces, PSR-12 coding standards, modern signatures and features to an increasing degree. This generally MUST run on PHP 7.4 and SHOULD run on recent PHP releases. This is where I actively pursue forward compatibility. Unit tests usually get a facelift to these standards and PHPStan coverage in a systematic fashion.
  • Glue code, utility code and interfaces are touched in a pragmatic fashion. Major rewrites come with updated standards and approaches, minor updates mostly ensure compatibility with the ever changing ecosystem.

If you maintain a large code base, you are likely know your own tradeoffs, the efforts you keep postponing in favour of more interesting or more urgent work until you have to. Your strategy might be different, porting everything to a certain baseline standard before approaching the next angle maybe. There is no right or wrong as long as it works for you.

bookmark_borderHorde Installer: Recent Changes

Posted on by Ralf

The maintaina-com/horde-installer-plugin has seen a few changes lately. This piece is run on every composer install or update in a horde installation. A bug in it can easily break everything from CI pipelines to new horde installations and it is quite time consuming to debug. I usually try to limit changes.

Two codebases merged

In the 2.3.0 release of November 2021 I added a new custom command horde-reconfigure which does all the background magic of looking up or creating config snippets and linking them to the appropriate places, linking javascript from addon packages to web-readable locations and so on. This is essentially the same as the installer plugin does but on demand. A user can run this when he has added new config files to an existing installation. Unfortunately the runtime environment of the installer plugin and the custom command are very different in terms of available IO, known paths and details about the package. I took the opportunity to clean up code, refactor and rethink some parts to do the same things but in a more comprehensible way. As I was aware of the risks I decided to leave the original installer untouched. I got some feedback and used it myself. It seemed to work well enough.

For the 2.4.0 release I decided to finally rebase the installer onto the command codebase and get rid of the older code. It turned out that the reconfigure command was lacking some details which are important in the install use case. Nobody ever complained because these settings are usually not changed/deleted outside install/update phase. As of v2.4.4 the installer is feature complete again.

New behaviour in v2.4

The installer has been moved from the install/update phase to the autoload-dump phase. It will now process the installation as a whole rather than one library at a time. This simplifies things a lot.reviously, the installer ran for each installed package and potentially did a few procedures multiple times. Both the installer and the horde-reconfigure command will now issue some output to the console about their operation and they will process the installation only once with the updated autoloader already configured. The changes will now also apply on removal of packages or on other operations which require a rewrite of the autoloader. The registry snippets now include comments explaining that they are autogenerated and how to override the autoconfigured values.

Outlook to 2.5 or 3.0

The composer API has improved over the last year. We need to be reasonably conservative to support OS distribution packaged older versions of composer. At some point in the future however I want to have a look at using composer for simplifying life

  • Improve Theme handling: Listing themes and their scope (global and app specific), setting default theme of an installation
  • Turning a regular installation into a development setup for specific libraries or apps
  • Properly registering local packages into composer’s package registry and autoloader (useful for distribution package handling).

Both composer’s native APIs and the installer plugin can support improving a horde admin’s or developer’s life:

  • Make horde’s own “test” utility leverage composer to show which optional packages are needed for which drivers or configurations
  • Expose some obvious installation health issues on the CLI.
  • Only expose options in the config UI which are supported by current PHP extensions and installed libraries
  • Expose a check if a database schema upgrade is needed after a composer operation, both human readable and machine consumable. This should not autorun.

The actual feature code may be implemented in separate libraries and out of scope for the installer itself. As a rule, horde is supposed to be executable without composer but this is moving out of focus more and more.

bookmark_borderMaintaina/Horde UTF-8 on PHP 8

Posted on by Ralf

On recent OS distributions, two conflicting changes can bring trouble.

MariaDB refuses connections with ‘utf-8’ encoding

Recent MariaDB does not like the $conf[‘sql’][‘charset’] default value of ‘utf-8’. It runs fine if you change to the more precise ‘utf8mb4’ encoding. This is what recent MySQL understands to be ‘utf-8’. You could also use ‘utf8mb3’ but this won’t serve modern users very well. The ‘utf8m3’ value is what older MariaDB and MySQL internally used when the user told it to use ‘utf-8’. But this character set supports only a subset of unicode, missing much-used icons like ☑☐✔✈🛳🚗⚡⅀ which might be used anywhere from calendar events sent out by travel agencies to todos or notes users try to save from copy/pasted other documents.

I have changed the sample deployment code to use utf8mb4 as the predefined config value.

Shares SQL driver does not understand DB-native charsets

The Shares SQL driver does some sanitation and conversion when reading from DB or writing to DB. The conversion code does not understand DB native encodings like “utf8mb4”. I have applied a patch to the share library that would detect and fix this case but I am not satisfied with this solution. First, this issue is bound to pop up in more places and I wouldn’t like to have this code in multiple places. Either the DB abstraction library horde/db or the string conversion library in horde/util should provide a go-to solution for mapping/sanitizing charset names. Any library using the config value should know that it needs to be sanitized but should not be burdened with the details. I need to follow up on this.

Update

See https://github.com/horde/Util/commit/7019dcc71c2e56aa3a4cd66f5c81b5273b13cead for a possible generalized solution.

bookmark_borderA new phase in life

Posted on by Ralf

TL;DR – I changed job and this will not affect ongoing maint. of anything Horde

I spent almost my whole work life with a single employer. It was quite a trip. I was part of it as a company grew from a hand full of guys into fifty, then hundred and ever more. I saw how structures developed, how people grew with their tasks and how a brand recognition built. It was a great time and I took advantage of all the opportunities and challenges that came along with it. How many places I traveled. The excitement of speaking at conferences, being a trainer, leading teams, designing architectures.

But after almost 15 years I am at a point where things need to change. Family is a priority now in a different way. Home has a different meaning. I needed a clean break. A few days ago I started a new job with one of Europe’s most relevant software companies. So far everything is shiny and new – I like it.

A little change comes along with it, too. Work life will not involve anything PHP or Horde anymore. There is this new, clear distinction between these things I do for fun or out of private interest on the one hand and earning money on the other. You cannot hire me for freelance work.

Nobody needs to worry. The Maintaina Horde fork is not going away. Development work on PHP 8.1 compatibility and features has not stopped.

Commercial Horde support at B1 Systems will still be around. I had the pleasure to work with an excellent team and that team is well capable of keeping up the expected quality and response times.
If you need any work done for hire, contact that company.

It’s an exciting summer after a pandemic winter. I will possibly take some weeks outside of mailing lists and bug reports to concentrate on things I must do now and things I like to do now. This is going to be fun. Stay tuned for updates.

bookmark_borderNet_DNS2 PHP 8.x compat issue

Posted on by Ralf

TLDR: When using horde/mail_autoconfig or other features using pear/net_dns2 under PHP 8.x, use the “master” branch or wait for a release of version 1.5.3 or higher.

While upgrading the Maintaina Horde codebase for PHP 8.1, I stumbled upon a problem:

[Sun May 15 12:15:15.026499 2022] [php:error] [pid 156] [client 172.23.0.1:43894] PHP Fatal error:  Uncaught ValueError: fread(): Argument #2 ($length) must be greater than 0 in /srv/www/horde/vendor/pear/net_dns2/Net/DNS2/Cache/File.php:147\nStack trace:\n#0 /srv/www/horde/vendor/pear/net_dns2/Net/DNS2/Cache/File.php(147): fread()\n#1 [internal function]: Net_DNS2_Cache_File->__destruct()\n#2 {main}\n  thrown in /srv/www/horde/vendor/pear/net_dns2/Net/DNS2/Cache/File.php on line 147, referer: http://localhost/horde/admin/config/config.php

It turns out Net_DNS2 as of latest version 1.5.2 has a problem in its filesystem cache when run under PHP 8.x – it tries to read the cache file content even if the file size is 0 or the file does not yet exist. In PHP 8.x this yields a ValueError because reading with a length 0 should not be attempted.

Mike Pultz, the maintainer of Net_DNS2 already has a fix in his master branch. I have kindly asked him to release a stable version with this fix. However I understand it may take a while or may not be very high on the daily priority list of todos. I am not sure what other changes the master branch contains which might impair its usefulness to your use case. I am hesitant to put work into a problem somebody else already fixed. I will only provide a downstream-packaged version in case no update happens within reasonable time.

bookmark_borderMaintaina Horde: Tumbleweed and PHP 8.1

Posted on by Ralf

PHP 8.1 is available off the shelf in openSUSE Tumbleweed. I will shortly prepare a PHP 8.1 / tumbleweed version of the maintaina Horde containers. These will initially be broken due to some outdated language constructs. As PHP 7.4 will EOL by the end of this year, I decided not to bother with PHP 8.0 and ensure compatibility with PHP 8.1 right away, while staying compatible with PHP 7.4 until end of year. This is not fun. PHP 8.x provides several features which allow for more concise code. I will not be able to use them.
This also means that for the time being I will produce code which you may find more verbose than necessary. While Constructor promotion is mostly about being less verbose, Readonly Properties and Enums kill some of the pro-method arguments in the eternal discussion if getter methods or public properties are more appropriate interfaces. Union Types and Intersection Types allow a flexibility of method interfaces which PHP 7.4 can only emulate. You can get far by type hints for static analysis combined with boilerplate guard code inside a method and dropping type hints all along or using insufficient surrogate interfaces. But it is really not shiny. Maintaining software which shows its age has its tradeoffs.

bookmark_borderSimplifying Routing / PSR-15 bootstrap in Horde

Posted on by Ralf

As you might remember from a previous post, Horde Core’s design is more complex than necessary or desirable for two main reasons:

  • Horde predates today’s standards like the Composer Autoloader and tries to solve problems on its own. Changing that will impair Horde’s ability to run without composer which we were hesitant to do, focusing on not breaking things previously possible.
  • Horde is highly flexible, extensible and configurable, which creates some chicken-egg problems to solve on each and every call to any endpoint inside a given app.

Today’s article concentrates on the latter problem. More precisely, we want to make routing more straight forward when a route is called.

A typical standalone or monolithic app usually is a composer root package. It knows its location relative to the autoloader, relative to the dependencies and relative to the fileroot of the composer installation. Moreover, the program usually knows about all its available routes. It will also have some builtin valid assumptions about how its different parts’ routes relate to the webroot.
None of this is true with a typical composer-based horde installation.

  • None of the apps is the root package
  • Apps are exposed to a web-readable subdir of the root package
  • While the relative filesystem location is known, each app can live in a separate subdomain, in the webroot or somewhere down the tree
  • Each app may reconfigure its template path, js path, themes path
  • The composer installer plugin makes a sane default. This default can be overridden
  • Each app can be served through multiple domains / vhosts with different registry and config settings in the same installation
  • Administrators can add local override routes
  • Parts of the code base rely on horde’s own runtime-configurable autoloader rather than composer.

This creates a lot of necessary complexity. The router needs to know the possible routes before it can map a request. To have the routes, the context described above must be established. Complexity cannot be removed without reducing flexibility. There is, however, a way out. The routing problem can be divided into three phases with different problems:

  • Development time – when routes are defined and changed frequently for a given app or service
  • Installation/Configuration time – when the administrator decides which apps’ routes will be available for your specific installation
  • Runtime – when a request comes in and the router must decide which route of which app needs to react – or none at all

Let’s ignore development time for now. It is just a complication of the other two cases. The design goal is to make runtime lean and simple. Runtime should initialize what the current route needs to work and as little as possible on top of that. Runtime needs to know all the routes and a minimal setup to make the router work. Complexity needs to be offloaded into installation time. Installation time needs to create a format of definite routes that runtime can process without a lot of setup and processing. As a side effect, we can gain speed for each individual call.

Modern Autoloaders are similar in concept: They have a setup stage where all known autoloader rules of the different packages are collected. In composer, the autoloader is re-collected in each installation or update process. The autoloader is exposed through a well-known location relative to the root package, vendor/autoload.php – it can be consumed by the application without further runtime setup. The autoloader can be optimized further by processing the autoloading rules into a fixed map of classes to filenames (Level 1), making these maps authoritative without an attempt to fail over on misses (Level 2a) and finally caching hits and misses into the in-memory APCu opcache. Each optimization process makes the lookup faster. This comes at the cost of flexibility. The mapping must be re-done whenever the installation changes. Otherwise things will fail. This is OK for production but it gets in the way of development. The same is true for the router.

The best optimization relies on the application code and configuration being static. The list of routes needs to be refreshed on change. Code updates are run through composer. The composer installer plugin can automatically refresh the router. Configuration updates can happen either through the horde web ui or through adding/editing files into the configuration area. Admins already know they need to run the composer horde-reconfigure command after they added new config files or removed files. Now they also need to run it when they changed file content. In development, routing information may change on the fly multiple times per hour. Offering a less optimized, more involved version of this route collection stage can help address the problem.

A new version of the RampageBootstrap codebase in horde/core is currently in development. It will offload more of horde’s early initialisation stages into a firmware stack and will reduce the early initialisation to the bare minimum. At the moment, I am still figuring out how we can do this in a backward compatible way.

bookmark_borderMaking horde/core more versatile

Posted on by Ralf

Hello, you may have seen my blog go quiet for a while. After two years of a global pandemic, I decided to take some more time with my family and also offload some knowledge and responsibilities around the Maintaina Horde codebase to my fellow team members at B1 Systems. I also took some time considering some very fundamental Horde design implications. One of these is horde/core.

You can use the Horde Framework and its libraries in two different styles. The first style I would call loosely coupled. Most libraries try to depend on as few as possible other horde libraries and concentrate on specific purposes. That makes them reusable outside of the wider context of the framework. For example, the backend for Michael Rubinsky’s blog uses the Horde Router, the Horde Controller library and the Horde Injector DIC as a kind of micro framework without much supporting code. A custom binder simulates setup of view paths normally performed by a horde environment.

The alternative is the “horde app” use case or tightly integrated use case. Your site will have a horde base app providing common capabilities and one or multiple modules or horde apps. They integrate with a framework-generated topbar, can use an inter-app API, react to a common styling choice etc. Developers benefit from being able to reuse solutions for preference storage and UI, configuration, permissions, user groups, predefined callbacks for webdav, rpc, language and presentation handling etc. This is all great if you just want to build some addon to your Horde Groupware installation. In other cases, Horde’s base does a lot of stuff you do not really need or want. I want to change that.

The horde/core library started out as a spinoff off the horde base app. It contains a lot of glue for putting together backends, drivers, config files, caches and it also provides base classes needed by the horde applications. The horde base app contains the endpoints for RPC, route-based UI, webdav, caldav. It also contains a micro bootstrapping file called horde/lib/core.php. This file installs an error handler, sets up some basic PHP sanity stuff, does some magic around Horde’s own autoloader, defines some constants and hooks into horde/config/horde.local.php, allowing the admin to inject some early-init custom magic. Our horde-installer-plugin for composer makes use of this hook to setup the composer autoloader (and some more constants) in a backward compatible way.

I think, outside of support for old-style code bases, none of this should happen. But our current implementation of the controller framework still depends on that magic and uses code in horde/core which also depends on it. In pre-composer environments, horde needs some tricks to find out basic facts about where it is, where everything else is, how to setup autoloading etc. In modern environments, we should neither pollute the global namespace with constants and global variables nor should we have an involved, tightly coupled setup process way before we even look at the app and route called for. We need to reorder how and when things are done.

Both developers and runtime operators benefit from this redesign. Developers can reduce boilerplate when relying on much, but not all of the Horde Framework and not exactly building a module for a horde installation. They are less entangled in conventions which do not make sense anymore. Operators will notice a lower memory, i/o and computation footprint, resulting in higher speed. This is because we throw out a lot of overhead unrelated to the current call. In the old setup we even created an IMAP connection when the current user was forbidden to use the mail component and the screen shown was for changing passwords or handling preferences. Yes, even when downloading an addressbook you would have triggered and IMAP connection handshake. We do not need that and by now, we have to tools to avoid it.

Stay tuned for more details in upcoming articles.

bookmark_borderRdo: Persistence is not your model

Posted on by Ralf

Remember that post on how your backend might betray you? You can store your Turba addressbook into an LDAP tree, but if the addressbook is manipulated from LDAP side, your CardDAV Sync may be ignorant of this. The bottom line is: You cannot trust the backend. Nor should the persistence model govern your application internal model.

The development team at B1 works with a lot of inherited code from different areas and eras of FOSS and closed-source development. We see all styles of code and we have made all sorts of bad micro decisions ourselves over the years. One particulary hard question that comes up time and again is what is the “true” model of data in an application. Developers who come from traditional, monolithic web applications may say the true model is what is in the (relational) database and point out that it will closely influence the application’s internal model. Developers with a background in APIs and distributed apps will tend to think the messages going in and out of a service are the main thing. Application will be modeled after the messages and persistence is a secondary thought. Then we have the school of DDD-trained developers who will say neither is right. The internal model should be mostly ignorant of persistence and external API is just another type of persistence.

Let’s explore the benefits and impacts of these options.

DB-centric applications

In a DB-centric application, your entities closely represent rows of data in DB tables. If you use ActiveRecord or DataMapper style ORMs, you will often use the ORM Entities as your business objects in the application. This works fairly well for CRUD style applications where most if not all fields in the database will be editable form fields on screen.

If your application does little beyond storing, retrieving, updating and deleting “records” of whatever type, you will have a straight-forward development experience. Some frameworks may support autogenerating create/view/edit forms and searchable lists (html tables) out of your DB schema. Because it is the easiest way to add features to these types of application, the object model tends to be rich in attributes and limited in relations and segregation.

For example, it would be common for a “customer” entity to include separate fields for the customer’s street, city, country, email address… some fields would be marked as mandatory and others as optional, each accessible via getters and setters or public properties. This becomes cumbersome for entities which have subtypes with different sets of attributes or behaviours. Fields may be mandatory for one subtype, but optional or even forbidden for other subtypes. Attribute values are usually primitives (string, int) that can be mapped to DB column types easily. Developing these types of apps is really fast and concise as long as you do nothing fancy. As soon as you have any real amount of business logic and variance, it becomes cumbersome to maintain and hard to test.

API-centric applications

A similar school of thoughts comes out of microservice development. In many cases, CRUD services can be prototyped from autogenerated code defined through a Swagger/OpenAPI definition file. Sometimes there is little to do after this autogeneration, including persistence to the relational database. If you use a non-relational persistence like CouchDB, you may even get along with some variance and depth in the schema. Even a json or yaml file on disk gets you very far (at least in the prototype stage).

However, you will have scenarios where the same application object looks quite different between API requests. Users may have limited permissions on querying details of an object, different APIs with different use cases may have limited needs on the deeper details of objects. Retrieving your list of customer IDs and customer names for populating a dropdown is different from being able to view their billing data or contact persons’ details.

Domain Model centric approach

If your application has to deal with structured aggregates and consistency rules, exhibits rich behaviour, services multiple types of backends or is otherwise complicated, this might be the right approach.

At its core, the (micro)application is fairly ignorant of both persistence, export formats and external API.
The domain model deviates from persistence model or message representations. It has internal constraints and business rules. A domain entity is retrieved from repository in valid and complete state and all transformations will result in valid state. Transformations usually happen through defined operations rather than accessing a set of getters and setters. Setters may even not exist for many types of domain objects.

Through this, you can trust your objects by contract and reduce validations in the actual operations. This comes at a price: In a domain centric application, your classes may double or triple compared to other approaches.

You have your business object with all the internal integrity aspects. You have another version of the same object which is used for I/O – the so-called Data Transfer Object or DTO. There are versions where DTOs are either typed classes or untyped plain objects with just arbitrary public attributes. They may even just be structured arrays. Each option comes with a drawback. These world-readable objects expose all their relevant data to some kind of I/O – views rendered on the server side, files to export, REST API messages coming in or going out. A third version of your entity may be the persistence model(s), especially in relational databases. They may decompose your domain entity into multiple database tables, LDAP tree nodes or even multiple representations in no-sql databases. The main effort is getting all these transformations right. This type of application may seem verbose and repetitive. On the other hand, a strong object model with hard internal constraints and few dependencies on unrelated aspects is very straight-forward to unit test. This makes it suitable for large scale applications.

Compromises and practice

You do not have to commit to one approach with all consequences. This might even make less sense in languages which do not really enforce type hints (like python) or have no enforced concept of private and protected properties (again, like python). In many cases, you can get very far by discipline alone. Just restrict usage of setters to the persistence and I/O parts of your application. Add methods and behaviour to your persistence layer entities and even better, add an interface. Hint your business logic against the interface. You can also make your ORM mapper double as a Repository. You can scale out to real domain objects step by step as needed. Add conversion methods which turn a domain entity into one or many ORM entities and commit them to the DB. If your aggregate is not very suitable for this, hide your ORM mappers in a dedicated repository class. You can scale out as little or as much as your needs dictate.

On the other hand, you may abuse your ORM entities to populate views or data formats like XML, json or yaml.

Why we move away from putting logic into Rdo entities

Rdo is Rampage Data Objects, Horde’s minimalistic ORM solution. We have come a long way using the described tricks to make Horde_Rdo_Mappers work as repository implementations and Horde_Rdo_Base entities as business objects. However, we run into more and more situations where this is not appropriate. Using and lazy-loading relations to child objects is fine in the read use case, but persisting changes to such structures becomes tedious and error prone. Any logic tightly coupled to Rdo objects is also hard to unit test. Rdo-centric development does not mix well with the Shares library or with handling users and identities. As Rdo entities carry around references to the mapper and the database driver, they tend to bloat debug output. More fundamentally, our object models are evolving to a design which does not really look like database tables at all. We still love Rdo and may resort to Rdo entities with interfaces here and there, especially in prototyping. But our development model has long shifted towards unittest-early or unittest-driven rather.

Beware of the edges

Domain Driven Design purists may emphasize how domain models will rarely have attribute setters and some will even argue you should minimize your getters (tell don’t ask). However, at some point we need to get raw attributes to compose messages to the edges, to answer API requests or to persist any objects. There are multiple ways to do it, with different implications. Remember the Data Transfer Objects? I tend to have method on my domain aggregate to “eject” a fully detailed world readable object. In many cases, it is even typed. This chatty object can now be used for Formatters to transform them into API messages or data files. So the operation would be: Construct the domain aggregate – and fail if data is invalid – and from the domain aggregate, construct the DTO. Use the DTO for chatty jobs.

But how about the other way around? Messages from the outside can be malformed in many ways. They may be tampered with, they may miss details by design. If I exported an object to a user’s limited API and he sends an update, how to handle that? If an API user does not even know advanced attributes that are subject to other views and use cases, how would he create new entities?

I tend to have a method on the repository which accepts these messages and tries to turn them into Domain Objects. If the partial message contains some ID field or a sufficient set of data to form a unique key, I will first ask the backend to get the original details of the object and then apply the message details – and fail if this violates the constraits of my aggregate. If no previous version exists in the backend, missing information is added from defaults and generators, including a unique id of some sort. The message is applied on top.

The is approach may be expensive. I am creating full domain objects just to render database content to another format which may not even contain most of the retrieved data. On the other hand, if I know an object exists in a relational db, I could use some cheap SQL to apply partial changes without ever constructing the full model.

Creating straight-to-output repos with optimized queries is a tradeoff. They mean additional maintenance burden whenever the model changes, additional parts to cover with tests. I only do this when performance actually becomes an issue, not by default.

Creating straight-to-persistence methods for partial messages is even less desirable. It is circumventing integrity check at code level. However, sometimes the message itself is the artifact to persist – to be later processed by a queue or other asynchronous process.