Tag: horde

bookmark_borderHorde 6: Return of the Git Tree

Posted on by Ralf

Over the last few weekends, Horde 6 code has been merged back from the Maintaina fork and from separate contributions to the former Horde development version, “master”. It was time to upgrade the development tool chain.

Back in the Horde 5 days, there was a utility called git-tools developed by Michael Rubinsky. It would checkout all horde library and application repositories of the github organization into a development directory – also called the git tree – and link into into another structure that resembled a regular Horde 5 installation – called the web tree. This required to hardcode some configurations what could otherwise be auto-detected but it allowed to organize the horde code base in a developer friendly flat structure and provide a ready to run test scenario that reflected the latest code changes even before they were committed to the official repositories. In the days of composer, half of what git-tools does is already covered by composer itself. The git-tools shell already had an integration for the horde-components tool which manages releases, change log, housekeeping tasks and some more.

It was obvious that I wanted a similar functionality for the composer-based setups. It took a while to get this right. The good news is I achieved that. Bear with me for another paragraph of incremental learning or skip to the next section with the results.

It’s getting better all the time

Originally I integrated a generator for “vcs” type repositories into the composer.json generation code. But these vcs repositories are relatively slow. This is OK for one or two apps or leave packages but it’s really really slow when you have a list of 80 or 120 dependencies each pulled from a separate vcs repository.

The next iteration involved generating a satis repository as a standin to releasing code into packagist. The satis repository would be updated by any commit or accepted pull request in any of the serviced repositories. Generating the satis repository is not particularly fast but I managed to scope refreshs to the individual component that needed updating. Reading from satis however is way faster than the vcs approach. Keeping an installation up to date with the latest development commits became much more viable. I also figured I could replace an already installed dependency with an actual git repository and composer would update the autoloader as needed. This works as long as you don’t change the autoloader rules, i.e. upgrade a package from PSR-0 to PSR-4.

Back to #1

Checking out individual libraries as root components and doing integration tests in the satis setup worked OK when the focus was on isolating individual pain points and solving bugs. For any development spanning changes on multiple libraries, it did not work out too well. I ended up implementing a new command.

horde-components github-clone-org

This will checkout a developer tree of git repositories containing apps, libraries or themes.

total 752
drwxr-xr-x 188 root root 4096 Jan 12 11:04 ./
drwxr-xr-x   4 root root 4096 Dec  4 17:44 ../
drwxr-xr-x   9 root root 4096 Jan 11 07:50 ActiveSync/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Alarm/
drwxr-xr-x   8 root root 4096 Jan 11 07:50 ApertureToAnselExportPlugin/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Argv/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Auth/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Autoloader/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Autoloader_Cache/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Backup/
drwxr-xr-x   7 root root 4096 Jan 11 07:51 Browser/
drwxr-xr-x   9 root root 4096 Jan 11 07:51 Cache/
drwxr-xr-x   9 root root 4096 Jan 11 07:51 Cli/
drwxr-xr-x   8 root root 4096 Jan 11 07:51 Cli_Application/
drwxr-xr-x   8 root root 4096 Jan 11 07:51 Cli_Modular/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Compress/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Compress_Fast/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Constraint/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Controller/
drwxr-xr-x  11 root root 4096 Jan 11 07:52 Core/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Crypt/
drwxr-xr-x   7 root root 4096 Jan 11 07:52 Crypt_Blowfish/
drwxr-xr-x   6 root root 4096 Jan 11 07:52 CssMinify/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Css_Parser/
...
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Xml_Element/
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Xml_Wbxml/
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Yaml/
drwxr-xr-x  14 root root 4096 Jan 11 07:50 agora/
drwxr-xr-x  17 root root 4096 Jan 11 07:50 ansel/
drwxr-xr-x  21 root root 4096 Jan 11 07:51 base/
drwxr-xr-x  13 root root 4096 Jan 11 07:51 beatnik/
drwxr-xr-x   7 root root 4096 Jan 12 21:33 bundle/
drwxr-xr-x  12 root root 4096 Jan 11 07:51 chora/
drwxr-xr-x  15 root root 4096 Jan 13 16:54 components/
drwxr-xr-x  11 root root 4096 Jan 11 07:52 content/
drwxr-xr-x   2 root root 4096 Jan 11 08:02 dev.horde.org/
drwxr-xr-x   3 root root 4096 Jan 11 08:02 dns/
drwxr-xr-x  15 root root 4096 Jan 11 08:03 folks/
drwxr-xr-x   6 root root 4096 Jan 11 08:03 git-tools/
drwxr-xr-x  12 root root 4096 Jan 11 08:03 gollem/
drwxr-xr-x   9 root root 4096 Jan 11 08:03 groupware/
drwxr-xr-x  13 root root 4096 Jan 11 08:03 hermes/
drwxr-xr-x   6 root root 4096 Jan 11 08:08 horde-installer-plugin/
drwxr-xr-x   7 root root 4096 Jan 11 08:08 horde-support/
drwxr-xr-x   9 root root 4096 Jan 11 08:10 horde-web/
drwxr-xr-x  10 root root 4096 Jan 11 08:10 hylax/
drwxr-xr-x   7 root root 4096 Jan 11 08:12 iPhoto2Ansel/
drwxr-xr-x  14 root root 4096 Jan 11 08:11 imp/
drwxr-xr-x  14 root root 4096 Jan 11 08:12 ingo/
drwxr-xr-x   3 root root 4096 Jan 11 08:12 internal/
drwxr-xr-x  15 root root 4096 Jan 11 08:12 jonah/
drwxr-xr-x  12 root root 4096 Jan 11 08:12 klutz/
drwxr-xr-x  13 root root 4096 Jan 11 08:12 kolab_webmail/
drwxr-xr-x   9 root root 4096 Jan 11 08:12 koward/
drwxr-xr-x  18 root root 4096 Jan 11 08:13 kronolith/
drwxr-xr-x  13 root root 4096 Jan 11 08:13 luxor/
drwxr-xr-x  17 root root 4096 Jan 11 08:13 mnemo/
drwxr-xr-x  18 root root 4096 Jan 11 08:14 nag/
drwxr-xr-x   9 root root 4096 Jan 11 08:14 operator/
drwxr-xr-x  13 root root 4096 Jan 11 08:14 passwd/
drwxr-xr-x  11 root root 4096 Jan 11 08:14 pastie/
drwxr-xr-x  10 root root 4096 Jan 11 08:15 sam/
drwxr-xr-x  13 root root 4096 Jan 11 08:15 sesha/
drwxr-xr-x  11 root root 4096 Jan 11 08:15 shout/
drwxr-xr-x  13 root root 4096 Jan 11 08:15 skeleton/
drwxr-xr-x   8 root root 4096 Jan 11 08:16 timeobjects/
drwxr-xr-x  14 root root 4096 Jan 12 09:08 trean/
drwxr-xr-x  16 root root 4096 Jan 11 08:16 turba/
drwxr-xr-x  11 root root 4096 Jan 11 08:16 ulaform/
drwxr-xr-x  15 root root 4096 Jan 12 11:06 vilma/
drwxr-xr-x  10 root root 4096 Jan 11 08:17 webmail/
drwxr-xr-x  18 root root 4096 Jan 11 08:17 whups/
drwxr-xr-x  15 root root 4096 Jan 11 08:17 wicked/

This tree does not look like a composer based installation and would not easily work in a web browser. I did not want to reinvent composer with its autoloader and other benefits so I wrapped it into an installer. This installer creates a new copy of the horde/bundle base project and registers all the other libraries as a special type of composer repository “path”.

 /srv/git/horde/components/bin/horde-components -c ~/horde-testme.conf.php install
[  INFO  ] Installation directory is missing: /srv/www/testme
[   OK   ] Created installation directory: /srv/www/testme
Creating a "horde/bundle" project at "../../www/testme"
Installing horde/bundle (dev-FRAMEWORK_6_0)
  - Installing horde/bundle (dev-FRAMEWORK_6_0): Mirroring from /srv/git/horde/bundle
Created project in /srv/www/testme

When running the composer install command on this prepared setup, composer will not download the horde components from packagist or github but will use your local checkout. Only external dependencies are still downloaded from the web.

# composer install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 92 installs, 0 updates, 0 removals
  - Locking horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17)
  - Locking horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4)
  - Locking horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
  - Locking horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7)
  - Locking horde/horde-installer-plugin (v2.5.5)
  - Locking horde/hordectl (v1.0.0alpha4)
  - Locking horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
  - Locking horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2)
  - Locking horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11)
  - Locking horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9)
  - Locking horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
  - Locking horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
  - Locking horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
  - Locking horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
  - Locking horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking pear/archive_tar (1.4.14)
  - Locking pear/console_color2 (0.1.2)
  - Locking pear/console_getopt (v1.4.3)
  - Locking pear/console_table (v1.3.1)
  - Locking pear/pear (v1.10.14)
  - Locking pear/structures_graph (v1.1.1)
  - Locking pear/xml_util (v1.4.5)
  - Locking php-extended/polyfill-php80-stringable (1.2.9)
  - Locking psr/container (2.0.2)
  - Locking psr/http-client (1.0.3)
  - Locking psr/http-factory (1.0.2)
  - Locking psr/http-message (2.0)
  - Locking psr/http-server-handler (1.0.2)
  - Locking psr/http-server-middleware (1.0.2)
  - Locking psr/log (3.0.0)
  - Locking sabre/dav (4.6.0)
  - Locking sabre/event (5.1.4)
  - Locking sabre/http (5.1.10)
  - Locking sabre/uri (2.3.3)
  - Locking sabre/vobject (4.5.4)
  - Locking sabre/xml (2.2.6)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 92 installs, 0 updates, 0 removals
  - Installing horde/horde-installer-plugin (v2.5.5): Extracting archive
  - Installing horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Util
  - Installing horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/Translation
  - Installing horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Exception
  - Installing horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Compress_Fast
  - Installing horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cache
  - Installing horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Wrapper
  - Installing horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Support
  - Installing psr/log (3.0.0): Extracting archive
  - Installing php-extended/polyfill-php80-stringable (1.2.9): Extracting archive
  - Installing horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Constraint
  - Installing horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9): Symlinking from /srv/git/horde/Log
  - Installing psr/container (2.0.2): Extracting archive
  - Installing horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11): Symlinking from /srv/git/horde/Injector
  - Installing horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Controller
  - Installing horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4): Symlinking from /srv/git/horde/Crypt_Blowfish
  - Installing horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Url
  - Installing horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Css_Parser
  - Installing horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/CssMinify
  - Installing horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Flowed
  - Installing horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Secret
  - Installing horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Idna
  - Installing horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Text_Filter
  - Installing horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Filter
  - Installing horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Stream
  - Installing horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Mime
  - Installing horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mail
  - Installing horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/ListHeaders
  - Installing horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Nls
  - Installing horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Date
  - Installing horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Icalendar
  - Installing horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Browser
  - Installing horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Data
  - Installing horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/HashTable
  - Installing horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Db
  - Installing horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/History
  - Installing horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/View
  - Installing horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Vfs
  - Installing horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Tree
  - Installing horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Token
  - Installing horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Diff
  - Installing horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Serialize
  - Installing horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Xml_Element
  - Installing horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Group
  - Installing horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Perms
  - Installing sabre/uri (2.3.3): Extracting archive
  - Installing sabre/xml (2.2.6): Extracting archive
  - Installing sabre/vobject (4.5.4): Extracting archive
  - Installing sabre/event (5.1.4): Extracting archive
  - Installing sabre/http (5.1.10): Extracting archive
  - Installing sabre/dav (4.6.0): Extracting archive
  - Installing psr/http-message (2.0): Extracting archive
  - Installing psr/http-factory (1.0.2): Extracting archive
  - Installing psr/http-client (1.0.3): Extracting archive
  - Installing horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Http
  - Installing pear/pear (v1.10.14): Extracting archive
  - Installing pear/xml_util (v1.4.5): Extracting archive
  - Installing pear/structures_graph (v1.1.1): Extracting archive
  - Installing pear/console_getopt (v1.4.3): Extracting archive
  - Installing pear/archive_tar (1.4.14): Extracting archive
  - Installing horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Template
  - Installing horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Share
  - Installing horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/SessionHandler
  - Installing horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Prefs
  - Installing horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Pack
  - Installing horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Notification
  - Installing horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Compress
  - Installing horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mime_Viewer
  - Installing horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/LoginTasks
  - Installing horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Lock
  - Installing horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/JavascriptMinify
  - Installing psr/http-server-handler (1.0.2): Extracting archive
  - Installing psr/http-server-middleware (1.0.2): Extracting archive
  - Installing horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2): Symlinking from /srv/git/horde/Http_Server
  - Installing horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Cli
  - Installing horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Autoloader
  - Installing horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Auth
  - Installing horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Alarm
  - Installing horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17): Symlinking from /srv/git/horde/Core
  - Installing horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Dav
  - Installing horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Rpc
  - Installing horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Image
  - Installing horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Form
  - Installing horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Argv
  - Installing horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7): Symlinking from /srv/git/horde/base
  - Installing horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Yaml
  - Installing horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cli_Modular
  - Installing horde/hordectl (v1.0.0alpha4): Extracting archive
  - Installing horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Routes
  - Installing horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Test
  - Installing pear/console_color2 (0.1.2): Extracting archive
  - Installing pear/console_table (v1.3.1): Extracting archive
81 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files
Applying /presets for absent files in /var/config
Looking for registry snippets from apps
Writing app configs to /var/config dir
Linking app configs to /web Dir
Linking javascript tree to /web/js
Linking themes tree to /web/themes
1 package you are using is looking for funding.
Use the `composer fund` command to find out more!

This composer environment works just like a regular installation. When you install turba, kronolith or passwd through composer, it will end up linking these apps and their library dependencies from the development tree.

Composer provides an option to copy files from the repositories rather than link the files. This would allow creating archives with artifacts for distribution packaging. The horde-components tool does not yet provide a switch to generate the necessary tweak to the repository files.

bookmark_borderMaintaina’s Horde 6 goes upstream

Posted on by Ralf

It’s been a while. It’s been much too long actually. Let’s look forward though.
I recently started to onboard into FOSS development again. People have been asking for PHP 8.2 support with Horde but capacity to deliver that was very limited. But we’re getting there.

I used the “Maintaina” fork to deliver fixed and upgraded customized versions of Horde beyond what was allowed in Horde’s master branch. It’s time to move forward though. Maintaina used to target PHP 7.4 to PHP 8.1, composer2 based install from a custom satis repo and some quite invasive changes to CalDAV/CardDAV support. Most prominently, Maintaina introduced library level compatibility with several PSRs (container, http middleware, logging) and the all new horde/http_server component.

At the moment I am importing the FRAMEWORK_6_0 branches and the alpha releases based on this branch to horde’s upstream repository. So far I have imported enough libraries to make the Horde Base application barely install from packagist.

Get the current state of affairs

Your magic carpet is:

composer create-project horde/bundle .

Originally I wanted to mimic maintaina’s setup with a separate satis server as a QA stage before release to packagist. I re-considered and dropped the separate satis server. Every update of the development branches and every tagged release is consumed via packagist. I will have to fix some of my tools and workflows to reflect that. The tagged alpha releases still use maintaina’s satis server. The upcoming releases won’t.

Onboarding Procedure


I leave the FRAMEWORK_5_2 and master branches mostly untouched. FRAMEWORK_6_0 is the new default branch on github for anything I am handling. I only edit other branches if they block packagist’s import. Usually I also rebase FRAMEWORK_6_0 on any latest commits of master, but in some cases I only cherry-picked from master branch. In some cases, some entries of the changelog between 2022 late and now (october 2023) might be missing. Pull Requests, bug reports and patches welcome.

What’s next

Before I move forward with the actual applications, I want to make sure the necessary infrastructure is in place. I need to fix some aspects of the FRAMEWORK_6 version of composer. The workflow files in each repo need some review, too. Does PHPUnit and PHPStan still pass? Can we improve management of 130+ repo’s workflows?

Finally https://dev.horde.org should index from packagist, not from the satis server we originally planned to use.

What will become of maintaina repos?

The maintaina repos have had direct contributions from some trusted maintainers from the company I used to work for. They service some customers out of these repos and the related SATIS server so I won’t actively break it. However, with the move to Horde upstream, maintaina has served its purpose for me and I will not actively support the fork anymore. I suggest once everything is ported to upstream, maintaina should be archived. I will need to consult other stakeholders of this fork and the satis server.

Maintenance cost

Over the lifetime of the fork I have explored and applied various strategies for keeping the effort in check. Still, a fork of 100+ repos and the accompanying infrastructure for testing and deploying is a major burden which detracts from actually developing and maintaining code. I am glad I can save on this now and actually contribute to Horde directly in a way that doesn’t slow down activity too much. We now have the chance to speed up the cycle of feedback and releases. I hope this attracts some occassional and regular contributors.

bookmark_borderLet’s have a date. Revised horde/date ideas

Posted on by Ralf

It’s bad but not as bad as you think.

The way we write dates is very different among cultures and technologies. Even countries of the same language family might have totally different notions where to put the year, where to put the month, where to put the day when writing down a date in numbers. Apart from the order, we might use hyphen -, dot . or slash / to mark the sections. Leading zero yes or no. It gets much worse when writing out a date, like “Vlad was born on the 13th of June” or “Monday, October 10 I will have a barbecue”. Mix in different languages and different ways to abbreviate Monday as “Mo”, “Mon”, “M”.
The PHP-native ways to deal with date math and date formatting are a little quirky and best wrapped in some code that hides away the nasty details but exposes useful capabilities.

The horde/date library has been around for ages and it does its job. Due to changes in the PHP 8.1+ engine, it requires some internal re-engineering. As it is also due for a conversion to namespaces and PSR-4, I decided to think about the interface a little more. These are rough ideas and I am still figuring what will be fun and safe to work it and what is possibly over-thinking it.

It’s DateTime all right.

PHP provides internal data types to store and manipulate a date: DateTime and DateTimeImmutable. Horde_Date behaves similar to DateTimeImmutable but implements a lot of intricate math on its own. Still, it uses DateTime for some conversions. It makes sense to delegate most mutation math to the DateTimeImmutable class and make it also hold most state. No need to manually manage minutes, hours, days of weeks etc in most cases. Preferring Immutable safes us some clone magic.

Wrap it up or extend it?

There are two possible approaches to dealing with the builtin type: Either extend it or hide it in a wrapper. The third option would be to implement the interface but this is not possible.
Extending any of the two DateTime types would be handy as it is the lingua franca between different library ecosystems. We could simply inject our Horde Date type into any library that uses it. There are down sides to this approach though. While DateTimeInterface is reasonably small and mostly useful, DateTime and DateTimeImmutable expose a lot of functionality. Some of it is awkwardly named. Some of it would have to be wrapped in extending to avoid inconsistencies in our own object. Some of it possibly clashes with own naming schemes and blocks us from using preferred signatures. Some of it might not fit into our own notion of what belongs where. There is also a risk of exposing different functionality based on PHP versions. This is undesirable. Extending is not the way to go. Hiding away the DateTimeImmutable object and exposing it explicitly might seem a little verbose. It offers some interesting applications.

Clock Date – Now is the right time.

A clock date type always emits now. Asking it again at a later point will yield a later time without having to manipulate the object. This is useful for tracing duration of processes or for emitting status messages. We can also make this clock the second element of a time span. Its duration will automatically expand. We can add a Stop method to the clock which will return a regular, fixed date for further processing.

No Date – When it’s not right for you.

Sometimes we cannot rely on a date being present in the input. It might be optional. It might be required by the current data model but used to be optional or malformed. Traditional options would be expressing the non-date as null value, throwing an exception or silently assuming no date means now. These are appropriate, good solutions in many cases. Sometimes you may prefer to have the “no date” information behave a little like a date – until it reaches a point where it cannot. For example, a “not a date” can be formatted many ways. It can be serialized to the backend, provided the backend can deal with it. It can however not be calculated, mutated or cast to a DateTimeImmutable. I am not yet sure how to handle this. Maybe it should be confined to Formatters and Readers.

In The Zone.

PHP provides an own finite list of Time Zone identifiers. Wrapping PHP’s timezone objects allow dealing with well known but unsupported timezone names. We can map them to known names. We can safe the originally provided name for later usage. We might not carry a PHP timezone at all but signal the other parts of the library that some custom code must be applied.

In Good Format.

There are a plethora of ways to express a given date. There are three builtin date renderers in PHP, IntlDateFormatter (not always installed), strftrime (deprecated) and DateTimeInterface::format (English only). You might add your own. Each has its own dependencies, arguments, requirements.
It is much simpler if there is a Formatter type. Implementations can just configure it and load it with a date. Consumers have a simple interface to work with them regardless of how they are implemented.This also allows to keep the dependency footprint of the core date library low and makes adding more output formats very easy. The same is true for reading data. Reading values from various formats should not be the Date object’s concern. Another object should turn arbitrary string, integer or other data into dates – including legacy Horde_Date objects.

bookmark_borderHorde on PHP 8.1 and Composer: Update

Posted on by Ralf

Regular readers of this blog and many other are aware that PHP 7.4 will stop receiving security updates when PHP 8.2 comes out in November. This has made many horde admins question if they can continue to run Horde. Some events in life have made progress slower than originally planned. So where are we?

Confirmed running under PHP 8.1 and composer 2.4

  • horde/base in Browser
  • essential Horde Base CLI tools like horde-db-migrate and hordectl
  • horde/base portal blocks and admin area
  • horde/components developer tool
  • horde/turba Addressbook App Reading and writing contacts in the UI
  • horde/mnemo Notes App UI and webdav
  • horde/nag Tasks Apps UI, webdav, caldav
  • horde/kronolith Calendar App UI, webdav, CalDAV
  • horde/passwd Password App – Changing passwords worked with the hordeauth driver
  • horde/gollem File Manager App – very limited testing so far
  • horde/imp Webmail – very limited testing.

I run on a setup with openssl3 and a recent mariadb against dovecot and postfix. You can also consume the openSUSE 15.4 based containers built nightly. There is still considerable log spam from deprecation notices: Mostly tentative return types and signatures, also some use of deprecated functionality like strftime. Each night a few of these disappear. They don’t stop you from running horde apps.

I also have an eye on PHP 8.2 compatibility – So far, there should not be too many surprises. I also check most unit tests against the development version of PHPUnit 10.

This code is quite solid on PHP 7.4 – production users run on it.
On PHP 8.1 I consider it ready for adoption tests. Breakage is expected, feedback is welcome. Be sure to have a backup of the database and of any mail accounts you connect to it.
There is a lot to be done over the next few weeks.

If it does not run for your combination of drivers, please contact me via the horde mailing list.

Known caveats:

  • imp config SHOULD have an explicit cache setting: Set it to false to disable caching or to ‘cache’ to use Horde’s default cache. The ‘sql’ option also seems to work but I do not recommend it.
  • The RPC interface has seen very little testing. The json-rpc protocol should work. I have no desire to look into xmlrpc though unless somebody voices his needs. Beware, the xmlrpc extension has moved out of mainstream into pecl.
  • I do not have the necessary setup to comment on ActiveSync currently
  • Kolab integration is very likely broken. I don’t think anybody really uses recent horde with ancient kolab versions.
  • Most likely the SyncMl and PHPGroupware drivers are useless. If anybody really uses that bridge, please give feedback
  • I usually test against sabre/dav 4.4 – if you use anything else and see bugs, let me know
  • I don’t currently test against postgresql. MariaDB, MySQL, PerconaDB should work.
  • As PHP’s LDAP extension has moved from resources to objects, the LDAP authentication and addressbook drivers likely need an update. I do not currently test against LDAP but this is something I want to change
  • I know my former colleagues run LDAP and Redis so likely they will give some feedback in that area – Cannot comment on the timeline. I will offer a redis option for the maintaina container setup soonish.

bookmark_borderModernizing horde/text_diff

Posted on by Ralf

If you ever read a github pull request or similar extension proposal, you will likely have seen side by side comparisons of the original and the changed file. You may also have seen some text format that highlights only differences and a little context but hides the unchanged rest of the file. Both of these formats are called Diff, named after the popular diff and patch utilities dating back to ancient Unix times. The git diff command does something very similar. The horde/text_diff library and its ancestor, the pear/text_diff library, are tools to generate and format such difference information for different usage scenarios.

Apart from Horde’s internal usage in its repository viewer, horde/chora, and its wiki software, horde/wicked, the tool is also used by external parties. WebSVN maintainer Michael O. approached me because he wanted to use a PHP 8.2 ready version of horde/text_diff to substitute an older component which did not do the job. Michael has been very helpful in getting me started, pointing me to some issues to solve and also providing his own solutions in some parts. The result is a conservative update of horde/text_diff that will run in the upcoming versions of PHP without causing any trouble. But this is only where I started.

Breaking bad habits

A closer look at the internal structure of the library showed that it deserved a major overhaul. The solution was to refrain from a verbatim upgrade to namespaces and the likes but to actually change some things. This meant breaking backwards compatibility. I go to great lengths to keep a conservative drop-in version of everything I touch in the lib/ folder. Sometimes it is just an interface or wrapper, sometimes the new and old code do not really share a lot.

I began with adding type hints to most methods. Targetting PHP 8.1+ for the src/ path allows to use union types and intersection types. A lot of knowledge hidden in phpdoc comments is moved into actual code and makes it more robust.

Exploring the code for base classes and interfaces, I noticed that some things I did not like.

Method signatures did not add up

Some method signatures did not add up. Depending on the type of Diff Engine, the diff() method would take different types of arguments. The interface was mixing the specifics of how the diff engine is set up with the command to create a list of operations objects. Loading the engine is now separated from running it. The running method is now always called in the same way.

Internal dependency creation

The Diff, Threeway Diff and Mapped Diff utilities all created their diff engine internally. To do this, they needed a very flexible constructor that allowed passing whatever is needed to set up the actual engine. That was bad enough but they also did it in different ways. The Differs’ constructor now only accepted a pre-constructed engine. For convenience, I created a factory which would take over the responsibilities originally assigned to the differs’ constructor: Building a differ from input and if no specific differ was chosen, selecting one by some priority logic. In the end it turned out that the Differ does not need the engine at all but rather needs the product of the Engine: A set of operations to transform document A to document B. Born is the OperationList class. I did not want to just pass an array. I added a small static method as a named constructor. It frees the actual constructor of too many responsibilities and allows to keep the interface clean and strict.

More explicit type juggling

Creation if diffs contains some interesting math. The algorithms use a lot of short variable names and operations that make sense if you know the underlying theory but otherwise look like garbage. I added some explicit conversions between string and integer and made some changes to ensure a number zero or an empty string is not mistaken for a “false” or “null” value which would have another meaning.
Overall it is now much easier for static analyzers to spot any issues.

Dual stacks have a price tag

Essentially maintaining two different sets of the library comes with some cost. One must ensure that unit tests targeted at the newer platform are not run when testing for compatibility with the older platform. The conservative lib/ is ready to run PHP 7.4 code but the modern version in src/ must be transpiled to be run in PHP 7.4 – I do that on release but still it is another aspect that needs minding. As I also use automated tools for some upgrade tasks, I need to ensure I do not upgrade the lib/ path. The price is worth it as I cannot convert the code base at once and I want to provide a good development experience to all who are caught in between maintaining an older release or creating new code. I am in that spot myself. Essentially it allows me to run two conflicting major versions of some critical libraries and pick the right one for different sub systems. The need will go away as code is gradually migrating towards the newer implementations. At some point a next major version will drop the conservative path. Anybody interested is free to maintain the older major version and keep using it.

Upcoming work

I consider the external interface of the newer horde/text_diff implementation fairly stable by now. Internally, however, there is a lot of room for improvement. Some functionality should move out of the base classes and into separate traits – which the base classes will use. Some getters should be added and used, preparing to move some public variables to internal state in a next major release. The new OperationList gets unloaded to plain arrays in several places – It needs to learn some tricks without degrading into a glorified array. None of this should stop early adopters from using the new code base. None of this is supposed to break any user code.

Out of scope for now

There are some items which I decided to postpone for now. One thing which bothers me is the amount of dependencies. While a dependency on horde/exception makes sense, it pulls in horde/translation for no good reason. Horde\Util is pulled in but really only used in two places: A horde/string call which could be reduced to a direct call to an internal function and one call to a helper for handling temporary files. That helper should maybe live in its own library, nicely decoupled from unrelated utilities. There was a reason why they were packed together but it is no longer relevant.

Also, some functionality is missing in the Xdiff-based engine. Most distributions do not even offer php-xdiff, including my own development platform. I will add that feature once I get it into CI and into the development setup. I do not want to delay other items to do that right now. Patches welcome 🙂

bookmark_borderTools to build better Tools faster

Posted on by Ralf

Behind every lofty architecture mantra there is mundane execution. This is best left to tools and I don’t mean anybody in particular but programs that help us make better programs. It basically goes like this: Build tool. Use tool. Build better tool. Build tool to build better tool. Build better tool to build better tool faster. And so on. Implementing this in practice can be quite boring but the alternative is to do boring things again and again and again and that’s enough already. So let’s see.

Maintaining 100+ libraries and programs involves doing a few things over and over again. Automating these seems natural but requires some thought. Developers want to spend their time in interesting and useful ways. Querying and manipulating git repositories is repetitive. Updating a changelog file with a select subset of messages also present in the git commits is repetitive. Rewriting project metadata and updating CI jobs for new PHPUnit and PHP versions or base operating systems is repetitive and requires no brains at all, why should I do this 100+ times?

Off the shelf tools

Using tools that already exist and are maintained by other parties is a no-brainer. Which tools can help?

  • PHPUnit helps us spot and eliminate regressions before any user is affected. The tool itself is maintained by Sebastian Bergmann but writing and upgrading the actual test code is a chore.
  • PHPStan or Psalm – I prefer PHPStan – are static analyzers which help developers spot places where signatures, types and assumptions don’t add up. To get the best out of it, either phpdoc annotations or parameter and return types must be added. No tests to write, which is good – but PHPStan is organized in progressively strict levels and each library needs to be checked against the level it is supposed to pass. Micromanaging that is boring as hell, tools are needed.
  • php-cs-fixer is developed by friendsofphp – it is a basic code manipulation tool which helps anywhere from adhering to PSR-12 or PER-1 to automatically upgrading from array() to [] notation. Configuring this beast is easy but ensuring the most current rules are used in every project is another burden.
  • rector is another tool that transpiles code either up or down to select standards. It will move implicit knowledge or phpdoc data into actual code or do the different thing. It will choose older ways to express something over new ones or vice versa. Configuring it to do only what is helpful is quite a challenge. Also ensuring the most recent config is used is just boring and cumbersome. Tool needed.

Homegrown tools

The horde project has some home grown tools that can help but need development themselves.

  • horde/git-tools by Michael Rubinsky used to be the way to assemble a bleeding edge developer copy from zillions of github repos. In a modern composer based installation this tool is less useful but it contains a lot of interesting capabilities that should be factored out
  • horde/components can generate composer and pear metadata from a self-defined yaml format. It can create tar archives from repositories, implements a basic workflow engine for release and quality check tasks and does some other things. Its internal architecture is rooted in history and while some of its functionality seems out of touch with 2022, many other parts deserve expanding or factoring out into modern self-contained libraries for reuse.
  • horde/hordectl is a command line tool to interact with a Horde installation. Inject users and passwords, configure permissions, groups or app-specific resources from yaml files and defaults. It needs some upgrading, it could do so much more to facilitate proof of concept, showcase or CI installations.
  • horde/horde-installer-plugin is a plugin for composer that helps bootstrap a horde installation and its web-readable part. Much of its code would best be moved out to separate libraries.

Building blocks

Existing and new libraries should inherit functionality moved out from existing tools or newly created

  • horde/vcs is a version control library. Its main origin are the horde/chora application and the installation/development tools. Recently I began to move or re-implement code from git-tools and horde/components into this library. I am less interested in the rcs, cvs and svn parts. The original library followed an approach abstracting the differences between git, cvs, svn & friends. This limits its usefulness. I see how it facilitates creating an application that consumes and shows code from these. Still, there should be a lower level of abstraction that provides the unique capabilities of git in a programmatic fashion. This is one thing I currently work on
  • horde/rampage used to be a dead end but I am reusing the library for deployment and introspection related code factored out from other tools.
  • horde/filesystem is a new library, focused on object-oriented filesystem traversal and manipulation. Still very immature but I hope to turn it into a standalone and reusable tool.
  • horde/registry is the stub of an upcoming redesign of the core bootstrapping process. No more globals, reliance on PSR-11 DI containers and PSR-4 autoloading – this registry will do less than its ancestors yet be much more powerful and easy to use. This is still much work.
  • horde/cli_modular is a tool to write extensible, pluggable commandline interfaces. It is used by horde/git-tools, horde/components, horde/hordectl and a few others. In the current upgrade cycle some redesign is necessary to make it viable for modern environments and free it from problems already solved by autoloaders or DI containers.

So much work to do but devoting some time to better tools is better than doing mindless conversions of existing code over and over.

bookmark_borderAuth Headaches

Posted on by Ralf

Back in the old days when rock musicians took the same drugs as your grandfather, authorisation and authentication might have been very simple. You had a user name, you had a password. Most likely you had one and the same password for each and everything. Congrats if you were smarter back then. Maybe your application had a notion of a super user flag or user levels. The more advanced had a permission system but who would ever need that? Well, there will only ever be the need for 5 computers, some researchers argued back in the old days, referring to an even older, albeit questioned, quote.

Today, authentication and authorization are much more complicated. People might still log into a system by user name and password. They might need a second factor like a One Time Password right away or later to perform advanced operations like committing orders. There might be elaborate permission or role-based systems restricting what the user can do on which resources. Users might not have a local password at all but a shadow identity linked to an authentication provider like Google or Github – who are the party assuring to the app that you are in fact the person you claim to be. In an enterprise context, devices might identify their users by SSL Certificates or bearer tokens. Finally the app might have long-lived remember-me cookies separate from their short-lived session tokens/session cookies. These might be bound to specific clients. Changing browsers may put you into a situation where login by user name and password will result in a more elaborate, email-based password.

And on a completely different level you might want to to authorize REST API access to entities either linked to a specific user account or to a specific outside service. Things got elaborate, things got complicated.

Basic Definition: Authentication

Authentication is the process of identifying who is dealing with the application.
This generally involves two orthogonal questions:

How is the authentication communicated?

This is usually achieved by presenting some evidence of authentication to the resource. Showing your passport, you might say. In a stateless API, the evidence is presented with each request. In classic HTTP Basic Authentication, the user first accesses a resource and the resource answers, “Authentication required”. Then the user agent (browser) presents the user a form to enter user name and password. The request against the resource is sent again, along with an authentication header containing the user and password in a transport-friendly envelope (BASE64 encoding) which provides no security by itself. The server will check this information and if it matches, grant access to the resource. The latter is actually authorization – see below. As far as authentication goes, presenting evidence and accepting it is the whole thing. More advanced systems may send a derivative of the actual authentication information. Digest Authentication sends a value computed from user name and password which the server knows or can check against something it knows. The server or any intruder can not deduce from that value what the actual password was. Another derivative mechanism is cookie or bearer token authentication. A new authentication credential is created, for example by sending user name and password to the server (or to a third party) only once. The credential is now sent along with each request to verify it’s you. You might need your passport or driver’s license to acquire a key to your hotel room but once you have it, the key is all you need to get in.

How authentication information is checked

The other major aspect is how the server side keeps the necessary information to verify authentication data. More simply put: How does the server know if your user name and password are legit? User name and password might be stored in a file. The password might better not be stored in the file but a derivative value like a computed hash. This way, if somebody steals the file, he will only know the users but cannot know the passwords. The password (or its hash) might be stored in a database or in an LDAP server. The credentials might be sent to an authentication API. In some cases, the server does not have to store any authentication data. This is true when the authentication data contains means to verify that it has been created by a trusted third party, is time-limited and has not been tampered with. Finally, the server might not care at all. A traditional chat service may receive your user name and create a session key. This key is used to understand who sent or asked for what. As long as you are logged in and keep communicating, no new session for this user can be created. Once you are out for long enough, the session expires and anybody can use the same name again. Having to deal with passwords may be an unwanted complexity. Authentication is identifying you by any (sufficient) means.

You know it’s drivin’ me wild – Confusion

Traditional systems have mixed emotions about their guardian angel. As said above, they may mix up knowing who asks (authentication) with knowing if they deserve to receive (authorization). They may also have a notion of an “authentication driver” which might emphasize one aspect over the other, assuming that it is either well-established or irrelevant how the password arrives at the server. New systems should have a clear understanding of both aspects and may link multiple combinations of both receiving and checking credentials to the same identity or user account.

Basic Definition: Authorization

Authorization is the process of deciding if a requester (who could be authenticated or anonymous) is authorized to interact with a resource or system. A concert hall or a renaissance faire may check your authorization to enter by a ticket, a stamp on the hand or ribbon, wrist band. They may not give an elaborate thought about who you are. Why should they care? At its core, a username/password authentication system is just checking by a password if you are authorized to identify as a certain user. It may be beneficial to tie a password to an identity. This identity may have permissions and other attached data which it will keep even when it changes its password. Other systems assign authorization to the token itself which is both the password and the identity. In this case, when the token expires, the identity will expire, too.

DAC: Discretionary Access Control and Permissions

Any system that discerns access by the identity of a user can be considered a DAC system. Permissions may be assigned to a user identity directly or to a named group. The users’ being part of the group can be verified through his identity, hence his access level. This can include special pseudy-groups like “all authenticated users” and “guests” or non-authenticated users. Most systems need to expose at least the means of authentication to a yet unauthenticated client. The horde/perms and horde/share libraries implement such a DAC system. Most DAC systems are cumulative: By being member of more groups, a user can only gain more privileges but not lose them. In practice, it might be easier to define a privilage and allow access if a user does NOT have it (and maybe have some other) rather than trying to work out how to allow negative privileges within the actual system. In a wider sense, countable limits like allowing to upload ten pictures or reading 5 articles per day can also be expressed in a DAC system.

MAC: Mandatory Access Control

Mandatory access control is an evolution from DAC in which acess is defined by policies. These allow or prevent a user from sharing a resource with a defined audience. There is little provision for individual exceptions.

RBAC: Role-Based Access Control

Role-Based Access Control systems combine previous concepts. Multiple permissions on resources are assigned to a role and subjects or identities are authorized for these roles. Who grants this authorization is not defined by the system – usually it is is the person with the role of “approver” on the specific “role” resource. A system may define that a user role is needed to even apply for further roles or application is not possible at all and roles are centrally assigned. Extended RBAC systems can model composed roles out of other roles. They may also define policies for mutually exclusive roles – A person may not apply for a role for which he is the approver or a person may not approve his own application for a role, even if he is allowed to apply for the role and has the authority to approve. A ticket system may ask a user if he is in the requester role or in the processor role and may access to different queues and commands based on that decision. In sports, you might be a player in one game or league and a referee in another but you are not allowed to combine both roles’ permissions during a game. This prevents undesirable situations. A system may ask the top administrator to choose if he is currently acting as the administrator or as a regular user and prevent him from mixing both types of access at once.

Two Factor Authentication and Weak Authentication

Many modern systems combine a primary authentication mechanism like username/password with additional aspects. A user may need to solve a captcha to gain the authority to enter his password.
A user may temporarily lose access to the login mechanism if the same IP address has tried to authenticate too many times within a time span. A user may be authenticated by a certificate or long living device cookie but needs to add password authentication or email verification before he has access to some functions, even if his user rank, role, permission, group membership or whatever is otherwise sufficient. One API call may be used in a UI scenario through a short lived session token and in an integration scenario using a separately scoped access token but not through a user / password combination. Finally there are One Time Password mechanisms which are only practical if they are limited either to specific requests like transferring money or are required periodically – like once every 24 hours. Keeping mechanisms nicely separated and combining requirements on a more abstract level is crucial. Trying to make a single mechanism powerful and flexible enough can end up making it overly complex and impractical to use. If you think of PSR-7 middlewares handling a HTTP request in PHP, a middleware’s job may be limited to fetching a credential from a header and calling into a backend or multiplexer. The result is stored back into the request as an extra attribute, leaving it to another middleware further down the line to process the result and implement consequences like an error message, a redirect to a login screen or determining which set of roles or permissions is whitelisted for this login type. By enabling or disabling middlewares for a specific request, complexity increases or decreases.

Challenging backend services

There is an obvious issue with scenarios in which multiple types of credentials may identify and authorize a user to access the system: In each case the system must be able to access its backend resources. This can be trivial for a global resource like a database accessed through a system wide application credential. It can be more tricky if you try to access a user-specific IMAP backend or an LDAP directory which has its own, completely separate notion of access control. There are several ways to tackle this but I will leave this to another article.

bookmark_borderHorde/Yaml: Graceful degradation

Posted on by Ralf

Tonight’s work was polishing maintaina’s version of horde/yaml. The test suite and the CI jobs now run successfully both on PHP 7.4 and PHP 8.1. The usual bits of upgrading, you might say.

However, I had my eye on horde/yaml for a reason. I wanted to use it as part of my improvements to the horde composer plugin. Composer famously has been rejecting reading any yaml files for roughly a decade so I need to roll my own yaml reader if I want to deal with horde’s changelog, project definition and a few other files. I wanted to keep the footprint small though, not install half a framework along with the installer utility.

You never walk alone – There is no singular in horde

The library used to pull in quite a zoo of horde friends and I wondered why exactly. The answer was quite surprising. There is no singular in horde. Almost none of the packages can be installed without at least one dependency. In detail, horde/yaml pulled in horde/util because it used exactly one static method in exactly one place. It turned out while that method is powerful and has many use cases, it was used in a way that resulted in a very simple call to a PHP builtin function. I decided whenever the library is not around I will directly call that function and lose whatever benefits the other library might grant over this. This pattern is called graceful degradation. If a feature is missing, deliver the next best available alternative rather than just give up and fail. The util library kept installing although the yaml parser no longer needed it. The parser still depended on the horde/exception package which in turn depended on horde/translation and a few other helpers. Finally horde/test also depended on horde/util. It was time to allow a way out. While all of these are installed in any horde centric use case, anybody who wants only a neat little yaml parser would be very unhappy about that dependency crowd.

Alternative exceptions

The library already used native PHP exceptions in many places but wrapped Horde exceptions for some more intricate cases. While this is all desirable, we can also do without it. If the horde/exception package is available, it will be used. Otherwise one of the builtin exceptions is raised instead. This required to update the test suite to make it run correctly either way. But what is the point if the test suite will install horde/util anyway?

Running tests without horde/test unless it is available

I noticed none of the tests really depended on horde/test functionality. Only some glue code for utilities like the horde/test runner or horde/components really did anything useful. I decided to change the bootstrap code so that it would not outright fail if horde/test was not around. Now the library can be tested by an external phpunit installation, phar or whatever. It does not even need a “composer install” run, only a “composer dump-autoload --dev” to build the autoloader file.

A standalone yaml parser

The final result is a horde/yaml that still provides all integrations when run together with its peer libraries but can be used as a standalone yaml parser if that is desirable. I hope this helps make the package more popular outside the horde context.

Lessons learned

Sometimes less is more. Factoring out aspects for reuse is good. Factoring out aspects into all-powerful utility libraries like “util”, “support” and the likes can glue an otherwise self contained piece of software together with too many other things. That makes them less attractive and harder to work with. Gracefully running nevertheless is one part. The other is redesigning said packages which cover too many aspects at once. This is a topic for another article in another night though.

bookmark_borderHorde Installer: Recent Changes

Posted on by Ralf

The maintaina-com/horde-installer-plugin has seen a few changes lately. This piece is run on every composer install or update in a horde installation. A bug in it can easily break everything from CI pipelines to new horde installations and it is quite time consuming to debug. I usually try to limit changes.

Two codebases merged

In the 2.3.0 release of November 2021 I added a new custom command horde-reconfigure which does all the background magic of looking up or creating config snippets and linking them to the appropriate places, linking javascript from addon packages to web-readable locations and so on. This is essentially the same as the installer plugin does but on demand. A user can run this when he has added new config files to an existing installation. Unfortunately the runtime environment of the installer plugin and the custom command are very different in terms of available IO, known paths and details about the package. I took the opportunity to clean up code, refactor and rethink some parts to do the same things but in a more comprehensible way. As I was aware of the risks I decided to leave the original installer untouched. I got some feedback and used it myself. It seemed to work well enough.

For the 2.4.0 release I decided to finally rebase the installer onto the command codebase and get rid of the older code. It turned out that the reconfigure command was lacking some details which are important in the install use case. Nobody ever complained because these settings are usually not changed/deleted outside install/update phase. As of v2.4.4 the installer is feature complete again.

New behaviour in v2.4

The installer has been moved from the install/update phase to the autoload-dump phase. It will now process the installation as a whole rather than one library at a time. This simplifies things a lot.reviously, the installer ran for each installed package and potentially did a few procedures multiple times. Both the installer and the horde-reconfigure command will now issue some output to the console about their operation and they will process the installation only once with the updated autoloader already configured. The changes will now also apply on removal of packages or on other operations which require a rewrite of the autoloader. The registry snippets now include comments explaining that they are autogenerated and how to override the autoconfigured values.

Outlook to 2.5 or 3.0

The composer API has improved over the last year. We need to be reasonably conservative to support OS distribution packaged older versions of composer. At some point in the future however I want to have a look at using composer for simplifying life

  • Improve Theme handling: Listing themes and their scope (global and app specific), setting default theme of an installation
  • Turning a regular installation into a development setup for specific libraries or apps
  • Properly registering local packages into composer’s package registry and autoloader (useful for distribution package handling).

Both composer’s native APIs and the installer plugin can support improving a horde admin’s or developer’s life:

  • Make horde’s own “test” utility leverage composer to show which optional packages are needed for which drivers or configurations
  • Expose some obvious installation health issues on the CLI.
  • Only expose options in the config UI which are supported by current PHP extensions and installed libraries
  • Expose a check if a database schema upgrade is needed after a composer operation, both human readable and machine consumable. This should not autorun.

The actual feature code may be implemented in separate libraries and out of scope for the installer itself. As a rule, horde is supposed to be executable without composer but this is moving out of focus more and more.

bookmark_borderMaintaina/Horde UTF-8 on PHP 8

Posted on by Ralf

On recent OS distributions, two conflicting changes can bring trouble.

MariaDB refuses connections with ‘utf-8’ encoding

Recent MariaDB does not like the $conf[‘sql’][‘charset’] default value of ‘utf-8’. It runs fine if you change to the more precise ‘utf8mb4’ encoding. This is what recent MySQL understands to be ‘utf-8’. You could also use ‘utf8mb3’ but this won’t serve modern users very well. The ‘utf8m3’ value is what older MariaDB and MySQL internally used when the user told it to use ‘utf-8’. But this character set supports only a subset of unicode, missing much-used icons like ☑☐✔✈🛳🚗⚡⅀ which might be used anywhere from calendar events sent out by travel agencies to todos or notes users try to save from copy/pasted other documents.

I have changed the sample deployment code to use utf8mb4 as the predefined config value.

Shares SQL driver does not understand DB-native charsets

The Shares SQL driver does some sanitation and conversion when reading from DB or writing to DB. The conversion code does not understand DB native encodings like “utf8mb4”. I have applied a patch to the share library that would detect and fix this case but I am not satisfied with this solution. First, this issue is bound to pop up in more places and I wouldn’t like to have this code in multiple places. Either the DB abstraction library horde/db or the string conversion library in horde/util should provide a go-to solution for mapping/sanitizing charset names. Any library using the config value should know that it needs to be sanitized but should not be burdened with the details. I need to follow up on this.

Update

See https://github.com/horde/Util/commit/7019dcc71c2e56aa3a4cd66f5c81b5273b13cead for a possible generalized solution.