Tag: Development

bookmark_borderHorde 6: Return of the Git Tree

Posted on by Ralf

Over the last few weekends, Horde 6 code has been merged back from the Maintaina fork and from separate contributions to the former Horde development version, “master”. It was time to upgrade the development tool chain.

Back in the Horde 5 days, there was a utility called git-tools developed by Michael Rubinsky. It would checkout all horde library and application repositories of the github organization into a development directory – also called the git tree – and link into into another structure that resembled a regular Horde 5 installation – called the web tree. This required to hardcode some configurations what could otherwise be auto-detected but it allowed to organize the horde code base in a developer friendly flat structure and provide a ready to run test scenario that reflected the latest code changes even before they were committed to the official repositories. In the days of composer, half of what git-tools does is already covered by composer itself. The git-tools shell already had an integration for the horde-components tool which manages releases, change log, housekeeping tasks and some more.

It was obvious that I wanted a similar functionality for the composer-based setups. It took a while to get this right. The good news is I achieved that. Bear with me for another paragraph of incremental learning or skip to the next section with the results.

It’s getting better all the time

Originally I integrated a generator for “vcs” type repositories into the composer.json generation code. But these vcs repositories are relatively slow. This is OK for one or two apps or leave packages but it’s really really slow when you have a list of 80 or 120 dependencies each pulled from a separate vcs repository.

The next iteration involved generating a satis repository as a standin to releasing code into packagist. The satis repository would be updated by any commit or accepted pull request in any of the serviced repositories. Generating the satis repository is not particularly fast but I managed to scope refreshs to the individual component that needed updating. Reading from satis however is way faster than the vcs approach. Keeping an installation up to date with the latest development commits became much more viable. I also figured I could replace an already installed dependency with an actual git repository and composer would update the autoloader as needed. This works as long as you don’t change the autoloader rules, i.e. upgrade a package from PSR-0 to PSR-4.

Back to #1

Checking out individual libraries as root components and doing integration tests in the satis setup worked OK when the focus was on isolating individual pain points and solving bugs. For any development spanning changes on multiple libraries, it did not work out too well. I ended up implementing a new command.

horde-components github-clone-org

This will checkout a developer tree of git repositories containing apps, libraries or themes.

total 752
drwxr-xr-x 188 root root 4096 Jan 12 11:04 ./
drwxr-xr-x   4 root root 4096 Dec  4 17:44 ../
drwxr-xr-x   9 root root 4096 Jan 11 07:50 ActiveSync/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Alarm/
drwxr-xr-x   8 root root 4096 Jan 11 07:50 ApertureToAnselExportPlugin/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Argv/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Auth/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Autoloader/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Autoloader_Cache/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Backup/
drwxr-xr-x   7 root root 4096 Jan 11 07:51 Browser/
drwxr-xr-x   9 root root 4096 Jan 11 07:51 Cache/
drwxr-xr-x   9 root root 4096 Jan 11 07:51 Cli/
drwxr-xr-x   8 root root 4096 Jan 11 07:51 Cli_Application/
drwxr-xr-x   8 root root 4096 Jan 11 07:51 Cli_Modular/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Compress/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Compress_Fast/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Constraint/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Controller/
drwxr-xr-x  11 root root 4096 Jan 11 07:52 Core/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Crypt/
drwxr-xr-x   7 root root 4096 Jan 11 07:52 Crypt_Blowfish/
drwxr-xr-x   6 root root 4096 Jan 11 07:52 CssMinify/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Css_Parser/
...
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Xml_Element/
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Xml_Wbxml/
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Yaml/
drwxr-xr-x  14 root root 4096 Jan 11 07:50 agora/
drwxr-xr-x  17 root root 4096 Jan 11 07:50 ansel/
drwxr-xr-x  21 root root 4096 Jan 11 07:51 base/
drwxr-xr-x  13 root root 4096 Jan 11 07:51 beatnik/
drwxr-xr-x   7 root root 4096 Jan 12 21:33 bundle/
drwxr-xr-x  12 root root 4096 Jan 11 07:51 chora/
drwxr-xr-x  15 root root 4096 Jan 13 16:54 components/
drwxr-xr-x  11 root root 4096 Jan 11 07:52 content/
drwxr-xr-x   2 root root 4096 Jan 11 08:02 dev.horde.org/
drwxr-xr-x   3 root root 4096 Jan 11 08:02 dns/
drwxr-xr-x  15 root root 4096 Jan 11 08:03 folks/
drwxr-xr-x   6 root root 4096 Jan 11 08:03 git-tools/
drwxr-xr-x  12 root root 4096 Jan 11 08:03 gollem/
drwxr-xr-x   9 root root 4096 Jan 11 08:03 groupware/
drwxr-xr-x  13 root root 4096 Jan 11 08:03 hermes/
drwxr-xr-x   6 root root 4096 Jan 11 08:08 horde-installer-plugin/
drwxr-xr-x   7 root root 4096 Jan 11 08:08 horde-support/
drwxr-xr-x   9 root root 4096 Jan 11 08:10 horde-web/
drwxr-xr-x  10 root root 4096 Jan 11 08:10 hylax/
drwxr-xr-x   7 root root 4096 Jan 11 08:12 iPhoto2Ansel/
drwxr-xr-x  14 root root 4096 Jan 11 08:11 imp/
drwxr-xr-x  14 root root 4096 Jan 11 08:12 ingo/
drwxr-xr-x   3 root root 4096 Jan 11 08:12 internal/
drwxr-xr-x  15 root root 4096 Jan 11 08:12 jonah/
drwxr-xr-x  12 root root 4096 Jan 11 08:12 klutz/
drwxr-xr-x  13 root root 4096 Jan 11 08:12 kolab_webmail/
drwxr-xr-x   9 root root 4096 Jan 11 08:12 koward/
drwxr-xr-x  18 root root 4096 Jan 11 08:13 kronolith/
drwxr-xr-x  13 root root 4096 Jan 11 08:13 luxor/
drwxr-xr-x  17 root root 4096 Jan 11 08:13 mnemo/
drwxr-xr-x  18 root root 4096 Jan 11 08:14 nag/
drwxr-xr-x   9 root root 4096 Jan 11 08:14 operator/
drwxr-xr-x  13 root root 4096 Jan 11 08:14 passwd/
drwxr-xr-x  11 root root 4096 Jan 11 08:14 pastie/
drwxr-xr-x  10 root root 4096 Jan 11 08:15 sam/
drwxr-xr-x  13 root root 4096 Jan 11 08:15 sesha/
drwxr-xr-x  11 root root 4096 Jan 11 08:15 shout/
drwxr-xr-x  13 root root 4096 Jan 11 08:15 skeleton/
drwxr-xr-x   8 root root 4096 Jan 11 08:16 timeobjects/
drwxr-xr-x  14 root root 4096 Jan 12 09:08 trean/
drwxr-xr-x  16 root root 4096 Jan 11 08:16 turba/
drwxr-xr-x  11 root root 4096 Jan 11 08:16 ulaform/
drwxr-xr-x  15 root root 4096 Jan 12 11:06 vilma/
drwxr-xr-x  10 root root 4096 Jan 11 08:17 webmail/
drwxr-xr-x  18 root root 4096 Jan 11 08:17 whups/
drwxr-xr-x  15 root root 4096 Jan 11 08:17 wicked/

This tree does not look like a composer based installation and would not easily work in a web browser. I did not want to reinvent composer with its autoloader and other benefits so I wrapped it into an installer. This installer creates a new copy of the horde/bundle base project and registers all the other libraries as a special type of composer repository “path”.

 /srv/git/horde/components/bin/horde-components -c ~/horde-testme.conf.php install
[  INFO  ] Installation directory is missing: /srv/www/testme
[   OK   ] Created installation directory: /srv/www/testme
Creating a "horde/bundle" project at "../../www/testme"
Installing horde/bundle (dev-FRAMEWORK_6_0)
  - Installing horde/bundle (dev-FRAMEWORK_6_0): Mirroring from /srv/git/horde/bundle
Created project in /srv/www/testme

When running the composer install command on this prepared setup, composer will not download the horde components from packagist or github but will use your local checkout. Only external dependencies are still downloaded from the web.

# composer install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 92 installs, 0 updates, 0 removals
  - Locking horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17)
  - Locking horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4)
  - Locking horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
  - Locking horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7)
  - Locking horde/horde-installer-plugin (v2.5.5)
  - Locking horde/hordectl (v1.0.0alpha4)
  - Locking horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
  - Locking horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2)
  - Locking horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11)
  - Locking horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9)
  - Locking horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
  - Locking horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
  - Locking horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
  - Locking horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
  - Locking horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking pear/archive_tar (1.4.14)
  - Locking pear/console_color2 (0.1.2)
  - Locking pear/console_getopt (v1.4.3)
  - Locking pear/console_table (v1.3.1)
  - Locking pear/pear (v1.10.14)
  - Locking pear/structures_graph (v1.1.1)
  - Locking pear/xml_util (v1.4.5)
  - Locking php-extended/polyfill-php80-stringable (1.2.9)
  - Locking psr/container (2.0.2)
  - Locking psr/http-client (1.0.3)
  - Locking psr/http-factory (1.0.2)
  - Locking psr/http-message (2.0)
  - Locking psr/http-server-handler (1.0.2)
  - Locking psr/http-server-middleware (1.0.2)
  - Locking psr/log (3.0.0)
  - Locking sabre/dav (4.6.0)
  - Locking sabre/event (5.1.4)
  - Locking sabre/http (5.1.10)
  - Locking sabre/uri (2.3.3)
  - Locking sabre/vobject (4.5.4)
  - Locking sabre/xml (2.2.6)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 92 installs, 0 updates, 0 removals
  - Installing horde/horde-installer-plugin (v2.5.5): Extracting archive
  - Installing horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Util
  - Installing horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/Translation
  - Installing horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Exception
  - Installing horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Compress_Fast
  - Installing horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cache
  - Installing horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Wrapper
  - Installing horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Support
  - Installing psr/log (3.0.0): Extracting archive
  - Installing php-extended/polyfill-php80-stringable (1.2.9): Extracting archive
  - Installing horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Constraint
  - Installing horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9): Symlinking from /srv/git/horde/Log
  - Installing psr/container (2.0.2): Extracting archive
  - Installing horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11): Symlinking from /srv/git/horde/Injector
  - Installing horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Controller
  - Installing horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4): Symlinking from /srv/git/horde/Crypt_Blowfish
  - Installing horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Url
  - Installing horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Css_Parser
  - Installing horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/CssMinify
  - Installing horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Flowed
  - Installing horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Secret
  - Installing horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Idna
  - Installing horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Text_Filter
  - Installing horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Filter
  - Installing horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Stream
  - Installing horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Mime
  - Installing horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mail
  - Installing horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/ListHeaders
  - Installing horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Nls
  - Installing horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Date
  - Installing horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Icalendar
  - Installing horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Browser
  - Installing horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Data
  - Installing horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/HashTable
  - Installing horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Db
  - Installing horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/History
  - Installing horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/View
  - Installing horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Vfs
  - Installing horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Tree
  - Installing horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Token
  - Installing horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Diff
  - Installing horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Serialize
  - Installing horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Xml_Element
  - Installing horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Group
  - Installing horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Perms
  - Installing sabre/uri (2.3.3): Extracting archive
  - Installing sabre/xml (2.2.6): Extracting archive
  - Installing sabre/vobject (4.5.4): Extracting archive
  - Installing sabre/event (5.1.4): Extracting archive
  - Installing sabre/http (5.1.10): Extracting archive
  - Installing sabre/dav (4.6.0): Extracting archive
  - Installing psr/http-message (2.0): Extracting archive
  - Installing psr/http-factory (1.0.2): Extracting archive
  - Installing psr/http-client (1.0.3): Extracting archive
  - Installing horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Http
  - Installing pear/pear (v1.10.14): Extracting archive
  - Installing pear/xml_util (v1.4.5): Extracting archive
  - Installing pear/structures_graph (v1.1.1): Extracting archive
  - Installing pear/console_getopt (v1.4.3): Extracting archive
  - Installing pear/archive_tar (1.4.14): Extracting archive
  - Installing horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Template
  - Installing horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Share
  - Installing horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/SessionHandler
  - Installing horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Prefs
  - Installing horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Pack
  - Installing horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Notification
  - Installing horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Compress
  - Installing horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mime_Viewer
  - Installing horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/LoginTasks
  - Installing horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Lock
  - Installing horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/JavascriptMinify
  - Installing psr/http-server-handler (1.0.2): Extracting archive
  - Installing psr/http-server-middleware (1.0.2): Extracting archive
  - Installing horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2): Symlinking from /srv/git/horde/Http_Server
  - Installing horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Cli
  - Installing horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Autoloader
  - Installing horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Auth
  - Installing horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Alarm
  - Installing horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17): Symlinking from /srv/git/horde/Core
  - Installing horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Dav
  - Installing horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Rpc
  - Installing horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Image
  - Installing horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Form
  - Installing horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Argv
  - Installing horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7): Symlinking from /srv/git/horde/base
  - Installing horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Yaml
  - Installing horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cli_Modular
  - Installing horde/hordectl (v1.0.0alpha4): Extracting archive
  - Installing horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Routes
  - Installing horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Test
  - Installing pear/console_color2 (0.1.2): Extracting archive
  - Installing pear/console_table (v1.3.1): Extracting archive
81 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files
Applying /presets for absent files in /var/config
Looking for registry snippets from apps
Writing app configs to /var/config dir
Linking app configs to /web Dir
Linking javascript tree to /web/js
Linking themes tree to /web/themes
1 package you are using is looking for funding.
Use the `composer fund` command to find out more!

This composer environment works just like a regular installation. When you install turba, kronolith or passwd through composer, it will end up linking these apps and their library dependencies from the development tree.

Composer provides an option to copy files from the repositories rather than link the files. This would allow creating archives with artifacts for distribution packaging. The horde-components tool does not yet provide a switch to generate the necessary tweak to the repository files.

bookmark_borderMaintaina’s Horde 6 goes upstream

Posted on by Ralf

It’s been a while. It’s been much too long actually. Let’s look forward though.
I recently started to onboard into FOSS development again. People have been asking for PHP 8.2 support with Horde but capacity to deliver that was very limited. But we’re getting there.

I used the “Maintaina” fork to deliver fixed and upgraded customized versions of Horde beyond what was allowed in Horde’s master branch. It’s time to move forward though. Maintaina used to target PHP 7.4 to PHP 8.1, composer2 based install from a custom satis repo and some quite invasive changes to CalDAV/CardDAV support. Most prominently, Maintaina introduced library level compatibility with several PSRs (container, http middleware, logging) and the all new horde/http_server component.

At the moment I am importing the FRAMEWORK_6_0 branches and the alpha releases based on this branch to horde’s upstream repository. So far I have imported enough libraries to make the Horde Base application barely install from packagist.

Get the current state of affairs

Your magic carpet is:

composer create-project horde/bundle .

Originally I wanted to mimic maintaina’s setup with a separate satis server as a QA stage before release to packagist. I re-considered and dropped the separate satis server. Every update of the development branches and every tagged release is consumed via packagist. I will have to fix some of my tools and workflows to reflect that. The tagged alpha releases still use maintaina’s satis server. The upcoming releases won’t.

Onboarding Procedure


I leave the FRAMEWORK_5_2 and master branches mostly untouched. FRAMEWORK_6_0 is the new default branch on github for anything I am handling. I only edit other branches if they block packagist’s import. Usually I also rebase FRAMEWORK_6_0 on any latest commits of master, but in some cases I only cherry-picked from master branch. In some cases, some entries of the changelog between 2022 late and now (october 2023) might be missing. Pull Requests, bug reports and patches welcome.

What’s next

Before I move forward with the actual applications, I want to make sure the necessary infrastructure is in place. I need to fix some aspects of the FRAMEWORK_6 version of composer. The workflow files in each repo need some review, too. Does PHPUnit and PHPStan still pass? Can we improve management of 130+ repo’s workflows?

Finally https://dev.horde.org should index from packagist, not from the satis server we originally planned to use.

What will become of maintaina repos?

The maintaina repos have had direct contributions from some trusted maintainers from the company I used to work for. They service some customers out of these repos and the related SATIS server so I won’t actively break it. However, with the move to Horde upstream, maintaina has served its purpose for me and I will not actively support the fork anymore. I suggest once everything is ported to upstream, maintaina should be archived. I will need to consult other stakeholders of this fork and the satis server.

Maintenance cost

Over the lifetime of the fork I have explored and applied various strategies for keeping the effort in check. Still, a fork of 100+ repos and the accompanying infrastructure for testing and deploying is a major burden which detracts from actually developing and maintaining code. I am glad I can save on this now and actually contribute to Horde directly in a way that doesn’t slow down activity too much. We now have the chance to speed up the cycle of feedback and releases. I hope this attracts some occassional and regular contributors.

bookmark_borderHorde Installer: Recent Changes

Posted on by Ralf

The maintaina-com/horde-installer-plugin has seen a few changes lately. This piece is run on every composer install or update in a horde installation. A bug in it can easily break everything from CI pipelines to new horde installations and it is quite time consuming to debug. I usually try to limit changes.

Two codebases merged

In the 2.3.0 release of November 2021 I added a new custom command horde-reconfigure which does all the background magic of looking up or creating config snippets and linking them to the appropriate places, linking javascript from addon packages to web-readable locations and so on. This is essentially the same as the installer plugin does but on demand. A user can run this when he has added new config files to an existing installation. Unfortunately the runtime environment of the installer plugin and the custom command are very different in terms of available IO, known paths and details about the package. I took the opportunity to clean up code, refactor and rethink some parts to do the same things but in a more comprehensible way. As I was aware of the risks I decided to leave the original installer untouched. I got some feedback and used it myself. It seemed to work well enough.

For the 2.4.0 release I decided to finally rebase the installer onto the command codebase and get rid of the older code. It turned out that the reconfigure command was lacking some details which are important in the install use case. Nobody ever complained because these settings are usually not changed/deleted outside install/update phase. As of v2.4.4 the installer is feature complete again.

New behaviour in v2.4

The installer has been moved from the install/update phase to the autoload-dump phase. It will now process the installation as a whole rather than one library at a time. This simplifies things a lot.reviously, the installer ran for each installed package and potentially did a few procedures multiple times. Both the installer and the horde-reconfigure command will now issue some output to the console about their operation and they will process the installation only once with the updated autoloader already configured. The changes will now also apply on removal of packages or on other operations which require a rewrite of the autoloader. The registry snippets now include comments explaining that they are autogenerated and how to override the autoconfigured values.

Outlook to 2.5 or 3.0

The composer API has improved over the last year. We need to be reasonably conservative to support OS distribution packaged older versions of composer. At some point in the future however I want to have a look at using composer for simplifying life

  • Improve Theme handling: Listing themes and their scope (global and app specific), setting default theme of an installation
  • Turning a regular installation into a development setup for specific libraries or apps
  • Properly registering local packages into composer’s package registry and autoloader (useful for distribution package handling).

Both composer’s native APIs and the installer plugin can support improving a horde admin’s or developer’s life:

  • Make horde’s own “test” utility leverage composer to show which optional packages are needed for which drivers or configurations
  • Expose some obvious installation health issues on the CLI.
  • Only expose options in the config UI which are supported by current PHP extensions and installed libraries
  • Expose a check if a database schema upgrade is needed after a composer operation, both human readable and machine consumable. This should not autorun.

The actual feature code may be implemented in separate libraries and out of scope for the installer itself. As a rule, horde is supposed to be executable without composer but this is moving out of focus more and more.

bookmark_borderMaintaina/Horde UTF-8 on PHP 8

Posted on by Ralf

On recent OS distributions, two conflicting changes can bring trouble.

MariaDB refuses connections with ‘utf-8’ encoding

Recent MariaDB does not like the $conf[‘sql’][‘charset’] default value of ‘utf-8’. It runs fine if you change to the more precise ‘utf8mb4’ encoding. This is what recent MySQL understands to be ‘utf-8’. You could also use ‘utf8mb3’ but this won’t serve modern users very well. The ‘utf8m3’ value is what older MariaDB and MySQL internally used when the user told it to use ‘utf-8’. But this character set supports only a subset of unicode, missing much-used icons like ☑☐✔✈🛳🚗⚡⅀ which might be used anywhere from calendar events sent out by travel agencies to todos or notes users try to save from copy/pasted other documents.

I have changed the sample deployment code to use utf8mb4 as the predefined config value.

Shares SQL driver does not understand DB-native charsets

The Shares SQL driver does some sanitation and conversion when reading from DB or writing to DB. The conversion code does not understand DB native encodings like “utf8mb4”. I have applied a patch to the share library that would detect and fix this case but I am not satisfied with this solution. First, this issue is bound to pop up in more places and I wouldn’t like to have this code in multiple places. Either the DB abstraction library horde/db or the string conversion library in horde/util should provide a go-to solution for mapping/sanitizing charset names. Any library using the config value should know that it needs to be sanitized but should not be burdened with the details. I need to follow up on this.

Update

See https://github.com/horde/Util/commit/7019dcc71c2e56aa3a4cd66f5c81b5273b13cead for a possible generalized solution.

bookmark_borderSimplifying Routing / PSR-15 bootstrap in Horde

Posted on by Ralf

As you might remember from a previous post, Horde Core’s design is more complex than necessary or desirable for two main reasons:

  • Horde predates today’s standards like the Composer Autoloader and tries to solve problems on its own. Changing that will impair Horde’s ability to run without composer which we were hesitant to do, focusing on not breaking things previously possible.
  • Horde is highly flexible, extensible and configurable, which creates some chicken-egg problems to solve on each and every call to any endpoint inside a given app.

Today’s article concentrates on the latter problem. More precisely, we want to make routing more straight forward when a route is called.

A typical standalone or monolithic app usually is a composer root package. It knows its location relative to the autoloader, relative to the dependencies and relative to the fileroot of the composer installation. Moreover, the program usually knows about all its available routes. It will also have some builtin valid assumptions about how its different parts’ routes relate to the webroot.
None of this is true with a typical composer-based horde installation.

  • None of the apps is the root package
  • Apps are exposed to a web-readable subdir of the root package
  • While the relative filesystem location is known, each app can live in a separate subdomain, in the webroot or somewhere down the tree
  • Each app may reconfigure its template path, js path, themes path
  • The composer installer plugin makes a sane default. This default can be overridden
  • Each app can be served through multiple domains / vhosts with different registry and config settings in the same installation
  • Administrators can add local override routes
  • Parts of the code base rely on horde’s own runtime-configurable autoloader rather than composer.

This creates a lot of necessary complexity. The router needs to know the possible routes before it can map a request. To have the routes, the context described above must be established. Complexity cannot be removed without reducing flexibility. There is, however, a way out. The routing problem can be divided into three phases with different problems:

  • Development time – when routes are defined and changed frequently for a given app or service
  • Installation/Configuration time – when the administrator decides which apps’ routes will be available for your specific installation
  • Runtime – when a request comes in and the router must decide which route of which app needs to react – or none at all

Let’s ignore development time for now. It is just a complication of the other two cases. The design goal is to make runtime lean and simple. Runtime should initialize what the current route needs to work and as little as possible on top of that. Runtime needs to know all the routes and a minimal setup to make the router work. Complexity needs to be offloaded into installation time. Installation time needs to create a format of definite routes that runtime can process without a lot of setup and processing. As a side effect, we can gain speed for each individual call.

Modern Autoloaders are similar in concept: They have a setup stage where all known autoloader rules of the different packages are collected. In composer, the autoloader is re-collected in each installation or update process. The autoloader is exposed through a well-known location relative to the root package, vendor/autoload.php – it can be consumed by the application without further runtime setup. The autoloader can be optimized further by processing the autoloading rules into a fixed map of classes to filenames (Level 1), making these maps authoritative without an attempt to fail over on misses (Level 2a) and finally caching hits and misses into the in-memory APCu opcache. Each optimization process makes the lookup faster. This comes at the cost of flexibility. The mapping must be re-done whenever the installation changes. Otherwise things will fail. This is OK for production but it gets in the way of development. The same is true for the router.

The best optimization relies on the application code and configuration being static. The list of routes needs to be refreshed on change. Code updates are run through composer. The composer installer plugin can automatically refresh the router. Configuration updates can happen either through the horde web ui or through adding/editing files into the configuration area. Admins already know they need to run the composer horde-reconfigure command after they added new config files or removed files. Now they also need to run it when they changed file content. In development, routing information may change on the fly multiple times per hour. Offering a less optimized, more involved version of this route collection stage can help address the problem.

A new version of the RampageBootstrap codebase in horde/core is currently in development. It will offload more of horde’s early initialisation stages into a firmware stack and will reduce the early initialisation to the bare minimum. At the moment, I am still figuring out how we can do this in a backward compatible way.

bookmark_borderRdo: Persistence is not your model

Posted on by Ralf

Remember that post on how your backend might betray you? You can store your Turba addressbook into an LDAP tree, but if the addressbook is manipulated from LDAP side, your CardDAV Sync may be ignorant of this. The bottom line is: You cannot trust the backend. Nor should the persistence model govern your application internal model.

The development team at B1 works with a lot of inherited code from different areas and eras of FOSS and closed-source development. We see all styles of code and we have made all sorts of bad micro decisions ourselves over the years. One particulary hard question that comes up time and again is what is the “true” model of data in an application. Developers who come from traditional, monolithic web applications may say the true model is what is in the (relational) database and point out that it will closely influence the application’s internal model. Developers with a background in APIs and distributed apps will tend to think the messages going in and out of a service are the main thing. Application will be modeled after the messages and persistence is a secondary thought. Then we have the school of DDD-trained developers who will say neither is right. The internal model should be mostly ignorant of persistence and external API is just another type of persistence.

Let’s explore the benefits and impacts of these options.

DB-centric applications

In a DB-centric application, your entities closely represent rows of data in DB tables. If you use ActiveRecord or DataMapper style ORMs, you will often use the ORM Entities as your business objects in the application. This works fairly well for CRUD style applications where most if not all fields in the database will be editable form fields on screen.

If your application does little beyond storing, retrieving, updating and deleting “records” of whatever type, you will have a straight-forward development experience. Some frameworks may support autogenerating create/view/edit forms and searchable lists (html tables) out of your DB schema. Because it is the easiest way to add features to these types of application, the object model tends to be rich in attributes and limited in relations and segregation.

For example, it would be common for a “customer” entity to include separate fields for the customer’s street, city, country, email address… some fields would be marked as mandatory and others as optional, each accessible via getters and setters or public properties. This becomes cumbersome for entities which have subtypes with different sets of attributes or behaviours. Fields may be mandatory for one subtype, but optional or even forbidden for other subtypes. Attribute values are usually primitives (string, int) that can be mapped to DB column types easily. Developing these types of apps is really fast and concise as long as you do nothing fancy. As soon as you have any real amount of business logic and variance, it becomes cumbersome to maintain and hard to test.

API-centric applications

A similar school of thoughts comes out of microservice development. In many cases, CRUD services can be prototyped from autogenerated code defined through a Swagger/OpenAPI definition file. Sometimes there is little to do after this autogeneration, including persistence to the relational database. If you use a non-relational persistence like CouchDB, you may even get along with some variance and depth in the schema. Even a json or yaml file on disk gets you very far (at least in the prototype stage).

However, you will have scenarios where the same application object looks quite different between API requests. Users may have limited permissions on querying details of an object, different APIs with different use cases may have limited needs on the deeper details of objects. Retrieving your list of customer IDs and customer names for populating a dropdown is different from being able to view their billing data or contact persons’ details.

Domain Model centric approach

If your application has to deal with structured aggregates and consistency rules, exhibits rich behaviour, services multiple types of backends or is otherwise complicated, this might be the right approach.

At its core, the (micro)application is fairly ignorant of both persistence, export formats and external API.
The domain model deviates from persistence model or message representations. It has internal constraints and business rules. A domain entity is retrieved from repository in valid and complete state and all transformations will result in valid state. Transformations usually happen through defined operations rather than accessing a set of getters and setters. Setters may even not exist for many types of domain objects.

Through this, you can trust your objects by contract and reduce validations in the actual operations. This comes at a price: In a domain centric application, your classes may double or triple compared to other approaches.

You have your business object with all the internal integrity aspects. You have another version of the same object which is used for I/O – the so-called Data Transfer Object or DTO. There are versions where DTOs are either typed classes or untyped plain objects with just arbitrary public attributes. They may even just be structured arrays. Each option comes with a drawback. These world-readable objects expose all their relevant data to some kind of I/O – views rendered on the server side, files to export, REST API messages coming in or going out. A third version of your entity may be the persistence model(s), especially in relational databases. They may decompose your domain entity into multiple database tables, LDAP tree nodes or even multiple representations in no-sql databases. The main effort is getting all these transformations right. This type of application may seem verbose and repetitive. On the other hand, a strong object model with hard internal constraints and few dependencies on unrelated aspects is very straight-forward to unit test. This makes it suitable for large scale applications.

Compromises and practice

You do not have to commit to one approach with all consequences. This might even make less sense in languages which do not really enforce type hints (like python) or have no enforced concept of private and protected properties (again, like python). In many cases, you can get very far by discipline alone. Just restrict usage of setters to the persistence and I/O parts of your application. Add methods and behaviour to your persistence layer entities and even better, add an interface. Hint your business logic against the interface. You can also make your ORM mapper double as a Repository. You can scale out to real domain objects step by step as needed. Add conversion methods which turn a domain entity into one or many ORM entities and commit them to the DB. If your aggregate is not very suitable for this, hide your ORM mappers in a dedicated repository class. You can scale out as little or as much as your needs dictate.

On the other hand, you may abuse your ORM entities to populate views or data formats like XML, json or yaml.

Why we move away from putting logic into Rdo entities

Rdo is Rampage Data Objects, Horde’s minimalistic ORM solution. We have come a long way using the described tricks to make Horde_Rdo_Mappers work as repository implementations and Horde_Rdo_Base entities as business objects. However, we run into more and more situations where this is not appropriate. Using and lazy-loading relations to child objects is fine in the read use case, but persisting changes to such structures becomes tedious and error prone. Any logic tightly coupled to Rdo objects is also hard to unit test. Rdo-centric development does not mix well with the Shares library or with handling users and identities. As Rdo entities carry around references to the mapper and the database driver, they tend to bloat debug output. More fundamentally, our object models are evolving to a design which does not really look like database tables at all. We still love Rdo and may resort to Rdo entities with interfaces here and there, especially in prototyping. But our development model has long shifted towards unittest-early or unittest-driven rather.

Beware of the edges

Domain Driven Design purists may emphasize how domain models will rarely have attribute setters and some will even argue you should minimize your getters (tell don’t ask). However, at some point we need to get raw attributes to compose messages to the edges, to answer API requests or to persist any objects. There are multiple ways to do it, with different implications. Remember the Data Transfer Objects? I tend to have method on my domain aggregate to “eject” a fully detailed world readable object. In many cases, it is even typed. This chatty object can now be used for Formatters to transform them into API messages or data files. So the operation would be: Construct the domain aggregate – and fail if data is invalid – and from the domain aggregate, construct the DTO. Use the DTO for chatty jobs.

But how about the other way around? Messages from the outside can be malformed in many ways. They may be tampered with, they may miss details by design. If I exported an object to a user’s limited API and he sends an update, how to handle that? If an API user does not even know advanced attributes that are subject to other views and use cases, how would he create new entities?

I tend to have a method on the repository which accepts these messages and tries to turn them into Domain Objects. If the partial message contains some ID field or a sufficient set of data to form a unique key, I will first ask the backend to get the original details of the object and then apply the message details – and fail if this violates the constraits of my aggregate. If no previous version exists in the backend, missing information is added from defaults and generators, including a unique id of some sort. The message is applied on top.

The is approach may be expensive. I am creating full domain objects just to render database content to another format which may not even contain most of the retrieved data. On the other hand, if I know an object exists in a relational db, I could use some cheap SQL to apply partial changes without ever constructing the full model.

Creating straight-to-output repos with optimized queries is a tradeoff. They mean additional maintenance burden whenever the model changes, additional parts to cover with tests. I only do this when performance actually becomes an issue, not by default.

Creating straight-to-persistence methods for partial messages is even less desirable. It is circumventing integrity check at code level. However, sometimes the message itself is the artifact to persist – to be later processed by a queue or other asynchronous process.

bookmark_borderPEAR down – Taking Horde to Composer

Posted on by Ralf

Since Horde 4, the Horde ecosystem heavily relied on the PEAR infrastructure. Sadly, this infrastructure is in bad health. It’s time to add alternatives.

Everybody has noticed the recent PEAR break-in.

A security breach has been found on the http://pear.php.net webserver, with a tainted go-pear.phar discovered. The PEAR website itself has been disabled until a known clean site can be rebuilt. A more detailed announcement will be on the PEAR Blog once it’s back online. If you have downloaded this go-pear.phar in the past six months, you should get a new copy of the same release version from GitHub (pear/pearweb_phars) and compare file hashes. If different, you may have the infected file.

While I am writing these lines, pear.php.net is down. Retrieval links for individual pear packages are down. Installation of pear packages is still possible from private mirrors or linux software distribution packages (openSUSE, Debian, Ubuntu). Separate pear servers like pear.horde.org are not directly affected. However, a lot of pear software relies on one or many libraries from pear.php.net – it’s a tough situation. A lot of software projects have moved on to composer, an alternative solution to dependency distribution. However, some composer projects have dependency on PEAR channels.

I am currently submitting some changes to Horde upstream to make Horde libs (both released and from git) more usable from composer projects.
Short-term goal is making use of some highlight libraries easier in other contexts. For example, Horde_ActiveSync and Horde_Mail, Horde_Smtp, Horde_Imap_Client are really shiny. I use Horde_Date so much I even introduced it in some non-horde software – even though most functionality is also somewhere in php native classes.

The ultimate goal however is to enable horde groupware installations out of composer. This requires more work to be done. There are several issues.

  • The db migration tool checks for some pear path settings during runtime https://github.com/horde/Core/pull/2 Most likely there are other code paths which need to be addressed.
  • Horde Libraries should not be web readable but horde apps should be in a web accessible structure. Traditionally, they are installed below the base application (“horde dir”) but they can also be installed to separate dirs.
  • Some libraries like Horde_Core contain files like javascript packages which need to be moved or linked to a location inside another package. Traditionally, this is handled either by the “git-tools” tool linking the code directory to a separate web directory or by pear placing various parts of the package to different root paths. Composer doesn’t have that out of the box.

Horde already has been generating composer manifest files for quite a while. Unfortunately, they were thin wrappers around the existing pear channel. The original generator even took all package information from the pear manifest file (package.xml) and converted it. Which means, it relied on a working pear installation. I wrote an alternative implementation which directly converts from .horde.yml to composer.json – Calling the packages by their composer-native names. As horde packages have not been released on packagist yet, the composer manifest also includes repository links to the relevant git repository. This should later be disabled for releases and only turned on in master/head scenarios. Releases should be pulled from packagist authority, which is much faster and less reliant on existing repository layouts. https://github.com/horde/components/pull/3

To address the open points, composer needs to be amended. I currently generate the manifests using package types “horde-library” and “horde-application” – I also added a package type “horde-theme” for which no precedent exists yet. Composer doesn’t understand these types unless one adds an installer plugin https://github.com/maintaina-com/installers. Once completed and accepted, this should be upstreamed into composer/installers. The plugin currently handles installing apps to appropriate places rather than /vendor/ – however, I think we should avoid having a super-special case “horde-base” and default to installing apps directly below the project dir. Horde base should also live on the same hierarchy. This needs some additional tools and autoconfiguration to make it convenient. Still much way to go.

That said, I don’t think pear support should be dropped anytime soon. It’s the most sensible way for distribution packaging php software. As long as we can bear the cost involved in keeping it up, we should try.

bookmark_borderhorde trustr – A new horde CA app step by step

Posted on by Ralf

Trustr is my current project to create a simple certificate management app.
I decided that it is just about the right scope to demonstrate a few things about application development in Horde 5.

I have not made any research if the name is already occupied by some other software. Should any problems arise, please contact me and we will find a good solution. I just wanted to start without losing much time on unrelated issues.

My goals as of now:

– Keep everything neat, testable, fairly decoupled and reusable. The core logic should be exportable to a separate library without much change. There won’t be any class of static shortcut methods pulling stuff out of nowhere. Config and registry are only accessed at select points, never in the deeper layers.
– Provide a CLI using Horde_Cli and Horde_Cli_Application (modeled after the backup tool in horde base git)
– Store to relational database using Horde_Db and Horde_Rdo for abstraction
– Use php openssl extension for certificate actions, but design with future options in mind
– Rely on magic openssl defaults as little as possible
– Use conf.xml / conf.php for any global defaults
– Show how to use the inter-app API (reusable for xml-rpc and json-rpc)
– Showcase an approach to REST danabol ds in Horde (experimental)

The app is intended as a resource provider. The UI is NOT a top priority. However, I am currently toying around with a Flux-like design in some unrelated larger project and I may or may not try some ideas later on.

Initial Steps: Creating the working environment

I set up a new horde development container using the horde tumbleweed image from Open Build Service and a docker compose file from my colleague Florian Frank. Please mind both are WIP and improve-as-needed projects.


git clone https://github.com/FrankFlorian/hordeOnTumbelweed.git
cd hordeOnTumbelweed
docker-compose -f docker-compose.yml up


This yields a running horde instance on localhost and a database container.
I needed to perform a little manual setup in the web admin ui to get the DB to run and create all default horde schemas.

Next I entered the developer container with a shell
docker exec -it hordeOnTumbelWeed_php_1 bash

There are other ways to work with a container but that’s what I did.

 

Creating a  skeleton app

The container comes with a fairly complete horde git checkout in /srv/git/horde and a clone of the horde git tools in /srv/git/git-tools

A new skeleton app can be created using

horde-git-tools dev new --author "Ralf Lang <lastname@b1-systems.de>" --app-name trustr

The new app needs to be linked to the web directory using

horde-git-tools dev install

Also, a registry entry needs to be created by putting a little file into /srv/git/horde/base/config/registry.d

cat trustr-registry.d.php

<?php
// Copy this example snipped to horde/registry.d
$this->applications['trustr'] = array(
'name' => _('Certificates'),
'provides' => array('certificates')
);

 

This makes the new app show up in the admin menu. To actually use it and make it appear in topbar, you also need to go to /admin/config and create the config file for this app. Even though the settings don’t actually mean anything by now, the file must be present.

I hope to follow up soon with articles on the architecture and sub systems of the little app.