Making plans is great but it’s just words. How are things going? Clearly moving towards goals but in unexpected ways.
Trains don’t only run on railroad tracks but supposedly on schedule. Commuters shall know every stop, arrival and departure, the order of cars and their position on the platform. If trains regularly arrive delayed, at another platform, composed of different or differently ordered cars it is a source of frustration. If they skip a station or get relayed to another course, it’s even worse. Software projects, on the other hand, are expected to detour regularly in very similar ways but hopefully arrive at certain way points more or less on time. Sometimes they get unplanned additional cars along the way. Sometimes you have to replace the engine. It never gets boring.
What is the goal?
In January, I anticipated there were some unknowns ahead but I did not detail how to tackle this.
Improve JMRI‘s support for the Märklin MCAN protocol and particularly the Mobile Station 2, Central Station 3 and Can-Digital-Bahn products. This might become a bit controversial.
The Märklin Hardware Throttles (Mobile Station 2) implement hot plugging and negotiate which one acts as the primary. They send a regular PING command announcing their type and serial and expect a PONG response from other devices on the bus. The most senior serial number is agreed to be the primary throttle even if the other one was the primary or single throttle until now. The primary offers its roster to the other throttles. If higher end Central Station systems are on the bus, one of the central station will always be the primary device and the throttles all behave a little different. This allows JMRI to passively listen to traffic and detect bus devices or actively send a PING and collect the PONGs.
The Can-Digital-Bahn devices have a similar PING/PONG mechanism used by their configuration utilities. Unfortunately there are different configuration utilities for different generations of their sensors, relays, turnout controllers, light controllers, switchboard components, mixed devices. These utilities are closed source Windows programs and you need different versions of the program for different versions of the components. Unfortunately, not all of the programs properly run on recent Windows systems.
What if I just had to push a button in the web browser and see all my MCAN throttles and devices in a table, loading the right configuration screen for each of them? Sounds much better, but how to achieve that?
Moving through the stack
To send the PING messages and receive the PONG messages a program needs to connect to the MCAN bus. This can be achieved by connecting the CC-Schnitte interface to the USB port of the computer or connecting to a Central Station device over TCP/IP. Both methods expose the raw MCAN bus to the software. Both options are already implemented in JMRI.
Next the necessary bus messages and response need to be added to the software. The MCAN PING command is already implemented but JMRI doesn’t implement the desired reactions to the answers. For can-digitial-bahn messages, support is still missing.
To use the found devices, a program needs to memorize some representation of them. JMRI currently does not have a generic way of handling external bus devices as such. Throttles and sensors are tracked in their specific roles, covering their common aspects as throttles or as sensors. For CBUS type connections, there is a node manager which looks very similar to what is needed for MCAN but the UI code and the table format probably deviates a bit. This is where it becomes a bit complicated.
Users need some kind of GUI to interact with found devices. There are many reasons to start out with a browser based UI instead of building it in JMRI’s native Java GUI. The browser based ui just works on any tablet or laptop connected to the network. It’s well decoupled from JMRI’s core. It can be styled or completely recomposed as needed without touching JMRI itself. JMRI does not need to bundle variants for different use cases but they can be installed separately, developed on their own schedule. JMRI only needs to expose the necessary interface protocol which also drives other parts of the browser based UI.
JMRI’s interface protocol is composed from JSON messages between the browser client and the JMRI server. These can be sent over the HTTP server port or through a separate WebSocket implementation. The latter provides better performance and less overhead.
JMRI’s WebSockets and HTTPS API
This is what I ended up doing. To familiarize myself with the API I built a simple demo use case to manipulate the displayed railroad company name. Previously this was only possible through the preferences screen in the native Java GUI. I ended up not integrating it in the regular browser based GUI but implement a separate demo page. JMRI core developers pointed out that special care is needed and unauthenticated calls to the API must not actually persist the changed configuration into JMRI.
Obviously this would not work for actually handling hardware. While JMRI already has a permissions and user authentication system, it is fairly new and does not cover the actual API messages. So my next step would be to implement the necessary messages in the JSON WebSocket API for retrieving an authentication token. This token would then be used to authorize further WebSocket calls which actually change something in JMRI. These new calls need to carry the authorization token as part of the data.
I plan to detail this development in a separate article later this month as I move through the process.
Seems like I should be getting somewhere
This free time project is a great adventure into the unknown between where I am and where I want to be next. In some ways this is similar to professional work. I commit to goals and target dates and I have a very clear understanding of next week and some ideas about the week after. Beyond that the way points and schedule dates become rarer.
It’s sometimes very challenging to tell project managers that there won’t be many super detailed milestones beyond the horizon of immediate next steps. Gladly, at work I have very smart project managers who know when to trust me and when to challenge me. Sadly, in free time I have a very harsh and unforgiving project manager who often won’t take no for an answer.
Paradox Development Studios is famous for their grand strategy games including the flagship Europa Universalis franchise. While a new main version is developed under code name Project Caesar, the latest Europa Universalis IV still gets new expansion packs (Download Content) and updates to the base game. Paradox even backported several former add-ons into the latest version of the base games. We are talking about a product originally released in 2013, almost a teenager now.
Their 2019 spin off Imperator: Rome turned out to be a much smaller commercial success. After a few add-ons and updates it got cancelled in 2021 due to dwindling sales. It was not their first foray into ancient history. Europa Universalis: Rome debuted in 2008 but it was not a long lived installment. Only one download content got produced before the product was relegated to the back catalog.
While Imperator: Rome is based on technology introduced for Europa Universalis 4 and the concurrent generation of spinoffs, Europa Universalis: Rome is clearly a member of the Europa Universalis 3 generation of the engine, UI design and scripting approaches. Imperator debuted the Jomini middleware between the Clausewitz engine and the actual game content. Jomini has since been used in later installments of various Paradox Grand Strategy games.
Even though Imperator product development has been cancelled in 2021, Paradox employees have still worked on bug fixes and feature patches. In April 2024, a new version 2.0.4 has been released for all supported platforms after it was available as an opt-in beta version for several months. In December 2024 a new patch 2.0.5 has been made available as public beta. The patch contains minor UI tweaks, some fixes to the computer player AI, some bug fixes and many additional hooks and exposed variables for the scripting engine. This allows builders of unofficial game modifications (“Mods”) to add new behaviours to the game like Trade Embargoes or forbidding some game characters to inherit provinces. Imperator: Invictus is the most prominent modification for the official Imperator game. It works both with the latest version of the base game and optionally all available official addons.
I find it very motivating and inspiring when a vendor continues to work with fans and open source community and supports their efforts to add more value to their discontinued commercial product.
As of January 2025 Imperator: Rome and all individual add-ons are available at a heavy discount from Paradox’ own store. At the moment they offer the individual components for less money than the official bundle. I am in no way affiliated with Paradox but I like the way they do this and the game is really fun.
Looking back on 2024, it was a challenging, demanding and sometimes crazy year. But let’s start with athe good things in life: There was so much joy to be had with the family, so much to learn and discover at work, so much progress on personal development and discovering new approaches to things that just won’t go away. All these precious moments I would not want to miss. It would be wrong to just skip over this without this little line of appreciation. I benefit very much from my family, friends and colleagues and I hope to give back a little bit of joy and a helping hand wherever I can. Thank you all.
But if you know me: I am ambitious, I am loyal to stuff I started, sometimes stubborn and I am an open source software guy. From this perspective, things were not going so well. While I am used to full schedules and stressful timelines at work, I had to re-prioritize my free time harshly in the middle of the first quarter. This got a lot of private projects into an undeclared hiatus and ultimately frozen for the rest of the year. It came unexpected and initially I did not grasp the extent of what was happening. To give you an idea, I even stopped reading mailing lists and most of my private inbox for a while. I did not plan any public talks and only visited FOSS events if they were job related. By coincidence, the main laptop with open source work setups also went defunct and I did not bother to replace it for several months.
If you feel you get into a similar situation, I can only advise you to take no time for regrets. Do what needs to be done and cut what you’d like to do on top until you feel ready again. Watch for the metrics: Do you start reading again? Is it more pages of fiction and non-fiction than last month? Do you take time for the occassional board or computer game? Can you make time again for you hobby, club or sports activities? Then maybe you’re set to come back to free time FOSS development – if you still like it. Don’t feel you need or must or have to. Maybe take another two weeks to question yourself before you start over. Nothing is won with spending hours in an exhausted or half detracted state.
That said, I am very positive to slowly bring everything back to a more happy status one at a time. A new FOSS work setup is available. Some of 2024’s learnings may help me to achieve more while still on a limited time budget. I am catching up with mails and yes, your pull requests will be handled. There’s also some clerical work to be done for my pet projects. It’s too early to announce a specific timeline or commitment and sometimes opportunity trumps priority. Let’s see how that works.
So what’s on the agenda?
Horde 6 Framework and main apps need to go to “stable”. I don’t want any ambiguity anymore. Make it run on the recent and upcoming PHP and try not to break it for any officially maintained release +1
PoC browser & thunderbird plugin with some very limited initial use case. Maybe a sidebar for Horde Mnemo Notes. Let’s gather some experience before making big plans.
Quality of life for aspiring developers: Documentation, test setup, reference setup
Modern authentication focus: I’ve done a custom 2FA/TOTP solution several years ago and I have been meaning to port it to Horde’s core for several years. The authentication stack probably needs to be revisited to make it a natural fit. I’d like to keep it nicely split into an interface part and the actual implementation so exchanging the TOTP app for some QR Code Challenge on your device, email confirmation or similar should be possible without modifying the core code.
Continue porting some traditional pages to the PSR-7 standard based controller framework
Be more active again in PHP-FIG discussions and practical maintenance work on the PSR/PER related repositories
Having loads of fun and learning
Improve JMRI‘s support for the Märklin MCAN protocol and particularly the Mobile Station 2, Central Station 3 and Can-Digital-Bahn products. This might become a bit controversial. The current implementation is very much centered on fulfilling JMRI’s minimal requirements for a connection and throttle implementation and much code is a straight port from the TAMS implementation. I’d rather build a java library with internal cohesion and a good match to the bus protocol and the device philosophy. I have read into Bob’s more specialized TrainControl and Frans Jacob’s standalone program. I appreciate how much more cohesive and maintainable their implementation is. They achieve this by not letting an external frame dictate their internal structure and mental model. Of course this will need some glue to match it with JMRI’s wider logic but it’s also a great opportunity to simplify unit testing.
That’s already a tall order and I am sure more ideas and desires pop up along the way. No explicit priority, no promises, no commitment.
Install git and github cli through winget to avoid using a browser or manually entering the microsoft store tool. Install vscode and WSL if you need it.
winget install -e --id GitHub.cliwinget install --id Git.Git -e --source winget
## if visual studio code is your thing
winget install vscode
winget install Microsoft.WSL
Issue: You can use FTDI based USB devices such as microcontroller boards and embedded products in Ubuntu Linux on physical devices and on full flegded VMs but under WSL2 the device does not work. You can successfully map the USB device in usbipd and you see it show up in lsusb, dmesg, journal. But no new serial device shows up under /dev/tty* and you cannot use the device.
Background: Ubuntu under Windows 11 WSL2 as of 2024/04 does not include the ftdi-sio driver in its default kernel builds. You cannot easily install that driver from the package manager either. Under WSL, similar to container environments and solaris zones, all linux guests use the same kernel.
Solution: Build and install a custom linux kernel with builtin FTDI drivers. At the time of writing, the default WSL kernel is 5.15 and the latest WSL branch is 6.1
Enter the Ubuntu WSL environment:
# If ubuntu is your only WSL or your default WSL, simply type "wsl"
# Double check with wsl -l if your ubuntu has another name like Ubuntu-22.04
# Mine is just called "Ubuntu"
wsl -d Ubuntu
Now download Microsoft’s modified kernel distribution and build it
git clone https://github.com/microsoft/WSL2-Linux-Kernel.git --depth=1 -b linux-msft-wsl-6.1.y
sudo apt update
sudo apt -y install build-essential flex bison libssl-dev libelf-dev bc
sudo apt -y install python3 pahole
cd WSL2-Linux-Kernel
## This line turns the FTDI driver from disabled or module to builtin
./scripts/config --file Microsoft/config-wsl --set-val CONFIG_USB_SERIAL_FTDI_SIO y
make -j$(nproc) KCONFIG_CONFIG=Microsoft/config-wsl
sudo make modules_install headers_install
## Use another target directory if you don't want the kernel in the root of C:\
cp arch/x86/boot/bzImage /mnt/c/bzImage-6.1
Save the file. Then run “wsl” or “wsl -d yourdistribution” to restart into the new kernel To verify inside wsl, run
uname -a
## Expect output similar to
## Linux W-PF384K2W 6.1.21.2-microsoft-standard-WSL2+ #1 SMP Tue Apr 2 22:11:36 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux
Note: In some cases, Windows will not actually terminate WSL and you need a full reboot of the host windows computer.
Over the last few weekends, Horde 6 code has been merged back from the Maintaina fork and from separate contributions to the former Horde development version, “master”. It was time to upgrade the development tool chain.
Back in the Horde 5 days, there was a utility called git-tools developed by Michael Rubinsky. It would checkout all horde library and application repositories of the github organization into a development directory – also called the git tree – and link into into another structure that resembled a regular Horde 5 installation – called the web tree. This required to hardcode some configurations what could otherwise be auto-detected but it allowed to organize the horde code base in a developer friendly flat structure and provide a ready to run test scenario that reflected the latest code changes even before they were committed to the official repositories. In the days of composer, half of what git-tools does is already covered by composer itself. The git-tools shell already had an integration for the horde-components tool which manages releases, change log, housekeeping tasks and some more.
It was obvious that I wanted a similar functionality for the composer-based setups. It took a while to get this right. The good news is I achieved that. Bear with me for another paragraph of incremental learning or skip to the next section with the results.
It’s getting better all the time
Originally I integrated a generator for “vcs” type repositories into the composer.json generation code. But these vcs repositories are relatively slow. This is OK for one or two apps or leave packages but it’s really really slow when you have a list of 80 or 120 dependencies each pulled from a separate vcs repository.
The next iteration involved generating a satis repository as a standin to releasing code into packagist. The satis repository would be updated by any commit or accepted pull request in any of the serviced repositories. Generating the satis repository is not particularly fast but I managed to scope refreshs to the individual component that needed updating. Reading from satis however is way faster than the vcs approach. Keeping an installation up to date with the latest development commits became much more viable. I also figured I could replace an already installed dependency with an actual git repository and composer would update the autoloader as needed. This works as long as you don’t change the autoloader rules, i.e. upgrade a package from PSR-0 to PSR-4.
Back to #1
Checking out individual libraries as root components and doing integration tests in the satis setup worked OK when the focus was on isolating individual pain points and solving bugs. For any development spanning changes on multiple libraries, it did not work out too well. I ended up implementing a new command.
horde-components github-clone-org
This will checkout a developer tree of git repositories containing apps, libraries or themes.
total 752
drwxr-xr-x 188 root root 4096 Jan 12 11:04 ./
drwxr-xr-x 4 root root 4096 Dec 4 17:44 ../
drwxr-xr-x 9 root root 4096 Jan 11 07:50 ActiveSync/
drwxr-xr-x 9 root root 4096 Jan 11 07:50 Alarm/
drwxr-xr-x 8 root root 4096 Jan 11 07:50 ApertureToAnselExportPlugin/
drwxr-xr-x 9 root root 4096 Jan 11 07:50 Argv/
drwxr-xr-x 9 root root 4096 Jan 11 07:50 Auth/
drwxr-xr-x 7 root root 4096 Jan 11 07:50 Autoloader/
drwxr-xr-x 7 root root 4096 Jan 11 07:50 Autoloader_Cache/
drwxr-xr-x 7 root root 4096 Jan 11 07:50 Backup/
drwxr-xr-x 7 root root 4096 Jan 11 07:51 Browser/
drwxr-xr-x 9 root root 4096 Jan 11 07:51 Cache/
drwxr-xr-x 9 root root 4096 Jan 11 07:51 Cli/
drwxr-xr-x 8 root root 4096 Jan 11 07:51 Cli_Application/
drwxr-xr-x 8 root root 4096 Jan 11 07:51 Cli_Modular/
drwxr-xr-x 8 root root 4096 Jan 11 07:52 Compress/
drwxr-xr-x 8 root root 4096 Jan 11 07:52 Compress_Fast/
drwxr-xr-x 8 root root 4096 Jan 11 07:52 Constraint/
drwxr-xr-x 8 root root 4096 Jan 11 07:52 Controller/
drwxr-xr-x 11 root root 4096 Jan 11 07:52 Core/
drwxr-xr-x 8 root root 4096 Jan 11 07:52 Crypt/
drwxr-xr-x 7 root root 4096 Jan 11 07:52 Crypt_Blowfish/
drwxr-xr-x 6 root root 4096 Jan 11 07:52 CssMinify/
drwxr-xr-x 8 root root 4096 Jan 11 07:52 Css_Parser/
...
drwxr-xr-x 7 root root 4096 Jan 11 08:17 Xml_Element/
drwxr-xr-x 7 root root 4096 Jan 11 08:17 Xml_Wbxml/
drwxr-xr-x 7 root root 4096 Jan 11 08:17 Yaml/
drwxr-xr-x 14 root root 4096 Jan 11 07:50 agora/
drwxr-xr-x 17 root root 4096 Jan 11 07:50 ansel/
drwxr-xr-x 21 root root 4096 Jan 11 07:51 base/
drwxr-xr-x 13 root root 4096 Jan 11 07:51 beatnik/
drwxr-xr-x 7 root root 4096 Jan 12 21:33 bundle/
drwxr-xr-x 12 root root 4096 Jan 11 07:51 chora/
drwxr-xr-x 15 root root 4096 Jan 13 16:54 components/
drwxr-xr-x 11 root root 4096 Jan 11 07:52 content/
drwxr-xr-x 2 root root 4096 Jan 11 08:02 dev.horde.org/
drwxr-xr-x 3 root root 4096 Jan 11 08:02 dns/
drwxr-xr-x 15 root root 4096 Jan 11 08:03 folks/
drwxr-xr-x 6 root root 4096 Jan 11 08:03 git-tools/
drwxr-xr-x 12 root root 4096 Jan 11 08:03 gollem/
drwxr-xr-x 9 root root 4096 Jan 11 08:03 groupware/
drwxr-xr-x 13 root root 4096 Jan 11 08:03 hermes/
drwxr-xr-x 6 root root 4096 Jan 11 08:08 horde-installer-plugin/
drwxr-xr-x 7 root root 4096 Jan 11 08:08 horde-support/
drwxr-xr-x 9 root root 4096 Jan 11 08:10 horde-web/
drwxr-xr-x 10 root root 4096 Jan 11 08:10 hylax/
drwxr-xr-x 7 root root 4096 Jan 11 08:12 iPhoto2Ansel/
drwxr-xr-x 14 root root 4096 Jan 11 08:11 imp/
drwxr-xr-x 14 root root 4096 Jan 11 08:12 ingo/
drwxr-xr-x 3 root root 4096 Jan 11 08:12 internal/
drwxr-xr-x 15 root root 4096 Jan 11 08:12 jonah/
drwxr-xr-x 12 root root 4096 Jan 11 08:12 klutz/
drwxr-xr-x 13 root root 4096 Jan 11 08:12 kolab_webmail/
drwxr-xr-x 9 root root 4096 Jan 11 08:12 koward/
drwxr-xr-x 18 root root 4096 Jan 11 08:13 kronolith/
drwxr-xr-x 13 root root 4096 Jan 11 08:13 luxor/
drwxr-xr-x 17 root root 4096 Jan 11 08:13 mnemo/
drwxr-xr-x 18 root root 4096 Jan 11 08:14 nag/
drwxr-xr-x 9 root root 4096 Jan 11 08:14 operator/
drwxr-xr-x 13 root root 4096 Jan 11 08:14 passwd/
drwxr-xr-x 11 root root 4096 Jan 11 08:14 pastie/
drwxr-xr-x 10 root root 4096 Jan 11 08:15 sam/
drwxr-xr-x 13 root root 4096 Jan 11 08:15 sesha/
drwxr-xr-x 11 root root 4096 Jan 11 08:15 shout/
drwxr-xr-x 13 root root 4096 Jan 11 08:15 skeleton/
drwxr-xr-x 8 root root 4096 Jan 11 08:16 timeobjects/
drwxr-xr-x 14 root root 4096 Jan 12 09:08 trean/
drwxr-xr-x 16 root root 4096 Jan 11 08:16 turba/
drwxr-xr-x 11 root root 4096 Jan 11 08:16 ulaform/
drwxr-xr-x 15 root root 4096 Jan 12 11:06 vilma/
drwxr-xr-x 10 root root 4096 Jan 11 08:17 webmail/
drwxr-xr-x 18 root root 4096 Jan 11 08:17 whups/
drwxr-xr-x 15 root root 4096 Jan 11 08:17 wicked/
This tree does not look like a composer based installation and would not easily work in a web browser. I did not want to reinvent composer with its autoloader and other benefits so I wrapped it into an installer. This installer creates a new copy of the horde/bundle base project and registers all the other libraries as a special type of composer repository “path”.
/srv/git/horde/components/bin/horde-components -c ~/horde-testme.conf.php install
[ INFO ] Installation directory is missing: /srv/www/testme
[ OK ] Created installation directory: /srv/www/testme
Creating a "horde/bundle" project at "../../www/testme"
Installing horde/bundle (dev-FRAMEWORK_6_0)
- Installing horde/bundle (dev-FRAMEWORK_6_0): Mirroring from /srv/git/horde/bundle
Created project in /srv/www/testme
When running the composer install command on this prepared setup, composer will not download the horde components from packagist or github but will use your local checkout. Only external dependencies are still downloaded from the web.
# composer install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 92 installs, 0 updates, 0 removals
- Locking horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
- Locking horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17)
- Locking horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4)
- Locking horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
- Locking horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7)
- Locking horde/horde-installer-plugin (v2.5.5)
- Locking horde/hordectl (v1.0.0alpha4)
- Locking horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
- Locking horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2)
- Locking horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11)
- Locking horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9)
- Locking horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
- Locking horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
- Locking horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
- Locking horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
- Locking horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
- Locking horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
- Locking horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
- Locking horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
- Locking horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
- Locking pear/archive_tar (1.4.14)
- Locking pear/console_color2 (0.1.2)
- Locking pear/console_getopt (v1.4.3)
- Locking pear/console_table (v1.3.1)
- Locking pear/pear (v1.10.14)
- Locking pear/structures_graph (v1.1.1)
- Locking pear/xml_util (v1.4.5)
- Locking php-extended/polyfill-php80-stringable (1.2.9)
- Locking psr/container (2.0.2)
- Locking psr/http-client (1.0.3)
- Locking psr/http-factory (1.0.2)
- Locking psr/http-message (2.0)
- Locking psr/http-server-handler (1.0.2)
- Locking psr/http-server-middleware (1.0.2)
- Locking psr/log (3.0.0)
- Locking sabre/dav (4.6.0)
- Locking sabre/event (5.1.4)
- Locking sabre/http (5.1.10)
- Locking sabre/uri (2.3.3)
- Locking sabre/vobject (4.5.4)
- Locking sabre/xml (2.2.6)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 92 installs, 0 updates, 0 removals
- Installing horde/horde-installer-plugin (v2.5.5): Extracting archive
- Installing horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Util
- Installing horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/Translation
- Installing horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Exception
- Installing horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Compress_Fast
- Installing horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cache
- Installing horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Wrapper
- Installing horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Support
- Installing psr/log (3.0.0): Extracting archive
- Installing php-extended/polyfill-php80-stringable (1.2.9): Extracting archive
- Installing horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Constraint
- Installing horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9): Symlinking from /srv/git/horde/Log
- Installing psr/container (2.0.2): Extracting archive
- Installing horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11): Symlinking from /srv/git/horde/Injector
- Installing horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Controller
- Installing horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4): Symlinking from /srv/git/horde/Crypt_Blowfish
- Installing horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Url
- Installing horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Css_Parser
- Installing horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/CssMinify
- Installing horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Flowed
- Installing horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Secret
- Installing horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Idna
- Installing horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Text_Filter
- Installing horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Filter
- Installing horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Stream
- Installing horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Mime
- Installing horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mail
- Installing horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/ListHeaders
- Installing horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Nls
- Installing horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Date
- Installing horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Icalendar
- Installing horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Browser
- Installing horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Data
- Installing horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/HashTable
- Installing horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Db
- Installing horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/History
- Installing horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/View
- Installing horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Vfs
- Installing horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Tree
- Installing horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Token
- Installing horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Diff
- Installing horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Serialize
- Installing horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Xml_Element
- Installing horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Group
- Installing horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Perms
- Installing sabre/uri (2.3.3): Extracting archive
- Installing sabre/xml (2.2.6): Extracting archive
- Installing sabre/vobject (4.5.4): Extracting archive
- Installing sabre/event (5.1.4): Extracting archive
- Installing sabre/http (5.1.10): Extracting archive
- Installing sabre/dav (4.6.0): Extracting archive
- Installing psr/http-message (2.0): Extracting archive
- Installing psr/http-factory (1.0.2): Extracting archive
- Installing psr/http-client (1.0.3): Extracting archive
- Installing horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Http
- Installing pear/pear (v1.10.14): Extracting archive
- Installing pear/xml_util (v1.4.5): Extracting archive
- Installing pear/structures_graph (v1.1.1): Extracting archive
- Installing pear/console_getopt (v1.4.3): Extracting archive
- Installing pear/archive_tar (1.4.14): Extracting archive
- Installing horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Template
- Installing horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Share
- Installing horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/SessionHandler
- Installing horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Prefs
- Installing horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Pack
- Installing horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Notification
- Installing horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Compress
- Installing horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mime_Viewer
- Installing horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/LoginTasks
- Installing horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Lock
- Installing horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/JavascriptMinify
- Installing psr/http-server-handler (1.0.2): Extracting archive
- Installing psr/http-server-middleware (1.0.2): Extracting archive
- Installing horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2): Symlinking from /srv/git/horde/Http_Server
- Installing horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Cli
- Installing horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Autoloader
- Installing horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Auth
- Installing horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Alarm
- Installing horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17): Symlinking from /srv/git/horde/Core
- Installing horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Dav
- Installing horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Rpc
- Installing horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Image
- Installing horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Form
- Installing horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Argv
- Installing horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7): Symlinking from /srv/git/horde/base
- Installing horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Yaml
- Installing horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cli_Modular
- Installing horde/hordectl (v1.0.0alpha4): Extracting archive
- Installing horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Routes
- Installing horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Test
- Installing pear/console_color2 (0.1.2): Extracting archive
- Installing pear/console_table (v1.3.1): Extracting archive
81 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files
Applying /presets for absent files in /var/config
Looking for registry snippets from apps
Writing app configs to /var/config dir
Linking app configs to /web Dir
Linking javascript tree to /web/js
Linking themes tree to /web/themes
1 package you are using is looking for funding.
Use the `composer fund` command to find out more!
This composer environment works just like a regular installation. When you install turba, kronolith or passwd through composer, it will end up linking these apps and their library dependencies from the development tree.
Composer provides an option to copy files from the repositories rather than link the files. This would allow creating archives with artifacts for distribution packaging. The horde-components tool does not yet provide a switch to generate the necessary tweak to the repository files.
By now you can enter into your browser: http://base.php82.horde and get to a message saying Please create a /var/www/base.php82.horde/var/config/horde/conf.php file and then run ‘composer horde-reconfigure’ to activate Horde
Starting with a default horde config
Start with a default horde config
cp -ar /var/www/base.php82.horde/vendor/horde/horde/config/conf.php.dist /var/www/base.php82.horde/var/config/horde/conf.php
cd /var/www/base.php82.horde/
composer horde-reconfigure
It’s been a while. It’s been much too long actually. Let’s look forward though. I recently started to onboard into FOSS development again. People have been asking for PHP 8.2 support with Horde but capacity to deliver that was very limited. But we’re getting there.
I used the “Maintaina” fork to deliver fixed and upgraded customized versions of Horde beyond what was allowed in Horde’s master branch. It’s time to move forward though. Maintaina used to target PHP 7.4 to PHP 8.1, composer2 based install from a custom satis repo and some quite invasive changes to CalDAV/CardDAV support. Most prominently, Maintaina introduced library level compatibility with several PSRs (container, http middleware, logging) and the all new horde/http_server component.
At the moment I am importing the FRAMEWORK_6_0 branches and the alpha releases based on this branch to horde’s upstream repository. So far I have imported enough libraries to make the Horde Base application barely install from packagist.
Get the current state of affairs
Your magic carpet is:
composer create-project horde/bundle .
Originally I wanted to mimic maintaina’s setup with a separate satis server as a QA stage before release to packagist. I re-considered and dropped the separate satis server. Every update of the development branches and every tagged release is consumed via packagist. I will have to fix some of my tools and workflows to reflect that. The tagged alpha releases still use maintaina’s satis server. The upcoming releases won’t.
Onboarding Procedure
I leave the FRAMEWORK_5_2 and master branches mostly untouched. FRAMEWORK_6_0 is the new default branch on github for anything I am handling. I only edit other branches if they block packagist’s import. Usually I also rebase FRAMEWORK_6_0 on any latest commits of master, but in some cases I only cherry-picked from master branch. In some cases, some entries of the changelog between 2022 late and now (october 2023) might be missing. Pull Requests, bug reports and patches welcome.
What’s next
Before I move forward with the actual applications, I want to make sure the necessary infrastructure is in place. I need to fix some aspects of the FRAMEWORK_6 version of composer. The workflow files in each repo need some review, too. Does PHPUnit and PHPStan still pass? Can we improve management of 130+ repo’s workflows?
Finally https://dev.horde.org should index from packagist, not from the satis server we originally planned to use.
What will become of maintaina repos?
The maintaina repos have had direct contributions from some trusted maintainers from the company I used to work for. They service some customers out of these repos and the related SATIS server so I won’t actively break it. However, with the move to Horde upstream, maintaina has served its purpose for me and I will not actively support the fork anymore. I suggest once everything is ported to upstream, maintaina should be archived. I will need to consult other stakeholders of this fork and the satis server.
Maintenance cost
Over the lifetime of the fork I have explored and applied various strategies for keeping the effort in check. Still, a fork of 100+ repos and the accompanying infrastructure for testing and deploying is a major burden which detracts from actually developing and maintaining code. I am glad I can save on this now and actually contribute to Horde directly in a way that doesn’t slow down activity too much. We now have the chance to speed up the cycle of feedback and releases. I hope this attracts some occassional and regular contributors.
ssh-agent does not persist over sessions in WSL2 with current openSUSE Linux and other common distributions. Linux native bashrc solutions don’t work here. Let’s use keychain instead
zypper in keychain
Then let’s add some snippet to .bashrc and remove any eval ssh-agent lines
vim .bashrc
# Once per key to load by default
/usr/bin/keychain -q --nogui $HOME/.ssh/id_rsa
/usr/bin/keychain -q --nogui $HOME/.ssh/id_host_www.ralf-lang.de
# Only once
source $HOME/.keychain/$HOST-sh
Then open a new window. It will ask you to enter your passphrase once. Enter any additional windows, it won’t ask you
My 2023 started in a bad way. I did not attend to any non-work twitter, email or other communication and FOSS work to deal with that. Don’t worry, I will catch up soon. Sorry for any delays and inconveniences.
