Making plans is great but it’s just words. How are things going? Clearly moving towards goals but in unexpected ways.
Trains don’t only run on railroad tracks but supposedly on schedule. Commuters shall know every stop, arrival and departure, the order of cars and their position on the platform. If trains regularly arrive delayed, at another platform, composed of different or differently ordered cars it is a source of frustration. If they skip a station or get relayed to another course, it’s even worse. Software projects, on the other hand, are expected to detour regularly in very similar ways but hopefully arrive at certain way points more or less on time. Sometimes they get unplanned additional cars along the way. Sometimes you have to replace the engine. It never gets boring.
What is the goal?
In January, I anticipated there were some unknowns ahead but I did not detail how to tackle this.
Improve JMRI‘s support for the Märklin MCAN protocol and particularly the Mobile Station 2, Central Station 3 and Can-Digital-Bahn products. This might become a bit controversial.
The Märklin Hardware Throttles (Mobile Station 2) implement hot plugging and negotiate which one acts as the primary. They send a regular PING command announcing their type and serial and expect a PONG response from other devices on the bus. The most senior serial number is agreed to be the primary throttle even if the other one was the primary or single throttle until now. The primary offers its roster to the other throttles. If higher end Central Station systems are on the bus, one of the central station will always be the primary device and the throttles all behave a little different. This allows JMRI to passively listen to traffic and detect bus devices or actively send a PING and collect the PONGs.
The Can-Digital-Bahn devices have a similar PING/PONG mechanism used by their configuration utilities. Unfortunately there are different configuration utilities for different generations of their sensors, relays, turnout controllers, light controllers, switchboard components, mixed devices. These utilities are closed source Windows programs and you need different versions of the program for different versions of the components. Unfortunately, not all of the programs properly run on recent Windows systems.
What if I just had to push a button in the web browser and see all my MCAN throttles and devices in a table, loading the right configuration screen for each of them? Sounds much better, but how to achieve that?
Moving through the stack
To send the PING messages and receive the PONG messages a program needs to connect to the MCAN bus. This can be achieved by connecting the CC-Schnitte interface to the USB port of the computer or connecting to a Central Station device over TCP/IP. Both methods expose the raw MCAN bus to the software. Both options are already implemented in JMRI.
Next the necessary bus messages and response need to be added to the software. The MCAN PING command is already implemented but JMRI doesn’t implement the desired reactions to the answers. For can-digitial-bahn messages, support is still missing.
To use the found devices, a program needs to memorize some representation of them. JMRI currently does not have a generic way of handling external bus devices as such. Throttles and sensors are tracked in their specific roles, covering their common aspects as throttles or as sensors. For CBUS type connections, there is a node manager which looks very similar to what is needed for MCAN but the UI code and the table format probably deviates a bit. This is where it becomes a bit complicated.
Users need some kind of GUI to interact with found devices. There are many reasons to start out with a browser based UI instead of building it in JMRI’s native Java GUI. The browser based ui just works on any tablet or laptop connected to the network. It’s well decoupled from JMRI’s core. It can be styled or completely recomposed as needed without touching JMRI itself. JMRI does not need to bundle variants for different use cases but they can be installed separately, developed on their own schedule. JMRI only needs to expose the necessary interface protocol which also drives other parts of the browser based UI.
JMRI’s interface protocol is composed from JSON messages between the browser client and the JMRI server. These can be sent over the HTTP server port or through a separate WebSocket implementation. The latter provides better performance and less overhead.
JMRI’s WebSockets and HTTPS API
This is what I ended up doing. To familiarize myself with the API I built a simple demo use case to manipulate the displayed railroad company name. Previously this was only possible through the preferences screen in the native Java GUI. I ended up not integrating it in the regular browser based GUI but implement a separate demo page. JMRI core developers pointed out that special care is needed and unauthenticated calls to the API must not actually persist the changed configuration into JMRI.
Obviously this would not work for actually handling hardware. While JMRI already has a permissions and user authentication system, it is fairly new and does not cover the actual API messages. So my next step would be to implement the necessary messages in the JSON WebSocket API for retrieving an authentication token. This token would then be used to authorize further WebSocket calls which actually change something in JMRI. These new calls need to carry the authorization token as part of the data.
I plan to detail this development in a separate article later this month as I move through the process.
Seems like I should be getting somewhere
This free time project is a great adventure into the unknown between where I am and where I want to be next. In some ways this is similar to professional work. I commit to goals and target dates and I have a very clear understanding of next week and some ideas about the week after. Beyond that the way points and schedule dates become rarer.
It’s sometimes very challenging to tell project managers that there won’t be many super detailed milestones beyond the horizon of immediate next steps. Gladly, at work I have very smart project managers who know when to trust me and when to challenge me. Sadly, in free time I have a very harsh and unforgiving project manager who often won’t take no for an answer.
Looking back on 2024, it was a challenging, demanding and sometimes crazy year. But let’s start with athe good things in life: There was so much joy to be had with the family, so much to learn and discover at work, so much progress on personal development and discovering new approaches to things that just won’t go away. All these precious moments I would not want to miss. It would be wrong to just skip over this without this little line of appreciation. I benefit very much from my family, friends and colleagues and I hope to give back a little bit of joy and a helping hand wherever I can. Thank you all.
But if you know me: I am ambitious, I am loyal to stuff I started, sometimes stubborn and I am an open source software guy. From this perspective, things were not going so well. While I am used to full schedules and stressful timelines at work, I had to re-prioritize my free time harshly in the middle of the first quarter. This got a lot of private projects into an undeclared hiatus and ultimately frozen for the rest of the year. It came unexpected and initially I did not grasp the extent of what was happening. To give you an idea, I even stopped reading mailing lists and most of my private inbox for a while. I did not plan any public talks and only visited FOSS events if they were job related. By coincidence, the main laptop with open source work setups also went defunct and I did not bother to replace it for several months.
If you feel you get into a similar situation, I can only advise you to take no time for regrets. Do what needs to be done and cut what you’d like to do on top until you feel ready again. Watch for the metrics: Do you start reading again? Is it more pages of fiction and non-fiction than last month? Do you take time for the occassional board or computer game? Can you make time again for you hobby, club or sports activities? Then maybe you’re set to come back to free time FOSS development – if you still like it. Don’t feel you need or must or have to. Maybe take another two weeks to question yourself before you start over. Nothing is won with spending hours in an exhausted or half detracted state.
That said, I am very positive to slowly bring everything back to a more happy status one at a time. A new FOSS work setup is available. Some of 2024’s learnings may help me to achieve more while still on a limited time budget. I am catching up with mails and yes, your pull requests will be handled. There’s also some clerical work to be done for my pet projects. It’s too early to announce a specific timeline or commitment and sometimes opportunity trumps priority. Let’s see how that works.
So what’s on the agenda?
Horde 6 Framework and main apps need to go to “stable”. I don’t want any ambiguity anymore. Make it run on the recent and upcoming PHP and try not to break it for any officially maintained release +1
PoC browser & thunderbird plugin with some very limited initial use case. Maybe a sidebar for Horde Mnemo Notes. Let’s gather some experience before making big plans.
Quality of life for aspiring developers: Documentation, test setup, reference setup
Modern authentication focus: I’ve done a custom 2FA/TOTP solution several years ago and I have been meaning to port it to Horde’s core for several years. The authentication stack probably needs to be revisited to make it a natural fit. I’d like to keep it nicely split into an interface part and the actual implementation so exchanging the TOTP app for some QR Code Challenge on your device, email confirmation or similar should be possible without modifying the core code.
Continue porting some traditional pages to the PSR-7 standard based controller framework
Be more active again in PHP-FIG discussions and practical maintenance work on the PSR/PER related repositories
Having loads of fun and learning
Improve JMRI‘s support for the Märklin MCAN protocol and particularly the Mobile Station 2, Central Station 3 and Can-Digital-Bahn products. This might become a bit controversial. The current implementation is very much centered on fulfilling JMRI’s minimal requirements for a connection and throttle implementation and much code is a straight port from the TAMS implementation. I’d rather build a java library with internal cohesion and a good match to the bus protocol and the device philosophy. I have read into Bob’s more specialized TrainControl and Frans Jacob’s standalone program. I appreciate how much more cohesive and maintainable their implementation is. They achieve this by not letting an external frame dictate their internal structure and mental model. Of course this will need some glue to match it with JMRI’s wider logic but it’s also a great opportunity to simplify unit testing.
That’s already a tall order and I am sure more ideas and desires pop up along the way. No explicit priority, no promises, no commitment.