bookmark_borderThe state of horde composer deployments

In early 2019 I first wrote a few lines on migrating horde to composer based setups.

Even though I was too occupied with other business for several months, I have something to show and it is time to review what we have and what we don’t have.

The good news is, it really runs as far as I am concerned. For the mainstream apps, the Web UI works correctly as well as caldav/carddav access, handling of db schema migrations, some critical commandline scripts. There is still a lot of scripts to check and possibly fix.

The Horde Components helper has been modified to be run as a part of a horde deployment or standalone. It can also still be run from git-tools. I added some code to git-tools to improve the way it wraps horde-components. However, most things git-tools provides are no longer that interesting to the casual developer or to deploying horde master tree installations.

I also improved the horde Controller/Routes core. It is now more successful in detecting which app handles a request. Also, a new option for unauthenticated routes and for “API style” auth handling has been included. This enables REST interfaces.

I finally did a small proof of concept app which is completely based on the controller/routes core and does not have a single traditional “page” file. Maybe this should be provided as an alternative skeleton.

How to run your own composer based install

Running a new horde install for development or CI purpose is as easy as:

cd your/web/root/
git clone https://github.com/maintaina-com/horde-deployment.git .
composer install
cp web/horde/config/conf.php.dist web/horde/config/conf.php

For using a database, provide a config with a db backend instead.
You might also want to run web/horde/bin/horde-db-migrate.

I intend to improve the installer a little to detect if pre-made configs have been put into expected locations of the deployment and use them.

Why you should not use this in production

This deployment comes “as is”. It is still dependent on a satis server not controlled by the horde organisation, running a no-warranties fork of the unreleased git master code. If you run this in production, you need to do a lot of QA for your specific use case. Patches welcome. You may also encounter features not part of official horde releases because they are incomplete, lacking tests or else.

I am still trying to get this upstream. Dealing with a downsteam horde distribution of 150+ repos is tedious work. I’d rather invest time in actual software development. In case you really need to, contact me about commercial support options.

bookmark_borderPEAR down – Taking Horde to Composer

Since Horde 4, the Horde ecosystem heavily relied on the PEAR infrastructure. Sadly, this infrastructure is in bad health. It’s time to add alternatives.

Everybody has noticed the recent PEAR break-in.

A security breach has been found on the http://pear.php.net webserver, with a tainted go-pear.phar discovered. The PEAR website itself has been disabled until a known clean site can be rebuilt. A more detailed announcement will be on the PEAR Blog once it’s back online. If you have downloaded this go-pear.phar in the past six months, you should get a new copy of the same release version from GitHub (pear/pearweb_phars) and compare file hashes. If different, you may have the infected file.

While I am writing these lines, pear.php.net is down. Retrieval links for individual pear packages are down. Installation of pear packages is still possible from private mirrors or linux software distribution packages (openSUSE, Debian, Ubuntu). Separate pear servers like pear.horde.org are not directly affected. However, a lot of pear software relies on one or many libraries from pear.php.net – it’s a tough situation. A lot of software projects have moved on to composer, an alternative solution to dependency distribution. However, some composer projects have dependency on PEAR channels.

I am currently submitting some changes to Horde upstream to make Horde libs (both released and from git) more usable from composer projects.
Short-term goal is making use of some highlight libraries easier in other contexts. For example, Horde_ActiveSync and Horde_Mail, Horde_Smtp, Horde_Imap_Client are really shiny. I use Horde_Date so much I even introduced it in some non-horde software – even though most functionality is also somewhere in php native classes.

The ultimate goal however is to enable horde groupware installations out of composer. This requires more work to be done. There are several issues.

  • The db migration tool checks for some pear path settings during runtime https://github.com/horde/Core/pull/2 Most likely there are other code paths which need to be addressed.
  • Horde Libraries should not be web readable but horde apps should be in a web accessible structure. Traditionally, they are installed below the base application (“horde dir”) but they can also be installed to separate dirs.
  • Some libraries like Horde_Core contain files like javascript packages which need to be moved or linked to a location inside another package. Traditionally, this is handled either by the “git-tools” tool linking the code directory to a separate web directory or by pear placing various parts of the package to different root paths. Composer doesn’t have that out of the box.

Horde already has been generating composer manifest files for quite a while. Unfortunately, they were thin wrappers around the existing pear channel. The original generator even took all package information from the pear manifest file (package.xml) and converted it. Which means, it relied on a working pear installation. I wrote an alternative implementation which directly converts from .horde.yml to composer.json – Calling the packages by their composer-native names. As horde packages have not been released on packagist yet, the composer manifest also includes repository links to the relevant git repository. This should later be disabled for releases and only turned on in master/head scenarios. Releases should be pulled from packagist authority, which is much faster and less reliant on existing repository layouts. https://github.com/horde/components/pull/3

To address the open points, composer needs to be amended. I currently generate the manifests using package types “horde-library” and “horde-application” – I also added a package type “horde-theme” for which no precedent exists yet. Composer doesn’t understand these types unless one adds an installer plugin https://github.com/maintaina-com/installers. Once completed and accepted, this should be upstreamed into composer/installers. The plugin currently handles installing apps to appropriate places rather than /vendor/ – however, I think we should avoid having a super-special case “horde-base” and default to installing apps directly below the project dir. Horde base should also live on the same hierarchy. This needs some additional tools and autoconfiguration to make it convenient. Still much way to go.

That said, I don’t think pear support should be dropped anytime soon. It’s the most sensible way for distribution packaging php software. As long as we can bear the cost involved in keeping it up, we should try.

bookmark_borderCurrent (10/2018) Tumbleweed on Raspberry Pi 1

Hallo,

I just had a little struggle getting the current tumbleweed to run on the original Raspberry Pi (first generation, though the revision with larger RAM).

Just in case this helps anybody: I did not have any luck with a fresh openSUSE Tumbleweed image of one of the current arm6 builds. Don’t know why.

Here’s what I did:
– Download a pretty old known-good OpenSUSE 13.1 built by Bernhard Wiedemann

http://www.zq1.de/bernhard/linux/opensuse/raspberrypi-opensuse-latest.img.xz

Unzip, dump it to SD Card

xz -d raspberrypi-opensuse-latest.img.xz
dd if=raspberrypi-opensuse-latest.img of=/dev/mmcblk0 bs=8M

Boot up, change to text console (CTRL + ALT + F2)
Log In (root/linux)

Change Password (passwd)

nano /etc/zypp/repos.d/oss131.repo
Change baseurl line to
baseurl=http:/download.opensuse.org/ports/armv6hl/tumbleweed/repo/oss/
Save and get out (CTRL+X, Y)

#Resize partition and FS as this build won’t do:
# Adjust to more if your card is larger or to less if you need a more advanced partitioning scheme

parted resize 3 16G

resize2fs /dev/mmcblk0p3

zypper ref

zypper up gzip rpm

zypper dup –download in-advance

#(super conservative, get all needed packages first) – This is going to take quite a while

reboot, power cycle

Note that you will end up with a system booting into X11 login. You should probably change the default systemd target and maybe also get rid of some software. And you really don’t want a server with ssh password “linux”, so better don’t skip changing the PW

 

bookmark_borderhorde trustr – A new horde CA app step by step

Trustr is my current project to create a simple certificate management app.
I decided that it is just about the right scope to demonstrate a few things about application development in Horde 5.

I have not made any research if the name is already occupied by some other software. Should any problems arise, please contact me and we will find a good solution. I just wanted to start without losing much time on unrelated issues.

My goals as of now:

– Keep everything neat, testable, fairly decoupled and reusable. The core logic should be exportable to a separate library without much change. There won’t be any class of static shortcut methods pulling stuff out of nowhere. Config and registry are only accessed at select points, never in the deeper layers.
– Provide a CLI using Horde_Cli and Horde_Cli_Application (modeled after the backup tool in horde base git)
– Store to relational database using Horde_Db and Horde_Rdo for abstraction
– Use php openssl extension for certificate actions, but design with future options in mind
– Rely on magic openssl defaults as little as possible
– Use conf.xml / conf.php for any global defaults
– Show how to use the inter-app API (reusable for xml-rpc and json-rpc)
– Showcase an approach to REST danabol ds in Horde (experimental)

The app is intended as a resource provider. The UI is NOT a top priority. However, I am currently toying around with a Flux-like design in some unrelated larger project and I may or may not try some ideas later on.

Initial Steps: Creating the working environment

I set up a new horde development container using the horde tumbleweed image from Open Build Service and a docker compose file from my colleague Florian Frank. Please mind both are WIP and improve-as-needed projects.


git clone https://github.com/FrankFlorian/hordeOnTumbelweed.git
cd hordeOnTumbelweed
docker-compose -f docker-compose.yml up


This yields a running horde instance on localhost and a database container.
I needed to perform a little manual setup in the web admin ui to get the DB to run and create all default horde schemas.

Next I entered the developer container with a shell
docker exec -it hordeOnTumbelWeed_php_1 bash

There are other ways to work with a container but that’s what I did.

 

Creating a  skeleton app

The container comes with a fairly complete horde git checkout in /srv/git/horde and a clone of the horde git tools in /srv/git/git-tools

A new skeleton app can be created using

horde-git-tools dev new --author "Ralf Lang <lastname@b1-systems.de>" --app-name trustr

The new app needs to be linked to the web directory using

horde-git-tools dev install

Also, a registry entry needs to be created by putting a little file into /srv/git/horde/base/config/registry.d

cat trustr-registry.d.php

<?php
// Copy this example snipped to horde/registry.d
$this->applications['trustr'] = array(
'name' => _('Certificates'),
'provides' => array('certificates')
);

 

This makes the new app show up in the admin menu. To actually use it and make it appear in topbar, you also need to go to /admin/config and create the config file for this app. Even though the settings don’t actually mean anything by now, the file must be present.

I hope to follow up soon with articles on the architecture and sub systems of the little app.

bookmark_borderBoneyard – a barebone horde “dynamic view” app based on “skeleton”.

Boneyard – a barebone horde “dynamic view” app based on “skeleton”.

In this article, I will show you some minimal setup for a “horde5 dynamic view” application as demonstrated by hermes time tracking and kronolith calendar

Pre-requisite
We have a working git checkout of a 5.2 or master installation of horde with some authentication and prefs backend working and the migrations inplace.
If your setup did not involve editing install_dev.conf, you probably have something else and I cannot guarantee this walkthrough will work for you without adopting some parts.

Let’s generate a fresh application called boneyard

maintaina:/srv/git/horde5-webmail/horde # php framework/bin/horde-generate-module boneyard "Ralf Lang "
Started new Module in /srv/git/horde5-webmail/horde/boneyard!
Register the new Module with a file in the config/registry.d directory:

<?php
$this->applications['boneyard'] = array('name' => _("Boneyard"));

We put a file with this oneliner into the directory as advised

maintaina:/srv/git/horde5-webmail/horde # vim horde/config/registry.d/boneyard.php

Now let’s re-run the script generating the links for the git checkout installation

maintaina:/srv/git/horde5-webmail/horde # php framework/bin/install_dev
EMPTYING old web directory /srv/www/vhosts.d/horde.ralf-lang.de

LINKING horde
Setting static directory permissions...
LINKING applications to web directory /srv/www/vhosts.d/horde.ralf-lang.de
LINKING sam
LINKING luxor
[.. snip ..]
LINKING pastie
LINKING ingo
LINKING boneyard
LINKING hvview
LINKING sesha
LINKING passwd
LINKING operator
LINKING nag
LINKING gollem
LINKING jonah
LINKING sueporter
LINKING ulaform

LINKING framework
[ INFO ] Source directory: /srv/git/horde5-webmail/horde/framework
[ INFO ] Framework destination directory:
/srv/www/vhosts.d/horde.ralf-lang.de/libs
[ INFO ] Horde directory: /srv/www/vhosts.d/horde.ralf-lang.de
[ INFO ] Create symbolic links: Yes

[ INFO ] Package(s) to install: ALL (129 packages)
[ INFO ] Installing package ActiveSync
[.. snip ..]
[ INFO ] Installing package xxhash

Now boneyard is set up in your web-accessible dir.
Let’s make the config dir web-writeable

chown wwwrun:www /srv/www/vhosts.d/horde.ralf-lang.de/boneyard/config

This is for SUSE – debian or redhat may have different user/group for the web server.

Next go to $yourdomain/admin/config/ the admin panel and generate the conf.php file by clicking on the “boneyard” entry and then the “create boneyard config” button.
At this point, we do not care about the actual contents of this config – the defaults are just fine.

If you only see “horde” and some library names, you most probably have not edited registry.local.php to contain something like:


<?php
// By default, applications are assumed to live within the base Horde
// directory (e.g. their fileroot/webroot will be automatically determined
// by appending the application name to Horde's 'fileroot'/'webroot' setting.
// If your applications live in a different base directory, defining these
// variables will change the default directory without the need to change
// every application's 'fileroot'/'webroot' settings.
$app_fileroot = '/srv/www/vhosts.d/horde.ralf-lang.de/';

Now “Boneyard” should appear in your horde topbar with some bogus buttons and content

Let’s create the structure of a “dynamic” application

* lib/Ajax.php – The Boneyard Ajax base class to load locale- and setting-dependent content into the browser’s javascript
* lib/Ajax/Application/Handler/Example.php – A handler for Ajax requests to load data from the server — we skip that for now
* lib/View/Sidebar.php – Boneyard_View_Sidebar – a sidebar for the dynamic view
* template/dynamic/sidebar.html.php – The template used by the sidebar view
* template/dynamic/index.inc – The main template of the dynamic view
* template/dynamic/example1.inc – One of our two example views in this demo
* template/dynamic/example2.inc – One of our two example views in this demo
* js/boneyard.js – The BoneyardCore object which contains the main click handler etc

We also need to touch the index.php file to enable the dynamic view and the lib/Application.php file to advertise that dynamic view exists.

See https://github.com/ralflang/horde-boneyard to view the code in detail.

bookmark_borderHorde_Rdo Many to Many relations and Horde DB Migrator

Many to Many relations btween to object types or table rows are usually saved to a database using a third table.

For example, if every server can have multiple services and each service can run on multiple computers, we need a third table to store the relations:

server table:
server_id | server_name
        1 | hoellenhund.internal.company.com
        2 | forellenfisch.internal.company.com
service table:
service_id | service_name
         1 | tomcat
         2 | dovecot
relation table:
service_id | server_id
         1 | 1
         2 | 2
         2 | 1

Horde’s ORM Layer Horde_Rdo supports creating, checking and changing such relations but it’s not very prominently documented.

Let’s look at an example.

First, we need to create the database schema. Note that the relations table has no autoincrement key, only the two columns used for lookup


/usr/share/php5/PEAR/www/horde/hvview/migration # cat 1_hvview_base_tables.php
<?php
/**
* Create Hvview base tables.
*
* Copyright 2015-2015 B1 Systems GmbH (http://www.b1-systems.de/)
*
* See the enclosed file COPYING for license information (GPL). If you
* did not receive this file, see http://www.horde.org/licenses/gpl.
*
* @author Ralf Lang
* @package Hvview
*/
class HvviewBaseTables extends Horde_Db_Migration_Base
{
/**
* Upgrade
*/
public function up()
{

$t = $this->createTable('hvview_technical_landscapes', array('autoincrementKey' => 'landscape_id'));
$t->column('landscape_name', 'string', array('limit' => 255, 'null' => false));
$t->column('period_id', 'integer', array('limit' => 11, 'null' => false));
$t->end();

$t = $this->createTable('hvview_resource_pools', array('autoincrementKey' => 'resource_pool_id'));
$t->column('pool_name', 'string', array('limit' => 255, 'null' => false));
$t->column('landscape_id', 'integer', array('limit' => 11, 'null' => false));
$t->column('period_id', 'integer', array('limit' => 11, 'null' => false));
$t->end();

$t = $this->createTable('hvview_hardware_pools', array('autoincrementKey' => 'hardware_pool_id'));
$t->column('pool_name', 'string', array('limit' => 255, 'null' => false));
$t->column('landscape_id', 'integer', array('limit' => 11, 'null' => false)); /* possibly redundant, but may speed up things */
$t->column('period_id', 'integer', array('limit' => 11, 'null' => false));
$t->end();

/*Relations table*/
$t = $this->createTable('hvview_rp_hwps', array('autoincrementKey' => false));
$t->column('resource_pool_id', 'integer', array('limit' => 11, 'null' => false));
$t->column('hardware_pool_id', 'integer', array('limit' => 11, 'null' => false));
$t->end();

$t = $this->createTable('hvview_periods', array('autoincrementKey' => 'period_id'));
$t->column('period_ts', 'integer', array('limit' => 11, 'null' => false));
$t->end();

/* We collapse hypervisor and blade server objects into one for now - let`s see if this scales well */
$t = $this->createTable('hvview_servers', array('autoincrementKey' => 'server_id'));
$t->column('period_id', 'integer', array('limit' => 11, 'null' => false));
$t->column('hardware_pool_id', 'integer', array('limit' => 11, 'null' => false));
$t->column('hostname', 'string', array('limit' => 100, 'null' => false));
$t->column('state', 'string', array('limit' => 20, 'null' => true));
$t->column('os_release', 'string', array('limit' => 20, 'null' => true));
$t->column('comment', 'string', array('limit' => 255, 'null' => true));
$t->column('hv_free_vcpu', 'integer', array('limit' => 11, 'null' => true));
$t->column('hv_free_memory', 'integer', array('limit' => 11, 'null' => true));
$t->column('hv_free_disk', 'integer', array('limit' => 11, 'null' => true));
$t->column('hv_total_vcpu', 'integer', array('limit' => 11, 'null' => true));
$t->column('hv_total_memory', 'integer', array('limit' => 11, 'null' => true));
$t->column('hv_excluded', 'integer', array('limit' => 1, 'null' => true));
$t->column('hv_vm_count', 'integer', array('limit' => 3, 'null' => true));
$t->end();

// Indices not before we have an idea which of them we need most
// $this->addIndex('hvview_items', array('item_owner'));

}

/**
* Downgrade
*/
public function down()
{
$this->dropTable('hvview_technical_landscapes');
$this->dropTable('hvview_resource_pools');
$this->dropTable('hvview_hardware_pools');
$this->dropTable('hvview_periods');
$this->dropTable('hvview_servers');
$this->dropTable('hvview_rp_hwps');
}
}

The relations are defined in a Horde_Rdo_Mapper class which also knows how to spawn objects from the rows.

The Objects

/usr/share/php5/PEAR/www/horde/hvview/lib/Entity # cat ResourcePool.php 
<?php

class Hvview_Entity_ResourcePool extends Horde_Rdo_Base {
}

/usr/share/php5/PEAR/www/horde/hvview/lib/Entity # cat HardwarePool.php 

The Mappers:

/usr/share/php5/PEAR/www/horde/hvview/lib/Entity # cat ResourcePoolMapper.php 
<?php
class Hvview_Entity_ResourcePoolMapper extends Horde_Rdo_Mapper
{
    /**
     * Inflector doesn't support Horde-style tables yet
     */
    protected $_classname = 'Hvview_Entity_ResourcePool';
    protected $_table = 'hvview_resource_pools';
    protected $_lazyRelationships = array(
             'hwps' => array('type' => Horde_Rdo::MANY_TO_MANY,
                          'through' => 'hvview_rp_hwps',
                          'mapper' => 'Hvview_Entity_HardwarePoolMapper')
    
            );

}

/usr/share/php5/PEAR/www/horde/hvview/lib/Entity # cat HardwarePoolMapper.php 
<?php
class Hvview_Entity_HardwarePoolMapper extends Horde_Rdo_Mapper
{
    /**
     * Inflector doesn't support Horde-style tables yet
     */
    protected $_classname = 'Hvview_Entity_HardwarePool';
    protected $_table = 'hvview_hardware_pools';
    protected $_lazyRelationships = array(
             'rps' => array('type' => Horde_Rdo::MANY_TO_MANY,
                          'through' => 'hvview_rp_hwps',
                          'mapper' => 'Hvview_Entity_ResourcePoolMapper')
    
            );

}

The relation is defined in both direction and only loaded on-demand ("lazy") as opposed to upfront when the item is created from the database rows.
Now let's fetch two items and link them:

You can do this through the mapper or through one of the two partners

Adding a relation to an object using the object

// $rm is a ResourcePoolMapper instance
// $hm is a HardwarePoolMapper instance
$rp = $rm->findOne(); // In reality, you would not pick a random item but add some criteria
$hwp = $hm->findOne();
$rp->addRelation('hwps', $hwp);

Adding a relation to an object using the mapper

// $rm is a ResourcePoolMapper instance
// $hm is a HardwarePoolMapper instance
$rp = $rm->findOne(); // In reality, you would not pick a random item but add some criteria
$hwp = $hm->findOne();
$rm->addRelation('hwps', $rp, $hwp);

bookmark_borderThimbleweed Park angekündigt

Ron Gilbert, der Mann, der uns Maniac Mansion, Monkey Island (1+2) und Total Annihilation brachte, hat zusammen mit Gary Winnick die Entwicklung eines klassischen Point&Click-Adventures begonnen. Thimbleweed Park wird für PC, Android und iOS erscheinen und (optionale) Sprachausgabe sowie Übersetzung von Boris Schneider-Johne, der schon Monkey Island übersetzte. Mitte Juni 2016 soll methenolone enanthate das Spiel im Handel erscheinen, so der erste Zeitplan. Ich bin gespannt.Tummy control ondergoed hoge taille afslanken bodybuilding vetverbranding zweet workout slip voor mannen beste online – newchic anabole kopen home – steroïden bodybuilding training, steroïden bodybuilding online – portaal autoškola.

Vorbestellungen sind zu 25 US-$ möglich über Amazon und Paypal.

http://blog.thimbleweedpark.com/

bookmark_borderSara Golemon (Facebook) announces PHP Language Specification for OSCON 2014

For more than 10 years, PHP core developers repeatedly raised the topic of providing a formal language specification for PHP. Now a team of facebook employees has written such a specification. The spec document is currently only available as a preview chapter a preview chapter . PHP veteran Sara Golemon announced on the “PHP internals” list that the full document will be ready for O’Reilly’s OSCON 2014. Sara Golemon published the standard book on “Extending and Embedding PHP” some years ago and now works for Facebook’s own PHP implementation HHVM. The PHP spec defines PHP version 5.6 in about 200 pages and contains all the odd and obscure quirks of the language core. Facebook’s own HHVM aims to be as close to the spec as possible. Currently, PHP developers discuss how amending the spec can become a mandatory part of the language development process. Though some are sceptic that all developers will embrace the change in the process, everybody on the list was happy to have the new document.

Software Architect Stas Malyshev:

Thank you Sara and Facebook team for doing something we’ve been talking
about for more than a decade and before that nobody actually attempting
to do. I think it is a great development and I hope to see the first
version soon.

http://dl.hhvm.com/resources/PHPSpec-SneakPeak.pdf

bookmark_borderSara Golemon (Facebook) kündigt PHP Language Specification auf OSCON 2014 an

Seit über 10 Jahren bringen immer wieder einige der PHP-Sprachentwickler den Plan an, eine formale Spezifikation für den Sprachkern bereitzustellen. Ein Team bei Facebook hat das nun getan. Die Spezifikation, die bisher nur als Vorschau vorliegt, wurde von Sara Golemon auf der Entwickler-Liste angekündigt und soll auf der OSCON 2014 vorgestellt werden. Sara Golemon veröffentlichte schon vor einigen Jahren ein Standardwerk über die Entwicklung von PHP-Erweiterungsmodulen und arbeitet mittlerweile an Facebooks eigener PHP-Version HHVM.

Das rund 200 Seiten starke Dokument orientiert sich an der PHP-Version 5.6 und enthält auch obskure Verhaltensweisen des PHP-Sprachkerns in seltenen Randfällen. Die Facebook-eigene PHP-Version HHVM soll sich möglichst eng an diese Vorgaben halten.

Die PHP-Community berät derzeit, wie sie die Fortschreibung der Spezifikation in den Entwicklungsprozess einbinden kann. Die Ankündigung wurde mit viel Begeisterung aufgenommen.

Software-Architekt Stas Malyshev:

Thank you Sara and Facebook team for doing something we’ve been talking
about for more than a decade and before that nobody actually attempting
to do. I think it is a great development and I hope to see the first
version soon.

http://dl.hhvm.com/resources/PHPSpec-SneakPeak.pdf