Ungathered Thoughts

Contents may differ

2026-08-31T00:00:00Z

WAHL product guide, advice that contents may differ from documentation in twenty-seven languages.

A gadget for clutter-filled homes

2026-08-31T00:00:00Z

Advertisement, Google News Showcase, 2026.

Mesh in Ōtepoti

2026-04-27T00:00:00Z

Hi Chris do you have a purchase link for that repeater? I'm looking to get one.

Yes I do, but I think Ali Express URLs are too long for Meshcore messages. So ... here we go!

Liberton Repeater - SenseCAP Solar Node P1 Pro

I got a Seeed SenseCap Solar Node P1 Pro. The name is a MOUTHFUL but the "Pro" is significant - the non-Pro version lacks batteries as well as GPS. (The batteries were convenient, the node is fixed so GPS less useful to me.)

I got mine from the Seeed store, I was holding off buying (as shipping to Aotearoa was hugely expensive) until they ran a Boxing Day (?) free global shipping sale and I jumped on it. If I was buying another today I'd consider Ali Express as shipping from there can be cheaper.

It's a good little unit, seems to be holding its battery OK here in Liberton as the skies are getting greyer (April / Autumn). I am not an RF expert, but it mounted up fine on roof vent pipe here in Liberton, has run reliably on Meshtastic (initially) and now Meshcore, and doesn't ask me for maintenance. No issues with weather; I could perhaps improve its location for better connectivity.

Warning - the mounting bracket on mine has sharp edges - I should give it a run over with a file next time I get it down. Currently I'm wearing a fibreglass cast on my wrist and not going up ladders ...

I appreciate that lots of locals seem to be building the mesh nodes out from parts, which I respect - it feels more in keeping with the resilience goals of the network. This unit worked out of the box (both as Meshtastic and later Meshcore) and is an easy place to start.

Wio Tracker L1 Pro

I want to like this one - cute little screen you can read messages on, joystick control. It has a decent antenna! The battery lasts ok, a few days. The joystick control ... works, but it's not a keyboard, so really you're as good without unless you want a very retro texting experience.

The USB C connection doesn't really hold, meaning this one needs care to charge. With the better antenna it has greater reach, and I favoured it at first ... but then with charging issues forgot where I left it (plugged into a charger which did work) for a month. During that time I started using Meshcore more; for incidental (misplacement) reasons this device is the last one on Meshtastic.

Seeed Studio - Wio Tracker L1 Pro

Seeed T-1000e tracker card

This was an extra when I purchased the above two; I needed to get the order north of a certain price to clear the free shipping offer.

The battery on it seems to quit at about 50%. It has a magnetic charger connector which works great - but means you need to know where the charge cable is for that one device.

The form factor is good, it's a sealed unit, so this one depends on use with a mobile phone or paired device.

Rak Wismesh tag

This replaced the Seeed T-1000e. I got it from Ali Express. Lives in my bag, charges every few days, is my primary mesh node now. It has internal antenna only so you need to be within a km or two of a repeater - as for the T-1000e.

Uses the same charger connector as the T-1000e, so now I have two of those cables ...

Ali Express - Rak Wismesh tag

Heltec v4

I grabbed a couple of these to play with mesh networks from Linux systems. One is attached to a Raspberry Pi running HomeAssistant, the other is on another Pi running regular linux. Playing with using these to observe mesh (via the HASS unit) and message channels (the other).

Ali Express - Heltec V4 OLED LoRa 32 Dev-Board ESP32-S3 SX1262 WiFi Bluetooth PSRAM 16MB

These come with a handy box that'll do for a case, once you add a hole each for USB-C and antenna. The antenna shown here is not the one the device comes with.

More local mesh notes

For me the appeal of mesh is about local-first networks, community and resilience.
Discord is where lots of chat happens (yes, this is ironic; still, go first where community is)
- In Ōtepoti Dunedin, I recommend the Make Stuff Dunedin server (hop into #rf-stuff), and
- for Meshcore I recommend the Meshcore Discord (look in "regional-channels")
Dunedin (and I hear, most of NZ) is using the "NZ (narrow)" settings - broadcast settings are not compatible, ie if you and your neighbour use different settings, you won't see each other at all.
Some of these items may say "Meshtastic" in the product listing, I see Seeed are selling some pre-flashed with Meshcore now as well. To use Meshcore, obtain the device, then use the flasher at https://meshcore.io/flasher ... it's the same hardware.
Lots of folks on the mesh locally are HAM radio folks, so https://www.zl4aa.org.nz/ also worth checking if you're in Otago / Southland.

All of this is current at time of writing ... I just wanted to quickly share purchase links for the person who asked! None of these links are referral links (as best I can tell, anyway).

You can probably find me on Meshcore as ChrisB and on Meshtastic as CMB0, in Discord I'm @xurizaemon. I don't have a fancy callsign yet :)

Corrections welcome!

Aureuo Marker 24 palette

2026-04-04T00:00:00Z

A palette of the Aureuo Acrylic Marker 24pk. This set of pens is amazing, the colours are so thick and wet.

White

Light grey

Grey

Lime green

Bright Yellow

Dark Yellow

Light Brown

Copper

Orange

Pink

Coral

Crimson

Lilac

Magenta

Purple

Viridian

Dark Blue

Ultramarine Blue

Aquamarine

Bright Green

Green

Dark Green

Gold

Black

Aureuo-Marker-24.gpl

The Gordon Harris product page lists the following colours: Crimson, Dark Yellow, Green, Purple, Black, Grey, White, Viridian, Light Brown, Ultramarine Blue, Gold, Orange, Lilac, Lime Green, Dark Green, Pink, Light Grey, Magenta, Light Blue, Red, Dark Blue, Bright Greren, Bright Yellow.

I have a copper colour which is not in the Gordon Harris list (AFAICT), and one of the greys is metallic-silver. So I've mostly used names from that website, but swapped a few where I didn't feel they fit quite, hence Coral, Aquamarine.

The colour codes here are sampled from a set of swatches I made, pics taken with a ten year old iPhone. This page pretty much came of me trying to work out what the colour names were, and making a fun little palette output from some front matter data in Eleventy.

So, you should know about: SearXNG

2026-02-18T00:00:00Z

It's interesting to reflect how much we've come to depend on tools like search engines in our daily lives, and I guess I
could be grateful to Google for making its centrality so stark when they introduced their AI Overviews - sometimes on
point, and quite often outlandishly wrong.

For a couple of months I switched to Duck Duck Go, hoping for a replacement which worked like I was used to. But it
did not hit the same, I looked around for more options, and found SearXNG. I'm impressed.

My searches now hit several engines including Google, all of which might withdraw service at any time, and the results I get are an aggregation of what's returned.
There's no AI Overview presented at all - great! The results feel much better - more like search from a few years ago.
There's no ads. There's no ads. Glorious.
Results come from a number of search engines, so it's somewhat more resilient, but I don't think of it as an entirely dependable long-term solution.
If or when it breaks for some reason, or essential engines block access, I might have to find an alternative. Oh well.
It doesn't try to innovate in UI, I guess it's "conservative" in that it presents as a fairly ordinary search engine.
It's capable, it does the job, and it's far less annoying than Google had gotten.
It's open source, so you can review its internals, run it yourself, and contribute fixes or extend with search providers.

I'm happy to have found it, even if it doesn't last forever - nothing does. It's the best alternative I found.

Using SearXNG

You don't have to run SearXNG yourself: there are public instances you can use listed at https://searx.space/

Running SearXNG

I'm running an instance myself, behind SSL and basic auth so it's All Mine and Only Just Mine, and it's working great.

Instructions at https://docs.searxng.org/ for the method that suits you. You don't really care how I run it :)

Why run a copy myself? I don't have to wonder who's running the instance I'm using, or what their privacy policy says,
or whether they might get bored of running it. My instance is private, so if I stop using it, I can turn it off.

Thoughts on organisational usage

For an organisation with a variety of internal systems, I think it has potential as an organisational search engine.
Imagine if your people could use a single search interface, and get results from both the internet at large and your
Sharepoint, wiki, or other systems. That seems like a great deal, even if it meant rolling your own plugin to support
whatever weird cheesy systems you needed to search.

Sandymount track

2026-02-16T00:00:00Z

We took a short walk along Sandymount Track to Sandfly Bay this weekend.

To test out a possible backpack I loaded it up and took it for the walk. It's a gentle track but a long steady (sandy) incline on the return.

There were lots of lepidoptera amongst the track and plants - we saw Coppers, a Magpie Moth, and a Red Admiral.

Gitlab CI Local

2026-02-15T00:00:00Z

I'm such a fan of firecow/gitlab-ci-local. It's a CLI tool which allows to execute Gitlab CI pipelines locally, without needing to push changes through Gitlab. For fast local development of pipelines, or targeted testing of jobs configured in pipelines, it's an amazing speed boost.

For example, in a local project I might have a job cypress, which perhaps depends on some prior build and install tasks. With gitlab-ci-local installed, from a CLI within the project directory I can execute -

gitlab-ci-local --needs cypress

to see the CI job and its dependencies run locally, in either docker or shell runners.

The tool tracks any files which are tracked by Git. You don't need to commit changes; it's sufficient that the files are tracked, and gitlab-ci-local will use the local versions of the files.

Artifacts from the CI jobs are preserved in /.gitlab-ci-local/artifacts by default, along with a handful of other useful directories which track output

If the job needs some additional CI variables set, I can pass --variable PATH=/some/path or add a /.gitlab-ci-local-variables.yml:

---
# CI_PIPELINE_SOURCE: merge_request_event
XDEBUG_CONFIG: client_host=192.168.178.56

gitlab-ci-local functions as a containerised testing environment with configuration syntax inherited from Gitlab CI. Since I'm already using that for most projects, adding gitlab-ci-local to the mix is fantastic - fast, no-push execution of targeted CI jobs, which I can push to Gitlab once I'm happy with the local tests.

Disk usage!

One thing to watch for with this tool is disk consumption via Docker volumes. When cleaning disk usage on my laptop, including deleting ~800 Docker volumes, 200 of which had names like gcl-{jobname}-{NNN}-(build|tmp) (eg gcl-cypress-126182-build, gcl-cypress-126182-tmp). Just normal Docker things really, but gitlab-ci-local was a bit of a multiplying factor ...

Save steps!

Useful detail to know: if you are changing the job "cypress", and it depends on the job "composer build"

You can run the job gitlab-ci-local --needs cypress first, to execute the build job

Then you can make changes to your code, and run just gitlab-ci-local cypress to skip the build job; if you're not changing composer.lock then the build job is skippable, and this lets you run pipelines again even faster than CI would.

drupal-ci-local

Documentation here for Drupal Gitlab Templates.

If you need extras:

drupal-ci-local --variable _GITLAB_TEMPLATES_REF=default-ref phpunit

Importing image directories to Eleventy

2026-02-15T00:00:00Z

My previous attempt at getting images into Eleventy didn't work out. I mean, it worked, but I wanted something different from what I'd built.

That package might be useful (eg if I wanted to publish all my cloud and sunset photos), but mostly I want to share a handful of photos like "Sandymount walk", which is a different use case.

Previous workflow:

Put images into Immich, manage/organise them there (eg into an album)
Get the album or media UUID out of Immich (manual step)
Use an or tag in my content

New workflow:

Copy images into ./src/media/ or directly to ./public/media/ (depends if I want to put the files in Git)
Run npm run image:ingest <ingest-directory> <output-directory> (eg npm run image:ingest ./src/media/sandymount-walk ./src/photos/sandymount-walk, where src/media/sandymount-walk is the directory containing the images, and src/photos/sandymount-walk is the generated markdown files)
Edit markdown content which is pre-filled with image tags

This is still pretty manual, but simplifies versus having content across disparate systems.

I generally echo '*' > src/media/<directory>/.gitignore and handle the media uploads with rsync, to keep the large files out of the repo.

Need to populate "description" in frontmatter too as that can provide alt= for the images.

2015 Corolla Fielder Hybrid G Aerotourer

2026-01-25T00:00:00Z

My old Toyota Carib bit the dust. We got by for a few months but eventually I had to bite the bullet and get a new car. This is what I got - a 2015 Toyota Corolla Fielder Hybrid G Aerotourer, imported from Japan.

Now I had questions!

How to change the time on the dashboard? I could not find it in the dashboard settings menu (despite being able to find and change the date there). Answer: There's a pair of odometer-reset looking buttons on the speedometer, they are HH and MM!
Where do I find a product manual? corollafielder_hybrid_201503.pdf
What's with the SIM card slot in the head unit? (miniB-CAS, for decoding TV, or is it T-Connect?)
What's happening with the map update microSD card?
What's Nanoe on the heater? There's a button labelled "Nanoe" - this was unfamiliar. It's a nano-electric heater that makes ozone by electrifying the air? Panasonic.

p225: Equipment inside RageJill (some translations)

Here's the product manual for my car (original and two rough translations). The translations are imperfect but useful if you can read through them.

Original version in Japanese

English translation A (via PDFMathTranslate)

English translation B (via Google Translate)

Locating manual PDFs and other docs:

Toyota.com links me from Owners' Manuals to 2015 Toyota Corolla owner's manual, but this is an AccessDenied XML error (for me, in Aotearoa). Same for 2014 Corolla Owner's Manual.
toyota.jp
- ⚠️ toyota.jp didn't work for me in firefox, used Chrome 🤷
- registration + use of in-browser translation got me to https://manual.toyota.jp/corollafielder/
- allowed me to find corollafielder_hybrid_201503.pdf
Found some owner's manuals from https://en.toyota-club.eu/manuals
Other PDFs I picked up along the way:
- 2010-corolla-axio-owners-manual-jpn.pdf - 2010 Corolla Axio
- 2012-toyota-corolla-fielder-jpn.pdf is a product brochure - has details of the Fielder spec variations (eg G, Aerotourer, WXB)
- corollafielder_hybrid_201308.pdf - 2013 manual from toyota.jp
- 2014-corolla-owners-manual-us.pdf.pdf from here is an owner's manual for the US 2014 model
- corolla-2015-owners-manual-us-canada.pdf
https://www.toyota.com/owners/warranty-owners-manuals/vehicle/corolla/2015/ not much of use here
Various somewhat helpful manuals sites that didn't have the correct model docs.

Reading the PDFs in English

All the docs I had so far were in Japanese. As for the dashboard and head unit, I could read the text on screen using my mobile + Google Translate app in camera mode, but for a 388 page PDF that's not practical.

I wanted an English translation that was passably readable. My first attempt to use Google Translate didn't work out, producing a PDF with garbage emoji where I hoped for text.

https://github.com/Byaidu/PDFMathTranslate is a translation tool for PDFs. Followed Docker instructions to bring it up at http://localhost:7860/ and used that interface to convert - this worked tidily, and showed me what I'd done wrong the first time with Google Translate.

I went back for a second pass at Google Translate (mostly for comparison now). Google Translate has a 300 page limit, so I split the original into two files:

pdftk corollafielder_hybrid_201503.pdf cat 1-200 output corollafielder_hybrid_201503.part1.pdf
pdftk corollafielder_hybrid_201503.pdf cat 201-388 output corollafielder_hybrid_201503.part2.pdf

Then I used Google Translate to translate the two files from Japanese to English, combining them back into a single PDF.

pdftk ~/Downloads/corollafielder_hybrid_201503.part1-1.pdf  ~/Downloads/corollafielder_hybrid_201503.part2.pdf cat output src/media/fielder-2015/corollafielder_hybrid_201503.google_translate.pdf

corollafielder_hybrid_201503.google_translate.pdf
corollafielder_hybrid_201503 is my car's manual. I wanted to read this but it's in Japanese.
In the car I had been using my mobile with Google Translate app in camera mode to read menus, and this works for viewing the PDF manual also (but slow to read without search).
File reads as "encrypted" to ocrmypdf so I first opened it in Evince and printed to PDF. This gave me corollafielder_hybrid_201503.printed.pdf
Then I was able to use OCRmyPDF to OCR the PDF: ocrmypdf --redo-ocr --language jpn 2010-corolla-axio-owners-manual-jpn.printed.pdf 2010-corolla-axio-owners-manual-ocr.pdf) to obtain corollafielder_hybrid_201503.orc.pdf
Also Tesseract? tesseract 2010-corolla-axio-owners-manual-jpn.pdf 2010-corolla-axio-owners-manual-eng.pdf -l jpn

Next step to try and translate the PDF to English,

https://github.com/Byaidu/PDFMathTranslate is a translation tool for PDFs. Followed Docker instructions to bring it up at http://localhost:7860/
Also Google Translate. won't do PDFs of more than 300 pages so we split it into two files.

ocrmypdf --redo-ocr --language jpn corollafielder_hybrid_201503.printed.pdf corollafielder_hybrid_201503.ocr.pdf
pdftk corollafielder_hybrid_201503.ocr.pdf cat 1-200 output corollafielder_hybrid_201503.ocr-part1.pdf
pdftk corollafielder_hybrid_201503.ocr.pdf cat 201-388 output corollafielder_hybrid_201503.ocr-part2.pdf

This output PDFs which looked like this (PDFTranslate first, then Google Translate):

Maybe no available fonts for the English translation?

Tried PDFMathTranslate again with the original PDF, without my ocrmypdf excursion, that worked great.

corollafielder_hybrid_201503-mono.pdf

Nanoe

Head unit

When I drove the car first, my notes included "seems ok, stereo yuck". I withdraw the latter - I think the NSZN-W64T head unit is fine. It plays CDs, radio, bluetooth audio, and has AUX input and microSD and SD card slots. (Haven't tested the cards; CDs work fine.)

The stereo works fine, and I would love to know what else is possible. I like the stereo's vibe as-is, can deal with (appreciate, even) labels in Japanese, and will not be rushing out to get a fancy Android head unit.

The head unit connects to WiFi ("AISIN CORPORATION" NIC), and has a Gracenote logo when playing CDs. I see that the stereo makes some HTTP requests when WiFi is configured, so one next step will be to capture the traffic to look at the request (eg user agent clues). Here's an nmap of the head unit connected to home wifi, with a host of "a.b.c.d":

Nmap scan report for a.b.c.d
Host is up (0.078s latency).
Not shown: 997 filtered tcp ports (no-response)
PORT     STATE  SERVICE
9001/tcp closed tor-orport
9002/tcp closed dynamid
9003/tcp closed unknown

Head unit: locked, update

After three months, the head unit got into a locked mode - looks something like the below. This was annoying. I did not want to buy an Android head unit, I just wanted my factory unit back. It took two weeks for me to get around to doing some research then taking a laptop out to the car.

Found a solution (amidst a LOT of YouTube videos selling services that would do the same):

Power on the car, so the head unit is powered.
Hold the PWR button down.
Turn the headlights on and off three times.
Follow the menu items in this video to access the ERC input screen.
Use Cyrus's ERC calculator to obtain an ERC code for the head unit's serial (displayed onscreen).
Enter the code on the ERC input screen.

Once this was done I navigated through the "Quick Setup" menu (translating what I saw using my phone) to configure the stereo. Boom!

More noise

2025-10-19T00:00:00Z

A music-playing weekend after enjoying Lines of Flight gig on Friday. Shoulda seen more of the events!

2025 ten one-nine

A rough drum pattern I've been messing with, in Strudel.

I plugged in the kid's old synth and got that talking to VMPK. Tried a couple of other synths, this one appeared to not want Jack, which suited me best.

Recorded in Audacity, I could see the levels were too high to begin with, no surprises how this sounds, haha now learn about levels Chris.

Recording

Looking like I have a bit to learn about audio on this system. Kid tells me the audio sounds "deep fried" 🤣

DCC & ORC candidate slogans, 2025

2025-10-05T00:00:00Z

It is local government elections 2025, here are some candidate slogans, do you know who they belong to?

Slogans below collected from riding the bus and taking notes, all errors my own!

More info:

Slogans

Click a slogan to see who has deployed it:

Less waste, more listening

Andrew Sutton

Delivering for Dunedin

Jules Radich

The real deal.

Sophie Barker

Turn this ship around!

Captain Hugh O'Neill

Care for tamariki

Marie Laufiso

Transformational leadership

Sophie Barker

Open. Honest. Fair.

Carmen Houlahan

Vote for the REEL deal

O'Neill & Chambers (double bill ad!)

You'd be barking mad not to have XXXXXX on council!

Brent Weatherall

Manaakitaka

Anna Knight

Proven, practical, committed

Bill Acklin

A breath of fresh air

Green Party candidates

Positive Action

Jules Radich

Follow your dog!

Jen Olsen

Sensible. Strong. Smart.

Sophie Barker

Strong leadership, proven track record

Andrew Simms

A compassionate approach to eliminating homelessness. No rates increases.

Daniel Rooney

Financial and ecological stability

Doug Hall Jr

Let's fix what's broken. Build what matters.

Lee Vandervis

What's news Dunedin

Richard Knights

Aroha ki te tākata

Marie Laufiso (/ Building Kotahitaka crew)

Committed to the well being of our city

Marita Johnson

For a healthy environment and future proof region

Tim Morgan

A beautiful Ōtepoti for all

Mandy Mayhem-Bullock

A Strong Voice for Dunedin

Jo Galer

Prosperous People, Thriving Environment

Chanel Gardner

I believe Dunedin can be better for all of us.

Lee Vandervis

I will bring more than classic Ford Falcons to Dunedin from across the Tasman

Benedict Ong

When Does It Stop

Lync Aronson

Submissions welcome, see contact page!

Also,

Thanks to all the folk who stuck googly-eyes on billboards across the city. Your work is appreciated.

Stupid small indie web

2025-09-13T00:00:00Z

Why write a silly page rating drink preferences?

It's throwing weird crumbs to crawlers. It's playing around with dumb content. It's a fancy that took me. It's a small git towards a weirder, smaller internet.

Mogu Mogu

2025-09-11T00:00:00Z

Mogu Mogu is a coco de nata (fermented coconut jelly) drink. It is quite sweet and feels super hydrating. Unimart has the best range in Ōtepoti.

Flavours

Blackcurrant

8/10. Sweet and refreshing. Looked unappealing, but turned out to be Blue Powerade flavour. Blackcurrant.

Coconut

7/10. OK sweetness, milky. Smokin taste, it's lovely.

Grape

5/10, not a huge fan of this one.

Lychee

10/10 or too much. Very sweet and rosy.

Mango

6/10. Okay. It's hard to do fake mango well.

Peach

7/10, love this.

Pineapple

8/10, sweet and refreshing. Love this.

Raspberry

4/10, somehow too sweet and too watery.

Strawberry

8/10, sweet & delicious.

Watermelon

6/10, maybe the same challenge as mango.

Untried

Apple
Orange
Pink Guava
Pina Colada
Yoghurt
Four Seasons

Strudel, a live coding music REPL

2025-09-06T00:00:00Z

Strudel is a music live coding environment for the browser, porting the TidalCycles pattern language to JavaScript. It's fun, thanks Simon for tipping me onto it!

https://strudel.cc

Check out the tutorial and, if it's your thing, enjoy - I am 🙂‍↕️ Here's a few early strudel baking results!

Temptation

Sweetest Taco

Simon asked me to do the Sweetest Taboo intro drums

Basic Rocket

I feel like this is a lot of rhythmic complexity for such a terse annotation? Basic sounds here.

Disabling data reporting in Lando

2025-08-20T00:00:00Z

I looked today for instructions on disabling data reporting in Lando, and thought to capture some notes on doing so.

Here's what worked for me:

Disabling usage reporting

Usage reporting is enabled by default, and lets the Lando team know how you're using Lando. At various Lando events, a report "ping" is sent to metrics.lando.dev. Here's an example of data reported, as shown by running lando restart -vvv:

example reporting stop action to url=https://metrics.lando.dev, report=true +31ms
example reporting data command=lando restart, context=local, devMode=false,
                       instance=abcdef, nodeVersion=v18.19.1, mode=cli,
                       type=Linux, platform=linux, landoPlatform=linux,
                       release=6.8.0-71-generic, arch=x64, isWsl=false,
                       isWslInterop=false, product=lando, version=3.23.26,
                       app=0123abc, type=drupal7,
                       services=[
                         api=3, type=drupal-php, version=7.4,
                         api=3, type=mariadb, version=10.3,
                         api=3, type=compose, version=custom,
                         api=3, type=mailhog, version=v1.0.0
                       ],
                       action=stop, created=2025-08-20T10:26:55.784Z +1ms

To disable it, add the following to your ~/.lando/config.yml file, as documented in https://docs.lando.dev/data.html

stats:
  - report: false
    url: https://metrics.lando.dev

You might read that yaml and think url might be redundant. It's not - and the reporting URL changed at some point in the past, so you might even have a different URL (https://metrics.devwithlando.io) disabled in your configuration, and be surprised to find that Lando has resumed reporting to its new URL.

I think it would be an improvement to permit a clearer global disable there (eg stats: false), to avoid a future change to the default reporting URL restoring stats reporting without signal to the user.

Disabling crash reports

Disabling crash reporting takes a slightly different format: ~/.lando/cache/report_errors will either not exist (prompts on next run), or contain "true" or "false". To disable:

echo "false" > ~/.lando/cache/report_errors

About that data

Data is reported to Amplitude and crash information stored in BugSnaq.

While it's really helpful to developers to have access to crash data and usage metrics, it's a decision to consider. I'm wary of the level of data reported from systems - in 2025 I'm far more cautious about, say, webserver access logs than I was in 2015 or 2005. The safest option is to minimise data gathered.

I still expect to use the excellent Lando toolkit, and to report detailed bug information manually.

Discovering local devices using HASS

2025-04-22T00:00:00Z

I've been working with Observability and the Grafana stack lately, and while having a re-organise got thinking about surveying and harvesting information about our home network.

Home network is a dynamic environment - devices come and go, are forgotten. I had observed that Home Assistant (HASS) was quite good at "discovering" new devices on the network, and got thinking about how I could utilise this myself.

Information recorded manually gets outdated and unreliable fast, while information retrieved by surveying gives you a current view but can lack historic context. The trick is combining what's discovered with knowledge you'd like to preserve; discovery can give you "a device is connected, named X and has details Y", while preserving knowledge can tell you who it belonged to or why it was introduced.

First question was, how to find hosts HASS sees. HASS has an API! Let's explore - we will need:

HASS_URL The URL of your Home Assistant instance
HASS_TOKEN A long lived token from Home Assistant

Let's see what the states are in HASS.

curl -X GET -H "Authorization: Bearer ${HASS_TOKEN}" \
  -H "Content-Type: application/json" \
  ${HASS_URL}/api/states

I get a response like:

[
  {
    "entity_id": "person.chris_burgess",
    "state": "home",
    "attributes": {
      "editable": true,
      "id": "chris_burgess",
      "device_trackers": [
        "device_tracker.chris_oneplus",
        "device_tracker.chriss_ipad_2",
        "device_tracker.nexus_5x",
        "device_tracker.cph2529"
      ],
      "latitude": -45.00,
      "longitude": 170.00,
      "gps_accuracy": 10,
      "source": "device_tracker.abcdef1",
      "user_id": "aabbccdd",
      "friendly_name": "Chris Burgess"
    },
    "last_changed": "2025-04-21T19:11:33.996324+00:00",
    "last_reported": "2025-04-21T22:36:42.920973+00:00",
    "last_updated": "2025-04-21T22:36:42.920973+00:00",
    "context": {
      "id": "01234",
      "parent_id": null,
      "user_id": null
    }
  },
  {
    "entity_id": "device_tracker.ubuntu_server_2",
    "state": "home",
    "attributes": {
      "source_type": "router",
      "ip": "10.0.0.2",
      "mac": "AB:CD:11:22:33:44",
      "host_name": "ahem",
      "last_time_reachable": "2025-04-21T22:43:16+00:00",
      "connected_to": "fritz.box",
      "connection_type": "LAN",
      "icon": "mdi:lan-connect",
      "friendly_name": "ahem"
    },
    "last_changed": "2025-04-21T19:12:32.411093+00:00",
    "last_reported": "2025-04-21T22:43:18.691542+00:00",
    "last_updated": "2025-04-21T22:43:18.691542+00:00",
    "context": {
      "id": "01234",
      "parent_id": null,
      "user_id": null
    }
  },
  {
    "entity_id": "device_tracker.the_nas",
    "state": "home",
    "attributes": {
      "source_type": "router",
      "ip": "10.0.0.3",
      "mac": "AB:CD:11:22:33:44",
      "host_name": "the-nas",
      "last_time_reachable": "2025-04-21T22:43:16+00:00",
      "connected_to": "fritz.box",
      "connection_type": "LAN",
      "icon": "mdi:lan-connect",
      "friendly_name": "nas"
    },
    "last_changed": "2025-04-21T19:12:32.429386+00:00",
    "last_reported": "2025-04-21T22:43:18.692900+00:00",
    "last_updated": "2025-04-21T22:43:18.692900+00:00",
    "context": {
      "id": "01234",
      "parent_id": null,
      "user_id": null
    }
  },
  // ...
]

Let's filter that down to hosts we're interested in:

curl -X GET -H "Authorization: Bearer ${HASS_TOKEN}" -H "Content-Type: application/json" ${HASS_URL}/api/states  | jq '.[] | select(.state == "home" and .attributes.host_name != null) | {entity_id, state, host_name: .attributes.host_name}'

How about as a Markdown list:

curl -X GET -H "Authorization: Bearer ${HASS_TOKEN}" -H "Content-Type: application/json" ${HASS_URL}/api/states  | jq -r '.[] | select(.state == "home" and .attributes.host_name != null) | "- \(.attributes.host_name) (\(.attributes.ip))"' | sort

Or as a table:

echo "| friendly name | host name | ip | notes |"
echo "|--|--|--|--|"
curl -X GET \
  -H "Authorization: Bearer ${HASS_TOKEN}" \
  -H "Content-Type: application/json" \
  ${HASS_URL}/api/states  | jq -r '.[] | select(.state == "home" and .attributes.host_name != null) | "| \(.attributes.friendly_name) | `\(.attributes.host_name)` | `\(.attributes.ip)` |  |"'

The output of the last might look like (after comments added):

friendly name	host name	ip	notes
9b7eb8b9-dff1-d401-5507-09a8aaaea195	`9b7eb8b9-dff1-d401-5507-09a8aaaea195`	`10.0.0.59`	?
a1f6389c-bb13-c10f-3af0-cd147e51a662	`a1f6389c-bb13-c10f-3af0-cd147e51a662`	`10.0.0.52`	?
aooga	`aooga`	`10.0.0.101`	VM running some services
baluga	`baluga`	`10.0.0.102`
Android-2	`Android-2`	`10.0.0.51`	Visitor phone
laptop-1	`work-laptop`	`10.0.0.54`	Work laptop
Chromecast-Audio	`Chromecast-Audio`	`10.0.0.59`	Best device
Chromecast	`Chromecast`	`10.0.0.103`
NAS-1	`NAS device`	`10.0.0.53`
DSP-W123	`DSP-W118`	`10.0.0.104`
ESP-ABCDE1	`ESP-ABCDE1`	`10.0.0.55`
ESP-123ABC	`ESP-123ABC`	`10.0.0.56`
ESP-DEF432	`ESP-DEF432`	`10.0.0.57`
esphome-web-809081	`esphome-web-809081`	`10.0.0.58`

This is useful, I now have a list of devices to review, and a table to keep notes in!

Our Friend the Computer

2025-04-08T00:00:00Z

I am currently devouring the Our Friend The Computer podcast, "a podcast exploring alternative computing histories and their relationship to society". It's interesting, engaging, and provides lots of stimulating avenues to consider the aspirations of early networks, their development, and their impact on society at the time.

I appreciate the social, cultural and art lenses that it applies. The first several episodes I've listened to - I'm approaching the end of the "Pre-Internet Networks" season - discuss and consider the social impacts and experiences of several networks, including CyberSyn, OGAS, Minitel, Bildschirmtext, NABU and Beltel.

Unlike today's internet, each of those networks was situated quite specifically to one particular nation. It's really interesting here to investigate the state-architected^[1] development of early computer networks, and the strong design impact of the specific culture and social priorities of the country this was happening in.

It's fascinating to see the impact of each nation's culture and language lead to different qualities in each of the networks discussed. With Bildschirmtext we explore the development of a network in Germany as the country re-integrates after the cold war separation.

Let's not assume that's always for good, either: in the Beltel episode we her of early computing networks used as tools of apartheid. The Beltel episode hit pretty hard, to be honest; as a worker in the tech industry, I am mindful of tech companies participating in apartheid and genocide. That was true then, and sadly it is true today. We must use our power to see this, to resist and undo the damage of colonisation.

I enjoy the exploration of how each culture led to the development of a network for each environment. I'm sure this eventually resulted in interoperability challenges and those gradually contributed to the less culturally situated, more globalised internet we have in 2025.

For me this brings a reflection on the development of non-state networks, also. If a group is not represented within the state/power systems in play, they might have to subvert available systems to develop a non-state network (eg BBS usage by resistance groups in South Africa as a response to government control of Beltel), or somehow create a network of their own. The internet hasn't solved this.

It's also interesting to apply this consideration to the development of government supported or sponsored platforms in current times. Technologies such as Common Web Platforms, or technology preferences as expressed through purchasing boards or panels, express the desires of states in modern technology. These may be technologies controlled by a single vendor, which tend to increase dependency and reduce agency, or they may be platforms open to participation, which offer at least an opportunity for self-determination and agency to the state, its component organisations, and the citizens within it.

The podcast hosts, Camila Galaz and Ana Meisel, explore somewhat familiar tech spaces through a lens that's richer and more human than would typically be applied. I'm really grateful for their work and it's inspiring me to consider how I can apply creative and more emotive expression to my own understanding of our work and impacts in society. Thanks Camila and Ana! Your reflections are delightful.

Worth considering that "state architected" here likely often unpacks with "by a small team of geeks within the relevant state telephony provider". ↩︎

GlitchTip API CLIent

2025-02-23T00:00:00Z

In GlitchTip's community chat a few days ago, a user asked about filtering event logging and issue reports in GlitchTip.

Depending on your GlitchTip / Sentry client library, you have various options to control what gets reported to Sentry or GlitchTip. The Python and Drupal Raven integrations support Sentry's before_send pattern, while other client libraries may or may not allow you to "silence" reports that might generate a lot of noise in your Sentry.

An example of this I've seen is reports for each 404 hit in some applications, which can result in a bunch of needless events and eventually issues in GlitchTip that are associated with the 404s.

There was a question raised in the GlitchTip channel about how to automatically silence messages in GlitchTip. The user asking didn't have access to filter clientside, so they were looking for a solution to remove entries within GlitchTip.

My take is that the preferred order of solving this is:

If at all possible, prevent issues being recorded from the client. Eliminating the event report means less data is gathered in GlitchTip. This is optimal.
If you can't handle it in the client configuration, you have to do it at the GlitchTip side.
1. A WAF-style filter in front of GlitchTip may be an option to prevent the submission reaching GlitchTip at all. This might be a configuration in a reverse proxy that GlitchTip sits behind.
2. Filtering could be introduced in GlitchTip to provide configurable filters - but event reporting is a performance critical pathway in GlitchTip, so this would need to be kept light.
3. Filtering can also be implemented in GlitchTip's ingest path - this was the maintainer's recommendation in thread. This is also a performance-sensitive path in GlitchTip so needs to be carefully tested.
4. Once the event is ingested and the item created, there's capacity to remove the item from GlitchTip.

It struck me that you could do something like:

# Get a token from /profile/auth-tokens in GlitchTip.
export GLITCHTIP_API_TOKEN=ABCDEF
curl -H 'Accept: application/json' \
  -H "Authorization: Bearer $GLITCHTIP_API_TOKEN" \
  'https://glitchtip.example.org/api/0/organizations/some-org/issues/?query=is:unresolved%20level:warning&project=2' \
  | jq '.[].id' \
  | while read ISSUE_ID ; do
  curl -H 'Accept: application/json' \
    -X DELETE \
    -H "Authorization: Bearer $GLITCHTIP_API_TOKEN" \
    'https://glitchtip.example.org/api/0/organizations/some-org/issues/$ISSUE_ID'

I figured I'd test this out, and the one-liner grew into a little bash script to show or delete issues from GlitchTip. Deleting issues should delete related events.

This is not the ideal solution - we want to reduce waste as a first step, recycle second; similarly we want to reduce event capture as a first step, and then as a second strategy remove events that we shouldn't have captured. (Maybe that wanted a fishing analogy, IDK.)

But, as a strategy for periodically clearing out events we don't want captured ... it might work. And that might be useful to me, or you. The tool is now available as xurizaemon/gt-cli.

I'm wondering at how I named it now, as I don't at all expect this to become the GlitchTip CLI. The Django app already has a local CLI, ./manage.py, and this has a couple commands only for the remote API. It's an experiment which I'm sharing just to share. If you find it useful, all the better!

A more efficient approach would likely be to PR some additional ./manage.py subcommands to filter out issues/events.

Drop your copyright ranges

2025-01-03T00:00:00Z

Day after New Years Day 2025, I saw a PR opened on a project I maintain. For historic reasons the hubotio/hubot project has a copyright statement with a date range, ex "Copyright 2011-2024, Github Inc".

Lots of projects have this format in their licensing, and I question if it's useful. I do not believe it has meaning - the project copyright wouldn't lapse if we missed updating the year, would it? I believe that in some media the second date signals that the last date that the work was substantitively modified (say, you re-edited and re-released a book text).

With an open source project, the change history of the project is highly visible, and a license date may even be less accurate than the commit history already visible. It's also a weird (to me) duplication, a kind of manual last-modified date that by being updated ensures the document has a last-modified date anyway.

I think projects would be better off dropping the end date. Your commit history should show when the project was modified, but doesn't need to show trivia like "Updated copyright year"; this is extra work for your contributors, and can be additional cognitive load^[1] for anyone working with your codebase.

Your project probably don't need additional MRs/PRs that make trivia changes, unless they serve some other purpose. Trivia changes might be useful as new contributor practice tasks, but if that's the aim, you won't want a once-a-year ritual for it.

Worse, if your project includes copyright in each file (as GNU.org's GPL guidance recommends^[2]), it is increasingly hard to discern meaningful changes in a file's history from trivia updates. I think GNU.org's advice is impractical here for most projects, and jurisdictions vary anyway.

Anyway - back to the PR opened on Hubot. I saw the PR, and thought "huh, why do we still have this copyright range here". The PR description said,

Yearly copyright bump. Padding my changed LOC! 😛

Uhoh. And yep, looking at the contributor's history, the account had no commits prior to this date, and had 1.5K similar open PRs across a bunch of projects ... 🙄 ... I searched for patience ... Perhaps they think it's helpful - but they do know what they're doing; they were capable of automating thousands of PRs, from a fresh Github account. I commented on the PR noting the user's PRs to other projects, suggesting that we should drop the end date instead. Joey closed the PR. Good.

Thing is, even if it was entirely well intentioned, this action probably wasted the time of hundreds of people. Even if they did no more than scan the email to decide whether it was worth their attention, that time cost in making bulk PRs multiplies out across hundreds or thousands of volunteers. Many of the PRs would be ignored, but even then it stacks up in the PR queue for that project, and will add to friction for future maintenance.

So, my messages:

Please don't make bulk contributions without a lot of care. I fully support new contributors making tentative PRs, even trivial ones, if they feel they are contributing something meaningful. I do not have patience for automated activity that is low-value and disregards related volunteer labour costs^[3].
If you maintain a project and have a "Copyright 2020-2024" which needs updating right now, consider removing the end date. If it's not useful to your project, removal may eliminate a busywork task for the future, and reduce cognitive load for developers.
If your project is languishing and only getting that annual license year update, perhaps stop by the oasis of No Maintenance Intended while you're at it. Nothing stopping you setting it down and picking it up again later :)

Update! This article discusses this and more, and makes a solid case for using the year of first publication for each file: https://matija.suklje.name/how-and-why-to-properly-write-copyright-statements-in-your-code

None of this is legal advice.

I probably don't live in your jurisdiction anyway.

Some prior art - years back in the CiviCRM project, we discussed (dev/core#878) and agreed to simplify the copyright headers including removing the year ranges - there was a long-form GPL license in each file's header with dates that required periodic updates. Simplifying the license in the files removed 47K of license text across 3.2K files (⚠️civicrm/civicrm-core#15843), making it much, much easier to read through future changes to any given piece of code's history by eliminating a class of changes which were not significant. ↩︎
I see that GNU.org's GPL HOWTO advises to introduce a year marker in the copyright for each release (guidance which I question, but I'll include here as it may explain why this pattern is widely observed).
The copyright notice should include the year in which you finished preparing the release (so if you finished it in 1998 but didn't post it until 1999, use 1998). You should add the proper year for each past release; for example, “Copyright 1998, 1999 Terry Jones” if some releases were finished in 1998 and some were finished in 1999. If several people helped write the code, use all their names.
For software with several releases over multiple years, it's okay to use a range (“2008-2010”) instead of listing individual years (“2008, 2009, 2010”) if and only if every year in the range, inclusive, really is a “copyrightable” year that would be listed individually; and you make an explicit statement in your documentation about this usage.
↩︎
What happened to that account and its PRs? I see today that the account, and almost all history of its activity (except signing this CLA! conda/infrastructure#1087), has been eliminated from Github. That's why this post mostly lacks URLs for the PRs and activity discussed. There's some mentions of their activity elsewhere. ↩︎

⭐ to relay

2024-12-30T00:00:00Z

I have mixed feelings about relaying between social networks: it's okay, but needs consideration.

For:

I own my rants, blorts, toots, skeets, posts micro, minor and major, and I will post them in multiple places.
I have connections on many networks and want to participate.
I'm against busywork, which includes copypasting between systems.

Against:

Relayed posts often hit badly when they are written with network A in mind and show up on network B.
That's especially true if the relayed account is not "active" - now you have a broadcast setup. (Is this how @bsky.brid.gy accounts work? Not sure.)
Special mention: No-consent relays such as bird.makeup suck hard. Do not do this. Set yourself a higher bar than "it's technically possible".

(Probably missing some bits there. Anyway.)

Currently I'm active in Fediverse (preferred, via the excellent, small, https://toot.cafe Mastodon instance) and Bluesky. I wanted the capacity to continue posting laundry appreciation, tech snark and life reflections to the Fediverse, and make it easy to circulate those posts from Fediverse to Bluesky when I wanted to^[1]. Since I'd be making that choice when posting, I control using the available interface - typically my mobile Mastodon client, Tusky, or the web interface for my home toot.cafe instance.

We can't extend the UI of apps we don't control ... usually. With ingenuity, we can re-use an existing UI element for our requirements: use the ⭐ "Fave" UI to flag a post from the posting UI for our own purposes, then read the post's faved status to determine whether the post should be relayed.

I had previously implemented this same functionality for a Fediverse / Twitter relay service, moa.party (fedstoa/moa#53). That project was no longer active but the solution could be reworked with a new service. Skymoth looked viable, and while it didn't share any DNA from Moa, I was able to implement a similar solution.

Taking a look at Skymoth, I could see notes in the README already relating to user control. The first step would be to see if this system was a fit for my usage. I ran up a copy locally using Docker Compose and tested it out; aside from the addition I planned to add, it seemed to work fine.

Next I opened an issue (https://github.com/thilobillerbeck/skymoth#55) to let the maintainer/community know I was interested in making this happen, posted some first thoughts on approach, and outlined my understanding of the existing implementation. My goal here was to avoid conflict with any existing WIP and to seek initial input.

Then I used the project docs to bring up a local development environment. This was interesting as the project uses a toolkit I hadn't tried before, Cachix DevEnv. It's pretty swish! It was super fast to bring up the local development environment, and this setup had a tidy TUI interface, process-compose, to manage and observe running processes.

Skymoth is written in TypeScript, using Fastify to provide the web app interface and Prisma + Postgres for storage. Of these, only Postgres was familiar territory. The DevEnv setup made it really easy to get up and running, and the toolchain was accessible and straightforward to get hacking on.

I mocked up some local changes to show the UX I would propose, and shared them on the issue queue. Then I set about implementing. The existing query structure made it pretty easy to work with. While I only wanted fave-to-relay for my own needs, I recalled that moa.party (and/or other relaying tools?) had supported relaying based on the presence or absence of a hashtag, so I included that functionality as well.

I liked that Skymoth's call to Mastodon API fetched 50 recent posts and iterated over them; that eliminated an ungainly part of how this worked in moa.party, where your post might not be relayed if you didn't fave it quickly enough, as the most recent post ID would be used as a high-water mark whether or not that post was relayed at the time of the poll.

The most challenging part was trying to follow some existing implementation - there was storage for user settings in the DB schema, but that didn't seem to be wired up in the application. I gave it a shot, but eventually left it and stored the details against the user; I think what I ran into was some incompletely implemented settings storage, not yet in a state to be used, making for a bit of a red herring! Once I decided to ignore that entirely, things progressed fine :)

The changes did what I wanted and the end result is great for me - I am happily posting to Fediverse (my preferred network), hitting ⭐ if I want to circulate it over to Bluesky, and able to engage with replies on both networks. Relayed posts handle the details of each network fine so far, including media, alt text, and splitting post bodies if the Mastodon post is too long for a single Bluesky post. I don't have to fave right away, self-replies work as desired, it's grand. And now that the PR is merged, the users of https://skymoth.app have the same functionality. Delighted!

I picked up some TypeScript knowledge along the way, and appreciated the simplicity of this tidy little app and the toolchain Thilo has selected for it. Nix, DevEnv, Fastify and Prisma all look interesting tools to use again in future if opportunity arises.

If you want to use this yourself, you can either use the hosted version at https://skymoth.app which now does this, or use the example.compose.yml in https://github.com/thilobillerbeck/skymoth/ to run your own copy locally.

Bluesky can join the Fediverse any time they like, of course, and save me the effort, but hey. ↩︎

Sunset over Wakari

2024-12-07T00:00:00Z

Te Whanaupaki / Flagstaff, Wakari, and Saddle Hill
As viewed from Pine Hill Road, 7 December 2024.
(Source: JS, CSS)

I appreciated finding the One O'Clock Jump: British Linocuts from the Jazz Age exhibition at Christchurch Art Gallery, and even timed it right to watch the curator intro - lucky!

One thing that caught my eye was artists highlighting distant hillside edges with a stroke of sunlight - might have been Rhona Haszard's "Spring in the Marne Valley" or some other piece, I was all eyes and agog in the exhibition - go see it if you can :)

The exhibition website above has a good selection of the pieces online but not all when I checked, and I couldn't find the piece I exactly recall seeing this on.

Arriving back in Dunedin we were out collecting elderflowers for cordial and I saw a similar effect looking across towards Wakari as the sun set.

I thought the image would be cool to try and produce a sketch of that using the Web Canvas API in the style of such a lino cut; I hadn't had a play with Canvas API before and I figured that this style would lend itself well to a first attempt. The time will be well spent getting familiar with some javascript interfaces and geometry.

Also in mind for me was the natural shapes which start to emerge when you have a Grafana dashboard in the corner of your eye.

I worked out how to draw a line across the screen and a linear gradient vertically from that to shade the hillside, then how to make the line randomly walk up and down a few pixels to give it an organic shape.

I then removed the jitter and shaped the lines with a couple of crude mathematical functions that could approximate two distant hills, one peeking from behind the other. I started with simple functions to define these shapes at first, then introduced wave and cubic functions so I could lift nicer shapes from maths magic.

I repackaged things so that I could paste drawHill() calls into the browser console to try different strokes. You can do this: open your browser console (Ctrl + Shift + K / Option + ⌘ + J) and paste this to add a new hill in the foreground of the image:

drawHill({
    initialY: 325,
    stepSize: 5,
    shadowY: 800,
    strokeStyle: '#cfc',
    lineWidth: 2,
    shape: wave(2.5, 0.6, -2),
});

Pasting again will draw a new stroke with a new hill in the foreground. If the canvas is redrawn (you can kinda do that by hitting reload now) it produces an animation not unlike a 2D flight sim valley.

Then I added a couple more functions to paint a gradient in the background for the atmosphere and fill a circle for the sun.

Along the way I ran into some interesting and amusing unexpected outputs when I tried numbers that didn't output normal-looking scenery, that was fun too :)

I hadn't had a phone with me when I saw the sunset, which meant I didn't have a reference for the colours of the scene I saw yesterday. Attempts to find it by nudging a colour selector about were unsatisfactory, and I like the reduced if flat greyscale palette I'd kicked off with, so I'm leaving it there. It has a 1990s Classic Mac feel I appreciate, even if it's an impossibly wide (to the smaller Mac displays of that era) 720 pixels.

Is it Wakari as seen from Pine Hill? It's far more a gestural sketch from my imagination, and it's a little mathematical in arrangement and tone, but I am happy with it!

This is the first piece of procedural art I've made in a long time and it's been fun.

Thinking forward:

Make the shading on the hillside account for the sunlight rather than vertical
Birds?
Trees & other shapes on the horizon

Reporting from Composer to Dependency Track from Gitlab CI

2024-11-25T00:00:00Z

For organisations using open source to build software, I believe a high-level view of the components being consumed is critical to maintenance and vulnerability response. Tools like this offer not just vulnerability detection, but also risk reduction, opportunity to manage dependencies more efficiently across projects, risk reduction, and license and policy compliance.

Dependency Track is a tool which organisations can use to gather and analyse their component usage. It does this by permitting submission of software bill of materials (SBOM) reports into a store, and combining this with automated import of vulnerability and component data alongside organisational policies.

I'd demoed Dependency Track in Drupal South Shorts in 2021, and figured it was worth another look as part of a project I'm working on.

Bringing up a Dependency Track instance

I used Dependency Track's instructions to deploy an instance using Docker Compose per their Deploying with Docker instructions. This works great - but do pay attention to the system requirements; Dependency Track looks lightweight on the surface, but behind the scenes it is importing and analysing a fair bit of data.

Dependency Track has a frontend and an API service. For reporting to Dependency Track, the API endpoint needs to be externally accessible.

Note that there's a short period after you bring the environment up when it's chugging away yoinking data from NIST and various other sources. Allow that time to ingest before expecting useful output.

The default creds are admin/admin, which you'll change before accessing the admin interface. Post-login, you probably want to set up a few things in the admin section; I recommend configuring Administration > Email for notifications. As I would be analysing Drupal projects for my instance, under Administration > Repositories > Composer, I added Drupal's Packagist endpoint https://packages.drupal.org/8 as well.

I then added a couple of Projects to test with. One of these is xurizaemon/commerce-demo, which is a public repository.

Generating SBOM in CI

I use CycloneDX/cyclonedx-php-composer for this. There are other tools, such as Gemnasium, and they might work equally well. In the Gitlab CI job, I'll install CycloneDX/cyclonedx-php-composer using composer global require (to keep it separate from the project-specific dependencies), then execute the new Composer command.

The job depends on some configuration, which I apply in Gitlab's CI/CD Variables settings. These are:

Variable key	Example	Description
`DEPENDENCYTRACK_API_URL`	`https://dt-api.example.org`	Base URL of the Dependency Track API service
`DEPENDENCYTRACK_API_KEY`	`abc_aaaabbbbccccddddEEEEFFFFGGGGHHHH`	API key for reporting
`DEPENDENCYTRACK_PROJECT`	`615683fe-abcd-1234-abcd-a04a7514e7f5`	Project identifier

For the API Key, in Dependency Track navigate Administration > Access Management > Teams; the "Automation" team should have an API key already assigned. For the Project ID, in Dependency Track navigate Projects > {selected project} > View Details and copy the Object Identifier.

If these aren't configured, the job will bail out with a configuration warning.

report dependencies:
  stage: check
  image: composer:2
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  needs:
    - composer install
  before_script:
    # Verify that the required variables have been configured in Gitlab CI/CD Variables before proceeding.
    - >
      if [ -z "${DEPENDENCYTRACK_API_KEY}" -o -z "${DEPENDENCYTRACK_API_URL}" -o -z "${DEPENDENCYTRACK_PROJECT}" ]; then
        echo "Configure DEPENDENCYTRACK_API_KEY, DEPENDENCYTRACK_API_URL and DEPENDENCYTRACK_PROJECT to report to Dependency Track."
        exit 1
      fi
    - composer global config --no-plugins allow-plugins.cyclonedx/cyclonedx-php-composer true
    - composer global require cyclonedx/cyclonedx-php-composer
  script:
    # Generate an SBOM in XML format for submission.
    - composer CycloneDX:make-sbom --output-format=XML --output-file="${CI_PROJECT_DIR}/composer-sbom.cdx.xml"
    # JSON for submission requires DT project identifier and b64encoded BOM XML.
    - >
      echo "{
        \"project\": \"${DEPENDENCYTRACK_PROJECT}\",
        \"bom\": \"$( base64 --wrap=0 ${CI_PROJECT_DIR}/composer-sbom.cdx.xml )\"
      }" > ${CI_PROJECT_DIR}/report-bom.json
    # PUT the JSON to Dependency Track endpoint.
    - >
      curl -X "PUT" "${DEPENDENCYTRACK_API_URL}/api/v1/bom" \
        -H "Content-Type: application/json" \
        -H "X-API-Key: ${DEPENDENCYTRACK_API_KEY}" \
        -d @${CI_PROJECT_DIR}/report-bom.json
  artifacts:
    paths:
      - composer-sbom.cdx.xml
      - report-bom.json
  allow_failure: true

With this correctly configured, we should see output from the Gitlab CI job (job from Commerce Demo project mentioned above) which shows the report being submitted to our Dependency Track service.

$ composer CycloneDX:make-sbom --output-format=XML --output-file="${CI_PROJECT_DIR}/composer-sbom.cdx.xml"
$ echo "{ # collapsed multi-line command
$ curl -X "PUT" "${DEPENDENCYTRACK_API_URL}/api/v1/bom" \ # collapsed multi-line command
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  825k  100    48  100  825k      8   152k  0:00:06  0:00:05  0:00:01  151k
{"token":"9090c0d5-6257-4b23-ae14-64d05fb4f349"}

And then in Dependency Track, we should see our gathered data.

I'll return to "what can we do with this information" in a later post maybe - until now, enjoy exploring Dependency Track.

Showing Composer dependency changes on MRs from a CI job

2024-11-12T00:00:00Z

I am a fan of davidrjonas/composer-lock-diff for making visible the dependency changes in a change, and often use its --md output to make this highly visible in a merge request.

I decided to make a quick job that would give me a one-click run of the tool in the merge request pipelines, so I could copy from there back to the MR.

composer changes:
  needs: [  ]
  stage: check
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - composer.lock
  allow_failure: true
  script:
    - PATH=$PATH:$( composer global config home )/vendor/bin
    - git fetch origin
    - composer global require davidrjonas/composer-lock-diff:^1.0
    - composer-lock-diff --from origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME --md | tee composer-changes.txt

Catching an artifact with tee isn't much help really, it's quicker to copy straight from the job output, but you can add that file as an artifact if you like.

(I'm still deciding whether this is best with or without a when: manual.)

Probably Gitlab has a more full-featured version of this in its fancy money tools - but this small addition is really handy.

It would be nice to extend this to automatically add a "Dependency changes" section to the MR description, or in a comment.

Check Chrome connectivity in Gitlab CI

2024-11-12T00:00:00Z

Wiring up Behat in a project the other day, I wanted to validate Chrome connectivity in the job environment quickly - without waiting for all the steps that would ordinarily depend on, such as composer install and other build processes.

I stripped a more full featured CI job down to the bare minimum to do this and figured it's worth capturing. Here's what I used:

zenika chrome:
  image: composer
  stage: test
  needs: [  ]
  services:
    - name: zenika/alpine-chrome:latest
      alias: chrome
      command:
        - '--disable-gpu'
        - '--headless'
        - '--remote-debugging-address=0.0.0.0'
        - '--remote-debugging-port=9222'
        - '--no-sandbox'
        - '--disable-web-security'
        - '--window-size=1080,1920'
  variables:
    CHROME_URL: "http://chrome:9222"
    HOST_PORT: "build:80"
  script:
    - >
      curl --silent --show-error --header "Host: localhost" ${CHROME_URL}/json/version | tee chrome-version.json  
  artifacts:
    when: always
    name: "artifacts-${CI_JOB_NAME}-${CI_COMMIT_SHA}"
    paths:
      - chrome-version.json

Here's that same job again, but for Pantheon's build-tools image. This one took a full minute longer to run - the image contains much more than just Chrome, so it takes longer to pull.

pantheon build tools:
  image: composer
  stage: test
  needs: [  ]
  services:
    - name: quay.io/pantheon-public/build-tools-ci:6.x
      alias: chrome
      command:
        - 'google-chrome'
        - '--disable-gpu'
        - '--headless'
        - '--remote-debugging-address=0.0.0.0'
        - '--remote-debugging-port=9222'
        - '--no-sandbox'
        - '--disable-web-security'
        - '--window-size=1080,1920'
  variables:
    CHROME_URL: "http://chrome:9222"
    HOST_PORT: "build:80"
  script:
    - >
      curl --silent --show-error --header "Host: localhost" ${CHROME_URL}/json/version | tee chrome-version.json  
  artifacts:
    when: always
    name: "artifacts-${CI_JOB_NAME}-${CI_COMMIT_SHA}"
    paths:
      - chrome-version.json

Lighthouse browser crashes

2024-08-26T00:00:00Z

For this site I have a small CI job (based on gitlab-ci-utils/lighthouse) which checks the built site in Lighthouse.

This job runs after the site has been built (using Eleventy) in test build:

# https://gitlab.com/gitlab-ci-utils/lighthouse
lighthouse mr:
  needs:
    - test build
  image: registry.gitlab.com/gitlab-ci-utils/lighthouse:latest
  stage: post
  script:
    - mkdir -p tmp
    - serve ./public/ > tmp/serve.log 2>&1 & wait-on http://localhost:3000/
    - lighthouse http://localhost:3000 --chrome-flags="--headless"
    - lighthouse http://localhost:3000/photos --chrome-flags="--headless"
    - lighthouse http://localhost:3000/projects --chrome-flags="--headless"
  rules:
    - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH
  artifacts:
    when: always
    paths:
      - localhost*.html
      - tmp

However, today it was failing. The final line in the job output showed the error:

Runtime error encountered: Browser tab has unexpectedly crashed.

I had a look at the Lighthouse CLI help via docker run -ti registry.gitlab.com/gitlab-ci-utils/lighthouse:latest bash and then lighthouse --help. I've seen memory issues pop up in CI before, so I was looking first for something to tell Lighthouse to adjust its memory expectations.

I also had a dig around and found a few issues (GoogleChrome/lighthouse-ci#1052: Browser crashes on lhci autorun in Docker #1052, puppeteer/puppeteer#1834: Add --disable-dev-shm-usage to default launch flags), but none that looked like an exact match for what I was seeing ... except it did just look like Chrome was crashing, and I've seen that before as part of behat-chrome/chrome-mink-driver use.

While there wasn't a direct flag in the lighthouse help, we have --chrome-flags, so I tried Chrome's --no-dev-shm-usage (ref Chromium command-line switches), which I've frequently found makes Chrome behave better in CI:

# https://gitlab.com/gitlab-ci-utils/lighthouse
lighthouse mr:
  needs:
    - test build
  image: registry.gitlab.com/gitlab-ci-utils/lighthouse:latest
  stage: post
  script:
    - mkdir -p tmp
    - serve ./public/ > tmp/serve.log 2>&1 & wait-on http://localhost:3000/
    - lighthouse http://localhost:3000 --chrome-flags="--headless --no-dev-shm-usage"
    - lighthouse http://localhost:3000/photos --chrome-flags="--headless --no-dev-shm-usage"
    - lighthouse http://localhost:3000/projects --chrome-flags="--headless --no-dev-shm-usage"
  rules:
    - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH
  artifacts:
    when: always
    paths:
      - localhost*.html
      - tmp

A little surprisingly, that worked, and the pipeline started passing ... huh!

It could be a little tidier with the repetition removed:

# https://gitlab.com/gitlab-ci-utils/lighthouse
lighthouse mr:
  needs:
    - test build
  image: registry.gitlab.com/gitlab-ci-utils/lighthouse:latest
  stage: post
  variables:
    BASE_URL: http://localhost:3000
    CHROME_FLAGS: "--headless --disable-dev-shm-usage"
  script:
    - mkdir -p tmp
    - serve ./public/ > tmp/serve.log 2>&1 & wait-on http://localhost:3000/
    - lighthouse ${BASE_URL} --chrome-flags="${CHROME_FLAGS}"
    - lighthouse ${BASE_URL}/photos --chrome-flags="${CHROME_FLAGS}"
    - lighthouse ${BASE_URL}/projects --chrome-flags="${CHROME_FLAGS}"
  rules:
    - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH
  artifacts:
    when: always
    paths:
      - localhost*.html
      - tmp

The job above, lighthouse mr, runs against the changed version of the site in the CI build environment. There's a separate one which runs on deploy to record results for the published version, so I have a reference for what's live as well.

As a side-effect of looking at this error, the site also has 10KB less CSS thanks to minification when the TailwindCSS job runs. 🎉

Visidata, a CLI powertool for CSV

2024-06-28T00:00:00Z

I've been messing around with spreadsheets a little more than usual the last week or two, and popping between shell environment and a spreadsheet was making me feel like I was shifting gears noisily. Spreadsheets are amazing tools, but when I'm happy building in some kind of flow, switching in and out of spreadsheet interface harshes my buzz. My IDE meanwhile: great at many things, but not tabular data.

I decided to give Visidata a spin, and found it delivered what I wanted - a quick, terminal-based editing interface to the tabular data world that didn't feel like too much of a shift from the shell. I had to look up the command basics, but that was not dissimilar to when I'm using a regular spreadsheet - I'm an infrequent visitor!

Soon enough I found an Interesting Task. I wanted to add an additional column filled with UUIDs, so that I could create content entities with a known reference and then round trip data in and out of the CSV. With Visidata, I was delighted to find I could use familiar UNIX tools to add myself a column of UUID to a CSV. My favourite solution the one I found in Visidata's help:

z; expr  create new column from bash expr, with $columnNames as variables

I entered z; then uuidgen, and Visidata added two new columns: uuidgen stdout (filled with UUIDs) and uuidgen stderr (empty). I renamed uuidgen stdout (select, ^, enter new value) and deleted the additional column (select, - I think?) and was on my way.

Nice and easy. Visidata looks like a great tool - will definitely keep it available, as it seems like it has a ton of potential for slicing and dicing data in this format and many more. Looking forward to exploring more!

Fond memories of my Mum using Visicalc on a TRS-80 ...

A UUID generation function is available in both LibreOffice Calc (=GUID()) and Google Sheets (=UUID()).
for ii in $( seq 20 ); do uuidgen ; done, copying the output and pasting into a blank column works fine too.

Oversharing with NPM

2024-06-19T00:00:00Z

Last weekend I released @xurizaemon/eleventy-immich. I used Github Actions to trigger an NPM release from a pushed tag, and after making the initial release with an arbitrary version of v0.1.0, I reviewed the @xurizaemon/eleventy-immich@v0.1.0 release page on NPMJS.

The "unpacked size" value was 28.5MB, and I realised I'd made a mistake.

I'm more familiar with Composer releases, and when you release a Composer package the built archive is constrained to the sourcecode of the tagged release. If you wanted to release a Composer package with some generated files - say, translations from an external source, or built/compiled CSS or JS, my understanding is that you need to build those additional files first and commit them to your Composer codebase.

NPM doesn't have this limitation; the release process can include build steps, and the release archive can include build-time results.

My package's build steps included running tests, and the tests performed end-to-end integration tests against an Immich instance I host. That 28.5MB of files included about 5KB of sourcecode, and some additional files. This counts as "oversharing", and I now needed to review whether I'd inadvertently shared anything I shouldn't have.

The test images weren't my concern. I'll document how the tests work in a more appropriate location, but I had good confidence that the images themselves weren't sensitive; the tests retrieve specific images individually and from a specific album. My concern was other details, such as the API keys used for the integration, and any other details I might find in the built archive. Also, I find being interested in problems is a really effective way to approach them, so my inclination was to learn more rather than to panic.

By the time I'd identified my mistake, I could also see that there were 9 downloads of my newly released package. How exciting! Do I think there were nine people that stumbled onto an interesting NPM package and typed npm install @xurizaemon/eleventy-immich straight away? No, I don't. Automated systems are watching new releases (and new Github pushes) for "interesting" details all the time, and in this world "interesting" means "profitable". I have to assume that any details shared will already have been seen by someone with potential for malicious use.

First action

My first action was to correct my mistake by adding some .gitignore entries (which NPM respects) to the project, and releasing v0.1.1 at a much more reasonable ~5KB in place of the previous 28MB.

Assessing the damage

With a replacement release out the door, my next step was to review what I'd shared.

Let's grab a copy of the files I published:

chris@thip: ~$ mkdir /tmp/throwaway && cd /tmp/throwaway
chris@thip:/tmp/throwaway$ npm init -y
# output skipped
chris@thip:/tmp/throwaway$ npm install @xurizaemon/eleventy-immich@v0.1.0

added 1 package, and audited 2 packages in 2s

found 0 vulnerabilities
chris@thip:/tmp/throwaway$ cat package.json
{
  "name": "throwaway",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "@xurizaemon/eleventy-immich": "^0.1.0"
  }
}

So, what made it into the release?

Firstly: does the downloaded files approximate what I think got released? Yes, it does.

chris@thip:/tmp/throwaway$ du -sh node_modules/@xurizaemon/eleventy-immich/
28M	node_modules/@xurizaemon/eleventy-immich/

Initial inspection

Let's have a look at what I need to explore. tree -da will show me the directories of a directory:

chris@thip:/tmp/throwaway/node_modules/@xurizaemon/eleventy-immich$ tree -ad
.
├── .cache
├── .github
│   └── workflows
├── .idea
│   ├── codeStyles
│   └── inspectionProfiles
├── public
│   └── media
│       └── img
└── test

10 directories

Looking at the file contents as well: tree -a

Click to see output

chris@thip:/tmp/throwaway/node_modules/@xurizaemon/eleventy-immich$ tree -a
.
├── .cache
│   ├── eleventy-fetch-13d629a2fb0ab7cea5b05589667f7e
│   ├── eleventy-fetch-13d629a2fb0ab7cea5b05589667f7e.buffer
│   ├── eleventy-fetch-317aeed09661ed23b93071dff930bd
│   ├── eleventy-fetch-317aeed09661ed23b93071dff930bd.json
│   ├── eleventy-fetch-67adde77a89e4672d3ac77c7313615
│   ├── eleventy-fetch-67adde77a89e4672d3ac77c7313615.json
│   ├── eleventy-fetch-77be3b5f8c2ec56bb3890fa6619ba6
│   ├── eleventy-fetch-77be3b5f8c2ec56bb3890fa6619ba6.json
│   ├── eleventy-fetch-7c85f08df0509d9d7825bf367d12d3
│   ├── eleventy-fetch-7c85f08df0509d9d7825bf367d12d3.json
│   ├── eleventy-fetch-a1fb53576fce4640adb4176510b7af
│   ├── eleventy-fetch-a1fb53576fce4640adb4176510b7af.buffer
│   ├── eleventy-fetch-a4f2ce9274ea106198f56d6355e3cc
│   ├── eleventy-fetch-a4f2ce9274ea106198f56d6355e3cc.json
│   ├── eleventy-fetch-ab5c0e42fecc55d7316494c515354d
│   ├── eleventy-fetch-ab5c0e42fecc55d7316494c515354d.buffer
│   ├── eleventy-fetch-b2e62be056ad837dbc96ebca8fb4ac
│   ├── eleventy-fetch-b2e62be056ad837dbc96ebca8fb4ac.buffer
│   ├── eleventy-fetch-b486d1283d68c1996d39e5c71be87f
│   ├── eleventy-fetch-b486d1283d68c1996d39e5c71be87f.json
│   ├── eleventy-fetch-c4f8f26c84ede2b465b604152680b1
│   ├── eleventy-fetch-c4f8f26c84ede2b465b604152680b1.buffer
│   ├── eleventy-fetch-c8a1319796be03a4a3a8fb0f82bd02
│   ├── eleventy-fetch-c8a1319796be03a4a3a8fb0f82bd02.json
│   ├── eleventy-fetch-f37c2de21a89f407822b63fbaa5f95
│   ├── eleventy-fetch-f37c2de21a89f407822b63fbaa5f95.json
│   ├── eleventy-fetch-f85c0fb9f6a0fd9713efd6f89f2eee
│   └── eleventy-fetch-f85c0fb9f6a0fd9713efd6f89f2eee.buffer
├── eslint.config.mjs
├── .github
│   └── workflows
│       ├── nodejs.yml
│       └── npm-publish.yml
├── .idea
│   ├── codeStyles
│   │   ├── codeStyleConfig.xml
│   │   └── Project.xml
│   ├── eleventy-immich.iml
│   ├── GitLink.xml
│   ├── inspectionProfiles
│   │   └── Project_Default.xml
│   ├── modules.xml
│   ├── php.xml
│   └── vcs.xml
├── immich.js
├── LICENSE
├── package.json
├── public
│   └── media
│       └── img
│           ├── 6F6xc8RT4F-300.jpeg
│           ├── 6F6xc8RT4F-600.jpeg
│           ├── 8wDxzJOqow-300.jpeg
│           ├── 8wDxzJOqow-600.jpeg
│           ├── AitV6V6-2X-300.jpeg
│           ├── AitV6V6-2X-600.jpeg
│           ├── heH3xr0I7G-300.jpeg
│           ├── heH3xr0I7G-600.jpeg
│           ├── oupuEcpEIs-300.jpeg
│           ├── oupuEcpEIs-600.jpeg
│           ├── rDX8xf_N04-300.jpeg
│           └── rDX8xf_N04-600.jpeg
├── README.md
└── test
    └── test.js

10 directories, 56 files

Reviewing the files on the surface

What can I see from filenames alone?

.cache/ contains filesystem caches generated by eleventy-fetch. These requests were authenticated, so i need to review them for sensitive data - eg a cached request header.
.idea/ indicates use of JetBrains tooling: My IDE (family) is revealed.
- An IDE plugin's configuration is present, revealing the use of GitLink. (Great for right-click and share link to colleague, btw.)
- Some other configuration is present in the .idea/ directory, eg code formatting configuration.

Inspecting file contents

And inspecting the files, what else can I see?

.cache/ entries:
- Each cache entry contains a pair of files: One with JSON metadata (eg .cache/eleventy-fetch-7c85f08df0509d9d7825bf367d12d3) with type and timestamp, and another with the cached response (eg .cache/eleventy-fetch-7c85f08df0509d9d7825bf367d12d3.json) with the retrieved data.
- For each of these pairs, the cached file is either a .json (Immich API responses) or a .buffer (Immich asset downloads, ie image data - these can be identified with file command, then opened as that file type).
- .cache/*.buffer file contents are already public; it's the same handful of images shared as an example from my page on the initial plugin concept.
- .cache/*.json entries are copies of the Immich API response and do reveal additional data. I can see:
  - My Immich account email (predictable), personal name, avatar colour (purple!)
  - Imported file filenames (filenames may leak metadata)
  - Exif data, including geolocation data. Fortunately I selected film scans as my test images, and only location tagged a single photo taken at Long Beach.
  - Immich "people" identification results for three people (myself and two family members), revealing the personal names I've tagged those faces with.
  - Immich "smartInfo" tags on a single image (["loupe", jeweler's loupe"] on the Dunedin Farmer's Market selfie - it's taken with a 170-degree lens, and Immich sees the bulbous shape of a magnifying glass). Interesting, and potentially revealing to see tags, but nothing to see here really.
public/ entries contains the set of images again, scaled to 300w and 600w.

Is any of this a worry to me?

Is there any of that I'm concerned about? Not really. There's a little extra sharing, but very little of concern. It has got me wondering whether using different emails to sign into a service would add any additional security to my accounts, but not the best return for effort versus other account protection strategies.

What other options did I have?

I could have unpublished the release (npm unpublish @xurizaemon/eleventy-immich@v0.1.0) from the registry. If I had been seriously concerned, I would have looked to that. However: at the time I knew of the error, I had already seen there were nine downloads. Unpublishing the release would not have addressed those, along with the fact that I'd supplied the release tarball to a couple of organisations along the way: Github and npm, Inc. I assumed seven of the nine downloads were malicious actors and moved on to threat assessment.

Shared, not redacted

Instead, I figure I should add a copy of the released files here as a reference: eleventy-immich-0.1.0.tgz

Or, you can get the original URL via npm view @xurizaemon/eleventy-immich@v0.1.0, or browse the v0.1.0 source files on npmjs.com

Newcombe 35 - a custom theme for Eleventy

2024-06-15T16:13:00Z

The image below shows "35mm Photo Technique" by H.S. Newcombe, a photography technique book first published in 1946. It was a whimsical purchase from a second hand store, after having the plates for depth of field catch my eye. My copy is the fourteenth edition, from 1960. It's a delight.

This photo of the cover doesn't show well the giant embossed "35" that the book sports.

The plates which first caught my eye in the store were the depth of field charts for various lense sizes - several pages of dense black ink with reversed white lines on paper.

There's rich information through the book, presented clearly and at a good level of amateur technical detail, sufficiently that it's very relevant to using a film camera today. For a book first printed in 1946, I find its relevance and utility in 2024 impressive. The technology of film is explained clearly and in ways that I find reveal a clear explanation.

For example, while using cameras recently I had been trying to envisage how an accurately timed exposure of up to 1/1000th of a second could be constructed using mechanical components in such a small space. The descriptions of various shutter mechanisms (pp 25-31) do a great job of outlining several common implementations in ways that I can understand by reading and thinking about the underlying technology. It's delightful and educating.

Anyway. I'm a bit in love with this book visually, and I decided it would be nice to use as the inspiration to style this site. Until now the site has been whatever the default look of a Tailwind CSS site is - and that didn't feel like much of a contribution. So!

Newcombe 35 is the style of this website

First things first: I'm no typographer. This probably won't look polished. It also probably won't look much like the printed book, since this is a website. My style of writing is far less dense - I use plenty of single-sentence paragraphs, H. S. Newcombe didn't have emoji at his disposal, there's no fixed paper size ... it's very much an "inspired by ..." thing I'm doing here. This design deviates substantially.

I've selected three fonts currently, all free fonts provided by The League of Movable Type (go check them out):

Ostrich Sans for the title, navigation etc - similar to the typography of the author and press name on the book cover; the exterior branding.
League Spartan for section headings and the "table of contents" on the front page
Fanwood for body text

I've implemented this where possible by customising the existing Tailwind-based theme I was using. I've used Tailwind on a few projects, but largely as a way to avoid doing much theming, by accepting a base design that was good enough. I'm finding it works OK, with a definite balance of making some basic things (line height and text width overrides) slightly less direct, then finding that the rest of the design tends to hold together well as a result.

A good time to ask myself then: Am I sold on Tailwind? It's ... working out, I guess. I do feel like I'm flying under the envelope of where Tailwind might pay off - if you had a team of designers and needed to keep a coherent alignment with lots of moving components. That's definitely not a problem I face here!

Highly legible fonts

2024-06-13T17:13:00Z

A conversation this evening reminded me that there are a few great fonts around that really assist legibility, that I appreciate having discovered these tools, and that others might appreciate them being shared on!

Here are a few fonts I appreciate for a legibility boost. These are not the fonts I've used on this site (see Newcombe 35 for that). I often use the first font below, Atkinson Hyperlegible, as the font for reading in my web browser.

Atkinson Hyperlegible

This is my preferred font for reading. The font focuses on letterform distinction to increase character recognition, ultimately improving readability. It aims

Where letters might otherwise be confusingly similar - especially in a reduced or low vision context - Atkinson Hyperlegible uses design techniques to ensure letter forms are more clearly distinguished.

https://brailleinstitute.org/freefont

B612 by Airbus

Aircraft cockpits and displays demand minimal chance of confusion, for which Airbus created B612, a highly legible open source font family designed and tested to be used on aircraft cockpit screens. Airbus worked with Intactile Design to produce an "Aeronautical font", to improve the display of information on the cockpit screens, in particular in terms of legibility and comfort of reading, and to optimize the overall homogeneity of the cockpit.

https://b612-font.com/

OpenDyslexic

I'll be honest: this one doesn't work for me. It does work for some, so I'm including it here. OpenDyslexic is created to help with some of the symptoms of dyslexia.

I see the design features which support those reading with Dyslexia. While I appreciate what it's doing, I find it too much for me, and that the font is less legible - and that's fine, since I guess it's not designed for my needs. I'm glad it exists and works for some folks.

https://opendyslexic.org/

Extending your access - JWT validation failure and auth bypass

2024-05-18T05:27:00Z

I collected my first HackerOne bounty just recently, and it's disclosed now which means I can share here too!

Bug #2472798 on HackerOne

The bug is fairly simple: the plugin implementation omitted to validate the signature of the provided JWT, which allowed supplying arbitrary data to authenticate as another user. There were a few mitigations to consider.

I spotted this issue on Github and thought it would be fun to try out the HackerOne reporting process. I had fun establishing clear steps to repro and capturing a reasonably clear video walkthrough.

Thanks to the Automattic security team for the quick turnaround and for rewarding the report with a bounty, which I intend to spend on some fun music toys!

The report

Summary:

The Newspack Extended Access plugin omits to validate JWT signing on the registration and login JSON endpoint. This permits registration of accounts with arbitrary (user-supplied) details, and auth bypass and account hijack if a target account email is known.

Platform(s) Affected:

Any website using Newspack Extended Access plugin.

Steps To Reproduce:

Create an unsigned JWT containing payload value {email: "target@example.org"}. Use a browser to supply this data to the Extended Access registration endpoint. Browser will be authenticated as the target user.

Alternative attack path: use lack of validation to create new accounts with "Customer" role via same endpoint using untrusted inputs. Potential for malicious inputs or DoS through unprotected user creation endpoint.

Requirements

Site installed and configured with Newspack Extended Access (+ dependencies incl WooCommerce, WC Memberships, Newspack).
Target user email is known (for account hijack)

Notes:

For account hijack, target user must be registered via SwG / Extended Access initially
For account hijack, target user is not Administrator or Editor (but may be any other privileged or unprivileged role)

Reproduction (account hijack)

Be logged out of the target website
Create a JWT token with value email set to the target account email. This can be done using the website https://token.dev - the email value is the only significant input for account hijack. Copy the resulting token value.
Visit the target website and use the following code in browser console to authenticate as the target user:

// Endpoint URL
let url = `${window.location.protocol}//${window.location.hostname}/wp-json/newspack-extended-access/v1/google/register`;
// JWT contents - this JWT contains email "test@example.org".
let token = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwiZW1haWwiOiJ0ZXN0QGV4YW1wbGUub3JnIiwiaWF0IjoxNzEzNjY2NjQ5LCJleHAiOjE3MTM2NzAyNDl9.I8D18nWsn5H6AylwJdak8727APyiMCWkcnXH95vMF_k";
// Provide token to authentication endpoint.
fetch(  
    url,  
    {  
       cache: 'no-store',  
       method: 'POST',  
       headers: {  
          'Content-type': 'text/plain',  
       },  
       body: token  
    }  
).then(response => {  
    console.log(response.json(), 'response');  
})

The browser will now be logged into the target account. Personal data (eg the target user's additional account details, billing address etc) will be visible to the attacker.

Reproduction (account creation)

Submitting new accounts via the above method is also possible. Accepting untrusted user submitted input here may allow for additional attack paths when user details are displayed to administrator. Potential for abuse of unprotected submissions via the endpoint would likely permit additional attacks by creating excessive user accounts.

Supporting Material/References:

JWT is accepted without validation of JWT signature. Ref https://github.com/Automattic/newspack-extended-access/blob/3daaae99d98d59dfed78ec68e6ba42c358ca7e82/includes/class-rest-controller.php#L81-L87

Mitigation: Newspack Extended Access prevents sign-in via this method if the user has role "Editor" or "Administrator", but does not prevent sign-in if the user has other privileged roles eg "Shop Administrator". Ref https://github.com/Automattic/newspack-extended-access/blob/3daaae99d98d59dfed78ec68e6ba42c358ca7e82/includes/class-rest-controller.php#L96-L99

Mitigation: Newspack restricts authentication to user accounts which have metadata indicating that Signin with Google was used for initial sign-in. Ref https://github.com/Automattic/newspack-plugin/blob/3daaae99d98d59dfed78ec68e6ba42c358ca7e82/includes/reader-activation/class-reader-activation.php#L681-L690

Impact

Registration of accounts with arbitrary (user-supplied) details
Authentication bypass if the target account email is known
Injection of untrusted data into user profiles

Preparing a video from OBS to MP4

2024-05-18T05:27:00Z

Filed a report, and the receiving party suggested to check for any sensitive details in the report before disclosing.

Included a short screencast and sure enough on re-watching I saw my browser bookmark toolbar was visible. No disclosure beyond three letter project codes, but decided to obscure that.

The location on screen to be redacted was static through the video, so I could use a static overlay to address it.

I extracted a single frame from the video to capture the image I needed to overlay: ffmpeg -i extended-access.mp4 -ss 00:00:05 -vframes 1 browser-chrome.png

Then I found that if you need specific timing, VLC's "Video > Take Snapshot" is good for this too!

I copied that image into an image editor, selected the toolbar and blurred it, inverted selection to delete the rest of the image, and saved it as a (mostly transparent) PNG.

In OpenShot I added the PNG as a project resource and inserted it as a new, frontmost layer in the video timeline.

I then scrubbed through the video and deleted (a little messily) the sections where the blurred overlay did not need to sit.

Exported as extended-access-source.mp4

By now I'd realised that the original recording was SUPER QUIET, my apologies to the person who had to turn their speakers up to hear my voiceover. Let's fix that too! FFMpeg will do it using an audio filter ("af").

ffmpeg -y -i extended-access-source.mp4 -af "volume=50" extended-access.mp4

From that I realised that I should learn how to make a set of subtitles for the video also.

How do we feel about Whisper? It's OpenAI so it feels a bit gross, but I can use it to produce subs pretty smartly from the MP4, and they are not bad.

whisper extended-access.mp4

I could use --output-format srt but I can also omit that and Whisper will turn out several different formats, which is fine. Then I watched through the video with the .srt open, made a few small changes, and away we went!

Here's the resulting video: Extending your access - JWT validation failure and bypass.

Drupal South + Potato Week

2024-04-01T10:45:00Z

A couple weeks ago I took leave to travel to Sydney for Drupal South, present a session on Recipes, and enjoy the community and culture that Drupal South makes possible.

I do feel so lucky to be part of this community. Especially these days, when travel is less frequent and things feel more isolated down here in Dunedin, I value highly the chance to be there, have personal conversations with people I interact with online, and soak in the ideas and vibes of the people who I share the space with. I also miss the folks who aren't there, or have even moved on. That's all a good feeling.

Often I've come back from a conference excited, tired, and dropped myself straight back into everyday workload. That is a terrible idea! What happens is all the amazing ideas and realisations get washed over by day to day work challenges, and between post-conf exhaustion and whatever's in your inbox on return, I find a lot can be lost. This year, the week following the conference was a short one in my region - Otago Anniversary Day and Good Friday had collaborated to set up a three day week. I wrote "Potato Week"^[1] on the calendar, booked that as leave, and took the time to let myself relax and reflect afterwards. (And to improve a few things around the house. It's been a great break!)

So - hmm, some thoughts on the conference.

A talk on Drupal Recipes

Recipes is a new tool for delivering configuration (more to come) to Drupal sites. It's exciting as it targets both a recurring need for site operation, and addresses a challenging specific detail of Drupal's current stack. I wanted opportunity to understand this better and even contribute to its progress, but didn't have a work opportunity to explore it in depth. I assigned myself some Conference Driven Development, where you propose a talk on a topic you don't yet know much about, then are obliged to learn enough to deliver a talk on the day. That was my first in-person conference presentation in a few years, and it was great - the room was nearly full and the feedback was engaged and positive.

Recipes aims to address a challenging feature of profile development, and I was happy to see Heike and Dave from Sparks were interested to see what that might offer the Sector Drupal distribution. If you're keen to explore Recipes more, let's chat!

Leaders dinner

There was a Drupal Leaders Dinner the Tueday before the conference, and I'd had opportunity to attend this the previous year in Wellington as well, due to my role on the Drupal South Steering Committee. The attendees were all leaders or decision makers in their respective organisations, including representation from lots of local agencies as well as Tim Doyle of the Drupal Association and Dries Buytaert the It's a great idea, based I think on the Drupal CEO dinners of Europe, and there's a really good energy of enthusiastic sharing and collaborative exploration between businesses. The discussions were interesting, the menu was amazing, and Mike kept things rolling with the Drupal Agency Survey results presented to the group. Valuable insights!

Something I noticed here especially, and which was tangible at the conference as well, is the strength of collaboration between industry businesses in Australia - where Drupal is doing great things! Australia's contribution game is strong, and the industry there seems really healthy. I love that, and I'm interested to know what we can do to develop this further in Aotearoa as well. (If you have thoughts, get in touch! Are there ways that the community could facilitate this better?)

Contribution cultivation

DrupalSouth has a sprint day, and while I was only able to be there the first half day, it was a great opportunity to share knowledge, assist debugging, and spin the flywheel a little on the tasks in my contribution queue. This spurred a few folks to respond in kind on the issues touched, which is a good return; recognising the multiplier effect available here, some reflections for me to study on cultivating contribution from community.

Resonances

I'm reflecting on the folks I know in the community who I resonate with, and on what.

What am I focused on? What tech did I come away with in my mind?

Areas I'm enjoying myself in development at the moment are -

Data transformations and Migration - I enjoy the "puzzle" aspects of sourcing and data transformations, and I have a fun personal project that I'm regularly knocking out new migrations to feed data into. I've also (post-conf) been checking out some additional tools such as Airbyte, Meltano, and Puppeteer with a mind to treating the web as a programmable entity - as it ought to be.
Testing and automation - I love making things reproducible, it makes my inner six year old very happy - and have been exploring a few tools outside my usual toolkit to survey the landscape. I enjoyed the session on Cypress and Drupal at DrupalSouth, and I've been testing out Puppeteer and n8n since then, comparing them with my more familiar toolkit. I really liked Huginn in the past as well, and I might spin up an instance of that soon.
For similar reasons, I'm keen to get a better handle on the various testing functionality available in core and contrib. I can usually muck my way through but I want to feel more confident and at home with it. Especially when Gitlab CI is something I'm really comfortable and confident with!

And more ...

There's much more to say, but I feel like I've infodumped a lot here already and I can't cover all of it! Personal reflections such as the resonances we discover in realising that someone else just is really into that thing you're into, huge and important topics such as how institutions can find deep trust in how their data is shaped and stored by the systems they select, and complicated things such as what it means for a software community to be community led, or to find common goals. Maybe I'll come back to those.

I'm grateful to have had the chance to be there - thanks team! Looking forward to next time - and to all the connections in between.

If any of this hits a note for you, reach out - you know where to find me :)

I really don't know where "Potato Week" came from but I may hang on to that too 🥔 ↩︎

Let's get cooking: Drupal Recipes

2024-03-22T03:45:00Z

This blog post has images of the slides and my notes. The format isn't perfect but I hope it's helpful to follow along or catch up later!

Tena koutou, thanks Kelvin for that presentation, thanks everyone for bringing yourselves along. I hope you’re all enjoying a great first day of DrupalSouth 2024!

Kia ora, I’ve come from Ōtepoti Dunedin, in the south of Te Waipounamu, Aotearoa New Zealand where I live. That’s my family in the top right - Saira, Hunter and Rowan.
I’m really grateful to be here today and part of this community, Drupal’s made a big difference to us.

I love data, and I think a lot about how we care for our own data. I also love throwing balls, and drawing, and learning music at the moment.

I’m a Senior Developer at Catalyst, where I enjoy some interesting and challenging projects in publishing, education and government as part of my work.

My name is Chris Burgess and you’ll find me online as @xurizaemon.

First up - some caveats.

I think the Recipes project work is really exciting! It’s an opportunity to expand what Drupal is to a broader pool of people.

Recipes is unfinished, not incomplete, things may change. And it’s open source, so things may change.

Recipes also does not try to deliver everything you can do with a Profile. That’s intentional.

I am just learning about Recipes myself, and I don’t want you to get the impression I think I’m an expert. Let me know.

Recipes is a way to apply composable configurations (and eventually content) to Drupal sites.

Drupal has twenty years of - a long history of success and lessons in empowering creators to re-use functionality with tools such as distributions, profiles, modules, and more.

Recipes are a new take on that, based on what we’ve learned so far.

Recipes promise to introduce less complexity to a site, and to be a less significant investment.

What was before Recipes? Lots, especially Profiles.

Profiles have been a cornerstone of Distributions, a great tool, a really valuable contribution to Drupal. Profiles are not the only method to deliver configuration to a suite of sites, but they are one of the leading solutions.

One challenge of Profiles in Drupal was they turned out to be a significant commitment for site operators - and profile maintainers - in lots of ways.

Profiles can be effective but their design locks derived sites and the profile together. They can require additional maintenance and affect decisions on both the site operator and maintainer sides, and generally complicate site operation.

So - Recipes are a response to that experience of wanting to deliver configuration to sites, usually a set of sites, without the commitment of tying them together.

How do Recipes offer this? They do less, and they do things differently.

Recipes are declarative: they say “do these things”, without reference to the current state of the system. By design, a Recipe is not informed of the state of the site.

Recipes can do several things.

Recipes may install modules and themes.

This implies they can add requirements for Drupal modules or themes they will install.

Recipes may apply a module’s default configuration, selectively, which means you can opt into some or all of the module’s out of the box config.

Recipes also can provide their own configuration.

And, Recipes can perform config actions. We’ll touch on that shortly.

All this configuration functionality brings new possibilities.

Most of those are features you know from earlier implementations.

What’s most different about Recipes is what it chooses not to do.

There’s no provision for code in a Recipe. It’s declarative, it won’t respond differently if module X is or isn’t installed.

It’s not dynamic at install time, and it can’t deliver behaviour changes at runtime except through the modules and configuration it applies. No hooks.

You can limit the configuration applied - you can opt into some, all or none of an installed module’s config. The Recipe - and the person using it - is responsible for a valid result!

And the killer feature is that after the Recipe is applied, it’s gone. The changes are in the site’s configuration and the Recipe can be forgotten.

At its core, a Recipe is a yaml file to define the recipe. There may also be a composer json file, and a directory of provided configuration.

Here’s a recipe configuration. This is our recipe dot yaml.

At the top we can see the recipe metadata - the name is required, the type (that can be “Site” for a distribution-tier Recipe), a description.

This recipe is a small example, showing an example of a small component which you might combine with others into a more fully featured recipe.

“Install” defines which modules to install; in this Recipe we’re installing Autologout module.

“Config” defines which configuration to apply.

The “import autologout star” means we’re opting to install all available configuration that Autologout module provides. That would also include config in a config/optional directory, if the module has it.
Instead of a single “star” value, we could explicitly provide an array of specific config items to import from that module. So we might say autologout dot settings there.
If we omitted that autologout module from the config entries, we should expect only the simple configuration items from autologout to be added, by default omitting configuration entities and optional configuration.

Then we’re using a config action - via the new Config Actions API - to apply a simple config update that changes the autologout timeout from the module’s default half hour to a full hour.

This recipe uses the module defaults as a start point, and changes the single value we want changed. Tidier.

A Recipe may also contain a directory of configuration to be imported.

If you want to replace or preconfigure an entire configuration, this is the way to do it.

If you aren’t changing much, I suggest using simple config update action to set only the values required, and accept the defaults elsewhere.

Let’s look at the recipe’s composer.json next.

After unpacking this - using a composer unpack command - into the project, the project’s composer.json takes responsibility for these requirements and the recipe can be removed.

We see a type of drupal-recipe here, similar to drupal-module and friends, this tells Composer where the recipe should be placed in the project.

We’ve added a requirement on Drupal Autologout module here as that’s the module we enable. I’ve specified a version constraint because I want to know the config I apply will be valid for the installed version.

That’s a single recipe, kept intentionally small. But I think that’s the right size for that component.

Then you can use these in higher level recipes by stacking them. This tree shows a recipe with components depending on components.

In this way you can build out your preferred start point, and own the lower level details too, and mix and match as appropriate.

One of several ways we’ll use this is to make the initial configuration of a Drupal site much quicker and easier. This puts the options - currently a choice between Minimal, Standard or a distribution - into the hands of the site owner.

It’s nice to save a few hours of clicking, but having a consistent start point will be a bigger benefit for agencies using this than just saving that setup time.

And Recipes can be applied later in the piece too. The pitch there is, if your blog takes off and you need to sell merch, you’ll apply a merch shop recipe.

Recipes is still in development. You can try it out today!

More complete instructions are in that link on the slide.

You’ll want a site to start with. It’s simplest to start with a minimal install, or you can test on an existing site.

The version-specific core patches from the project will make your site Recipes capable.

Composer configuration is required to install Recipes with patches.

When your site is patched and you have a recipe to test, the drupal CLI is used to apply recipes.

After applying a recipe, the changes can be captured from Composer and config export.

With some confidence in juggling config changes, this can be used on a local copy of a real site.

Remember that Recipes are applied, so the core patch and Composer changes don’t need to be retained once you’ve finished applying them.

Recipes does less, and allows Drupal to do more, together with other new functionality.

The Project Browser will be a big part of this. It’ll allow ambitious site builders to discover and build with community provided recipes.

Automated Updates I think is also a big part of what can allow Drupal to be managed by the broad pool of ambitious builders I’d like to see.

And I think that persona of the Ambitious Site Builder is something Recipes fits so well with.

Unreliable narrator

2024-03-09T19:55:00Z

I have been putting together a set of ETL migrations^[1] which capture data points from online systems I interact with day to day. I think of this as a narrator, but rather than telling a coherent story, it observes events that occur without understanding them in context.

Many of the applications and systems we use - social media websites, health and fitness trackers, finance systems, business tools, conveniences and entertainments, all interoperate with us online via data endpoints.

By capturing a stream of data from various sources, we can observe the digital/data footprint that these sources relate to, and visualise it.

Why (re)capture these data points? I became interested thinking of the record of each day which could be reconstructed from services I use as part of regular daily process or daily habits and associated data streams. I do enjoy the challenge of "solving" ETL transforms (puzzles, think Sudoku), so it was easy to keep building from there. The results are entertaining!

I'm really interested in the portability and visibility of data as it relates to our ownership and responsibilities to the same. Consciousness of "our data" may feel like new territory to be plundered, but our stories are what make us who we are. How do these streams of personal digital record interplay with each of our stories?

Whatever we might collect with a handful of sources and transformations will be only a tiny fraction of available data in a sea of analytics and tracking information.

Beneath the surface, in deep, dark data lakes, swamps and oceans, digital systems dream targetable entities of each of us. Computer toucher guardians spill data from place to place, confident in their ability to extract value without harm, echoing the errors of their ancestors.

Physical and digital devices study us to determine how best to feed their appetites. We exist in their realm, perhaps unaware of, perhaps uncomfortably disregarding their watchful gaze.

The animation on this page decontextualises a subset of my own recently captured data. I wouldn't dare expose the feed directly ... here I'm teasing, baiting those predatory devices which crawl online spaces with a crude depiction of event data I hold about myself.

A sample of my most recent events are depicted, represented by an emoji per service type. What do these images represent, what might they tell you about my day? Why would you care? Machines are hungry for this knowledge - why?

🌅

Events are harvested in a Drupal 10 site via Migrate, stored as entities. JSON API is used to expose these to the build process for the Eleventy site. The Eleventy site retrieves the entities and embeds an associated service emoji and UUID here. There's no need for the UUID. Animations are applied in javascript + CSS but use no source data beyond the symbol animated. ↩︎

Automating commit messages better

2024-03-08T18:56:00Z

I use Git^[1] on the daily and in most contexts^[2] I care a lot about the commit messages being meaningful.

I also have a small handful of repositories where commits are automated (automated data feeds; daily timesheets; my lab journal, etc).

In the case of these repos, I want to track the incoming changes, but do not want the effort of composing a message. There's no benefit to me in manually composing a history entry, either because I'm not present, or because a descriptive change wouldn't add to the record.

(Sometimes that's not true; when I backfilled hundreds of dates into old journal entries, I did write a commit message for the change. But not for when it's happening from cron.)

For a long time I used a git commit -m "journal update on ${HOSTNAME} @ ${DATE}" to automate; it gave me visibility on the origin and recency of the changes, and I could check the history for more information if necessary.

But I want more. I want to be able to look at a commit message and see something more like:

Changed: 2024/03/09.md, Presentations/DS2024/Recipes.md

- 2024/03/09.md:
  - Modified section "Tasks"
  - Added section "Git commit message"
- Presentations/DS2024/Recipes.md
  - Added section "Config validation"

That seems to match what Martin Fowler has referred to as Semantic Diff (like Martin, I ask that if you know a better name, please tell me).

I don't have that part yet. I expect it wouldn't be terrible to implement; in Markdown you'd be looking for the parent "element" (ie, heading), and maybe reporting back on the parent of that if you wanted it.

I see also that are some LLM-based tools which will generate a commit message for you based on staged changes, which is perhaps nice for some but I'll give a hard no on feeding my daily work notes to an LLM for the provider to train against.

But to improve on the lack of information in "journal update on ${HOSTNAME} @ ${DATE}" I decided to at least capture a high-level view. Here's what I came up with.

A Git prepare-message-hook with the following content:

#!/usr/bin/env bash

COMMIT_MSG_FILE=$1
COMMIT_SOURCE=$2
SHA1=$3

ADDED=$(git status --porcelain | grep '^A' | wc -l)
DELETED=$(git status --porcelain | grep '^D' | wc -l)
MODIFIED=$(git status --porcelain | grep '^M' | wc -l)
RENAMED=$(git status --porcelain | grep '^R' | wc -l)
OTHER=$(git status --porcelain | grep -Ev '^[ADMR]' | wc -l)
HOSTNAME=$(hostname)
SUBJECT="Files changed on $HOSTNAME: "
ORIG_SUBJECT=$SUBJECT

function push_subject() {
  if [ "$SUBJECT" != "$ORIG_SUBJECT" ]; then
    echo "$SUBJECT, $1"
  else
    echo "$SUBJECT $1"
  fi
}

if [ "$ADDED" != "0" ]; then
  SUBJECT=$( push_subject "$ADDED added" )
fi
if [ "$DELETED" != "0" ]; then
  SUBJECT=$( push_subject "$DELETED removed" )
fi
if [ "$MODIFIED" != "0" ]; then
  SUBJECT=$( push_subject "$MODIFIED modified" )
fi
if [ "$RENAMED" != "0" ]; then
  SUBJECT=$( push_subject "$RENAMED renamed" )
fi
if [ "$OTHER" != "0" ]; then
  SUBJECT=$( push_subject "$OTHER other" )
fi

echo "$SUBJECT" > "$COMMIT_MSG_FILE"
echo >> "$COMMIT_MSG_FILE"
git diff --cached --stat >> "$COMMIT_MSG_FILE"

Will generate commit messages like this:

commit 7f9cb0ff06afaa5db8733869ca5552c86aa7fc99 (HEAD -> main, origin/main, origin/HEAD)
Author: Chris Burgess <chris@giantrobot.co.nz>
Date:   Sat Mar 9 07:16:04 2024 +1300

    Files changed on thip:  44 added, 1 removed, 23 modified, 1 renamed
    
     2021/03/23/Some Notes.md          | 374 +++++++++++++++++++++
     2023/09/04.md                     |   2 +-
     2024/02/04.md                     | 260 +++++++++++++-
     2024/02/05.md                     | 121 +++++++
     ...

Or this:

commit 36a61c68c8b0031ad556c6be15ffe8399c2903a9 (HEAD -> main, origin/main, origin/HEAD)
Author: Chris Burgess <chris@giantrobot.co.nz>
Date:   Sat Mar 9 07:26:37 2024 +1300

    Files changed on thip:  1 added
    
     2024/03/09.md | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
     1 file changed, 82 insertions(+)

It's not perfect. It doesn't handle git commit -m <msg>, for one thing, and I hope to explore further a diff that is capable of precis-ing a set of changes to Markdown files. But it'll improve the history I see when working with those repos where I'm not writing commit messages for each change.

I'll report back on whatever annoys me about this first quick implementation 😀

If you use another SCM, please substitute yours for "Git" in this. I'm not saying "SCM" here, picking clarity over correctness. ↩︎
On a WIP branch, I will push any and all manner of rubbish commit messages, and rebase later. ↩︎

GlitchTip filtering by log levels

2024-02-25T00:00:00Z

This morning I was looking at some log entries in GlitchTip and went to filter by log severity to see what was being recorded.

GlitchTip, by the way, is an open source error tracking tool. It's pretty fancy, I have a lightweight instance I run to capture debug output from a handful of projects I operate personally. Errors or log messages from applications can be relayed to GlitchTip for review, notification, and other similar features.

I was surprised to see that there wasn't any UI for how to filter down to eg "error" or "debug" level entries. I thought this was existing functionality but was (perhaps of course) thinking of other tools. I also noticed that the level of each message (or issue) was not represented textually, instead showing only as a colour bar to the side of the entry in the UI.

I opened the browser's inspector and copied the API requests out as a curl command, then tried some variations for filtering syntax - ?level=info, ?query=level:info - to see if the API supported it. It didn't appear to. There was no indication of support in the front end either, while I saw other UX hints from autocomplete on environment: etc.

This now had my interest, and I checked out the GlitchTip project Gitlab to see if I could find an issue where this was being requested or worked on. Searching for "severity", "level", "type" led me to glitchtip/glitchtip-frontend#161: Add UI for error levels to issue list page which at least confirmed for me that the colours I'd noticed earlier did relate to the levels of the entries.

I checked into GlitchTip's chat to ask if this was an existing feature request, and was fortunate to have a few pointers from one of GlitchTip's core contributors, David Burke, including an existing issue glitchtip/glitchtip-frontend#185: Feature Request: Filter "message events" out of the issues list, which the keywords I'd tried searching for hadn't surfaced.

Since I didn't know Django at all, I figured the best first step would be to write test coverage for the desired functionality. I made a first attempt at a test, and ran the tests with docker compose exec web ./manage.py test apps.issue_events. I expected my changes to have a heavy foreign accent, not being much familiar with Django or even Python. To my incredulity not only did the proposed code pass GlitchTip's lint checks, but the tests passed as well. I was pretty confident this meant my tests were flawed.

David kindly showed me I had implemented a test which tested the wrong thing - whether the issues were tagged with named levels, rather than whether the issues had levels assigned.

I already had the backend component running locally, and wanted to be able to poke at the API directly with these changes in place to test. Getting an API key however looked like it would require attaching the frontend service component. I had the feeling it might be a diversion but was told this was the easiest way to get an authentication token. Then I found that the frontend setup steps didn't work for me, exiting with a "bus error" each time I ran npm install.

This had me stuck for a little bit, so after talking with David some more about the error I saw, I went and tidied up the yard for a bit, trimmed some edges and mowed some grass. When I came back I had realised that it was unnecessary to set up a build environment at all ... I could download a fresh set of built files from the frontend project's build-assets CI job! So I grabbed those, copied them into the local backend development instance I had available, and was finally able to see the GlitchTip UI.

Now I needed to get a login to the instance. It had come up with an unknown admin email address. A bit of research led me to Django's createsuperuser command, which I used via ./manage.py:

$ docker compose exec web ./manage.py createsuperuser
Email: test@example.com
Password: 
Password (again): 
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.

Once this was done I could log into my local GlitchTip instance and configure it with the required data to get access to a token for testing the behaviour of the client.

I now rewrote the test so it now (correctly, I believe) failed. I then added a few lines to the Issue model to implement the filter, and once more, to my pleasant surprise, the test now passed. I tried a few additional test changes, confirming various changes or inputs could fail the test, to confirm things were working as intended, which it seems like it does. Wow. That was too easy?

The GlitchTip codebase has a rewrite under way to make event and issues APIs more efficient, and David had pointed me at that branch (with some special instructions on how to test it). The tests were passing both locally and in CI, locally with the rewrite changes active and in CI without. I was ready to push the changes for consideration and did so. The resulting MR is at glitchtip/glitchtip-backend!767.

I really enjoyed this first brush with the GlitchTip codebase. I was impressed by the project's clear guidelines in CONTRIBUTING.md, with a good section on code style and philosophy to align contributions. I especially liked -

Economical over completeness. Make running GlitchTip as easy and simple as possible, especially for small and medium sized projects. Be wary of introducing additional external dependencies. The entire project must be maintained on a budget of 4 person-hours per week. When introducing a large new feature, offer to help with maintenance in addition.

I also thought that guidance like what to avoid was a really helpful step, eg:

Avoid: Inefficient database calls - GlitchTip must work for both small self hosters and 100 million event projects. Always assume scale. If you need to edit every user, assume there are 100 million users and the queries much be chunked in batches. Neither one query per user nor one giant query that takes too long to execute.

(See the full CONTRIBUTING.md for other guidance - it's a sign of a healthy setup to me that this has been reflected on, and is at the fore.)

I enjoyed working with the project today and found the project layout and Django's experience really well formed. I'm be interested to learn more about this framework I've long heard of and not engaged until now 🐍 and excited to look forward to a second round of feedback on the MR.

A positive reflection was that when the missing functionality caught my eye, my initial response was one of interest - both that the feature wasn't there, and how it might be implemented. That attitude supported me to take the step to trying a commit in a system outside my usual toolkits, and I appreciate the experience and outcome. If that first mental/emotional had been disappointment, annoyance, frustration or even resignation ... I think I wouldn't have tried half of the steps I did, and that would have limited the outcomes. It's not all attitude, but it sure helps.

Update: A day later, glitchtip/glitchtip-backend!767 was accepted and will be available in the first release of GlitchTip 4.0! 🎉

Eleventy 🤝 Immich

2024-02-12T00:00:00Z

Lately I've been exploring self-hosting personal data, and photo storage, management, discovery and sharing is one service I'm keen to move from a megacorp-hosted option to self-hosted. Several weeks back I gave PhotoPrism a trial, and I'm now checking out Immich. I'm using Eleventy for my personal website and one thing I wanted to test was the ability to integrate the two.

June 2024 update ✨ ⭐

The plugin referred to in this post is now available via
npm install @xurizaemon/eleventy-immich or
https://github.com/xurizaemon/eleventy-immich

February 2026 update ✨ ⭐

I'm not using the plugin below now, I found the benefits of dynamically updating content in this site from that
remote source weren't as great as copying images to this repo's media directory, generating frontmatter, and then
working with markdown here.

Immich permits album sharing, but for public display I prefer to selectively pull some albums or images into my static build pipeline - I'd rather put my faith in a static publishing output than take on the maintenance and securing of a complex service - life is too precious and short. I want to be able to reference photos from my image collection into my writing.

This last weekend I've got a prototype up and running of embedding Immich assets in Eleventy, and here's an update! All images in this post are film photos taken Summer 2023/2024 using a Lomo Fisheye and Orca black and white 110 film.

Shortcodes for `immich_image` and `immich_album`

Here's an image I'm embedding which was sourced from Immich when building this site. A custom shortcode has been used to generate an image tag using the UUID of the image in Immich.

{% immich_image '3d0bd5db-a237-4a5e-8994-6ff91fa62878' %}

[ removed ]

Here's an album via the immich_album shortcode. I haven't attempted to format this in a meaningful way, only to emit some details about the album and then a series of photos. This part will call for templates that can be overridden in the project.

{% immich_album 'ef43ecca-f88d-4f5a-b590-533d0611e045' %}

[ removed ]

Next steps

I implemented a collection so I could iterate through Immich photo albums - that was a start point, but it's removed now as I didn't want to leave example code in place that would reveal updates to the photo albums over time. I will come back to that to implement filtering (ie "show me albums that are marked public or shared").
It would be interesting to provide front matter such as immich_header_image to pull an image in for the page's feature image?
I am keen to experiment with turning the output into .webc components instead of shortcodes.
This can become an installable Eleventy plugin ... but it's not ready yet! Watch https://github.com/xurizaemon/eleventy-immich for updates.
Performance wants review. While I'm using Eleventy Fetch and I think it's caching, the build time has spiked from ~30s to ~2min, and we only have 24 images in this post. The .cache/ directory seems to be getting populated and staying about the same size.
Direct embeds are demonstrated above, calling an image or album into a post by some reference. I'd like to explore configuring a set of albums and having Eleventy build out a section of my website with those pages based on the data retrieved, and with URLs for each album or image media.

Implementation

Here's what I currently have.

Configuration

Right now, the plugin supports only a single Immich instance and user account, via environment variables. Your build environment must be able to make HTTP requests to your Immich API URL. You'll also need an API key from your Immich instance - see /user-settings there to create a key for your account. Set the required variables in your build environment:

IMMICH_BASE_URL=https://immich.example.org
IMMICH_API_KEY=ABCD1abcd1ABCD1abcd1ABCD1abcd1ABCD1abcd1AB

I'm running this by adding ./immich.js to my codebase, next to the Eleventy configuration file. I've copied the code to https://github.com/xurizaemon/eleventy-immich as well - but I haven't even switched my own use to that yet.

./immich.js contents:

/**
 * A plugin to provide Immich shortcodes for Eleventy.
 */

const EleventyFetch = require("@11ty/eleventy-fetch");
const Image = require("@11ty/eleventy-img");

module.exports = function immich(eleventyConfig) {
    let required_vars = ['IMMICH_BASE_URL', 'IMMICH_API_KEY'];
    for (name of required_vars) {
        if (!process.env[name]) {
            console.error(`Immich plugin requires setting the following env vars: ${required_vars.join(', ')}`);
            // @TODO Can I bail out here somehow?
        }
    }

    let immichUrl = `${process.env.IMMICH_BASE_URL}`;
    let apiKey = `${process.env.IMMICH_API_KEY}`;
    let defaultFetchOptions = {
        headers: {
            'x-api-key': apiKey,
        }
    };

    /**
     * Fetches an album and renders each asset.
     *
     * @param uuid
     * @returns {Promise<string>}
     */
    async function immichAlbumShortcode(uuid) {
        let albumUrl = `${immichUrl}/api/album/${uuid}`;

        let albumData = await EleventyFetch(albumUrl, {
            duration: "10m",
            type: "json",
            fetchOptions: {
                ...defaultFetchOptions,
                ...{ accept: 'application/json' }
            }
        });

        let html = `<div class="immich-album"><h2>${albumData.albumName}</h2>`;
        if (!!albumData.description) {
            html += `<p>${albumData.description}</p>`;
        }
        if (albumData.assets.length) {
            html += '<div class="immich-album-assets">';
            for (asset of albumData.assets) {
                html += await immichImageShortcode(asset.id);
            }
            html += '</div>';
        }
        html += '</div>';
        return html;
    }

    /**
     * Fetches image data and file from a given UUID and generates HTML for the image element with specified attributes.
     *
     * @param {string} uuid - The UUID of the image asset.
     * @returns {Promise<string>} - A promise that resolves to the generated HTML for the image element.
     */
    async function immichImageShortcode(uuid) {
        let assetDataUrl = `${immichUrl}/api/asset/assetById/${uuid}`;
        let assetFileUrl = `${immichUrl}/api/asset/file/${uuid}`;
        let fetchOptions = defaultFetchOptions;

        fetchOptions.headers.accept = 'application/json';
        let assetData = await EleventyFetch(assetDataUrl, {
            duration: "10m",
            type: "json",
            fetchOptions: fetchOptions
        });

        fetchOptions.headers.accept = 'application/octet-stream';
        let assetFile = await EleventyFetch(assetFileUrl, {
            duration: "1w",
            type: "buffer",
            fetchOptions: fetchOptions
        });

        let alt = assetData.exifInfo.description ?? 'No image description available';

        let metadata = await Image(assetFile, {
            widths: [300, 600],
            formats: ["jpeg"],
            outputDir: 'public/media/img/',
            urlPath: '/media/img/'
        });

        let attributes = {
            alt: alt,
            sizes: [],
            loading: "lazy",
            decoding: "async",
        };

        return Promise.resolve(Image.generateHTML(metadata, attributes));
    }

    /**
     * Retrieves a collection of Immich albums.
     *
     * @returns {Promise<Array>} A promise that resolves to an array of albums.
     *                            Each album is represented as an object in the array.
     *                            If an error occurs, an empty array is returned.
     */
    async function immichAlbumsCollection() {
        try {
            let albums = await EleventyFetch(`${immichUrl}/api/album`, {
                duration: "1d",
                type: "json",
                fetchOptions: {
                    ...defaultFetchOptions,
                    ...{ accept: 'application/json' }
                }
            });

            for (album of albums) {
                collection.push(album);
            }
            return collection;
        }
        catch(e) {
            console.log( "Failed getting Immich albums" );
            console.log(e, 'exception');
        }
    }

    // eleventyConfig.addCollection("immich_albums", immichAlbumsCollection);
    eleventyConfig.addShortcode("immich_album", immichAlbumShortcode);
    eleventyConfig.addShortcode("immich_image", immichImageShortcode);
};

We add the plugin to Eleventy by modifying the eleventy configuration file, eg this .eleventy.js addition:

module.exports = (eleventyConfig) => {
    // ...

    eleventyConfig.addPlugin(require('./immich'));

    // ...
};

PS. Hopefully you saw the update at the top - this plugin is now available as an NPM module, see https://github.com/xurizaemon/eleventy-immich or npm install @xurizaemon/eleventy-immich - it's still WIP and contributions are welcome!

Yakfilling dates

2024-02-04T00:00:00Z

I have a journal / lab book markdown collection with many entry file paths by date (2015/01/23.md). Only the more recent files also had inline date metadata.

I wanted to backfill missing day properties for these markdown files into the yaml frontmatter. Initially my notes were just text, and over time recorded additional metadata into the entries, so making the date records consistent adds context to metadata already in the files.

Tools such as MarkdownDB, DataView, or SSGs such as Eleventy and Hugo permit querying a markdown corpus for data. To see records for those dates, migrating day "data" from file path into yaml frontmatter made access more consistent.

Thinking lots about data preservation and the maintenance costs of running dynamically generating systems at the moment ...

Here's the bash script I came up with to sort this quickly and not prettily. I tested it out in a copy of the repo (I'm not stupid!) and was delighted with the results.

This really accounts for a pretty simple markdown structure only, making assumptions about the structure of the documents. I know what this specific corpus mess is and isn't and accept the T&C of any breakages.

#!/usr/bin/env bash

SOURCE=$1

if [[ "$SOURCE" == "" ]] ; then
  echo "Usage: add-dates.sh <path>"
  exit 1
fi

find $SOURCE -name '*.md' | while read FILE ; do
  DAY=$( echo $FILE | sed -n -e 's|^.*\([0-9]\{4\}\)/\([0-9]\{2\}\)/\([0-9]\{2\}\).*|\1-\2-\3|p' )
  if [[ "$DAY" == "" ]] ; then
    echo "File $FILE does not match date pattern"
    continue
  fi
  # Use -Pzo to see matched output for testing.
  if grep -Pzl "^---\n(.|\n)*\n---" "$FILE" > /dev/null
  then
    # echo "$FILE has frontmatter."
    if ! grep -E -- '^day:' "$FILE" > /dev/null
    then
      echo "$FILE has no day entry, should be $DAY."
      PROPERTY="day: $DAY"
      echo "Inserting: $PROPERTY"
      sed -i "0,/^---/a $PROPERTY" "$FILE"
    # else
    #   # echo "$FILE has day entry."
    #   grep -E -- '^day:' "$FILE"
    fi
  else
    echo "$FILE has no frontmatter, should be $DAY."
    METADATA="---\nday: $DAY\n---"
    echo "Inserting: $METADATA"
    sed -i "1i $METADATA" "$FILE"
  fi
done

Usage: add-dates.sh /path/to/journal. Because of the layout of my journal (folders with and without date-based naming) I targetted each year one by one, working only on a separate copy so I could review changes and merge via Git when happy.

I hope this script may be useful or interesting for you to build on. This script is No Maintenance Intended :)

It's a pretty crude approach, but it suffices nicely. I was happy to quickly see results like this:

A happy accident of this implementation was that some notes with paths like 2019/02/03/Some topic.md were also correctly dated. A win for UNIX-y simplicty I guess. I'd forgotten that at one point I tried such a layout briefly ... it didn't really work for me, so I'd dropped it.

Feeling grateful for the workshop on the Atom Journal plugin at OS//OS in 2015 which got me to start using date-based filenames!

Whew, 882 modified files!

On review, I found one case that hadn't been handled and which wanted tidying. Where the original markdown file had an empty frontmatter block -

---
---

the result was a duplicate frontmatter entry would be introduced.

I used grep -riPzl -- "---\n---" <path> to locate seven affected files like that, and fixed it by hand-editing. All were from a similar date range ... I didn't bother handling that in the script above.

Gull feet

2024-01-28T00:00:00Z

A gull walks on a canvas roof, seen from below; the gull's feet are orange, while its silhouette against the canvas moves smoothly above the shape of the feet. A second recording of a gull walking on a canvas roof, seen from below; the gull's feet are orange, while its silhouette against the canvas moves smoothly above the shape of the feet.

We stopped at Punakaiki on our Toastie Coastie road trip. I loved this gull's feet through the canvas roof and plan to make some kind of animation based on it.

Send shrugs 🤷

2023-12-10T00:00:00Z

I find Matrix's /confetti, /fireworks, /hearts, /rainfall, /snowfall, and /spaceinvaders commands fun and entertaining (and too noisy to use often).

I've long been fascinated by the ways in which limiting vocabulary affects the way folks communicate and ultimately think. Consider the period when Facebook permitted the singular reaction "Like" to respond to news events, or whatever set of half a dozen emotional quick responses are available to respond to a LinkedIn post. Or Orwell's Newspeak, in "1984", a language created by a totalitarian state to limit the range of thought.

Sometimes - especially in the business world - it's important to express calm acceptance of a situation, and perhaps even celebrate this. For this I thought it'd be good to add a /shruggiestorm command to Matrix.

When I looked into this, I learned the Matrix team are not accepting additional effects (probably for quite reasonable reasons).

I wanted to see it in action; now we can. That button below will rain shruggies for you.

Source (being lazy, just linking):

Thankyou for your interest! For a more immersive experience, enjoy the Shrugs generator page.

Shroomcam 🍄

2023-12-09T00:00:00Z

Your browser does not support the video tag.

We grew some mushrooms! We're having them in tacos tonight.

These are native Phoenix Oyster mushrooms (pleurotus pulmonarius). If you know the right reo Māori kupu I'm interested.

I got the kit from the farmers markets here in Ōtepoti, from Brightside.

🍄🍄🍄

Lando + Memcache + Memcacheadmin

2023-12-06T00:00:00Z

I had not heard of hatamiarash7/Memcached-Admin before today, and got nerdsniped by someone telling me they had trouble setting it up for their local development with SilverStripe. So, here's a recipe.

name: test
services:
  memcached:
    type: memcached
  memcacheadmin:
    type: compose
    app_mount: false
    services:
      image: hatamiarash7/memcached-admin:latest
      command: docker-php-entrypoint apache2-foreground
      ports:
        - 80
      environment:
        # This needs configuring to the hostname shown from `lando info`
        MEMCACHED_HOST: "memcached.test.internal"

This will bring up a small Lando stack with only memcached and memcached-admin, and the only URL output will be for the admin interface. This example is mostly for lifting into an existing project, and as an example of how to utilise a Docker image as part of a Lando stack using a compose service.

Note that MEMCACHED_HOST contains the name of the project in the hostname - this will need updating to match the project it's being dropped into.

Oh and another thing

I also want to give a shout out to how you can overlay configs in Lando, so I can add .lando.local.yml to .gitignore, and then put the "extras" into the local file only. This means I can have a regular project .lando.yml:

.lando.yml:

# Lots of project specific config here.
name: whatever
type: lamp

And then isolate whatever whimsical things I'm doing - eg test out services or tooling in a local sandbox - without it showing up in the main configuration.

.lando.local.yml:

# Stuff I'm messing with and don't want to track in git.
services:
  memcached:
    type: memcached
  memcacheadmin:
    type: compose
    app_mount: false
    services:
      image: hatamiarash7/memcached-admin:latest
      command: docker-php-entrypoint apache2-foreground
      ports:
        - 80
      environment:
        # This needs configuring to the hostname shown from `lando info`
        MEMCACHED_HOST: "memcached.whatever.internal"
# Random example tooling made up to demo adding that as well.
tooling:
  install-netcat:
    service: memcached
    cmd:
      - apt -y update
      - apt -y install netcat
    user: root
  query-memcache:
    service: memcached
    cmd:
      - echo "version" | nc memcached.test.internal 11211
      - nc memcached.test.internal 11211

This to me is a tidy way to stash things in a file that means I can retain a modification locally while hopping between branches. I can copy the day's tooling or services into the main .lando.yml when the time is right.

SensorWatch days of the week in reo Māori

2023-11-11T21:54:00Z

The Sensor Watch is an open source, custom replacement board for the CASIO F-91W and similar watches. I bought one on a whim off Crowd Supply back in the fog of COVID and work that is the recent past. The project is by Joey Castillo / Oddly Specific Obbjects and collaborators.

The SensorWatch permits custom watchfaces, or you can select from a range of existing custom faces, and it comes with a default custom arrangement. The remaining hardware of the watch is unmodified which establishes some interesting boundaries to what's possible - there are three buttons available for operation, and the original CASIO LCD for output.

I'd purchased the board (and an additional temperature sensor, which is on the board in the later "red" edition), but did not do anything with it for eleven months until a recent isolation had me idle for a few days in my room. (I'm always happy to find my former self has sent cool things through time to me!)

The interface of the watch once it has new faces onboard is really interesting to me. At first the interface felt extremely familiar, because it's the three buttons of a CASIO watch, so the actions are familiar. The watchfaces however were unfamiliar - but the familiarity of the buttons makes the interface discoverable. Something I found interesting (via grep -rE "\bEVENT_" . in the source) is that long presses are supported. I got to wondering if combined press events are supported?

But first: it took me nearly a year to swap the board into the F91-w on my wrist. Lazy bones. I knew it required a solder and was timid at first. Eventually when I was in iso a few days I did put the board into my everyday watch, and then a week or two later took it apart again to transfer the piezo connector from the original watch board (which went fine, as it should) and finetune my custom ROM.

One minor change I had wished for was having the days of the week ("MO", "TU", "WE") appear in Te Reo Māori ("HI", "TU, "AP"). It's distinctive and something special that reminds of my place in the world as we spin through the cosmos. The localisation change is extremely minor - a single array needs replacing in the source:

index 9e524762..0225dee0 100644
--- a/watch-library/shared/watch/watch_utility.c
+++ b/watch-library/shared/watch/watch_utility.c
@@ -26,7 +26,8 @@
#include "watch_utility.h"

const char * watch_utility_get_weekday(watch_date_time date_time) {
-    static const char weekdays[7][3] = {"MO", "TU", "WE", "TH", "FR", "SA", "SU"};
+    static const char weekdays[7][3] = {"HI", "TU", "A ", "PA", "ME", "HO", "TA"};
    return weekdays[watch_utility_get_iso8601_weekday_number(date_time.unit.year WATCH_RTC_REFERENCE_YEAR, date_time.unit.month, date_time.unit.day) - 1];
}

Rāhina, Rātu, Rāapa, Rāpare, Rāmere, Rāhoroi, Rātapu.

Hey, why's Rāapa just a single "A"? The original LCD doesn't have separately addressable segments for all of the shapes in the letter. If "AP" is displayed, the LCD shows "AF". Of the possible displays, I felt "A " was clearest and least wrong.

"AF" is amusing and I was tempted to keep it: Wednesday As Fuck. But, it wasn't what I wanted.

I also customised the selection of faces - currently I have simple clock, tomato, countdown (3min), moon phase, prefs, set time. Here's a link to my current custom ROM source.

The SensorWatch is a super cool board and the result is a customisation of a classic simple watch that I'm happy with. I'm really enjoying it and plan on ordering at least one more as a gift :)

It might be interesting to consider a custom face which supports localisation, or even introducing a localisation layer to the main ROM, for a more full featured support.

Update, next day: I'd popped into the Sensor Watch Discord channel at the end of October with a question, and shared my WIP at the time. After writing this post yesterday I see joeycastillo/Sensor-Watch#323 has been created by wesleyac, which brings in a compile-time setting for day language, with the initial options being English (default), Spanish, German and reo Māori. Neat!

Cypress in Gitlab CI

2023-10-28T00:00:00Z

I had a few days downtime and found myself trying out an unfamiliar testing framework, Cypress. I was interested to see how it compared to other test frameworks.

I'd seen folks posting about trouble getting it running in containerised environments - local dev with DDev or Lando, and CI, which interested me.

I figured I'd give it a spin in Gitlab CI. My first attempt was bringing Cypress up alongside the existing environment by pulling Cypress's dependency chain (NodeJS) into the PHP service which the site's tests already ran on. This worked fairly well.

Refer to xurizaemon/commerce-demo!20 for details.

Lando v4 preview

2023-09-30T19:49:00Z

I was fortunate to join a Lando v4 preview call back at the start of October and get a look at what's coming up in v4. Mike Pirog and Alec walked a group of us through some of what's exciting and how it's coming along. Here's my notes!

Lando v3.20 / v4 sponsor preview

The new services API sounds really nice, less bash build steps and more tooling to drop in eg PHP extensions, faster (caching layers instead of writing docker-compose files which require a full build on rebuild)

Orchestrator / Build engine (currently Docker Compose) becomes swappable, opens door for Colima, K8s etc - great news for folks who want that flexibility.

Will still use Docker / Docker Compose syntax generally, translate to other orchestrators & build engines, permits per-project build engine / orchestration (if desired).

Permits using full users instead of user mapping, decouples recipes more from current base images (eg bitnami/* for DBs).

Lots of existing but prev undocumented switches now documented in https://docs.lando.dev/core/v3/lando-service.html

Much nicer debug output (eg seeing results of failed healthchecks directly, debug output shows full output).

Stealth launch today of WIP v4 docs https://docs.lando.dev/core/v4/

lando service type looks great - it's compose service but image key can be any of: local dockerfile path, inline dockerfile, registry image, or configurations (eg "php type, but add xsl" only without copypasta of apt steps).

Version becomes easier to swap cos you can update services to Lando v4 individually - sounds good for maintenance, no big jumps.

Couple mentions of "Lando Team" for shared/common config - I didn't catch what this product / offering is, but it sounds interesting. It makes sense for Lando to be offering this sort of value extension.

Service groups (eg share configuration chunks like "get abc php extensions" between several services) could help tidy up some of our currently inheriting configurations.

All in all - looking really good, and grateful to the Lando team for sharing!

Convert MR patches to local patches

2023-09-12T20:48:00Z

Many projects use pull or merge requests to collaborate on fixes to software components. Tooling can enable us to apply these patches during build phase, so we can say "Use some/module v1.2.3, but apply the fixes I need from issues #4 and #5 as well".

Collaborating on these changes is great, and it's helpful to be able to pull patchs from patch URLs (Github, Gitlab, Drupal docs) directly when things are in a state of lots of change. You might need to add additional fixes as you go, and tracking that with a local patch copy is a bit of extra work.

It's convenient to use the patch URLs when things are very dynamic, but that adds some risks - other collaborators can push changes which break things unexpectedly, and an external actor could even introduce malicious code to your builds when you're pulling unvalidated code from the internet like that.

So, before things start to "set", you want to turn those URLs into local copies of patches, which aren't mutable by external parties. To that end, here's a quick PHP script to convert remote MR patch/diff URLs to local patches.

This example is hardcoded for a build using cweagans/composer-patches, with a filename of composer.patches.json and merge requests from Drupal's Gitlab, but can be readily adjusted to suit your needs.

#!/usr/bin/env php
<?php

/**
 * For remote patches which are at mutable URLs, we make a local copy.
 *
 * This counters risk of breakage or attack via update to an open MR.
 */
if ($config = json_decode(file_get_contents('composer.patches.json'))) {
  foreach ($config->patches as $project => &$patches) {
    foreach ($patches as $description => &$url) {
      if ($parsed_url = parse_url($url)) {
        if (isset($parsed_url['scheme']) && stristr('git.drupalcode.org', $parsed_url['host'])) {
          $prefix = preg_replace('#/#', '_', $project);

          $filename = preg_replace('/[^0-9a-zA-Z_]/', '-', "{$prefix}-{$description}");
          $filename = preg_replace('/-+/', '-', $filename);
          $filename = trim($filename, '-');

          $filename = "patches/{$filename}.patch";
          $cmd = escapeshellcmd("curl -sSo {$filename} {$url}");
          if (shell_exec($cmd) == "") {
            print "Retrieved {$url} to {$filename}." . PHP_EOL;
            $url = $filename;
          }
          else {
            print "Unable to retrieve {$url} for project {$project}." . PHP_EOL;
          }
        }
        // OK to skip; it's not a remote URL.
      }
      else {
        print "Unable to parse patch URL {$url} for project {$project}." . PHP_EOL;
      }
    }
  }
  file_put_contents('composer.patches.json', json_encode($config, JSON_UNESCAPED_SLASHES|JSON_PRETTY_PRINT));
}

Debugging a Sieve detail that was confounding me

2023-09-12T20:48:00Z

This Sieve behaviour had been bugging me, I just solved it and needed to share.

Sieve (Wikipedia) is a mail filtering tool. It's brutally effective and if your mail provider supports it (as my work and personal emails do), I recommend it highly to sort your email for you (and more).

However ... a specific filter seemed to be misbehaving. The intended behaviour was:

For emails from Gitlab, if the project path contains "clientname", place the message in my "Projects/Clientname" folder^[1].
Unless I'm explicitly mentioned, mark the message as "Seen"^[2].

Here's what I used to have:

  if header :contains "X-GitLab-Project-Path" [
    "clientname/"
  ] {
    fileinto "INBOX.Projects.Clientname";
    if not body :contains "@chrisburgess" {
      setflag "\\Seen";
    }
    stop;
  }

My gut feel was that this worked fine - but I could see evidence (emails marked unread which didn't @mention me) it wasn't working now. A slightly hard bug to be confident about as there's a lot of emails, and it felt like it worked most of the time. I had that Heisenbug feeling. This configuration was still filing Gitlab messages into the correct folder ... but it seemed like it was inconsistently marking items as Seen.

I wondered if Gitlab's emails contained the string @chrisburgess in the headers - not how I expected body :contains to work, but I did start to wonder. I checked the email source and found no matches.

Perhaps I was considering the @ symbol wrong for matching purposes?

Then I found, via a StackOverflow question, a link to Fastmail's Sieve tester. That tool appears to be available whether or not you're logged into Fastmail.

Using a web-based (or CLI^[3]) tester, I can much more quickly iterate over testing changes than I could if each test required an email delivery (and in this case, a Gitlab issue update). Similar to the difference between a full setup and execution to validate, and testing in a REPL.

With a hosted web-based tool, I do need to consider confidentiality (eg not submitting client names in my Sieve configuration OR the sample email source) and implementation (Fastmail's Sieve has a couple of plugins missing relative to my work email).

With those things in mind I was able to reduce my full Sieve configuration to a version that reproduced the incorrect behaviour - the web interface showing the results of my filter:

"filing message into 'INBOX.Projects.ClientName'".

I was then able to test a few things out and quickly identify a fix - a trivial one! Faster test iteration lets us try things out more freely.

Fastmail now reported the desired outcome:

"filing message into 'INBOX.Projects.ClientName' with flags '\seen'".

Here's the solution:

  if header :contains "X-GitLab-Project-Path" [
    "clientname/"
  ] {
    if not body :contains "@chrisburgess" {
      setflag "\\Seen";
    }
    fileinto "INBOX.Projects.Clientname";
    stop;
  }

What changed? I moved the setflag above the fileinto, establishing that ordering of setflag and fileinto is significant; setflag action applies to subsequent fileinto actions.

Now the messages will all be filed in the per-client folder (searchable), but only marked unread if they explicitly @mention me.

Reviewing the rest of my Sieve configuration for the same fix, I saw that other similar filters had setflag before fileinto - which probably contributed to my uneasy sense that this rule configuration had worked in the past, but inconsistently today; I hadn't twigged to that detail being out of order for this one customer.

I don't see this ordering detail mentioned in the published version of Sieve's RFC5232, but once I knew what to look for, I saw it was mentioned in an earlier draft, section 3.1.

I have similar filters for other issue trackers such as Redmine and WRMS. ↩︎
This condition depends on whether I'm active on the project; it lets me hear only about updates that someone wants to put on my radar. ↩︎
I see there's also sieve-test available as a CLI option. ↩︎

pa11y in Lando

2023-06-25T00:00:00Z

Reading a post by Morgan on web performance and accessibility standards and I'm reminded of pa11y, a CLI tool for accessibility checking that we've used before.

Tools can't match expert assessment, but they can be useful to automate and frequency of checking can be a real boon to raising standards.

Here's pa11y in a Lando configuration - you can add the service and tooling entries below to some existing Lando environment, or use this entire config as-is for a standalone tool.

name: pa11y
services:
  pa11y:
    type: node
    overrides:
      image: registry.gitlab.com/gitlab-org/ci-cd/accessibility:latest
tooling:
  pa11y:
    service: pa11y
    cmd: pa11y --config /pa11y-configs/pa11y.json

Once those are in an environment, you can use pa11y alongside your project:

$ lando pa11y --sitemap https://my.lndo.site/sitemap.xml
$ lando pa11y https://example.org

This uses the image made available via Gitlab's Accessibility Testing tools; you can use their configuration to test accessibility in your Gitlab CI pipelines too.

chris@thip:~/Projects/lando-pa11y$ lando pa11y https://example.org --reporter cli

Welcome to Pa11y

 > Running Pa11y on URL https://example.org

Results for URL: https://example.org/

 • Error: The html element should have a lang or xml:lang attribute which describes the language of the document.
   ├── WCAG2AA.Principle3.Guideline3_1.3_1_1.H57.2
   ├── html
   └──  Example Domai...</html><br><br>1 Errors</code></pre>

Lighthousing

2023-06-25T00:00:00Z

I've just gotten around to running some Lighthouse reports over this site for the first time. It scored at 93, 96, 100, 100, 22, which is interesting. There are a couple of improvements I can make, and a bunch of suggestions that won't make it to prioritisation.

That is an OK first result, but I'm not taking particular pride in something I've done very little to achieve. The credit for the coffee I'm drinking at home right now goes to those who grew, prepared, shipped and roasted the beans. Likewise, for this site I selected some ingredients (markdown, Eleventy, Tailwind) which made sense for my needs, and I've added as little technical extra as I could get away with.

For me this is the right approach. This is a tool to share some thoughts, and the tooling is here to enable me and then stay out of my way. I particularly don't want a high maintenance burden looking after it. So if anything, in that 93 score I can take a little credit for picking tooling that's delivered decent results out of the box.

The 22 is the PWA ranking in Lighthouse, and something which highlights criteria which make sense for a limited range of sites. It's a useful reminder that Lighthouse metrics definitely aren't to be pursued just to get "all green" ratings. Few of those checks are for things which would add meaningful value to this site.

On the other hand, there are a couple of improvements identified in the first test which I could knock off pretty easily, and did so.

Seeing that loading the PrismJS libraries from a CDN was the biggest single wait time nudged me to finally fix code formatting here. Until now I'd shrugged and used Github Gist embeds - also fine. Fixed.
One of the accessibility checks flagged that I had not correctly set the language metadata in the root HTML tag. Fixed.

Result: 95, 100, 100, 100, 22, and more importantly two minor annoyances fixed. For this content page now, a slightly higher rating - 99, 100, 100, 100, 22.

Thanks Lighthouse, that was helpful after all!

And not from the numbers but the screenshot, a nudge for me to sort out the site title and menu which overlap at present. Not fixed yet.

XDebug, Lando and a CLI service

2023-06-10T00:00:00Z

I use Lando to simplify management of my local development environments.
I use PHPStorm to edit, comprehend and debug complex PHP codebases.
I use XDebug as a debug tool

If you've not discovered how helpful XDebug can be when debugging PHP, I strongly recommend you give it a try. If you don't code in PHP, find the equivalent tooling!

The initial setup for XDebug can be complex as a number of things have to be aligned to function correctly. A challenge for me is that when I try to configure it I've typically just run into a gnarly problem on a project; once I get it running my monkey brain turns straight back to debugging and forgets to record what steps I took to configure the debugger. Over time I'd collected a scattered set of "today, x worked" notes in my lab journal. This Saturday morning, a person asked in the Lando Slack asked how to get it running, and I figured I'd document a few details that often catch me out.

How does XDebug work?

A quick outline works as I understand it.

When PHP starts executing something, if the XDebug extension is loaded then it can be activated
XDebug and a debug client (your IDE, etc) may then communicate
XDebug will connect to the configured IDE via IP and port
The IDE will decide whether to handle this request
The IDE can tell XDebug about breakpoints in code

Docs

First, the proper docs, which you should refer to before my own notes.

My notes

Check XDebug module is enabled

lando ssh then php -i | grep -i xdebug - you'll see XDebug's configuration from PHP config here

Environment variables Lando sets

Among other variables, Lando sets these when XDebug is enabled for a service.
- XDEBUG_MODE
- XDEBUG_CONFIG,
- PHP_IDE_CONFIG
- PHP_IDE_KEY

Check environment in PHP service

lando ssh then env | grep XDEBUG
Check XDEBUG_MODE and XDEBUG_CONFIG

Config to check

client_host - should point at the IP of the host the IDE is running on
discover_client_host - can use HTTP headers to discover the IP of the browser and connect to IDE on that IP address
idekey is often 'phpstorm' but may only need to be set?
serverName is the XDebug server name passed to PHPStorm

Config to check - IDE

XDebug will pass the file path and other execution information to the IDE. The IDE needs to know which paths correspond from your IDE OS and the environment it's running in. PHPStorm has a settings screen (Settings > PHP > Servers) where you can map the project's files to the path as seen within the environment XDebug runs in. Here I've set the project root to map to /app in Lando.
![[attachments/Pasted image 20230610152023.png]]

XDebug on a CLI only Lando

Lando service configuration

If PHP is running on another service than "appserver", the global Lando configuration option may not be applied. (I think this is here.) If you want XDebug on a service not named "appserver", the global xdebug option won't apply and you should set xdebug: on the service instead.

Lando restart after switching networks

Lando's XDEBUG_CONFIG variable won't update if your laptop switches IPs until you restart. This can mean that XDebug stops working after you move locations. lando restart -y should refresh the IP of the Lando service you want to debug.

PatchWatch progress report

2023-05-29T00:00:00Z

Thursday before last at DrupalSouth 2023 codesprint I knocked together a prototype of PatchWatch, a tool to consolidate information about patches applied to a set of projects. [Earlier post on patchwatch here]((https://chris.bur.gs/composer-drupal-patches-report/).

This last weekend, I revisited the prototype and fleshed it out some.

The tool now is a Symfony console command with three or four commands implemented:

patchwatch:report-composer-patches reports on cweagans/composer-patches patches from a composer.json or patches-file
patchwatch:report-drush-make reports on patches from an old style drush.make file
patchwatch:report-patch-directory reports on a directory of patches
patchwatch:report-patch-package reports on a directory of patches in patch-package format

It has some test coverage in tests/src/ which parses sample input data from tests/fixtures/.

I also added a caching layer in the Guzzle client, so we don't annoy HTTP endpoints too much while querying for data about issues related to patches.

The core function of "read patch input data and output something structured" works, now. The tool does not yet collate several sources into a single "high level" view, but that should be straightforward to add. I want to spend a bit of time reflecting on the structure of the patch listings before I do that.

I've added some Gitlab CI and this includes a "demo" job which shows the output of the commands when using the test fixture samples as inputs. See the demo job in the latest pipeline for this.

I also had a play with taking the generated output and throwing it at a store ... having heard a bit of buzz about Metabase from some team members lately, I threw the generated JSON at MongoDB and then yoinked that into Metabase, which was interesting. See Lando + Metabase for some notes on that!

Next steps

Configurable patterns

I am thinking about how to take a listing of patches (which will have some basic data such as a patch URL/path and possibly a label), and "enrich" the list of patches into something more full featured. Since different organisations will treat this differently, there is probably room for configurable pattern matching, eg to help an org discern that #123: Bugfix for frobnitz adapter. It may also make sense here to specify that there's no default behaviour and that all references should be canonical (drupal/drupal#123) ... but flexibility here would make adoption easier.

Plugins for informing patch data

The other thing that's on my mind is in the Patch reader where we apply "informers" to the source data. At this point we might want to pass the patch data (an array with a filename) into a set of transformers which might "enhance" the data ("I recognise an issue number in that filename pattern; I will add to the patch data information retrieved from an upstream tracker."). There's only one such plugin at present (for Drupal issues) and I want to work out the right pattern for making this easier to adapt, and possibly allow for user-specific plugins as well.

Support for forks as well

Later I plan to add support for forked repositories as well. Initially I was taking the opinionated position that forks are a worse strategy, and this tool should focus on patch handling. Now that I've implemented a few source adapters, I realise that implementing Git support wouldn't be that hard, and it would make the tool useful to folks who have a local fork of an upstream project.

(I still recommend to not adopt the strategy of long-lived forks, but it's a thing people do, and we can allow for that.)

Credits

Thanks to the authors of the following open source tools which I've been reading and considering as I implement this. I'm grateful to be building on what you've shared.

Lando + Metabase

2023-05-29T00:00:00Z

Having heard buzz on it from colleagues, I wanted to give Metabase a quick try. I knocked together a quick .lando.yml to test it out. I'm impressed. Here's the .lando.yml I used:

name: metabase
proxy:
  metabase:
    - metabase.lndo.site:3000
services:
  metabase:
    type: compose
    app_mount: false
    services:
      image: metabase/metabase
      ports:
        - '3000'
      environment:
        MB_DB_TYPE: postgres
        MB_DB_DBNAME: metabase
        MB_DB_PORT: 5432
        MB_DB_USER: postgres
        MB_DB_PASS: postgres
        MB_DB_HOST: database
  database:
    type: postgres
    creds:
      database: metabase

This brought up a Metabase service at http://metabase.lndo.site for me. I expected to be able to load data directly into Metabase, but I found you can mostly connect Metabase to external sources. So I added a MongoDB service to the above, and threw my JSON into that, and connected Metabase to the MongoDB store.

So - my first misconception I think was that Metabase was a store of data. From my initial look, this seems to be wrong - while Metabase does store data internally (ok, metadata I guess), it seems to want to give you a view of external data. Which may or may not have been yanked into Metabase to support your insights.

Then I realised I had other data sources I could wire into Metabase. I had a local Commerce site running in Lando, and expected that Metabase from Lando would be able to see the DB from that commerce site via Lando's .internal network hosts. I ran lando info in the commerce project, snagged the DB creds, and fed those into Metabase; it indexed the data and fed back a kind of "Facebook memories" of my commerce DB.

And some WIP visibility on a set of patches from Patchwatch:

All in all, a decent first experience for trying it out. Looking forward to learning more.

Fixing the Element notification sound

2023-05-25T07:03:00Z

If you use Element you might have noticed that the default notification sound is ... jarring? janky? It's loud, and it's sharp, and it's honestly kind of agitating to me.

Matrix message sound

First, a restful tone

I mean, you can use anything you like. Bon Jovi. Go wild. Me, I wanted something restful to hear during the day - these are challenging times.

Like the Tokyo Train Dream Sound set, created by Zeethrogise, originally released circa 2004 as an Adium soundset. I picked CSnd.mp3.

Matrix message sound

Get a room

Element doesn't let you set a global notification via the UI yet (vector-im/element-web#20286: Add interface for setting a global notififcation sound). We need to assign the sound to a single room, then apply that sound to the global configuration.

Pick any room you're in. I picked a PM I have with a bot I run.

In the room, locate the "Room Info" button (top right corner for me), and then select "Notifications". Upload your sound here.

Now:

Say /devtools in that room
Click "Explore Account Data"
Click the "Account Data" button
Click im.vector.web.settings

Copy the notificationSound block, e.g.

"notificationSound": {
    "name": "Tokyo Train Dream Ringtones - CSnd.mp3",
    "type": "audio/mpeg",
    "size": 18560,
    "url": "mxc://example.org/aPLnafWKcouYthRwKbHSqNHC"
}

Click "Back"
Click "Room Data"
Click im.vector.web.settings
Click "Edit"
Insert the copied json object into this object somewhere, ensuring that the result is valid JSON. (Each entry should be followed by a comma, except the last one.)
Click "Send"

Sounds should now change globally. To test, you'll need someone to message you (only, in a room which you're not watching).

Enjoy! This small change may vastly improve my Matrix experience.

What patches are we running?

2023-05-19T00:00:00Z

I'm at a code sprint with members of the Drupal community today, and we're looking for tasks to work on. The agency company I work for manages Drupal sites for a number of clients, and across those sites we might apply patches for the same issues to various projects.

Patches are applied using cweagans/composer-patches and defined per-project using an entry in a composer patches file.

By scanning these files and looking at the related issues, we can get a high-level view of the patches in play, and start to see some interesting things.

My goal for this is to build a Composer plugin (xurizaemon/patchwatch) which can be used as a command from the project's CI, but for today (and mostly due to network issues while travelling) I shot for a quick and dirty PHP script which gave me some serviceable output.

Here's some excerpted and redacted output, to show what we can see afterwards. This is a pretty crude PoC.

And here's the quick script, somewhat munged to hopefully share my rough initial script in a single executable .php. The repository for this project will be https://gitlab.com/xurizaemon/patchwatch

(I haven't embedded the output and script here because I apparently haven't sorted out code embeds on my own blog yet ... sorry!)

The interesting things we can start to see from the output above are:

Some issues or patches are indeed common to multiple projects, and good candidates to focus on to reduce patch debt!
Some projects have diverged patches, which points to a maintenance task for those sites. (They may not have matching versions of core, or the differing patches may be correct - but it's worth checking.)
I included issue status somewhat on a whim, but this turned out really interesting. We see:
- Patches for issues marked "Closed (fixed)", which suggests to check if there's an option to update the component and drop the patch.
- Patches for issues marked "Closed (won't fix)", which suggests that we should review our own use of the patch (and then perhaps revisit our issue communication). This may point to a future conflict between our project with patch, and the module's development direction.
- Patches for issues marked "Closed (can't reproduce)", where we can probably contribute by supplying instructions to reproduce (because we definitely recorded why we introduced this patch, right ...).

Next steps are:

Consider patching for other projects, eg I recently learned of Node's patch-package from the Mahara project ...
Implement this suitably for CI, possibly as a Composer plugin (with close integration to cweagans/composer-patches), or possibly with capacity to consider other build/patch solutions and sources (this version is Drupal-specific and ignores patches to components which don't have a drupal/ prefix).
Integrate with systems for managing internal knowledge of which patches are significant to our organisation, so we have visibility for our team to identify issues of interest and can reduce duplicated effort
Consider whether this tool could be generally useful to capture metrics of patch consumption eg for Drupal.org
Consider whether the output of this tool is indicating opportunity for us to contribute in other ways, eg by funding the maintainers of modules which we have lots of patches for, or by stepping up to maintain ourselves, ... or reconsidering use of modules which aren't getting patches landed.

Something which might be interesting - if Drupal.org wanted to surface it - would be the download stats / popularity of patch URLs. This might indicate patches which are "popular" and being applied by lots of sites, and good candidates for review.

If this looks useful, or you know of a similar approach to this which I don't or are interested to explore further - get in touch! I'd love to hear from you.

Update: progress report here

Upgrading from Drupal 6 or 7 to 9

2023-05-09T00:00:00Z

There is an auto-discovery process built in, using the Migrate Drupal module in core. It will generate an initial set of migrations for you, for both site configuration (eg content types, fields) and content (the nodes themselves, and friends).

https://www.drupal.org/docs/upgrading-drupal

You can use that to generate a pretty good starting point - and you can customise from there too.

It's really important IMO to consider this as development of a process to migrate sites, rather than development of a migrated site. Be comfortable with the idea of running the migration repeatedly while developing.

With Migrate Plus module, you can take those migrations generated by Migrate Drupal and fine tune them. Core's auto-upgrade handles field types that are in core, but you might need to adjust where the Drupal 7 module needs more customisation.

From Jekyll & Hugo to Eleventy

2023-03-06T00:00:00Z

Well, I seem to have made it through a Jekyll and Hugo to Eleventy conversion, and come out the other side with a set of posts going back a decade. Hail plain text formats!

Eleventy has been an accessible (for me) platform to do this. What have I had to tackle?

A variety of sources

The posts collated together here cover a few years of a "personal engineering" Hugo blog I've kept internally at Catalyst, and about as many years of Jekyll blog I'd stopped updating when I joined Catalyst. Two similar (markdown-based static site generator) systems with subtly different data sets. The frontmatter for both engines was different, and I did a little juggling with perl find and replace to rearrange things. I had both categories from Jekyll and tags from Hugo, for example.

It could be tempting to perform a "migration" from Hugo and Jekyll to Eleventy, but one lovely aspect of Eleventy here was that it was flexible and fast enough for me to fix things on the fly. So I didn't need to do that.

Dates

For a long time on at least one of these blogs, I was hand-entering something like ISO dates. And sometimes got them wrong, cos I forget small things. So I wrote a couple lines of filter to handle most any date that the posts would throw Eleventy's way.

eleventyConfig.addFilter("postDate", function(dateInput) {
  let dateObj = DateTime.fromJSDate(dateInput);
  if (dateObj.invalid) {
    dateObj = DateTime.fromISO(dateInput);
  }
  if (dateObj.invalid) {
    dateObj = DateTime.fromFormat(dateInput);
  }
  return dateObj.toLocaleString(DateTime.DATE_MED);
});

Syntax highlighting

Eleventy had PrismJS built in and that worked well once set up, but I had to nudge a few formats about - in my Hugo blog I'd used "cron" format which PrismJS didn't like, and so on. So, a little content mop-up required.

This isn't quite working right yet - I'm seeing things highlighted but combined into a single line on deployment. TBD.

Tailwind CSS

I tried Tailwind CSS, and I've decided it's perfectly good. I wasn't sure at first. It's a whole vocabulary I'd need to either pick up or look up to operate with, which is something. But it's produced a tidy, painless layout which I'd otherwise have probably spent ages getting approximately right.

I'm using eleventy-plugin-tailwindcss which seems to work great.

Drafts

I had posts with draft: true set, and didn't want to drop them all on the world. I used eleventy-plugin-ignore.

Permalinks

My previous Jekyll blog had dates in the filename (posts/2023-03-06-jekyll-to-eleventy) but not in the paths (/jekyll-to-eleventy). I didn't want to have to handle the redirects, so I used Eleventy's per-directory JSON to rewrite paths.

In .eleventy.js:

eleventyConfig.addFilter('stripDatePrefix', str => {
  return str.replace(/^[0-9-]+/, '')
});

In src/posts/posts.json:

{
  "permalink": "/{{ page.fileSlug | stripDatePrefix }}/" 
}

Gists

I'd previously used a {% gist <url> %} and needed to convert this from Jekyll.

eleventyConfig.addShortcode("gist", function(url) {
  return `<script src="${url}.js"></script>`;
});

Large media and hosting

I'm already deploying this to Gitlab Pages, easy. However ... I want to have a slightly more flexible and funcitional hosting environment. So I'll wire up a deploy over SSH/rsync from Gitlab CI.

Commit credit carry-on

2023-03-05T00:00:00Z

Drupal was an early proponent of crediting community contribution visibly. Today the community faces an interesting consequence of this policy - a large number of (perceived to be) low-value contributions are being submitted through the issue queues, with the apparent aim of increasing promotion via the Drupal.org contribution recognition systems.

Organisations which contribute to Drupal are recognised and have placement benefits on the Drupal.org website. This creates a motivation to earn commit credits, which some feel has recently resulted in submissions which are of low value (eg "Rename README.txt to README.md", or unnecessary patch re-rolls), which in turn increases maintainer workload in dealing with the influx of submissions.

Bad patch re-rolls and poor review process have the risk of introducing bugs and all increased workload has a human cost in maintainer workload. The increased effort can only be worthwhile if it's appropriately focused, or the impact to the wider community will be negative.

It's worth noting that the organisational recognition system through commit credits is a form of hybrid, source-unavailable leaderboard - agencies can sponsor Drupal.org to a certain tier, then I think have their promotional placement increased by achieving a given number of contribution recognitions. Leaderboards can be problematic, and because they encode both community and implementation decisions, are gamable interfaces. Much to say here about metrics too, and we've faced leaderboards before.

I have appreciated that Drupal's contribution system was flexible enough for people to use in innovative ways, eg how Talking Drupal and several working groups use d.o issues for crediting those involved in non-code labour. That's cool. The credit system also can't ever perfectly reflect contribution valuing, because a community is complicated. As a community, I think we need to build a solid understanding of what commits are most valuable and effective.

One response I've found helpful in response to community behaviour I find challenging is to have a ready reply which is generic, simple and clear ... and re-usable! Eg I have a no email / private support gist to copy paste when a community support discussion gets redirected to a PM support request I can't meet. I saw a comment by @jrockowitz which I thought was a well-phrased response and that got me thinking about how to have a clear, community response to correcting a behaviour we want to reduce.

(It's so important to keep in mind that this is a value judgement, that there may be more complexity to the situation, that )

I'm not sure if Drupal has made a "community comment" on the behaviour yet - it's complicated, and from conversations I've seen it looks like there's been direct communication at least with some of the agencies doing it. I also see maintainers struggling with getting 40 issue queue updates a night who are being prevented from doing their regular maintainer workload. At that point it'll be hard to maintain cool or even continuing maintaining components and we risk real community loss. Even so, I've observed some "low-value" contributions being worthwhile and I try to give a fair response.

I'm concerned that if a brigade against entry-level contributions was set off, that could be really harmful for the community.

The Drupal community was a fore-runner of commit credits and even leaderboards - something we know can have significant impact for good and bad. I've seen proposed that we track negative credits and even display those against "offending" organisations to disincentivise, which I fear would have unintended consequences of its own. I'm realising the interesting complexity of Drupal's contribution data, how it reflects (or doesn't reflect) values across a diverse community, and the responsibility of having an established process for this which we'll have to continue to improve upon over time as long as it exists.

For me the reflection is about how we best redirect people who have time and energy to contribute from "commit credit carry-on" to outcomes which are those which are appreciated.

I worry we might redirect people to review, then see an uptick in low-effort RTBC reviews.
I worry that as tooling improves, automated low-effort contributions will increase
I think we can predict that even more automated (eg LLM-generated) contributions will be on their way

I think there's a danger that a change in the balance of labour will affect thinking here. Agencies committing to contribution efforts will do well to be present and focus their contributions effectively.

We can be confident we'll see more automated commits in the future. Maybe getting ahead of that wave offers some relief but I'm guessing it won't be a refuge.

Responsible consumption of Open Source Software

2023-03-04T00:00:00Z

Consuming OSS components is a digital value transfer from community to the entity using the component.

With digital goods, reproduction costs are much lower compared to physical artifacts, but production and maintenance costs can be high to the provider of the good. As we account for the full cost of ownership, we discover the cost of maintenance is much higher than it might have seemed, especially when we factor in the ongoing cost of maintenance.

This has an analogy in extracting resources from the environment in early times when business doesn't account for the long term impact, and then later we start to appreciate a more full accounting of the impact of our work.

It's incumbent on users of open soruce to recognise and nurture the communities that provide the software they depend upon.

We should also appreciate the special power of open source, that we are able to contribute directly to projects. There are lots of ways to contribute - organising, coding, reviewing, testing, funding, directing, and more I'm sure. When we contribute we enable ourselves to engage in direct advocacy and have agency over the functionality and future of the project.

When building on OSS components, it can help to have ways of modifying them which support upstreaming our changes. Patching or forking workflows which are in public support and encourage visibility and sharing. (There's more to communicating than just forking in public, of course.) Visibility of working in public also asks us to code securely, to not commit sensitive data to a codebase, and to think carefully about how our code is structured for general usage - all valuable practices to work securely and with confidence.

Developing an active presence in the community helps us work with the software better through familiarity in both directions. By becoming familiar with community approaches, we expend less effort trying to configure systems in ways which don't work in alignment with the project's intended use, and don't rabbithole ourselves inventing novel solutions to solved problems. Presence in the community also gives others familiarity with our work, which supports us when there's reason to propose a change to the way things work. Both are beneficial outcomes to working visibly.

We use a lot of software and can't be expected to actively engage in every single component or community we make use of. For those of special significance to our business or process, and especially those we actively develop with, there are lots of opportunities for companies to contribute in ways that benefit the provider and consumer mutually.

In Memoriam Instagram

2023-03-03T00:00:00Z

Recently helped a friend extract her own Instagram account's images from her hijacked account after it got hacked.

The hijackers of her account had not yet deleted her previous posts, and using instaloader we were able to download her visible posts to a digital archive.

My friend was upset to have lost access to a lot of special images which she didn't have copies of elsewhere.

Losing control of an account can be upsetting for the loss of networks and connections we share, or experiences we're attached to. It's a form of identity loss I guess. Digital security is a privilege not all people have access too either.

She wasn't yet able to recover her account but I was glad that we could retrieve those images she'd lost access to. I posted them back to her yesterday by regular post. It's good that her images can be where they belong, with her, not attached to a lost account.

I'm mindful of Sumana Harihareswara's "All maintainers eventually leave (or stop leading) projects.". This is true of all digital accounts. All digital accounts in time will be cut from the identity attached to them. (Amusing corollary: there are now digital accounts, such as @CocaCola but not just corporate, which may now last for generations of humans.)

Before I got it together to send the digital archive by post (is the postal service a cloud of sorts?), I wanted to reassure her that the images were safe. So I had the idea of combining a year's images into a few seconds of video and messaging that to her - used ffmpeg, it looked cool.

The video at the top of this post is me having finally got round to doing the same for my own (admittedly shorter) feed.

The envelope is suitably sombre and has been cut from building paper which I had scraps of in the garden.

Anyway - after posting this today, I went back and made a backup of my (much smaller) Instagram history. And then I found the ffmpeg command to turn it into a brief video. My own posts only makes a ten second video, but watched on loop it's quite fun, and various things leap out as you watch it.

I've hidden it behind a reveal click below because otherwise it's a bit of an eyeful!

Click here to see my instagram account's entire feed in a ten second loop

Islandora and ISLE DC

2023-03-02T00:00:00Z

For work reasons I ran up an Isle DC instance or two on Thursday. OK, first let's unpack that!

Islandora is an open source digital asset manager and digital heritage repository. It's composed of a lot of services.
ISLE is Islandora: Enterprise 2 (I think)
ISLE DC is a Docker Compose environment to spin up Islandora sites

The ISLE DC installer uses a Makefile (love it) and brings up several containers to run the various services that make an Islandora environment. (You can configure lots of this, eg swapping in your preferred DB backend service for the Matomo or Drupal components).

The environment which it brings up has lots to it. I'm planning on making some post-it notes to list out the services that it include, since I look at the list and boggle.

We use Lando quite a lot on other projects, and both Lando and ISLE DC drop in a Traefik proxy which wants (out of the box) to bind port 80. I'd love to be able to have both run alongside each other, so I'm interested to see whether we can do that. Ideally we want to be able to switch quickly between projects without needing to shut down all the others. The conflict is really around the default ports, which can be resolved of course, and this is a challenge you can run into with running (say) Vagrant and Lando sites in parallel.

I wonder if we can have the two systems co-operate on a common Traefik proxy ... there are other options.

The installer for the demo Islandora site takes a wee while, during which my everyday laptop lagged crazily. But it came up and it's an impressive work of orchestration to bring each of those several services up and connect them. Really impressed. I can see that the orchestration here is keeping deployment to other platforms in mind too, and it got me wondering about how we plan to deploy Islandora ourselves.

I don't require local development environments to faithfully reproduce hosted environments, I see that as an anti-pattern. It's beneficial to not couple too tightly to a specific deployment implementation, and so I like to treat local dev and CI as a loose parallel which may run up only the required services. If I'm not developing or testing against Matomo, I'll choose to omit that from the dev and CI layouts unless it's of significance. To my thinking, this actually tests that things work in a broader array of configurations.

Thoughts

A few notes on the deployment it brought up. I've only had a brief introduction to the software so far, so I'm not going to leap ahead and propose changes yet. These are just first impressions!

Use of `sudo`

I was a bit uncomfortable with the Makefile prompting me for my system password part way through the process. This doesn't feel like it should be necessary, and is always worth a careful review. On doing so I saw this code which is triggered from the presence of an environment variable in .env:

CMD := $(shell [ $(IS_DRUPAL_PSSWD_FILE_READABLE) -eq 1 ] && echo 'tee' || echo 'sudo -k tee')

It's a little hard to read - I think this is only using sudo to read the contents of the file, right? But it makes me nervous seeing sudo scattered around as a workaround for correct file permissions configuration, and until I read that carefully I wondered if I could exec something by setting DRUPAL_DEFAULT_ACCOUNT_PASSWORD to "password"; adduser baddie <...>.

Docker Compose or otherwise, calling sudo commands on the host OS are something to be wary of. Lando does some fancy shenanigans behind the scenes to keep file ownership aligned, which is nice. Many services which are fully self-contained don't need this treatment at all; for those which permit direct edits in a volume mount, it's important to get right. (For Drupal media, configuring $settings['skip_permissions_hardening'] = TRUE; in local dev helps with this.)

I disagree with `update_settings_php`

On first impression I don't agree with update_settings_php.

I don't like that it places configuration AFTER include settings.local.php, which is supposed to be at the very bottom of Drupal's settings.php. settings.local.php should IMO still be able to override any of the values set in that snippet. The snippet generated should be placed above that final include, or in a separate include if it's intended to be generated per environment/build rather than committed to the codebase.
I don't like that the implementation is a bash function stashed away inside a built container, which can hide how things are working if you're trying to understand what's going on by reading the source of your Islandora codebase.

Traefik and sharing proxy

As mentioned above, this is on my mind ... I see in ISLE DC that there's a configuration INCLUDE_TRAEFIK_SERVICE, which apparently permits the environment to use an external Traefix proxy if set false. Lando also has a global config proxy which can be set "OFF" to disable Lando's Traefik

Do we need all this?

Some of the containers may not be necessary? I don't know yet which, but I was glad that the inclusion of VS Code server is not activated by default (!!!), and I would probably turn Matomo off by default. The services which did come up had related settings for memory limits, and the total of all those memory limits was in the neighbourhood of 50GB for services which did come up in my test. Whoa!

Resource demand

The fcrepo service certainly gave my work laptop a run for its money and I killed some updating daemons at one point when the system lagged to a standstill with a loadavg just shy of 400. I may have to requisition an appropriate 64GB RAM stick, or a standalone environment! But I'd rather be able to hack on my laptop and not remotely. Maybe that's why VS Code server got in there ... With things as they are, I probably won't casually put an Islandora environment on my personal laptop to test things out.

In summary

This project is exciting. The above criticisms / thoughts are just an early response, and my bigger feeling is - wow, this is really well put together, and reminds me of other community efforts where people find something that works and make progress. I'm stoked to see that the Islandora community has got things so well put together so far and I'm excited to see what a deployment of this looks like. We have a handful of existing Islandora projects I look forward to trying it out on.

The Islandora community looks really exciting to be a part of, and I look forward to getting to work on some of these projects in future, and building a connection with the community.

Working with digital heritage and culture artifacts is something I'd love to see myself doing more in the next few years, and I'd be excited to work directly engaged with institutions implementing these systems. I think it's so important to have a strong connection to the people involved in working with the objects of significance in order to best support building tools that reflect both the community best practice and the special nature of the heritage at hand.

I'm thankful to Jonathan Hunt and Ant Brown for making the introduction, and giving me a good roadmap to begin my journey with Islandora!

Eleventy!

2023-02-28T00:00:00Z

OK, my blog at chris.bur.gs had gone pretty idle since joining Catalyst in 2017.

This is an interesting note - I think I felt that starting in a new role, I wasn't confident of my voice and paused posting. At Catalyst I started an internal blog once I got the hang of things, and realised I wanted a place to share thoughts. I highly rate this approach, tbh - I find writing helps me consolidate thinking, and it's an unobtrusive way to work openly and occasionally surprise your future self.

I've long had a pocket journal, notebook or digital journal / blog and honestly I think I mostly write for my own future self, but with others in mind where I can?

Anyway - I've pulled some of those internal articles across to this as a continuation of my process.

It's nice that so much of my former blog converted readily to Eleventy - testament to markdown as a format really. I'm more and more enamoured of the static site approach.

I've not really implemented any themeing at this point ... exploring Tailwind, just out of interest.

Using Drupal Migrate to report on content

2022-11-20T23:19:00Z

A customer requested a report of:

How many times a given token ([foo:some-value:3]) could be found in a content corpus
The specific IDs of each value in each content item

The content corpus in question is a decent size (150K content items) stored in a Drupal 7 DB. While this request might be deliverable via a SQL query, the risk of bypassing application logic is that we might not account for some detail of implementation that means we miss some content items.

Since we're already writing migrations for this Drupal 7 site, why not use Drupal Migrate to report on it?

Here's my plan, and how it worked out:

Add a temporary field to the Drupal 9 destination content fields
Add a migration which extracts the tokens only from the source content and stores it to that field.
Run the migration and see

Implementation

This request came in before we had the Drupal 9 site codebase for the site in question ready, so I opted to hijack another similar site's DB for the purpose.

I brought up the Drupal 7 dev environment in Lando, and reconfigured the destination site's Migrate source connection to use the same credentials but with a changed hostname. Lando makes the DB for an environment available at something like database.name.internal where name is derived from the D7 site's name value.

Drupal 9 web/sites/default/settings.php:

if (getenv('MIGRATION_DATABASE_NAME')) {
  $databases['migrate']['default'] = [
    'database' => getenv('MIGRATION_DATABASE_NAME'),
    'username' => getenv('MIGRATION_DATABASE_USERNAME'),
    'password' => getenv('MIGRATION_DATABASE_PASSWORD'),
    'prefix' => '',
    // 'host' => getenv('MIGRATION_DATABASE_HOST'),
    'host' => 'database.SITENAME.internal',
    'port' => getenv('MIGRATION_DATABASE_PORT'),
    'driver' => getenv('MIGRATION_DATABASE_DRIVER'),
  ];
}

I confirmed that I could now see the alternate D7 source data via drush migrate:status.

Side quest: performance challenge!

This drush migrate:status command was really really slow. In another terminal I connected to the D7 source system and executed watch -n5 "mysql -uroot -e 'SHOW FULL PROCESSLIST'" to see what was going on. The query was checking content for regex matches against %src="/file%, and sure enough this led me to the Media Migrate plugin and a performance issue I'd raised 11months ago, which now had a patch. Win!

I didn't want to have to iterate through the entire content set to develop, so I used drush migrate:import migration_id --limit=1 --migrate-debug to identify the first node we'd get out of the system. This would become my sacrificial test node, even though it didn't originally contain the tokens in question.

I modified the D7 DB state directly so that I would be able to retrieve a matching result from the first source row. It's intentional that the tokens aren't identical.

update field_revision_body set body_value='blah\n\n[foo:grouped-media:0]\n\n[foo:grouped-media-table:1]\n\n[foo:grouped-media:2]\n\nblah' where entity_id=74
update field_data_body set body_value='blah\n\n[foo:grouped-media:0]\n\n[foo:grouped-media-table:1]\n\n[foo:grouped-media:2]\n\nblah' where entity_id=74

I reconfigured the D9 site to have an additional field field_tmp_token_count (text) to store the report values. This can be later used in Views to report for the client. I exported the Drupal 9 configuration to preserve the field configuration.

I now created a node classic migration for the content type we were interested in. I modified it to process the body field directly into the temporary field.

  field_tmp_token_count: body/0/value

I commented out almost all other process pipelines; I only wanted the essentials: nid, vid, title, and anything else required for the migration to work as expected.

I then wrote the process plugin, testing my preg_match_all() using php -a (or drush core-cli).

web/modules/custom/foo_migrate/src/Plugin/migrate/process/TokenCount.php:

<?php

namespace Drupal\foo_migrate\Plugin\migrate\process;

use Drupal\migrate\MigrateException;
use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\node\Entity\Node;

/**
 * Count the number of tokens in a string.
 *
 * @MigrateProcessPlugin(
 *   id = "token_count"
 * )
 *
 * @code
 * field_token_count:
 *   plugin: token_count
 *   source: body/value
 * @endcode
 */
class TokenCount extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   *
   * @throws \Drupal\migrate\MigrateException
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $matches = [];
    $pattern = '/\[foo\:[^\[]*\]/';
    $res = preg_match_all($pattern, $value, $matches);
    if ($res) {
      return implode(' ', $matches[0]);
    }
  }

}

Now I can use my process plugin in the migration:

  field_tmp_token_count:
    -
      plugin: token_count
      source: body/0/value

I imported this configuration change using drush config:import and re-ran the migration with drush migrate:import migration_id --update --limit=1 --migrate-debug.

I could see the matched tokens extracted and stored in the destination field.

Now I could kick off the full migration and let it run. This took two and a half hours.

 [notice] Processed 48046 items (19490 created, 22960 updated, 0 failed, 5596 ignored) in 8539.5 seconds (337.6/min) - done with 'foo_d7_node_resource'

The migration yaml

uuid: 91a9f5db-d4fc-4387-9adc-24d672a89286
langcode: en
status: true
dependencies: {  }
id: foo_d7_node_resource
class: Drupal\node\Plugin\migrate\D7NodeTranslation
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
  - Content
migration_group: migrate_drupal_7
label: 'Node (Resource)'
source:
  plugin: d7_node_complete
  node_type: resource
  revision_log_append: '@logMessage'
process:
  nid:
    -
      plugin: get
      source: tnid
  vid:
    -
      plugin: get
      source: vid
  langcode:
    -
      plugin: default_value
      source: language
      default_value: und
  moderation_state:
    -
      plugin: static_map
      default_value: draft
      source: '@moderation_state'
      map:
        draft: draft
        published: published
        archive: archived
        needs_review: draft
  title:
    -
      plugin: get
      source: title
  uid:
    -
      plugin: get
      source: node_uid
  status:
    -
      plugin: get
      source: status
  field_tmp_token_count:
    -
      plugin: token_count
      source: body/0/value
destination:
  plugin: 'entity_complete:node'
  translations: true
  default_bundle: resource
migration_dependencies:
  required:
    - foo_d7_media_document
    - foo_d7_media_image
    - foo_d7_media_video
    - foo_d7_taxonomy_term_author_corporate
    - foo_d7_taxonomy_term_content_status
    - foo_d7_taxonomy_term_copyright_licensing
    - foo_d7_taxonomy_term_region
    - foo_d7_taxonomy_term_resource_types
    - foo_d7_taxonomy_term_solr_keywords
    - foo_d7_taxonomy_term_tags
    - foo_d7_taxonomy_term_viewpoint
    - foo_d7_user
    - foo_embedded_taxonomy_term

Drupalsouth Talks

2022-10-30T22:10:00Z

I didn't make it to DrupalSouth this year sadly, but it sounds from some attendee reports like it was a tight conference and the industry is going great over the water. An interesting challenge of this is that Australia is in a hiring frenzy for Drupal folk, and it'll be a challenge for us to compete with that.

I applied to attend when my talk proposal was accepted, but was disappointed to find that there was no remaining budget for this.

Some good talks coming out on video now anyway! I get the impression that the Drupal industry is really moving ahead in Australia.

How to lead talented technical teams

Keynote by Sarah Jane Peterschlingmann. Some good insights on team culture & development, particularly interesting to see her comments on how trying to improve team makeup using pay can have unintended consequences, and appreciated her articulation of the "expert leader" trap.

Drupal 9, Migrations, Decoupled APIs and Online Collections / Morgan Strong

QAGOMA is the Queensland Art Gallery / Gallery of Modern Art. A really interesting look at a complex project which Drupal is one part of, and how Migrate is helping them.

Scaling Drupal Through a Pandemic

Kim Pepper (PNX, Skpr) talks about handling massive web traffic during the pandemic, sharing some of the lessons and wins they found along the way.

Beyond plain language: Techniques for communicating complex concepts and processes

Really valuable; not about technology but communication.

There's lots more - I'd love to hear what you found valuable!

https://www.youtube.com/playlist?list=PL31TzQhFjH2Lk9JjkudM7prduSDfIx30e

Growing Catalyst Cloud Volume

2022-10-29T17:35:00Z

So, I ran up a couple of cloud instances recently, and this week looked back and both had full volumes. Le sigh.

(No, I probably wouldn't have run into this if I'd had one of our excellent ops team deploy an instance for us.)

Time to grow the available diskspace for them!

Docs

Catalyst Cloud has an entry on this in their Block Storage FAQ.

On first read I actually missed what looks like the easy instructions at the top. Second time around I'm going to use those better.

Documentation

I logged into https://dashboard.cloud.catalyst.net.nz and found my instances, making a note of the storage size and type (b1.standard is the default, and that uses spinning disks which are cheaper but slower).

I have an instance with a 10GB disk and another with a 40GB disk. I'll tackle the 10GB first, then repeat for the other once I'm happy with round one.

The FAQ answer to growing boot volumes give three answers:

Deploy a new instance and configure from scratch - good advice generally, not doing that thanks
Use a volume snapshot - looks easy but says I can never delete the source volume, which seems wrong
"Old fashioned method" is what we'll be doing today, then

Am I deterred by the label "old-fashioned method"? Why doesn't it describe the process ("clone data to new volume"), but instead makes a comment on the process ("old-fashioned method")? It feels like this ought to dissuade us - everyone knows old-fashioned means bad, right? Well, no, I'm not deterred - option one is labour intensive, option two has a weird gotcha, let's try option three.

Process followed (v1)

I suggest you read the whole article before following this, as the first attempt didn't work out.

In the Catalyst Cloud dashboard, I create a new volume, with a size of 40GB and a type of b1.sr-r3-nvme-1000. The default label is b1.standard^[1] which I believe from experience means a spinning disk storage - much slower. I can see "nvme" in this label, and while Catalyst Cloud doesn't specify what the types are in the UX, I find clues on the pricing page - "3 replicas on NVME with 1000 IOPS burst limit".

I don't need to answer the "Volume name" question, it'll get a UUID generated if I leave it blank. And the instructions say this new volume needs to be bootable; that option isn't available when creating the new volume, but it is if I create the volume, then edit it and check the "Bootable" option.

I can now attach the volume to my running instance, by using the menu on the volume. This asks just for the compute instance to connect to. It also asks for a device ID and the placeholder value shown is /dev/vdc, but I notice that when I leave that blank and attach it, the volume shows up as /dev/vdb. Also fine, just one to keep an eye on.

Now I can SSH into that running instance.

First we want to make a filesystem on the new volume, and mount it at a target destination to be copied to. (This isn't stated in the docs. If you omit this, you'll end up copying to the same disk.)

mkfs.ext4 /dev/vdb
mkdir /media/newvolume
mount /dev/vdb /media/newvolume

The docs have a command for us to perform the copy at the top of the page, but it seems to be explicitly referring to non-boot volumes. Let's see what that rsync command does:

rsync -avxHAXSW --numeric-ids --info=progress2 /data/ /mnt/data_new/

-a  : all files, with permissions, etc..
-v  : verbose, mention files
-x  : stay on one file system
-H  : preserve hard links (not included with -a)
-A  : preserve ACLs/permissions (not included with -a)
-X  : preserve extended attributes (not included with -a)
-S  : handle sparse files, such as virtual disks, efficiently
-W  : copy files whole (w/o delta-xfer algorithm)
--info=progress2 : will show the overall progress info and transfer speed
--numeric-ids : don't map uid/gid values by user/group name

I'd probably drop the -v here since I don't need to see all files listead (I wonder if that plays nicely with --info=progress2?). -x is going to take care of skipping mounts like /dev, /proc, /sys. -HAX looks good. -S seems reasonable? -W I guess?

Let's try it with -v first:

rsync -avxHAXSW --numeric-ids --info=progress2 / /media/newvolume/

root@deploy-instance:~$ rsync -avxHAXSW --numeric-ids --info=progress2 / /media/newvolume/
sending incremental file list
./
bin -> usr/bin
lib -> usr/lib
lib32 -> usr/lib32
lib64 -> usr/lib64
libx32 -> usr/libx32
sbin -> usr/sbin
boot/
boot/System.map-5.15.0-52-generic
     12,494,544  11%   20.67MB/s    0:00:00 (xfr#2, ir-chk=1039/1065)
boot/config-5.15.0-52-generic
     13,018,266  11%   19.17MB/s    0:00:00 (xfr#4, ir-chk=1037/1065)
boot/initrd.img -> initrd.img-5.15.0-52-generic
boot/initrd.img-5.15.0-52-generic
     52,744,423  48%   24.63MB/s    0:00:02  ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(503) [generator=3.2.3]

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(703) [sender=3.2.3]

OK, that's doing per-file progress updates. I don't need that at all.

rsync -axHAXSW --numeric-ids --info=progress2 / /media/newvolume/

Much nicer - now I've got a percentage progress bar (which leaps around a bit?) and not too much output. Just right.

The copy does its thing.

Launch with new volume

Since I was doing "old fashioned method" I figured this next would work as though for a physical machine - I'd detach the old volume and leave only the newer one connected.

I found I couldn't detach the original volume from the existing instance. The instructions actually say "Attach the new volume and the original to another instance.".

I'd already done the clone so instead I launched a new instance with the new volume.

Using Launch Instance

Details
- name: deploy-instance-2
- source: volume deploy-volume
- flavor: c2r4
- networks: deploy-network
- secgroups: deploy-secgroup
- key pair: chrisburgess

Then I'll need to move the allocated IP over from the original. Under Floating IPs

Disassociate the existing IP (lol)
Associate it to the new instance

This didn't boot up (ok, maybe I did need to run grub, sigh).

Couldn't detach the new volume from the new instance to put it back on the old instance. Ugh.

Tried uploading the volume to image, no dice. Tried a few things to see if I could "unlock" the volume (unlock the instance, set the volume not bootable) but nope. Huh.

That didn't work, let's try again

I deleted the new instance created above, so I had just the original additional volume. I attached that to the original instance. (Thank heck I didn't get eager and delete the instance early.)

Inspect the attached volume's partition table:

root@deploy-instance:~# fdisk -l /dev/vdb
Disk /dev/vdb: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@deploy-instance:~# fdisk -l /dev/vda
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6845360D-8391-4A07-9961-267A575B1271

Device      Start      End  Sectors  Size Type
/dev/vda1  227328 20971486 20744159  9.9G Linux filesystem
/dev/vda14   2048    10239     8192    4M BIOS boot
/dev/vda15  10240   227327   217088  106M EFI System

Partition table entries are not in disk order.

Clone the old partition to the new volume's partition:

root@deploy-instance:~# dd if=/dev/vda of=/dev/vdb bs=16M status=progress
10737418240 bytes (11 GB, 10 GiB) copied, 239 s, 44.9 MB/s
640+0 records in
640+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 240.109 s, 44.7 MB/s

Take another look at the partition table on the new volume:

root@deploy-instance:~# fdisk -l /dev/vdb
GPT PMBR size mismatch (20971519 != 83886079) will be corrected by write.
The backup GPT table is not on the end of the device.
Disk /dev/vdb: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6845360D-8391-4A07-9961-267A575B1271

Device      Start      End  Sectors  Size Type
/dev/vdb1  227328 20971486 20744159  9.9G Linux filesystem
/dev/vdb14   2048    10239     8192    4M BIOS boot
/dev/vdb15  10240   227327   217088  106M EFI System

Partition table entries are not in disk order.

Expand the partition on the new volume:

root@deploy-instance:~# growpart /dev/vdb 1
CHANGED: partition=1 start=227328 old: size=20744159 end=20971487 new: size=83658719 end=83886047

Take another look of the changed partition table:

root@deploy-instance:~# fdisk -l /dev/vdb
Disk /dev/vdb: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6845360D-8391-4A07-9961-267A575B1271

Device      Start      End  Sectors  Size Type
/dev/vdb1  227328 83886046 83658719 39.9G Linux filesystem
/dev/vdb14   2048    10239     8192    4M BIOS boot
/dev/vdb15  10240   227327   217088  106M EFI System

Partition table entries are not in disk order.

Mount in the required things to chroot and run grub:

root@deploy-instance:/mnt# mount /dev/vdb1 /mnt
root@deploy-instance:/mnt# for i in /dev /dev/pts /proc /sys /run ; do mount -B $i .$i ; done
root@deploy-instance:/mnt# chroot /mnt
root@deploy-instance:/# mount
/dev/vdb1 on / type ext4 (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1989100k,nr_inodes=497275,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=401816k,mode=755,inode64)

Grub it:

root@deploy-instance:/# grub-install /dev/vdb
Installing for i386-pc platform.
Installation finished. No error reported.
root@deploy-instance:/# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50-cloudimg-settings.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.0-52-generic
Found initrd image: /boot/initrd.img-5.15.0-52-generic
Found linux image: /boot/vmlinuz-5.15.0-48-generic
Found initrd image: /boot/initrd.img-5.15.0-48-generic
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
root@deploy-instance:/#
exit

Now detach the new volume and launch a new compute instance, selecting "Volume" under Source. If the volume doesn't appear in the list you may need to edit the volume first and check "Bootable".

Yay, and it came up OK! Now I can shelve then later delete the old instance.

This got me thinking about driving built versions of the site from Gitlab CI => Catalyst Cloud Openstack. I know that's what Cove offers but it requires K8s and the config overhead feels pretty big.

Easy for me to say after making a LEMP stack though, there's a lot more to maintaining infra than spinning it up.

I think the entry tier nVME storage would be a better default option honestly - it's similarly priced ($0.21/GB/month for spinning disk, $0.23/GB/month for nVME with double the burst limits). Current defaults will mean newly onboarded customers experience spinning disk until they migrate away. ↩︎

Gitlab CI Runner

2022-10-23T04:55:00Z

This doesn't need to be on the same instance as you're deploying to - in fact it shouldn't be. For the purposes of this prototype I did re-use the same instance. You could split to a second instance if you want.

SSH into the instance you're going to host the Gitlab CI runner on.

Install Docker

We're following the Docker documentation for install Docker Engine on Ubuntu here.

sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo usermod -a -G docker ubuntu

Log out and back in to get the new group docker for your user. Test Docker.

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:18a657d0cc1c7d0678a3fbea8b7eb4918bba25968d3e1b0adebfa71caddbc346
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

Great, that worked.

Install & run Gitlab Runner

Now we're going to follow Gitlab's run Gitlab Runner in a container docs.

You might want to put the configuration directory somewhere other than /etc/gitlab-runner, up to you.

$ sudo mkdir /etc/gitlab-runner
$ docker run -d --name gitlab-runner --restart always \
  -v /etc/gitlab-runner:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

Obtain a registration key from your Gitlab instance and register the runner (docs) with your Gitlab instance.

$ docker run --rm -it -v /etc/gitlab-runner:/etc/gitlab-runner gitlab/gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=6 revision=0d4137b8 version=15.5.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.example.org
Enter the registration token:
GR123456abcdefg-dddddEEEEEfff
Enter a description for the runner:
[123abc456def]: deploy-runner
Enter tags for the runner (comma-separated):
docker
Enter optional maintenance note for the runner:

Registering runner... succeeded                     runner=GR123456abcdefg-x
Enter an executor: docker, shell, virtualbox, docker+machine, docker-ssh+machine, custom, docker-ssh, parallels, ssh, instance, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.7):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

Once your runner shows up as connected in Gitlab on Settings > CI > Runners, you can move on to configure a deployment pipeline from CI to webserver.

LEMP stack

2022-10-23T04:54:00Z

We've got a fresh Ubuntu install by way of create a Catalyst Cloud compute instance. Now we'll install a simple LEMP stack (Linux, NginX, MariaDB, PHP).

First, decide your project name - eg I'm using this to test for CLIENT, so I'll use client where PROJECT appears below. You also need the IP address of your webserver. The instance IP from is shown below as x.x.x.x below, but you should use your provisioned IP from create a Catalyst Cloud compute instance.

SSH into your instance.

ssh ubuntu@x.x.x.x

Install MariaDB.

$ sudo apt update && sudo apt install -y nginx mariadb-server
$ sudo mysql_secure_installation

Create a DB and grant access.

$ sudo mariadb

MariaDB [(none)]> create database client;
Query OK, 1 row affected (0.001 sec)
MariaDB [(none)]> grant all on client.* to client@localhost identified by 'client';
Query OK, 0 rows affected (0.009 sec)
$ mariadb -u client -p

Install PHP and required libraries for Drupal 9.

$ sudo apt install -y php-fpm php-mysql php-curl php-gd php-json php-mbstring php-xml php-curl php-gd php-mbstring php-xml

Create nginx config in /etc/nginx/sites-available/PROJECT using config/nginx/drupal.conf as a reference implementation. Symlink to the NginX config to enable it.

sudo ln -s /etc/nginx/sites-available/client /etc/nginx/sites-enabled/

Reload NginX.

sudo nginx -t && sudo nginx -s reload

Create a test .php file.

mkdir /var/www/PROJECT/current/web
echo '<?php phpinfo();' > /var/www/PROJECT/current/web/index.php

Verify that the phpinfo works when accessed over http (use the IP address) by visiting http://x.x.x.x in your browser.

If it works, we'll remove it to prepare for the next step.

$ rm /var/www/client/current/web/index.php

Now we need to ensure the deployment can SSH in and exec commands as a user. In this example we'll use ubuntu as the deploy and login user. In a real setup we'd use separate users.

Generate an SSH key on your local system.

$ ssh-keygen -t ecdsa -f ~/.ssh/deploy-key
Generating public/private ecdsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ubuntu/.ssh/deploy-key
Your public key has been saved in /home/ubuntu/.ssh/deploy-key.pub
The key fingerprint is:
SHA256:abcdefabcdefabcdef/abcdefabcdef ubuntu@deploy-instance
The key's randomart image is:
+---[ECDSA 256]---+
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
+----[SHA256]-----+

Add the SSH key to deploy user .ssh/authorized_keys on the compute instance.

Test SSH access from your computer.

ssh -i ~/.ssh/deploy-key ubuntu@x.x.x.x

Next up: Set up a Gitlab CI runner.

Catalyst Cloud Instance

2022-10-23T04:53:00Z

These are very brief instructions to get an instance up and running with a recent Ubuntu release. This is basically what's documented at more length in Catalyst Cloud Docs: . The Ubuntu image we use will mean we end up with a default user account of ubuntu.

We need to be logged into a Catalyst Cloud account with some available resources. I'm using test.chc.cat-it.co.nz here, in region nz-por-1.

Log into Catalyst Cloud
Create an SSH key (or you can re-use an existing key of yours)
Create a network
- name = deploy-network
- subnet = 192.168.x.0/254
Create a router
- name = deploy-router
Add an interface to the router, connecting to the network's subnet
Create a security group (or re-use an existing again)
- name = deploy-secgroup
- ingress http, https, ssh for 0.0.0.0/0
Create and launch an instance
- instance name = deploy-prototype
- source = ubuntu-22.04-x86_64
- flavor = c1.c2r4
- network = deploy-network
- security group =
- ssh key = yours
- launch instance
- associate a floating IP to the instance

When it's done, you should be able to ssh ubuntu@x.x.x.x where x.x.x.x is the IP associated with the compute instance

Next: Set up a webserver environment

Contriblog

2022-09-25T04:26:00Z

ContribLog (adaptdk/contrib) is a tool for capturing contribution records on the commandline.

On first run the user does some interactive setup of their organisation and first project. After that, users can select the project and interactively add contributions. Trying it out, it's a lightweight but manual method for capturing a personal record of contributions.

www-data@ea69a85560cf:~$ ./contriblog add
The "contributions.yml" file does not exist yet. Do you want to generate initialize it? (y/n) y
[1/2] What is the name of the organization? Catalyst IT
What is main URL for the organization? https://catalyst.net.nz
[1/7] Which project received the contribution?
  [0] New project
 >
What is the machine name of the project? (E.g. drupal/migrate_plus): dmore/behat-chrome-extension
What is the name of the project? (E.g. Migrate Plus): Behat Chrome Extension
What is the main URL of the project? (E.g. https://www.drupal.org/project/migrate_plus): https://gitlab.com/behat-chrome/behat-chrome-extension
Please provide tags for the project, e.g. drupal (one per line)
Use EOL to finish, e.g. Ctrl+D on an empty line to finish input
behat
testing

[2/7] Please give the contribution a title: Some thing
[3/7] What was the main type of the contribution? [default: code]
  [code            ] Writing Code
  [code_review     ] Reviewing Code
  [triage          ] Bug Triaging
  [qa              ] Quality Assurance and Testing
  [security        ] Security-Related Activities
  [localization    ] Localization/L10N and Translation
  [event           ] Event Organization
  [documentation   ] Documentation Authorship
  [community       ] Community Building and Management
  [teaching        ] Teaching and Tutorial Building
  [troubleshooting ] Troubleshooting and Support
  [creative        ] Creative Work and Design
  [ux              ] User Interface, User Experience, and Accessibility
  [social          ] Social Media Management
  [user-support    ] User Support and Answering Questions
  [write           ] Writing Articles
  [public_relations] Public Relations - Interviews with Technical Press
  [speak           ] Speaking at Events
  [marketing       ] Marketing and Campaign Advocacy
  [website         ] Website Development
  [legal           ] Legal Council
  [financial       ] Financial Management
 > teaching
[4/7] Who is making the the contribution?
  [0] New person
 >
What is the name of the person? (E.g. José Saramago): Chris Burgess
What is the identifier for the person? (E.g. Jose): chris
[5/7] When was the contribution first published? (E.g. 2020-01-22) [default: 2022-09-25]:
[6/7] How would you describe the contribution? (multiline)
Use EOL to finish, e.g. Ctrl+D on an empty line to finish input
Teaching a workshop
[7/7] Please provide public links related to the contribution? (one per line)
Use EOL to finish, e.g. Ctrl+D on an empty line to finish input
https://gitlab.com/behat-chrome/behat-chrome-extension/-/issues/123
The file "contributions.yml" was updated correctly

On second run, we won't get so many questions:

www-data@ea69a85560cf:~$ ./contriblog add
[1/7] Which project received the contribution?
  [0] New project
  [1] dmore/behat-chrome-extension
 >

And contributions are logged to a local .yml file:

www-data@ea69a85560cf:~$ cat contributions.yml
organization:
    name: Catalyst IT
    url: 'https://catalyst.net.nz'
people:
    xurizaemon: 'Chris Burgess'
projects:
    dmore/behat-chrome-extension:
        name: 'Behat Chrome Extension'
        url: 'https://gitlab.com/behat-chrome/behat-chrome-extension'
        tags:
            - drupal
contributions:
    -
        project: catalyst-it/something
        title: 'Some thing'
        type: teaching
        who: chris
        start: 2022-09-25T00:00:00+00:00
        description: 'Deliver presentation'
        links:
            - 'https://github.com/catalyst-it/something/issues/123'

What I like about the tool is that it enables tracking of non-code contributions on an equal standing with code ones. This tool could suit CLI users if we asked staff to manually track contributions. It uses PHP, but could be packaged as a phar for distribution.

Engari ko te raruraru

The chief challenge of this tool for me is ... it's a lot like timesheeting, right? Timesheeting is effort, and introducing a new task steals energy from other tasks. I sense anyone who's ever had to chase timesheets knows it's not reliable either.

That's why I think that if we can do so responsibly, we should prefer unobtrusive tools to gather data in the first instance, and bolster automated approaches with ways of capturing this information together - perhaps a discussion roundup at the end of the week, in which we might capture a handful of contributions. A team's contriblog output might be captured there and consumed by a tool like GrimoireLab.

There are some other challenges/opportunities too - eg I see that the contribution is associated with an individual in the example above? Perhaps the tool could push contributions entered to a centralised (organisation or community) service as well. The UX is that of a decent prototype and could be improved.

Chaoss Metrics Test Run

2022-09-25T03:23:00Z

CHAOSS Metrics is a series of proposed metrics for a project to assess its health. A metric might be something like "how many clones of this repo were made in the most recent 90 days". Metrics such as these, proposed by CHAOSS for measuring OSS ecosystem health, could be useful for organisations or OSS communities to self-assess. This has the advantage that process doesn't need to be invented by each organisation individually, and that organisations can use a common yardstick to compare success.

Metrics have their challenges (Goodhart's law: "When a measure becomes a target, it ceases to be a good measure.", or many of these quotes), so we have to use them carefully and appropriately. Measuring things is helpful when appropriate and these metrics seem to be a pretty well considered set. It's interesting to consider how these metrics might compare when evaluated against the needs of Catalyst as a business, and the needs of the OSS communities we engage with.

An example metric: Contributions - Clones

Number of clones in a time period is available from both Github and Gitlab APIs, apparently. A Git server would be able to report on this too - most would already have the data recorded in its logs.

Trying to extract the data from Gitlab - using Gitlab's GraphQL explorer, I don't see this stat available in GraphQL:

# Query for project details.
query {
  project(fullPath: "behat-chrome/behat-chrome-extension") {
    id
    statistics {
      containerRegistrySize
      pipelineArtifactsSize
      snippetsSize
      uploadsSize
      wikiSize
    }
  }
}

Curl has it:

curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/projects/3329075/statistics

But, it's forbidden for projects you aren't admin of? Got a 403 for https://gitlab.com/api/v4/projects/278964/statistics (gitlab's repo).

Drupal Core patch testing locally with Lando

2021-12-22T22:33:00Z

Testing patches for Drupal core can be fiddly, but it's a process worth making easy so we can better engage in upstream fixes.

Here's my current approach (as of 2021).

Prerequisites

You'll need Lando installed, which requires Docker as well.

Upstream codebase

I keep a long-lived copy of the Drupal core codebase in ~/Projects/drupal-core. This is a git clone of https://git.drupalcode.org/project/drupal.git and the workflow is either to create patches for upload (old style) or use Gitlab's merge requests (new style) to submit proposed changes.

So, step one: clone a copy of Drupal core! Note that I change the cloned repo name to "drupal-core" for my own sanity.

cd ~/Projects # or whereever you'd like to keep it
git clone --branch=9.4.x https://git.drupalcode.org/project/drupal.git drupal-core

You can switch between branches of upstream, but we want to default to the current dev branch for proposed changes. Most patches will apply to your current version.

Local Lando environment

I like Lando and this setup works for me. We need to add a handful of files into the core codebase, and then not accidentally commit those files. These files are:

.lando.yml
.lando.base.yml
phpunit.xml
sites/default/settings.php

It may feel unfamiliar that the Drupal core codebase has the codebase as the root of the repo - there's no web/ subdirectory. For projects, we use the web/ subdirectory, but for core, we don't use that. 🤷 It's OK!

Additions

These are the files I put into the Drupal codebase:

`.lando.yml`

This file and the next could be combined, and in this project there's little advantage to the separation. However, I track these Lando configs across multiple projects, so the separation is an advantage to me in that I can copy changes to the base file more easily across various Drupal projects.

name: drupal-core
config:
  # Change to `true` and run `lando rebuild` to enable XDebug.
  xdebug: false

`.lando.base.yml`

name: unconfigured
recipe: drupal9
env_file:
  - .env
config:
  webroot: .
  database: mysql
  via: nginx
services:
  appserver:
    build_as_root:
      - apt update && apt install libxslt1-dev -y && docker-php-ext-install xsl
    build:
      - composer install --prefer-dist
  mailhog:
    type: mailhog
  node:
    type: node:14
tooling:
  phpunit:
    service: appserver
    cmd: "php /app/vendor/bin/phpunit -c /app/phpunit.xml"
  phpcs-install:
    service: appserver
    cmd:
      - composer require --dev squizlabs/php_codesniffer
      - phpcs --config-set installed_paths /app/vendor/drupal/coder/coder_sniffer
  phpcs:
    service: appserver
    cmd: /app/vendor/bin/phpcs --standard=Drupal,DrupalPractice
  phpcbf:
    service: appserver
    cmd: /app/vendor/bin/phpcbf --standard=Drupal,DrupalPractice
  site-install:
    service: appserver
    cmd: drush site:install -y

`phpunit.xml`

This configures PHPUnit to run in this environment.

<?xml version="1.0" encoding="UTF-8"?>
<!-- TODO set checkForUnintentionallyCoveredCode="true" once https://www.drupal.org/node/2626832 is resolved. -->
<!-- PHPUnit expects functional tests to be run with either a privileged user
 or your current system user. See core/tests/README.md and
 https://www.drupal.org/node/2116263 for details.
-->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" bootstrap="/app/core/tests/bootstrap.php" colors="true" beStrictAboutTestsThatDoNotTestAnything="true" beStrictAboutOutputDuringTests="true" beStrictAboutChangesToGlobalState="true" failOnWarning="true" printerClass="\Drupal\Tests\Listeners\HtmlOutputPrinter" cacheResult="false" xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.3/phpunit.xsd">
  <php>
    <!-- Set error reporting to E_ALL. -->
    <ini name="error_reporting" value="32767"/>
    <!-- Do not limit the amount of memory tests take to run. -->
    <ini name="memory_limit" value="-1"/>
    <!-- Example SIMPLETEST_BASE_URL value: http://localhost -->
    <env name="SIMPLETEST_BASE_URL" value="http://appserver_nginx" force="true"/>
    <!-- Example SIMPLETEST_DB value: mysql://username:password@localhost/databasename#table_prefix -->
    <env name="SIMPLETEST_DB" value=""/>
    <!-- Example BROWSERTEST_OUTPUT_DIRECTORY value: /path/to/webroot/sites/simpletest/browser_output -->
    <env name="BROWSERTEST_OUTPUT_DIRECTORY" value=""/>
    <!-- To have browsertest output use an alternative base URL. For example if
     SIMPLETEST_BASE_URL is an internal DDEV URL, you can set this to the
     external DDev URL so you can follow the links directly.
    -->
    <env name="BROWSERTEST_OUTPUT_BASE_URL" value=""/>
    <!-- To disable deprecation testing completely uncomment the next line. -->
    <!-- <env name="SYMFONY_DEPRECATIONS_HELPER" value="disabled"/> -->
    <!-- Example for changing the driver class for mink tests MINK_DRIVER_CLASS value: 'Drupal\FunctionalJavascriptTests\DrupalSelenium2Driver' -->
    <env name="MINK_DRIVER_CLASS" value=""/>
    <!-- Example for changing the driver args to mink tests MINK_DRIVER_ARGS value: '["http://127.0.0.1:8510"]' -->
    <env name="MINK_DRIVER_ARGS" value=""/>
    <!-- Example for changing the driver args to webdriver tests MINK_DRIVER_ARGS_WEBDRIVER value: '["chrome", { "chromeOptions": { "w3c": false } }, "http://localhost:4444/wd/hub"]' For using the Firefox browser, replace "chrome" with "firefox" -->
    <env name="MINK_DRIVER_ARGS_WEBDRIVER" value=""/>
  </php>
  <testsuites>
    <testsuite name="unit">
      <file>./core/tests/TestSuites/UnitTestSuite.php</file>
    </testsuite>
    <testsuite name="kernel">
      <file>./core/tests/TestSuites/KernelTestSuite.php</file>
    </testsuite>
    <testsuite name="functional">
      <file>./core/tests/TestSuites/FunctionalTestSuite.php</file>
    </testsuite>
    <testsuite name="functional-javascript">
      <file>./core/tests/TestSuites/FunctionalJavascriptTestSuite.php</file>
    </testsuite>
    <testsuite name="build">
      <file>./core/tests/TestSuites/BuildTestSuite.php</file>
    </testsuite>
  </testsuites>
  <listeners>
    <listener class="\Drupal\Tests\Listeners\DrupalListener">
    </listener>
  </listeners>
  <!-- Filter for coverage reports. -->
</phpunit>

`sites/default/settings.php`

//$settings['config_sync_directory'] = '../config/drupal/sync';
$databases['default']['default'] = array (
  'database' => 'drupal9',
  'username' => 'drupal9',
  'password' => 'drupal9',
  'prefix' => '',
  'host' => 'database',
  'port' => '',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

How to use it

With these in place, I should be able to execute lando start and bring up a workable Drupal 9 site.

More importantly, I should be able to execute lando phpunit to run core tests.

I'll now follow the instructions from the issue to check out a local working branch with the existing MR or patch applied.

I can use PHPUnit's --group and --filter to run specific tests (eg those related to the issue, or those already failing).

Dependency Track

2021-11-10T19:51:00Z

Below are my notes for the lightning talk above, given at DrupalSouth Shorts 2021.

The situation

We like components, they allow composition of complex things
This means we need to understand the components we're using
As an organisation, we'll have many projects with many components
There are advantages to understanding this - reducing complexity by consolidation

Our context, briefly

Many DrupalSouths ago (2010?!), Catalyst released Archimedes
Over time, Archimedes became Mataara
Other tools exist for various spaces - what system packages are installed, host info etc
We want to consolidate on process and reduce complexity - better team process, less siloing/divergence
For the same reasons as we might use a tool like this, 11 years later we don't need to write our own tool

Software composition analysis for development

For a Drupal project, hundreds (to thousands) of packages in the dependency tree
Applies outside Composer - NPM, frameworks, containers, OS, firmware
What are we running? What else is running x?
What vulnerabilities do we need to care about?
What license considerations are there?

Dependency Track

Enter Dependency Track, an OWASP project to analyse, process and report on composition
v4.3.6 today
Ran it up in Catalyst Cloud for purpose of demo

How it works

BOM = Bill of Materials
Uses a BOM format called CycloneDX
Turns composer.json/lock, package.json/package-lock.json into BOM
Automated inbound data (via your CI)
Imports into Dependency Track, analyses & extracts metadata (version, license, ...)
Sources vulnerability intelligence externally - public and licensed
Configure to apply your organisational rules - license, reporting, alerts
Docker install was pretty easy, maybe configure SSL first cos CORS failure + static frontend took a minute to spot
Give it lots of RAM, it has to eat all the vulns that came out in 2005 - failed at 4GB, worked at 16GB, thanks Catalyst Cloud :)
Give it half an hour to haul all the data in initially - you can use it in that time - and let it chug through analysis on its own time
Feed it from CI or manually at first

What do you get?

High level reporting of vulnerabilities
Alerting / notification for newly identified vulnerabilities
Identify policy violations on licensing
Drill into projects, components, search, report
Export report data for signoff

What can this mean

Reference for developers considering a new component
Added review step for new components - firming up our approach here

What else is there?

Tidelift provides a similar paid, hosted service
... tell me about other things?

What did you learn?

Some projects aren't licensed!
Some repackage licensed things without add a license!
Vulnerabilities view shows all known vulns?
Lots of CLI tools to handle turning everybody's Gemfile/Composer/whatever into a BOM
Had to remove Byte Order Markers from Bill of Materials XML

Update. Your. Shit.

2021-09-27T22:05:00Z

A few years back, I was working on a migration project. A project which shall not be named. Tonight I was reminded of a very particular thing I ran into there.

As part of the migration process, I had to regularly reload DBs from the customer source. That process took a long time. The compressed DBs weren't big, but the DB loading took ... longer than felt right.

Eventually, and with the time that you end up with waiting on a slow task, I made an effort to ask myself why. Looking, I saw the DB load spent ages writing to a particular table: {epsacrop_files}.

This table stored image crop settings for every image in the Drupal content DB. The intent was to store an array of dimensions, but a wrong JS expectation in the module's admin UI -

var presets = [];
presets[1234] = { x: 123, y: 456 };
// presets.length = 1235

had resulted in every image stored being padded with a vast number of NULL values before you got to the dimensions.

This bug (epsacrop#1823940: Simplify data storage to avoid massive amounts of null values was fixed in 2015, but the copy of the site we'd been supplied was still using a version that predated the fix, and so the busy content DB had many GiB of NULL in it. A massive DB table full of almost entirely NULL, NULL, NULL, NULL.

What a waste of developer time, leaving that in the system for years!

Drupal Core Patches With Lando

2021-09-13T01:55:00Z

I have been working on a tidy setup for contribution using available community tools, and today I needed the same for core. Core patches and Contrib have slightly different workflows.

Anyway, it turned out really easy to do this.

Get your codebase

I already had a copy of the Drupal core codebase in ~/Projects/src/drupal, checked out to my branch (3232222-pager-no-h4).

Copy in `.lando.*`

I copied in the following two Lando .yml files:

.lando.yml

name: drupal-core
config:
  # Change to `true` and run `lando rebuild` to enable XDebug.
  xdebug: false

.lando.base.yml

name: unconfigured
recipe: drupal9
env_file:
  - .env
config:
  webroot: .
  database: mysql
  via: nginx
services:
  appserver:
    build_as_root:
      - apt update && apt install libxslt1-dev -y && docker-php-ext-install xsl
    build:
      - composer install --prefer-dist
  memcache:
    type: memcached:1
  mailhog:
    type: mailhog
  node:
    type: node:14
tooling:
  phpunit:
    service: appserver
    cmd: "php /app/vendor/bin/phpunit -c /app/core/phpunit.xml.dist"
  # lando composer require --dev squizlabs/php_codesniffer
  # lando phpcs --config-set installed_paths /app/vendor/drupal/coder/coder_sniffer
  phpcs:
    service: appserver
    cmd: /app/vendor/bin/phpcs --standard=Drupal,DrupalPractice
  phpcbf:
    service: appserver
    cmd: /app/vendor/bin/phpcbf --standard=Drupal,DrupalPractice

Start your engines

lando start

Run tests

PHPUnit has various ways of filtering which tests to run. In this case I had only one test I needed to run:

lando phpunit core/tests/Drupal/KernelTests/Core/Theme/ConfirmClassyCopiesTest.php

It's fast :)

$ lando phpunit core/tests/Drupal/KernelTests/Core/Theme/ConfirmClassyCopiesTest.php
PHPUnit 8.5.19 by Sebastian Bergmann and contributors.

Testing Drupal\KernelTests\Core\Theme\ConfirmClassyCopiesTest
.....                                                               5 / 5 (100%)

Time: 2.23 seconds, Memory: 6.00 MB
OK (5 tests, 2481 assertions)

Upload your changes

Pushed my changes to the merge request per the instructions on the issue. Done!

For more information on contributing to Drupal, see Contributor Guide: Quick Info.

Testing a fork of Sector distribution

2021-08-03T12:44:00Z

This is my notes on how to test a fork of Sector. This is tricky because we are using composer create-project and Composer needs to recursively resolve package dependencies, and we're testing a change in a required package.

Background

We want to test that running composer create-project sparksinteractive/sector-project will do the right thing with a patch which we've submitted to a dependency, sparksinteractive/sector-distribution. Our change will be that we are modifying composer.json in the distribution project to use an updated patch to another contrib module.

sector-project is the template, which is used with composer create-project
sector-distribution is the profile, which depends on a large number of contrib modules

Preparation

Identify your working directory
Clone both projects there
Modify sector-project repository to incorporate the local copy of sector-distribution
Modify sector-distribution to incorporate the upstream patch
Run create-project with some additional args
Verify that the patch is applied

Caveats

If Composer appears to not notice your changes, clear your Composer cache with composer clearcache. This can add ~5min to your next download run.

Process

Identify your working directory

I'm going to do this in /tmp so all paths are short.

Clone both projects there

cd /tmp
git clone https://github.com/sparksi/sector-distribution.git sector-distribution
git clone https://github.com/sparksi/sector-project.git sector-project

Modify `sector-project` to reference the local copy of `sector-distribution`

In /tmp/sector-project, modify composer.json to add a new local repository entry in the repositories key.

commit ee8af146a1643f3cfce4de6fafe00af77ab227a2 (HEAD -> local-distro)
Author: Chris Burgess <chris.burgess@catalyst.net.nz>
Date:   Tue Aug 3 23:44:53 2021 +1200

    Use local copy of sector-distribution

diff --git a/composer.json b/composer.json
index d594d36..20e3bf0 100644
--- a/composer.json
+++ b/composer.json
@@ -39,6 +39,16 @@
         }
     },
     "repositories": {
+        "sector-distribution-local": {
+            "type": "path",
+            "url": "/tmp/sector-distribution",
+            "options": {
+                "symlink": false,
+                "versions": {
+                    "sparksinteractive/sector-distribution": "9.1.4"
+                }
+            }
+        },
         "packagist.drupal": {
             "type": "composer",
             "url": "https://packages.drupal.org/8"

key	value	description
type	path	use the current state of this local directory
url	/tmp/sector-distribution	the local path to your modified distribution codebase
options.symlink	false	copy the files into place
options.versions	sparksinteractive/sector-distribution: 9.1.4	increment above the current release

Modify `sector-distribution` to reference the upstream patch we want

In /tmp/sector-distribution, modify composer.json to update the patch URL we want to change. (This could be any other change to the local distro codebase.)

commit a5dd78ffbb61667f5a191467f6d767d980b6f01c (HEAD -> patch-updates)
Author: Chris Burgess <chris.burgess@catalyst.net.nz>
Date:   Tue Aug 3 23:47:49 2021 +1200

    Update patch for Menu Block

diff --git a/composer.json b/composer.json
index be7ab2e..aa931e4 100644
--- a/composer.json
+++ b/composer.json
@@ -103,7 +103,7 @@
         "patches": {
             "drupal/menu_block": {
                 "No contextual link to edit menu": "https://www.drupal.org/files/issues/2799821-2.menu_block.contextual_links.patch",
-                "Hide menu block title if empty": "https://www.drupal.org/files/issues/2018-05-08/menu_block-hide_block_if_no_links-2757215-9.patch"
+                "Hide menu block title if empty": "https://www.drupal.org/files/issues/2020-02-12/menu_block-hide_block_if_no_links-2757215-13.patch"
             },
             "drupal/entity_usage": {
                 "Better tracking per entity": "https://www.drupal.org/files/issues/2020-06-16/per-node-tracking_3151920_4.patch"

Run `create-project` with the additional args required

Switch back to the parent directory, and run composer create-project with a --repository parameter which will instruct Composer to discover the local copy of sector-project and create the templated site in subdirectory sector-test.

cd /tmp
composer create-project --prefer-dist --repository='{"type":"path", "url":"/tmp/sector-project","options":{"symlink":false, "versions":{"sparksinteractive/sector-project":"9.1.4"}}}' sparksinteractive/sector-project sector-test

Keys and values here are much as documented in the changes for sector-project.

Verify that the output contains the correct applied patch

  - Applying patches for drupal/menu_block
    https://www.drupal.org/files/issues/2799821-2.menu_block.contextual_links.patch (No contextual link to edit menu)
    https://www.drupal.org/files/issues/2020-02-12/menu_block-hide_block_if_no_links-2757215-13.patch (Hide menu block title if empty)

Review Achievements

2021-04-29T21:34:00Z

At the end of 2020 I was disappointed to realise with COVID that we hadn't dived into our usual annual review, and I did my own personal annual review over the new years break which was really helpful to me. (Also, it didn't just cover work things!)

When I talked to my manager in early 2021 about the GROW/GLOBE/GROWTH worksheet, I realised I had this in my journal upstairs. While my review was a lot due to some challenging things I'd been dealing with, I was really pleased to have a solid list of achievements for the year at hand!

Here's what I had:

Achievements

Maintained & improved my development and project delivery standards and practices over the year, with reference to the wider open source community and Catalyst's internal standards eg The Catalyst Test.

Worked to deliver positive change through leading by example. This included:

Continuing program of establishing or improving test / CI coverage across all projects I engaged with.
Advocating for and normalising code review as an requirement of project process.
Improving project documentation to ensure clear guidance on process and future maintainability.

Contributed to RFPs for Ministry of Education, Treasury, LINZ and (less significantly) NZ Police. This was a new and challenging area of work to contribute to, and I felt that my efforts were valued and we achieved good outcomes. I enjoy the communication and writing aspects of this work.

Provided peer support and training. Supported team members to become more familiar with testing and CI process, active presence in chat channels to assist developers on their journey.

Ensured that I documented clearly whenever possible. I made 384 edits to the Catalyst Wiki in 2020!

Organised and facilitated workshop in Christchurch to share knowledge around CI and testing process. Followed up with additional support on CI improvements across projects for other team members.

Maintained & developed presence in Drupal community by learning and sharing in the open. Attended DrupalSouth 2021 virtual conference, awarded one of several event prizes for engagement. Regularly active in regional and topical Drupal chat channels.

Continued contributing to various open source projects & communities. See:

Contributed to Matrix chat rollout with testing, documentation, planning and bot development. My Catathon project from 2018 formed the basis for a Matrix-compatible bot implementation, which includes thorough documentation. Wormy is risen!

Engaged in collaborative technical leadership through active sharing of development process & learning - see Working Loud blog.

Established personal support / development relationship with staff from another team for co-mentoring & peer development.

Asked to guide and direct establishment of reproducible Drupal project processes and practices in the scope of the Treasury project.

In the first quarter of 2021 I've been nominated and elected to the Eight and invited to participate in the Inclusivity Team. I'm really happy to be engaging in these places as they add depth to my technical work and allow me to support some of the special culture Catalyst has.

I've worked to ensure our team are co-ordinating and communicating in our various systems - organising team meetings, including all members correctly in email distribution lists, raising allocation gaps on projects, and contributing to our best practices WRs and issue queue.

Work and Trails

2021-04-28T06:55:00Z

I had an experience today where someone said, "can I do that, I have a special way to do it". Then they took over the task and made a small change to the process which improved it. Job done, and great - since I saw the approach, I can use that next time.

It got me thinking about types of task though, and allocation within teams. Let's turn this into a model for deciding who should take that ticket off the board.

Let's say there's ten tasks on the board. Some of those tasks demand in depth knowledge and creative solutions, and some are more ordinary problems. Some are trail-blazing, and some are walks on a trail we know.

I don't like "junior/senior" terminology but I'll use it here for now.

If our senior developer picks up a task that the junior could have done with a little guidance, then the senior is wasting both their own resource and a learning opportunity for the team.

So, when you allocate a task, ask yourself: do we need to trailblaze^[1] here, or could we show someone to the start of the path and let them take it from there?

There's a whole lot more to discuss in resisting the desire to innovate when solutions exist. Another time! ↩︎

Team Resources

2021-04-15T07:20:00Z

I've been gathering a bunch of reading on teams and figured I might collect some of it in a post.

Who's in the room

Hire More Women In Tech
Gender Decoder and JobLint, tools to check your job listing for gendered smells
There's more to consider here than gender!

Interviewing

Culture Queries - questions to ask if you're being interviewed for a position. Also if you're reflecting on your current environment.

Getting Better

Values

Gitlab's Values - I love this, myself.
Github Values - some good stuff here too, but Gitlab's resonates with me a lot more.

Drush SQL Sanitize Config

2021-04-13T08:38:00Z

A handy snippet to sanitize DB snippets:

# drush/drush.yml
# ...
sql:
  structure-tables:
    # Data to exclude from default sync/dump.
    common:
      - cache
      - 'cache_*'
      - history
      - queue
      - 'search_*'
      - 'session'
      - watchdog
      - 'webform_*'
  skip-tables:
    # Whole tables to exclude from default sync/dump.
    common:
      - 'migration_*'
      - 'migrate_*'
command:
  sql:
    dump:
      options:
        structure-tables-key: common
        skip-tables-key: common
    sync:
      options:
        structure-tables-key: common
        skip-tables-key: common

Via @sime on Drupal Slack.

Demo: Commerce Demo & Commerce DPS

2021-03-29T03:06:00Z

Last week Frederik asked about DPS on Drupal Commerce (Commerce DPS is a Drupal payment module I maintain, but which I hadn't had a chance to road test the Drupal 8 version of). I figured I should get familiar!

I ran up a Commerce Demo site with DPS payments and threw a Behat test on it locally. Then made that all execute in CI.

Cos I'm thinking about Treasury re-usable process I used my drupal-builder tool to kick things off. This is a tool which implements the "New Drupal Site" process, but is IMO tidier and more reproducible on account of demonstrating ideas from Drupal new project proposal.

I also implemented inserting creds into Drupal configuration from Gitlab variables to CI runs, because I didn't want to expose them on a public project.

I included some example things such as taking screenshots (see CI/CD => latest pipeline => test behat => files) and checking out via configured DPS gateway.

Next step: fix the Commerce DPS tests :)

Drupal Code Checks

2021-03-26T18:02:00Z

There are a bunch of Drupal code check tools about the place (and then plenty more generic PHP ones). These are a few ... I should write them up in more detail?

Hugo archetypes

2021-03-02T04:05:00Z

I should update this version of Hugo and get myself using archetypes to create new posts ...

https://gohugo.io/content-management/archetypes/

Then I won't have to manually set the date every time.

...

OMFG, it works already. I no longer need to set the date. See false.

hugo new --kind post content/posts/2021-02-25-bleh-bleh.md

And now I no longer need to trim out the date at the start of the title.

Automating Gitlab comments in Hugo

2021-02-24T20:05:00Z

OK, let's think about Gitlab as a comments platform for Hugo.

I have an environment where all staff (theoretically) have access to Gitlab.

If they don't, I'll happily post comments on behalf.

Gitlab API permits opening issues and fetching / rendering issues.

This should happen via JS so the comments are current. If not via JS, then we just link to the thread. Maybe copy the article text into the issue description?

nb (notebook), a CLI taker of notes, journal, etc

2021-02-24T04:05:00Z

nb is a command line note-taking, bookmarking, archiving, and knowledge base application with:

plain-text data storage,
encryption,
filtering and search,
Git-backed versioning and syncing,
Pandoc-backed conversion,
global and local notebooks,
customizable color themes,
extensibility through plugins,

and more, all in a single portable, user-friendly script.

See https://xwmx.github.io/nb/ for screenshots etc, it looks cool! No, I haven't tried it (as of writing this).

Benefits of code review for the reviewer

2021-02-24T04:05:00Z

On otsukare

Code Review is an essential part of the process of publishing code. We often talk about the benefits of code review for projects and for people writing the code. I want to talk about the benefits for the person actually reviewing the code.

Understanding The Project

Learning How To Code

Instant Gratification

(read the full post, it's short)

I like Karl's use of Github issues, and should get around to automating same for comments on these posts.

Reproducing Gitlab CI failures in Docker Compose locally

2021-02-11T08:46:00Z

OK, this is a fun situation.

A piece of code works and passes tests locally (local dev environment is a Vagrant) but fails the Drupal install phase in Gitlab CI runner.

The failure is that in the Gitlab CI job, drush site-install task bails out with:

drush site:install -y --db-url=pgsql://example:example@db/example --existing-config

 You are about to:
 * DROP all tables in your 'example' database.

 // Do you want to continue?: yes.                                                                                 

 [notice] Starting Drupal installation. This takes a while.
 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_settings_form
 [notice] Performed install task: install_write_profile
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [warning] The "block_content:b42edb07-dcb6-4be4-aa3b-480fc6e06ccf" was not found
 [notice] Options titles do not exist.
 [notice] Options state_province_names do not exist.

In FieldConfig.php line 312:
                                                                                                
  Attempt to create a field field_not_quite_logged_in that does not exist on entity type user.  
                                                                                                

site:install [--db-url DB-URL] [--db-prefix DB-PREFIX] [--db-su DB-SU] [--db-su-pw DB-SU-PW] [--account-name [ACCOUNT-NAME]] [--account-mail [ACCOUNT-MAIL]] [--site-mail [SITE-MAIL]] [--account-pass ACCOUNT-PASS] [--locale [LOCALE]] [--site-name [SITE-NAME]] [--site-pass SITE-PASS] [--sites-subdir SITES-SUBDIR] [--config-dir CONFIG-DIR] [--existing-config] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--notify] [--druplicon] [--xh-link XH-LINK] [--] <command> [<profile>]...

I tried to reproduce this locally in the existing Vagrant environment, and it did not appear there; drush site-install completed as expected.

I emulate the Gitlab CI environment (which uses a Docker CI runner environment) using Docker Compose to reproduce the failure. Here's the Gitlab CI yaml snippet:

# Reduced version of https://gitlab.example.org/example/example/-/blob/<hash-here>/.gitlab-ci.yml
image: gitlab.example.org:4567/drupal/docker/php:7.2-latest

test behat:
  stage: test
  needs:
    - build theme
    - build drupal
  tags:
    - docker
  services:
    - name: postgres:10.12
      alias: db
  variables:
    HOST_PORT: "0.0.0.0:8888"
    DRUPAL_ENVIRONMENT: ci
    POSTGRES_DB: example
    POSTGRES_USER: example
    POSTGRES_PASSWORD: example
  before_script:
    - echo '<?php $environment = getenv("DRUPAL_ENVIRONMENT");' > web/sites/default/settings.local.php
    - drush site:install -y --db-url=pgsql://example:example@db/example --existing-config
    - php -S $HOST_PORT -t ./web &> tmp/php.log.txt &
    - until curl -s http://$HOST_PORT; do true; done > /dev/null
    - curl -o tmp/index.html -s http://$HOST_PORT
    - cd tests/behat
  script:
    - behat --profile=gitlab_ci --tags='~@broken&&~@drush' -f junit -o $CI_PROJECT_DIR/behat-junit -f pretty -o std --colors

Here's the docker-compose.yml I use to reproduce what the above will do:

version: "3.1"
services:
  db:
    image: postgres:10.12
    environment:
        POSTGRES_DB: example
        POSTGRES_USER: example
        POSTGRES_PASSWORD: example
  app:
    image: gitlab.example.org:4567/drupal/docker/php:7.2-latest
    environment:
        HOST_PORT: "0.0.0.0:8888"
        DRUPAL_ENVIRONMENT: ci
    working_dir: /app
    command: ['php', '-S', '0.0.0.0:8888', '/app/web']
    volumes:
        - ./:/app

I then need to reproduce the behaviour of the Gitlab CI before_script and script steps.

I set the environment in settings.local.php:

/app # echo '<?php $environment = getenv("DRUPAL_ENVIRONMENT");' > web/sites/default/settings.local.php

Then I execute the step which fails:

/app # drush site:install -y --db-url=pgsql://example:example@db/example --existing-config

 You are about to:
 * DROP all tables in your 'example' database.

 // Do you want to continue?: yes.                                                                                 

 [notice] Starting Drupal installation. This takes a while.
 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_settings_form
 [notice] Performed install task: install_write_profile
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [warning] The "block_content:b42edb07-dcb6-4be4-aa3b-480fc6e06ccf" was not found
 [notice] Options titles do not exist.
 [notice] Options state_province_names do not exist.

In FieldConfig.php line 312:
                                                                                                
  Attempt to create a field field_not_quite_logged_in that does not exist on entity type user.  
                                                                                                

site:install [--db-url DB-URL] [--db-prefix DB-PREFIX] [--db-su DB-SU] [--db-su-pw DB-SU-PW] [--account-name [ACCOUNT-NAME]] [--account-mail [ACCOUNT-MAIL]] [--site-mail [SITE-MAIL]] [--account-pass ACCOUNT-PASS] [--locale [LOCALE]] [--site-name [SITE-NAME]] [--site-pass SITE-PASS] [--sites-subdir SITES-SUBDIR] [--config-dir CONFIG-DIR] [--existing-config] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--notify] [--druplicon] [--xh-link XH-LINK] [--] <command> [<profile>]...

This fails, which is a great success! From here, I debugged a little further to establish it was something like a timing issue on first run. Those details ended up in the related client-specific issue, and I kept these notes for sharing here.

The real takeaway for me was that I could readily use Docker Compose to replicate issues I saw in Gitlab CI, and knowing that a significant amount of variation between environments could be eliminated by using the same container images, environment variables, and exact build steps to set up a reproduction.

This experience gave me a decent nudge towards getting rid of long-lived Vagrant instances for development and coalescing practice towards more consistent (but not identical!) environments for development, test and hosting.

Youtube watches

2020-12-21T00:56:00Z

How to classify and sort more than 200 documents in a single bulk upload (DrupalSouth 2020) - this actually didn't deliver, except to inform me that GovCMS permits custom code only via the admin theme, resulting in people using it to do complex form processing. I get why, I get how, I still think it's horrid.
Highly processed migrations - Eric Goodwin (DrupalSouth Hobart 2019)
Understanding Automated Tests in Drupal (DrupalSouth 2020)
Paragraphs Without Pain - Angus Gordon (DrupalSouth 2020)

Trying CodeLingo playbooks

2020-12-16T22:00:00Z

I had a demo of CodeLingo's Playbooks today - an interesting tool they've pivoted to from their static analysis (I think) tool.

CodeLingo are a startup based in Dunedin who built a tool for reviewing and I guess "enshrining" development processes, to support team communication and knowledge sharing about correct procedure. We all like that, right?

CodeLingo's v1 tool looked like this, if I get it right:

on a merge request, a reviewer might identify a block of code that has a "code smell"
reviewer adds a note, saying "This block should use the internal HTTP client rather than this more verbose instantiation of a custom client"
reviewer pings a bot in the MR, saying @codelingobot add rule prefer internal HTTP client
codelingobot adds a rule (name?) to the project ruleset
team expands the rule docs, test and guidance
in future, the code smell is automatically identified

This was done with a custom AST and a query language which comprehends syntax using the AST, so that your rule ends up as an XPath-like description of the incorrect pattern (finding a custom HTTP client instantiation inside classes known to have an HTTP client available).

CodeLingo Playbooks are a variation of this; it's now about capturing blocks of code and describing them in playbooks, so focusing less on rules and more on documenting and communicating (but with the rules capability still available).

This feels partly like functionality I'd enjoy, and partly like functionality I'm already familiar with - I can link to code in context using line numbers so that's pretty easy to communicate amongst team members. But the ability for this documentation to track the referenced code via the AST query selectors is cool.

I've committed to give it a go documenting some things and see how useful it is to me as part of my workflow.

I do think the "playbook" name is a bit confusing to me, and it probably conveys an aspect of the product I'm missing.

DrupalGov Sprint and Sector

2020-11-04T05:23:00Z

I joined the DrupalGov Sprint yesterday - it was my first online-only sprint ... not quite the same as an IRL one, that's for sure!

However :) I got to meet one key contact, Daniel Veza (team lead on Sector) and we had a good chat, he seemed enthusiastic about contributions and was open to

Here's my planned schedule for Thursday:

9:20 KEYNOTE: Government as a Platform
10:05 Unsure
10:30am How to classify & sort more than 200 documents in a single bulk upload
10:55am Unsure ... Meet the Moment: Delivering Results with WebOps?
11:15am Drupal 9 and CiviCRM
11:40am KEYNOTE: 2020. That was epic! A tale of gargantuan scale.
1:10pm Drupal Security Panel
1:35pm Managing large (and smaller) site portfolios with Lagoon Insights
2:00pm Strategies for Managing a Fleet of Drupal Projects
2:20pm Networking & Break
2:40pm Securing and Scaling Drupal with Kubernetes Panel
3:25pm Networking & Break
3:05pm Understanding Automated Tests in Drupal OR GovCMS and Drupal 9, roadmap and Q&A
4:10pm Closing Remarks

Renovate - Automated update MRs

2020-10-27T06:45:00Z

I found the docs and error output quite unclear, to be honest - perhaps because the first result referred to a Github token, and I wasn't using Github at all. Turned out (I think) the Github token is required for reading package changelogs ... OK then.

So:

GITHUB_COM_TOKEN=<token from github>
RENOVATE_TOKEN=<token for your project>

The Gitlab token can be a project token with API access, rather than a user token, so I'd recommend that; then the MRs appear to come from the bot rather than your user.

The tool acts on the remote repository, not your local codebase. I spent a while trying renovate . and wondering why it didn't see the renovate config ... cos that config wasn't in the repo.

Which Drupal config split is currently active

2020-09-25T00:00:00Z

This information is also shown at admin/config/development/configuration/config-split. I have to figure this one out periodically for CLI debugging, so here's a note.

$environments = [
  'development',
  'ci',
  'staging',
  'uat',
  'production',
];
foreach ($environments as $environment) {
  if (\Drupal::config("config_split.config_split.$environment")->get('status')) {
    print "$environment";
  }
}

Patching modules from D8 to D9

2020-09-25T00:00:00Z

Currently there's an interesting obstacle to working with Drupal 9: if a D8 module doesn't yet have D9 support, and a patch is sitting in the issue queue, cweagans/composer-patches can't handle the scenario ... because the composer dependency tree prevents the module version being installed installed. What can you do?

Contribute! Step up as a co-maintainer. But if the module is already slow to merge D9 patches, you might find it's not maintained.
Fork the module to modules/custom ... please don't do this.
Work around it in composer.json

The last option looks like this:

Add a package entry specifically for this Drupal module to your composer.json:

{
    "type": "package",
    "package": {
        "name": "drupal/cache_control_override",
        "type": "drupal-module",
        "version": "1.0.0",
        "source": {
            "type": "git",
            "url": "https://git.drupalcode.org/project/cache_control_override.git",
            "reference": "8db91684a427366d8f9c51f60cbac10c2d586d95"
        }
    }
},

We've used a Git SHA so Composer won't bother with version specifics, and the presence of the package entry means we won't look this up in packages.drupal.org

Gitlab's Directed Acyclic Graph functionality

2020-07-14T00:00:00Z

Gitlab has a DAG (Directed Acyclic Graph) feature for managing dependencies between jobs, which improves on the previous behaviour of stage precedence. The old behaviour is that jobs are grouped in stages, and jobs from a later stage only start if the previous stages complete successfully. The new behaviour is that with a depends: property on a job, you can name other jobs which much be completed, and that job can commence as soon as its dependencies have been met.

By way of example, I tested this on a project which had three stages: lint, build, test.

Stage lint
- Job grumphp would run composer install then grumphp, run time ~ 60".
Stage build
- Job drush make would build the site codebase, run time ~ 1'45".
Stage test
- Job behat would install the site (using MySQL service) then run Behat, run time ~ 1'11".

This was totalling to about 3'30" on average (excluding the Behat test time).

In the above, behat depends on drush make, composer install and the DB being loaded. Because Gitlab CI services currently can't be shared between jobs, the DB load has to happen in the same job as Behat. composer install can be decoupled from grumphp, but this appears to offer little time saving: initialising an additional job adds some overhead, so you might think there's not much to be gained from splitting the two. With stages only that's true - but in this case it also permits us to run composer install and drush make simultaneously. And that's where the win is!

Reorganised:

Stage build
- Job composer install prepares for running behat and grumphp, run time ~ 1'.
- Job drush make, run time ~ 1'15".
Stage lint
- Job grumphp checks the codebase, run time ~ 45".
Stage test
- Job behat loads the DB and runs tests, run time ~ 1'20".

The real advantage in the above, which totals to a longer run time (and more charges against your Gitlab allocated CI minutes, if that's a consideration), is that we can start composer install and drush make immediately, and then start grumphp and behat as soon as the first two jobs are completed. Totalling the change, the behat task can be taken as a constant since it doesn't shift, and what we can vary is how fast behat starts. In the first scenario, completing grumphp then drush make took ~ 2'30" - 2'45". In the second scenario, that's down to 2'.

Samples

job name	run times observed	notes
grumphp + composer	45", 46", 46", 52"	includes composer install
composer install	50", 57", 58", 58"
grumphp	32", 36", 38", 39"
drush make	1'05", 1'16", 1'16", 1'16", 1'17", 1'19", 1'20", 1'27"
behat	(not significant here)

This is a really simple example of the advantages of the DAG model. In this case it's somewhat constructed; rearranging with the grumphp into the test stage could have the same time saving. But for more complex CI layouts where you might have developers waiting or attending to the outcome of results, this is a real win in terms of keeping focus up and development progressing. Attention switching is a cost and long CI run times can be costly for that.

I can't explain why the measured run times of a task which included composer install and grumphp went up when the task was reduced to just composer install. Maybe with more samples this would iron itself out ... I'm intrigued. Also keen to measure a few CI runs of a job which does just echo hello world so I get a feel for the overhead of any single job added to the CI process - this obviously depends on the image used, and artifacts each end of the job too.

The DAG feature isn't (yet, as of July 2020) visible / available on Catalyst's Gitlab instances; the CE docs indicate that it's something to manually enable while in beta. I'm looking forward to being able to use it!

Details & links:

Behat, Drupal and related tools

2020-07-07T00:00:00Z

This is really a brain dump for a developer joining one of many projects I'm using Behat, Drupal & some related tools on at the moment. But ... if I write it up generally instead of as a stream in some chat program, it won't be lost quite as quickly. Or I can link to it next time.

First of all: Why Behat? For me, after a few times clicking through the same webform, I start to feel a little ... impaired. Seriously. I cannot manually test effectively; my brain is gone elsewhere. But that's fine, because repetitive manual testing is not something we need humans to be doing, and we can deliver it repeatably and reliably for Drupal using Behat.

Behat abstracts the steps of testing into Feature and Scenario definitions written in Cucumber syntax. I recommend absorbing the way that Cucumber / Gherkin is written 1, 2,

Scenario: Wilson posts to his own blog
  Given I am logged in as Wilson
  When I try to post to "Expensive Therapy"
  Then I should see "Your article was published."

The Steps in the Scenario above (and the associated components like Feature, Background etc) describe in plain language how something is used.

Developing with Behat

What's ideally going to happen in your development cycle looks like this:

You're asked to implement a feature
You understand enough about the requirements to write a meaningful Feature / Scenario
This won't pass at first, and that's OK
You'll then modify the application / website to implement the requirements
The test will begin to pass, and you'll improve the test as the implementation develops
You'll submit an MR with the test and the code changes
The test will be proven to pass in CI before it is merged
You'll have confidence that the functionality continues to work in future

Understanding Behat structure

<project>
+ composer.json
  + tests
    + behat
      - behat.yml
      - features/bootstrap/FeatureContext.php
      - features/*/foo.feature

composer.json - In composer.json we reference composer libraries for Behat, and a set of other supporting components. Some of these might be:

behat/behat - the main Behat library
behat/mink - mink drives various browser implementations such as Goutte, Chromedriver, Selenium.
behat/mink-goutte-driver - Goutte is a PHP-only "browser" which implements the HTTP spec and can do basics, but lacks JS support or a rendering display.
bex/behat-screenshot - A Behat extension which watches for failed Steps and takes screenshots on failure
dmore/behat-chrome-extension - Allows Behat to drive Chromium via Chromedriver
rpkamp/mailhog-behat-extension - Connects Behat to Mailhog, so we can capture and read emails sent during testing
drupal/drupal-extension - integration layer between Behat, Mink and Drupal. Provides many common Drupal-specific Steps (eg Given I create an Article node with title "Foo")
drupal/drupal-driver - Drupal Driver for Behat to talk to a Drupal site. Contains Drupal, Drush and Blackbox connectors.

Install Behat libraries using composer require --dev so they don't get sent to production.

behat.yml - This is the configuration file for Behat. It has configuration sections for Behat packages, and supports multiple profiles so you can (for example) switch profiles for different environments. It can sprawl a bit, but mostly you won't need to change it.

features/bootstrap/FeatureContext.php - each project has a FeatureContext like this, although you can add your own custom contexts either in the Behat directory or as extensions. For example, if you had a set of sites to test together you could create an BauersiteContext class which encapsulated functionality specific to that suite of sites.

features/*/foo.feature - this is where the Features live. Organise as you wish (if they don't auto-load, check the top of your behat.yml). Tag your features and use tags to run specific features: behat --tags=@payment. I often tag tests with the associated WR during development, so I can behat --tags=@wr277853.

Writing good Scenarios

This is advice to myself too and in some of the linked examples I won't have done perfectly. It's OK to do what works! But this is a list of what's good too.
Write idempotent tests. Don't hardcode in Given I log in as user test@example.com with password "fruity" because your tests will break when the content (user account) they depend on
Try to avoid When I click on the element with ID "#user-1-send-message", instead writing When I click on "Send message" for user 1 and creating a StepDefinition to match. Your Scenarios should ideally not incorporate low-level details such as DOM IDs because these things may change in the application / content and you don't want that to break your Scenario. Ideally your StepDefinition should parse the document rather than just packaging up the lower-level details, but even the latter is better than putting hard to comprehend language into your Scenarios.

Good blog posts / docs / videos

Behat's Writing Scenarios is worth your time.
Making your life easier with Behat + Mink + Drupal Extension (youtube)
Automated testing in Drupal with Behat (youtube)
Mink at a glance to understand Mink's place in things
Drupal 8 and Behat tests - introductory, but a decent walkthrough / intro
Docs for the Drupal Extension

To do

Things I could expand on more here.

How to use a clean install to run your tests (no requirement for a DB dump) - ideal scenario
How to use DB dumps and use those (no examples yet)
- How to sanitise / reduce DBs in preparation for CI runs
- Kirill recommends to not do this - I think for existing sites this is a pragmatic path towards testing; otherwise the leap can be too big for projects to justify
Behat in CI (probably another post), incl
- Switching configuration for CI (eg showing backtraces, capturing error logs)
- Long install times / short job runs

Drupal configuration management and sensitive credentials

2020-07-04T00:00:00Z

Working out loud?

2020-02-28T00:00:00Z

I was introduced to the book Working Out Loud at an open source conference in Denver 2016, when Matt Thompson keynoted. The ideas behind Working Out Loud seemed so familiar and he expressed them so well.

Working Out Loud's blurb makes it sound like it's a career development book but to me it's more about us all being better together. If you're reading this (hosted as it is internally to Catalyst), I can assume you're familiar with open source.

So - what's Working Loud, this blog, for me? It's a process of encouraging myself to share my learning and my work by trying to be open by default. If I'm discovering how to implement something, I'll try to share my journey somehow. This might be a write-up of a technique, or some thoughts about how I might do better in future.

Working Open (as a philosophy, as opposed to this blog) is the idea that we learn more when we share more, and that there's rarely a good reason to make things hard to discover. I learned so much from open source and am grateful for it; it seems odd to not share what I've freely been given by this community of people working together. (Sure, I've had to work to learn too, but that's not a reason to not pass the learning on.)

Working Open has side effects, such as being more able to pick up from where I left off in six months (treating my future self as a team member), and reducing the effort to share process when I find I need to. It's refreshing and rewarding to answer a question with "oh, I wrote that up, here you go" and give someone all the details they need; it feels like a win.

Working Open also reduces siloing and improves bus factoring (I forget which direction the bus factor is supposed to go, but if I've kept notes, I've moved it the right way).

That's what this is all about.

Breaking YouTube & friends

2020-02-26T00:00:00Z

We have a couple of young kids (currently 10 and 12) and "devices" in the home. We have a pretty good arrangement but one thing that's hard for them - one in particular - to resist is YouTube in the mornings.

We also have a PiHole device on the network, and it's quite a bit more flexible (being "plain old unix", as far as I care) for scripting behaviours.

I added a couple of cronjobs to prevent access to certain sites during certain hours. This has resulted in the occasional "Dad, SITENAME doesn't work" but I'm firm that Dad doesn't do technical support before 9am (not domestic tech support anyway).

Here's the recipe - nothing special, but it works nicely enough for my needs.

/etc/cron.d/blocked-sites

0 6 * * * pi bash /usr/local/bin/pihole --regex '(^|\.)(googlevideo|snapchat|tiktok|youtube)\.com$'
0 9 * * * pi bash /usr/local/bin/pihole --regex --delmode '(^|\.)(googlevideo|snapchat|tiktok|youtube)\.com$'

Previously this used a file /etc/pihole/blocked-sites but that got fiddly and required wrapper scripts to work from cron ... this Just Works.

I haven't found a commenting solution for this static-hosted blog yet, you can use chrisburgess/working-load issue queue for comments if you have a suggestion.

Hugo custom shortcode for WR links

2020-02-26T00:00:00Z

false adds a shortcode implementation for Hugo.

This commit adds a shortcode implementation for commits in this repo 😁

You can use it like: false.

See layouts/shortcodes for this.

Hugo, Gitlab, let's go!

2020-02-24T00:00:00Z

I caught Ben Bradshaw's Pizza Thursday talk on How to turn Gitlab into a '90s web host and decided I should get around to doing this.

Here's how I set this up:

Cloned the example Hugo Gitlab Pages repo to a local folder
Blew away and re-inited the git repo in that folder
Created a new personal repo on the internal Gitlab
Messed around with this repo until I was happy
Added the personal repo as the remote origin
Pushed my code and watched the CI task build
Via the repo sidebar, I found the Pages URL
Updated config.toml to reflect the base URL

I'm leaving Fuzion today!

2017-06-06T07:45:00Z

Today is my last day working for Fuzion! It's been a great experience. For the last four years I've enjoyed being part of the Fuzion team, supporting non-profit and charitable orgs (and the odd business!), working to support their efforts in online campaigns. I've also had a fantastic opportunity to work with the CiviCRM community worldwide, and it's been a privilege to be part of this.

I joined Fuzion in 2013, but my connection to Pete runs back to a decade ago, and hopefully far into the future too. He's been a hugely supportive employer and really has his priorities screwed on right. Thanks Pete :D

I've enjoyed the Drupal & CiviCRM parts of our work, but more importantly the community aspects of what we achieve. Not just "today I helped an org I believe in", but more that through our contributions to the CiviCRM project, we're able to help today's organisation, and many around the world, and some in future we don't even anticipate today. The most important part of that is not the code we contribute, but the community which builds that code.

I'm so grateful for the support to learn and contribute that Fuzion has offered me in my time here, and for all the many lessons I've learned along the way.

I don't know yet whether my next gig will include any CiviCRM requirements, but I do plan to stay involved. Civi folks - should you ever find yourself near the bottom of the globe in Dunedin, get in touch! We have some excellent breweries I'd like to show you ...

Hubot SSLLabs

2017-06-06T07:45:00Z

Tonight I spent a few minutes on the couch rolling a hubot-ssllabs plugin.

It uses Qualys SSL Labs API to check your site's SSL config and report back in your project chat.

Enjoy :)

[ lost image ]

CiviCon 2017 recap

2017-05-21T21:32:00Z

Oops.

John Key is watching ...

2017-05-21T21:32:00Z

It seems I made this at some point, with the help of my 8yr / 6yr assistants. That's the (then) Prime Minister of NZ enjoying your viewing selections.

(I think between when I made it and now, browsers started supporting clip-path properly?)

Mobile view

Blackbird nest & rPi cam

2015-12-04T17:45:00Z

Outside our bedroom window and beside the front door, a blackbird has nested in the rose bushes. After watching her for a few days at breakfast and saying hello out the bedroom window, we added a nest cam for a better viewpoint.

nest cam on YouTube

I was cautious about installing the camera, as I thought there were no eggs and had read that birds are more likely to abandon their nesting if they aren't invested fully. Turned out there were four eggs but the nest was quite deep and they weren't visible when holding a camera out the window to get a view.

If making any changes near the nest, I'll watch her from the breakfast table, and get to work with any changes when she breakfasts soon after dawn. She knows we're here as she can see us in the bedroom window, so I think she's decided we're mostly harmless, if noisy.

There was a bit of crawling around in the roof to get a Raspberry Pi within range of the nest, and some small holes drilled - yay for being a homeowner, I can drill where I like! Have to go back up there and wire the Pi power off the porch light so I can remove the extension cable snaking into the laundry ceiling.

The rPi is running Raspbian Jessie, and the camera is a Microsoft HD-3000. From the feel of the base I thought I'd be able to drill mounting holes into the camera, but it turns out the base was steel and broke a drill bit. I make a small wooden mount that the base slots into, so I can affix that to the underside of the eave.

Initially I used Motion to capture the stream, which worked pretty much out of the box. For security & bandwidth reasons I decided to push the stream out to a CDN for viewing, so after a day of Motion I swapped to avconv last night, and am now pushing it to YouTube (hence the embed above).

Since night is too dark to see activity, I'm deactivating the stream from dark (10pm) until dawn (5:30am). Running the avconv stream inside a tmux session to observe output - expected to have to do more to keep it running TBH, since video encoding, but so far it's fine.

There's not really a lot of "how to" to this, since it worked out of the box after installing libav-tools. Still I can share the successful results of some trial and error :)

Cron entries to start & stop a named tmux session for the stream.

Shell script to run the actual encode.

YouTube seems to ignore streams without audio, so this adds silent audio channel.
Submits FLV - YouTube grumbles about "unsupported codec" occasionally in the video manager, but -c:v libx264 doesn't encode efficiently on the Pi. Need to play with this.

Pushing off to CDN and back for viewing introduces a short delay, so if I need to adjust the camera I swap back to Motion for a few minutes - it's much faster for local only delivery. It's probably possible to run the two side by side.

Quite impressed with what the rPi and cheap camera can do here, and happy that YouTube appear to be offering unlimited free rtmp:// broadcast. I did check out Twitch.tv and Ustream, but found it easier to navigate YouTube's live broadcast setup than the others.

Letter to Santa

2015-12-01T18:45:00Z

Just a quick post 'cos I'm doing #nzsecretsanta this year and have been wishing my secretsanta recipient posted more about what they want. So here's me paying it forward by saying what I like/where I'm at!

Last year I got some Sharpie sunglasses and a set of a dozen Sharpies - I really enjoyed the strong bright tones of the pens, which went well with my contour drawing style and liking for strong blocks of colour.

In 2016 I will be -

Crafting with wood (rediscovering this)
Sketching
Playing with technology lots
Building (or finding) a team environment that supports ongoing learning
Studying Mandarin (with Rowan, 6)
Practicing Tae Kwon Do (with Hunter, 8)
Riding bikes and playing outdoors with family
Playing futsal / indoor netball / anything with balls
Watching blackbirds hatch outside our window

We're going up to Auckland in a few weeks, looking forward to a summer in the north.

Right now I'm building the 2016 election website for the Australian Green Party, enjoying working with the various tools in play. They have a great distributed team and a lively mix of skills on deck.

I may update this, it feels like a good "forward reflection" :)

KoalaSafe first impressions

2015-11-13T22:36:00Z

A few months back I purchased a KoalaSafe off Kickstarter - at the time I didn't feel a pressing need for it, but the project hit a chord with me and I wanted to support it.

After it arrived I reflected on how we use devices as a family, and decided not to plug it in. Cue forward to a couple of months later (last night) and Saira and I were talking about Minecraft, playing versus watching videos, and how to positively engage kids towards active engagement.

I proposed that instead of prescriptive, negative approach ("get off the device", "you should be playing not watching"), we should be embracing our kids interests and engaging positively ("show me how to build a snow golem", "tell me what's happening here").

Saira asked me to keep an eye on usage over the weekends (peak times - I'd estimate the boys have about 3H/day screen time on weekends, and about 1H weekdays), and for us to review it after a bit of play. That reminded me that the KoalaSafe recorded usage stats (this device spent this much time on youtube) and that it was pretty much for this purpose.

Also, parents use devices too, and seeing that usage might help us engage our own habits in ways that set good examples for our kids.

So I found the KoalaSafe (which I'd taken to work and stashed), and plugged it in this morning. It's going OK.

Internet seems to be a bit flaky on some devices.
Wish I could budget an amount to be consumed.
Not sure how to control from laptop!
Can I hack on the platform?
Can I change the SSID?
Shared device - Parent mode

https://www.youtube.com/watch?v=HUgv5MDF0cQ

Mandarin lessons

2015-11-13T22:16:00Z

Hunter got a do-bok for Tae Kwon Do and I knew Rowan was impressed with it, so in Australia I picked up a kung-fu suit for Rowan at the Sydney markets.

About a week later he announced at dinner that he'd like to learn Chinese, please. Saira taught him to count to ten, say hello and thankyou, and that was as best we could do with not knowing much Chinese to speak.

Rowan persisted in asking, and via the NZ China Friendly Society's free weekly adult introductory classes (contact details) I found out that there is a Chinese school on Sundays in Dunedin, where you can learn Chinese art, Mandarin language, and Chinese dance.

We tried a language lesson, and he was keen but the teacher indicated it would be hard for a student with no background to support him to catch up with a class of students who have Mandarin at home. Saira and I talked about this and decided to see if the art classes held his interest and gave him a foothold, so we did a couple of weeks of that.

He thought that was ok, but after a couple of weeks announced from the shower the night before a Sunday session that he'd be going back to the language class now. I'm impressed by this determination.

So - I've doubled down on learning Mandarin myself (enjoying the HelloChinese app on mobile), Rowan is practicing hard and his teacher is delighted to see that he's as keen as he is.

I didn't expect to end up learning Mandarin myself, but after studying Te Reo Maori last year I'm finding I really enjoy the challenge and the experience. Even though I'm not officially a member of the class - just sitting in to help Rowan get his feet on the ground for now.

(There are also free adult classes in Dunedin, I haven't checked them out yet.)

Dunedin, let's fly the flags

2015-09-28T06:38:00Z

A year or two back, walking Dunedin and enjoying the details of our buildings, I started tuning into the the abandoned flagpoles that spike the skyline of our city. With our great stock of older architecture, we have a lot of flagpoles - and most of them are bare today.

My proposal to Dunedin City Council and the property owners of Dunedin is that we run flags up all those poles.

Right now, New Zealand is discussing the flag we'll fly in the future, and Dunedin has an opportunity to highlight our city's architecture and at the same time invite our citizens to engage more in discussions about the future of New Zealand's flag.

Most New Zealanders have seen static representations of the various flags under discussion, in print or online. But flags aren't static things, they twist and fold and hang and stand in the breeze, and they are seen against the sky - not on paper or screen.

Imagine if, a few weeks from the start of the referendum, every one of those flagpoles across our city had one of the six flags above flying from it. For the next several weeks, flags could swap places every few nights, giving people across the city the chance to see each of the new flags flying. Many would also notice their environment and surrounds anew.

During the second referendum, we'd repeat the process with the current and proposed flags.

People here would notice aspects of the city anew as their eyes were drawn to the sky. We'd build appreciation of the urban environment, which I think would benefit property owners. And over the summer tourism period it would be a stand-out feature of our city for visitors to notice.

By engaging people in the voting process in late 2015 (and early 2016), familiarity with postal voting would be boosted close to the local elections in late 2016. I think boosting postal vote turnout in the flag referendum could only benefit turnout in the local elections to follow.

What would it take to make this happen? I imagine getting the property owners of buildings across the city to see value in it, and a backer like the DCC supporting it by providing official backing, numerous flags and crew to keep the flags flying around the city. Probably some other ingredients too, but it seems within reach and worth trying.

Those bare poles have long had me toying with the idea of how an art project might repurpose the abandoned open spaces where the flags once flew through the city ... I'd love to see our city skies come to life like that.

Image credits: Bad photoshoppery by yours truly, the middle four via Flag Consideration Project, current NZ flag via Four Corners Classroom, Red Peak via Evening Report NZ. No subliminal placement, only for my convenience.

Online Voting Trial - submission to Dunedin City Council & outcome

2015-09-21T09:20:00Z

Until recently, a government working group and several local councils had been (fairly quietly) contemplating a live trial of online voting at the 2016 local body elections. Aside from a few articles in the ODT back in 2013, there had been little coverage of this.

Dunedin City Council's meeting was today, and over the weekend I had the opportunity to talk to Chris Morris of the ODT and share some concerns about what online voting might mean. I spoke in the public forum at the meeting, and was really happy with the engaged and informed questions that the councillors asked.

Sadly I don't have the questions asked in reply, but here's my submission to the council from today -

Mayor Cull, Councillors.

Thankyou for the opportunity to speak today. I want to applaud your motivations in contemplating online voting. I believe that you seek to improve representation by increasing access and engagement. I wish that online voting would address those issues, but I am here today because the internet is not a secure environment for democracy.

I'm an IT developer with clients in many countries, all of whom demand security for their data. Our work has been subjected to penetration testing and to active attacks on the internet, and I'm familiar with the issues involved in securing systems online from the different methods of interference that can be attempted.

I'm concerned about a move to online voting because I feel the significant risks involved are being underplayed, and because many of the benefits cited do not hold up to investigation.

Security researchers and experts generally agree that online voting has substantial risks of vulnerability. A secret ballot poses an especially hard problem for securing when there is no physical evidence of the actual votes placed. There are multiple opportunities for attackers, and little scope for remedy or recourse if the electoral process is abused.

A group of American researchers describe their investigation of a 2010 online voting trial as follows: "Within 48 hours of the system going live, we had gained near- complete control of the election server. We successfully changed every vote and revealed almost every secret ballot. Election officials did not detect our intrusion for nearly two business days—and might have remained unaware for far longer had we not deliberately left a prominent clue." (Halderman et al. "Attacking the Washington, D.C. Internet Voting System" Proc. 16th Conference on Financial Cryptography & Data Security, Feb. 2012)

Security expert Bruce Schneier summed up the state of his field in 2001. "Building a secure Internet-based voting system is a very hard problem, harder than all the other computer security problems we've attempted and failed at. I believe that the risks to democracy are too great to attempt it." (Bruce Schneier, Crypto-gram newsletter, 15 Feb 2001)

Online voting depends on the security not only of the systems running the voting process, but also the systems used to vote - and the majority of those systems, home and work PCs and mobile devices in the community are vulnerable to multiple software vulnerabilities.

Users of computers fall prey to online deception frequently; a year ago, large numbers of users being tricked into opening a malicious attachment triggered a nationwide outage for Spark, our largest internet provider. (NZ Herald: Spark users experience internet meltdown).

With online voting, individual users will be susceptible to a targetted attack, especially when a large number of internet users are attempting an unfamiliar task together. An attacker can falsify apparent voting environments and put those in front of voters, hijacking credentials and then re-using them to make the attackers vote selections on the live system.

Attackers would have multiple ways to interfere with online voting - preventing votes, falsifying votes, destroying vote records, or denying access to voting systems. All of these techniques could be used to affect the outcome of an election in various ways.

Securing and assuring a secret ballot in a digital environment is a particularly hard issue, and one which many nations and larger organisations than ours have attempted to address and walked away from. Australia experienced a significant security issue; in France faked votes were demonstrated; the Netherlands banned online voting outright as a result; Norway abandoned their attempt; Portugal, Spain and the UK all discontinued their efforts. Washington DC's trial - an actual trial, not a live test - was hacked within 48 hours. Most security experts agree that this is no easy task, and I see risks but little advantage for us in breaking new ground here.

Proponents of online voting claim it may increase engagement - but experience has not shown this is a result to expect. When surveyed about reasons for not voting in recent general elections, NZ voters primary reason was "didn't get around to it, not interested or forgot", and disengagement was the underlying cause for 40% of non-voters. (Stats NZ: Non-voters in 2008 and 2011 General Elections)

Engagement, not access, is the primary issue for voter turnout. If the council seeks to improve vote turnout, then that is where I believe the council should direct this funding. Again, I do applaud the DCC's interest in increasing engagement from citizens and in considering this approach.

$165,000 to be part of a test with actual council elections is a significant investment, but online voting will carry a much higher pricetag for full implementation.

My recommendation is that Dunedin not proceed with the proposed online voting pilot.

Thanks for your attention.

Beau Murrah and Stu Fleming also spoke against online voting, and representatives from the NZ Blind Foundation (I think Diane, and another man whose name I forgot) spoke in favour of it on the grounds of how it enables more full representation of all voters. Councillor Wilson identified, I think correctly, that "accessible voting" and "online voting" are separate decisions, but I do appreciate that having had a taste of accessible voting online, the appeal would be great. I hope we'll see solutions here which don't require the risks of OV.

The meeting decision was that Dunedin decided to not continue with the trial, which was the outcome I hoped for. Councillors discussed it in depth and I was glad after following the discussions in other cities to observe that many councillors understood the concerns that informed experts share about moving democratic process online. I was concerned to hear a few flag that cost rather than security was a primary motivator, but learned that in our democratic system, it comes down to how people vote - not why they vote!

There was some media coverage, but the best moment of my day was getting an unexpected text from my Dad, probably having heard my comments on the radio, saying that he couldn't agree with me more. That felt great!

I'm really happy to have taken this first step in engaging with local council - not something I'd aspired to, but it was inspiring and empowering to have an opportunity to be heard by the council members, and to see that they followed what I was saying. And I think that the outcome was a very positive thing for most Dunedin citizens.

Some related coverage -

ODT: Fears for online voting security - Chris Morris. After the last interview a few weeks ago, I managed my habit of double-negative kiwi-isms ("not such a stupid idea" => "a good idea"). Great!
National Radio NZ: Dunedin Council pulls out of online voting trial - Ian Telfer, on Jim Mora's Afternoons show. (Six jurisdictions abandoned the process, but only one banned it.)
Channel 9: DCC will not take part in nationwide online voting trial; bonus points for using "for the LOLz" in a TV interview!
ODT: Council says no to online voting trial - Craig Borley
DCC should put the council meeting on their YouTube eventually, should you want to watch the whole thing.

Archimedes - a tool for tracking Drupal Sites

2015-09-01T09:36:00Z

Visited the Drupal Wellington Meetup tonight and did a quick presentation of a couple of integrations we're using with Archimedes to monitor a network of Drupal sites. Here are my notes on the same, in case anyone wants to look further!

Setup

Running Archimedes involves setting up an Archimedes server (profile) and installing a copy of the Archimedes module (module) on each client site. (Run drush pm-update --security-only on the server at a minimum, since I've not been updating the modules in the profile. You might also have to add some Bootstrap libs?)

You'll then need to configure each site by copying the Public Key from the server to each client, and setting the Submission URL to point to the server's $base_url + /archimedes/update. These details are available from the server at admin/reports/archimedes/settings.

Updates

Updates will happen daily per default, or whenever you run drush arch-send on the client site. The server cron will process the incoming submissions via Drupal cron, so you may want to run the Archimedes's server's cron task frequently when there are updates in the queue - typically early morning.

Additional Integrations

I'll publish more about this shortly, but we use some additional Drupal Views to expose data over the web to -

Hubot
Dashing
Munin
Nagios

Here's Hubot reporting on which sites use a module, and what versions they have installed. This is helpful on Thursdays, when security updates drop.

[ lost image ]

Here's Dashing telling us the current state of play on a Thursday, after some core and contrib security updates have shipped. We can click on these numbers to see a (protected) View of which sites need attention.

[ lost image ]

I don't have demo images for these, but we also generate Nagios and Munin config snippets from data gathered in Archimedes, and that lets us monitor site outages, track site load times, and graph additional figures like DB size, user registrations and other indicators of site activity.

Resources

Archimedes Server, the server profile.
Archimedes, the client module.
Archimedes Extras, some additional checks you might want to explore :)
Dashing, an exceptionally handsome dashboarding toolkit.
Hubot, your personal robot assistant.
Munin and Nagios, tools to monitor your things.
Frosty Meadow, which I used to bulk-rename our client sites for the purpose of demonstration. I used its metal mode.

Credits

We did a bit of this work, but Catalyst did most of the work with the release of Archimedes v1, which Josh Waihi presented back in 2010 (?) at Drupal Down Under in Brisbane.

GovHack 2015 NZ Awards!

2015-08-17T06:32:00Z

Sooooo we went to the GovHackNZ awards, and came home with some awards. It was fun! Bit of a delayed update here, on account of a hangover and flights home yesterday.

GovHackNZ flew us (Vincent Rijlaarsdam, Beau Murrah and I) to Wellington for the NZ awards ceremony at Te Papa this last weekend.

We arrived and met some of the other teams & organisers - as mentioned previously we were the only team at the Dunedin event so we'd not met any other teams as part of the weekend.

Several of the projects that I liked most (School Stories, Project Lighthouse and GrocerFree) had been shortlisted, and this was especially good because I got to catch up with Robin from GrocerFree there and heard that Hyun-Jin and he were now seven weeks from the due date of their child. That moment was probably the absolute high point, and it was funny that Robin and I had been talking about many other things by email / IRC this year without touching on it.

Anyway, as you may see from the image above, our weekend's work earned second place for Best National Team and a National Spirit of GovHack award, as well as (without competition!) the Local Spirit of GovHack award for Dunedin.

We had a great night, enjoyed meeting the other participants, and felt really boosted by taking away lots of people commenting positively on the concept of the project. In particular Cherie gave some really kind comment when awarding the Spirit of GovHack award, observing - I'm going from memory here -

A lot of projects focussed on "home and heart", as it's clearly a high priority to ensure the best possible place to live, schools to send your children to and so forth.
This project is one that makes it possible for the average person to go further than those decisions - it enables people to not only understand the legislative process but have an opinion and a say in how the greater decisions are being made and how the country is run.
It is truly important that everyone has that opportunity, which makes this project a shining example of what the spirit of GovHack really is.

That was ... wow, exciting to hear. I had an awesome time, we were very happy to come away with some awards, and are excited about continuing this year's project & helping make GovHack something bigger next year.

And this evening I've got an email from the Parliamentary Counsel Office (who run legislation.govt.nz) asking about what we're doing next, so tomorrow I'll have to gather some thoughts for that conversation!

Getting TiddlyBot on the wireless

2015-08-01T03:56:00Z

TiddlyBot arrived in the mail a week or two back! The boys and I had a fun time assembling it (really simple kit, no breakages) although there was a few days pause when we tried to mount the Pi and realised we needed a v2 Pi not a v1 ... nicegear shipped it quickly and we picked up the next weekend. Once we had it assembled, the AP mode didn't work out of the box, so I'm writing a quick post on how we got it to work so far.

The TiddlyBot image v1.1 contains hostapd v0.8.x_rtw_r7475.20130812_beta while Debian currently has v1.0; the Pibot crew recommend the Edimax EW-7811Un and confirm it works with a few others. We had a miscellany of wifi devices for pi, all of which have worked out of the box on Raspbian. I didn't want to wait for another part to ship, so here's the process I'm using to bring up hostapd starting with TiddlyBot and a v1.1 image.

My wifi card is: Bus 001 Device 004: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter - this device requires the nl80211 driver in hostapd.

The version of hostapd in TiddlyBot v1.1 doesn't include RT5370 support, so I want to swap in the binary from Debian's current package of the same without changing Tiddly's startup configuration by installing the full Debian package.

My current process is to -

ethernet the device
find Tiddly's DHCP assigned ethernet IP from the router
ssh into Tiddly
start raspi-config and grow the filesystem (default install is close to 4GB)
reboot, ssh back in
aptitude upgrade all the packages
aptitude install libnl-route-3-200
apt-get download hostapd
dpkg -x the downloaded hostapd .deb to a temp dir
copy the deb's binaries over the shipped ones in /usr/local/bin
modify /etc/hostapd/hostapd.conf to use the correct driver for this USB wifi
sudo hostapd -dd /etc/hostapd/hostapd.conf to check it works

If the above process works for you and you can connect to Tiddly over wifi at this point, then the steps to turning the bot on and connecting are still a little longer than "just works".

ethernet the device, turn it on
ssh in as above
sudo service hostapd restart
connect to Tiddly wifi

I suspect at this point hostapd is starting on boot, but works on second / delayed start; I don't yet know why. Still fiddly but will do for tonight - I have a working TiddlyBot today instead of when a new USB device would have arrived.

Might not hurt to have hostapd logging to a specific file, that way we have a bit more to look at when things don't work. I've been running it with -dd in a tmux session but not out of the init scripts.

I'd recommend asking questions on the TiddlyBot forum rather than posting questions as comments here :)

GovHack 2015: We're Going to Wellington!

2015-07-31T12:51:00Z

My phone rang today - I was home in bed with a cracker of a headache and wearing inflight sleep goggles to shield my brain from the terrible sunlight. An unfamiliar number, Cherie from GovHack calling to tell us that we'd been shortlisted for a national prize as well as winning the regional Spirit of GovHack award.

(If we picked up the regional award by entering a project at all because there were no other teams in the South Island, we may owe that to @hackdunedin for organising the event here - thanks again Phil and Josh! On the other hand, this kicked in the impostor syndrome klaxon: what if turning up was all we really achieved? But I don't really believe that.)

Anyway - I'd already booked my tickets figuring that participating in the awards is part of the package, but now all three of us - myself, Vin and Beau - will be able to get to the awards ceremony.

My motivation to enter was to get to do cool things, meet some other local hackers and work on a team. Being invited to be there for the awards as well is super exciting.

CiviCRM Contribute Form JS: Don't Rely on Priceset IDs.

2015-07-28T22:32:00Z

Since a lot of CiviCRM work is around supporter donations / memberships, at Fuzion we end up doing quite a lot of customisation. It's tempting when writing stylesheets and javascript to depend on the IDs and values provided in the page, but with CiviCRM this can lead to trouble down the track.

CiviCRM manages price sets internally as a list of values in a DB table, and the assigned IDs for each price option may change if the priceset is edited. This can mean that a client change on a live site breaks a customisation that was previously working, for no reason other than them changing a different price on the same form.

Where possible, try to check something which is not one of CiviCRM's autogenerated IDs relating to the priceset or the priceset value (=id of that price option). The above change is hardly any better - if someone changes "3 years" to "3 yrs" it'll still break, but at least it's more tied to the actual criteria we're checking than an ephemeral id from the database.

NZ PHP Conf 2015 Speaker Lineup

2015-07-16T08:28:00Z

In a few weeks NZ will have it's second (I think?) PHP conference. The topics look good and it's one of a few opportunities for NZ PHP developers to get together and learn. I saw a post on a mailing list promoting it today, and thought this would be a good opportunity to get together with other NZ developers and share some knowledge. (I know a few local developers through Codecraft, but it doesn't feel to me like there's a community of PHP devs in Dunedin.)

So - I headed over to the site, and checked out the speakers page to see what was on the cards, and I found myself thinking twice about going. I opted not to jump on the early bird pricing after all (I'm not saying I won't still go).

I reflected a while before posting this, because I'm not confident that it's for me to speak on diversity at conferences when I'm white & male, and I'm not confident it's my job to look at a set of headshots and count gender or race. I'm genuinely unsure about this tension; if I'm silent then I may be tacitly enabling the status quo; if I speak out am I "white knighting" or denying someone who is more directly affected by lack of diversity the chance to speak?

I am OK with sharing my feels (even if/when I'm wrong), and I'm a human goddamnit, and when I look at the speakers page like that, I boggle at how un-representative of humanity a group of 22 people could possibly be after someone flys them to NZ from around the world.

I guess this makes me really appreciate the great work that conferences I've attended in the last twelve months (Kiwicon, NetHui, OS//OS, DrupalSouth, DevMob) have done towards keeping diversity in mind when they are organised, and I'm intrigued to see how Kiwi PyCon 2015 (same weekend!) will do on the same front. Maybe I'll diversify myself to a Python conf instead.

GovHack 2015: Legolas

2015-07-06T22:32:00Z

(see also: GovHack 2015: My writeup)

Law is commonly represented as a rigid "thing" that happens to us, and changes reduced to simple catchphrases that don't mean the same things that laws say. We become polarised and we are disengaged from taking part in how laws are built.

Making law a conversation which all citizens can engage in might just enable the kinds of behaviour that open source has demonstrated; open source empowers users to contribute to fix bugs, and open laws can enable citizens to build a better society.

Laws are a code, and our GovHack project for 2015 seeks to bring citizens onboard as stakeholders.

How it works (the concept)

Legislation is imported into an environment which supports free and nuanced discussion. This platform aims to provide -

clarity and visibility to inform citizens,
a discussion model to permit informed public debate,
SCM-style presentation of "fork and submit" to clarify proposed changes to laws,
side-by-side "diff" views of amendments and legislative changes over time,
submission tools to permit community organisations to canvas submissions, and
support for additional data layers to allow all stakeholders to overlay additional information related to legislation.

From there, we support various uses of the platform according to different needs -

Susan wants to understand her legal responsibilities as an employer. Using Legolas search and comment tools, she can use identify using terms familiar to her ("holiday", "sick leave") and find both specific legislation and citizens experiences on relevant topics.
Bruce and Alice are discussing a technical point of law on social media. They can refer to specific aspects of various laws, linking directly to specific clauses of existing legislation as references in their discussion. Legislative changes over time are exposed using a Github-like "diff" interface which clarifies the dates and amendments made to laws.
Dave feels that local laws on keeping chickens are unjust. He can use Legolas to inform himself of the specifics of poultry law in his region, and to discover that there is already a lively community active around driving change in this area.
Members of a local RSA wish to submit to a bill on pension amendments. The platform lets them gather for a meeting and read through the bill as a group, formulating responses to each section. When their input is gathered they can submit their group feedback in a clear format, contextualised alongside the proposed changes.
Rebecca is an MP who wants to better understand how citizens feel about an amendment before the house. She can review comments on the existing law over the last few years, and identify contentious and troublesome aspects which need to be addressed in the upcoming debate.
A lobby group wants to provide a library of legal advice to members. The platform provides methods for additional data layers to be displayed in context with legislation, informing citizens of their responsibilities and rights under the law in ways which can be contextualised to their current needs.
Bob is a practicing lawyer who refers frequently to criminal law. On his mobile device he has an app which exposes Legolas with additional overlays for both case law references retrieved from a popular commercial case law service and his company's internal team practice notes.
Samwise is a citizen who aims to bring change to existing legislation on small business. As well as discussing the law on the platform, he can submit a special form of comment which proposes an amendment to legislation; this is visible to others and can be discussed separately without having any special legal significance.

Legolas is a proposal to support legislative documentation with tools for online engagement of the community - discussion relating to specific legislative "elements", history as legislation evolves over time, search and crowd-sourced supplementary documentation, and whatever other APIs, extensions, hooks and elements may prove useful over time.

How the prototype works (under the hood)

a nodejs importer which retrieves legislation and does various things with it (commit XML & PDF to Git repo, push XML to storage on Drupal site)
an XSL stylesheet which renders legislation to output formats
a Drupal site (source) which as of now supports search, display and commenting

GovHack 2015: my writeup

2015-07-04T13:02:00Z

GovHack 2015 was a fun weekend.

We gathered on Friday night at the Sargood Centre in Logan Park - great venue, a large open space for working, and a room full of desks on wheels arranged exactly as you'd expect to find them, especially if you factor in that the students are physed students.

When I arrived there were two humans (@philwheeler and @vinlew) and a lot of chocolate biscuits.

I decided to jump onboard GovHack for multiple reasons. Firstly, as a hacker with a focus in social / political sphere, it's right up my alley. Secondly, my work environment has felt pretty isolated of late, and I wanted to do something as part of a team. Thirdly, I wanted a chance to explore some technology that's outside my regular work tasks.

So I signed up pretty much immediately on hearing that it was happening in Dunedin, but I wasn't sure I'd be able to do it until close to the last minute - it all fell together because Saira took leave from work expecting family to visit, and then they couldn't come at the last minute and I was able to commit my time to the weekend. Fortunate for me!

Friday night, and there were three of us - Phil the organiser, and Vin and I the participants. Vin and I decided we should team up. Great! A team, first objective nailed. We were in a large nautically themed room, and decided to retire to the room full of desks and whiteboards to plan goals. We sketched out a handful of concepts that might work.

Another participant arrived, Ross; he explained he was just checking it out, and probably wouldn't commit to the full weekend. (I don't think he came back.)

We bounced the ideas around. I can't remember what the other ideas were, but one of them I put forward was something that had been in my head for a few years, the idea that our legislation should be more open, more of a discussion, more forum and less rulebook. This was at the forefront of my mind after OS//OS in Wellington a few weeks before, where amongst many other inspirational exchanges I saw Ben Balter talk about open government and where it's going in the U.S.A.

Huh, I'm rambling and I haven't even touched on the idea of the project yet. Maybe I should split that out into another post. This is what happened, and it's late and I'm typing like crazy. (see also: GovHack 2015: My writeup)

We bounced ideas around. One seemed to stand out, but I was a bit precious about it since it was an idea that I'd been keeping quite close to my heart for a long time, and I wasn't sure it was the right thing to burn on a weekend project. Also I'd come in hoping to contribute to someone else's idea! But ... well, we decided to go for it.

(I don't think we kept a record of the other ideas.)

So the basic concept was: A platform for making legislation a conversation.

What's that mean? Well, I feel like citizens could engage more in the laws that we are bound by. To me, it seems the reason that "bad" laws stay on the rulebooks so long is that they are not obvious to us. If law was a thing you could check out, look at and discuss, would you feel differently about the laws that affect you? Would laws be written in language that were more accessible to citizens? Would laws change to reflect attitudes more quickly?

So, while we officially have input into our legislation, the real process is harder than it needs to be for citizens.

Damnit, I was going to write a separate post on that part.

So. We picked a goal. We started doing some research and fleshing out concepts. Phil Wheeler gave us some great guidance on how to achieve results over the course of the weekend - identifying the whole set of tasks, which would include making a video and other requirements of entry before the close of play Sunday evening.

I can't remember what time we left the Sargood Centre, but I went home and stayed up late, pulling apart some prior works, exploring the government dataset (NZ legislation) we were working with, and knocking together a crawler that could haul it into some platform to host discussion.

Saturday morning came around, and I was up early and hacking, then took an hour to walk in to Logan Park on the slightly icy roads. Walking helps me think, especially when I haven't had a full complement of sleep. I reached out to Beau and got him onboard; three is better than two, and Beau's legal training and perspective brought a lot of useful understanding to what Vin and I had achieved.

Saturday and Sunday were a whirl. I remember

turning up at the venue Sunday before the organisers had it open, tapping away on my laptop.
reluctantly deciding to use Drupal as a content store - turned out it was a fine choice for what we needed it to do, and thanks to existing community contributions (thanks Dan Morrison) came with the tools to handle the legislation XML and output it nicely. (Writing your own XSL is another story, though, and an ongoing one.)
deciding in the last few hours that it was OK to make something with just demo in mind.
Clare Curran visiting us and saying heartwarmingly encouraging things. Thanks Clare!
enjoying hacking in NodeJS a lot.
finding XSL a headfuck.
regretting the choice of project name pretty much from the get-go.

And I learned some interesting things too.

I enjoyed working with NodeJS; I still really like Javascript.
Drupal Services is pretty cool for remote actions; I'll be hanging on to this tool.
Drupal Services API npm package does a great job.
With legislation whizzing past my eyes, I got to ponder about things like our treatment of undischarged bankrupts.
I occasionally sat back to keep up via Twitter as other teams nationwide experienced the full gamut of GovHack feels.

And then it was Sunday evening, and we'd turned in a video and made some commits in a repo, and imported a bunch of legislation and served it up, and thrown a site up around that to "sell" the idea, and made friends, and eaten all the chocolate biscuits, and we had to down tools.

Today it's the school holidays - I probably won't have a chance to revisit this for a while, but it was great fun & you should get in on it next year.

Here's a Storify of our weekend, probably missing bits.

Thanks @hackdunedin for organising it, and Phil Wheeler & Josh Jenkins for all the time you put into helping make it happen for us.

South Dunedin Flooding

2015-06-13T00:00:00Z

There was a lot of rain, and it flooded. Parts of Dunedin were not built for this - reclaimed swampland wants to be swamp once more.

I got to see it first hand while sandbagging flooded houses, and it made me angry to see how much housing had been built with disregard for the environment it was being put in - concrete floor pensioner flats at ground level, with not even a door jamb to stop the water.

My favourite moment of the night was carrying sandbags past an elderly couple, lights on and watching the show happen around them, on the other side of a ranch slider door. They seemed to be having fun, and their recliner chairs were positioned to watch through their glass window, and I hoped they were warm and appreciating a shared moment as I was carrying bags of sand to the houses behind them.

(You might see a video below this line? If I haven't stuffed up the embed, and you aren't blocking content from that site ... or it's here but it's not all that fancy anyway.)

Archimedes - Drupal platform for managing Drupal platforms

2015-06-13T00:00:00Z

When you have Drupal sites you have Drupal updates, and if you have update notification emails turned on then you'll have lots of notices in your INBOX too.

At first I thought more informative update notices would help, but it can be a weekly deluge. We used Droptor for a while too, but it was hosted so it only did what was in the box.

Archimedes allows you to centralise this deluge of notices / "critical business intelligence" into a single website. It allows Drupal sites to report in to a central (Drupal) service which collates info about each site - update notices but also other site data (number of nodes, users; DB, file sizes; etc).

The site data comes in as nodes for each website and records upgrade status for each module / theme so you can view your sites in cross-section instead of digging through emails -

which production sites are reporting views at version less than 7.x-3.11?

Once this data is collected together you can query it much more easily, and make pretty dashboards or chatbot commands to look up useful facts about how bad thursday just got.

Archimedes was first presented at Drupal Down Under 2011 by Josh Waihi, who developed it as an internal tool for Catalyst NZ. I started using it that year and

Atom Remote Edit UI & speed of opening remote files

2015-04-24T09:28:00Z

I quite like Atom as an editor - it ticks most of the boxes for an editor I'll enjoy using. I've been using Emacs for a Very Long Time and it's one of the harder habits to kick in this world.

Atom has a dialog UI which it seems to encourage - most Atom plugins follow suit and use the same interface to present a list of options. It's very reminiscent of the sheet that Sublime Text presented for option lists, and maybe in turn that interface came from another editor I don't recognise.

I use Tramp in Emacs to open files on remote servers. Sometimes you want to edit a file on a remote server from the editor instance you're already using. I noticed how much slower this experience seemed to be when using Atom than when using Emacs, so I made a quick screencapture of it and was surprised to see how much longer it really was.

Note the benefit of tab completion here too - in opening the same file path using Tramp, I made two typos which were corrected so fast I didn't recall making them until watching the animation play back.

Emacs + Tramp

[ lost image ]

Atom + Remote Edit

[ lost image ]

The time to set up the SSH connection is similar both times, but the delay IMO is introduced where Atom's sheet interface doesn't have as quick a tab-selection (even though it presents a visual list of options to choose from, a feature Tramp lacks).

Another interesting observation watching those animations is the time Atom's Remote Edit spends saying "no matches found". While the plugin isn't "selling" anything, the psychology of showing a "no matches found" message while waiting for files to show might be looked into - I expect just changing the message would improve perceived performance. (Brings this recent article about website experience and perception management to mind for me.)

This has also shown me how much I tend to keep a memory of file paths on remote systems; to me the visual interface just slows things down, but I also tend to perceive unexpected layouts as a bit of a bug (mostly this comes up when a predictable structure isn't kept to consistently). If I didn't have that internal reference, I'd probably not mind picking from a list each time. As it is, I haven't the patience!

Both Atom & its Remote Edit plugin are great tools - thanks @sveale and the Atom team for your work. I'm still flapping between Atom and Emacs on the desktop, but I'll be sticking with Tramp for this task. Implementing a Tramp style UX on Atom would be a fun project.

Trajectory, 2015

2015-02-19T08:44:00Z

Hi 2015! I've got plans for you.

Life's good at present - I enjoyed 2014 working with Fuzion Aotearoa, I'm excited to see what 2015 can bring.

Workwise I've spent a lot of the summer break thinking about our company's improving our internal systems for collaboration and teamwork - we're a small distributed team and I'm

OK so here's a set of 2015 goals.

Walk _____m / week
Stretch session n times a week

Here's a set of intents, things I'm still working out how to make measurable.

Teamwork
Diligence

Teamwork

I like that 2015 is the Year of the Sheep. Henry tells me "it's the worst year of the Chinese Zodiac, they hate it over there". I love Caprinae, and the isolation of remote work was a big takeaway from 2014 for me (and a driver for focussing on improving our company teamplay). Sheep and goats are about community for me, and one of my goals for this year is being part of an everyday community.

Diligence

Recently I took a day to focus on my professional development trajectory for the year, and as part of the day's reflection took a walk through the Botanical Gardens. In the flowing waters of the Leith was a forgotten shopping trolley, which I sketched, and ended up with this.

I have no idea why. Anyway ...

2014 Year in review

2015-01-06T23:14:00Z

It's the start of 2015 and I figured why not follow on from the year
in reviews offered everywhere with my own. Hi! I'm writing this for my
own reflection more than for public consumption, but sharing it might
just encourage me to be a bit more engaged. MAYBE.

At the start of 2015 I'm thinking most about family - we had Christmas
in Dunedin with Saira's parents and my brother James visited, but we
missed being in Auckland with the rest of our families. Living in
Dunedin puts us further from our siblings and parents, but living here
means more time for our family nucleus - something we both feel pretty
strongly about. Times like Christmas throw this choice into pretty
stark relief though.

In 2014 I reflected a lot on my career path after some conversations
with our school principal when she had her regular principal's
appraisal. I saw her approach to career which takes a longer term view
to professional development than I had considered, and wanted to
establish similar perspective for myself. This forced me to ask
whether a larger workplace might give more opportunity to learn from
teammates - currently I'm in a solitary office with a team of four~six
distributed about NZ, and I sense that I could be learning more FIXME
"be the lions tail".

My approach to this was heavily influenced by a single tweet FIXME
which I saw from @br3nda while attending FIXME. I value both our
family's happy life in Dunedin and my supportive employer, and I
realised that by doubling down on that commitment and improving my
existing situation rather than seeking to replace it, I could ramp up
my learning and improve things for our team.

I got sunk into improving our processes - something I'd felt strongly
about since joining the team a year or two ago, but had lacked
confidence about driving change with, not wanting to impose change
on a team that I was the newest member of.

In 2015 I'll work to strengthen this aspect of my work, and extend it
into other areas of my life.

One element of the work environment which I think we'd struggled with
was a sense of teamplay - making sure that team members are informed
of changes and the reasons for them, that configurations are
documented and so forth. My aim was to improve visibility across the
organisation so that we all had a sense of what the wider team was
engaged in. This is an ongoing project but I think the results have
been good for us so far. Over the summer break I've taken advantage of
the quiet time to further improve systems by automating monitoring,
reporting and informing - aspects of the business infrastructure which
I think had previously left a lot of the operation "off the radar".

Election 2014 was a big part of the year. Being in the engine room of
the Green Party's election process also helped inform some of my
thoughts on teamplay in projects. The election result wasn't stellar
but it was the predictable outcome.

I studied Te Ara Reo Maori level 2 with Te Wananga o Aotearoa, which
was a really good experience. I recommend the course massively (it was
a zero-fee course, in the local community rooms with about two dozen
students). Kia ora Ritchie and Shaun for your great work this year
... I learned a lot, including a good reminder that being a student
requires commitment. I came away with much more confidence than I
started, which was what I hoped to achieve; I don't think I'll enrol
again next year - I'd love to but I want to focus my efforts closer to
family and work for now.

Election 2014
BoT chair
BCAC rates
Happy teams tweet
NZG computing, wordpress hack
Drupal South 2014

Learning where you are

2014-11-25T07:20:00Z

I've been reflecting on work lately. I enjoy the team I work with, the crews we collaborate with and our clients who are all pretty damn good people. But I feel like I could be learning a lot more, and wondering about something more challenging and which I'll learn more from.

Then: how can I make my current work more challenging and more learning oriented, so that I'm learning and being challenged in new ways without having to lose the great team I'm with.

Tonight was graduation ceremony for TWoA Te Ara Reo Maori Level 2 which I studied this year. Great to complete the course (I had a lot of re-learning to do with regard to study habits!) but tonight signalled a new space in my life too (for now - Level 4 next year?).

So for now I can see a few opportunities to build things up where I'm at -

Continuing improvements to our technical systems, which I'm enjoying
Improving team collab within the company, which I think is something we could improve on
Growing things to a point where we can scale up beyond the current size, which I hope is a goal we share
Learning new tech to apply within our company, which I realise was easier to justify when I was self-employed

Not quite the same as being on team with people I can learn from, but I'm finding opportunities for improvement where I'm at all the same. And there are lots of good reasons to be happy here too.

Also: Tonight I started jotting down small things I've learned during the day, partly to combat the nag that "I'm not learning anything". I am - it's just easy to not notice sometimes. (Maybe I want to be learning things I value more; maybe I want to be learning more deeply.)

IBM Watson Personality Insights is a horoscope machine at the Victoria Park Markets

2014-11-23T06:32:00Z

Here's what it says about me, based on being fed a randomly handy post from this here blog.

You are sentimental.

You are empathetic: you feel what others feel and are compassionate towards them. You are proud: you hold yourself in high regard, satisfied with who you are. And you are assertive: you tend to speak up and take charge of situations, and you are comfortable leading groups.

You are motivated to seek out experiences that provide a strong feeling of connectedness.

You are relatively unconcerned with tradition: you care more about making your own path than following what others have done. You consider helping others to guide a large part of what you do: you think it is important to take care of the people around you.

I suspect it's probably fairly right, for the instance of me which existed while writing that post, but these sweeping "you are X" claims seem to have all the credibility of this yellow machine at the Victoria Park Markets which took your DOB and spat out a page of dotmatrix horoscopic proclamations.

I kinda liked that machine too, and computers are pretty good at horoscopes anyway since it's just maths right? But I didn't trust it.

Making team-talk a part of everyday work

2014-11-23T06:32:00Z

Lately I've been studying my task focus, and noticing which activities are "interruptive" in the sense that they break the flow of productive work. Number one on my hitlist is Skype, which is the primary comms channel at Fuzion currently.

I don’t know why Skype is “more interruptive” for me than other apps. I know that I’ve talked before with Eileen about preferring IRC, where I feel quite comfortable ignoring a question until I finish the task at hand - and that’s fine IRC etiquette. Doesn’t feel so on Skype. For me, one of these is asynchronous I/O and one of them is not.

(Something here about the tyranny of the red badge notification icon. Another time.)

Anyway, we're a distributed team, we need a way to communicate as one, and teamtalk is important I feel. When Skype is the primary comms, and it's quite an interruptive one, it encourages siloed communication - I'm talking to Eileen, Pete's talking to Gemma, but it's direct and 1:1 unless we drag everybody in.

Team conversation isn't like that; it's something that everyone can opt into, and of which we can comfortably tune out as appropriate. The same rules that apply to overhearing workmates converse across the room; you can chime in and drop out and there's virtually no mental cost to doing so, because it's part of being a human.

(Very relevant: this 'shard' from #devmob by @br3nda today, localcopy - thanks Brenda for sharing as much of your process as you do.)

I'd like to see each of our team members to be part of a shared company space, make being present as a team a thing that is part of normal daily process, and have a place where we can discuss what's happening. There are great integrations available for us to leverage a tool like Slack or HipChat with the other tools that are part of our daily flow.

So! I proposed today via email to the team that we try swapping out Skype for HipChat for a month or so, and see how it works out. I'll update after a month and give our experience (well, my experience and what I hear back).

I'm also +1 on Eileen's response to this suggestion, that we should try and reconvene around our daily standups. Again I think having a common meeting place is critical to understanding where your fellow humans are at. A chatroom and a standup gather us in different dimensions; with standups a daily roundup of "what's happening for me" and especially sharing of what we've been learning is really valuable. Working alongside the principal at our school has shown me how important it is to value shared learning and open reflection.

I found the idea of standups challenging (don't put me on the spot!) when I first experienced it at Webscope, but I now think it's so important to support that space and make it a fixture of process, at least until it becomes something natural.

I'm enjoying engaging this happy team thing - it's reinvigorated me with regard to my work, and I'm hoping that I can make a real positive change in our work environment.

Speedcurve

2013-10-10T21:22:00Z

A couple of weeks ago I got to try out Speedcurve, which has just launched today.

It's a smooth product which gives both a good high-level view of site performance over time, review the major speed factors (differing browser render time etc) and phases of page load, and index against other competing sites and the wider "ecosystem" of your site's sector. Also, the historic detail is like having a daily request trace (super useful) along with a very-low-resolution archive.org / screengrab (awww!) all of your own.

I've been doing some basic monitoring of site performance using Munin, and what I missed in Speedcurve were the ability to monitor the same set of sites (from previous posts, it's probably obvious that I'm currently obsessing over NZ political party sites in the leadup to the NZ election) and the ability to manually adjust colours (I know, but it just doesn't make sense if the Green Party are orange and the National Party are green).

Very small complaints there - one a beta account limitation (Speedcurve plans and pricing), the other down the bottom of a list of things people want.

The one-off monitoring options will be a really good way to give sites a periodic review ... congrats to Mark on getting it out the door!

Cloudspotter

2013-10-10T11:38:00Z

cloudspotter

A Cloudspotter Spottings tag on Github.

Might get pretty big - the list of images I mean, not the Jekyll plugin - so probably don't want this as a gallery so much as an ongoing stream. Will do for a first stab.

Should have called it "CLOUDTAG" though.

Based on Thomas Mango's Flickr Set plugin for Jekyll.

Pow

2013-09-10T21:16:00Z

I know it's pretty old news, but I just have to say that as someone who's messed around with hosts files and apache configs for some years now, I'm freaking love the convenience and lazy simplicity of Pow.

Even when it's just proxying back to LAMP apps, it's still saving me a bunch of effort. (I've automated that effort with .sh in the past, but Pow is just soooo easy.)

Political sites speedtested (again!)

2013-07-08T12:36:00Z

I'm not bored of this yet. I'm just getting started. Trying WebPageTest.org this time to compare results - it isn't quite as pretty but I think it's more detailed.

Times are initial / second load. These results are substantially different to the previous two rounds.

Labour

1.871s / 1.270s.

United Future

2.876s / 1.094s

Greens

2.983s / 1.900s

NZ First

4.310s / 2.405s

ACT

6.545s / 4.272s

Mana

7.514s (did we lose the test machine with the initial load time?)

National

10.121s / 6.093s

Maori

14.619s / 4.078s

PGP in Parliament.

2013-06-28T12:34:00Z

With greater awareness of the visibility of email content and traffic, I got wondering whether any NZ elected representatives had published PGP keys in order to give the public a means of communicating with some level of privacy.

S/MIME? I don't know of a way to look up Parliamentary certificates to obtain the recipients' cert, and otherwise you'd have to get an email from an MP or EA with their cert. (If such a directory exists, I'll be happy to update this.)

I checked the MIT PGP keyserver for results in the @parliament.govt.nz domain as well as seven @partyname.org.nz domains (sorry if I forgot your favourite party, I've got that wrong before). The results are few, out of date, and not encouraging.

PGP keys in @parliament.govt.nz

Rodney Hide
EAs belonging to (I think) ACT, Labour and National (1 each)
A few others

PGP keys in @national.org.nz

One person

PGP keys for @labour.org.nz, @greens.org.nz, @act.org.nz, @maoriparty.org, @unitedfuture.org.nz, @mana.org.nz

Nobody.

Friends in Parliamentary Service tell me that encryption is a hard proposition for use within Parliament, because there is tight control over what software can and can't be installed. I'm not sure what encryption options are offered by Parliamentary Services IT, but it doesn't look like it's PGP.

I emailed a handful of MPs to enquire about their current approaches to private / secure comms with consituents. National MP Amy Adams, the Communications and Information Technology Minister, referred me to Parliamentary Services. Through a response from PS I learned about SEEMail ("Secure Electronic Environment Mail"), which provides secure mail gateways to communicate between SEEMail-using agencies. By my reading of this document) it seems like SEEMail

is only available to participating organisations,
has mail gateways apply rules to communications (eg recipient
constraints on a document marked [RESTRICTED],
outsources PKI management to Dimension Data,
encrypts and secures based on agency certificates rather than
certificates for individual recipients

References -

Hello Pete & CiviCRM.

2013-06-28T10:41:00Z

Pete and I met several years back - I'd been doing a bit of consulting
for the Green Party of Aotearoa NZ and Pete was working for them at
Parliamentary Services.

They were looking towards a system for managing their contact
systems. Nandor had got a toehold for open source representation in
Wellington.

Someone (probably Mark) had asked me for input on the options on the
table, and realising that no-one was batting for a web-based, open
source CRM option, I went to the AGM that year in Nelson to show what
you could do with CiviCRM.

(Until that point, I hadn't heard of CiviCRM, but it seemed like a
better philosophical fit for the org than the options they were
considering - all very windows, not very open in either the web
delivered or open source aspects.)

I demoed CiviCRM, and then skipped the rest of the AGM (introvert!).

Saira and I went over to stay near Farewell Spit, far away from the
meetings and technology and stuff. When we came back the party had
decided that CiviCRM was a good option and could they have one
please. And that was how I ended up doing a lot of work with CiviCRM
over the five years since.

Friday night, delighted.

2013-06-28T10:41:00Z

It's Friday night, I'm delighted. Home and quiet, the fire warms the
room, it's midwinter cold outside. About a month ago I started working
for Fuzion, run by a long-time colleague and
friend Peter Davis. I'm loving it - a great crew to work with, it
feels like a good fit.

Right now, my delight is 100% because I get to leave work at
work. I've always had a "protestant work ethic" thing going - I'm not
a particularly hard worker, mind - just kind of driven to work in
order to reassure myself that I'm doing my bit.

So tonight, I finished work around five, and then I went on a spree of
custom code removal (11K LoC gone in 1:15, yo - what a freaking mess
that was), and then I hopped the bus home. And I'm home, and putting
the kids to bed, and feeling absolutely healthy because I don't feel
like I have a pathological need to pick up some unfinished business
later on in the evening.

(Sorry, peeps with unfinished business, see ya Monday! Unless it's
urgent enough that you really need me. You've got my number.)

I love having a job. It took me a long time to realise it was a
healthy option for me.

A second survey of NZ political party sites

2013-06-28T09:47:00Z

A second survey, to follow the first unscientific survey of NZ political sites. On the eve of the Ikiroa-Rawhiti by-election no less.

I'm only following this up out of duty to my past and future selves, you understand. Also, it occurs to me that FPT probably offers a permalink to the report, and I could skip the images copypasta. But hey, consistency?

I am a little bit excited to see Mr Zeman's been busy putting together Speedcurve, it looks pretty fab from the demo. Hi Mark!

EDIT: Oh dear! I forgot ACT existed both times I ran this survey. I'm belatedly adding them in third place below, well done Berend, fastest of the Drupal sites here.

United Future

Supermodel

[ lost image ]

Mana

Wordpress

[ lost image ]

ACT

Drupal 7

[ lost image ]

Maori Party

Website Administration System

[ lost image ]

NZ Labour

Drupal 6

[ lost image ]

NZ First

Drupal 7

[ lost image ]

NZ Greens

Drupal 6. Lots of room for improvement here!

[ lost image ]

NZ National Party

VAdmin 3.0 Enterprise

[ lost image ]

NZ673 flight deck

2013-06-05T04:36:00Z

The lads got invited onto the flight deck of NZ673 by the burser, and met Arjen the pilot.

Turns out that on the domestic 747-400s at least, it's not all shiny shiny Pan Am Raybans up front. Just raw functional machinery.

Lucky boys - I'd thought that cockpit visits were a thing of the past, but we're not that far gone.

Letting go is easy

2013-04-05T23:37:00Z

CiviCRM blows me away. I love the people who use it and the development team, but I kind of hate the codebase. It's insane that I've been working with it as long as I have.

Still I'm drawn back ... why is that? I'm not sure.

CiviCRM, payment, and processing

2013-04-05T23:37:00Z

CiviCRM's contribution pages are no Obama campaign. Certainly not out of the box. On the backend, there are issues too - like the fact that CiviCRM's forms layer drops submitted data to the DB while processing submissions, and that contribution form submissions contain credit card details. (It's OK, provided your contribution doesn't fail, CiviCRM will clear your PAN and CVV really soon, promise.)

Unless a client has a dedicated security team (in the CiviCRM space, probably only the EFF) I push them to use a hosted processor. I'd rather you use PayPal than stored PANs in the DB. But onsite transactions are a thing, and if we're building modern contribution pages, we're not going to 302 you off to paypal mid process.

So - how to get some contributions into CiviCRM and sidestep the horrors of CiviCRM's forms layer? Glad you asked :)

For the AU Green Party in the 2010 Australian Federal Election, we added an API method to CiviCRM so that we could access transaction processing via the API. (Nothing to do with PAN storage, all about the horrors of Smarty at the time.) We used Drupal FAPI to build a simple form for the donate interface, then called civicrm_api() from Drupal's form submit hook to do the CiviCRM layer stuff.

This year is election year in Australia as well, and so the same question comes up. I expect we'll face this in 2014 with both Labour and the Greens in NZ using the Drupal/CiviCRM combo.

If I was rewriting that three years later, I'd be thinking about these things -

How can CiviCRM's Contribute forms be better modularised? Last I saw of the contribute flow, it was a sprawling mess (and getting messier) of pledge blocks, profile includes, contribution and donation fieldsets, event and membership generation, and all sorts of other things.
What parts can't be done yet at API layer? This is the real kicker. I'd really rather not have to clean up the various elements of form processing, but I can help make sure the right replacement elements come into play.
How can payment processors expose themselves to the API? Currently CiviCRM payment processors implement some magic function names, but really seem to be a morass of special cases and checks.

CiviCRM, Smarty looking better

2013-04-05T12:24:00Z

Tim Gummer asked me to look at a template tweak for Natural Areas a few days ago - they wanted to display a little extra contact info (membership type, status, what have you) on the contact summary page.

I'd been resistant to CiviCRM's Smarty syntax from days of yore when it was a crazy nested mess of .tpl files, and a right pain to substantially rejig in any shape except the defaults. I did not expect to have much fun ... in fact my initial impulse was to sidestep Smarty entirely and use JS + CiviCRM REST API to get the data. Yeah, that resistant.

Well, since Tim asked nicely I figured we should take a look at it. And as it turned out, it wasn't so bad after all. It was still pretty crazy - I didn't look to see what happens on the backend, but we end up with three crmapi calls here. It's still Smarty, and it's still CiviCRM.

That worked.

CiviCRM upgrades via Vagrant

2013-04-05T12:24:00Z

The last few weeks I've been working for the NZ Green Party on an update to their CiviCRM DB. I expect a bit of suffering in any CiviCRM upgrade, and I wasn't disappointed. (I accept suffering as inevitable, but that doesn't mean I need to embrace it.)

I wanted to kick off with a recipe which would let me reproducibly bring up upgraded versions of CiviCRM with a fresh copy of the site data. (See @todo Obfuscating CiviCRM test data for more on that.)

(@todo I should write an embargo plugin for Jekyll so I can say embargo: +3mo in my post's YAML.)

(@todo I should write a @todo plugin too.)

Ideally, all this would also build towards a more reproducible infrastructure for the NZ Green Party to hand out to developers. The current setup (a dev / staging box) was bottlenecking new developer contributions, and there were no quick answers to the "how do we bring a new team member onboard" available.

I look forward to a situation where a new volunteer can be given instructions to deploy a Drupal/CiviCRM instance, fetch a sample dataset (defanged of sensitive info), and start hacking for world domination.

The tools chosen so far for this round turned out to be -

Lots of RAM.
Vagrant and VirtualBox for VMs, chef-solo to deployed instances. (Baby steps.)
Librarian to manage a set of fairly standard LAMP (& drush) cookbooks.
Drush make to build the Drupal / CiviCRM "app".
A custom DB upgrade script, which output on provision & called on demand.
Patience.

The interesting elements turned out to be -

CiviCRM upgrades are fraught. If your DB isn't in the exact configuration CiviCRM expects, it chokes hard on upgrade. CiviCRM upgrades and DB issues.
CiviCRM relocation requires a few gentle taps in the right place. Deploying CiviCRM.
Obfuscating DB content for sharing.

(@todo I plan on writing more.)

Just the medicine

2013-04-05T12:22:00Z

For the last several years I've been rigging websites largely for NGOs and dot-orgs who are campaign focussed - typically political or non-govt, and as often as possible lefty liberals who are trying to save the world.

Aldo asked me once to put my finger on what my aims were - I'd said I wanted to do good, but not what. His question was the right one; I realised that rather than finding something to be passionate about, I'd outsourced my motivations to orgs that were doing something "about right".

Realising that I was letting the situations and client pressures dictate what tools I was using was a good first step. I'm not toolyvangelist but you need to be selective about what you're working with, especially if you want to maintain passion about your work.

And I'm not all that passionate about some of the tools I've been using. I can make use of that - it's good to be critical and to be able to be dispassionate - but you don't want to go cold, and I want to relish working with the things on my bench.

My favourite people are tool makers, not tool users. That's an evolutionary thing.

And dogs. Dogs are good.

Oh, the medicine thing in the title, that was another post. It can stay.