sdoconnell

Just the other day, I decommissioned my bespoke, self-hosted Wireguard VPN and AdGuard infrastructure, and replaced them with a Tailscale network, Mullvad exit nodes, and NextDNS. Today, I decommissioned both my internal and external Gitea servers, and moved all of my repos (public and private) back to Github.

I honestly don't know if I'm having a mid-life crisis, or if I've simply reached the apex of my growing intolerance for tilting at windmills. I used to be such an ardent self-hoster, and these days I just don't care to hassle with it in the slightest. (And it is a hassle. It's not by coincidence that this Github move follows on the heels of a multi-day outage with my public repo server, which required a non-trivial amount of my attention to resolve.) Recently, I've even caught myself actively looking for things I can hand off to someone else, and just pay them to deal with it.

I think this all really comes down to personal bandwidth, and the value I increasingly place on it these days. Perhaps there is some underlying awareness of my own mortality at play (I'm not a spring chicken, after all). Or maybe it's the practical recognition that there are only so many hours in the day, and there are things I want to accomplish that are far more important to me than running a farm of personal servers to provide services that someone else is more than willing to provide me for pennies a day.

Older and wiser? Or just lazier? I will concede that either possibility (and perhaps, even both!) may be truth.

So I had a bit of a monkey-wrench thrown at me earlier this week. On Wednesday, my home firewall decided to (quite suddenly) die on me about 15 minutes before my wife (who, like me, works from home) was about to login to attend a business meeting. Isn't that always the way?

Now the thing is, I only just installed this new firewall about six weeks ago, so I was more than a little miffed that the thing went and gave up the ghost already. But luckily, I still had my old firewall in the closet with an only slightly older version of OPNsense and a slightly older config. I also had a backup of the current config that was only a week old. Recovery was a relatively quick and simple effort of digging out the old firewall, plugging it in, updating the firmware, and restoring my backup config. About 40 minutes or so, and at least ten minutes of that was dealing with a tangle of power cables.

At this point, you might guess that this post will be a rant about how crappy my new firewall was, and how the company sucks, and how you should never, ever buy their garbage product. But you'd be wrong. I come not to bury the vendor, but to praise them.

Six weeks ago, I bought a Protectli Vault. Specifically, the FW2B—a fanless, two-port mini-PC designed to run open source firewall platforms such as OPNsense and pfsense. I bought a barebones model from Amazon (because I already had compatible memory and mSATA disks on hand). And since I already ran OPNsense on my existing mini-PC firewall (a Shuttle DH310), the replacement involved simply installing my components into the FW2B, installing OPNsense, restoring my existing config, and swapping hardware. Less than an hour of work.

[Side note: the only real issue I had with the Shuttle box was the fan noise, which over the years has gotten louder, and a little grindy due to worn bearings. I could have probably just swapped the fan but it would have been more trouble, and the box is really overkill for my firewall needs. Plus, the FW2B uses less power and produces less waste heat, so there were other advantages to the replacement.]

Getting back to Wednesday's failure, once I had the old firewall in place I began to do some diagnostics on the FW2B and found that I had a complete failure of one of the two onboard NICs, as well as an intermittent problem where the device would occasionally lock up on boot. After I determined the problem wasn't being caused by either the memory or disk (i.e., my components), I opened a support ticket with Protectli to start an RMA and get a replacement for the failed device.

Here's where things get good. I submitted a ticket with all of the troubleshooting info I had gathered, a screenshot of my order on Amazon, the serial number of the device, and a request for hardware replacement. Now, keep in mind that I didn't buy this unit directly from Protectli's online store, and that I had no support contract of any kind. Given my prior experience with consumer electronics companies, I expected a few things:

1. A long delay in response, if I got a response at all.
2. A ton of questions, back and forth communication, and pointless instructions to perform troubleshooting I had already done and documented.
3. A lot of hassle and runaround.

In fact, I pretty much considered the US$230 I'd spent on the device to be a loss, and I'd already written it off.

So you can imagine my surprise when less than 15 minutes later I received a response asking a single question:

What's your shipping address?

That's right. This company—who didn't know me from Adam—responded to my ticket in less time than it takes to get a pizza delivered. And without giving me the runaround or demanding any kind of payment information first, they bench-tested a replacement unit and shipped it out to me same day. In addition, the communications from the support engineer were not only professional, but were also personable, friendly, and conveyed in clear, understandable English. (Shout-out to Skip Star. You're awesome, dude.) I received a (AFAICT) NIB replacement unit yesterday (Saturday) along with a prepaid return label to send back the failed device at my convenience.

There are multibillion dollar enterprise technology companies from whom I have received much poorer RMA service (sup, Cisco... how you been?).

Look, hardware is a commodity. Sure, my device failed. It's an SBC made in China, like practically every other consumer electronics product on the market today. A non-zero failure rate is more or less expected. What matters to me is how the vendor handles those failures, and Protectli took care of me. So unless that company starts e.g., chucking kittens into a woodchipper live on Youtube, they just earned a customer for life. In fact, I already ordered a second firewall on Wednesday to keep as a shelf spare (knowing that hardware does fail, and if it's critical, you should have a backup). The device is sitting on my desk right now, awaiting OS load and a configuration restore.

I don't give endorsements often, but I can wholeheartedly say this: if you're in the market for a SOHO firewall, and specifically an OPNsense or pfsense device, you owe it to yourself to at least consider buying a Protectli Vault. The hardware may not be perfect (realistically, no hardware is), but at least the vendor will have your back if and when things go sideways. I'm very happy with my own firewall, and I have been nothing but impressed with the level of support I've received from the company behind it.

I needed a little diversion from writing this week (you can only hammer on something for so many hours in a day for so many days in a row), and so I decided to do some much-needed work on my website. My intention was to discontinue use of my custom static site generator and replace it with something off the shelf—most likely, my old nemesis Wordpress.

Then I setup a Wordpress dev instance and almost immediately remembered why I stopped using it in the first place.

After looking around at few other options, I quickly came to the conclusion that pretty much all "off the shelf" solutions suffer from a common problem. If you are creating a site from scratch, Wordpress and others will give you a fairly simple onboarding process. Pick a theme you like, customize some colors and fonts, maybe add some widgets (if supported by the theme), and then away you go. Just add your content (images, pages, and posts).

However, if you already have a website—and you like the way it currently looks and works—recreating that site in those off the shelf content management systems is a royal pain in the ass. Because basically, every CMS uses their own theming system, and if you want your new CMS-based site to look and work like your existing site, well... you're in for a metric crapton of work.

• Step 1: spend a week or more poring over (sometimes sparse, misleading, or outdated) documentation to learn how to make a theme for your choice of CMS that closely approximates your existing site.
• Step 2: spend a week to a month writing and debugging said theme.

As Kimberly "Sweet Brown" Wilkins once famously opined, "...ain't nobody got time for that."

There were a few reasons I started using a static site generator in the first place:

• Tighter security with no database backend and no server-side code execution.
• Better performance because I'm only serving static files.
• Easier administration meant no need for admin panels, etc.
• More streamlined content creation because everything is just a text file, and all I need to create or edit, is an editor.

But along with those upsides, came a few issues:

1. Static sites can be a little too static. Sometimes it's handy to be able to run some server-side code to enable a feature or to make your site a better experience for the reader. With a static site, any dynamic behavior has to be coded client-side, which means that you have to accommodate cases where the user has disabled Javascript in their browser. That involves either gracefully degrading functionality, or just popping up a message saying "sorry, Javascript is required."
2. File sprawl. Because everything is a static page, for paginated content like blog posts, you end up with index.html, index1.html, index2.html, etc. Which can become problematic when a visitor bookmarks your website, not realizing they bookmarked index37.html instead of the root document. And what's worse is that as you add more posts, the contents of index37.html will change. A month later the content that was on index37.html is now on index39.html and the user's bookmark is invalidated.
3. The edit/build/deploy workflow may (depending on your tooling) require some very specific environment dependencies which can become brittle over time. System updates, library updates, moving to a new computer, etc., can all break your build environment. And if you don't have the best documentation or some composable implementation, it can be tricky to recreate the same setup you cobbled together a year prior.
4. SSGs can become a bit of a time-sink. You write up a post, kick off your build, and then get an error. Maybe it's a quick fix, maybe it's several hours to get the thing working again. Meanwhile, you just wanted to make a blog post...

Anyway, so I wanted to get off the SSG but the road to Wordpress or another CMS would just be too long. I can divert off my book project for a few days, not a few weeks. What I came up with was basically a hybrid solution.

• A PHP-driven backend "CMS-lite" based around the assembly of webpages from component parts via includes. This was fairly easy to put together just by taking my current SSG templates and CSS, and replacing Jinja code with PHP code (sort of).
• A flat-file data model consisting of JSON files. While not as human-readable as YAML, JSON is still plain text. And actually, after dealing with some of the peculiarities and ambiguities that arise from YAML's human-friendliness, I've actually come to prefer JSON's more regimented structure and predictable behavior. Even if you can't wrap long strings (which kind of sucks).
• A self-contained LAMP stack wrapped up in a Podman container. Which makes deploying my site (along with all of its dependencies) to a new host a simple matter of rsyncing a directory and adding a systemd unit file. For the frontend, I still proxy behind NGINX like all my other (static) sites.

Adding content to my new site is a matter of writing the content into a file, and then adding an entry to the JSON "database." Deploying the change involves copying/updating both files to a directory on my host and running a server-side script to update the Atom feed. With about 30 seconds of work, I wrote a bash script to automate a new post deployment. So, write post, update database file, run script. Done.

Making edits is just "edit file, run script again." Which I've done about five times so far (I should have proofread better, sorry).

All of the site content, and everything related to my website (even the Apache config files) is stored and version-controlled in a local git repo and backed up offsite with the rest of my stuff.

With this system, I get a flexible, dynamic site where I can add more advanced features, but I also keep a flat-file, read-only (to the app layer) data model with better performance and (somewhat) better security than with something like Wordpress. And I was able to convert my old static site with minimal style/structure changes in a matter of about three days. That's not a huge time investment. And I think I spent more time tweaking CSS for a visual update than I did writing backend code...

The only real downside of this new solution is that it's still bespoke code that I'll have to maintain (which is something I was trying to get away from). But since the version of the containerized backend is pinned, the "host" of my site is more or less frozen in carbonite, and I shouldn't incur any issues with unplanned changes occurring from some random system update.

Unless an update brings a breaking change with Podman itself, but I've only had that occur once—with an in-place upgrade from Debian 11 to Debian 12. And because that upgrade was planned (and I expected some kind of issue to occur) I had already set aside the time to address any failures that followed. (Plus, simply recreating the affected pods resolved the problem, anyway. Quick fix.)

Anywhooo... we'll see how it goes with this new site model. Now back to finishing my book (sometime this year, please).

Wow, it's been almost a year since I posted anything to this site. I have a few reasons for that, but the primary one is that I've just been too busy working.

In no particular order, here are some updates:

• I've made some pretty significant changes in my personal tech stack over the last year or so. There's a bit of a story there, which I may write up in an article at some point, if and when I ever have enough spare time. But in short:
  • I've moved my desktop and mobile computing platform from homebrew Linux-based systems to Apple products and solutions.
  • I've moved all of my critical services (email, messaging, calendar, etc.) to cloud-hosted services (again, Apple).
  • My on-prem servers and storage have been replaced with Synology NAS appliances.
  • I continue to self-host a handful of non-critical services on my VPS infrastructure, but almost all of my bespoke tech stack has been decommissioned. As much as possible, the remainder has been moved to container-based workloads.
  • At some point, I will likely be converting this site from my custom SSG code to something off-the-shelf (probably Wordpress, but maybe something else).
  • I do continue to use Linux-based operating systems for a number of purposes. However, I have divested my personal use of Linux distributions based on or affiliated with commercial entities (e.g. Canonical, Red Hat, and SUSE) in favor of a return to Debian, a community-based distribution. Professionally, I continue to support commercial distributions (this is a change for reasons of practicality, and not a religious conversion).
  • UPDATE 2023-09-28: I've actually had a bit of change of heart on that last item, and have moved my personal servers to CentOS Stream 9. I've also started using Fedora for my Linux workstation/desktop needs (such as they are). There a few reasons for this, perhaps I'll write an article about it. But suffice to say, I've come to realize that I modify stock Debian post-install to a considerable degree. And by the time I'm done, it's almost indistinguishable from CentOS/RHEL. So, really... I might as well just install CentOS to start with, and in doing so, avoid the many debianisms I often find so annoying.
• I have purged a lot of old note content from this site, and may purge some old articles as well. Moving forward with the site updates, I may or may not even post notes in the future. I just don't have that much to say anymore. Or rather, I'm saying it in a different way (see below).
• I spent six months (roughly 2,000 hours of work) in 2021 developing the nrrdtools application suite (which amounts to ~25k LOC), and... yeah, I basically don't use it anymore. I'm still maintaining it as an "active" project for now, but it's no longer being dogfooded. I really don't know what I'm going to do with that one. I think there's a lot of really useful code there (particularly the ICS handling integration for mail clients like mutt) that I spent a lot of time debugging. So I'll probably not just throw it away or anything. I don't know, maybe now I finally have a reason to write a CalDAV/CardDAV sync component for the suite so I can continue to toy with it. But honestly... don't hold your breath for that.
• I've largely dropped out of the FOSS scene. I don't really listen to tech podcasts anymore, I've let my membership lapse in the FSF, and I'm not really following too many FOSS projects any longer. I've not gone anti-FOSS (exactly), but I've backed away from what I would call "active" community involvement. My personal code still continues to be released under GPL for the time being, and any changes to that would be to a more permissive license (e.g. MIT license), not to a less permissive or closed source license. I'll continue to make FOSScan donations, because I do continue to use Open Source software.
• I am no longer an active Emacs user. "I still have the feels, but baby I got to quit you." Again, there's probably an article here if I ever get the time to write one.
• I'm currently in the process of preparing for an eventual move back to Las Vegas, although the timetable for that is not set. The current guesstimate is within 12 months. My wife and I are very much looking forward to that happening. Both of us have grown weary of the cold winters and humid summers of Arkansas, and the desert is calling us home.

So, yeah. Lots of stuff has been going on during the period of my digital sequestration. However, you may have noticed that the above doesn't really explain what has been keeping me so busy. Well, for the last 18 months, the near entirety of my focus has been devoted to the task of writing a novel. In fact, I'm currently on the third draft and still deep in the editing process, but I hope to have a finished product within the next six months. At which time, the journey toward publishing will commence. As I plan to publish under a pseudonym, I'll not mention the title or synopsis here, but I will say that it is not a short novel and has been a massive undertaking (far more involved than I thought it would be when I started writing a still undefined "short story kind of thing" in January of 2022).

Anyway, that's it. Still alive. Working hard. Talk to you again in six months (hopefully) when I can pull my head out of the word processor.

Looking over at my #FOSScan this morning I saw it was full once again. Time to count it up and make some FOSS donations!

The change collected since I last emptied the can 14 months ago amounted to US$75.00.

Today, I've made the following donations:

• $40 to the Let's Encrypt organization for their terrific (and free!) Certificate Authority service.
• $35 to the Homebrew project. Their awesome package manager for FOSS applications has made my transition to the macOS platform so much simpler.

(Yes, you read that last part correctly.)

Anyway, this brings my #FOSScan donations to a total of US$239.94 since I started the project in 2019.

Do you have a #FOSScan of your own? If not, eat a can of soup (or beans, veggies.. whatever), make your own #FOSScan, and start collecting spare change for Free Software and related projects today! Together, we can make a difference in Free Software funding.

If you haven't heard from me pretty much at all in the second half of this year, it's because I've been heads down on a passion project for the last six months. A project which has just hit a major milestone today with its first release.

I am super happy to announce the launch of the nrrdtools project.

nrrdtools is a suite of terminal-based productivity applications for managing your calendar, tasks, contacts, reminders, bookmarks, notes, journal, and time logging - all from the shell. This project came about due to my prolonged dissatisfaction with existing command-line/text-mode solutions, coupled with my general lack of faith in the future of graphical application development for the Linux desktop. There's probably an article there, I just haven't written it yet.

((frustration + cynicism) * hubris) / sparetime = code

I'm coming away from this project with the self-assurance that this has been a most productive sabbatical. I guess now I'll take a couple days off before picking up my next contract for 2022 and going back to actual work.

Looking over at my #FOSScan this morning I saw it was full. Time to count it up and make some FOSS donations!

The change collected since I last emptied the can (wow) 16 months ago amounted to US$62.00. Yikes! I guess the lockdowns over the last year plus have really stifled the opportunities to go out and collect spare change.

After a bit of frustration, I've made the following donations today:

• $28.84 to the Late Night Linux podcast. While not a FOSS project per se, they do a lot when it comes to communicating to and evangelizing for the FOSS community. And I've been listening for years and never donated, so this is a bit of a make-good.
• $13.12 to the NeoMutt project.
• $20.04 to the MATE Desktop project.

If those amounts seem kind of strange it's because of USD to GBP conversion.

I wanted to donate to the XFCE Desktop project but they made it too goddamn difficult to give them money. They only take funds via Bountysource, so I actually went out of my way and tried to create a Bountysource account. Unfortunately the sign-up process is completely broken (at least, at the moment), doesn't send email confirmation or password reset emails, and didn't give me any link or button to actually give the project money (probably because my email address wasn't confirmed).

Hey XFCE team (and several other projects I visited today), put a fucking PayPal "donate" button on your website and maybe you'll get some contributions next time. Not everyone wants to sign up for Patreon or the payment processor du jour to give you money. PayPal is pretty damn ubiqitous so make it easy for donators to actually donate, okay?

Anyway, this count brings my #FOSScan donations to a total of US$164.94 since I started the project in 2019. Considering this is in addition to my regular annual donations to specific projects and organizations, I guess an extra $55 $82 a year in FOSS funding is not too bad. I'm going to have to work on finding opportunities to add to the can, and see if I can't bump that closer to $100 a year in spare change donations.

Do you have a #FOSScan of your own? If not, eat a can of soup (or beans, veggies.. whatever), make your own #FOSScan, and start collecting spare change for Free Software and related projects today! Together, we can make a difference in Free Software funding.

EDIT: math fail.

Man, it's June and this is the first time I've posted to my website in 2021. What can I say, Valheim is a hell of a drug.

Kidding (sort of). Yes, I lost a month of spare time to that game but really the reason I haven't posted in a while is because I've been doing some server consolidation and site moves/changes. As a part of that project, I've been wanting to convert my LAMP-driven personal website to a static site for quite some time. Security is prime driver for that decision, but portability is also a factor (more on that in a bit).

Unfortunately, in doing research and evaluation of a number of the popular static site generators I found that all existing SSGs suck*. So... I decided to write my own (why do I get the feeling that this is how all static site generators get started?).

I'm leveraging Python and Jinja2 templating to convert Markdown and YAML source files to static html. I had started exporting and reformatting all of the content on my website back in January or February, but then sort of lost inertia on the project. However, in the meantime I didn't want to add any new content until I'd gotten the new SSG built (otherwise I'd have to convert even more stuff), hence my lack of recent posts.

About a week ago I finally forced myself to sit down and knock out the first version of my custom SSG. It's less than 800 lines of code but it replicates my former website pretty faithfully, including features like my photo albums. One of my design goals for this new iteration was for the website to work both online and from a local directory, and some changes were required to paths and filenames to accomplish that. One of the reasons for this design requirement is that my near-to-medium term goals include being able to publish both on the web and via IPFS. In order to do that, the site needs to also work from a local filesystem directory.

For the most part, I was able to preserve existing URIs using server-side redirects for the old paths, pointing them to the new files. The primary exception is that the photo album structure was completely revised but that shouldn't be a big deal. The URIs for the photos themselves remain unchanged, and I'm more worried about links to previous notes or articles continuing to work.

This post comes shortly after the first push of the new static content. I've already been fixing a few bugs here and there as I write this, I'm sure there will be more.

* Okay, they don't necessarily suck but let's just say I couldn't find one SSG that would build my website content and maintain the structure I currently have without practically re-writing the SSG itself. At which point, it just made sense to write my own from scratch.