Tinker, break and fix

RHCSA 9 changes

2024-01-21T01:45:00+00:00

For anyone else labbing with Rhel 9.3 and messing with /etc/default/grub GRUB_CMDLINE_LINUX= options not taking effect, such as removing options "rhgb quiet", after running grub2-mkconfig, you now need to add --update-bls-cmdline to grub2-mkconfig or set GRUB_ENABLE_BLSCFG=true to GRUB_ENABLE_BLSCFG=false.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.3_release_notes/index

section 4.9 boot loader

Alternatively, we can use grubby, such as:

[root@localhost ~]# grubby --update-kernel=ALL --args="bogus_kernel_arg"
[root@localhost ~]# grubby --info /boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64
index=0
kernel="/boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64"
args="ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap bogus_kernel_arg"
root="/dev/mapper/rhel-root"
initrd="/boot/initramfs-5.14.0-362.8.1.el9_3.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.14.0-362.8.1.el9_3.x86_64) 9.3 (Plow)"
id="98edf952bc814fc39e39cb067e0381e8-5.14.0-362.8.1.el9_3.x86_64"
[root@localhost ~]# grubby --update-kernel=ALL --remove-args="bogus_kernel_arg" /boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64
[root@localhost ~]# grubby --info /boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64
index=0
kernel="/boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64"
args="ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap"
root="/dev/mapper/rhel-root"
initrd="/boot/initramfs-5.14.0-362.8.1.el9_3.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.14.0-362.8.1.el9_3.x86_64) 9.3 (Plow)"
id="98edf952bc814fc39e39cb067e0381e8-5.14.0-362.8.1.el9_3.x86_64"

RHEL 9 hidden grub menu

2024-01-21T01:45:00+00:00

For anyone taking RHCSA9 exam soon, this might be helpful.

On newly installed Rhel9.3 virtual machines, the grub2 boot menu gets hidden after successful boots.

This makes test exam tasks more difficult that wants you to reset root passwords, by going to emergency mode and remounting /sysroot rw.

This can be undone by unsetting the menu_auto_hide grub2 option with grub2-editenv, such as:

[user@hostname ~]$ sudo grub2-editenv list
saved_entry=841c0cb68442436a89b4246c45e8c3a0-5.14.0-362.8.1.el9_3.x86_64
menu_auto_hide=1
boot_success=0
boot_indeterminate=0
[user@hostname ~]$ sudo grub2-editenv - unset menu_auto_hide
[user@hostname ~]$ sudo grub2-editenv list
saved_entry=841c0cb68442436a89b4246c45e8c3a0-5.14.0-362.8.1.el9_3.x86_64
boot_success=0
boot_indeterminate=0

Ansible RSS feeds

2023-12-05T04:45:00+00:00

I've looked for an rss feed link for the Ansible Community release (not ansible-core) that I usually install with pip, but could never find.

I should have looked on pypi.org first, from https://pypi.org/project/ansible you can find Release history tab, which gives this link that can be subscribed to:

https://pypi.org/rss/project/ansible/releases.xml

For ansible-core, visit either https://pypi.org/project/ansible-core/#history (the rss link being https://pypi.org/rss/project/ansible-core/releases.xml) or find https://github.com/ansible/ansible/releases.atom from https://github.com/ansible/ansible/releases.

SMR drives suck

2023-03-18T14:45:00+00:00

Never again will I buy any SMR hdds.

I cheaped out on 2 drives years ago and I've been regretting it ever since.

Weekend

2023-03-18T04:45:00+00:00

Project 1 for the weekend:

replace shorthand image references in my ansible/docker-compose playbooks from eg:

'traefik:2.9'

'docker.io/library/traefik:2.9'

I also need to change them in my Dockerfiles, yay.

They took our jobs! .. err Dilbert comics

2023-03-14T09:21:00+00:00

Dilbert.com now redirects to https://linktr.ee/scottadams, making my Dilbert comic scraper useless.

The scraper collected a total of 12382 comics over the past 6-7 years I was working on it. I had to save the last 2 available comics from archive.org.

So the collection has a grand total of 12384 comics, measuring 1490224578 bytes(1.4GiB), starting from the first one from 1989-04-16, to the last one from 2023-03-12.

PAM_FAILLOCK

2023-03-12T14:21:00+00:00

In the Sander Van Vugt RHCSA video rcsa_01_06_08.mp4 video(L6: Managing Users and Groups/Managing Password Properties), he mentions looking into the pam_tally2 module.

What is pam_faillock and how to use it in Red Hat Enterprise Linux?

The kb above says pam_tally2 has been replaced with pam_faillock.

Its man page can be seen at:

man pam_faillock

Finding a Repo's Github Release RSS link

2022-09-01T19:21:00+01:00

I've been following a few projects' releases over RSS to determine see new releases and when to upgrade docker images I use locally.

For some software, it's not worth it to upgrade straightaway to a new release(eg on a new v13.0 release). I like to see if there are any new bugfix releases afterwards in the coming days or weeks.

On Github.com, a project/repo's releases RSS link follows this format:

https://github.com/{username,orgname}/reponame/releases.atom

Rss for tag feeds are at:

https://github.com/{username,orgname}/reponame/tags.atom

I know for Docker you can/could use Watchtower to upgrade your software or send notifications when upgrades are available. I should look into watchtower anyway and its email notifications .

2022-08-08 Keep an eye on Gitlab storage

2022-08-08T22:15:00+01:00

As someone who uses Gitlab CI a lot, I ought to take a look at the following link soon.

https://docs.gitlab.com/ee/user/usage_quotas.html#namespace-storage-limit-enforcement-schedule

2022-08-08 Updating my docker images to multiarch builds

2022-08-08T22:00:00+01:00

I've been updating my dockerfiles on my several git repos across Gitlab and Github.

Some of them have working images now for armv7(32 bit arm), armv8(64 bit arm) and amd64(most desktops these days).

I only use armv7 and amd64 images for my raspberry pies and my pc-s …

2022-05-10 Change of email service provider

2022-05-10T22:00:00+01:00

I've changed my email service provider to Migadu from Google Gsuite.

After 2 years, I've found I paid too much for email hosting.

I also used storage provided by gsuite for backup purposes, fortunately never needed to use it.

Migration of my emails was easy.

Changed over DNS settings with terraform.

Job done.

2022-04-13

2022-04-13T00:00:00+01:00

New services to add

Yay, more things to keep track of!

Note taking

Right now my note taking for personal stuff is a mess between vscode md files, local files and some files on my phone.

Also something to sync them? Syncthing or just some service like onenote or evernote?

Local monitoring

I have no local monitoring services used, as I didn't feel them necessary for a desktop and a laptop.

I need something lightweight than runs on a pi 2 or 3.

Web page change monitor

Some service that watches for changes in web pages.

Something that notify me if it detects changes in search results on that page, instead of just straight html changes(which would be pointless).

Some projects to consider here are:

https://github.com/PipedreamHQ/pipedream

https://github.com/huginn/huginn

https://github.com/home-assistant

2022-04-12

2022-04-12T12:00:00+01:00

Just wonderful

Things are just great in 2022 so far in the UK, aren't they?

droneci-https

2021-05-29T12:00:00+01:00

Droneci and SSL

Finally managed to get Drone working properly with traefik v2 in docker.

I had to go back and forth while I managed to track down my issues, which were:

wrong hostname configured on the drone docker label parts
play with docker env vars while I get the oath2 redirect working between gitea, drone and traefik

SSL and redirection required me to change up config somewhat, which looks like this below:

note: it's from an ansible playbook

- name: Run drone docker-compose
  docker_compose:
    project_name: droneci
    state: present
    pull: yes
    remove_orphans: yes
    remove_volumes: yes
    restarted: yes
    remove_images: all
    debug: true
    definition:
      version: '2'
      networks:
        ci:
          external: true
      services:

        drone:
         container_name: drone
         image: drone/drone:1
         restart: unless-stopped
         hostname: drone
         networks:
           - ci
         environment:
           - DRONE_GITEA_SERVER={{ DRONE_GITEA_SERVER }}
           - DRONE_GITEA_CLIENT_ID={{ DRONE_GITEA_CLIENT_ID }}
           - DRONE_GITEA_CLIENT_SECRET={{ GITEA_CLIENT_SECRET }}
           - DRONE_RPC_SECRET={{ DRONE_RPC_SECRET }}
           - DRONE_SERVER_HOST={{ DRONE_SERVER_HOST }}
           - DRONE_SERVER_PROTO=https
           - DRONE_DATADOG_ENABLED=false
         labels:
           - "traefik.enable=true"
           - "traefik.http.routers.drone.rule=Host(`drone.sub.domain.tld`)"
           - "traefik.http.routers.drone.entrypoints=websecure"
           - "traefik.http.routers.drone.tls.certresolver=myresolver"
         volumes:
           - /var/lib/drone:/data

        drunner:
         container_name: drunner
         image: drone/drone-runner-docker:1
         restart: unless-stopped
         hostname: drunner
         ports:
           - "3000:3000"
         networks:
           - ci
         environment:
           - DRONE_RPC_PROTO=https
           - DRONE_RPC_HOST=drone
           - DRONE_RPC_SECRET={{ DRONE_RPC_SECRET }}
           - DRONE_SERVER_HOST=drone
           - DRONE_RUNNER_CAPACITY=2
           - DRONE_DATADOG_ENABLED=false
         volumes:
           - /var/run/docker.sock:/var/run/docker.sock

        traefik:
         image: traefik:2.2
         restart: unless-stopped
         container_name: traefik
         command:
           - "--providers.docker=true"
           - "--providers.docker.exposedbydefault=false"
           - "--entrypoints.web.address=:80"
           - "--entrypoints.websecure.address=:443"
           - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
           - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
           - "--certificatesresolvers.myresolver.acme.dnschallenge=true"
           - "--certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudflare"
           - "--certificatesresolvers.myresolver.acme.email={{ traefik_var_CF_API_EMAIL }}"
           - "--certificatesresolvers.myresolver.acme.storage=/acme.json"
           - "--global.sendAnonymousUsage=false"
         environment:
           - CF_API_EMAIL={{ traefik_var_CF_API_EMAIL }}
           - CF_API_KEY={{ traefik_var_CF_API_KEY }}
         ports:
           - 80:80
           - 443:443
         networks:
           - ci
         volumes:
           - /var/run/docker.sock:/var/run/docker.sock
           - /var/docker/traefik/acme.json:/acme.json

Somethings remain to be done

I'm just testing what needs to be fixed and tweaked now that drone runs on ssl.

A few days ago I only set up 1 repo to test, so it needed its webhooks updated to point to the https new url.

Also need to fix my test runner as it seems it can't connect to the drone server so no ci runs are performed. Great.

Finally got the runner working with the server, had to configure the runner to use:

DRONE_RPC_PROTO=http

instead of

DRONE_RPC_PROTO=https

as above.

I also need to decide how many and what repos I want to locally test.

Also want to know whether drone can run my ansible roles with molecule.

Also I'm not a fan of mounting the docker socket into any containers, but these are running in their own virtual machine at this time.

Also want to try to upgrade to traefik v2.4.

Also had to figure out an issue with drone not being able to connect to my gitea server anymore.

Turned out it had some routing issues, and I have no idea what messed up the routing table inside the drone vm as only ansible was modifying it.

Useful reminder:

ip route add 192.168.123.0/24 via 192.168.122.1

Another issue that came up while trying to get an ansible role working was the concept of trusted repos in droneci.

I forgot to add the docker socket into the pipeline, so my droneci looks awful like this:

---
kind: pipeline
type: docker
name: default

steps:
  - name: local build
    image: qwe1/dind-ansible-molecule:2.9-root
    volumes:
      - name: dsock
        path: /var/run/docker.sock
    commands:
      - ls -lah
      - pwd
      - mkdir -p ~/.ansible/roles && cp -aR "$(pwd)" ~/.ansible/roles/richardskumat.ansible_role_user
      - molecule test

volumes:
  - name: dsock
    host:
      path: /var/run/docker.sock

I didn't specify an admin user for drone when setting up, so when setting up an ansible role to be tested in drone, I was getting the following error:

default: linter: untrusted repositories cannot mount host volumes

So added the following env var to my drone container and it was happy afterwards:

DRONE_USER_CREATE=username:user,admin:true

username:user in my case had to match the gitea user I was using.

droneci

2021-05-23T12:00:00+01:00

Droneci

I used to run gitlab a while ago, but it was way too resource heavy for my requirements.

So far a long time I didn't run any local CI.

Today, I started playing around Droneci.

I got it working after some time in its own Debian vm.

The server runs inside docker and the runner also runs in the same vm.

My local git server is a gitea instance, running also in docker.

Link

Link for setting up drone with the gitea instance:

drone docs gitea link

Issues

I setup the instance of Drone to run on http.

Which was fine, until I couldn't get the oauth2 auth working between Gitea and Drone.

Turns out, I setup the Redirect URI as the base http url instead of http://IP/login.

Finally managed to get it working, then activated my blog's git repo, which is a pelican website.

Then setup the drone runner in docker, which was simple.

I created a drone.yml file in my git repo, and pushed it, but nothing happened.

I looked at my local drone webpage, however it wasn't registering any activity from the git repo.

I looked at the git repo's settings and the webhook was configured to post at HTTPS://IP/what&ever.

Changed that to http://IP/what&ever and it started working!

So in my droneci.yml, I configured one step only, which was:

make publish

using my pelican docker image(qwe1/docker-pelican).

The makefile in the repo contains a step which clones the pelican plugins into the git working directory, because building my website requires some plugins.

Anyway, at this step I got permission denied errors.

The user inside my pelican:latest image is uid 1000.

Turns out, drone clones the source repo as root.

I found this issue on their forums:

post

I couldn't find a way to chown/chmod the cloned git directory to uid 1000 from root inside the droneci run, because I was already running as uid 1000 inside the build, so even trying I got permission denied errors.

I probably could have used a new tag inside my image that has sudo and add uid 1000 as a passwordless sudoer but that seemed excessive for this use case.

So instead I made a new tag which runs as root(yay, great) and my test ci managed to build my website on my 23rd try!

What a waste of time.

To-do?

Add a proper hostname to Drone configuration and start using https and hostnames instead of http and ip-s.

Spring cleaning

2021-05-23T07:30:00+01:00

Idea of cleaning up projects

I've got like 35 or 36 git projects, both public and private that need some attention.

In an earlier post, I said I'd do maintenance on them, but that led me to not even trying anything new.

If I want to learn more, I have to dig out new stuff, install it, and play with things.

I'm talking mainly about kubernetes and also newer ansible versions.

I also want to see which projects of mine are worth working on, which to be honest I don't think any of them are.

My favourite git repo is still my Dilbert repo, working to this day somewhat wonky with updates over the years.

It'd be nice to have all the dilbert downloaded comics locally, allegedly which that script does.

I had the idea of rewriting the dilbert scraper in python years ago, but has also been on hold forever.

I get these big ideas or thing I want to study or look at/into, and after I come home from work, I lose all my interest in learning or doing anything really.

I remember my French teacher telling me that I should have used my high school years learning new languages because when I'll grow up I won't have the time or energy to pursue learning too much stuff.

It turns out she was right all along.

The point of this post is self-reflection somewhat, that I need to do things I want and need and also learn new skills I want to be more valuable and get ahead with my life.

Sadly I've been stuck for too long.

Will that change? Who knows.

Black friday

2020-11-27T18:00:00+00:00

Buy buy buy

As a good little consumer I haven't bought anything for black friday.

I planned on getting a cheap xiaomi phone again, but nothing's in sight yet.

I also thought about building a new pc, but why would I?

The last desktop I built was about 4 years ago, and it's still OK. Not great anymore, but still does the job.

I noticed I haven't posted anything in 4 months, but with this lockdown happening nothing's worth writing about, which is pretty fscking depressing.

A nifty find one liner

2020-07-16T11:00:00+01:00

What this does

This find command finds and moves all directories to /dest/dir that matches the 'dirnamepattern'.

'dirnamepattern' could be aws, go or tmp, or whatever.

Actual command

find . -type d -name '*dirnamepattern*' -exec mv -t /dest/dir {} +

Test before try

I'm not sure if there's a dry run to find's exec, but the above command can be modified to output matching directories with a simple echo variation:

find . -type d -name '*dirnamepattern*' -exec echo {} +

Debian Jessie(8) end of life

2020-07-15T20:00:00+01:00

Special release

Jessie was my first version of Debian I've installed on my acer laptop back in 2015.

Nowadays, all my things run Buster, but it's still awful to think how fast time goes and Debian releases come and go.

An issue I ran into when creating a new Gitlab repo with terraform

2020-07-02T20:45:00+01:00

Gitlab + Terraform

I try to manage my gitlab repos via terraform lately.

Yesterday, I created a new project called ansible-role-chrome.

This was done with terraform.

The project has a gitlabci file that runs molecule test in two Debian 9&10 images of mine.

Today I got an email saying that my CI failed on this new repo, whereas I had molecule tests working locally.

I found out that when creating the repo with terraform apply, shared runners were disabled for this repo.

After looking at the gitlab project resource docs, I saw there's a shared_runners_enabled argument.

My resource block of course didn't contain the following line:

shared_runners_enabled = true

The whole block now looks like this:

resource "gitlab_project" "ansible-role-chrome" {
    name = "ansible-role-chrome"
    description = "this is the ansible-role-chrome repo"
    visibility_level = "public"
    request_access_enabled = false
    default_branch = "master"
    merge_requests_enabled = true
    issues_enabled = true
    wiki_enabled = false
    shared_runners_enabled = true
}

So now I've got shared runners enabled for this project and now of course my molecule tests passed. Yay.

Some of the docker images I use

2020-06-26T13:15:00+01:00

My own Docker images

I maintain a couple of docker images for myself on Dockerhub(about 9 active) at the following link, under user qwe1:

https://hub.docker.com/u/qwe1

Other images used

I use many images from linuxserver.io, such as sonarr, radarr, jackett and jellyfin.

Bazarr is also great to get subtitles and to be honest, it works OK most of the time.

I also like to use mariadb:10.4 and traefik:2.2.

It took me some time to get a working config for the new traefik 2.x docker image.

I had to ditch my previous method of using a toml file and some ansible hackery.

Nowadays, I just run traefik with a long command line from a docker compose file.

A traefik example from an ansible docker-compose file:

        traefik:
         image: traefik:2.2
         restart: unless-stopped
         container_name: traefik
         command:
           - "--providers.docker=true"
           - "--providers.docker.exposedbydefault=false"
           - "--entrypoints.web.address=:80"
           - "--entrypoints.websecure.address=:443"
           - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
           - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
           - "--certificatesresolvers.myresolver.acme.dnschallenge=true"
           - "--certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudflare"
           - "--certificatesresolvers.myresolver.acme.email={{ var1 }}"
           - "--certificatesresolvers.myresolver.acme.storage=/acme.json"
           - "--global.sendAnonymousUsage=false"
         environment:
           - CF_API_EMAIL={{ var1 }}
           - CF_API_KEY={{ var2 }}
         ports:
           - 80:80
           - 443:443
         networks:
           - desktop_network
         volumes:
           - /var/run/docker.sock:/var/run/docker.sock
           - /var/docker/traefik/acme.json:/acme.json

Some other images I use are php:7.3-apache-buster for 2 dokuwiki instances.

I also have a gitea instance running with mariadb database.

I like to wait a bit before upgrading my gitea instance to see if there are issues with a release.

I also run a pihole instance with host networking on a laptop, and it functions OK as a dns and dhcp server.

I made a mistake by leaving my pihole tag at latest, so when the latest release came out(v5.0), I accidently got upgraded to 5.0.

I haven't issues with the new version, it's just I wanted to wait a few weeks for issues to be ironed out.

I also have a draw.io image running, image:

fjudith/draw.io:alpine

I run a local version of js dencoder, which is useful for decoding some garbage URLs found on the web with httpd:alpine image.

What's special about the httpd:alpine image is that the default web root is /usr/local/apache2/htdocs, instead of /var/www/html as found in the debian based httpd images.

This caught me as a surprise when I tried running the alpine image the first time.

How I try to track docker images tags and releases

I am subscribed to a bunch of Github RSS feeds and frequently review new releases for any issues before doing any major upgrades.

Fortunately, linuxserver.io has Github repos for docker images and their git tags match their docker tags.

This is all manual and can be tedious, but I try to to avoid using latest tags where possible.

pgrep

2020-06-20T14:30:00+01:00

Pgrep

I've got a few shell scripts in a private repo and I recently wrote another to kill my vlc instances because it seems to hang up sometimes on Debian 10 when playing files from my junk hp microserver.

So I wrote something idiotic like this to kill vlc:

kill -9 $(ps auxf | grep '[v]lc' | awk '{print $2}')

Then comes my Gitlab ci result saying I'm an idiot and should use pgrep instead.

So now I've got something like this which the linter is happy with:

kill -9 "$(pgrep -u "$(whoami)" vlc)"

Improvements to linting

I need to look into how I could do linting before making git commits from inside vscode, where I code mostly.

I've found the following extension:

https://github.com/magicmark/pre-commit-vscode

https://marketplace.visualstudio.com/items?itemName=MarkLarah.pre-commit-vscode

I'll have to test it how well it integrates with latest vscode.

Recent things

2020-06-19T19:30:00+01:00

What's been going

I haven't posted in 2 months, as I was focusing on work.

In the meantime, I was focusing on maintaining my private code repos and thinking about ways of what I could improve in my public and private projects.

What I'm going to do

I've got nothing specific to write here, it's just I feel that stuff I already have running at home are running OK, not great but not bad either.

What I could improve on is automating more things(like bare metal reinstalls) and setting up a centralized logging system.

Simple ansible logrotate script

2020-04-18T10:00:00+01:00

Ansible logs

Ansible logs details of playbooks runs into files defined with the:

log_path = /path/text.log

variable if defined in the ansible.cfg file.

For my simple personal use cases, I turned logging on and for a while did nothing with the logfile.

After several months, the first log file turned into several thousand lines long.

I wrote a fairly simple shell scrip to do basic logrotating with the ansible log file.

I uploaded this script to a Github gist, with the raw script link being:

ansible-log-rotate.sh

There's also a Gitlab copy of this script:

ansible-log-rotate.sh

Cron job

I run a cron job monthly, which looks like this:

#Ansible: monthly ansible logrotate
@monthly bash $HOME/path/scripts/ansible-log-rotate  > ${LOG_FILE_PATH}/ansible-logrotate-$(date +\%Y-\%m-\%d-\%H-\%M-\%S).log 2>&1

This way the log rotate scripts runs every month, on the first of month at midnight.

I've been running this script with cron for a few months, and it resulted in the following tar.gz files, which are readable with zcat:

ansible.log             ansible.log-2019-12-01-00-00-01.tar.gz
ansible-log-201907310913.tar.gz     ansible.log-2020-01-01-00-00-01.tar.gz
ansible.log-2019-09-01-00-00-01.tar.gz  ansible.log-2020-02-01-00-00-01.tar.gz
ansible.log-2019-10-01-00-00-01.tar.gz  ansible.log-2020-03-01-00-00-01.tar.gz
ansible.log-2019-11-01-00-00-01.tar.gz  ansible.log-2020-04-01-00-00-01.tar.gz

Of course, a lot better way would be to ship these log files to a central log server, but for a personal use case I find these adequate.

Gmail settings for Thunderbird with a Gsuite account

2020-04-15T13:00:00+01:00

My issue with sending emails

I've been using Thunderbird(TB) for 3-4 years with a couple email addresses and it has served me well so far.

Last year, I've moved my email address over to Google's Gsuite service and when sending emails sometimes I'd get an error that TB could not login to Gmail.

So then I'd login and enable the "unsecure" "Access to less secure apps" to ON.

However, Google turns this off if it thinks you're not using it, eg not sending emails all the time. And there's no option to change this behaviour.

I was getting the same error as the poster on the mozilla forums.

Namely, the following error:

error message from send: Sending of the message failed.
Failed due to unexpected error 80004005. No description is available.
The message could not be sent using Outgoing server (SMTP) smtp.gmail.com for an unknown reason.
Please verify that your Outgoing server (SMTP) settings are correct and try again.

What I've tried

On the linux version of TB, I went to Edit/Account Settings and had a look at my primary email's "Server settings".

The following is how the incoming settings are:

Server Type: Imap
Server name: imap.gmail.com
Port: 993
Username: whatever@mydomain.tld

Security Settings:
Connection Security: SSL/TLS
Authentication method: OAuth2

All of this seem to have continued working since my email migration from OVH's email service.

On the Outgoing Server (SMTP) part of TB settings, I had my Gsuite labelled gmail settings configured as follows:

Description: Gsuite
Server name: smtp.gmail.com
Port: 465

Security & Authentication:
Connection Security: SSL/TLS
Authentication method: Normal Password
User name: whatever@mydomain.tld

As per the first poster on the link above, I've changed Auth method to Oauth2.

Then proceeded to turn off "Access to less secure apps".

So now my updated SMTP settings look like this:

Description: Gsuite
Server name: smtp.gmail.com
Port: 465

Security & Authentication:
Connection Security: SSL/TLS
Authentication method: Oauth2
User name: whatever@mydomain.tld

So now I can happily send messages without getting the dreaded "unexpected error 80004005".

Relevant links

G suite updates blog

theregister's take on the above announcement

k9-mail oauth2 support

k9-mail oauth2 pr

Terraform git repo removal with Gitlab provider

2020-04-04T10:30:00+01:00

New changes on Gitlab

I haven't deleted a git repo on Gitlab for some time, so I didn't know about a "feature" Gitlab created.

Now on the public Gitlab.com instance, if you try to remove/delete a repo, it defaults into an archived form for 7 days before it's removed.

Here's the Gitlab issue about this:

gitlab-org/gitlab/issues/32935

Also if you try to delete a fork of a repo, it also goes into read-only mode for 7 days as per this issue.

Other relevant issues

Gitlab/Immediately remove a soft-deleted project

Github,TF/Gitlab introduced soft delete of projects and groups

Github,TF/Project Roadmap

Working with the Gitlab provider for Terraform

2020-03-18T16:00:00+00:00

More Terraform work

I've got a few git repositories on Gitlab.com that have been manually made over time, with different configurations, ci schedules and purposes.

I consider them to be a giant mess at this point, so with some of my recent work I wanted to reorganise all of it for the better.

Terraform(TF) with Gitlab

Since I like both TF and Gitlab, I thought it'd be a great time to make them work together.

TF has a Gitlab provider that I could use.

After a lot of trial and errors, I managed to use a few features of the Gitlab provider for my needs and have posted some sample code to a Gitlab.com repo for others to view.

What's in the repo

There's some sample terraform(.tf) code, some shell utility scripts and also some Ansible code I used for templating some terraform resource configuration blocks.

I learned a lot of new things doing this for myself and am happy with the outcome.

Molecule tests

2020-02-16T16:00:00+00:00

Update on 2020-02-23

Turns out this doesn't work at all, all the time.

Some notes for myself regarding molecule testing

I recently removed my everyday user from the docker group, so programs running under my user can't run arbitrary docker commands anymore. Wooho.

Sudo with docker instead

The original post I had here wasn't working after all, so I decided to remove it.

I don't know how I got it to work.

Finished working on my Docker ansible role for now

2020-02-02T16:00:00+00:00

Merged some commits in my docker role

After some work on the docker ansible role, I merged my own commits from dev branch to master.

The list commits can be found here with comments in the git log detailing the changes.

Here's the Gitlab.com link for commit history.

Of course, I wouldn't recommend using my own role for anyone but myself, instead I would point people to Jeff Geerling's docker role on Github.

His role has a lot more work, commits and features added.

New Docker images

2020-01-12T16:00:00+00:00

Two new docker images

I'll be building some of my own CentOS and Debian images for molecule testing, hopefully to cut down on molecule testing time.

These are:

Debian

CentOS

Source code to repos

docker-debian

docker-centos

The build logs are public as well.

Now they're done

At least they're now usable for my Ansible & Molecule tests.

These images are built from upstream Debian/CentOS images and Gitlab CI pushes them to Dockerhub and Gitlab registries.

Update on my Docker Ansible role

2020-01-12T15:30:00+00:00

New year, old projects

People love working on new things all the time, however I've got lots of repos on Gitlab.com that I want keep updated for myself.

One of those projects is this blog, which source code I've taken private permanently.

However, other projects that I've recently updated are my docker images' projects and some of my Ansible roles.

Why update my own Ansible roles that I only use myself

Times change and new distros come out.

Plus most of my roles are just bare enough to get my own personal systems running, however they wouldn't get any praise in a work environment.

I've learned lots from Jeff Geerling's Ansible roles on Github, also from https://github.com/weareinteractive.

I'm currently trying to update my Docker ansible role to make it a bit better.

In hindsight, I should have just used Jeff Geerling's docker repo.

What's being updated in my Docker role

I'm trying to add Centos support and improve some other aspects of it.

I want to add a new task and make it configurable whether to add/remove any users to/from the docker group.

Here's a list of issues for this role.

I'm currently working on the dev branch.

I've posted an update in the following post.

Day-off project: Retropie

2019-12-11T21:00:00+00:00

Madness in boxes

So around black friday I was running through my card board boxes that have all sorts of cables and other PC accessories.

I wanted to a buy a new mouse for my desktop, because the previous one's scroll wheel was pushing me into madness.

But I managed to find a usb mouse that was from ASDA and it was still in its original box.

Here, I got the idea that I wanted to put my other boxed/packaged things to use as well.

I've two Raspberry Pis from years ago and I used them a lot for years.

Then I put them into these boxes, because managing four x64 machines were a lot simpler after moving,than mixing armhf and x64 arches.

I'm mostly referring to building my own docker images and starting to write my own ansible playbooks.

Focusing only on one arch at the time seemed to be a better idea.

Raspberry Pis

I wrote a post on wanting to make these Pis useful again here.

As update to that previous post, I managed to find a functioning SD card for the Pi2.

Where I got the idea to try out Retropie

One day at lunch break at work, I was talking to my colleagues about having a Nintendo as a kid, and remembering playing a tank game.

Turns out they also remember playing this game, but nobody could recall what is was called.

I had talked about this game to some other people for more than years, but nobody knew the name of this game.

So I started searching on Google, and it turns out my favourite game on Nintento was called Battle City.

I found it funny enough, that it took me about 2 minutes of effort to find something I was looking for about 10-20 years.

I found a Nintento NES emulator called Nesoid for Android on Fdroid, however the last uploaded version is years old.

However, it still ran on Android 9/Miui 11. I must say that a touch screen is no way to play any NES games.

Retro games on modern Raspberry Pi 2b

So the following day, I burned Retropie 4.5.1 to an SD card and slot it into the Pi 2.

The following example with dd should work, when replacing of=/dev/sdx with the correct hard drive location.

sudo dd if=$HOME/Downloads/retropie-4.5.1-rpi2_rpi3.img of=/dev/SDZXXZZX;sync

My old pi booted up and I manually started messing with Retropie's Emulation Station.

Since I never used Emulaton Station/Retropie, the interface was kind of strange at first.

I also don't have any controllers, so for the time being I'm setup with a keyboard.

I configured WASD for up, left, down and right.

Then right arrow for A and down arrow for B.

F for start, and G for select.

Game ROMs

Well, this is an interesting topic, and I'm sure people will have their own sources to get game ROMs from.

Some links to note:

My setup with Retropie

Here are the items I have for my Retropie:

Raspberry pi 2b
8GB Sd card
RTL8188CUS 802.11n WLAN Adapter
USB keyboard
Monitor with HDMI/VGA converter
5v@2.1A micro-usb charger

Most of these things can be bought on Amazon/Asda/B&M/Ebay.

What's interesting is the free memory reported on the Pi is ~750MB and I thought it had 1GB memory.

Then I found this page on the Github wiki , which explains why.

Using Ansible on Retropie

Since Retropie's just lots of packages on top of Raspbian Stretch, it means I could run Ansible against it.

Putting Windows 10 on UEFI bootable USB

2019-12-02T19:00:00+00:00

Windows 10 on hardware

I need Windows 10 today to try to root my Xiaomi Android phone, but first I have to unlock the bootloader, which is the most annoying process I have ever been through.

I tried many programs found on XDA-developers or /r/xiaomi, but none of them seem to work.

I also tried to do USB passthrough my phone to a Windows vm, but that didn't work out.

I've got no choice, but to try Xiaomi's own software to do the unlocking.

What's needed for old school USB booting

Format the USB drive with GPT partitioning.

On Debian, there's gparted which makes this easy.

Once the USB drive is selected, select:

Device/Create Partition Table/gpt from "Select new partition table type" then apply.

From there, create a new fat32 partition.

With my USB drive, I let the fat32 to use 100% of the drive's space.

Then find the Windows ISO, mount it locally, and copy all contents to the USB drive's folder.

On the target machine, set BOOT mode to UEFI native(not hybrid).

Test it if it works

It did! The laptop booted nicely, and the installation worked great.

Putting old Raspberry Pis to good use

2019-11-29T02:00:00+00:00

Update to this post

I put my Raspberry Pi2 to use by installing Retropie on a functioning SD card.

The issues I mention below were most likely due to failing/corrupt sd cards.

Why use a Raspberry pi at all?

They're great devices for learning a different operating system when all one is used to are either Windows or Apple's Osx.

I got my first pi in 2015, a pi 2b.

My main machine was an Acer laptop at the time running Windows 7.

I remember using putty and Win32 Disk Imager(had to look this name up) to write the Raspbian Wheezy image to the sd card I had at the time.

They're also great to experiment on, install this, enter this shell(bash, python) or whatever.

If you mess up the OS, you can always just write a new Raspbian image to the card and restart.

Times change

Over time, I moved to Debian on my acer laptop and started using Debian Jessie on it.

Then in 2016, I got a desktop and ran Debian Sid for a while. Then testing, and stable when Stretch got released in 2017.

Kernel 4.9+ was really required to run the AMD polaris cards I bought in late 2016 and early 2017(rx 460+480).

After using the Pi2b+ for a while, I got a second Pi3b, also running Raspbian.

They ran a couple of things for me in my lan between 2015 - early 2019, running Squid, Nginx, Dokuwiki, Dnsmasq and a couple other things I can't remember.

I was really satisfied with both Pis, however I was dumb enough to buy the starter kits for both on Amazon.

I ended up spending a lot more money than I should have.

Also turns out the Sandisk Ultra sd cards I got were not so ultra great after all.

Putting them to good use again in late 2019

Because it's black friday tomorrow, I've been thinking about buying stuff that I would end up regretting anyway.

Then it occurred to me, why not use something that I already have.

I already have 4 Intel machines on my Lan doing various things, but I have nothing to monitor them.

So I thought, using one or both of the Pis could be used for something great.

Rummaging through boxes and cables

To get the 2 Pis working again, I'd need:

2 ethernet cables
2 micro-usb power cables(5v@2.5A are recommended)

I had no issues with getting ethernet cables, I've got plenty of them.

However, I was having problems with finding power cables that are good enough for the Pis.

Reading around on various forums, sites and raspberrypi.org, it seems people recommend using a power cable that supplies 5v@2.5A.

I found one of the PSU that with one of those starter packs, and it was rated 5.0v@2.0A(Vilros AC adapter, model: XHY050200LBCW).

The second PSU I've got that would be good for a Pi is my phone charger.

So temporarily, I powered up both Pis, started them both with Raspbian Buster.

My Pi3 works fine with Buster and I'll be using to monitor my other machines.

My Pi2 had several issues.

With two SD cards, load average would spike to 10-12 while doing a apt-get update & upgrade combo.

On the third card, on first boot it would fail to resize the second partition on the sd card.

On the second boot with the third card, I ran raspi-config to try to resize the partition, however it would also fail.

It would error out, citing a syntax error on line 151.

Since raspi-config is a bash script(2048 lines on buster), I had a look with nano:

sudo nano -l /usr/bin/raspi-config

-l shows the lines with nano.

The relevant section from raspi-config:

 150   LAST_PART_NUM=$(parted /dev/mmcblk0 -ms unit s p | tail -n 1 | cut -f 1 -d:)
 151   if [ $LAST_PART_NUM -ne $PART_NUM ]; then
 152     whiptail --msgbox "$ROOT_PART is not the last partition. Don't know how to expand" 20 60 2
 153     return 0
 154   fi

By looking at this part of the script, I tried to manually run the shell that defined $LAST_PART_NUM.

So when running:

parted /dev/mmcblk0 -ms unit s p | tail -n 1 | cut -f 1 -d:

I would end up getting segmentation faults with parted every time.

There would also be some other issues with the Pi2 that I just ended up shelving it.

To-do for Pi3

What I've already done:

erase the pi user
setup my user
disable root password and ssh login
ran my ansible playbook against the pi3

Figure out what monitoring tool to use.

I'll probably try out librenms or zabbix.

If I end up with librenms, I'll have to add snmp to my machines and set up snmp communities, probably with v3.

If zabbix wins, then all my other nodes will get zabbix agents.

Trying out old CentOS images

2019-11-24T02:00:00+00:00

Trying out the oldest available CentOS images

I use Debian all the time, however I also like CentOS for a couple of reasons.

It's nice that Yum/DNF has transaction history and the ability to undo/redo a certain installation/removal of packages.

What is however great is the longer(~10 years) supported life time of a release, compared to Debian's 3-5 years.

CentOS 2.1

This is the first release I could get my hands on.

From the CentOS vault, I had tried to download the ISO files to no avail.

However, these mirrors are linked from the 2.1 download page:

USA:
http://archive.kernel.org/centos-vault/
rsync://archive.kernel.org::centos-vault/

Europe:
http://mirror.nsc.liu.se/centos-store/
rsync://mirror.nsc.liu.se::centos-store/

http://linuxsoft.cern.ch/centos-vault/
rsync://linuxsoft.cern.ch/centos-vault/

I actually used the nsc.liu.se links from above.

Both disc1 and disc2 images are needed for the installation to complete.

I also pulled down the MD5SUMS file to check the ISO-s I got were OK.

CentOS 2.1 on KVM with virt-manager

I've started a local install with disc1.

Then in the OS type field entered linux, the field "Include EOL OS-s" were already ticked.

From there, all I had to was find RHEL 2.1.

Selecting RHEL 2.1 will bring up some default settings which worked OK for this test VM.

The defaults for the disk & cd bus were ide and the network adapter model defaulted to e1000e.

The installation was fairly straight-forward, however using the mouse inside the console was finicky, I ended up using the keyboard(space, tab and enter are great here) instead.

While installing the packages, the installer asked for disk2.

I forgot to add a second disk drive before starting the installer, so what I did was swap the ISO file to disc2 from virt-manager's "Show virtual hardware details" section.

Then the installer carried finished and all that was left to reboot the VM.

Connecting to CentOS 2.1

By default the Debian ssh client couldn't connect to SSH 3.1 on C2.1.

The following command let me connect:

ssh -c aes128-cbc -oKexAlgorithms=+diffie-hellman-group1-sha1 root@192.168.122.16

I tried using ssh-copy-id for C2.1, but only using ~/.ssh/config as a workaround I managed to get this working. Relevant ssh-config snippet:

Host c2
    Hostname 192.168.122.16
    User root
    Port 22
    Identityfile ~/.ssh/id_rsa
    Ciphers aes192-cbc
    KexAlgorithms=+diffie-hellman-group1-sha1

Then all it was just ssh-copy-id c2 and enter the password.

To enable the use of yum, I had to swap the repos in /etc/yum.conf to the mirror where I got the iso from.

In my case, /etc/yum.conf looks like the following:

[root@localhost root]# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest

[base]
name=CentOS-2 base
baseurl=http://mirror.nsc.liu.se/centos-store/2.1/i386/CentOS/

[updates]
name=CentOS-2 updates
baseurl=http://mirror.nsc.liu.se/centos-store/2.1/updates/

[extras]
name=CentOS-2 extras
baseurl=http://mirror.nsc.liu.se/centos-store/2.1/extras/

This let me update all the packages on the system to latest.

[root@localhost root]# yum -y update
Gathering package information from servers
Getting headers from: CentOS-2 base
Getting headers from: CentOS-2 extras
Getting headers from: CentOS-2 updates
Finding updated packages
Downloading needed headers
No Packages Available for Update
No actions to take

Now my Centos 2.1 VM is so secure!

Now let's enable and start httpd, because what could go wrong?

What's interesting is that the package used to be called apache, but the service was already called httpd like it is nowadays.

[root@localhost root]# service httpd start
Starting httpd:                                            [  OK  ]
[root@localhost root]# chkconfig httpd on
[root@localhost root]# chkconfig --list httpd
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
[root@localhost root]# httpd -v
Server version: Apache/1.3.27 (Unix)  (Red-Hat/Linux)
Server built:   Jan 16 2008 12:08:50

Here are some screenshots:

Centos 3.9

Centos 3 originally was released in 2004/03/19 as 3.1.

I grabbed both i386 and amd64 versions of 3.9 from the liu.se mirror.

The installation was a bit simpler than doing Centos 2.

To connect with SSH, I've set the following settings in ~/.ssh/config, which is a lot similar to what I have for c2:

Host c3
        Hostname 192.168.122.137
        User root
        Port 22
        Identityfile ~/.ssh/id_rsa
        Ciphers aes256-cbc
        KexAlgorithms=+diffie-hellman-group1-sha1

Afterwards, I add my SSH keys to my C3 vm with ssh-copy-id and I'm in!

There were 4 commands required to setup yum to work properly today.

First making a backup of /etc/yum.conf. This is just optional.

cp /etc/yum.conf{,.bak}

Second changing the repo urls in /etc/yum.conf to the mirror I use.

sed -i 's#mirror.centos.org/centos#mirror.nsc.liu.se/centos-store#g' /etc/yum.conf

sed -i 's#$releasever#3.9#g' /etc/yum.conf

I was surprised to see '#' to work for gnu sed 4.0.7.

Last, yum was missing a gpg key (warning: rpmts_HdrFromFdno: V3 DSA signature: NOKEY, key ID 025e513b), which has to be imported:

rpm --import http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-3

I also tried getting httpd working on 3.9, here's how it went.

First, the package name has changed to httpd instead of apache.

The service and chkconfig commands are the same as for 2.1.

Also, apache was at version 2.0 at this time.

However, for this VM, I forgot to add http for to be allowed during installation, so had to update the firewall:

iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
service iptables save

Compared to Centos 2, v3 didn't require a second cd to finish the installation.

A couple screenshots for Centos 3:

Centos 4

Using the 4.9 installation media, the installer process is better than that of 3.

To install C4, the first ISO("CentOS-4.8-x86_64-bin1of5.iso") was enough.

A couple lines of sed were required to get yum working:

cp /etc/yum.repos.d/CentOS-Base.repo{,.bak}
# uncomment mirror list
sed -i 's/mirrorlist=/#mirrorlist=/g' /etc/yum.repos.d/CentOS-Base.repo
# use baseurl instead
sed -i 's/#baseurl=/baseurl=/g' /etc/yum.repos.d/CentOS-Base.repo
# change centos mirror to lie.se mirr
sed -i 's#mirror.centos.org/centos/$releasever#mirror.nsc.liu.se/centos-store/4.9#g' /etc/yum.repos.d/CentOS-Base.repo

Getting httpd to run was the same process as it was for CentOS 3.

[root@localhost ~]# chkconfig httpd on
[root@localhost ~]# service httpd start
Starting httpd:                                            [  OK  ]
[root@localhost ~]# httpd -v
Server version: Apache/2.0.52
Server built:   Oct 20 2011 14:31:42

Screenshots:

Migrating from OVH Pro Email to Gsuite

2019-11-19T21:00:00+00:00

Migrating email from one service to another

I'm currently working on migrating from OVH's pro email service to Gsuite.

Experience with OVH's mail service

My monthly cost with OVH's service has been £2.49 since last year.

I've started using Mail Pro since end of September last year.

From my experience, it has been very reliable, however I've been having email delivery issues to Microsoft and Gmail email addresses.

Also, I couldn't find a way to setup DKIM/DMARC with this service.

However, I've had the following SPF record, but I'm not sure how much it helped with email delivery rates.

v=spf1 a mx include:mx.ovh.com -all

My reasons to migrating to Google Gsuite

Well, email delivery is the biggest one.

I rarely send emails but I got the feeling over the past year that people haven't been receiving/reading them.

Second, I believe I can setup DKIM/DMARC records so this will help with email delivery.

Third, is having access to Gsuite's presumably unlimited Google drive service, even with one user. Who knows how long this will last.

Fourth, I'd like to try something new.

Fifth, it's the experience with Gsuite. I've seen people mentioning on different IT news sites/forums of using Gsuite. By using it personally, I might gain valuable experience in a future role.

The only two downsides I can think of are:

increased monthly costs
hosting personal email with Google

Migration experience for a personal/single user domain

First, I had to add a txt record to my DNS zone at Cloudflare to prove ownership of my domain.

After, I've done that, I had to start adding users. The 14 days trial period allows I believe adding 10 users if needed for testing.

Since it was just for me, I've already had my user setup.

I had a gotcha here however.

I've got a different Google account with my primary email address which is for testing Google Cloud's trial year with the $300 credit(~£226 at the time).

Since my email was already in use in Google's system, I had to use another email for the initial account creation with Gsuite.

So instead of my old/regular email of whatever@domain.tld, I had to use whatever1@domain.tld.

Fortunately, in the admin panel/user settings I could setup email aliases, so my old address will be an alias of my new address.

I'm still exploring the admin control panel, so that will some take time getting used to.

Imap on Gmail

I use Thunderbird(TB) 68.x on Debian 10, so I need IMAP access to my email address.

I had to turn on IMAP access for the domain, as per kb105694.

Then enable IMAP access for "less secure devices" in Gmail.

I added a "New account" in Thunderbird, with the Gmail imap and smtp records.

Then TB opened up a new window with a Google Oauth page asking me to grant Thunderbird access to my Gmail account, which I did.

Before migrating emails over, I kept my old email address at OVH in TB.

Made 3 local folders for backup purposes.

One for the main inbox, one for sent and for an archived folder.

Migrating 5000+ emails

I've got over 5000 emails gathered over the last few years.

The first thing I've tried was trying to copy from local folders to Inbox, Sent and Archived folder inside Thunderbird itself.

However, this turned out to be unreliable with Gmail because only a fraction of my emails would be uploaded to Gmail. I'd get "Connection timed out to imap.gmail.com" error messages.

Then I've tried the Data Migration wizard inside the Gsuite admin panel, however this turned out to be way too slow.

I aborted the Migration after around 6%.

Then I remembered Imapsync from Gilles Lamiral.

I also remember that he had a docker image up at gilleslamiral/imapsync.

I've pulled down his image, then ran an instance of it and put into background:

docker run --name im -d -t --rm gilleslamiral/imapsync sleep infinity

Then attached into the container called im inside a tmux session:

docker exec -ti im bash

Once inside the container, I started the imapsync process with:

imapsync \
    --noreleasecheck \
    --host1 pro1.mail.ovh.net --ssl1 --port1 993 --user1 email@domain.tld --password1 hunter2 \
    --gmail2 --user2 email1@domain.tld --password2 hunter3

This resulted in the following:

Host1: Expunging folder archive_folder
ETA: Tue Nov 19 19:00:33 2019  1 s  1/5799 msgs left
++++ End looping on each folder

Folders sizes after the synchronization.
You can remove this foldersizes listing by using  "--nofoldersizesatend"
...

Host1 Nb folders:                    16 folders
Host2 Nb folders:                    14 folders

Host1 Nb messages:                 5798 messages
Host2 Nb messages:                 5709 messages

Host1 Total size:             289009755 bytes (275.621 MiB)Host2 Total size:             200927369 bytes (191.619 MiB)                                                                                                                           [0/1889]

Host1 Biggest message:         20405072 bytes (19.460 MiB)
Host2 Biggest message:         27913861 bytes (26.621 MiB)

Time spent on sizing:        33.8 seconds
++++ Statistics
Transfer started on                     : Tue Nov 19 17:32:36 2019
Transfer ended on                       : Tue Nov 19 19:01:07 2019
Transfer time                           : 5311.1 sec
Folders synced                          : 16/16 synced
Messages transferred                    : 5705
Messages skipped                        : 92
Messages found duplicate on host1       : 48
Messages found duplicate on host2       : 0
Messages found crossduplicate on host2  : 0
Messages void (noheader) on host1       : 0
Messages void (noheader) on host2       : 0
Messages found in host1 not in host2    : 1 messages
Messages found in host2 not in host1    : 1 messages
Messages deleted on host1               : 0
Messages deleted on host2               : 0
Total bytes transferred                 : 200801589 (191.499 MiB)
Total bytes skipped                     : 13109007 (12.502 MiB)
Message rate                            : 1.1 messages/s
Average bandwidth rate                  : 36.9 KiB/s
Reconnections to host1                  : 0
Reconnections to host2                  : 0
Memory consumption at the end           : 275.2 MiB (started with 218.4 MiB)
Load end is                             : 0.18 0.17 0.21 6/659 on 2 cores
Biggest message                         : 27913861 bytes (26.621 MiB)
Memory/biggest message ratio            : 10.3
Start difference host2 - host1          : -5750 messages, -326607299 bytes (-311.477 MiB)
Final difference host2 - host1          : -89 messages, -88082386 bytes (-84.002 MiB)
The sync is not finished, there are 1 identified messages in host1 that are not on host2.
There is no unidentified message
The sync is not strict, there are 1 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync. (5679 identified messages in host2)
Detected 3 errors

Check if a new imapsync release is available by adding --releasecheck
Homepage: https://imapsync.lamiral.info/
++++ Listing 3 errors encountered during the sync ( avoid this listing with --noerrorsdump ).
Err 1/3: Host2 Folder Outbox: Could not select: 31 NO [NONEXISTENT] Unknown Mailbox: Outbox (now in authenticated state) (Failure)
Err 2/3: Host2 Folder archive_folder: Could not select: 38 NO [NONEXISTENT] Unknown Mailbox: archive_folder (now in authenticated state) (Failure)
Err 3/3: - msg INBOX/5329 {23820} could not append ( Subject:[whatever], Date:["15-Oct-2017 12:13:35 +0100"], Size:[34930], Flags:[\Seen] ) t
o folder INBOX: * BYE Invalid character in literal (4x)
Exiting with return value 111 (EXIT_WITH_ERRORS) 3/50 nb_errors/max_errors

Many thanks for Gilles Lamiral for this brilliant tool.

Example Terraform Code for Cloudflare provider ~v1.x

Here's a sample code repo:

Google Drive for Rclone

I plan on using Gdrive with Rclone with a crypted remote for remote backups to the cloud.

My current local backup procedure isn't the best, so adding Gdrive to the mix might help with reorganising my thinking & scripts.

To-dos:

write backup scripts
setup rclone remotes
decide whether to backup from both my desktop and nas or just from the nas

Servers and clients in my homelab

2019-11-12T12:00:00+00:00

My homelab computers

I have 4 computers at home that each have their own use.

I have a desktop that I built in 2016/11 that is my main computer and which I use the most and code from.

I also have a HP Probook 6570b which is my laptop, which I take with me on occasions and watch movies from.

Then I've got my first laptop, an Acer Aspire 5736z, which only runs Pihole and serves as the DHCP and DNS server on the network and nothing else.

And the last device is my HP Proliant Gen8 Microserver that runs about 5-10 docker images and serves as file server on the network.

Probably should mention the ISP provided "fantastic high quality" modem/router that serves as the router and wifi AP at this point in time.

I've got two Raspberry Pis, a 2b and a 3b+.

I used to use DNSMasq and Squid proxy on these Pis in my previous place, but then I realized I already four Intel machines to use.

In the long term, I'd like to reduce my devices to maybe a desktop and a laptop.

But for the time being, having only Intel machines(as in having all of them on amd64 operating systems) means simpler Ansible playbooks, roles and Docker images.

Network Diagram

Here's the homelab network diagram I've made with draw.io:

homelab network diagram

Playing with draw.io diagrams

2019-11-11T22:00:00+00:00

Draw.io is nice!

It's a program that make nice diagrams and so many other things.

Works OK in a browser, on desktops and it's cross platform.

I've seen many nice network diagrams on /r/homelab and saw draw.io recommended in the comments.

Running it locally

There's a handy docker image at fjudith/draw.io that I started using.

I made my homelab network diagram with it, then exported a png image from it.

A command like this can run it locally, as non-root with user:

docker run --user 1234:1234 --rm -d --name=draw -p 8080:8080 fjudith/draw.io:alpine

Here's my first diagram

This is the first diagram I've made with it:

homelab network diagram

More details on this image can be found in this post:

My home network

Releasing Dilbert scraper's docker image

2019-10-18T12:00:00+01:00

Dilbert.com scraper

I wrote a dilbert.com comics scraper in bash about 3 years ago because I really love Dilbert comics.

While I assume people will scream if they look at the code in dilbert.sh and date generator python script, they have worked fairly reliably for 3 years, with the only breakage that occurred was when dilbert.com changed to https as my parsing code is lame.

Put it into a docker image because why not

So today also I got a working image up on Dockerhub and Gitlab's docker registry.

For usage instructions, please read the readme provided on Gitlab.

As anything I'd write here or on dockerhub would eventually become outdated.

But to save some clicks, here are two one-liners:

Here's a sample command to download all comics:

docker run -d --name dilbert --rm -v $HOME/Pictures/dilbert:/home/user/Pictures/dilbert qwe1/dilbert:latest -a

Here's a sample command to download today's comic:

docker run -d --name dilbert --rm -v $HOME/Pictures/dilbert:/home/user/Pictures/dilbert qwe1/dilbert:latest -t

Upgrading to Debian Buster

2019-09-30T12:00:00+01:00

First impressions

I've been running Debian systems at home for about 3+ years.

I really like the distribution for the package availability and quality.

Buster(Debian 10) comes with kernel 4.19, which is a long term release(LTS) kernel.

On kernel.org, it's said to be supported until December 2020.

As is the case with kernel 3.16(used in Jessie) and 4.9(used in Stretch), I assume the support date will be adjusted to be in line with Buster's End of Life date.

Xfce, the desktop environment I use, is still on 4.12 in this release. 4.14 has been released recently but it didn't make into buster.

There's also a newer libreoffice release(6.1.5.2).

Firefox-ESR, Thunderbird(60 ESR to 69 ESR) and Chromium will be upgraded with newer versions with time.

There's also keepassxc in this release, which I've just started using since keepassx's last release was in 2016/02.

Issues and workarounds

Xfce4 with i915 Intel igpu

I want to leave a link to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929834 and a comment on that bug report as I've run into this issue on a laptop with Intel gpu.

/etc/X11/xorg.conf.d/20-intel.conf:

Section "Device"
  Identifier "Intel Graphics"
  Driver "intel"
EndSection

I'll have to add this file/templating task to my playbooks from now on for this release.

Update: Finally added this a couple days ago.

GPU passthrough issues with amdgpu and rx 480 card

For some reason, the same vfio setup I've used in Stretch stopped working after upgrading.

At first, I thought the issue was withe Windows 10 virtual machine I've migrated from the previous install.

It didn't even occur to me to check the lspci -nnk output.

I'd start the VM, but libvirtd/virt-manager would hang and panic the kernel when trying to reboot.

At anyrate, lspci -nnk output looked like this:

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev c7)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Radeon RX 480 [1002:0b37]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

Amdgpu kernel module would grab the graphics controller, but not the sound card.

What I needed was that vfio-pci would grab the entire card(02:00.0 and 02:00.1).

After digging around with searches Google and Reddit, I've found a possible workaround.

As a reference, I've based my configs for Stretch on the pages from https://scottlinux.com/ and the Arch linux wiki.

For Buster, I've needed to add to my conf file in /etc/modprobe.d/vfio.conf:

softdep amdgpu pre: vfio-pci

so now it looks like this:

#Make vfio-pci a pre-dependency of the usual video modules
softdep amdgpu pre: vfio-pci
# rx 480
options vfio-pci ids=1002:67df,1002:aaf0

High IO delay with LUKS encrypted drives

I've noticed serious UI freezes when downloading larger files over a gigabit lan.

My mouse pointer would stop responding to movements for several seconds.

Here are two screenshots I made while downloading a larger file with firefox on my lan:

I have no workaround for this issue, I just reinstalled my desktop without LUKS.

Pihole Custom Domain Config

2019-09-08T16:00:00+01:00

Running Pihole at home

I run pihole via docker on an old laptop.

Pihole Docker container as DHCP and DNS server

I got it configured via the pihole docs.

To see my ansible playbook that works with Pihole and is run via compose, look at this playbook.

I had to set the container into host networking mode, so it can directly answer DNS and DHCP requests.

Due to host networking mode, I had to comment out all port mappings from the compose config.

Via docker envs, it's also configured so that dnsmasq answers on all network interfaces.

Also the container required to have the NET_ADMIN capabilities added to it.

What I haven't figured out yet is how I can put traefik in front of the pihole web UI to avoid port conflicts with pihole's own webserver that's reachable via port 80 on the host.

Pihole Logs and Stats

It shows some really useful info on your network.

Examples of such info are, when full logging are enabled:

what domains are queried and blocked
what devices are making the requests
what kind of requests devices are making(A, AAAA, PTR and so on)
what DNS servers/forward destinations are used for the queries in what proportion(google, cloudflare, etc), these stats include the blocklist and local cache
there are lists for top permitted and blocked domains
and also lists for top clients and a subset of top blocked clients
my top performers on the blocked client list are a Windows 10 VM and my android phone

Pihole DHCP settings

I was most interested in these parts of the settings as my ISP's router is very limited.

When the DHCP is server is enabled, you can define a range of IP addresses to use on the network.

Since my laptop(pihole host) is not the router on the network, I set the ISP's device as the gateway.

This page also shows the currently active DHCP leases and also any static DHCP lease configurations.

On the right side, there's a section where you can define what domain to use on the LAN.

Note: this domain does need to have any A/AAAA records.

Dynamic DNS

This domain setting will put a search sub.domain.tld line into each Linux device's /etc/resolv.conf:

# Generated by NetworkManager
search home.domain.tld
nameserver 192.168.0.69

As a result, each device will be resolvable on the lan thanks to Dynamic DNS.

So I can connect to each device by either their hostname(eg acerlaptop) or full hostname which would be acerlaptop.home.domain.tld.

This is most useful, when used with traefik and other containers. If configured properly, traefik can then issue wildcards for each device where it runs.

Resulting in the following wildcard certs, valid for:

*.device.home.domain.tld

So a web host might be resolvable this way on the lan:

servicename.device.home.domain.tld

Thanks to Traefik, it will have a valid SSL certificate.

However, this results in long hostnames in the browsers, which will annoy some people.

Adding Custom Domain names to Pihole

Since pihole uses Dnsmasq, we can also apply Dnsmasq configs of our own.

In my case, I've placed my custom config file that has device specific subdomains defined inside via an Ansible template task.

An example of such task is:

    - name: Template pihole conf file for local wildcard domains
      template:
        src: wildcard-local.conf.j2
        dest: /var/docker/pihole/etc-dnsmasq.d/wildcard-local.conf
        owner: root
        group: root
        mode: 0644
      tags:
        - docker

This configuration file can be placed into the container's /etc/dnsmasq.d directory.

If a static volume mapping is used for a container's config, eg:

volumes:
  - /var/docker/pihole/etc-pihole:/etc/pihole
  - /var/docker/pihole/etc-dnsmasq.d:/etc/dnsmasq.d

then our config file needs to be placed inside /var/docker/pihole/etc-dnsmasq.d on the host that runs pihole. Then inside the container, it will look like that a new config file has appeared.

If I want to restart the dnsmasq process inside the container, without restarting the entire container, I can just simply exec:

docker exec ctname pihole restartdns

inside the container. This will apply our new configuration file straight away.

Running Docker-compose from Ansible playbooks

2019-09-01T20:00:00+01:00

Running Docker-compose from Ansible playbooks

On my home devices, I run and maintain my docker containers(ct-s with my Ansible playbooks.

But why use Docker-compose and not Docker Swarm or Kubernetes

Because I only have four devices that I run docker ct-s on, and each serve a dedicated function at my place.

Here are the following devices:

Old Acer laptop
HP Gen8 microserver
Custom built desktop
HP laptop

Since I've got each devices dedicated to a certain function, I just prefer the simplicity of knowing what docker ct-s each device is running as declared in a playbook with a docker_compose task.

How

My current playbooks are in https://gitlab.com/richardskumat/ansible-playbooks-homelab.

Each device is assigned into a group.

Each group has a group_vars directory for variables and secrets. These directories are not public.

In each groups directory, where their playbooks reside, I have symlinked the homelab/group_vars directory so I can use them with vagrant testing. When I do a vagrant up in a group's directory, there's a task in the Vagrantfile that runs the group's playbooks.

I didn't have the group_vars directory symlinked into each subdirectory, then the vagrant's playbook test runs fail with XYZ variable undefined and so on.

Here's a tree output of how my playbooks look are structured:

user@host:~/wherever/homelab$ tree
.
├── acer
│   ├── acerlaptop.yml
│   ├── group_vars -> ../group_vars
│   ├── interfaces.j2
│   ├── readme.md
│   ├── Vagrantfile
│   └── wildcard-local.conf.j2.example
├── add_remotes_back.sh
├── client
│   ├── client.yml
│   ├── group_vars -> ../group_vars
│   ├── readme.md
│   ├── sysctl-local.conf.j2
│   └── Vagrantfile
├── group_vars
│   ├── acer
│   ├── all
│   ├── desktop
│   ├── hpg8
│   ├── laptop
│   └── rpi
├── hosts
├── hpg8
│   ├── group_vars -> ../group_vars
│   ├── hpg8-dc.yml
│   ├── hpg8.yml
│   ├── readme.md
│   └── Vagrantfile
├── LICENSE
├── README.md
├── reboot.yml
├── rpi
│   ├── group_vars -> ../group_vars
│   ├── raspi-debug.yml
│   ├── readme.md
│   ├── rpi3-dc.yml
│   └── rpi.yml
└── shutdown.yml

I'm currently going to change my playbooks to use import_task for the docker_compose playbook.

Then each playbook is structured according to the following thinking:

establish connections details(host groups, remote user and become method)
roles: each device runs a couple of Ansible roles
tasks: extra tasks needed done

The docker_compose task is also part of these tasks, however the indentation requirements for yaml are unpleasant.

Fortunately, VSCode is really good in showing indentation and syntax errors.

I want to move them into separate yml files, so they're more readable.

Traefik for proxying and Let's Encrypt(LE) certificates(cert)

I also run traefik on each device and it serves as a reverse proxy for the other containers having a web interface.

It's also configured to create LE certs with DNS-01 challenge, using Cloudflare as a DNS provider.

Looking into better alternatives

I'm really satisfied with the simplicity of ansible and the docker_compose module.

The ct-s I run are documented in each group's docker_compose files/tasks so I know what personal device runs what containers.

There are better alternatives, such as Docker Swarm and Kubernetes.

I intend to look Kubernetes and Swarm in the future, maybe test them out in their own virtual machines.

Swarm is also part of Docker CE, so if I wanted to, I could use it straight away.

Playing around with Gitlab.com's CI and Docker Registry

2019-08-15T12:15:00+01:00

Dabbling with Gitlab

I've got a Gitlab Community Edition(CE) instance running at home with 3 runners on separate devices.

I'd like to say I've become familiar with the basics of Gitlab's ci, however I haven't done much with the continuous delivery part.

What I'd like to do with the CI part is how I could test a repo simultaneously on different versions of software/OS-s at the same time, without too much copy-pasting of same block of code.

Eg:

testing the same ansible role with different releases of Debian
building docker images with different versions of Debian releases for comparison purposes

The only CD so far I have been doing at home is pushing my own docker images to dockerhub, which is not complicated at all.

Pushing Docker images from home

I've got a few images on Dockerhub, which I've been building, testing and pushing from home.

I've got daily scheduled jobs running in my private Gitlab instance that rebuild and push the docker images early in the morning.

However, this requires me to keep my own machines powered on all the time, even if I wanted to go away someplace.

I could just power them off while gone, but then the images wouldn't be rebuilt daily.

So now I came to the conclusion, it'd be smarter to utilize Gitlab.com's free runner instances to build and push images.

At this time, I only publish on Dockerhub, but this way I could push both to Gitlab.com's docker registry and dockerhub to provide a bit of redundancy for myself.

My only concern with building docker images on Gitlab.com would be needing to store my Dockerhub login details in each repo's CI/CD vars settings, which would be really cumbersome to do manually.

Terraform and Gitlab.com

Looking at Terraform's documentation in the providers section, there is a VCS/Gitlab provider.

Further reading up on these pages could save me some time in the future.

Gitlab repo playground

I made a dummy git repo on both on public Gitlab.com and my private Gitlab instance.

The purpose of this repo is to test/play with Gitlab's CI.

What I've managed to do at this time was to build and push a docker image to Gitlab's internal CI without specifying any CI/CD vars in settings manually.

Here's the raw version of the Gitlab CI file that made this possible.

Here's a link to Gitlab.com's doc page of predefined environment variables.

Some comments below for the vars/commands in that CI file above:

"$CI_REGISTRY_USER": this seems to return gitlab-ci-token in the build logs, yet this is the only var that let me log into Gitlab's docker registry, yet when pushing the docker image to Gitlab's registry, I'd get requested access to the resource is denied errors.
"$CI_REGISTRY": this returns the registry on the Gitlab instance if it's activated
echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin: This is more secure than passing -p password to docker login for some reason
docker push $CI_REGISTRY/$GITLAB_USER_LOGIN/$CI_PROJECT_NAME:$CI_COMMIT_SHA: This seems to work only if the person initiating the build is also the owner of the project, because of $GITLAB_USER_LOGIN being: The login username of the user who started the job. Eg on my private instance, my projects are under a non-admin user. If I run a pipeline as a different user, the build fails.
$CI_PROJECT_NAMESPACE: The project namespace (username or groupname) that is currently being built. This seems to be a better choice than $GITLAB_USER_LOGIN to change in the docker push URL as it also works when a pipeline is launched by a different user.

Second blog post

2019-08-08T23:15:00+01:00

Second blog post

For the time being this site will be hosted on netlify, at https://blog.richardskumat.com.

A fun project would be to setup the blog subdomain to automatically fail over to either Github or Gitlab pages if Netlify's servers were down or if they terminated my account for some reason, but at this point in time it'd be a waste of time, and I'm not even sure where I should get started with this idea.

Configuration

So after setting up Gitlab and Github repos with the blog repo, I've decided to configure my netlify account to publish from the Gitlab repo.

Also configured my DNS records with terraform to point the blog subdomain with a CNAME record to netlify's subdomain where the site is hosted.

Then added a custom domain on netlify's dashboard, and since the cname record already existed, it was easy enough to enable SSL for the website.

First blog post

2019-08-08T21:15:00+01:00

First blog post

This is the first blog post to see how the new blog will look.

The blog posts will be written in Markdown and published with Pelican.

To-do:

Settle on a theme
Settle on a site structure
Setup CI/CD and decide which host to use(leaning towards Gitlab or Netlify at this point)
Setup DNS via terraform for the site
Start publishing
Check up on my netlify account

Notes on pushing to Github pages

Based on https://docs.travis-ci.com/user/deployment/pages/ and https://pages.github.com/.

To publish to Github pages via Travis CI, ideally the following things are required:

a working CI config file, the .travis.yml in this repo can be used as a base
set $GITHUB_TOKEN found in this repo's on https://github.com/settings/tokens/new, select Scope repo/public_repo. Selecting this scope should be enough
having gh-pages branch in the repo, so travis-bot can force push the created static files to this branch. this can achieved by: git branch gh-pages git push github gh-pages
on https://github.com/user/project/settings, set Github Pages source to gh-pages branch which then sets the gh-pages branch as the source of Github Pages. This is the same branch where Travis pushes to. So Travis reads from master, then pushes the output of pelican to gh-pages, then Github publishes from gh-pages branch

Notes on publishing to Gitlab pages

Requirements:

a working CI config file, the .gitlab-ci.yml in this repo can be used as a base

See https://gitlab.com/pages/pelican for a pelican example, which this site is based on.

Source code

My own links for publishing:

Canonical URL of my blog, as of 2019-12-05:

https://www.richardskumat.com/

Used to be blog.domain instead of www, but changed that later on.

Tinker, break and fix

RHCSA 9 changes

RHEL 9 hidden grub menu

Ansible RSS feeds

SMR drives suck

Weekend

They took our jobs! .. err Dilbert comics

PAM_FAILLOCK

Finding a Repo's Github Release RSS link

2022-08-08 Keep an eye on Gitlab storage

2022-08-08 Updating my docker images to multiarch builds

2022-05-10 Change of email service provider

2022-04-13

New services to add

Note taking

Local monitoring

Web page change monitor

2022-04-12

Just wonderful

droneci-https

Droneci and SSL

Somethings remain to be done

droneci

Droneci

Link

Issues

To-do?

Spring cleaning

Idea of cleaning up projects

Black friday

Buy buy buy

A nifty find one liner

What this does

Actual command

Test before try

Debian Jessie(8) end of life

Special release

Related link

An issue I ran into when creating a new Gitlab repo with terraform

Gitlab + Terraform

Some of the docker images I use

My own Docker images

Other images used

How I try to track docker images tags and releases

pgrep

Pgrep

Improvements to linting

Recent things

What's been going

What I'm going to do

Simple ansible logrotate script

Ansible logs

Cron job

Gmail settings for Thunderbird with a Gsuite account

My issue with sending emails

What I've tried

Relevant links

Terraform git repo removal with Gitlab provider

New changes on Gitlab

Other relevant issues

Working with the Gitlab provider for Terraform

More Terraform work

Terraform(TF) with Gitlab

What's in the repo

Molecule tests

Update on 2020-02-23

Some notes for myself regarding molecule testing

Sudo with docker instead

Finished working on my Docker ansible role for now

Merged some commits in my docker role

New Docker images

Two new docker images

Source code to repos

Now they're done

Tags

Update on my Docker Ansible role

New year, old projects

Why update my own Ansible roles that I only use myself

What's being updated in my Docker role

Day-off project: Retropie