Posts on Lead DevOps

fixing memory leaks in python services: diagnostics and dump collection

Sun, 04 Oct 2026 00:00:00 +0000

Python services under load can silently consume memory until the cgroup limit triggers an OOM kill. Without systematic dump collection and introspection, root-cause analysis devolves into hypothesis spinning. Below is a practical set of commands and scripts for diagnostics, heap dump collection, and leak mitigation.

1. Memory Consumption Diagnosis Commands

Basic level — psutil. Installed in one line and works without process restart.

pip install psutil

Current process consumption:

SSH key best practices

Thu, 01 Oct 2026 00:00:00 +0000

SSH keys are the de facto standard for authenticating to infrastructure, but poor management turns every deployment into a potential vulnerability. This note collects proven practices: from key generation to revocation and rotation without service downtime.

Introduction

SSH keys function as long‑lived credentials, and their lifecycle directly impacts supply‑chain security. Unlike passwords, keys are often created once and forgotten, leading to an accumulation of “zombie” keys with privileges that exceed current needs. Proper generation, binding to an agent, and regular rotation minimize the attack surface and enable auditing of changes. The following sections describe concrete commands and configurations used in operations.

kubectl cheatsheet: essential commands for Kubernetes

Thu, 24 Sep 2026 00:00:00 +0000

kubectl is the primary interface to a Kubernetes cluster. This cheatheet covers routine operations — from context setup to pod debugging and namespace switching. All commands are verified against current kubectl versions (1.28+).

Installation and Context Setup

Installation depends on your OS. On Linux, use the package manager or the binary directly:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
kubectl version --client

Configuration lives in ~/.kube/config. A context defines the cluster, user, and default namespace.

Creating User and Role in Kubernetes and Binding Them via RBAC

Wed, 23 Sep 2026 00:00:00 +0000

In Kubernetes there are no “users” in the traditional sense — there are ServiceAccounts and certificates, bound to roles through RBAC. Without proper configuration, anyone holding a kubeconfig gets full access to the cluster. Below is the complete cycle: create a ServiceAccount, define permissions, bind them, and verify.

Creating ServiceAccount and Generating kubeconfig

Start by creating a ServiceAccount in the target namespace:

kubectl create serviceaccount devops-sa -n staging

To generate a kubeconfig, extract the token from secrets and build the config file:

Git Tag: Marking Releases and Bookmarks in History

Wed, 23 Sep 2026 00:00:00 +0000

Git Tag: Marking Releases and Bookmarks in History

Tags are Git’s mechanism for assigning meaningful labels to specific commits. Unlike branches, tags don’t move — they’re pinned to a commit and serve as anchors for releases, versions, and critical checkpoints. Without tags, release history devolves into a hash search — and that’s a direct path to deployment errors.

Types of Tags: Lightweight vs Annotated

There are two types of tags. Lightweight is just a name attached to a commit, with no additional metadata. Annotated is a full Git object with author, date, message, and signing capability.

Deploying with a post-receive Git Hook

Sat, 19 Sep 2026 00:00:00 +0000

Deploying through CI is great, but sometimes you just need to push code to a server with a single git push. The post-receive hook in a bare repository handles this without extra dependencies: push to the server, and the hook automatically checks out files into the working directory.

Schema: bare repo as a deploy trigger

The logic is straightforward:

A bare repository is created on the server (e.g., /srv/deploy/app.git).
The developer adds it as a remote and runs git push origin main.
Git receives the data and triggers hooks/post-receive.
The hook script runs git --work-tree=/var/www/app --git-dir=/srv/deploy/app.git checkout -f main.

Note

A bare repository has no working directory. That is why post-receive must explicitly pass --work-tree so checkout knows where to write files.

ulimit and systemd LimitNOFILE — why ulimit -n inside a unit doesn't stick

Sat, 19 Sep 2026 00:00:00 +0000

What is nofile and where it lives

nofile is the maximum number of open file descriptors per process. That’s not just regular files — it covers sockets, pipes, stdin/stdout/stderr, logs shipped through journald — everything counts. When nginx or a Go app crashes with too many open files, this is the limit to blame.

Limits live at three levels:

Level	Where to check	What it controls
Kernel (system-wide)	`/proc/sys/fs/file-max`, `/proc/sys/fs/nr_open`	Absolute ceiling for the whole system
PAM / login	`/etc/security/limits.conf`, `/etc/security/limits.d/`	For sessions via `pam_limits.so`
systemd	`LimitNOFILE=` in unit, `DefaultLimitNOFILE=` in `system.conf`	For systemd-managed services

Note

/proc/sys/fs/nr_open is the upper bound you can raise nofile to for a single process. It defaults to 1073741816 (≈1B) on most distros, but in practice you rarely need more than 1048576.

coredumpctl: Finding a Binary Crash

Fri, 18 Sep 2026 00:00:00 +0000

What is coredumpctl and how it works

When a binary crashes with SEGV, the kernel can save a core dump — a snapshot of process memory at the moment of the crash. In systemd-based distributions, coredumpctl handles collecting, storing, and searching these dumps. It’s a wrapper around systemd-coredump, which stores dumps in /var/lib/systemd/coredump/ and indexes metadata through journald.

Note

Requires systemd-coredump and an active journald. In minimal containers without systemd, this tool is unavailable.

curl --resolve and SNI: Testing Virtual Hosts Without /etc/hosts

Fri, 18 Sep 2026 00:00:00 +0000

When you need to test a virtual host on a specific IP but don’t want to edit /etc/hosts — whether due to permissions, conflicts with other services, or just the habit of keeping the file clean — curl --resolve solves both problems at once: it overrides DNS resolution and sends the correct SNI in the TLS handshake.

Problem: virtual host without editing /etc/hosts

Multiple virtual hosts can live on a single IP, and the server picks the right one based on the Host header (HTTP/1.1) and SNI (TLS). Without an entry in /etc/hosts, curl first tries to resolve the name through DNS — getting the wrong IP, or no response at all.

Docker logs and journald: choosing a logging driver

Fri, 18 Sep 2026 00:00:00 +0000

When a container crashes, logs are the first thing you need to see. docker logs looks simple, but under the hood different logging drivers are at work, and the choice affects how logs are stored, rotated, and accessed. Here is what you should know before trusting the default.

How docker logs works

The docker logs <container> command reads the container’s stdout/stderr stream and outputs it to the terminal. Behind this sits a logging driver — a component that determines where the data actually goes. By default it is json-file: each container gets a JSON file on the host into which every output line is written.

scp — Secure Copy Over SSH

Fri, 18 Sep 2026 00:00:00 +0000

scp — a utility for copying files over SSH using the SSH protocol. It works from the terminal, requires no extra server setup — just a running sshd and working authentication. In an era of rsync and bat, SCP survives as a simple tool for one-off transfers when you don’t want to deal with daemons or configuration files.

Syntax and Basic Scenarios

General form:

scp [flags] source destination

Source and destination can be local paths or remote addresses in the format user@host:path.

Sudoers: NOPASSWD Without Holes

Fri, 18 Sep 2026 00:00:00 +0000

Unrestricted NOPASSWD in sudoers is a misconfiguration that grants root access without a password, turning any user script or library vulnerability into a full system compromise. The correct approach limits NOPASSWD to specific commands only.

Why NOPASSWD + ALL Is a Hole, Not a Solution

%admin ALL=(ALL) NOPASSWD: ALL — the most common sudoers error. The user receives unlimited root access without a password. Any script, any utility, any vulnerability in the user’s environment becomes a direct path to full machine control. NOPASSWD without command restrictions is not convenience; it is a backdoor in plain sight.

ThinLinc: Remote Access to Linux Desktops

Fri, 18 Sep 2026 00:00:00 +0000

ThinLinc: Remote Access to Linux Desktops

In enterprise environments, remote access to Linux desktops often comes down to VNC with flaky encryption or RDP proxies held together with workarounds. ThinLinc by Cendio is a full-featured remote desktop solution that runs on top of VNC, supports RDP clients, and provides a web-based administration interface without the usual hassle with certificates and firewalls.

What Is ThinLinc

ThinLinc is a remote desktop access solution with a server-plus-clients architecture. The server runs VNC sessions on the backend, and clients connect through a web browser or native ThinLinc clients. The protocol between client and server is tunneled over SSH, which solves the encryption problem out of the box.

Chrony Instead of ntpd

Thu, 17 Sep 2026 00:00:00 +0000

Chrony has replaced ntpd as the default NTP client in most modern Linux distributions. It converges to accurate time faster, handles intermittent network connections better, and consumes fewer resources. If your machine still runs ntpd, switching takes only a few minutes.

Installation

On RHEL-based systems:

sudo dnf install chrony -y
sudo systemctl enable --now chronyd

On Debian/Ubuntu:

sudo apt install chrony -y
sudo systemctl enable --now chronyd

If ntpd was running on this machine before, stop and disable it to avoid port conflicts:

fail2ban: SSH Jail Configuration

Thu, 17 Sep 2026 00:00:00 +0000

Securing SSH against brute-force attacks is one of the first steps in hardening any server. fail2ban scans logs, detects repeated failed login attempts, and blocks the source via iptables or nftables. This note covers the sshd jail — from installation to fine-tuning ban durations.

Installation and Basic Configuration

Install from the standard repository:

# Debian/Ubuntu
apt install fail2ban

# RHEL/CentOS
yum install fail2ban

Enable and start the service:

journalctl: filters and follow

Thu, 17 Sep 2026 00:00:00 +0000

Systemd’s journal is the first place to look when a service crashes or a node starts burning CPU. journalctl does far more than dump the entire log in sequence: it can filter by units, priorities, time windows, and stream in real time. Below is the working set I use daily.

Follow in real time

Behavior similar to tail -f, but aware of journald’s structured format:

nftables: Basic Rule Set

Thu, 17 Sep 2026 00:00:00 +0000

Note

All commands were verified on Debian/Ubuntu with the nftables package and on RHEL/CentOS 8+. On older systems you may need apt install nftables or yum install nftables.

nftables replaced iptables, but documentation for a basic rule set is often scattered. Here is the reference I use when bringing up a firewall on a new host.

Creating the inet filter table

The inet family table unifies IPv4 and IPv6 under a single namespace. This is the preferred approach when both stacks are active on the host.

Debian — The Swiss Army Knife of Linux

Tue, 15 Sep 2026 00:00:00 +0000

Debian isn’t the flashiest distribution, but it’s the most reliable foundation in the Linux world. Behind it stands the largest community of volunteer developers, and behind its track record are decades of uninterrupted operation on servers, embedded systems, and cloud infrastructure. If you manage Linux machines in production, Debian (or one of its derivatives) is already in your stack — whether you’ve acknowledged it or not.

Managing Packages: apt and dpkg

Two tools form the core of Debian’s packaging system. apt is the high-level interface for working with repositories, resolving dependencies, and upgrading the system. dpkg is the low-level engine that installs, removes, and inspects individual .deb files without contacting repositories.

pipx: Isolated Python CLI Tools Without the Mess

Thu, 10 Sep 2026 00:00:00 +0000

pipx solves a simple but chronic problem: you need to run a Python utility once or occasionally, and pip install pollutes the global environment or leaves behind a virtual environment you forget to clean up. pipx creates an isolated venv for each utility, installs dependencies there, and makes the binary available in $PATH. One command and the tool works without conflicting with anything.

What Is pipx and Why You Need It

pipx installs and runs Python applications in isolated virtual environments. Each utility lives in its own venv under ~/.local/pipx/venvs/, and its console-scripts are symlinked into ~/.local/bin/.

uv — Fast Python Package Manager

Wed, 09 Sep 2026 00:00:00 +0000

What is uv

uv is a Python package manager written in Rust. It solves one problem: the standard pip is slow at resolving dependencies, and poetry adds its own project model on top of PEP 621. uv works with pyproject.toml, is compatible with PEP 621 and PEP 508, and does it significantly faster.

Under the hood is a caching resolver written in Rust that reuses data from pip-compatible indexes (PyPI by default). uv can replace pip, pip-tools, virtualenv, and poetry in a single tool.

logrotate for Custom Daemons

Tue, 08 Sep 2026 00:00:00 +0000

Log rotation is missing for your daemon, the log file has grown to dozens of gigabytes, the disk is full, and monitoring is screaming. systemd-journald and syslog-ng rotate on their own, but if your custom daemon writes directly to a file, rotation falls to logrotate. Here is how to configure it for a specific service.

Why write a custom logrotate config

Packages from the repository usually drop their config into /etc/logrotate.d/, but for self-built daemons or those compiled from source, there is none. Without a config the file grows without limits. logrotate runs via a systemd timer (logrotate.timer) or cron and reads all files from /etc/logrotate.d/. Creating a single file is enough to start the rotation cycle.

Rsync: Backing Up a Directory Over SSH

Tue, 08 Sep 2026 00:00:00 +0000

Rsync: Backing Up a Directory Over SSH

The classic way to copy a directory to a remote machine is rsync over SSH. No extra ports to open, traffic is encrypted, and the tool itself handles incremental transfers and metadata preservation. One command and the backup is ready.

Basic rsync Command Over SSH

The minimal invocation to copy a local directory to a remote host:

Setting Up Your Own SSH Bastion Server

Tue, 08 Sep 2026 00:00:00 +0000

Why You Need a Bastion and Where It Lives

A bastion is the single entry point into a private network segment. Instead of exposing SSH on every server to the internet, you funnel traffic through one hardened host with a strict access policy. Typical layout: internet → bastion (public IP) → internal servers (only private subnet, SSH listening on 127.0.0.1 or a private interface).

The bastion sits in a demilitarized zone (DMZ) or a public subnet provided by your hosting platform. Internal machines have no route to the internet through the bastion — return traffic flows only over established connections. This is a baseline model you can deploy on any VPS in about 15 minutes.

tmux on Prod After Screen

Tue, 08 Sep 2026 00:00:00 +0000

Why We Switched from Screen to tmux

Screen was our primary tool for about five years. After migrating the cluster to new servers it became obvious: screen drops sessions on SSH disconnect when hardstatus isn’t configured, and screen -r recovery sometimes hits a race condition when multiple admins connect simultaneously. tmux solves both problems out of the box — sessions live in server memory, are bound to a socket, and reconnection doesn’t depend on the TCP connection state.

bpftrace: one-liners that replace strace in production

Mon, 07 Sep 2026 00:00:00 +0000

strace halts a process on every syscall. On a live server at 2000 RPS, that means timeouts and alerts. bpftrace runs through eBPF in the kernel — tracing happens in parallel, without stopping anything. The overhead difference is orders of magnitude.

Installation

# Debian / Ubuntu
sudo apt install bpftrace

# RHEL / CentOS / Fedora
sudo dnf install bpftrace

# Arch
sudo pacman -S bpftrace

# Verify
sudo bpftrace -V

Full probe coverage requires debug symbols:

ip: Network Setup and Diagnostics in CLI

Mon, 07 Sep 2026 00:00:00 +0000

When ifconfig returns nothing and configuring a route requires a separate command, that’s not a system bug. It’s iproute2 — the package that replaced net-tools in modern Linux distributions. The ip utility from iproute2 is the standard interface for managing the Linux network stack. It covers interfaces, addresses, routes, ARP cache, routing policies, and namespace isolation.

Why iproute2 replaced net-tools

net-tools (ifconfig, route, arp, netstat, nameif) originated in BSD and migrated to Linux in the 1990s. By the 2000s it became clear: they cannot handle VLAN, IPsec, QoS, multicast routing, or Policy Routing. Each task required a separate command with unrelated syntax.

nslookup and drill: DNS resolution in terminal

Mon, 07 Sep 2026 00:00:00 +0000

The server won’t resolve a domain, but pings fly through. No familiar dig at hand — the BIOS is already loading a minimal busybox. Or on a host without bind-tools. nslookup and drill fill this gap: the first one is built into almost everything, the second gives more context when debugging.

nslookup: interactive and one-liner modes

nslookup ships with bind-utils and isc-dhcp-client. It works in two modes.

journalctl: Filtering and Formatting systemd Logs

Sun, 06 Sep 2026 00:00:00 +0000

Logs disappeared. Server rebooted, and the familiar less /var/log/syslog returns nothing. On modern distros with systemd, logs are collected by journald and read with journalctl. Without knowing its filters, system debugging turns into guesswork.

Why Logs Disappear After Reboot

By default, journal stores data in /run/log/journal/ — a tmpfs that wipes on reboot. To make logs survive reboots, create the directory:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

Then restart systemd-journald:

OpenSSL: TLS Certificate Verification and Parsing in CLI

Sun, 06 Sep 2026 00:00:00 +0000

Certificates expiring on prod at the worst moment — a familiar story. OpenSSL answers TLS certificate questions faster than any marketplace checker. Here are the key scenarios without the fluff.

Basic Certificate Parsing

The first command for any diagnostics is the text dump:

openssl x509 -text -noout -in cert.pem

Output shows Subject, Issuer, validity dates, signature algorithm, and public key. For a quick summary without the wall of text:

ProxyJump and bastion hosts via ~/.ssh/config

Sun, 06 Sep 2026 00:00:00 +0000

Sometimes a server sits in a private network with no public IP. The only entry point is a bastion host with a public address. Typing ssh -J user@bastion user@private every time gets old fast. Here’s how to configure everything in ~/.ssh/config so you can reach private networks in one command.

Why you need a bastion host

A bastion (jump host, jump box) is an intermediate server with public access that proxies connections to infrastructure without external IPs. The typical topology:

auditd: file access and syscall logging

Sat, 05 Sep 2026 00:00:00 +0000

Linux doesn’t write every access to /etc/shadow or every unlink call to syslog. For incident investigation and compliance this is critical. auditd solves this: the Linux Audit kernel subsystem records system calls, file access, and more.

Installation and Startup

auditd comes in the audit package available in any distribution.

# Debian/Ubuntu
apt install auditd

# RHEL/CentOS/Alma
yum install audit

# Arch
pacman -S audit

After installation, start the service via systemd.

logrotate: automatic log rotation and archiving

Sat, 05 Sep 2026 00:00:00 +0000

Application logs fill up disk space within a week, and manually running rm *.log is a recipe for trouble. logrotate handles this automatically: it rotates, compresses, and deletes old files on a schedule. Let’s see how it works and how to set it up in five minutes.

How It Works

logrotate runs daily through cron. The default config lives in /etc/logrotate.conf, and additional configs are included from /etc/logrotate.d/. During rotation, the current file gets renamed, a new empty one is created, old copies are compressed and numbered.

sshd_config: baseline for a test stand

Sat, 05 Sep 2026 00:00:00 +0000

SSH access to a test stand often gets opened in a hurry, and then the logs fill with brute-force attempts. A baseline sshd_config that blocks common attack vectors fits into five parameters and twenty minutes.

Why Change Defaults

Distribution-provided sshd ships with permissive settings: root login via password, no user restrictions, three authentication attempts. On a test stand this is tolerable until the logs show:

systemd-run: Run Services Without Unit Files

Sat, 05 Sep 2026 00:00:00 +0000

Sometimes you need to run a process under systemd’s control without writing a unit file — maybe you’re in a container without systemd, on someone else’s machine, or just need a quick one-off. That’s where systemd-run comes in.

Why systemd-run

The tool creates a transient unit — a unit that exists only in systemd’s memory, with no file on disk. This is useful when you need:

curl: HTTP Debugging in CLI

Fri, 04 Sep 2026 00:00:00 +0000

cURL is the standard tool for debugging HTTP in the terminal. It ships out of the box on Linux and macOS and is present in most Docker images. Need to quickly check an API, inspect response headers, or trace a redirect issue — one command line is enough.

Basic Debug Flags

The most common scenario: get a response and see what the server returned. The -i flag prints headers before the body, -v enables verbose mode with connection details.

ethtool: network interface diagnostics and tuning

Fri, 04 Sep 2026 00:00:00 +0000

Network issues hide well — interface is up, IP is assigned, iptables is quiet, yet packet loss or micro-freezes only surface under load. ethtool gives direct access to hardware state, driver behavior, and offload mechanisms that neither ip nor netstat expose.

Basic Output: Link State

Installation is straightforward:

# RHEL/Alma/Rocky
sudo dnf install ethtool -y

# Debian/Ubuntu
sudo apt install ethtool

Running ethtool without flags prints a summary:

lsof: which processes listen on port and hold file

Fri, 04 Sep 2026 00:00:00 +0000

Service won’t start — port 8080 is already bound. You dig into who’s holding it, and discover the same process has your config file open while you’re trying to edit it. lsof answers both questions: which processes opened which files and sockets.

Listening Ports

The classic task — find who is listening on a specific port.

lsof -i -n -P

Flag	Effect
`-i`	Show internet sockets
`-n`	Skip DNS resolution (show IP instead of hostname)
`-P`	Skip port-to-service conversion (show 80 instead of http)

Without -n -P, lsof wastes time on DNS lookups and resolves ports through /etc/services. On production hosts that’s unnecessary seconds.

mc — MinIO Client S3 CLI

Fri, 04 Sep 2026 00:00:00 +0000

S3-compatible object storage is the default choice for buckets, backups, and static assets. When AWS CLI feels excessive and the web console is too clunky, MinIO Client (mc) fills the gap. This CLI tool works with any S3-compatible storage: MinIO, Yandex Cloud, AWS S3, Backblaze B2. Zero dependencies, works out of the box, configured in under a minute.

Installation

Download the binary and make it executable:

nftables: Modern Linux Firewall

Fri, 04 Sep 2026 00:00:00 +0000

Warning

Before changing nftables, make sure you have physical or console access to the server. A misconfigured input chain can block SSH and lock you out.

nftables replaced iptables in the Linux kernel starting with version 3.13. If you’re still writing rules in iptables style, it’s time to reconsider. nftables performs better, has built-in dual-stack IPv4/IPv6 support, and lets you manage the entire ruleset as a whole instead of entering commands one by one.

ss: socket statistics instead of deprecated netstat

Thu, 03 Sep 2026 00:00:00 +0000

When netstat hangs on a server with tens of thousands of connections, it’s time to switch to ss. Part of the iproute2 package, ss queries the kernel directly via netlink instead of parsing /proc/net/*. The result is instant output with minimal overhead.

Why switch from netstat

netstat from net-tools relies on a deprecated approach: it reads from /proc/net/tcp, /proc/net/unix and converts numeric IDs to symbolic names. On a server with active connections, this takes seconds and spikes CPU usage.

SSH Config: Wildcards and Dynamic Variable Substitution

Thu, 03 Sep 2026 00:00:00 +0000

SSH reads ~/.ssh/config line by line, but without variables the file quickly becomes copy-paste hell. Here’s how Host patterns, Match exec, and substitution tokens like %h, %r, %l cut config size by orders of magnitude while covering real scenarios — from dynamic routing to agent forwarding through bastion hosts.

Templates and wildcards in SSH config

SSH supports glob-like patterns in the Host directive. The most common is Host *, but compound patterns work too.

strace: System Call Tracing for Diagnosing Hangs and Leaks

Thu, 03 Sep 2026 00:00:00 +0000

When a service hangs, standard tools like top, htop, and ps show the state but not the cause. If a process is in state D (uninterruptible sleep), it’s waiting on a syscall. strace attaches to a live process and outputs every system call in real time. This turns a mysterious hang into a specific syscall, its arguments, and return code.

Note

strace uses ptrace, the kernel’s debugging mechanism. On production, tracing slows a process by 2–10x. Use it sparingly, targeting a single PID.

tcpdump and tshark: Packet Capture in CLI

Thu, 03 Sep 2026 00:00:00 +0000

When debugging network issues in Linux infrastructure, ping and curl are not enough. Sometimes you need to see what is actually traveling over the wire. tcpdump is the standard tool for capturing packets from the CLI. tshark is its sibling from the Wireshark suite, convenient for scripting.

Quick Start with tcpdump

Check that packets are reaching the host:

tcpdump -i eth0 host 10.0.0.5

The utility puts the interface into promiscuous mode and prints one line per packet passing through. By default it works with the first interface it finds, but specifying explicitly is better.

CasaOS: Web Dashboard for Home Lab

Wed, 02 Sep 2026 00:00:00 +0000

CasaOS is a lightweight web dashboard for managing Docker containers on a single host. If you’re currently accessing each container through its own port, this replaces that mess with a unified interface where you can install apps with a couple of clicks, monitor resources, and manage storage — without touching Nginx or dealing with Portainer’s complexity.

Installation

CasaOS installs on a clean system with one command. Supported distros: Debian 10+, Ubuntu 18.04+, Raspbian. If Docker is already running, remove it first or accept that CasaOS will take over the Docker daemon.

cockpit-ufw-module: Uncomplicated Firewall in Cockpit

Wed, 02 Sep 2026 00:00:00 +0000

UFW on a home or small server is usually configured over SSH: ufw status numbered, then ufw allow 443/tcp. cockpit-ufw-module covers the same cycle in the browser: package, status, policies, rules. It is one panel from the cockpit-modules group — UFW under Tools.

The UI is Russian, in PatternFly v5. You need Cockpit 264+ and administrator rights for writes. MIT license; current release is tag v.1.0.1.

Why a panel if `ufw` already exists

The client is already there: system ufw. What is missing is a view without a terminal and safe input: port, CIDR, rule number.

Creating a Custom Systemd Service

Wed, 02 Sep 2026 00:00:00 +0000

Your application needs to start on boot, restart on crash, and log output. Shell scripts in /etc/rc.local give you none of that. Systemd solves all three with a single declarative file.

Why Write a Custom Unit File

Supervisord and init scripts are overkill for most cases. Systemd provides a unified interface for service management: socket-based activation, dependency tracking, resource limits, and built-in logging via journald. You get all of it without additional tooling.

kubectl whoami and Service Account Permission Checks

Wed, 02 Sep 2026 00:00:00 +0000

When deploying an application to Kubernetes, the most common failure is a service account that cannot do what it should. Permission denied when creating a secret, rejection on list pods, refusal on update. kubectl whoami and kubectl auth can-i let you quickly identify exactly who cannot do what.

kubectl whoami plugin

Note

kubectl whoami is not a built-in kubectl command. It is a plugin from krew or a standalone binary. Install with kubectl krew install whoami or download from GitHub.

ngrep: grep for Network Packets in Real Time

Wed, 02 Sep 2026 00:00:00 +0000

Ngrep applies grep-style pattern matching to network packets. When you need to see exactly what two services are exchanging over the wire and tcpdump drowns you in noise, ngrep isolates the payload content you care about.

Installation

Ngrep ships in the standard repositories of most distributions.

# Debian/Ubuntu
apt install ngrep

# RHEL/CentOS/Alma
yum install ngrep

# macOS
brew install ngrep

Running ngrep requires root privileges or the CAP_NET_RAW and CAP_NET_ADMIN capabilities.

socat: Forwarding Unix Sockets Over TCP

Wed, 02 Sep 2026 00:00:00 +0000

Sometimes you need to reach a Unix socket from a host where that socket doesn’t physically exist. SSH tunnels won’t help — they only work with TCP ports. socat solves this: it opens a TCP listener and forwards connections to a Unix socket, and the client just connects over the network.

Installation

The package is available in every major distribution. On Debian/Ubuntu:

apt install socat

On RHEL/CentOS:

SSH Escape Sequences: Reviving a Frozen Terminal

Wed, 02 Sep 2026 00:00:00 +0000

SSH session froze, Ctrl+C does nothing, Ctrl+D spits out garbage — familiar situation. Before closing the terminal and losing the session, try built-in escape sequences. They operate at the SSH client level before data reaches the remote host.

How to invoke escape sequences

The default escape character is tilde (~). The combination works only at the beginning of a line. Press Enter, then ~, then the desired symbol. For example, ~. terminates the connection.

What is Self-Hosted and Why It's So Popular

Wed, 02 Sep 2026 00:00:00 +0000

You’re paying for Notion, Dropbox, and Google Drive. Then Notion raises prices, Dropbox caps storage, and Google “improves” the Docs interface. Self-hosted is taking infrastructure into your own hands and breaking the vendor dependency loop.

Definition: Not Your Cloud

Self-hosted means deploying and operating applications on your own servers or VPS instead of using SaaS alternatives. The server can sit at home, in a data center, or be a VM at a hosting provider—the point is that the hardware is under your control, not a third party’s.

cockpit-modules: web panels for day-to-day operations

Tue, 01 Sep 2026 00:00:00 +0000

Cockpit covers basic Linux administration in the browser: services, logs, networking, accounts. Firewall, fail2ban, cron, and Let’s Encrypt sit outside that set — either there is no panel, or it is too generic.

The cockpit-modules group is a set of separate modules for those jobs, plus a store that installs them on the host. Each module lives in its own repository. This is a map of the group, not a walkthrough of UI and commands. Individual panels get their own articles.

dig: DNS Query Debugging in CLI

Tue, 01 Sep 2026 00:00:00 +0000

DNS resolvers return the wrong address, clients don’t see updates, or it’s unclear which server is handling requests. dig (Domain Information Groper) is the standard CLI tool for DNS diagnostics. Works on Linux, macOS, and Windows via WSL.

Installation

# Debian/Ubuntu
apt install dnsutils

# RHEL/CentOS/Alma
dnf install bind-utils

# macOS — ships with the system
# Windows — via WSL or ISC official binaries

Basic Flags

dig has two classes of options: short flags (start with -) control query behavior, while keywords with + control output format.

MkDocs: Project Documentation from Markdown

Tue, 01 Sep 2026 00:00:00 +0000

Documentation in a repository ages faster than anyone reads it: README links lead nowhere, sections are scattered across docs/, wiki/, and Confluence, and site search doesn’t work. MkDocs solves this predictably — it takes a folder of .md files and builds a static site. One config, one command for the deploy, familiar Markdown.

What is MkDocs

MkDocs is a static site generator for documentation written in Python. Input: a directory of Markdown files and a YAML config. Output: a ready site/ directory with HTML, served by any web server or hosted on GitHub Pages, GitLab Pages, S3. MkDocs core handles rendering; the theme defines look and features. The de-facto standard is Material for MkDocs.

Squid: Internet Forwarding to Remote VM

Tue, 01 Sep 2026 00:00:00 +0000

Your VM in the cloud has no public IP or internet access is blocked via NAT, but the deployment needs wget/curl from inside. Squid on an intermediate host with a decent uplink solves this in ten minutes.

Why This Is Needed

I forward internet through Squid when a VM sits in an isolated network segment. An intermediate host with a public IP and network access becomes the proxy server. The application on the remote machine routes traffic through the tunnel.

SSH certificates instead of authorized_keys

Tue, 01 Sep 2026 00:00:00 +0000

authorized_keys works fine for a handful of servers. Once you hit a dozen, it becomes a liability. Onboarding a new developer means manually distributing their public key across every machine. SSH certificates flip this model: one CA signs all public keys, and authorized_keys stays empty.

Why authorized_keys breaks at scale

authorized_keys requires your public key to exist on every target server. Scaling this creates:

a separate deployment step for key distribution during onboarding
no centralized revocation — removing a key means touching each host
key rotation touches every machine
no expiration means stale access accumulates

An SSH certificate is a CA signature on your public key. The server only needs to trust the CA — your key never needs to be present locally.

systemd-timer: scheduling instead of cron

Tue, 01 Sep 2026 00:00:00 +0000

cron works, but its logs are flat text files with no structure, and service dependencies require workarounds like embedding Requires= logic inside shell scripts. systemd-timer fixes this: unified management interface, logs in journald, dependencies through the familiar After= and WantedBy= directives — all in one stack.

Structure: .service and .timer

A timer is a separate unit that triggers a .service. The separation is intentional: the service can be invoked manually or on a schedule.

Cron setup: a practical walkthrough

Mon, 31 Aug 2026 00:00:00 +0000

Cron is the standard job scheduler in Linux, found in every infrastructure. Jobs pile up, logs accumulate, and the environment breaks things. Let’s walk through how to configure cron reliably and where it falls short.

When to Use Cron and When to Avoid It

Cron fits simple recurring tasks: backups, log rotation, temp file cleanup, periodic notifications. It’s a daemon that sleeps between runs — no resource consumption.

GNU Screen: sessions that survive an SSH drop

Mon, 31 Aug 2026 00:00:00 +0000

A long apt upgrade, a migration, a build — then the laptop sleeps. SSH dies, the process gets SIGHUP and dies with it. GNU Screen keeps the terminal on the server: disconnect, come back, the work is still there.

It is not a nohup replacement and not “another SSH”. It is a multiplexer: named sessions, several windows inside one, a shared session for two people. On older RHEL/Debian boxes screen is often already installed when tmux is not.

Kafka: Cluster Health Check

Mon, 31 Aug 2026 00:00:00 +0000

A Kafka cluster in KRaft mode doesn’t forgive neglect until the first incident. Health checks need to be regular and quick — no graphs or dashboards, just the terminal. Here’s the command set that covers the typical checklist: processes, quorum, partition leaders, ISR, and a quick status report in one shot.

Checking KRaft Processes

KRaft mode has no separate ZooKeeper — the controller role is either co-located with the broker or isolated on dedicated nodes. First, verify the JVM processes are alive and see which mode each node started in.

kind: Local Kubernetes in Docker

Mon, 31 Aug 2026 00:00:00 +0000

kind creates a Kubernetes cluster from Docker containers: control-plane and worker nodes are kindest/node images. Primary use cases are local development and CI. GitHub Actions has an official create-kind action that makes pipelines with K8s tests straightforward.

Compared to minikube, kind has no hypervisor dependency and natively supports multi-node topologies. Compared to k3d, it requires no Rancher and talks to CRI/containerd directly.

Installation

macOS, Linux, and Windows (via WSL2) — download the binary from GitHub Releases.

Too many authentication failures: SSH ran out of tries

Mon, 31 Aug 2026 00:00:00 +0000

Received disconnect from 10.0.0.5 port 22:2: Too many authentication failures followed by Permission denied (publickey) is not a broken server, and it is not necessarily a wrong password. The client spent the attempt budget while walking agent keys and never reached the method you meant to use.

The budget is MaxAuthTries on the server (6 by default). Each public key offered is one try. Five keys in ssh-agent plus one more wrong key — the connection dies before the right key or a password prompt.

Trusting a custom CA: system store, browsers, and CLI

Mon, 31 Aug 2026 00:00:00 +0000

A TLS error that says the certificate is untrusted almost never means the certificate is “broken”. The trust anchor is in the wrong store.

curl, openssl, Git, Python, and the browser are different clients. Some keep their own root lists. Updating the OS bundle on Linux will not fix Chrome or Firefox by itself.

What to import

You trust the CA root that signed the server certificate, not localhost.crt / app.example.internal itself.

ssh-connection-manager: a TUI for hosts in ~/.ssh/config

Sun, 30 Aug 2026 00:00:00 +0000

When ~/.ssh/config holds dozens of stands, bastions, and jump hosts, memorizing aliases stops being fun. ssh-connection-manager is a TUI on top of ordinary OpenSSH: a host list, a filter, a connect via the system ssh, and a way to append a new block to the config.

The CLI command is ssh-connect. Repository: gitlab.com/unsorted-projects/ssh-connection-manager.

Why not another SSH client

The client already exists: the system ssh. What is missing is navigation over the config file.