Devops
52 pagesfixing memory leaks in python services: diagnostics and dump collection
Python services under load can silently consume memory until the cgroup limit triggers an OOM kill. Without systematic dump collection and introspection, root-cause analysis devolves into hypothesis spinning. Below is a practical set of commands and …
Python services under load can silently consume memory until the cgroup limit triggers an OOM kill. Without systematic dump collection and introspection, root-cause analysis devolves into hypothesis spinning. Below is a practical set of commands and …
kubectl cheatsheet: essential commands for Kubernetes
kubectl is the primary interface to a Kubernetes cluster. This cheatheet covers routine operations — from context setup to pod debugging and namespace switching. All commands are verified against current kubectl versions (1.28+). Installation and …
kubectl is the primary interface to a Kubernetes cluster. This cheatheet covers routine operations — from context setup to pod debugging and namespace switching. All commands are verified against current kubectl versions (1.28+). Installation and …
Git Tag: Marking Releases and Bookmarks in History
Git Tag: Marking Releases and Bookmarks in History Tags are Git’s mechanism for assigning meaningful labels to specific commits. Unlike branches, tags don’t move — they’re pinned to a commit and serve as anchors for releases, versions, and critical …
Git Tag: Marking Releases and Bookmarks in History Tags are Git’s mechanism for assigning meaningful labels to specific commits. Unlike branches, tags don’t move — they’re pinned to a commit and serve as anchors for releases, versions, and critical …
Creating User and Role in Kubernetes and Binding Them via RBAC
In Kubernetes there are no “users” in the traditional sense — there are ServiceAccounts and certificates, bound to roles through RBAC. Without proper configuration, anyone holding a kubeconfig gets full access to the cluster. Below is the complete …
In Kubernetes there are no “users” in the traditional sense — there are ServiceAccounts and certificates, bound to roles through RBAC. Without proper configuration, anyone holding a kubeconfig gets full access to the cluster. Below is the complete …
ulimit and systemd LimitNOFILE — why ulimit -n inside a unit doesn't stick
What is nofile and where it lives nofile is the maximum number of open file descriptors per process. That’s not just regular files — it covers sockets, pipes, stdin/stdout/stderr, logs shipped through journald — everything counts. When nginx or a Go …
What is nofile and where it lives nofile is the maximum number of open file descriptors per process. That’s not just regular files — it covers sockets, pipes, stdin/stdout/stderr, logs shipped through journald — everything counts. When nginx or a Go …
Deploying with a post-receive Git Hook
Deploying through CI is great, but sometimes you just need to push code to a server with a single git push. The post-receive hook in a bare repository handles this without extra dependencies: push to the server, and the hook automatically checks out …
Deploying through CI is great, but sometimes you just need to push code to a server with a single git push. The post-receive hook in a bare repository handles this without extra dependencies: push to the server, and the hook automatically checks out …
ThinLinc: Remote Access to Linux Desktops
ThinLinc: Remote Access to Linux Desktops In enterprise environments, remote access to Linux desktops often comes down to VNC with flaky encryption or RDP proxies held together with workarounds. ThinLinc by Cendio is a full-featured remote desktop …
ThinLinc: Remote Access to Linux Desktops In enterprise environments, remote access to Linux desktops often comes down to VNC with flaky encryption or RDP proxies held together with workarounds. ThinLinc by Cendio is a full-featured remote desktop …
Sudoers: NOPASSWD Without Holes
Unrestricted NOPASSWD in sudoers is a misconfiguration that grants root access without a password, turning any user script or library vulnerability into a full system compromise. The correct approach limits NOPASSWD to specific commands only. Why …
Unrestricted NOPASSWD in sudoers is a misconfiguration that grants root access without a password, turning any user script or library vulnerability into a full system compromise. The correct approach limits NOPASSWD to specific commands only. Why …
scp — Secure Copy Over SSH
scp — a utility for copying files over SSH using the SSH protocol. It works from the terminal, requires no extra server setup — just a running sshd and working authentication. In an era of rsync and bat, SCP survives as a simple tool for one-off …
scp — a utility for copying files over SSH using the SSH protocol. It works from the terminal, requires no extra server setup — just a running sshd and working authentication. In an era of rsync and bat, SCP survives as a simple tool for one-off …
Docker logs and journald: choosing a logging driver
When a container crashes, logs are the first thing you need to see. docker logs looks simple, but under the hood different logging drivers are at work, and the choice affects how logs are stored, rotated, and accessed. Here is what you should know …
When a container crashes, logs are the first thing you need to see. docker logs looks simple, but under the hood different logging drivers are at work, and the choice affects how logs are stored, rotated, and accessed. Here is what you should know …
curl --resolve and SNI: Testing Virtual Hosts Without /etc/hosts
When you need to test a virtual host on a specific IP but don’t want to edit /etc/hosts — whether due to permissions, conflicts with other services, or just the habit of keeping the file clean — curl --resolve solves both problems at once: it …
When you need to test a virtual host on a specific IP but don’t want to edit /etc/hosts — whether due to permissions, conflicts with other services, or just the habit of keeping the file clean — curl --resolve solves both problems at once: it …
coredumpctl: Finding a Binary Crash
What is coredumpctl and how it works When a binary crashes with SEGV, the kernel can save a core dump — a snapshot of process memory at the moment of the crash. In systemd-based distributions, coredumpctl handles collecting, storing, and searching …
What is coredumpctl and how it works When a binary crashes with SEGV, the kernel can save a core dump — a snapshot of process memory at the moment of the crash. In systemd-based distributions, coredumpctl handles collecting, storing, and searching …
nftables: Basic Rule Set
Note All commands were verified on Debian/Ubuntu with the nftables package and on RHEL/CentOS 8+. On older systems you may need apt install nftables or yum install nftables. nftables replaced iptables, but documentation for a basic rule set is often …
Note All commands were verified on Debian/Ubuntu with the nftables package and on RHEL/CentOS 8+. On older systems you may need apt install nftables or yum install nftables. nftables replaced iptables, but documentation for a basic rule set is often …
journalctl: filters and follow
Systemd’s journal is the first place to look when a service crashes or a node starts burning CPU. journalctl does far more than dump the entire log in sequence: it can filter by units, priorities, time windows, and stream in real time. Below is the …
Systemd’s journal is the first place to look when a service crashes or a node starts burning CPU. journalctl does far more than dump the entire log in sequence: it can filter by units, priorities, time windows, and stream in real time. Below is the …
fail2ban: SSH Jail Configuration
Securing SSH against brute-force attacks is one of the first steps in hardening any server. fail2ban scans logs, detects repeated failed login attempts, and blocks the source via iptables or nftables. This note covers the sshd jail — from …
Securing SSH against brute-force attacks is one of the first steps in hardening any server. fail2ban scans logs, detects repeated failed login attempts, and blocks the source via iptables or nftables. This note covers the sshd jail — from …
Chrony Instead of ntpd
Chrony has replaced ntpd as the default NTP client in most modern Linux distributions. It converges to accurate time faster, handles intermittent network connections better, and consumes fewer resources. If your machine still runs ntpd, switching …
Chrony has replaced ntpd as the default NTP client in most modern Linux distributions. It converges to accurate time faster, handles intermittent network connections better, and consumes fewer resources. If your machine still runs ntpd, switching …
Debian — The Swiss Army Knife of Linux
Debian isn’t the flashiest distribution, but it’s the most reliable foundation in the Linux world. Behind it stands the largest community of volunteer developers, and behind its track record are decades of uninterrupted operation on servers, embedded …
Debian isn’t the flashiest distribution, but it’s the most reliable foundation in the Linux world. Behind it stands the largest community of volunteer developers, and behind its track record are decades of uninterrupted operation on servers, embedded …
pipx: Isolated Python CLI Tools Without the Mess
pipx solves a simple but chronic problem: you need to run a Python utility once or occasionally, and pip install pollutes the global environment or leaves behind a virtual environment you forget to clean up. pipx creates an isolated venv for each …
pipx solves a simple but chronic problem: you need to run a Python utility once or occasionally, and pip install pollutes the global environment or leaves behind a virtual environment you forget to clean up. pipx creates an isolated venv for each …
uv — Fast Python Package Manager
What is uv uv is a Python package manager written in Rust. It solves one problem: the standard pip is slow at resolving dependencies, and poetry adds its own project model on top of PEP 621. uv works with pyproject.toml, is compatible with PEP 621 …
What is uv uv is a Python package manager written in Rust. It solves one problem: the standard pip is slow at resolving dependencies, and poetry adds its own project model on top of PEP 621. uv works with pyproject.toml, is compatible with PEP 621 …
tmux on Prod After Screen
Why We Switched from Screen to tmux Screen was our primary tool for about five years. After migrating the cluster to new servers it became obvious: screen drops sessions on SSH disconnect when hardstatus isn’t configured, and screen -r recovery …
Why We Switched from Screen to tmux Screen was our primary tool for about five years. After migrating the cluster to new servers it became obvious: screen drops sessions on SSH disconnect when hardstatus isn’t configured, and screen -r recovery …
Setting Up Your Own SSH Bastion Server
Why You Need a Bastion and Where It Lives A bastion is the single entry point into a private network segment. Instead of exposing SSH on every server to the internet, you funnel traffic through one hardened host with a strict access policy. Typical …
Why You Need a Bastion and Where It Lives A bastion is the single entry point into a private network segment. Instead of exposing SSH on every server to the internet, you funnel traffic through one hardened host with a strict access policy. Typical …
Rsync: Backing Up a Directory Over SSH
Rsync: Backing Up a Directory Over SSH The classic way to copy a directory to a remote machine is rsync over SSH. No extra ports to open, traffic is encrypted, and the tool itself handles incremental transfers and metadata preservation. One command …
Rsync: Backing Up a Directory Over SSH The classic way to copy a directory to a remote machine is rsync over SSH. No extra ports to open, traffic is encrypted, and the tool itself handles incremental transfers and metadata preservation. One command …
logrotate for Custom Daemons
Log rotation is missing for your daemon, the log file has grown to dozens of gigabytes, the disk is full, and monitoring is screaming. systemd-journald and syslog-ng rotate on their own, but if your custom daemon writes directly to a file, rotation …
Log rotation is missing for your daemon, the log file has grown to dozens of gigabytes, the disk is full, and monitoring is screaming. systemd-journald and syslog-ng rotate on their own, but if your custom daemon writes directly to a file, rotation …
ip: Network Setup and Diagnostics in CLI
When ifconfig returns nothing and configuring a route requires a separate command, that’s not a system bug. It’s iproute2 — the package that replaced net-tools in modern Linux distributions. The ip utility from iproute2 is the standard interface for …
When ifconfig returns nothing and configuring a route requires a separate command, that’s not a system bug. It’s iproute2 — the package that replaced net-tools in modern Linux distributions. The ip utility from iproute2 is the standard interface for …
ProxyJump and bastion hosts via ~/.ssh/config
Sometimes a server sits in a private network with no public IP. The only entry point is a bastion host with a public address. Typing ssh -J user@bastion user@private every time gets old fast. Here’s how to configure everything in ~/.ssh/config so you …
Sometimes a server sits in a private network with no public IP. The only entry point is a bastion host with a public address. Typing ssh -J user@bastion user@private every time gets old fast. Here’s how to configure everything in ~/.ssh/config so you …
journalctl: Filtering and Formatting systemd Logs
Logs disappeared. Server rebooted, and the familiar less /var/log/syslog returns nothing. On modern distros with systemd, logs are collected by journald and read with journalctl. Without knowing its filters, system debugging turns into guesswork. Why …
Logs disappeared. Server rebooted, and the familiar less /var/log/syslog returns nothing. On modern distros with systemd, logs are collected by journald and read with journalctl. Without knowing its filters, system debugging turns into guesswork. Why …
systemd-run: Run Services Without Unit Files
Sometimes you need to run a process under systemd’s control without writing a unit file — maybe you’re in a container without systemd, on someone else’s machine, or just need a quick one-off. That’s where systemd-run comes in. Why systemd-run The …
Sometimes you need to run a process under systemd’s control without writing a unit file — maybe you’re in a container without systemd, on someone else’s machine, or just need a quick one-off. That’s where systemd-run comes in. Why systemd-run The …
sshd_config: baseline for a test stand
SSH access to a test stand often gets opened in a hurry, and then the logs fill with brute-force attempts. A baseline sshd_config that blocks common attack vectors fits into five parameters and twenty minutes. Why Change Defaults …
SSH access to a test stand often gets opened in a hurry, and then the logs fill with brute-force attempts. A baseline sshd_config that blocks common attack vectors fits into five parameters and twenty minutes. Why Change Defaults …
logrotate: automatic log rotation and archiving
Application logs fill up disk space within a week, and manually running rm *.log is a recipe for trouble. logrotate handles this automatically: it rotates, compresses, and deletes old files on a schedule. Let’s see how it works and how to set it up …
Application logs fill up disk space within a week, and manually running rm *.log is a recipe for trouble. logrotate handles this automatically: it rotates, compresses, and deletes old files on a schedule. Let’s see how it works and how to set it up …
mc — MinIO Client S3 CLI
S3-compatible object storage is the default choice for buckets, backups, and static assets. When AWS CLI feels excessive and the web console is too clunky, MinIO Client (mc) fills the gap. This CLI tool works with any S3-compatible storage: MinIO, …
S3-compatible object storage is the default choice for buckets, backups, and static assets. When AWS CLI feels excessive and the web console is too clunky, MinIO Client (mc) fills the gap. This CLI tool works with any S3-compatible storage: MinIO, …
ethtool: network interface diagnostics and tuning
Network issues hide well — interface is up, IP is assigned, iptables is quiet, yet packet loss or micro-freezes only surface under load. ethtool gives direct access to hardware state, driver behavior, and offload mechanisms that neither ip nor …
Network issues hide well — interface is up, IP is assigned, iptables is quiet, yet packet loss or micro-freezes only surface under load. ethtool gives direct access to hardware state, driver behavior, and offload mechanisms that neither ip nor …
curl: HTTP Debugging in CLI
cURL is the standard tool for debugging HTTP in the terminal. It ships out of the box on Linux and macOS and is present in most Docker images. Need to quickly check an API, inspect response headers, or trace a redirect issue — one command line is …
cURL is the standard tool for debugging HTTP in the terminal. It ships out of the box on Linux and macOS and is present in most Docker images. Need to quickly check an API, inspect response headers, or trace a redirect issue — one command line is …
tcpdump and tshark: Packet Capture in CLI
When debugging network issues in Linux infrastructure, ping and curl are not enough. Sometimes you need to see what is actually traveling over the wire. tcpdump is the standard tool for capturing packets from the CLI. tshark is its sibling from the …
When debugging network issues in Linux infrastructure, ping and curl are not enough. Sometimes you need to see what is actually traveling over the wire. tcpdump is the standard tool for capturing packets from the CLI. tshark is its sibling from the …
SSH Config: Wildcards and Dynamic Variable Substitution
SSH reads ~/.ssh/config line by line, but without variables the file quickly becomes copy-paste hell. Here’s how Host patterns, Match exec, and substitution tokens like %h, %r, %l cut config size by orders of magnitude while covering real scenarios — …
SSH reads ~/.ssh/config line by line, but without variables the file quickly becomes copy-paste hell. Here’s how Host patterns, Match exec, and substitution tokens like %h, %r, %l cut config size by orders of magnitude while covering real scenarios — …
What is Self-Hosted and Why It's So Popular
You’re paying for Notion, Dropbox, and Google Drive. Then Notion raises prices, Dropbox caps storage, and Google “improves” the Docs interface. Self-hosted is taking infrastructure into your own hands and breaking the vendor dependency loop. …
You’re paying for Notion, Dropbox, and Google Drive. Then Notion raises prices, Dropbox caps storage, and Google “improves” the Docs interface. Self-hosted is taking infrastructure into your own hands and breaking the vendor dependency loop. …
SSH Escape Sequences: Reviving a Frozen Terminal
SSH session froze, Ctrl+C does nothing, Ctrl+D spits out garbage — familiar situation. Before closing the terminal and losing the session, try built-in escape sequences. They operate at the SSH client level before data reaches the remote host. How to …
SSH session froze, Ctrl+C does nothing, Ctrl+D spits out garbage — familiar situation. Before closing the terminal and losing the session, try built-in escape sequences. They operate at the SSH client level before data reaches the remote host. How to …
socat: Forwarding Unix Sockets Over TCP
Sometimes you need to reach a Unix socket from a host where that socket doesn’t physically exist. SSH tunnels won’t help — they only work with TCP ports. socat solves this: it opens a TCP listener and forwards connections to a Unix socket, and the …
Sometimes you need to reach a Unix socket from a host where that socket doesn’t physically exist. SSH tunnels won’t help — they only work with TCP ports. socat solves this: it opens a TCP listener and forwards connections to a Unix socket, and the …
Creating a Custom Systemd Service
Your application needs to start on boot, restart on crash, and log output. Shell scripts in /etc/rc.local give you none of that. Systemd solves all three with a single declarative file. Why Write a Custom Unit File Supervisord and init scripts are …
Your application needs to start on boot, restart on crash, and log output. Shell scripts in /etc/rc.local give you none of that. Systemd solves all three with a single declarative file. Why Write a Custom Unit File Supervisord and init scripts are …
cockpit-ufw-module: Uncomplicated Firewall in Cockpit
UFW on a home or small server is usually configured over SSH: ufw status numbered, then ufw allow 443/tcp. cockpit-ufw-module covers the same cycle in the browser: package, status, policies, rules. It is one panel from the cockpit-modules group — UFW …
UFW on a home or small server is usually configured over SSH: ufw status numbered, then ufw allow 443/tcp. cockpit-ufw-module covers the same cycle in the browser: package, status, policies, rules. It is one panel from the cockpit-modules group — UFW …
systemd-timer: scheduling instead of cron
cron works, but its logs are flat text files with no structure, and service dependencies require workarounds like embedding Requires= logic inside shell scripts. systemd-timer fixes this: unified management interface, logs in journald, dependencies …
cron works, but its logs are flat text files with no structure, and service dependencies require workarounds like embedding Requires= logic inside shell scripts. systemd-timer fixes this: unified management interface, logs in journald, dependencies …
SSH certificates instead of authorized_keys
authorized_keys works fine for a handful of servers. Once you hit a dozen, it becomes a liability. Onboarding a new developer means manually distributing their public key across every machine. SSH certificates flip this model: one CA signs all public …
authorized_keys works fine for a handful of servers. Once you hit a dozen, it becomes a liability. Onboarding a new developer means manually distributing their public key across every machine. SSH certificates flip this model: one CA signs all public …
Squid: Internet Forwarding to Remote VM
Your VM in the cloud has no public IP or internet access is blocked via NAT, but the deployment needs wget/curl from inside. Squid on an intermediate host with a decent uplink solves this in ten minutes. Why This Is Needed I forward internet through …
Your VM in the cloud has no public IP or internet access is blocked via NAT, but the deployment needs wget/curl from inside. Squid on an intermediate host with a decent uplink solves this in ten minutes. Why This Is Needed I forward internet through …
MkDocs: Project Documentation from Markdown
Documentation in a repository ages faster than anyone reads it: README links lead nowhere, sections are scattered across docs/, wiki/, and Confluence, and site search doesn’t work. MkDocs solves this predictably — it takes a folder of .md files and …
Documentation in a repository ages faster than anyone reads it: README links lead nowhere, sections are scattered across docs/, wiki/, and Confluence, and site search doesn’t work. MkDocs solves this predictably — it takes a folder of .md files and …
dig: DNS Query Debugging in CLI
DNS resolvers return the wrong address, clients don’t see updates, or it’s unclear which server is handling requests. dig (Domain Information Groper) is the standard CLI tool for DNS diagnostics. Works on Linux, macOS, and Windows via WSL. …
DNS resolvers return the wrong address, clients don’t see updates, or it’s unclear which server is handling requests. dig (Domain Information Groper) is the standard CLI tool for DNS diagnostics. Works on Linux, macOS, and Windows via WSL. …
cockpit-modules: web panels for day-to-day operations
Cockpit covers basic Linux administration in the browser: services, logs, networking, accounts. Firewall, fail2ban, cron, and Let’s Encrypt sit outside that set — either there is no panel, or it is too generic. The cockpit-modules group is a set of …
Cockpit covers basic Linux administration in the browser: services, logs, networking, accounts. Firewall, fail2ban, cron, and Let’s Encrypt sit outside that set — either there is no panel, or it is too generic. The cockpit-modules group is a set of …
Trusting a custom CA: system store, browsers, and CLI
A TLS error that says the certificate is untrusted almost never means the certificate is “broken”. The trust anchor is in the wrong store. curl, openssl, Git, Python, and the browser are different clients. Some keep their own root lists. Updating the …
A TLS error that says the certificate is untrusted almost never means the certificate is “broken”. The trust anchor is in the wrong store. curl, openssl, Git, Python, and the browser are different clients. Some keep their own root lists. Updating the …
Too many authentication failures: SSH ran out of tries
Received disconnect from 10.0.0.5 port 22:2: Too many authentication failures followed by Permission denied (publickey) is not a broken server, and it is not necessarily a wrong password. The client spent the attempt budget while walking agent keys …
Received disconnect from 10.0.0.5 port 22:2: Too many authentication failures followed by Permission denied (publickey) is not a broken server, and it is not necessarily a wrong password. The client spent the attempt budget while walking agent keys …
kind: Local Kubernetes in Docker
kind creates a Kubernetes cluster from Docker containers: control-plane and worker nodes are kindest/node images. Primary use cases are local development and CI. GitHub Actions has an official create-kind action that makes pipelines with K8s tests …
kind creates a Kubernetes cluster from Docker containers: control-plane and worker nodes are kindest/node images. Primary use cases are local development and CI. GitHub Actions has an official create-kind action that makes pipelines with K8s tests …
Kafka: Cluster Health Check
A Kafka cluster in KRaft mode doesn’t forgive neglect until the first incident. Health checks need to be regular and quick — no graphs or dashboards, just the terminal. Here’s the command set that covers the typical checklist: processes, quorum, …
A Kafka cluster in KRaft mode doesn’t forgive neglect until the first incident. Health checks need to be regular and quick — no graphs or dashboards, just the terminal. Here’s the command set that covers the typical checklist: processes, quorum, …
GNU Screen: sessions that survive an SSH drop
A long apt upgrade, a migration, a build — then the laptop sleeps. SSH dies, the process gets SIGHUP and dies with it. GNU Screen keeps the terminal on the server: disconnect, come back, the work is still there. It is not a nohup replacement and not …
A long apt upgrade, a migration, a build — then the laptop sleeps. SSH dies, the process gets SIGHUP and dies with it. GNU Screen keeps the terminal on the server: disconnect, come back, the work is still there. It is not a nohup replacement and not …
Cron setup: a practical walkthrough
Cron is the standard job scheduler in Linux, found in every infrastructure. Jobs pile up, logs accumulate, and the environment breaks things. Let’s walk through how to configure cron reliably and where it falls short. When to Use Cron and When to …
Cron is the standard job scheduler in Linux, found in every infrastructure. Jobs pile up, logs accumulate, and the environment breaks things. Let’s walk through how to configure cron reliably and where it falls short. When to Use Cron and When to …
ssh-connection-manager: a TUI for hosts in ~/.ssh/config
When ~/.ssh/config holds dozens of stands, bastions, and jump hosts, memorizing aliases stops being fun. ssh-connection-manager is a TUI on top of ordinary OpenSSH: a host list, a filter, a connect via the system ssh, and a way to append a new block …
When ~/.ssh/config holds dozens of stands, bastions, and jump hosts, memorizing aliases stops being fun. ssh-connection-manager is a TUI on top of ordinary OpenSSH: a host list, a filter, a connect via the system ssh, and a way to append a new block …