Quick Access to my selection Last Update: 2020-09-21 22:47:40 IST Unix epoch: 1600724860
Cloud Computing Operations Engineering/DevOps/SRE Engineering Miscellaneous
Back End Programming Commodity Hardware ZFS Architecture
Python Open Source Utilities Bash Open Source Utilities My Tech Talks My Books
  • count_repeated_pattern_in_logs.sh
    A easy way to see errors that are repeating, e.g.: NFS/iSCSI timeouts.
    2020-May Views: 911 views
  • backup_partition_in_files.sh
    Compressing an unmounted partition to a image file while compressing on the fly, and breaking into 1GB gz files.
    Also explains in a funny way about STDIN, STDOUT, STDERR and methodology investigating in deep.
    2020-May Views: 1,171 views
  • iostat_bandwidth.sh
    See the aggregated bandwidth used by all the drives, and the maximum speed achieved.
    2020-Aug Views: 343 views

How to recover access to your Amazon AWS EC2 instance if you loss your Private Key for SSH

This article covers the desperate situation where you had generated one or more instances, instructed Amazon to use a SSH Key Pair certs where only you have the Private Key, your instances are running, for example, an eCommerce site, running for months, and then you loss your Private Key (.pem file), and with it the SSH access to your instances’ Data.

Actually I’ve seen this situation happening several times, in actual companies. Mainly Start ups. And I solved it for them.

Assuming that you didn’t have a secondary method to access, which is another combination of username/password or other user/KeyPairs, and so you completely lost the access to the Database, the Webservers, etc… I’m going to show you how to recover the data.

For this article I will consider an scenario where there is only one Instance, which contains everything for your eCommerce: Webserver, code, and Database… and is a simple config, with a single persistent drive.

Warning: be very careful as if you use ephemeral drives, contents will be lost is you power off the instance.

Method 1: Quicker, launching a new instance from the previous

Step1: The first step you will take is to close the access from outside, using the Firewall, to avoid any new changes going to the disk. You can allow access to the instance only from your static Ip in the office/home.

Step 2: You’ll wait for 5 minutes to allow any transaction going on to conclude, and pending writes to be flushed to disk.

Step 3: From Amazon AWS Console, EC2, you’ll request an Snapshot. That step is to try to get extra security. Taking an Snapshot from a live, mounted, filesystem, is not the best of ideas, specially of a Database, but we are facing a desperate situation so we’re increasing the numbers of leaving this situation without Data loss. This is just for extra security and if everything goes well at the end you will not need this snapshot.

Make sure you select No reboot.

Step 4: Be very careful if you have extra drives and ephemeral drives.

Step 5: Wait till the Snapshot completes.

Step 6: Then request a graceful poweroff. Amazon will try to poweroff the Server in a gentle way. This may take two minutes.

Step 7: When the instance is powered off, request a new Snapshot. This is the one we really want. The other was just to be more safe. If you feel confident you can just unclick No Reboot on the previous Step and do only one Snapshot.

Step 8: Wait till the Snapshot completes.

Step 9: Generate and upload the new key you will use to AWS Console, or ask Amazon to generate a key pair for you. You can do it while creating the new instance through the wizard.

Step 10: Launch a new instance, based on your snapshot AMI. This will generate a copy of your previous instance (using the Snapshot) for the new one. Select the new Key pair. Finish assigning the Security groups, the elastic ip…

Step 11: Start the new instance. You can select a different flavor, like a more powerful instance, if you prefer. (scale vertically)

Step 12: Test your access by login via SSH with the new pair keys and from your static Ip which has access in the Firewall.

ssh -i /home/carles/Desktop/Data/keys/carles-ecommerce.pem ubuntu@

Step 13: Check that the web Starts correctly, check the Database logs to see if there is any corruption. Should not have any if graceful shutdown went well.

Step 14: Reopen the access from the Firewall, so the world can connect to your instance.

Method 2: Slower, access the Data and rebuild whatever you need

The second method is exactly the same until Step 6 included.

Step 7: After this, you will create a new instance based on your favorite OS, with a new pair of Keys.

Step 8: You’ll detach the Volume from the eCommerce previous instance (the one you lost access).

Step 9: You’ll attach the Volume to the new instance.

Step 10: You’ll have access to the Data from the previous instance in the new volume. type cat /proc/partitions or df -h to see the mountpoints available. You can then download or backup, or install the Software again and import the Database…

Step 11: Check that everything works, and enable the access worldwide to the Web in the Firewall (Security Group Inbound Rules).

If you are confident enough, you can use this method to upgrade the OS or base Software of your instance, making it part of your maintenance window. For example, to get the last version of Ubuntu or CentOS, MySQL, Python or PHP, etc…

Programs I use for Windows in my Workstations

I love Linux and Linux tools and I’m a big fan of it, using it for Servers since 1995.

However some companies use Windows for the Workstations, and that’s not necessarily bad.

So I describe here the tools I use to maximize productivity.


That really depends on my employers. I’ve my opinion about several of them.

Apache Directory Studio

For working with LDAP.


This is a Commercial Software to capture video. I record bugs, tutorials, internal web training sessions…

Docker Desktop for Windows

But not in the laptops cause the hyper-v may conflict with the BitLocker drive encryption and may cause the entire drive to be lost.


Very powerful for doing replacement over large CSV files.





Free Database Manager for MariaDB, MySQL, SQLite, PostgreSQL and Microsoft SQL.

Is compatible with Wine, so you can use it on Linux.



With zmodem, sftp, SSH, tunnel….


A MySQL debugger. Specially useful for Stored Procedures.

Is compatible with Wine, so you can use it on Linux.


https://obsproject.com/ is a screen recorder Software, for Linux, Mac Os X and Windows.

One Note


Specially useful the option of using a VPN.



Remote Desktop Connection



Video editor

Yes, I always have a Linux VM.


For the Diagrams.


Video player

VMWare Player


Normally RealVNC.



News from the blog 2020-08-19

  • I assisted to the OpenZFS leadership meeting.

I tried to continue following it since I left Sanmina. ZFS is really an amazing Software and it’s lead by an amazing Community of super cool Engineers and companies. I would like to continue contributing ASAP.

I bought some new hard drives in order to work a bit on this. You don’t need to have dedicated hardware if you want to test features. You can run in a VirtualBox or VMWare Workstation.

  • I received more books about DevOps and Python

None is perfect. I see flaws in all of them and bad architecture practices*, however from all I learn interesting things.

*I guess that’s why I wrote my own book :)

You know, I study every day. At least 30 minutes, after work. As part of my healthy routines.

But I also study and learn during the work, as we have time available for this.

I’m very fortunate that Blizzard gives me time every day to study. That’s amazing. They also send us to events paying the ticket, travel, hotel, expenses… now with covid-19 we only go to virtual events, but the company still pay for this and give free days. Is a very nice company.

I use a lot Linux Academy too:

I continue having purchases of my book, and I’m very happy about that. I’m working on improving it and providing more contents and samples going from the scratch, with step by step code samples. From spaghetti code reading CSV files, to OOP with Full Coverage.

  • My application for a Higher degree Computer Science Cloud Computing (Level 8) has been accepted. The Irish government pays me 90% of the degree, and Blizzard will pay me the other 10% after I pass the first year course.

I’m really grateful to this beautiful country, Ireland.

Having an Irish degree is something that brings me an special illusion.

  • I have updated CTOP.py with some interesting features

It allows to pass a fixed width and height for the terminal render. That’s very useful when you run CTOP in a Docker non interactive session, or from a Cron, with the –iterations=1 so the output can be captured programmatically.

  • Jetbrains has provided me with a Free License of all their products, in order to support my work in Open Source projects. That’s very nice. I’m using now mainly PyCharm and PhpStorm.
  • At the beginning of the covid-19 I wrote a simulator in Python. That’s why I was able to anticipate that the number of cases and deaths would be very much higher when nobody around me knew what was going to happen. My first simulations were simple, and the algorithms were growing in complexity until I had a full rich Object Oriented modeler. Maybe I’ll write an article about this someday.
    • I based my data in https://www.worldometers.info/coronavirus/#countries
    • I studied the evolution of several countries and I was working with simulations in Spain until their government started blocking the information and stop providing transparent and accurate metrics.
  • I’m seeing how the covid is affecting and transforming several kind of business:
    • Meetup.com I see meetups with more than 1,000 users closing, as they are no meeting anymore
    • Airlines, obviously
    • Hotels, offering less services
    • Metasearchers and OTAs (Online travel agencies)
    • I can imagine the impact on airbnb
    • Discos, nightclubs are closing doors
    • Restaurants, they will lose the Christmas season (with families and companies doing lunch and dinners)
  • At the same time, other companies are hitting records in sales
  • After doing a Masterclass to some colleagues about Refactor, Code Reliability, Quality, The non-happy path and Unit Testing, I’m preparing some contents that I’ll publish to the Community soon. So far I created this repo, where I added the source code for lesson 0: starting to program in Python videos that I created few months ago to help beginners.


I also added some contents to lesson 1, where we refactor pure spaghetti code with no error control, to something more elaborated with unit testing and full code coverage. Still procedural, but I will jump to next class in two weeks, where we will move to OOP and Dependency Injection.

  • Here my “Harley” assembled :)


If you use ZFS with spinning drives and you share iSCSI, you will need to use a SLOG device for ZIL otherwise you’ll see your iSCSI connections interrupted.

What is a ZIL?

  • ZIL: Acronym for ZFS Intended Log. Logs synchronous operations to disk
  • SLOG: Acronym for (S)eperate (LOG) Device

In ZFS Data is first written and stored in-memory, then it’s flushed to drives. This can take 10 seconds or more.

So without SLOG it can happen that if a power loss occurs, you may loss the last 10 seconds of Data submitted.

The SLOG device brings security that if there is a power loss, after remounting the pool, the information in the SLOG, acknowledged to iSCSI clients, is not lost and flushed to the Hard drives conforming the pool. Basically this device keeps the writings that come from network and flushes to the Hard drives and then clears this data from the SLOG.

Normally I’m describing configurations with a fast device for SLOG ZIL, like one or a pair of NVMe drive or SAS SSD, most commonly in mirror a pool of 12 HDD drives or more SAS preferentially, maybe SATA, with 14TB or more each.

As the SLOG device will persist your Data if there is a power off, and submit to the pool the accepted transactions, it is clear that you cannot spare your SLOG ZIL device. It is needed to bring security when remotely writing.

But what happens if we have a kind of business where we don’t care about that the last 10 seconds writings may be lost? (ZFS will never get corrupted due to its kinda journal system), just because we are filling a Server the fastest possible, migrating from another, or because we are running workouts that can be retaken is some data is lost… do we really need to have the speed constrain of an SSD?. Examples are a Hadoop node, or a SETI@Home client. Tasks will be resumed if something failed.

Let me put in another way, we have 2 NIC 100Gbps, in bonding, so 200Gbps (equivalent to (25GB/s Gigabytes per second), 90 HDD drives that can work in parallel up to 250 MB/s each (22.5GB/s) and our Server has a pair or SAS SSD ZIL in mirror, that writes at 900 MB/s (Megabytes per second, so 0.9 GB/s), so our bottleneck or constraint is the SLOG ZIL.

Adding one RAMDISK, or better two RAMDISKs in mirror, we can get to much more highers speeds. I cannot tell you how much, but in my tests with regular configurations (8D+3P) I was achieving more than 2 GB (Gigabytes) per second sustained of Data to the pool. Take in count that the speed writing to the pool does not only depend on the speed on the ZIL, and the speed of the HDD spinning drives (slow, between 100 and 250 MB/s), but also about the config of the pool (number of vdevs, distributions of data and parity drives) and the throughput of your IOC (Input Output Controller), and the number of them.

Live real scenarios use to be more in the line of having 2x10GbpE cards, combined in bonding making 20Gbps, so being able to transmit 2.5GB/s. So to get the max speed of our Network this Ramdrive will do it. Also NVMe devices used as ZIL will do it.

The problem with the NVMe is that they are connected to the PCI Express bus, and so they are not hot swap. If one dies, you cannot replace without stopping the Server.

The problem with the SSD is that they are not made for writing, they will die, so you need at least a mirror and for heavy IO I strongly recommend you to go with Enterprise grade SAS SSD drives. Those are made to last.

SSD Enterprise grade are double price versus one common SSD, but that peace of mind and extra lasting is worth it. And you don’t need a very big device, only has to hold 10 seconds of Data at max speed. So if you can ingest Data through the Network at 20 Gbps (2.5GB/s) you only need approximately 25 GB of space of the SLOG. 50 GB if you want to be more than safe.

Also you can use partitions instead of complete devices for the SLOG (like for the ZFS pool, where you can add complete drives, or partitions).

If you write locally, and you have 4 IOC’s capable of delivering 8 GB/s each, and you write to a Dataset to the pool, and not to a ZVOL which are slow by nature, you can get astonishing combined speed writing to the drives. If you are migrating a Server to another new, where you can resume if power goes down, then it’s safe to disable sync (set async) while this process runs, and turn sync on when going live to production. If you use async you don’t need to use a SLOG.

4 IOC’s able to deliver 8 GB/s are enough to provide sustained speed to 90 HDD SAS drives. 90x200MB/s=18GB/s required at max speed or 90x250MB/s=22.5GB/s.

The HDD drives provide different speeds in the inner and in the outer areas of the drive, so normally those drives up to 8TB perform between 100 and 200 MB/s, and the drives from 10TB SAS to 14TB SAS perform between 145 and 250 MB/s. I cannot tell about the 16 TB as I’ve not tested them.

The instructions to set a Ramdrive and to assign to a pool are like this:

#!/usr/bin/env bash

if [[ $(id -u) -ne 0 ]] ; then
    echo "Please run as root"
    exit 1

modprobe brd rd_nr=1 rd_size=${RAM_DRIVE_SIZE_IN_BYTES} max_part=0

echo "Use it like: zpool add carlespool log ram0"

If you created more than one Ramdisk you can add a mirror for the slog to the pool with:

zpool add carlespool log mirror /dev/ram0 /dev/ram1

You can partition the Ramdrive and add a partition but we want to add the whole ram device.

Obviously you cannot put other things to that Ramdisk (like the Metadata) as you need persistence for that.

In any case, please, avoid JBODs loaded of big HDD drives with low bandwidth micro SATA like 3Gbps per channel to the Server, and RAID. The bandwidth is too low. Your rebuilds will take forever.

With ZFS you’ll resilver (rebuild) only the actual data, not the whole drive.

News from the blog 2020-08-10

  • Atlassian tells employees they can work from home indefinitely. Which is nice and follow the steps of other giants before.


This new scenario challenges the old companies and is driving to some internal tensions in the companies that resist to allow employees to freely Remote Work.

  • I published this script to read the combined bandwidth, and peak max, from all drives.


  • I got new sales of my book in LeanPub and I’ve to say that this really makes me happy

I’ve been working in adding new information and I released a new update this week, the version 0.77. Talking about mutable and immutable objects when passing to a function, and references.

  • Bought new books and resumed my routines to study daily in the Coffee Shop (recently open)
  • I bought some new hardware
    • An Arm to support the laptop and a Vesa Monitor. That’s my Desk, actually

It is exactly this model:

  • I bought also an HDMI switch with 3 inputs and PinP
The most cool feature is the PinP. Is a simple model with reduced but does what I wanted perfectly and cheap. Worth the price.
  • I bought also this Mic/headphone USB dondle with a single jack. Very cool
Does exactly what I wanted, adding a new device. I’m using on a Windows 10 Enterprise box.
  • I’ve bought a static bicycle for the WFH / lockdown covid-19.

A simple trick to find your Git Submodules imports in Python by adding to Syspath

If you are using Git Submodules, is very probable that at some point you will create you own libraries. Probably those libraries will have their own structure, even with their own tests/ folder and you’re adding into a subfolder into your new project and maybe you have problems using relative imports.

This is a trick you can use to add the relevant root folder of your project to the System Path, so the libraries are found, specially when you call by command line from anywhere in the filesystem. This works for Python2 and Python3.

#!/usr/bin/env python3

import sys
import os

s_path_program = os.path.dirname(__file__)
sys.path.append(s_path_program + '../../')

from clib.src.argsutils import ArgsUtils
from clib.src.datetimeutils import DateTimeUtils
from clib.src.fileutils import FileUtils

This sample can be found in my book Pythom Combat Guide.

iostat_bandwitdth.sh – Utility to calculate the bandwidth used by all your drives

This is a shell script I made long time ago and I use it to monitor in real time what’s the total or individual bandwidth and maximum bandwidth achieved, for READ and WRITE, of Hard drives and NMVe devices.

It uses iostat to capture the metrics, and then processes the maximum values, the combined speed of all the drives… has also an interesting feature to let out the booting device. That’s very handy for Rack Servers where you boot from an SSD card or and SD, and you want to monitor the speed of the other (SAS probably) devices.

I used it to monitor the total bandwidth achieved by our 4U60 and 4U90 Servers, the All-Flash-Arrays 2U and the NVMe 1U units in Sanmina and the real throughput of IOC (Input Output Controllers).

I used also to compare what was the real data written to ZFS and mdraid RAID systems, and to disks and the combined speed with different pool configurations, as well as the efficiency of iSCSI and NFS from clients to the Servers.

You can specify how many times the information will be printed, whether you want to keep the max speed of each device per separate, and specify a drive to exclude. Normally it will be the boot drive.

If you want to test performance metrics you should make sure that other programs are not running or using the swap, to prevent bias. You should disable the boot drive if it doesn’t form part of your tests (like in the 4U60 with an SSD boot drive in a card, and 60 hard drive bays SAS or SATA).

You may find useful tools like iotop.

You can find the code here, and in my gitlab repo:


#!/usr/bin/env bash

AUTHOR="Carles Mateo"

# Changelog
# 1.4
# Added support for NVMe drives
# 1.3
# Fixed Decimals in KB count that were causing errors
# 1.2
# Added new parameter to output per drive stats
# Counting is performed in KB

# Leave boot device empty if you want to add its activity to the results
# Specially thinking about booting SD card or SSD devices versus SAS drives bandwidth calculation.
# Otherwise use i.e.: s_BOOT_DEVICE="sdcv"
# If this value is positive the loop will be kept n times
# If is negative ie: -1 it will loop forever
# Display all drives separatedly
# Display in KB or MB

# Init variables

# Internal variables

# So if you run in screen you see colors :)
export TERM=xterm

# ANSI colors

for i in "$@"
    case $i in
        shift # past argument=value
        shift # past argument=value
        shift # past argument=value
        # unknown option

if [[ "${i_UNKNOWN_OPTION}" -eq 1 ]]; then
    echo -e "${s_COLOR_RED}Unknown option${s_COLOR_NONE}"
    echo "Use: [-b|--boot_device=sda -l|--loop_times=-1 -a|--all-separatedly=1]"
    exit 1

if [ -z "${s_BOOT_DEVICE}" ]; then
    i_NUMBER_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | wc --lines`
    s_LIST_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | awk '{printf $1" ";}'`
    echo -e "${s_COLOR_BLUE}Excluding Boot Device:${s_COLOR_NONE} ${s_BOOT_DEVICE}"
    # Add an space after the name of the device to prevent something like booting with sda leaving out drives like sdaa sdab sdac...
    i_NUMBER_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | grep -v "${s_BOOT_DEVICE} " | wc --lines`
    s_LIST_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | grep -v "${s_BOOT_DEVICE} " | awk '{printf $1" ";}'`

for s_DRIVE in ${AR_DRIVES};

echo -e "${s_COLOR_BLUE}Bandwidth for drives:${s_COLOR_NONE} ${i_NUMBER_OF_DRIVES}"
echo -e "${s_COLOR_BLUE}Devices:${s_COLOR_NONE} ${s_LIST_OF_DRIVES}"
echo ""

while [ "${i_LOOP_TIMES}" -lt 0 ] || [ "${i_LOOP_TIMES}" -gt 0 ] ;
    # In MB
    # s_IOSTAT_OUTPUT_ALL_DRIVES=`iostat -d -m -y 1 1 | grep "sd\|nvm"`
    # In KB
    s_IOSTAT_OUTPUT_ALL_DRIVES=`iostat -d -y 1 1 | grep "sd\|nvm"`
    if [ -z "${s_BOOT_DEVICE}" ]; then
        s_IOSTAT_OUTPUT=`printf "${s_IOSTAT_OUTPUT_ALL_DRIVES}" | awk '{sum_read += $3} {sum_write += $4} END {printf sum_read"|"sum_write"\n"}'`
        # Add an space after the name of the device to prevent something like booting with sda leaving out drives like sdaa sdab sdac...
        s_IOSTAT_OUTPUT=`printf "${s_IOSTAT_OUTPUT_ALL_DRIVES}" | grep -v "${s_BOOT_DEVICE} " | awk '{sum_read += $3} {sum_write += $4} END {printf sum_read"|"sum_write"\n"}'`

    if [ "${i_ALL_SEPARATEDLY}" -eq 1 ]; then
        for s_DRIVE in ${AR_DRIVES};
            s_IOSTAT_DRIVE=`printf "${s_IOSTAT_OUTPUT_ALL_DRIVES}" | grep $s_DRIVE | head --lines=1 | awk '{sum_read += $3} {sum_write += $4} END {printf sum_read"|"sum_write"\n"}'`
            i_IOSTAT_READ_KB=`printf "%s" "${s_IOSTAT_DRIVE}" | awk -F '|' '{print $1;}'`
            i_IOSTAT_WRITE_KB=`printf "%s" "${s_IOSTAT_DRIVE}" | awk -F '|' '{print $2;}'`
            if [ "${i_IOSTAT_READ_KB%.*}" -gt ${AR_DRIVES_VALUES_READ_MAX[i_COUNTER_LOOP]%.*} ]; then
                echo -e "New Max Speed Reading for ${s_COLOR_BLUE}$s_DRIVE${s_COLOR_NONE} at ${s_COLOR_RED}${i_IOSTAT_READ_KB} KB/s${s_COLOR_NONE}"
            if [ "${i_IOSTAT_WRITE_KB%.*}" -gt ${AR_DRIVES_VALUES_WRITE_MAX[i_COUNTER_LOOP]%.*} ]; then
                echo -e "New Max Speed Writing for ${s_COLOR_BLUE}$s_DRIVE${s_COLOR_NONE} at ${s_COLOR_RED}${i_IOSTAT_WRITE_KB} KB/s${s_COLOR_NONE}"


    i_IOSTAT_READ_KB=`printf "%s" "${s_IOSTAT_OUTPUT}" | awk -F '|' '{print $1;}'`
    i_IOSTAT_WRITE_KB=`printf "%s" "${s_IOSTAT_OUTPUT}" | awk -F '|' '{print $2;}'`

    # CAST to Integer
    if [ "${i_IOSTAT_READ_KB%.*}" -gt ${i_READ_MAX%.*} ]; then
    # CAST to Integer
    if [ "${i_IOSTAT_WRITE_KB%.*}" -gt ${i_WRITE_MAX%.*} ]; then

    if [ "${s_DISPLAY_UNIT}" == "M" ]; then
        # Get MB

    # When a MAX is detected it will be displayed in RED
    echo -e "READ  ${s_READ_PRE_COLOR}${i_IOSTAT_READ_UNIT} MB/s ${s_READ_POS_COLOR} (${i_IOSTAT_READ_KB} KB/s) Max: ${i_READ_MAX_MB} MB/s (${i_READ_MAX} KB/s) (${s_READ_MAX_DATE})"
    if [ "$i_LOOP_TIMES" -gt 0 ]; then

News from the Blog 2020-07-19

I start this section to mention things that are worth.

  1. I read this article from Slack outage in May. It is really nice:


2. I read this interesting article about Netflix and Hexagonal Architecture:


3. This new from Amazon is interesting for everybody in the sector:


4. I updated CTOP.py to v.0.7.6 adding some command line switches to:

  • Limit the number of loops (like just getting the info and exiting, for scripting)
  • Work mode in black and white
  • Sleep time between retraces, to avoid flickering over SSH, and reduce bandwidth

5. I’ve been asked by an editorial to print my books in paper, for all the world. I’m thinking about it.

6. I’ve been asked to talk in a DevOps event.

7. I’ve created a new section in the blog, for adding the score of the videogames I played.

LDAPGUI a LDAP GUI program in Python and Tkinter

I wanted to automate certain operations that we do very often, and so I decided to do a PoC of how handy will it be to create GUI applications that can automate tasks.

As locating information in several repositories of information (ldap, databases, websites, etc…) can be tedious I decided to create a small program that queries LDAP for the information I’m interested, in this case a Location. This small program can very easily escalated to launch the VPN, to query a Database after querying LDAP if no results are found, etc…

I share with you the basic application as you may find interesting to create GUI applications in Python, compatible with Windows, Linux and Mac.

I’m super Linux fan but this is important, as many multinationals still use Windows or Mac even for Engineers and SRE positions.

With the article I provide a Dockerfile and a docker-compose.yml file that will launch an OpenLDAP Docker Container preloaded with very basic information and a PHPLDAPMIN Container.

Installation of the dependencies


sudo apt-get install python3.8 python3-tk
pip install ldap3

Windows and Mac:

Install Python3.6 or greater and from command line.

For Mac install pip if you don’t have it.

pip install ldap3

Python Code

#!/bin/env python3

import tkinter as tk
from ldap3 import Server, Connection, MODIFY_ADD, MODIFY_REPLACE, ALL_ATTRIBUTES, ObjectDef, Reader
from ldap3.utils.dn import safe_rdn
from lib.fileutils import FileUtils

# sudo apt-get install python3.8 python3-tk
# pip install ldap3

class LDAPGUI:

    s_config_file = "config.cfg"

    s_ldap_server = "ldapslave01"
    s_connection = "uid=%USERNAME%, cn=users, cn=accounts, dc=demo1, dc=carlesmateo, dc=com"
    s_username = "carlesmateo"
    s_password = "Secret123"
    s_query = "location=%LOCATION%,dc=demo1,dc=carlesmateo,dc=com"

    i_window_width = 610
    i_window_height = 650
    i_frame_width = i_window_width-5
    i_frame_height = 30
    i_frame_results_height = 400

    # Graphical objects
    o_window = None
    o_entry_ldap = None
    o_entry_connection = None
    o_entry_results = None
    o_entry_username = None
    o_entry_location = None
    o_button_search = None

    def __init__(self, o_fileutils):
        self.o_fileutils = o_fileutils

    def replace_connection_with_username(self):
        s_connection_raw = self.o_entry_username.get()
        s_connection = self.s_connection.replace("%USERNAME%", s_connection_raw)
        return s_connection

    def replace_query_with_location(self):
        s_query_raw = self.o_entry_location.get()
        s_query = self.s_query.replace("%LOCATION%", s_query_raw)
        return s_query

    def clear_results(self):
        self.o_entry_results.delete("1.0", tk.END)

    def enable_button(self):
        self.o_button_search["state"] = tk.NORMAL

    def disable_button(self):
        self.o_button_search["state"] = tk.DISABLED

    def ldap_run_query(self):


        s_ldap_server = self.o_entry_ldap.get()
        self.o_entry_results.insert(tk.END, "Connecting to: " + s_ldap_server + "...\n")

            o_server = Server(s_ldap_server)
            o_conn = Connection(o_server,

            if isinstance(o_conn.last_error, str):
                s_last_error = o_conn.last_error
                self.o_entry_results.insert(tk.END, "Last error: " + s_last_error + "\n")

            if isinstance(o_conn.result, str):
                s_conn_result = o_conn.result
                self.o_entry_results.insert(tk.END, "Connection result: " + s_conn_result + "\n")

            s_query = self.replace_query_with_location()
            self.o_entry_results.insert(tk.END, "Performing Query: " + s_query + "\n")

            obj_location = ObjectDef('organizationalUnit', o_conn)
            r = Reader(o_conn, obj_location, s_query)
            self.o_entry_results.insert(tk.END, "Results: \n" + str(r) + "\n")

            #print(0, o_conn.extend.standard.who_am_i())
            # https://github.com/cannatag/ldap3/blob/master/tut1.py
            # https://github.com/cannatag/ldap3/blob/master/tut2.py

            self.o_entry_results.insert(tk.END, "There has been a problem" + "\n")

            if isinstance(o_conn.last_error, str):
                s_last_error = o_conn.last_error
                self.o_entry_results.insert(tk.END, "Last error: " + s_last_error + "\n")

            self.o_entry_results.insert(tk.END, "Closing connection\n")
            self.o_entry_results.insert(tk.END, "Problem closing connection" + "\n")


    def create_frame_location(self, o_window, i_width, i_height):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)

        o_lbl_location = tk.Label(master=o_frame,
        o_lbl_location.place(x=0, y=0)

        o_entry = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry.insert(tk.END, "")
        o_entry.place(x=100, y=0)

        return o_frame, o_entry

    def create_frame_results(self, o_window, i_width, i_height):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)

        o_entry = tk.Text(master=o_frame, fg="grey", bg="white", width=75, height=20)
        o_entry.insert(tk.END, "")
        o_entry.place(x=0, y=0)

        return o_frame, o_entry

    def create_frame_username(self, o_window, i_width, i_height, s_username):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)

        o_lbl_location = tk.Label(master=o_frame,
        o_lbl_location.place(x=0, y=0)

        o_entry_username = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry_username.insert(tk.END, s_username)
        o_entry_username.place(x=100, y=0)

        return o_frame, o_entry_username

    def create_frame_ldapserver(self, o_window, i_width, i_height, s_server):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)

        o_lbl_ldapserver = tk.Label(master=o_frame,
                                    text="LDAP Server:",
        o_lbl_ldapserver.place(x=0, y=0)
        # We don't need to pack the label, as it is inside a Frame, packet
        # o_lbl_ldapserver.pack()

        o_entry = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry.insert(tk.END, s_server)
        o_entry.place(x=100, y=0)

        return o_frame, o_entry

    def create_frame_connection(self, o_window, i_width, i_height, s_connection):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)

        o_lbl_connection = tk.Label(master=o_frame,
        o_lbl_connection.place(x=0, y=0)
        # We don't need to pack the label, as it is inside a Frame, packet
        # o_lbl_ldapserver.pack()

        o_entry_connection = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry_connection.insert(tk.END, s_connection)
        o_entry_connection.place(x=100, y=0)

        return o_frame, o_entry_connection

    def create_frame_query(self, o_window, i_width, i_height, s_query):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)

        o_lbl_query = tk.Label(master=o_frame,
        o_lbl_query.place(x=0, y=0)

        o_entry_query = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry_query.insert(tk.END, s_query)
        o_entry_query.place(x=100, y=0)

        return o_frame, o_entry_query

    def create_button(self):
        o_button = tk.Button(


        return o_button

    def render_screen(self):
        o_window = tk.Tk()
        o_window.title("LDAPGUI by Carles Mateo")
        self.o_window = o_window
        self.o_window.geometry(str(self.i_window_width) + 'x' + str(self.i_window_height))

        o_frame_ldap, o_entry_ldap = self.create_frame_ldapserver(o_window=o_window,
        self.o_entry_ldap = o_entry_ldap
        o_frame_connection, o_entry_connection = self.create_frame_connection(o_window=o_window,
        self.o_entry_connection = o_entry_connection

        o_frame_user, o_entry_user = self.create_frame_username(o_window=o_window,
        self.o_entry_username = o_entry_user

        o_frame_query, o_entry_query = self.create_frame_query(o_window=o_window,
        self.o_entry_query = o_entry_query

        o_frame_location, o_entry_location = self.create_frame_location(o_window=o_window,

        self.o_entry_location = o_entry_location

        o_button_search = self.create_button()
        self.o_button_search = o_button_search

        o_frame_results, o_entry_results = self.create_frame_results(o_window=o_window,
        self.o_entry_results = o_entry_results


    def load_config_values(self):
        b_success, d_s_config = self.o_fileutils.read_config_file_values(self.s_config_file)
        if b_success is True:
            if 'server' in d_s_config:
                self.s_ldap_server = d_s_config['server']
            if 'connection' in d_s_config:
                self.s_connection = d_s_config['connection']
            if 'username' in d_s_config:
                self.s_username = d_s_config['username']
            if 'password' in d_s_config:
                self.s_password = d_s_config['password']
            if 'query' in d_s_config:
                self.s_query = d_s_config['query']

def main():
    o_fileutils = FileUtils()

    o_ldapgui = LDAPGUI(o_fileutils=o_fileutils)


if __name__ == "__main__":


FROM osixia/openldap

# Note: Docker compose will generate the Docker Image.
# Run:  sudo docker-compose up -d

LABEL maintainer="Carles Mateo"

ENV LDAP_ORGANISATION="Carles Mateo Test Org" \

COPY bootstrap.ldif /container/service/slapd/assets/config/bootstrap/ldif/50-bootstrap.ldif


version: '3.3'
        context: .
        dockerfile: Dockerfile
        LDAP_ADMIN_PASSWORD: test1234
        LDAP_BASE_DN: dc=carlesmateo,dc=com
        - 389:389
        - ldap_data:/var/lib/ldap
        - ldap_config:/etc/ldap/slapd.d
      image: osixia/phpldapadmin:0.7.2
        - 8090:80
        PHPLDAPADMIN_LDAP_HOSTS: ldap_server
        PHPLDAPADMIN_HTTPS: 'false'


# Config File for LDAPGUI by Carles Mateo
# https://blog.carlesmateo.com

# Configuration for working with the prepared Docker

# Carles other tests
#connection=uid=%USERNAME%, cn=users, cn=accounts, dc=demo1, dc=carlesmateo, dc=com


In order to bootstrap the Data on the LDAP Server I use a bootstrap.ldif file copied into the Server.

Credits, I learned this trick in this page: https://medium.com/better-programming/ldap-docker-image-with-populated-users-3a5b4d090aa4

dn: cn=developer,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: developer
givenname: developer
sn: Developer
displayname: Developer User
mail: developer@carlesmateo.com
userpassword: developer_pass

dn: cn=maintainer,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: maintainer
givenname: maintainer
sn: Maintainer
displayname: Maintainer User
mail: maintainer@carlesmateo.com
userpassword: maintainer_pass

dn: cn=admin,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: admin
givenname: admin
sn: Admin
displayname: Admin
mail: admin@carlesmateo.com
userpassword: admin_pass

dn: ou=Groups,dc=carlesmateo,dc=com
changetype: add
objectclass: organizationalUnit
ou: Groups

dn: ou=Users,dc=carlesmateo,dc=com
changetype: add
objectclass: organizationalUnit
ou: Users

dn: cn=Admins,ou=Groups,dc=carlesmateo,dc=com
changetype: add
cn: Admins
objectclass: groupOfUniqueNames
uniqueMember: cn=admin,dc=carlesmateo,dc=com

dn: cn=Maintaners,ou=Groups,dc=carlesmateo,dc=com
changetype: add
cn: Maintaners
objectclass: groupOfUniqueNames
uniqueMember: cn=maintainer,dc=carlesmateo,dc=com
uniqueMember: cn=developer,dc=carlesmateo,dc=com


You can download this file from the lib folder in my project CTOP.py

More information about programming with tkinter:




About LDAP:


A script to backup your partition compressed and a game to learn about dd and pipes

This article is more an exercise, like a game, so you get to know certain things about Linux, and follow my mental process to uncover this. Is nothing mysterious for the Senior Engineers but Junior Sys Admins may enjoy this reading. :)

Ok, so the first thing is I wrote an script in order to completely backup my NVMe hard drive to a gziped file and then I will use this, as a motivation to go deep into investigations to understand.

Ok, so the first script would be like this:

TARGET_PATH="/media/carles/Seagate\ Backup\ Plus\ Drive/BCK/"
sudo bash -c "dd if=${SOURCE_DRIVE} | gzip > ${TARGET_PATH}${TARGET_FILE}.gz"

So basically, we are going to restart the computer, boot with Linux Live USB Key, mount the Seagate Hard Drive, and run the script.

We are booting with a Live Linux Cd in order to have our partition unmounted and unmodified while we do the backup. This is in order to avoid corruption or data loss as a live Filesystem is getting modifications as we read it.

The problem with this first script is that it will generate a big gzip file.

By big I mean much more bigger than 2GB. Not all physical supports support files bigger than 2GB or 4GB, but even if they do, it’s a pain to transfer this over the Network, or in USB files, so we are going to do a slight modification.

TARGET_PATH="/media/carles/Seagate\ Backup\ Plus\ Drive/BCK/"
sudo bash -c "dd if=${SOURCE_DRIVE} | gzip | split -b 1024MiB - ${TARGET_PATH}${TARGET_FILE}-split.gz_"

Ok, so we will use pipes and split in order to generate many files as big as 1GB.

If we ls we will get:

-rwxrwxrwx 1 carles carles 1073741824 May 24 14:57 nvme.img-split.gz_aa
-rwxrwxrwx 1 carles carles 1073741824 May 24 14:58 nvme.img-split.gz_ab
-rwxrwxrwx 1 carles carles 1073741824 May 24 14:59 nvme.img-split.gz_ac

Then one may say, Ok, this is working, but how I know the progress?.

For old versions of dd you can use pv which stands for Pipe Viewer and allows you to know the transference between processes using pipes.

For more recent versions of dd you can use status=progress.

So the script updated with status=progress is:

TARGET_PATH="/media/carles/Seagate\ Backup\ Plus\ Drive/BCK/"
sudo bash -c "dd if=${SOURCE_DRIVE} status=progress | gzip | split -b 1024MiB - ${TARGET_PATH}${TARGET_FILE}-split.gz_"

You can also download the code from:


Then one may ask himself, wait, if pipes use STDOUT and STDIN and dd is displaying into the screen, then will our gz file get corrupted?.

I like when people question things, and investigate, so let’s answer this question.

If it was a young member of my Team I would ask:

  • Ok, try,it. Check the output file to see if is corrupted.

So they can do zcat or zless to inspect the file, see if it has errors, and to make sure:

gzip -v -t nvme.img.gz
nvme.img.gz:        OK

Ok, so what happened?, because we were seeing output in the screen.

Assuming the young Engineer does not know the answer I would had told:

  • Ok, so you know that if dd would print to STDOUT, then you won’t see it, cause it would be sent to the pipe, so there is something more you’re missing. Let’s check the source code of dd to see what status=progress does

And then look for “progress”.

Soon you’ll find things like everywhere:

  if (progress_time)
    fputc ('\r', stderr);

Ok, pay attention to where is the data written: stderr. So basically the answer is: dd status=progress does not corrupt STDOUT and prints into the screen because it uses STDERR.

Other funny ways to get the progress would be to use:

watch -n10 "ls -alh /BCK/ | grep nvme | wc --lines"
So you would see in real time what was the advance and finally 512GB where compressed to around 336GB in 336 files of 1 GB each (except the last one)

Another funny way would had been sending USR1 signal to the dd process:

Hope you enjoyed this little exercise about the importance of going deep, to the end, to understand what’s going on on the system. :)

Instead of gzip you can use bzip2 or pixz. pixz is very handy if you want to just compress a file, as it uses multiple processors in parallel for the tasks.

xz or lrzip are other compressors. lrzip aims to compress very large files, specially source code.