Category Archives: Containers

Podman – unable to pull image

Today, while pulling the latest Nextcloud container image from the docker.io repository I noticed that it is failing. Looking at the read: connection reset by peer error I assume it could be primarily due to network failure; maybe poor quality of the connection.

Note that Mauritius is far from "everywhere" else and connectivity issues are nothing new.

ish@coffee-bar:~> podman pull docker.io/library/nextcloud
Trying to pull docker.io/library/nextcloud...
Getting image source signatures
Copying blob e0276193a084 done  
Copying blob eb2d00c10344 done  
Copying blob 3a60f364b0c5 done  
Copying blob e0d3d1244592 done  
Copying blob f54006e0dc29 done  
Copying blob 8559a31e96f4 done  
Copying blob b22875b95a2a done  
Copying blob f65316e96b10 done  
Copying blob 3e309988c00b done  
Copying blob 0c78caf16ec3 [======================================] 13.1MiB / 13.2MiB
Copying blob 4fc30aae7ee5 done  
Copying blob 37b016cacdc6 done  
Copying blob fd56bf3cc539 done  
Copying blob e3cd35f544b5 done  
Copying blob 467fea8f6f80 done  
Copying blob 0dc7444f9282 done  
Copying blob 547ae6684264 done  
Copying blob 3139b6de5be7 done  
Copying blob 00e77223b529 done  
Copying blob 2b37e3b3a856 done  
  read tcp 192.168.100.6:33536->104.18.124.25:443: read: connection reset by peer
Error: error pulling image "docker.io/library/nextcloud": unable to pull docker.io/library/nextcloud: unable to pull image: Error writing blob: error storing blob to file "/var/tmp/storage720474498/10": read tcp 192.168.100.6:33536->104.18.124.25:443: read: connection reset by peer

Podman does not retry to copy the image in case of failures.

A few days ago there was a suggestion to implement a similar feature in Podman that is present in Buildah, which provides the image copy retry functionality.

For the curious, the implementation can be seen here.

Container images that are pulled by Buildah are stored in the local repository which can also by accessed by Podman, so that's an advantage. I tried pulling the Nextcloud container image using Buildah and it completed successfully.

ish@coffee-bar:~> buildah pull nextcloud
Getting image source signatures
Copying blob e0d3d1244592 done  
Copying blob 8559a31e96f4 done  
Copying blob eb2d00c10344 done  
Copying blob 3a60f364b0c5 done  
Copying blob f54006e0dc29 done  
Copying blob e0276193a084 done  
Copying blob f65316e96b10 done  
Copying blob b22875b95a2a done  
Copying blob 3e309988c00b done  
Copying blob 0c78caf16ec3 done  
Copying blob 4fc30aae7ee5 done  
Copying blob 37b016cacdc6 done  
Copying blob e3cd35f544b5 done  
Copying blob fd56bf3cc539 done  
Copying blob 467fea8f6f80 done  
Copying blob 0dc7444f9282 done  
Copying blob 547ae6684264 done  
Copying blob 2b37e3b3a856 done  
Copying blob 00e77223b529 done  
Copying blob 3139b6de5be7 done  
Copying config 327476ebe3 done  
Writing manifest to image destination
Storing signatures
327476ebe3280c7b570d8463edd136956eab120959976b643cb7dbfaa73f98c1

Now, the downloaded container image is also accessible by Podman.

ish@coffee-bar:~> podman images
REPOSITORY                     TAG      IMAGE ID       CREATED        SIZE
docker.io/libreoffice/online   latest   0586fecfa3c1   28 hours ago   2.84 GB
docker.io/library/nextcloud    latest   327476ebe328   3 days ago     774 MB

To conclude, while we are waiting that a retryCopyImage function is available in Podman, we can use Buildah to pull container images that are troublesome due to network issues.

A quick intro to Podman by Estu Fardani

Estu Fardani is a helpful & jovial fellow whom I met at the openSUSE Asia Summit last year in Bali, Indonesia. Recently, for the openSUSE Virtual Summit, Estu did a short presentation on Podman. His presentation video is available on YouTube and slides deck available on Google Slides.

While I am not a fan of alias docker=podman I believe Estu added that part in his slide below to make the Podman transition a bit smoother for users already familiar with Docker commands.

Slide from Estu's presentation

Note that the latest release of openSUSE Leap, i.e version 15.2, comes with support for Podman through the libcontainers-common package.

Previously, if one needed to experiment with Podman and/or deploy in production then openSUSE Tumbleweed, Kubic or MicroOS were the supported choices. Now, one may deploy Podman containers on their existing Leap infrastructure (after upgrading to the latest version 15.2).

Debugging on MicroOS made easier with toolbox

Until now, it was not easy to run debugging tools on openSUSE MicroOS or Kubic for the simple reason that installing any package using the transactional-update in package_name command would require a reboot for the program to be available on the system. Why so? MicroOS & Kubic use transactional updates and thus, the root filesystem is mounted as read-only. Packages that are installed are available in a "snapshot" which is selected at the next boot. openSUSE MicroOS & Kubic fall in the category of immutable operating systems.

One couldn't just reboot the system every time a simple program was needed - e.g using Vi to edit a YAML config or using database client utils for a rapid inspection.

But now we have a toolbox script available in MicroOS & Kubic that allows us to launch a container and use debugging utilities that can interact with the host. The script is written in Bash and it is based on CoreOS/toolbox.

Thorsten Kukuk, SUSE Engineer & Architect (SLES / MicroOS), announced the availability of the toolbox container on the openSUSE Kubic mailing list in October.

The toolbox script resides in the system $PATH and thus can be called directly. On its first execution, the script will pull the toolbox container from the openSUSE Container Registry.

microos:~ # toolbox
Trying to pull registry.opensuse.org/opensuse/toolbox...
Getting image source signatures
Copying blob 92a022f7c0d8 [>-------------------------------------] 2.8MiB / 122.9MiB
Copying blob f0c8107beec9 [==>-----------------------------------] 3.0MiB / 36.4MiB

The host root filesystem can be accessed at /media/root/ and since the toolbox container is spawned as the root user, the files will be modifiable.

toolbox:/ # ls /media/root/
bin   dev  home  lib64	opt   root  sbin     srv  tmp  var
boot  etc  lib	 mnt	proc  run   selinux  sys  usr

Non-root toolbox containers will have read-only access to the host files, unless the --root flag is provided.

ish@microos:~> toolbox --root

Next, use zypper to install the required utilities and interact with the host and other containers.

Exiting toolbox will stop and the container and next time the script is called, the container is started again. Therefore, previously installed utilities remain available.

Each toolbox container is named by appending the username calling it, e.g toolbox-ish.

microos:~ # podman ps --format "{{.ID}} {{.Status}} {{.Names}}"
7f61fba95487 Up 10 minutes ago toolbox-ish
00ada606ec4f Up 33 minutes ago toolbox-root

Understand networking in Podman

I received a message on Twitter on 17 October from a fellow who attended the openSUSE Asia Summit 2019. Strangely, I didn't get any notification about it and it's only today that I read the message. He also attended my workshop on openSUSE MicroOS and had some questions regarding inter-Pod communication.

As a quick response, I explained him very breifly about "container networking" and pointed him to the Kubernetes documentation on IP allocation. I do realize though, that most of the times, documenation can be lengthy and {boring}, and that you would just want a simple article or blog post that clears your doubts.

Tell me about Podman networks

Rootless containers (i.e containers started using Podman as a regular user) do not obtain an IP address. Podman uses slirp4netns to allow Internet connectivity inside the container.

Communication with a rootless container is achieved by mapping the container ports to the host, e.g using -p 8080:80 to map a webserver port 80 to the host on port port 8080.

$ podman run -dt --name webserver -p 8080:80 nginx
$ curl http://localhost:8080

Therefore, two rootless containers can communicate over their published ports on the host. Let's experiment this by starting an openSUSE Leap container and installing the telnet package.

$ podman run -dt --name leap leap
$ podman exec -it leap bash

4a0f95e011b9:/ # zypper in telnet

We run ip a s on the host to find its IP address. Say the IP address is 192.168.100.8. Now, from within Leap container let's telnet port 8080 over the host IP.

4a0f95e011b9:/ # telnet 192.168.100.10 8080
Trying 192.168.100.10...
Connected to 192.168.100.10.
Escape character is '^]'.

The connection went through successfully, meaning from the Leap container we've been able to access the Nginx container through it's mapped port on the host.

This same experiment can be repeated using two different pods, say you have a pod that contains your web services and another pod that contains your databases.

$ podman pod create --name webservice -p 8080:80
$ podman run -dt --name webserver --pod webservice nginx

$ podman pod create --name db -p 3306:3306
$ podman run -dt --name mariadb --pod db -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" mariadb

The Nginx container will be able to reach the MariaDB database over 192.168.100.10:3306 as the same port is mapped on the host.

Ideally, these two containers could have been created in the same pod and therefore share the same network space. Then, the Nginx container would reach the database over localhost:3306 easily.

I used the above Nginx/MariaDB example to explain rootless inter-Pod communication, which was the question that was asked to me initially.

What about rootfull containers?

Rootfull containers are those that are created using Podman with root privileges, either by the root user itself or using sudo privilege.

Containers created using Podman with root privileges obtain an IP address. Podman then uses the Container Network Interfec (CNI) instead of slirp4netns for networking provisioning.

Details about the network subnet is found in the CNI config file.

$ cat /etc/cni/net.d/87-podman-bridge.conflist

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "10.88.0.0/16",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
      },
      {
        "type": "portmap",
        "capabilities": {
          "portMappings": true
        }
      }
    ]
}

So, let's start a container with root privileges and see.

$ sudo podman run -dt --name db postgres

$ sudo podman inspect -f "{{.NetworkSettings.IPAddress}}" db
10.88.0.30

The podman inspect ... command returns the container's IP address and the same is ranged within the subnet specified in the CNI config. The PostgreSQL database can be accessed over 10.88.0.30:5432 from the host or from within any other container started using root privileges.

$ telnet 10.88.0.30 5432
Trying 10.88.0.30...
Connected to 10.88.0.30.
Escape character is '^]'.

Podman 101 at the Middlesex University Mauritius

Podman 101 at the Middlesex University Mauritius

Thanks to Senior Lecturer and Programme Coordinator, Aditya Santokhee, at the Middlesex University Mauritius, my colleague Chittesh & I got to deliver guest talks at the university today.

Podman 101 at the Middlesex University Mauritius

Chittesh, also our local Mozillian, spoke about the Internet Health Report and raised privacy concerns surrounding major online platforms. He mentioned the Cambridge Analytica scandal in particular, especially how that raised public awareness about digital privacy. He added that following the enforcement of the European Union's General Data Protection Regulations (EU GDPR), tens of thousands of violation complaints have been filed. Google was fined €50 million for GDPR violations in France.

His talk was more a message to the young students for them to review their online habits and take a moment to think about digital privacy.

On the other hand I had a semi-technical talk about Pods & Containers. Although we had a lecture theatre fully packed with Computer Science students, I was told they were mostly in second year, I realise that not everyone would be acquainted to Linux containers, or yet Linux itself.

Therefore, I started with a gentle introduction about operating systems and where Linux stands. I asked the students whether they are familiar with virtualization and part of the room answered yes. It made it easier to compare between having an "architecture emulator" to run a virtual machine and a simple isolated environment comprising of a bunch of files. That's the simplest explanation I could give to make the students comfortable with the idea of containers.

Podman 101 at the Middlesex University Mauritius
Podman 101 at the Middlesex University Mauritius

Then, I moved to the topic of Podman, while doing some demos and helping the students learn how podman run ... could make things easier for them. I asked them a few questions about university assignments such as developing an application and the need of having an "environment" to build or test the application. Instead of having a full Linux environment with a bunch of packages installed & configured simply to serve a web application, a single-line of podman could serve those files in an Nginx container. Especially, doing this without a big-fat-daemon, pun intended! 🤓

I briefly talked about Skopeo and I invited the more adventurous students to peek inside of containers and get a better understanding of what they are made of. The key lesson being, magic is for the users not for the engineer.

I ended the talk with this nice slide copied from my openSUSE MicroOS workshop deck from the openSUSE Asia Summit 2019. 😊

Podman 101 at the Middlesex University Mauritius

Slides available at speakerdeck.com/ishwon/podman-101.