I'm using podman on RHEL8.6 to deploy a pod composed of two containers, zabbix-server and postgres-server. To deploy the containers, podman play kube is used on a .yaml file to import and run the pod configuration. This is a rootless setup.
The containers need to communicate over an internal network using DNS. I use podman create network mon to create a bridged network called 'mon' that has DNS enabled.
With podman play kube monitoring.yaml --network mon, the containers are not able to communicate with each other over DNS. I've tested the setup without using podman play kube by creating the pod and containers manually. In the manual setup, the containers in the pod are able to find each other using DNS.
manual setup:
podman pod create --name monitoring
podman network create mon
podman run -d --rm --network=mon --pod=monitoring --name postgres-server docker.io/timescale/timescaledb:latest-pg14
podman run -d --rm --network=mon --pod=monitoring --name zabbix-server docker.io/zabbix/zabbix-server-pgsql
mon network config:
[
{
"cniVersion": "0.4.0",
"name": "mon",
"plugins": [
{
"bridge": "cni-podman1",
"hairpinMode": true,
"ipMasq": true,
"ipam": {
"ranges": [
[
{
"gateway": "10.89.0.1",
"subnet": "10.89.0.0/24"
}
]
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"type": "host-local"
},
"isGateway": true,
"type": "bridge"
},
{
"capabilities": {
"portMappings": true
},
"type": "portmap"
},
{
"backend": "",
"type": "firewall"
},
{
"type": "tuning"
},
{
"capabilities": {
"aliases": true
},
"domainName": "dns.podman",
"type": "dnsname"
}
]
}
]
monitoring.yaml file:
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-3.4.7
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2022-05-24T13:40:30Z"
labels:
app: monitoring
name: monitoring
spec:
containers:
- args:
- postgres
image: docker.io/timescale/timescaledb:latest-pg14
name: postgres-server
resources: {}
securityContext:
capabilities:
drop:
- CAP_MKNOD
- CAP_NET_RAW
- CAP_AUDIT_WRITE
- args:
- /usr/sbin/zabbix_server
- --foreground
- -c
- /etc/zabbix/zabbix_server.conf
image: docker.io/zabbix/zabbix-server-pgsql:latest
name: zabbix-server
resources: {}
securityContext:
capabilities:
drop:
- CAP_MKNOD
- CAP_NET_RAW
- CAP_AUDIT_WRITE
volumeMounts:
- mountPath: /var/lib/zabbix/export
name: bdc06ccb2b092148bb78c353fccc104255a81a8735eb3a14974f6ede0e6516e8-pvc
- mountPath: /var/lib/zabbix/snmptraps
name: 196d7983e1b62995bae07beb7942e18ac46dcf6d3346e43fff2604c9a7bc30ec-pvc
restartPolicy: Never
volumes:
- name: bdc06ccb2b092148bb78c353fccc104255a81a8735eb3a14974f6ede0e6516e8-pvc
persistentVolumeClaim:
claimName: bdc06ccb2b092148bb78c353fccc104255a81a8735eb3a14974f6ede0e6516e8
- name: 196d7983e1b62995bae07beb7942e18ac46dcf6d3346e43fff2604c9a7bc30ec-pvc
persistentVolumeClaim:
claimName: 196d7983e1b62995bae07beb7942e18ac46dcf6d3346e43fff2604c9a7bc30ec
status: {}
Troubleshooting
The podman docs indicate that attaching a network created with podman network create is supported: https://docs.podman.io/en/latest/markdown/podman-play-kube.1.html#network-mode-net
Opening a shell ( podman exec -it <container> /bin/bash ) on the containers created manually, I can ping both zabbix-server and postgres-server:
$ podman exec -it zabbix-server /bin/bash
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.89.0.3 c61fdf070064 zabbix-server
10.89.0.1 host.containers.internal
bash-5.1$ ping postgres-server
PING postgres-server.dns.podman (10.89.0.2) 56(84) bytes of data.
64 bytes from postgres-server.dns.podman (10.89.0.2): icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from postgres-server.dns.podman (10.89.0.2): icmp_seq=2 ttl=64 time=0.062 ms
64 bytes from postgres-server.dns.podman (10.89.0.2): icmp_seq=3 ttl=64 time=0.100 ms
^C
--- postgres-server.dns.podman ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.051/0.071/0.100/0.021 ms
On the play kube containers, pinging either monitoring-<name>-server or <name>-server fails. Pinging the DNS resolver on both the manual and the play kube setups returns successfully.
bash-5.1$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.89.0.2 monitoring be3df46ca68d-infra
10.89.0.1 host.containers.internal
bash-5.1$ ping postgres-server
ping: postgres-server: Name does not resolve
bash-5.1$ ping monitoring-postgres-server
ping: monitoring-postgres-server: Name does not resolve
bash-5.1$ cat /etc/resolv.conf
search dns.podman
nameserver 10.89.0.1
Using podman inspect <container>, both the manual and kube play setups have the 'mon' network attached. However, on the play kube setup, the output for network setup is the following:
"NetworkMode": "container:49e7b0bafd4619b9bcb50b1b841aeee3cb910bf3c555a9788dc297e71c948092"
This differs from the manual network setup:
"NetworkMode": "bridge"
It almost appears as if 'play kube' is ignoring the --network flag and instead creating an internal container network. Looking at the containers network setup with kube play, the ips on both containers are the same. On the manual setup, the IPs are different:
"Networks": {
"mon": {
"EndpointID": "",
"Gateway": "10.89.0.1",
"IPAddress": "10.89.0.3",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "3e:30:b7:e2:1d:00",
"NetworkID": "mon",
Networks": {
"mon": {
"EndpointID": "",
"Gateway": "10.89.0.1",
"IPAddress": "10.89.0.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "fa:1a:6b:f3:02:3a",
"NetworkID": "mon",
I've tried podman kube play with and without SELinux in Enforcing mode. I've also tried running this setup as root to no avail.
I've tested this setup with the following versions:
- podman: 3.4.7, 4.0.2
- RHEL: 8.5, 8.6
Any help would be much appreciated!
