Tricks

ssh

Docker images should run as non-root arbitrary users, but that makes anything that uses ssh (like git) a pain in the ass. ssh expects a username for the running UID, which you might not have at runtime. Fake it out and pretend to be root with libuid-wrapper and a homedir that you can create at runtime as the running user:

Dockerfile:
RUN apt update && apt -y install ... libuid-wrapper
RUN sed -i 's|:/root:|:/var/tmp/user:|' /etc/passwd

Not using 'usermod -d' there because it doesn't work for root.

ENTRYPOINT/CMD wrapper script:
export HOME=/var/tmp/user
mkdir -p "$HOME/.ssh"
chmod 700 "$HOME"
chmod 700 "$HOME"/.ssh
export LD_PRELOAD=libuid_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1
/app/do-ssh-stuff.py

Note that this doesn't change your uid, it just makes ssh (and everything else that uses [g]libc) look up usernames and homedirs for root instead of the running uid.

git

If you're pushing/pulling from ssh urls, use the ssh trick above and add this to the entrypoint/cmd wrapper script as well so ssh doesn't moan about host keys:

git config --global core.sshCommand 'ssh -o StrictHostKeyChecking=no'

If you're worried about MITM attacks, add the host pubkeys to /var/tmp/user/.ssh/known_hosts in your Dockerfile instead. And if you're really worried about MITM attacks, stop reading random wikis you found on the internet looking for ssh/git advice, and hire someone who knows what they're doing you moron.

python

Python stdout is buffered to non-tty file handles, so watching 'docker logs -f' will be a waste of everyone's time. Switch off buffering in your wrapper script with:

export PYTHONUNBUFFERED=1
/app/blah.py

Debug

Processes

Find docker processes from system shell

# docker ps
# ps -eo pid,cgroup,cmd | grep <first 8 chars of containerID>

Network

Look at the network config from a container's namespace

# nsenter -t $(docker inspect --format '{{.State.Pid}}' <containerid>) -n ip {addr|route|...}

or run bash instead of ip to get a shell in the namespace.

Known Issues

DNS failing in containers

I don't know why, but sometimes (just sometimes) dockerd has problems creating iptables rules for the in-daemon resolver.

Check that 127.0.0.11 is listed in the container's resolv.conf as it should be, then check:
nsenter -n -t $(docker inspect --format {{.State.Pid}} <container>) iptables -t nat -nvL
There should be two DNAT and two SNAT rules for port 53 (upd & tcp). If they're missing, you'll need to restart dockerd to make it work.

This only affects newly started containers - if DNS works at container-boot but fails later, it's something else.