6. Frequently asked questions (FAQ)¶
6.1. About the project¶
6.1.1. Where did the name Charliecloud come from?¶
Charlie — Charles F. McMillan was director of Los Alamos National Laboratory from June 2011 until December 2017, i.e., at the time Charliecloud was started in early 2014. He is universally referred to as “Charlie” here.
cloud — Charliecloud provides cloud-like flexibility for HPC systems.
6.1.2. How do you spell Charliecloud?¶
We try to be consistent with Charliecloud — one word, no camel case. That is, Charlie Cloud and CharlieCloud are both incorrect.
6.2. Errors¶
6.2.1. How do I read the ch-run
error messages?¶
ch-run
error messages look like this:
$ ch-run foo -- echo hello
ch-run[25750]: can't find image: foo: No such file or directory (ch-run.c:107 2)
There is a lot of information here, and it comes in this order:
- Name of the executable; always
ch-run
. - Process ID in square brackets; here
25750
. This is useful when debugging parallelch-run
invocations. - Colon.
- Main error message; here
can't find image: foo
. This should be informative as to what went wrong, and if it’s not, please file an issue, because you may have found a usability bug. Note that in some cases you may encounter the default messageerror
; if this happens and you’re not doing something very strange, that’s also a usability bug. - Colon (but note that the main error itself can contain colons too), if and only if the next item is present.
- Operating system’s description of the the value of
errno
; hereNo such file or directory
. Omitted if not applicable. - Open parenthesis.
- Name of the source file where the error occurred; here
ch-run.c
. This and the following item tell developers exactly wherech-run
became confused, which greatly improves our ability to provide help and/or debug. - Source line where the error occurred.
- Value of
errno
(see C error codes in Linux for the full list of possibilities). - Close parenthesis.
Note: Despite the structured format, the error messages are not guaranteed to be machine-readable.
6.2.2. Tarball build fails with “No command specified”¶
The full error from ch-docker2tar
or ch-build2dir
is:
docker: Error response from daemon: No command specified.
You will also see it with various plain Docker commands.
This happens when there is no default command specified in the Dockerfile or
any of its ancestors. Some base images specify one (e.g., Debian) and others
don’t (e.g., Alpine). Docker requires this even for commands that don’t seem
like they should need it, such as docker create
(which is what trips
up Charliecloud).
The solution is to add a default command to your Dockerfile, such as
CMD ["true"]
.
6.2.3. ch-run
fails with “can’t re-mount image read-only”¶
Normally, ch-run
re-mounts the image directory read-only within the
container. This fails if the image resides on certain filesystems, such as NFS
(see issue #9). There are
two solutions:
- Unpack the image into a different filesystem, such as
tmpfs
or local disk. Consult your local admins for a recommendation. Note that Lustre is probably not a good idea because it can give poor performance for you and also everyone else on the system. - Use the
-w
switch to leave the image mounted read-write. This may have an impact on reproducibility (because the application can change the image between runs) and/or stability (if there are multiple application processes and one writes a file in the image that another is reading or writing).
6.3. Unexpected behavior¶
6.3.1. --uid 0
lets me read files I can’t otherwise!¶
Some permission bits can give a surprising result with a container UID of 0. For example:
$ whoami
reidpr
$ echo surprise > ~/cantreadme
$ chmod 000 ~/cantreadme
$ ls -l ~/cantreadme
---------- 1 reidpr reidpr 9 Oct 3 15:03 /home/reidpr/cantreadme
$ cat ~/cantreadme
cat: /home/reidpr/cantreadme: Permission denied
$ ch-run /var/tmp/hello cat ~/cantreadme
cat: /home/reidpr/cantreadme: Permission denied
$ ch-run --uid 0 /var/tmp/hello cat ~/cantreadme
surprise
At first glance, it seems that we’ve found an escalation – we were able to read a file inside a container that we could not read on the host! That seems bad.
However, what is really going on here is more prosaic but complicated:
- After
unshare(CLONE_NEWUSER)
,ch-run
gains all capabilities inside the namespace. (Outside, capabilities are unchanged.) - This include
CAP_DAC_OVERRIDE
, which enables a process to read/write/execute a file or directory mostly regardless of its permission bits. (This is why root isn’t limited by permissions.) - Within the container,
exec(2)
capability rules are followed. Normally, this basically means that all capabilities are dropped whench-run
replaces itself with the user command. However, if EUID is 0, which it is inside the namespace given--uid 0
, then the subprocess keeps all its capabilities. (This makes sense: if root creates a new process, it stays root.) CAP_DAC_OVERRIDE
within a user namespace is honored for a file or directory only if its UID and GID are both mapped. In this case,ch-run
mapsreidpr
to containerroot
and groupreidpr
to itself.- Thus, files and directories owned by the host EUID and EGID (here
reidpr:reidpr
) are available for all access withch-run --uid 0
.
This is not an escalation. The quirk applies only to files owned by the
invoking user, because ch-run
is unprivileged outside the namespace,
and thus he or she could simply chmod
the file to read it. Access
inside and outside the container remains equivalent.
References:
6.3.2. Why does ping
not work?¶
ping
fails with “permission denied” or similar under Charliecloud,
even if you’re UID 0 inside the container:
$ ch-run $IMG -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: permission denied (are you root?)
$ ch-run --uid=0 $IMG -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: permission denied (are you root?)
This is because ping
needs a raw socket to construct the needed
ICMP ECHO
packets, which requires capability CAP_NET_RAW
or
root. Unprivileged users can normally use ping
because it’s a setuid
or setcap binary: it raises privilege using the filesystem bits on the
executable to obtain a raw socket.
Under Charliecloud, there are multiple reasons ping
can’t get a raw
socket. First, images are unpacked without privilege, meaning that setuid and
setcap bits are lost. But even if you do get privilege in the container (e.g.,
with --uid=0
), this only applies in the container. Charliecloud uses
the host’s network namespace, where your unprivileged host identity applies
and ping
still can’t get a raw socket.
The recommended alternative is to simply try the thing you want to do, without
testing connectivity using ping
first.
6.3.3. Why is MATLAB trying and failing to change the group of /dev/pts/0
?¶
MATLAB and some other programs want pseudo-TTY (PTY) files to be group-owned
by tty
. If it’s not, Matlab will attempt to chown(2)
the file,
which fails inside a container.
The scenario in more detail is this. Assume you’re user charlie
(UID=1000), your primary group is nerds
(GID=1001), /dev/pts/0
is the PTY file in question, and its ownership is charlie:tty
(1000:5
), as it should be. What happens in the container by default
is:
- MATLAB
stat(2)
s/dev/pts/0
and checks the GID. - This GID is
nogroup
(65534) becausetty
(5) is not mapped on the host side (and cannot be, because only one’s EGID can be mapped in an unprivileged user namespace). - MATLAB concludes this is bad.
- MATLAB executes
chown("/dev/pts/0", 1000, 5)
. - This fails because GID 5 is not mapped on the guest side.
- MATLAB pukes.
The workaround is to map your EGID of 1001 to 5 inside the container (instead
of the default 1001:1001), i.e. --gid=5
. Then, step 4 succeeds because
the call is mapped to chown("/dev/pts/0", 1000, 1001)
and MATLAB is
happy.
6.4. How do I …¶
6.4.1. My app needs to write to /var/log
, /run
, etc.¶
Because the image is mounted read-only by default, log files, caches, and other stuff cannot be written anywhere in the image. You have three options:
- Configure the application to use a different directory.
/tmp
is often a good choice, because it’s shared with the host and fast. - Use
RUN
commands in your Dockerfile to create symlinks that point somewhere writeable, e.g./tmp
, or/mnt/0
withch-run --bind
. - Run the image read-write with
ch-run -w
. Be careful that multiple containers do not try to write to the same files.
6.4.2. Which specific sudo
commands are needed?¶
For running images, sudo
is not needed at all.
For building images, it depends on what you would like to support. For example, do you want to let users build images with Docker? Do you want to let them run the build tests?
We do not maintain specific lists, but you can search the source code and
documentation for uses of sudo
and $DOCKER
and evaluate them
on a case-by-case basis. (The latter includes sudo
if needed to invoke
docker
in your environment.) For example:
$ find . \( -type f -executable \
-o -name Makefile \
-o -name '*.bats' \
-o -name '*.rst' \
-o -name '*.sh' \) \
-exec egrep -H '(sudo|\$DOCKER)' {} \;
6.4.3. OpenMPI Charliecloud jobs don’t work¶
MPI can be finicky. This section documents some of the problems we’ve seen.
6.4.3.1. mpirun
can’t launch jobs¶
For example, you might see:
$ mpirun -np 1 ch-run /var/tmp/mpihello -- /hello/hello
App launch reported: 2 (out of 2) daemons - 0 (out of 1) procs
[cn001:27101] PMIX ERROR: BAD-PARAM in file src/dstore/pmix_esh.c at line 996
We’re not yet sure why this happens — it may be a mismatch between the OpenMPI
builds inside and outside the container — but in our experience launching with
srun
often works when mpirun
doesn’t, so try that.
6.4.3.2. Communication between ranks on the same node fails¶
OpenMPI has many ways to transfer messages between ranks. If the ranks are on the same node, it is faster to do these transfers using shared memory rather than involving the network stack. There are two ways to use shared memory.
The first and older method is to use POSIX or SysV shared memory segments.
This approach uses two copies: one from Rank A to shared memory, and a second
from shared memory to Rank B. For example, the sm
byte transport
layer (BTL) does this.
The second and newer method is to use the process_vm_readv(2)
and/or
process_vm_writev(2)
) system calls to transfer messages directly from
Rank A’s virtual memory to Rank B’s. This approach is known as cross-memory
attach (CMA). It gives significant performance improvements in benchmarks,
though of course the real-world impact depends on the application. For
example, the vader
BTL (enabled by default in OpenMPI 2.0) and
psm2
matching transport layer (MTL) do this.
The problem in Charliecloud is that the second approach does not work by default.
We can demonstrate the problem with LAMMPS molecular dynamics application:
$ srun --cpus-per-task 1 ch-run /var/tmp/lammps_mpi -- \
lmp_mpi -log none -in /lammps/examples/melt/in.melt
[cn002:21512] Read -1, expected 6144, errno = 1
[cn001:23947] Read -1, expected 6144, errno = 1
[cn002:21517] Read -1, expected 9792, errno = 1
[... repeat thousands of times ...]
With strace(1)
, one can isolate the problem to the system call noted
above:
process_vm_readv(...) = -1 EPERM (Operation not permitted)
write(33, "[cn001:27673] Read -1, expected 6"..., 48) = 48
The man page
reveals that these system calls require that the process have permission to
ptrace(2)
one another, but sibling user namespaces do not. (You can
ptrace(2)
into a child namespace, which is why gdb
doesn’t
require anything special in Charliecloud.)
This problem is not specific to containers; for example, many settings of kernels with YAMA enabled will similarly disallow this access.
So what can you do? There are a few options:
We recommend simply using the
--join
family of arguments toch-run
. This puts a group ofch-run
peers in the same namespaces; then, the system calls work. See the ch-run man page for details.You can also sometimes turn off single-copy. For example, for
vader
, set the MCA variablebtl_vader_single_copy_mechanism
tonone
, e.g. with an environment variable:$ export OMPI_MCA_btl_vader_single_copy_mechanism=none
psm2
does not let you turn off CMA, but it does fall back to two-copy if CMA doesn’t work. However, this fallback crashed when we tried it.The kernel module XPMEM enables a different single-copy approach. We have not yet tried this, and the module needs to be evaluated for user namespace safety, but it’s quite a bit faster than CMA on benchmarks.
6.4.4. How do I run X11 apps?¶
X11 applications should “just work”. For example, try this Dockerfile:
FROM debian:stretch
RUN apt-get update \
&& apt-get install -y xterm
Build it and unpack it to /var/tmp
. Then:
$ ch-run /scratch/ch/xterm -- xterm
should pop an xterm.
If your X11 application doesn’t work, please file an issue so we can figure out why.
6.4.5. How do I create a tarball compatible with Charliecloud?¶
In contrast with best practices for source code, Charliecloud expects an image
tarball to have either no top-level directory or a top-level directory that is
exactly .
(dot). This is inherited from the format of docker
export
tarballs. If you’re creating tarballs by other means, you may run into
this issue.
For example, let’s try to re-pack the chtest
image directory. This
fails with a rather opaque error message.
$ cd $CH_TEST_IMGDIR
$ tar czf chtest2.tar.gz chtest
$ tar tf chtest2.tar.gz | head
chtest/
chtest/var/
chtest/var/run/
chtest/var/empty/
chtest/var/spool/
chtest/var/spool/cron/
chtest/var/spool/cron/crontabs
chtest/var/opt/
chtest/var/local/
chtest/var/log/
$ ch-tar2dir chtest2.tar.gz .
$ ls chtest2
chtest dev mnt WEIRD_AL_YANKOVIC
$ ch-run ./chtest2 -- echo hello
ch-run[28780]: can't bind /etc/passwd to /var/tmp/images/chtest2/etc/passwd: No such file or directory (charliecloud.c:132 2)
The workaround is to create the tarball from within the image directory. (If
you do this immediately after the above, you’ll need to remove the
chtest2
directory first.)
$ ch $CH_TEST_IMGDIR/chtest
$ tar czf ../chtest2.tar.gz .
$ cd ..
$ tar tf chtest2.tar.gz | head
./
./var/
./var/run/
./var/empty/
./var/spool/
./var/spool/cron/
./var/spool/cron/crontabs
./var/opt/
./var/local/
./var/log/
$ ch-tar2dir chtest2.tar.gz .
$ ls chtest2
bin etc lib mnt root sbin sys tmp var
dev home media proc run srv test usr WEIRD_AL_YANKOVIC
$ ch-run ./chtest2 -- echo hello
hello
We are working on usability enhancements for this process.