Container Image¶
The Novelty container image holds Novelty, a Java runtime and 18 Ubuntu packages, so a vulnerability scanner has few operating system packages to report on. This page covers what the image contains, which tools a running container has and which it lacks, how the image's health check finds the web server, and how to build an image of your own that adds packages.
What the image contains¶
The image is a minimal Ubuntu 24.04 root filesystem with the Java runtime and Novelty on top.
| Part | Detail |
|---|---|
| Java runtime | Eclipse Temurin 21 JRE in /opt/java/openjdk, including java and keytool |
| Shell | bash, also available as sh |
| Metrics agent | The Prometheus JMX exporter (/jmx_prometheus_javaagent.jar, with its configuration in /exporter.yaml) |
| File and text tools | ls, cat, head, tail, grep, sed, cut, tr, find, xargs, stat, mkdir, mktemp, touch, mv, rm, rmdir, ln, readlink, chmod, chown, basename, dirname, base64 |
| Other tools | sleep, timeout, env, printenv, pwd, date, id, uname, nproc, test, locale |
| Ubuntu packages | base-files, base-passwd, bash, coreutils, findutils, gcc-14-base, grep, libacl1, libattr1, libc-bin, libc6, libcap2, libgcc-s1, libpcre2-8-0, libselinux1, libstdc++6, libtinfo6, sed |
| Locales | en_US.UTF-8 (the default) and C.UTF-8 |
| Time zone | Etc/UTC |
| User | root |
| Data volume | /var/thatdot |
The image does not have curl, wget, ps, top, tar, cp, wc or sort. It has no package manager (apt or dpkg), no OpenSSL, and no system certificate bundle in /etc/ssl/certs.
Work in a running container¶
docker exec -it <container> bash and kubectl exec -it <pod> -- bash open a shell in the container. The tools in What the image contains work there, for example ls /var/thatdot and cat /etc/os-release.
A command that starts a tool the image doesn't have fails. docker exec <container> curl, for example, reports exec: "curl": executable file not found in $PATH, and a script or probe that runs curl in the container fails the same way.
Bash covers some of the missing tools:
| Task | Command |
|---|---|
| Copy a file | cat <source> > <destination> |
| List processes | grep -H Name /proc/[0-9]*/status |
| Test a TCP port | (exec 3<>/dev/tcp/127.0.0.1/8080) && echo open |
When that is not enough, pick the option that fits:
- Send a request to Novelty: send it from outside the container, to the port the container publishes. A Kubernetes
httpGetprobe also runs outside the container. -
Inspect a pod with tools: attach a debug container whose image has them. The debug container joins the pod, and
--targetgives it the process namespace of the Novelty container:kubectl debug -it <pod> --image=busybox --target=<container> -
Run a probe inside the container: use the
/dev/tcpredirection of bash, as the image's own health check does.
Copy files to and from a container¶
kubectl cp does not work with this image, because it runs tar inside the container. It fails with exec: "tar": executable file not found in $PATH. Use kubectl exec with cat:
# From the pod to your machine
kubectl exec <pod> -- cat /tmp/heap.hprof > heap.hprof
# From your machine to the pod
kubectl exec -i <pod> -- bash -c 'cat > /tmp/data.json' < data.json
docker cp works as usual, because it does not run a command in the container.
Run as a non-root user or with a read-only root filesystem¶
The container runs as root by default. Novelty also runs as another user, such as --user 1000:1000 or a pod with runAsNonRoot, when that user can write to the data volume. The image has no named user other than the system users, so the user has no name and its home directory is /.
With a read-only root filesystem, mount a writable directory on /tmp as well as on the data volume. Novelty unpacks native libraries into /tmp and loads them from there, so the mount must allow execution. A Kubernetes emptyDir does. With docker run --read-only, use --tmpfs /tmp:exec, because a Docker tmpfs does not allow execution by default.
Health check¶
The image defines a Docker health check. Every 10 seconds, it sends a Process Liveness: GET /api/v2/system/liveness request over plain HTTP to port 8080 of 127.0.0.1 inside the container. The container is healthy when the response has a 2xx status. Each check times out after 2 seconds, and the start period is 5 seconds.
Set QUINE_HEALTHCHECK_PORT when the web server uses another port or TLS
When you change thatdot.novelty.webserver.port, the health check still asks port 8080, and Docker reports the container as unhealthy while Novelty is working. Set the QUINE_HEALTHCHECK_PORT environment variable of the container to the same port. For a web server on port 8081, add -e QUINE_HEALTHCHECK_PORT=8081 to docker run, or set the variable under environment: in a Compose file.
The health check uses plain HTTP. When use-tls is on for the web server, Docker reports the container as unhealthy while Novelty serves HTTPS. Setting both the SSL_KEYSTORE_PATH and SSL_KEYSTORE_PASSWORD environment variables also turns use-tls on. In that case, enable the health endpoints, which Novelty serves over plain HTTP on 127.0.0.1, port 8081 by default, and set QUINE_HEALTHCHECK_PORT to that port:
thatdot.novelty.webserver.use-mtls.health-endpoints.enabled = true
The health status matters to Docker and Docker Compose, where depends_on with condition: service_healthy and docker compose up --wait both wait for a healthy container. Kubernetes does not run the image's health check. It uses the probes in the pod specification. The default probes of the Novelty Helm chart are httpGet requests.
The health check is this bash command. A probe of your own that has to run inside the container can use the same approach:
exec 3<>"/dev/tcp/127.0.0.1/${QUINE_HEALTHCHECK_PORT:-8080}" \
&& printf 'GET /api/v2/system/liveness HTTP/1.0\r\nHost: localhost\r\nConnection: close\r\n\r\n' >&3 \
&& read -r _ code _ <&3 && code=${code%$'\r'} && [[ $code == 2[0-9][0-9] ]]
Run a probe like this with /bin/bash -c. In a Compose file, write each $ of the command as $$. The same form works wherever a Docker health check command is set, such as an Amazon ECS task definition: ["CMD", "/bin/bash", "-c", "<command>"].
Add packages to the image¶
An image built FROM the Novelty image can't install packages, so a RUN apt-get install instruction fails. A RUN instruction can use bash, java, keytool and the tools in What the image contains.
To add an Ubuntu package, cut it with chisel in a separate build stage and copy the result onto the Novelty image. Chisel is the tool that builds the root filesystem of the Novelty image. It installs a named part of a package, called a slice, together with the slices that part depends on, into a directory.
Each package you add brings its dependencies with it, and a vulnerability scanner reports on all of them.
curl brings vulnerabilities with it
The example below adds curl, because it is the package most often asked for. curl depends on many libraries, and recent scans of curl and those libraries have found vulnerabilities, some with no fix yet in Ubuntu. That is why the Novelty image does not include it. Scan your image after you add it.
A plain HTTP check does not need curl. To send a request to Novelty from inside the container, use the /dev/tcp redirection of bash, as the image's own health check does.
This Dockerfile adds curl and the certificate bundle it needs for HTTPS:
FROM ubuntu:24.04 AS extra
ARG TARGETARCH
ARG CHISEL_VERSION=v1.5.1
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl jq zstd
RUN curl -fsSL "https://github.com/canonical/chisel/releases/download/${CHISEL_VERSION}/chisel_${CHISEL_VERSION}_linux_${TARGETARCH}.tar.gz" \
| tar -xz -C /usr/local/bin chisel
# Cut the slices into /extra
RUN mkdir /extra \
&& chisel cut --release ubuntu-24.04 --root /extra \
base-files_chisel curl_bins ca-certificates_data
# Record each package for vulnerability scanners
RUN set -eu; \
records=/extra/var/lib/dpkg/status.d; \
mkdir -p "$records"; \
zstd -dc /extra/var/lib/chisel/manifest.wall > /tmp/manifest.jsonl; \
jq -r 'select(.kind == "package") | "\(.name) \(.version) \(.arch)"' /tmp/manifest.jsonl > /tmp/packages; \
while read -r name version arch; do \
src=$(apt-cache show "$name=$version" | awk '/^Source:/ { print $2; exit }'); \
{ echo "Package: $name"; \
echo "Status: install ok installed"; \
echo "Architecture: $arch"; \
echo "Version: $version"; \
if [ -n "$src" ]; then echo "Source: $src"; fi; \
} > "$records/$name"; \
done < /tmp/packages; \
rm -rf /extra/var/lib/chisel
FROM registry.license.thatdot.com/thatdot/novelty:<version>
COPY --from=extra /extra /
The parts of the Dockerfile that matter:
--release ubuntu-24.04: the slices must come from the Ubuntu release of the Novelty image.curl_bins ca-certificates_data: the slices to add. Replace them with the slices of the package you need. The slices of each package are defined in the chisel-releases repository, in one file for each package. A package with no file there can't be added this way.base-files_chisel: makes chisel write the list of packages it cut (manifest.wall), which the next instruction reads.- The package records: chisel does not write the package database that scanners read. The last
RUNinstruction writes one record for each package into/var/lib/dpkg/status.d/, which is where the Novelty image keeps its own records. Without them, a scanner doesn't see the packages you added.
To add tools for debugging only, a single statically linked binary is enough, because it needs no libraries from the image. This Dockerfile adds BusyBox. Run its tools as busybox ps, busybox tar and so on:
FROM registry.license.thatdot.com/thatdot/novelty:<version>
COPY --from=busybox:stable-musl /bin/busybox /usr/local/bin/busybox
A vulnerability scanner doesn't see a binary that is copied in this way, because no package record describes it.
Trust a private certificate authority¶
Novelty uses the trust store of the Java runtime, /opt/java/openjdk/lib/security/cacerts. The image has no system certificate bundle and no update-ca-certificates command.
To trust a private certificate authority, add its certificate to the Java trust store in an image of your own:
FROM registry.license.thatdot.com/thatdot/novelty:<version>
COPY my-ca.pem /tmp/my-ca.pem
RUN keytool -importcert -noprompt -cacerts -storepass changeit \
-alias my-ca -file /tmp/my-ca.pem
Time zone and locale¶
The default time zone of Novelty in the container is Etc/UTC. To use a different one, set the TZ environment variable of the container, for example -e TZ=America/New_York on docker run.
The default locale is en_US.UTF-8. The image also has C.UTF-8, and no other locale.
Vulnerability scanning¶
The image identifies itself as Ubuntu 24.04 in /etc/os-release, and records its Ubuntu packages in /var/lib/dpkg/status.d/, with one file for each package. This is the layout that distroless images use. Trivy reads it and reports the 18 Ubuntu packages listed in What the image contains.
The Java runtime is not an Ubuntu package, so it is not in these records, and Trivy does not list it or report vulnerabilities in it. A tool that reads the runtime's release file does list it: Syft, for example, reports it as openjdk with its version.