Picture by Editor
# Introduction
Reproducibility fails in boring methods. A wheel compiled in opposition to the “mistaken” glibc, a base picture that shifted underneath your ft, or a pocket book that labored as a result of your laptop computer had a stray system library put in from six months in the past.
Docker can cease all of that, however provided that you deal with the container like a reproducible artifact, not a disposable wrapper.
The methods beneath deal with the failure factors that truly chunk knowledge science groups: dependency drift, non-deterministic builds, mismatched central processing items (CPUs) and graphics processing items (GPUs), hidden state in photos, and “works on my machine” run instructions no one can reconstruct.
# 1. Locking Your Base Picture on the Byte Stage
Base photos really feel secure till they quietly should not. Tags transfer, upstream photos get rebuilt for safety patches, and distribution level releases land with out warning. Rebuilding the identical Dockerfile weeks later can produce a unique filesystem even when each utility dependency is pinned. That is sufficient to change numerical conduct, break compiled wheels, or invalidate prior outcomes.
The repair is easy and brutal: lock the bottom picture by digest. A digest pins the precise picture bytes, not a transferring label. Rebuilds develop into deterministic on the working system (OS) layer, which is the place most “nothing modified however every part broke” tales truly begin.
FROM python:slim@sha256:REPLACE_WITH_REAL_DIGEST
Human-readable tags are nonetheless helpful throughout exploration, however as soon as an atmosphere is validated, resolve it to a digest and freeze it. When outcomes are questioned later, you might be not defending a obscure snapshot in time. You might be pointing to a precise root filesystem that may be rebuilt, inspected, and rerun with out ambiguity.
# 2. Making OS Packages Deterministic and Preserving Them in One Layer
Many machine studying and knowledge tooling failures are OS-level: libgomp, libstdc++, openssl, build-essential, git, curl, locales, fonts for Matplotlib, and dozens extra. Putting in them inconsistently throughout layers creates hard-to-debug variations between builds.
Set up OS packages in a single RUN step, explicitly, and clear apt metadata in the identical step. This reduces drift, makes diffs apparent, and prevents the picture from carrying hidden cache state.
RUN apt-get replace
&& apt-get set up -y --no-install-recommends
build-essential
git
curl
ca-certificates
libgomp1
&& rm -rf /var/lib/apt/lists/*
One layer additionally improves caching conduct. The atmosphere turns into a single, auditable choice level reasonably than a sequence of incremental adjustments that no one desires to learn.
# 3. Splitting Dependency Layers So Code Modifications Do Not Rebuild the World
Reproducibility dies when iteration will get painful. If each pocket book edit triggers a full reinstall of dependencies, folks cease rebuilding, then the container stops being the supply of reality.
Construction your Dockerfile so dependency layers are secure and code layers are unstable. Copy solely dependency manifests first, set up, then copy the remainder of your challenge.
WORKDIR /app
# 1) Dependency manifests first
COPY pyproject.toml poetry.lock /app/
RUN pip set up --no-cache-dir poetry
&& poetry config virtualenvs.create false
&& poetry set up --no-interaction --no-ansi
# 2) Solely then copy your code
COPY . /app
This sample improves each reproducibility and velocity. Everyone rebuilds the identical atmosphere layer, whereas experiments can iterate with out altering the atmosphere. Your container turns into a constant platform reasonably than a transferring goal.
# 4. Preferring Lock Recordsdata Over Unfastened Necessities
A necessities.txt that pins solely top-level packages nonetheless leaves transitive dependencies free to maneuver. That’s the place “identical model, completely different outcome” typically comes from. Scientific Python stacks are delicate to minor dependency shifts, particularly round compiled wheels and numerical kernels.
Use a lock file that captures the complete graph: Poetry lock, uv lock, pip-tools compiled necessities, or Conda express exports. Set up from the lock, not from a hand-edited record.
For those who use pip-tools, the workflow is easy:
- Keep necessities.in
- Generate a totally pinned necessities.txt with hashes
- Set up precisely that in Docker
COPY necessities.txt /app/
RUN pip set up --no-cache-dir -r necessities.txt
Hash-locked installs make provide chain adjustments seen and scale back the “it pulled a unique wheel” ambiguity.
# 5. Encoding Execution as A part of the Artifact With ENTRYPOINT
A container that wants a 200-character docker run command to breed outcomes will not be reproducible. Shell historical past will not be a constructed artifact.
Outline a transparent ENTRYPOINT and default CMD so the container paperwork the way it runs. Then you possibly can override arguments with out reinventing the entire command.
COPY scripts/practice.py /app/scripts/practice.py
ENTRYPOINT ["python", "-u", "/app/scripts/train.py"]
CMD ["--config", "/app/configs/default.yaml"]
Now the “how” is embedded. A teammate can rerun coaching with a unique config or seed whereas nonetheless utilizing the identical entry path and defaults. CI can execute the picture with out bespoke glue. Six months later, you possibly can run the identical picture and get the identical conduct with out reconstructing tribal information.
# 6. Making {Hardware} and GPU Assumptions Specific
{Hardware} variations should not theoretical. CPU vectorization, MKL/OpenBLAS threading, and GPU driver compatibility can all change outcomes or efficiency sufficient to change coaching dynamics. Docker doesn’t erase these variations. It may disguise them till they trigger a complicated divergence.
For CPU determinism, set threading defaults so runs don’t differ with core counts:
ENV OMP_NUM_THREADS=1
MKL_NUM_THREADS=1
OPENBLAS_NUM_THREADS=1
For GPU work, use a CUDA base picture aligned together with your framework and doc it clearly. Keep away from obscure “newest” CUDA tags. For those who ship a PyTorch GPU picture, the CUDA runtime selection is a part of the experiment, not an implementation element.
Additionally, make the runtime requirement apparent in utilization docs. A reproducible picture that silently runs on CPU when GPU is lacking can waste hours and produce incomparable outcomes. Fail loudly when the mistaken {hardware} path is used.
# Wrapping Up
Docker reproducibility will not be about “having a container.” It’s about freezing the atmosphere at each layer that may drift, then making execution and state dealing with boringly predictable. Immutable bases cease OS surprises. Steady dependency layers maintain iteration quick sufficient that folks truly rebuild. Put all of the items collectively and reproducibility stops being a promise you make to others and turns into one thing you possibly can show with a single picture tag and a single command.
Nahla Davies is a software program developer and tech author. Earlier than devoting her work full time to technical writing, she managed—amongst different intriguing issues—to function a lead programmer at an Inc. 5,000 experiential branding group whose shoppers embody Samsung, Time Warner, Netflix, and Sony.
Picture by Editor
# Introduction
Reproducibility fails in boring methods. A wheel compiled in opposition to the “mistaken” glibc, a base picture that shifted underneath your ft, or a pocket book that labored as a result of your laptop computer had a stray system library put in from six months in the past.
Docker can cease all of that, however provided that you deal with the container like a reproducible artifact, not a disposable wrapper.
The methods beneath deal with the failure factors that truly chunk knowledge science groups: dependency drift, non-deterministic builds, mismatched central processing items (CPUs) and graphics processing items (GPUs), hidden state in photos, and “works on my machine” run instructions no one can reconstruct.
# 1. Locking Your Base Picture on the Byte Stage
Base photos really feel secure till they quietly should not. Tags transfer, upstream photos get rebuilt for safety patches, and distribution level releases land with out warning. Rebuilding the identical Dockerfile weeks later can produce a unique filesystem even when each utility dependency is pinned. That is sufficient to change numerical conduct, break compiled wheels, or invalidate prior outcomes.
The repair is easy and brutal: lock the bottom picture by digest. A digest pins the precise picture bytes, not a transferring label. Rebuilds develop into deterministic on the working system (OS) layer, which is the place most “nothing modified however every part broke” tales truly begin.
FROM python:slim@sha256:REPLACE_WITH_REAL_DIGEST
Human-readable tags are nonetheless helpful throughout exploration, however as soon as an atmosphere is validated, resolve it to a digest and freeze it. When outcomes are questioned later, you might be not defending a obscure snapshot in time. You might be pointing to a precise root filesystem that may be rebuilt, inspected, and rerun with out ambiguity.
# 2. Making OS Packages Deterministic and Preserving Them in One Layer
Many machine studying and knowledge tooling failures are OS-level: libgomp, libstdc++, openssl, build-essential, git, curl, locales, fonts for Matplotlib, and dozens extra. Putting in them inconsistently throughout layers creates hard-to-debug variations between builds.
Set up OS packages in a single RUN step, explicitly, and clear apt metadata in the identical step. This reduces drift, makes diffs apparent, and prevents the picture from carrying hidden cache state.
RUN apt-get replace
&& apt-get set up -y --no-install-recommends
build-essential
git
curl
ca-certificates
libgomp1
&& rm -rf /var/lib/apt/lists/*
One layer additionally improves caching conduct. The atmosphere turns into a single, auditable choice level reasonably than a sequence of incremental adjustments that no one desires to learn.
# 3. Splitting Dependency Layers So Code Modifications Do Not Rebuild the World
Reproducibility dies when iteration will get painful. If each pocket book edit triggers a full reinstall of dependencies, folks cease rebuilding, then the container stops being the supply of reality.
Construction your Dockerfile so dependency layers are secure and code layers are unstable. Copy solely dependency manifests first, set up, then copy the remainder of your challenge.
WORKDIR /app
# 1) Dependency manifests first
COPY pyproject.toml poetry.lock /app/
RUN pip set up --no-cache-dir poetry
&& poetry config virtualenvs.create false
&& poetry set up --no-interaction --no-ansi
# 2) Solely then copy your code
COPY . /app
This sample improves each reproducibility and velocity. Everyone rebuilds the identical atmosphere layer, whereas experiments can iterate with out altering the atmosphere. Your container turns into a constant platform reasonably than a transferring goal.
# 4. Preferring Lock Recordsdata Over Unfastened Necessities
A necessities.txt that pins solely top-level packages nonetheless leaves transitive dependencies free to maneuver. That’s the place “identical model, completely different outcome” typically comes from. Scientific Python stacks are delicate to minor dependency shifts, particularly round compiled wheels and numerical kernels.
Use a lock file that captures the complete graph: Poetry lock, uv lock, pip-tools compiled necessities, or Conda express exports. Set up from the lock, not from a hand-edited record.
For those who use pip-tools, the workflow is easy:
- Keep necessities.in
- Generate a totally pinned necessities.txt with hashes
- Set up precisely that in Docker
COPY necessities.txt /app/
RUN pip set up --no-cache-dir -r necessities.txt
Hash-locked installs make provide chain adjustments seen and scale back the “it pulled a unique wheel” ambiguity.
# 5. Encoding Execution as A part of the Artifact With ENTRYPOINT
A container that wants a 200-character docker run command to breed outcomes will not be reproducible. Shell historical past will not be a constructed artifact.
Outline a transparent ENTRYPOINT and default CMD so the container paperwork the way it runs. Then you possibly can override arguments with out reinventing the entire command.
COPY scripts/practice.py /app/scripts/practice.py
ENTRYPOINT ["python", "-u", "/app/scripts/train.py"]
CMD ["--config", "/app/configs/default.yaml"]
Now the “how” is embedded. A teammate can rerun coaching with a unique config or seed whereas nonetheless utilizing the identical entry path and defaults. CI can execute the picture with out bespoke glue. Six months later, you possibly can run the identical picture and get the identical conduct with out reconstructing tribal information.
# 6. Making {Hardware} and GPU Assumptions Specific
{Hardware} variations should not theoretical. CPU vectorization, MKL/OpenBLAS threading, and GPU driver compatibility can all change outcomes or efficiency sufficient to change coaching dynamics. Docker doesn’t erase these variations. It may disguise them till they trigger a complicated divergence.
For CPU determinism, set threading defaults so runs don’t differ with core counts:
ENV OMP_NUM_THREADS=1
MKL_NUM_THREADS=1
OPENBLAS_NUM_THREADS=1
For GPU work, use a CUDA base picture aligned together with your framework and doc it clearly. Keep away from obscure “newest” CUDA tags. For those who ship a PyTorch GPU picture, the CUDA runtime selection is a part of the experiment, not an implementation element.
Additionally, make the runtime requirement apparent in utilization docs. A reproducible picture that silently runs on CPU when GPU is lacking can waste hours and produce incomparable outcomes. Fail loudly when the mistaken {hardware} path is used.
# Wrapping Up
Docker reproducibility will not be about “having a container.” It’s about freezing the atmosphere at each layer that may drift, then making execution and state dealing with boringly predictable. Immutable bases cease OS surprises. Steady dependency layers maintain iteration quick sufficient that folks truly rebuild. Put all of the items collectively and reproducibility stops being a promise you make to others and turns into one thing you possibly can show with a single picture tag and a single command.
Nahla Davies is a software program developer and tech author. Earlier than devoting her work full time to technical writing, she managed—amongst different intriguing issues—to function a lead programmer at an Inc. 5,000 experiential branding group whose shoppers embody Samsung, Time Warner, Netflix, and Sony.
