Worker Images

Worker images are Docker containers that define the reproducible execution environment for your notebooks, pipelines, live apps, and dev environments in MINEO. They encapsulate every Python package, system library, and tool your code needs, so each run uses the identical environment and solves a critical challenge in data science and ML workflows: environment consistency.

The Problem: "It works on my machine" syndrome. Different team members have different package versions, code breaks in production, notebooks fail to reproduce results.

The Solution: Worker images are Docker containers that define exactly which Python packages, system libraries, and tools your code needs. Once defined, every execution—whether it's you, your teammate, or production—uses the identical environment.

Key Benefits:

• 🔒 Reproducibility: Results are consistent across runs and team members
• 🚀 Faster onboarding: New team members don't spend days setting up environments
• 🛡️ Version control: Track and rollback environment changes like code
• 🎯 Specialized environments: Different projects can use different package versions without conflicts

Usage contexts

Worker images operate in one of two usage contexts:

Context	Description	Use Cases
Notebook	Default context for data science and ML workflows	Notebooks, Pipelines
Live Apps	Context for interactive web applications	Live Apps
Dev Environment	Context for cloud-based development environments. Only predefined images available	Dev Environments (VS Code)

When creating or filtering worker images, the usage context determines where the image can be used. Each context has its own set of predefined base images optimized for that purpose.

In the worker image General tab, the Type field selects this usage context. It controls which predefined base images you can pick in the Dockerfile editor and where the resulting image becomes available for selection.

Quick start

New to Worker Images? Follow this 5-minute guide:

1. Go to Project Settings → Workers → Images
2. Click New Worker Image and give it a name
3. In the General tab, paste this Dockerfile:

   FROM <MINEO_BASE_IMAGE_PLACEHOLDER>
   RUN pip install pandas==2.0.3 scikit-learn==1.3.0

4. (Optional) Set the Docker build context to a specific folder in your project if your Dockerfile references local files (e.g., COPY or ADD instructions). By default it uses the project root (/)
5. Select a Worker environment (CPU, RAM, GPU) that fits your build workload
6. Click Build button
7. Wait for build to complete (check the status column)
8. Use the new worker image when creating or editing resources (notebooks, pipelines, live apps)

What just happened? You created a custom environment with specific package versions that will now be used every time you run notebooks, pipelines, live apps, etc... with this image.

How worker images work

The worker image workflow follows these steps:

Workflow Steps:

1. Define: Write a Dockerfile specifying your dependencies (Python packages, system libraries, tools)
2. Build: MINEO builds the Docker image within its infrastructure (no external setup needed)
3. Tag: Each successful build creates a versioned tag you can reference
4. Assign: Select the worker image when creating notebooks, pipelines, live apps, or dev environments
5. Execute: Your code runs in the exact environment you defined

The build process happens entirely within MINEO's infrastructure, so you don't need to configure external build environments or Docker registries.

Understanding worker images

Worker images in MINEO serve several important purposes:

• Isolated Execution Environment: Each notebook, pipeline, or dev environment runs in its own containerized environment
• Dependency Management: Install and manage custom Python packages and system dependencies
• Reproducibility: Ensure code executes consistently across different machines and environments
• Version Control: Tag and track different image versions for different workloads

MINEO provides both predefined base images and the ability to create custom worker images with your specific requirements. Additionally, MINEO includes predefined global images for each context, optimized for their specific purpose.

Creating a worker image PRO

To create a new worker image in your project, you have multiple options:

1. From Project Settings (main method):

   • Navigate to your project in MINEO
   • Go to Settings → Workers → Images
   • Click the New Worker Image button
   • Enter a name and confirm

2. Via the Resource Creation Dialog:

   • When creating a new notebook, live app, or pipeline, click "New Worker Image" in the creation dialog

3. Through Resource Properties:

   • Open an existing notebook, live app, or pipeline properties
   • Look for the worker image settings to create a new one

After creating the image, you will be redirected to the worker image detail page where you can configure the Dockerfile, build the image, and manage tags.

Configuring worker images

General tab

In the General tab, configure the basic parameters of your worker image:

• Active: Enable or disable the worker image. Disabled images won't be visible for selection
• Description: Add a description to help your team understand the purpose of this image
• Worker environment: Select machine resource options (RAM, CPU, GPU) for the image
• Dockerfile: Define your custom Docker configuration with a standard Dockerfile
• Docker build context: The folder within your project that will be available to the Docker build process. Defaults to / (project root). This is the directory that COPY and ADD instructions in your Dockerfile will resolve relative to. Set it to a specific subfolder if you want to limit the build scope or reference files from a particular location in your project

You can start your Dockerfile in two ways:

Option 1: Using MINEO base images

FROM <MINEO_BASE_IMAGE_PLACEHOLDER>

# Add your custom packages
RUN pip install scikit-learn pandas matplotlib seaborn

# Install system dependencies if needed
RUN apt-get update && apt-get install -y \
    ffmpeg \
    && rm -rf /var/lib/apt/lists/*

Option 2: Using public Docker images

FROM python:3.11-slim

# Install your required packages
RUN pip install scikit-learn==1.3.0 pandas==2.0.3 numpy==1.24.3

# Add custom system dependencies
RUN apt-get update && apt-get install -y \
    libgomp1 \
    && rm -rf /var/lib/apt/lists/*

Advanced Dockerfile examples

Multi-stage build (recommended for production)

Multi-stage builds significantly reduce final image size by separating build dependencies from runtime dependencies:

# Build stage - includes build tools
FROM python:3.11-slim as builder
WORKDIR /app

# Install build dependencies
RUN pip install --user --no-cache-dir \
    pandas==2.0.3 \
    scikit-learn==1.3.0 \
    numpy==1.24.3

# Runtime stage - minimal image
FROM python:3.11-slim
WORKDIR /app

# Copy only installed packages from builder
COPY --from=builder /root/.local /root/.local

# Update PATH to find installed packages
ENV PATH=/root/.local/bin:$PATH

# Verify installation
RUN python -c "import pandas; import sklearn; print('Success!')"

Benefits:

• 30-50% smaller final image size
• Faster resource startup times
• Only runtime dependencies in production

MINEO Assistant

MINEO provides an AI-powered assistant to help you write and optimize your Dockerfiles. Located in the Dockerfile editor, you'll find two helpful tools:

• Optimize button: Automatically optimizes your Dockerfile for better performance, smaller size, and best practices (e.g., layer caching, multi-stage builds, pinned versions)
• Ask assistant (... button): Open a prompt dialog where you can request specific changes, such as:
  • "Switch to multi-stage build and pin all versions"
  • "Add TensorFlow with GPU support"
  • "Minimize the image size"
  • "Add system dependencies for video processing"

The assistant understands Dockerfile best practices and MINEO's specific requirements, making it easier to create optimized worker images.

Building your image

Once you've configured your Dockerfile in the General tab, click the Build button (located at the top of the General tab) to start building your worker image. The build process runs within MINEO's infrastructure.

Important: Build time counts towards your project's usage time. Optimize your Dockerfiles to minimize build duration and costs.

Builds tab

The Builds tab shows the history and status of all your builds:

1. View the build history of your worker image
2. Monitor build status (Pending, Running, Completed, Failed)
3. View the size of each build to optimize for faster resource creation times
4. Access detailed build logs for troubleshooting failed builds

Builds are executed within MINEO's infrastructure, so you don't need to configure external build environments. Each successful build generates a new tag for your image. Smaller image sizes result in faster startup times when creating notebooks, pipelines, or live apps.

Tags tab

The Tags tab allows you to:

1. View all available tags (versions) of your worker image
2. See the size of each tag to identify opportunities for optimization
3. Set a specific tag as the default for your project
4. Delete unused or outdated tags
5. View when each tag was created and last used

The size column helps you identify which tags are most efficient. Smaller images lead to faster resource creation and lower storage costs.

Permissions tab

The Permissions tab controls the image's visibility scope. Organization administrators set the Access to one of:

• Project: only members of the image's project can use it (default).
• Organization: every member of the organization can use it, in any project.

Global images (the built-in MINEO catalog) are read-only and their access can't be changed.

Organization-wide sharing

Organization administrators can share a worker image with the whole organization, making it available across every project.

• Share: Set Access to Organization in the image's Permissions tab, or open the ... menu in the project's worker image list and select Visibility → Organization. The image then appears in every project's worker image selector and in the organization's Workspace tab.
• Stop sharing: Set Access back to Project in the Permissions tab, or open the ... menu on the image in the organization's Workspace tab and select Visibility → Project.
• Badge: Shared images display an Organization badge and are marked with a team icon in selectors.

The image always keeps its original project association, so it returns there when you stop sharing.

Stopping sharing detaches other projects

When you stop sharing, the image becomes private to its own project again. The confirmation dialog lists every resource currently using the image, then on confirm:

• Resources (notebooks, pipelines, live apps, dev environments) in other projects are unlinked and fall back to their project's default image.
• Other projects that used it as their default image are reset to the global default image.
• Resources in the image's own project are unaffected.

Deleting a shared image

An organization-shared image can't be deleted from the project worker image list. Stop sharing it first (set Access to Project), then delete it.

Using worker images

Once you've created and built a worker image, you can use it in:

1. Notebooks: Select the worker image when creating a new notebook or change it in notebook settings
2. Pipelines: Specify the worker image in pipeline configuration
3. Live Apps: Choose the worker image when deploying a live app
4. Project Settings: Set a default worker image for all new notebooks in your project
5. Dev Environments: Currently only predefined images are available for dev environments

Performance guidelines

Understanding image size and build performance helps optimize your workflow and reduce costs.

Image size recommendations

Size	Startup Time	Status	Use Case
< 500 MB	10-20s	✅ Optimal	Basic Python packages
500 MB - 1 GB	20-30s	✅ Optimal	Standard data science stack
1 GB - 2 GB	30-60s	⚠️ Acceptable	ML frameworks (scikit-learn, XGBoost)
2 GB - 3 GB	1-2 min	⚠️ Acceptable	Deep learning (TensorFlow, PyTorch)
> 3 GB	2+ min	❌ Too large	Needs optimization

Impact: Smaller images mean faster startup when creating notebooks, pipelines, or live apps. Every team member and every execution benefits from optimized images.

Resource limits

• Maximum image size: 10 GB (contact support if you need more)
• Maximum build time: 60 minutes
• Build timeout: Automatic cancellation after 60 minutes of inactivity
• Concurrent builds: Limited per organization (builds queue automatically)

Optimization tips

1. Use multi-stage builds: Reduce final image by 30-50%
2. Add --no-cache-dir to pip: Saves 100-500 MB
3. Clean apt cache: Add && rm -rf /var/lib/apt/lists/* after apt-get
4. Order layers wisely: Put frequently changing code last
5. Use MINEO Assistant: Click "Optimize" for automatic improvements

Best practices

• Keep images small: Only include necessary dependencies to reduce build time and image size. Smaller images significantly reduce resource creation times when launching notebooks, pipelines, or live apps. Use the size column in Builds and Tags tabs to monitor your image size.
• Use the MINEO Assistant: Leverage the Optimize button to automatically apply Dockerfile best practices like multi-stage builds, layer caching, and version pinning
• Pin package versions: Specify exact versions (e.g., pip install pandas==2.0.3) to ensure reproducibility
• Document your images: Include comments in your Dockerfile and add a clear description
• Version control: Use meaningful tag names and clean up unused tags regularly
• Test thoroughly: Verify your worker image with test notebooks before using in production
• Share across projects: Promote worker images to the organization level to make them available to every project in your organization — ideal for standard base images your whole team uses

Security best practices

Use trusted base images

• Use official images: python:3.11-slim, ubuntu:22.04, or <MINEO_BASE_IMAGE_PLACEHOLDER>
• Always specify version tags (avoid :latest)
• Avoid unverified third-party images

Never include sensitive data

# ❌ NEVER hardcode secrets
ENV API_KEY=sk-1234567890abcdef
COPY .env /app/

# ✅ Use MINEO's secrets management instead

Pin package versions

# ✅ Always pin versions for security and reproducibility
RUN pip install pandas==2.0.3 requests==2.31.0

Keep images minimal and updated

• Use slim base images (python:3.11-slim)
• Only install necessary packages
• Rebuild regularly to get security patches
• Use multi-stage builds to remove build tools

Troubleshooting

If you encounter issues with your worker image:

1. Check the build logs for error messages
2. Verify that your Dockerfile syntax is correct
3. Ensure all required dependencies are properly installed
4. Test with a simple notebook to isolate problems
5. Look for compatibility issues between package versions

Conclusion

Worker images are a powerful feature in MINEO that give you complete control over your execution environment. By creating custom images tailored to your specific needs, you can ensure reliable and reproducible execution for all your notebooks, pipelines, live apps, and dev environments.

See also

• Notebooks — run code in the environment your worker image defines.
• Pipelines — orchestrate scheduled workflows using a worker image.
• Live Apps — deploy interactive apps backed by a worker image.
• Dev Environments — cloud-based VS Code environments built on predefined images.