Environments 1.0

Content

Environments 1.0#

Python Environments#

Standardized Configuration#

While tools execute the installation, the environment and project specifications must be declaratively defined. The modern Python packaging standard dictates the use of a pyproject.toml file. This file serves as the centralized configuration matrix for the project, replacing legacy files such as requirements.txt, setup.py, and setup.cfg.

# ./pyproject.toml
[project]
name = "your_project_name"
authors = [
  { name="Your Name", email="your.email@example.com" },
]
description = "Brief description of your project"
readme = "README.md"
requires-python = "~=3.13.0"
license.file = "LICENSE"
dependencies = [ "numpy==2.3.5", ]
[project.urls]
Homepage = "https://github.com/j-i-l/pythonProject"

[dependency-groups]
test = [ "pytest~=7.4.3", ]
docs = [ "Sphinx==8.1.3", ]
dev = [{include-group = "test"},
       {include-group = "docs"},
       "black>=23.0", ]

[tool.hatch.version]
source = "vcs"

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

Comprehensive Project Configuration

The pyproject.toml structure is divided into specific tables that govern distinct aspects of the software project.

1. Build System Instructions ([build-system]) This table defines the backend required to package the codebase into installable distributions (wheels and source distributions). Standard backends include hatchling, flit-core, or setuptools.

2. Project Metadata and Versioning ([project]) Core project metadata is strictly formalized under PEP 621. This table standardizes the declaration of the project’s name, versioning scheme (which can be static or dynamically derived from Git tags), description, designated entry points, license types, and authorship information. It acts as the primary source of truth for repository information.

3. Dependency Declarations Within the [project] table, the runtime execution context is defined:

requires-python: Dictates the compatible Python interpreter versions.
dependencies: An array of the core external packages strictly required for the software to function.

4. Optional Dependencies ([project.optional-dependencies]) Optional dependencies are utilized to define distinct, compartmentalized execution contexts without polluting the core runtime environment. They permit the specification of additional packages required exclusively for specific workflows, such as executing test suites (e.g., including pytest and code coverage modules), generating documentation (e.g., sphinx), or configuring developer environments.

5. Tool Configuration ([tool.*]) The configuration of external developer tools is consolidated within the pyproject.toml. Tables such as [tool.pytest.ini_options] or [tool.ruff] dictate the behavior of test runners, static type checkers, linters, and code formatters, ensuring uniform configurations across all execution environments.

Interpreter vs. Compilers

An interpreter is a program that directly executes instructions written in a programming language, line by line, without requiring prior compilation into machine code. In Python’s case, the interpreter (e.g., python3) reads and runs .py files at runtime. Compilers translate the entire program at once, creating an executable file that runs quickly, whereas interpreters translate and execute code line-by-line, making them slower but easier for debugging.

Alternative Ecosystems

Standard Python tooling occasionally fails when projects require complex non-Python dependencies, such as specific C++ compilers, system-level libraries, or GPU drivers. In these instances, alternative ecosystems are utilized.

Conda The conda package manager isolates environments at the system level rather than strictly at the Python package level. It installs Python itself alongside necessary system binaries. Conda typically relies on an independent configuration format (environment.yml).

Pixi pixi is a modern package manager that leverages the Conda ecosystem for binary resolution but integrates directly into the standard Python configuration workflow. It bridges the gap between Python standards and Conda binaries by extending the pyproject.toml.

To utilize pixi, a dedicated [tool.pixi] table is appended to the pyproject.toml. This defines Conda-specific channels, dependencies, and system requirements without violating standard Python packaging rules:

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["linux-64", "osx-arm64"]

[tool.pixi.dependencies]
python = "3.11.*"
cuda-toolkit = "11.8.*"

Additional resources:

Structure and format of TOML files en.wikipedia.org/wiki/TOML, toml.io/en/latest

Python Environment Tooling#

System-wide Python package installations lead to dependency version conflicts across concurrent projects. This is mitigated by establishing isolated, project-specific virtual environments, ensuring that dependencies, interpreters, and execution contexts remain fully self-contained.

Standard Tooling: `venv`, `pip`, and `dotenv`#

The standard methodology relies on tools integrated directly into the core Python distribution, supplemented by external CLI utilities for environment management.

venv: A standard library module utilized to generate an isolated environment structure (typically named .venv). This directory contains a localized Python interpreter and standard library clone.
pip: The standard package installer. Upon manual environment activation (e.g., source .venv/bin/activate), pip routes all installed packages into the localized .venv directory rather than global system paths.
dotenv: To adhere to the Separation of Concerns principle, environment variables should not be loaded within the Python code itself. Instead, the dotenv CLI tool (provided by the python-dotenv package) is utilized to wrap the execution, injecting the variables from the .env file directly into the script’s environment:

dotenv run -- python scripts/data_processing.py

This baseline approach requires the target Python version to be manually pre-installed on the host system prior to environment creation.

Modern Tooling: `uv`#

Modern workflows frequently utilize uv, a comprehensive package and project manager. It replaces individual standard tools by consolidating interpreter management, environment creation, and dependency resolution into a single utility.

Standard Compatibility Despite operating as an independent manager, uv is strictly compatible with standard Python conventions. It generates a standard .venv directory. Users retain the ability to manually activate the environment (source .venv/bin/activate) and interact with it using standard commands.

Automated Synchronization (uv sync) The uv sync command parses the project’s configuration file to automatically download the specified Python interpreter, instantiate the .venv, and install all declared dependencies.

Isolated Execution (uv run) To execute code, uv run is utilized. This command automatically identifies the project’s virtual environment and executes the specified script strictly within that isolated context, bypassing the requirement for manual environment activation.

To prevent implicit state and unexpected behaviors, uv run intentionally ignores .env files by default. However, uv allows to pragmatically include environmental variables by setting the UV_ENV_FILE variable:

export UV_ENV_FILE=.env
uv run scripts/data_processing.py

To load local environment variables during execution, the target file can also be explicitly declared via the CLI flag:

uv run --env-file .env scripts/data_processing.py

Python Environment Lifecycle & Constraints#

While environment tooling automates package installation, the lifecycle and state management of these Python environments adhere to strict operational constraints. Failure to observe these constraints frequently results in corrupted execution states or breached reproducibility.

The Python-Domain Limitation#

A fundamental limitation of standard virtual environments is their strict confinement to the Python domain. In scientific computing, Python frequently functions merely as an API layer for underlying compiled libraries (e.g., C, C++, Fortran, Rust) utilized by packages such as NumPy, PyTorch, pandas, or SciPy.

While the Python wrapper is isolated within the .venv, the underlying non-Python dependencies (e.g., CUDA drivers, shared C libraries like glibc, system-level geospatial tools) are not isolated. They remain globally shared and dependent on the host operating system. Therefore, an identical Python environment configuration may fail to execute on a different compute node if the underlying OS or global system libraries differ.

Architecture and OS Incompatibility#

Beyond standard dependencies, a virtual environment is strictly bound to the Operating System and CPU architecture upon which it was created. When a .venv is initialized, system-specific compiled binaries (e.g., .so files on Linux, .dll on Windows) are downloaded.

Consequently, a .venv generated on a local workstation (e.g., macOS ARM or Windows) is fundamentally incompatible with a high-performance compute cluster (typically Linux x86_64). Environments cannot be transferred across different architectures; only the declarative configuration files (pyproject.toml, lockfiles) are transferred, and the .venv must be compiled natively on the target execution node.

Path Dependency and Portability#

Virtual environments are inherently non-portable. When a .venv directory is generated, absolute system paths are hardcoded into its internal activation scripts and executable shebangs (e.g., #! /absolute/path/to/.venv/bin/python).

Moving or renaming a .venv directory, or relocating the parent project directory, severs these paths and corrupts the environment. If a project is relocated, the existing .venv must be deleted and entirely recreated. Due to this localized footprint and the thousands of system-specific binaries contained within, the .venv directory must always be excluded from version control via the project’s .gitignore file.

State Capture and Determinism#

A primary objective of environment isolation is exact reproducibility across different compute nodes. The methodology for capturing the environment state defines the degree of determinism achieved.

Loose State Capture (pip freeze) In standard workflows, the state of an active Python environment is manually captured utilizing command-line outputs (e.g., pip freeze > requirements.txt). This approach frequently fails to rigorously pin sub-dependencies, allowing upstream package updates to silently alter the execution context upon subsequent installations.

Deterministic Lockfiles (uv.lock) Modern package managers enforce strict reproducibility via lockfiles. When uv sync is executed, a strict lockfile (uv.lock) is generated. This file captures the exact versions, sub-dependencies, and cryptographic hashes of the entire installed Python ecosystem. When the codebase is deployed to a new compute node, the lockfile guarantees the recreation of an identical Python execution state, immune to upstream package mutations.

The Execution Context Disconnect (Jupyter & IDEs)#

A critical misconception involves the scope of environment activation. Activating a virtual environment in a terminal merely alters the shell’s $PATH environment variable, prioritizing the local Python executable. It does not create an isolated system container.

This mechanism frequently causes execution disconnects when utilizing external interfaces such as IDEs (e.g., VS Code) or Jupyter Notebooks. Executing a Jupyter server from a global environment while a local .venv is “activated” in the terminal background will result in the notebook executing against the global environment, causing ModuleNotFoundError exceptions. To utilize an isolated environment within Jupyter, the environment must be explicitly registered as an executable kernel (e.g., via ipykernel), or the Jupyter server itself must be installed and executed strictly within the local .venv.