Post

How to Setup Custom PyPI Index URL in Different Environments

Posted
By R G
8 min read

How to Setup Custom PyPI Index URL in Different Environments

When working with private Python packages or using a custom PyPI repository (like Azure Artifacts, AWS CodeArtifact, JFrog Artifactory, or a self-hosted PyPI server), you need to configure pip to use a custom index URL. This guide covers how to set up a private PyPI index URL across different environments.

Overview

The PyPI index URL tells pip where to download packages from. By default, pip uses https://pypi.org/simple, but you can configure it to use a private repository for proprietary packages or to mirror public packages.

1. Setup for GitHub Actions (PIP_INDEX_URL)

GitHub Actions supports environment variables that pip respects automatically. The PIP_INDEX_URL environment variable is the recommended approach.

Basic Setup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

name: Python CI

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        env:
          PIP_INDEX_URL: $
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

With Authentication

If your private PyPI server requires authentication, include credentials in the URL:

1
2
3
4
5
6

- name: Install dependencies with authentication
  env:
    # Format: https://username:password@your-pypi-server.com/simple
    PIP_INDEX_URL: $
  run: |
    pip install -r requirements.txt

Using Extra Index URL

If you want to use both the public PyPI and a private repository:

1
2
3
4
5
6

- name: Install dependencies with extra index
  env:
    PIP_INDEX_URL: https://pypi.org/simple
    PIP_EXTRA_INDEX_URL: $
  run: |
    pip install -r requirements.txt

Best Practices for GitHub Actions

  1. Store credentials in GitHub Secrets: Never hardcode URLs with credentials
  2. Use repository secrets: Go to Settings → Secrets and variables → Actions
  3. Format: https://username:password@pypi.example.com/simple

2. Setup for Dockerfile with pip install

When building Docker images, you can pass the PyPI index URL as a build argument to keep credentials secure.

Using Build Arguments

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

FROM python:3.11-slim

# Define build argument for PyPI index URL
ARG PIP_INDEX_URL=https://pypi.org/simple

# Set environment variable for pip
ENV PIP_INDEX_URL=${PIP_INDEX_URL}

WORKDIR /app

# Copy requirements file
COPY requirements.txt .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

CMD ["python", "app.py"]

Building with Custom Index URL

1
2
3
4

# Build with custom PyPI index
docker build \
  --build-arg PIP_INDEX_URL=https://username:password@pypi.example.com/simple \
  -t myapp:latest .

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# syntax=docker/dockerfile:1

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .

# Mount secret during build
RUN --mount=type=secret,id=pip_index_url \
    PIP_INDEX_URL=$(cat /run/secrets/pip_index_url) \
    pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Build with secret:

1
2
3
4
5
6
7
8
9
10

# Store your URL in a file
echo "https://username:password@pypi.example.com/simple" > /tmp/pip_index_url

# Build with secret
DOCKER_BUILDKIT=1 docker build \
  --secret id=pip_index_url,src=/tmp/pip_index_url \
  -t myapp:latest .

# Clean up
rm /tmp/pip_index_url

Multi-stage Build Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Build stage
FROM python:3.11-slim as builder

ARG PIP_INDEX_URL
ENV PIP_INDEX_URL=${PIP_INDEX_URL}

WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Runtime stage
FROM python:3.11-slim

WORKDIR /app

# Copy installed packages from builder
COPY --from=builder /root/.local /root/.local
COPY . .

# Update PATH
ENV PATH=/root/.local/bin:$PATH

CMD ["python", "app.py"]

3. Setup for pyproject.toml

For projects using modern Python packaging with pyproject.toml, you can configure pip through several methods.

Create or edit pip.conf or pip.ini:

Linux/macOS: ~/.config/pip/pip.conf or ~/.pip/pip.conf

Windows: %APPDATA%\pip\pip.ini or %HOME%\pip\pip.ini

1
2
3
4
5
6
7
8
9

[global]
index-url = https://username:password@pypi.example.com/simple

# Or use extra-index-url to include public PyPI
# index-url = https://pypi.org/simple
# extra-index-url = https://username:password@pypi.example.com/simple

[install]
trusted-host = pypi.example.com

Using Environment Variables

1
2

export PIP_INDEX_URL=https://username:password@pypi.example.com/simple
pip install .

In pyproject.toml (for Poetry)

If you’re using Poetry, configure the private repository:

1
2
3
4
5
6
7
8
9
10
11
12
13

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = "My project"

[[tool.poetry.source]]
name = "private-pypi"
url = "https://pypi.example.com/simple"
priority = "primary"

[[tool.poetry.source]]
name = "pypi"
priority = "supplemental"

Configure authentication separately:

1
2

poetry config repositories.private-pypi https://pypi.example.com/simple
poetry config http-basic.private-pypi username password

For PDM

1
2
3
4

[[tool.pdm.source]]
name = "private"
url = "https://pypi.example.com/simple"
verify_ssl = true

Configure credentials:

1
2
3

pdm config pypi.private.url https://pypi.example.com/simple
pdm config pypi.private.username myusername
pdm config pypi.private.password mypassword

4. Setup for Docker Compose

Docker Compose allows you to set environment variables and build arguments for services.

Using Environment Variables

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        PIP_INDEX_URL: ${PIP_INDEX_URL}
    environment:
      - PIP_INDEX_URL=${PIP_INDEX_URL}
    ports:
      - "8000:8000"
    volumes:
      - .:/app

Using .env File

Create a .env file in the same directory as docker-compose.yml:

1
2

# .env file
PIP_INDEX_URL=https://username:password@pypi.example.com/simple

Important: Add .env to .gitignore to avoid committing credentials!

# .gitignore
.env
*.env

With Build Secrets

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      secrets:
        - pip_index_url
    ports:
      - "8000:8000"

secrets:
  pip_index_url:
    file: ./secrets/pip_index_url.txt

Complete Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - PIP_INDEX_URL=${PIP_INDEX_URL:-https://pypi.org/simple}
    environment:
      - PYTHONUNBUFFERED=1
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    command: python manage.py runserver 0.0.0.0:8000
    
  worker:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - PIP_INDEX_URL=${PIP_INDEX_URL:-https://pypi.org/simple}
    command: celery -A myapp worker -l info
    environment:
      - PIP_INDEX_URL=${PIP_INDEX_URL}
    volumes:
      - .:/app

Run with:

1
2
3
4
5

# Set environment variable
export PIP_INDEX_URL=https://username:password@pypi.example.com/simple

# Or use .env file
docker-compose up --build

5. Setup for Local Development Environment

For local development, you have several options to configure pip.

Option 1: Environment Variables (Temporary)

1
2
3

# For current session only
export PIP_INDEX_URL=https://username:password@pypi.example.com/simple
pip install -r requirements.txt

Option 2: pip Configuration File (Persistent)

Create a configuration file for pip:

Linux/macOS:

1
2
3
4
5
6
7
8
9
10

mkdir -p ~/.config/pip
cat > ~/.config/pip/pip.conf << EOF
[global]
index-url = https://username:password@pypi.example.com/simple
trusted-host = pypi.example.com

[install]
# Optional: add extra index for public packages
# extra-index-url = https://pypi.org/simple
EOF

Windows:

mkdir %APPDATA%\pip
echo [global] > %APPDATA%\pip\pip.ini
echo index-url = https://username:password@pypi.example.com/simple >> %APPDATA%\pip\pip.ini
echo trusted-host = pypi.example.com >> %APPDATA%\pip\pip.ini

Option 3: Project-specific Configuration

Create pip.conf in your project root:

1
2
3
4
5
6

[global]
index-url = https://pypi.example.com/simple
extra-index-url = https://pypi.org/simple

[install]
trusted-host = pypi.example.com

Use with:

1

pip install --config-file=pip.conf -r requirements.txt

Option 4: Using Command-line Arguments

1
2
3
4
5
6
7
8

# Single index
pip install -r requirements.txt \
  --index-url https://username:password@pypi.example.com/simple

# With extra index (fallback to public PyPI)
pip install -r requirements.txt \
  --index-url https://pypi.org/simple \
  --extra-index-url https://username:password@pypi.example.com/simple

Option 5: Virtual Environment with direnv

Install direnv and create .envrc:

1
2

# .envrc
export PIP_INDEX_URL=https://username:password@pypi.example.com/simple

Allow the directory:

1

direnv allow .

Important: Add .envrc to .gitignore!

Option 6: Using keyring for Credential Management

For better security, use keyring to store credentials:

1
2
3
4
5
6
7
8

# Install keyring
pip install keyring

# Store credentials
keyring set https://pypi.example.com username

# Configure pip to use keyring
pip config set global.index-url https://pypi.example.com/simple

Create ~/.config/pip/pip.conf:

1
2
3

[global]
index-url = https://pypi.example.com/simple
keyring-provider = auto

Security Best Practices

  1. Never commit credentials: Always use environment variables or secrets
  2. Use .gitignore: Add .env, *.env, pip.conf with credentials
  3. Rotate credentials regularly: Change passwords periodically
  4. Use token authentication: Prefer tokens over username/password
  5. Limit token scope: Create tokens with minimum required permissions
  6. Use HTTPS: Always use secure connections to package repositories
  7. Verify SSL certificates: Don’t disable SSL verification unless absolutely necessary

Common Issues and Troubleshooting

Issue: SSL Certificate Verification Failed

1
2
3
4
5
6

# Temporary workaround (NOT recommended for production)
pip install --trusted-host pypi.example.com package-name

# Better: Add to pip.conf
[global]
trusted-host = pypi.example.com

Issue: Authentication Failed

Verify URL format:

1

https://username:password@pypi.example.com/simple

URL encode special characters in username/password:

1
2
3
4

from urllib.parse import quote
username = quote("user@example.com")
password = quote("p@ssw0rd!")
url = f"https://{username}:{password}@pypi.example.com/simple"

Issue: Package Not Found

Check index priority:

1
2
3

pip install --index-url https://pypi.example.com/simple \
           --extra-index-url https://pypi.org/simple \
           package-name -v

Summary

This guide covered setting up custom PyPI index URLs across different environments:

  • ✅ GitHub Actions: Use PIP_INDEX_URL environment variable
  • ✅ Dockerfile: Use build arguments and Docker secrets
  • ✅ pyproject.toml: Configure via pip.conf or tool-specific configuration
  • ✅ Docker Compose: Use environment variables and .env files
  • ✅ Local Development: Multiple options including pip.conf, environment variables, and keyring

Choose the method that best fits your workflow and always prioritize security when handling credentials.

CI-CD
python pip pypi docker dockerfile docker-compose github-actions CI/CD
This post is licensed under CC BY 4.0 by the author.