SelfHosting WandB

Self-Hosting Weights & Biases (wandb)

Weights & Biases (wandb) is a popular tool used by machine learning practitioners to track, visualize, and manage experiments. While wandb is typically used as a cloud-based service, many users and organizations prefer to self-host the platform for reasons related to data privacy, customization, and cost control. This article provides an in-depth look into whether and how you can self-host wandb.

Overview of wandb

wandb (Weights and Biases) is an open-source tool designed to facilitate experiment tracking, visualization, and organization of machine learning experiments. It offers features such as experiment dashboards, real-time metrics, hyperparameter tuning, and collaboration tools.

Cloud vs. Self-Hosting

By default, wandb is offered as a cloud-based SaaS (Software as a Service), where users create an account on wandb servers, and their experiments are stored and managed remotely. However, wandb also provides an option for self-hosting, allowing organizations to run the platform on their infrastructure.

Is Self-Hosting Supported?

Yes, wandb officially supports self-hosting. This is particularly useful for organizations that need to keep data on-premises, require custom integrations, or want to avoid vendor lock-in.

Requirements for Self-Hosting

Hardware Requirements

• A server machine with sufficient CPU and RAM to handle the expected load.
• Storage capacity to store experiment data, models, logs, and artifacts.
• Network setup to enable access within the organization.

Software Requirements

• Operating System: Linux (recommended, e.g., Ubuntu 20.04+)
• Docker: For containerized deployment (highly recommended)
• PostgreSQL: For database storage (can be hosted externally if preferred)
• Redis: For caching and message brokering

Dependencies

• Python 3.7+ (for client scripts and optional integrations)
• Other dependencies as specified in the wandb self-hosting documentation.

Setting Up a Self-Hosted wandb

Step 1: Obtain the wandb Self-Hosting Package

Wandb provides a Docker image that simplifies deployment. You need to pull the wandb server Docker image and set up the containers.

Step 2: Configure the Environment

Configure environment variables such as database URL, Redis URL, and internal domain name.

Step 3: Deploy Using Docker Compose

Create a docker-compose.yml file that specifies the services (wandb, PostgreSQL, Redis). Example configuration:

<syntaxhighlight lang="yaml"> version: '3' services:

 wandb:

image: wandb/server

ports:

  - "8080:8080"

environment:

  - WANDB''API''KEY=your''api''key

  - WANDB''DB''URL=postgresql://user:password@db''host/db''name

  - WANDB''REDIS''URL=redis://redis:6379

depends''on:

  - db

  - redis

 db:

image: postgres:13

environment:

  - POSTGRES''USER=user

  - POSTGRES''PASSWORD=password

  - POSTGRES''DB=wandb

 redis:

image: redis:alpine

</syntaxhighlight>

Step 4: Launch the Services

Run docker-compose up -d to start all services.

Step 5: Access the Platform

Open the browser and navigate to the domain or IP address where the server is hosted. You should see the wandb interface.

Additional Configuration and Security

• Enable HTTPS for secure communication.
• Set up user authentication and access controls.
• Regularly update containers and dependencies.

Limitations and Considerations

• Self-hosting requires maintenance, updates, and security management.
• It may involve significant initial setup and ongoing administration.
• Performance depends on hardware and network setup.

Conclusion

Self-hosting wandb is a viable option for organizations with specific data privacy, customization, or control needs. It requires proper infrastructure, ongoing maintenance, and technical expertise. By following the official documentation and best practices, you can successfully deploy and operate your own wandb server.

For more detailed instructions and updates, refer to the official wandb self-hosting documentation or contact support.

---

This page will be periodically updated to reflect new features and best practices.