Miner deployment
Deploying a miner to popular cloud providers.
RunPod Guide
This step-by-step guide covers the entire BitMind Subnet miner onboarding process using RunPod. We will walk you through topics including compute acquisition, subnet installation, miner model training, registration, and deployment.
Deploy Your Pod
RunPod interface displaying available pods.Action: Navigate to the "Pods" page on RunPod and click the "Deploy" button to initiate a new pod setup.
Access Pod Settings
Detailed view of pod specifications including GPU, RAM, CPU, and disk storage.Action: Click the hamburger icon next to your pod to edit its settings Select the hamburger icon to edit your pod settings.
Configure Storage and Network
Editing storage space and TCP port settings in the pod configuration.Action: Ensure the volume disk has sufficient storage for downloads. In the network settings, request port 70000 for TCP to enable proper connectivity. Click "Save" to finish configuring your settings.
Connect to the Web Terminal
Options to connect to the pod's web terminal.Action: Click "Connect," then "Start Terminal," and finally "Connect to Web Terminal" to access the command line interface.
Clone and Navigate to Repository
RunPod Web Terminal displaying successful repository cloning.Action: In the terminal, enter the following command to clone the BitMind subnet repository and navigate into it:
git clone https://github.com/bitmind-ai/bitmind-subnet.git && cd bitmind-subnet
Set Up Conda Environment
Action: Install Conda using the quick command-line instructions provided by Miniconda:
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
Terminal showing the creation of a conda environment named "bitmind".Action: After installation, exit and reconnect to the web terminal to create a new Conda environment:
conda create -y -n bitmind python=3.10 ipython jupyter ipykernel
Manage Conda Environment
Action: Activate your newly created virtual environment with:
conda activate bitmind
To deactivate it, use:
conda deactivate
Configure Python Package Installer
Disabling pip's cache within the activated "bitmind" conda environment.Action: Before installing any Python packages, set the environment variable to disable pip's cache:
export PIP_NO_CACHE_DIR=1
Handle Missing 'sudo' Command
Action: Update your package list and install sudo with:
apt update
apt install sudo
Install System Dependencies
Action: Run the following commands to make the install script executable and execute it. This step may take several minutes.
chmod +x setup_miner_env.sh
./setup_miner_env.sh
Download Datasets and Train (optional)
Downloading open-source datasets for training a miner.Action: If you intend on training a miner, you can download our open source datasets by running:
python bitmind/download_data.py
Attention: Downloading may take a few minutes depending on your connection and our dataset access points.
Action: You can then run our training script for a given base miner model, with any desired modifications.
Train NPR model (CVPR 2024 architecture): cd base_miner/NPR/ python train_detector.py The weights with the lowest validation loss will be saved to base_miner/checkpoints/experiment_name/model_epoch_best.pth Train UCF (ICCV 2023 architecture, highest scoring on DeepfakeBench): cd base_miner/UCF/ python train_detector.py The weights with the lowest validation loss will be saved to logs/training/latest_time_stamp/ckpt_best.pth along with training configuration config.yaml. Tip: To avoid having your training job die if your ssh session is terminated, start your training job with pm2, which was installed in step 6.
pm2 start train_detector.py --name train --interpreter python3 --no-autorestart
You can view running pm2 jobs with pm2 list, or view logs with pm2 logs train.
Tip: If you happen to run into this warning (below) while using CUDA on TensorDock,
UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW try running the following commands in the terminal.
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia_uvm
sudo rmmod nvidia
For other potential solutions, see https://stackoverflow.com/a/72443701.
Set Miner Variables
After installing system dependencies, two files called validator.env and miner.env should have been created in the root repo directory (bitmind-subnet/). For mining, you will only need to consider the latter.
Action: Edit the miner.env fields to the correct values for your wallet (coldkey), hotkey, and symmetric port for the miner axon. If you want to use a different detector than the provided base miner defaults, also change the DETECTOR and DETECTOR_CONFIG fields accordingly. Tip: We recommend consulting the README in base_miner/ to learn about the extensibility and modular design of our base miner detectors.
DETECTOR=CAMO # Options: CAMO, UCF, NPR
DETECTOR_CONFIG=camo.yaml # Configurations: camo.yaml, ucf.yaml, npr.yaml
DEVICE=cpu # Options: cpu, cuda
# Subtensor Network Configuration:
NETUID=34 # Network User ID options: 34, 168
SUBTENSOR_NETWORK=finney # Networks: finney, test, local
SUBTENSOR_CHAIN_ENDPOINT=wss://entrypoint-finney.opentensor.ai:443
# Endpoints:
# - wss://entrypoint-finney.opentensor.ai:443
# - wss://test.finney.opentensor.ai:443/
WALLET_NAME=default
WALLET_HOTKEY=default
# Miner Settings:
MINER_AXON_PORT=8091
BLACKLIST_FORCE_VALIDATOR_PERMIT=True # Default setting to force validator permit for blacklisting" > miner.env
Attention: The detector type corresponds to the module name of the DeepfakeDetector subclass registered in base_miner/registry.py's DETECTOR_REGISTRY (e.g. UCF). The associated detector config file lives in base_miner/deepfake_detectors/configs/ (e.g. ucf.yaml). For UCF only: You will need to set the train_config field in the detector configuration file to point to the training configuration file. This allows the instantiation of UCFDetector to use the settings from training time to reconstruct the correct model architecture.
Attention: The NETUID and SUBTENSOR_NETWORK fields default to the Mainnet values, but if your wallet is registered on Testnet and you want to deploy a miner there, use the values in the code comments above. That is, Mainnet: NETUID=34, SUBTENSOR_NETWORK=finney Testnet: NETUID=168, SUBTENSOR_NETWORK=test
Bittensor wallet setup using btcli
Action: Refer to Bittensor's official documentation to set up your wallets (create or regenerate) using btcli. You must have a coldkey and an associated hotkey in order to register and receive a UID.
Miner Registration
To mine, you must have a registered hotkey on the correct subnet. Pictured: testnet registration.Action: Testnet (uses test tao, which can be requested in the Bittensor "Requests for Testnet TAO" Discord channel): btcli s register --netuid 168 --wallet.name [wallet_name] --wallet.hotkey [wallet.hotkey] --subtensor.network test Mainnet (costs tao): btcli s register --netuid 34 --wallet.name [wallet_name] --wallet.hotkey [wallet.hotkey] --subtensor.network finney
Deploy Miner
Action: Run the following command to begin mining with the base miner. pm2 start run_neuron.py -- --miner Attention: You will notice two pm2 processes running if you have auto update and self healing on (default behavior). You can check with pm2 list To stop your miner, run pm2 delete miner To turn off auto update and self healing, which respectively watch git for new updates and restarts your miner process every 6 hours, you can use the associated flags when running your initial miner process: pm2 start run_neuron.py -- --miner --no-auto-update --no-self-heal
TensorDock Guide
Select GPU or CPU instance
Select a type of machine (GPU/CPU only). If you're deploying a machine to train, you'll need a GPU. If you're deploying our base model, or one that is similarly lightweight, you only need a CPU.
Action: Click "New GPU Server" or "New CPU-only Server"
Configure Resources
Action: Select the number of cores, amount of RAM and storage, and (optionally) your GPU type.
Note: The BitMind team has tested training on RTX 30 and 40 series GPUs, so these are recommended, but not required.
Action: Based on your chosen resources, select a desired region.
Action: Select "Ubuntu + Everything" your OS template, for both CPU and GPU instance types. This comes with everything you need to train or deploy your miner.
Configure Network Settings
Action: Set your machine's name (just for display purposes in TensorDock), your password, and importantly, make sure to expose at least 1 symmetrical port mapping. Adding this mapping will allow your miner to receive requests from BitMind Subnet validators, by ensuring that the port number communicated by your miner axon matches the port number the external world sees.
Deploy your Machine and SSH in
Action: Click deploy and give TensorDock a few minutes to spin up your machine
Action: Once your machine is up and running, you can find your ssh command under My Servers > MyBitmindMiner > Connect. Copy and paste this command into your local terminal, and enter the password you set in step 3 when prompted.
Clone and Navigate to Repository
Action: In the terminal, enter the following command to clone the BitMind subnet repository and navigate into it:
git clone https://github.com/bitmind-ai/bitmind-subnet.git && cd bitmind-subnet
Set Up Conda Environment
If you selected the "Ubuntu + Everything" operating system template, your machine will be preconfigured with anaconda.
Action: Run the following command to create a conda environment named bitmind
conda create -y -n bitmind python=3.10 ipython jupyter ipykernel
Manage Conda Environment
Action: Activate your newly created virtual environment with:
conda activate bitmind
To deactivate it, use:
conda deactivate
Configure Python Package Installer
Action: Before installing any Python packages, set the environment variable to disable pip's cache:
export PIP_NO_CACHE_DIR=1
Install System Dependencies
Action: Run the following commands to make the install script executable and execute it. This step may take a few minutes.
chmod +x setup_miner_env.sh
./setup_miner_env.sh
Tip: If you run into warnings such as
Waiting for cache lock: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4295you can run
sudo rebootand reconnect to your VM to kill any incomplete processes causing the issue: https://askubuntu.com/a/1433045
Download datasets and train (optional)
Action: If you intend on training a miner, you can download our open source datasets by running:
python bitmind/download_data.py
Attention: Downloading may take a few minutes depending on your connection and our dataset access points.
Action: You can then run our training script for a given base miner model, with any desired modifications.
Train NPR model (CVPR 2024 architecture):
cd base_miner/NPR/
python train_detector.py
The weights with the lowest validation loss will be saved to base_miner/checkpoints/<experiment_name>/model_epoch_best.pth
Train UCF (ICCV 2023 architecture, highest scoring on DeepfakeBench):
cd base_miner/UCF/
python train_detector.py
The weights with the lowest validation loss will be saved to logs/training/<latest_time_stamp>/ckpt_best.pth along with training configuration config.yaml.
Tip: To avoid having your training job die if your ssh session is terminated, start your training job with pm2, which was installed in step 6.
pm2 start train_detector.py --name train --interpreter python3 --no-autorestart
Tip: If you happen to run into this warning while using CUDA on TensorDock:
UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HWtry running the following commands in the terminal:
sudo rmmod nvidia_drm sudo rmmod nvidia_modeset sudo rmmod nvidia_uvm sudo rmmod nvidiaFor other potential solutions, see https://stackoverflow.com/a/72443701.
Set Miner Variables
After installing system dependencies, two files called validator.env and miner.env should have been created in the root repo directory (bitmind-subnet/). For mining, you will only need to consider the latter.
Action: Edit the miner.env fields to the correct values for your wallet (coldkey), hotkey, and symmetric port for the miner axon. If you want to use a different detector than the provided base miner defaults, also change the DETECTOR and DETECTOR_CONFIG fields accordingly.
Tip: We recommend consulting the README in base_miner/ to learn about the extensibility and modular design of our base miner detectors.
DETECTOR=CAMO # Options: CAMO, UCF, NPR
DETECTOR_CONFIG=camo.yaml # Configurations: camo.yaml, ucf.yaml, npr.yaml
DEVICE=cpu # Options: cpu, cuda
# Subtensor Network Configuration:
NETUID=34 # Network User ID options: 34, 168
SUBTENSOR_NETWORK=finney # Networks: finney, test, local
SUBTENSOR_CHAIN_ENDPOINT=wss://entrypoint-finney.opentensor.ai:443
# Endpoints:
# - wss://entrypoint-finney.opentensor.ai:443
# - wss://test.finney.opentensor.ai:443/
# Wallet Configuration:
WALLET_NAME=default
WALLET_HOTKEY=default
# Miner Settings:
MINER_AXON_PORT=8091
BLACKLIST_FORCE_VALIDATOR_PERMIT=True # Default setting to force validator permit for blacklisting
Attention: The detector type corresponds to the module name of the DeepfakeDetector subclass registered in base_miner/registry.py's DETECTOR_REGISTRY (e.g. UCF). The associated detector config file lives in base_miner/deepfake_detectors/configs/ (e.g. ucf.yaml).
For UCF only: You will need to set the train_config field in the detector configuration file to point to the training configuration file. This allows the instantiation of UCFDetector to use the settings from training time to reconstruct the correct model architecture.
Attention: The NETUID and SUBTENSOR_NETWORK fields default to the Mainnet values, but if your wallet is registered on Testnet and you want to deploy a miner there, use the values in the code comments above. That is,
Mainnet: NETUID=34, SUBTENSOR_NETWORK=finney
Testnet: NETUID=168, SUBTENSOR_NETWORK=test
Bittensor wallet setup using btcli
Action: Refer to Bittensor's official documentation to set up your wallets (create or regenerate) using btcli. You must have a coldkey and an associated hotkey in order to register and receive a UID.
Miner Registration
Action:
Testnet (uses test tao, which can be requested in the Bittensor "Requests for Testnet TAO" Discord channel):
btcli s register --netuid 168 --wallet.name [wallet_name] --wallet.hotkey [wallet.hotkey] --subtensor.network test
Mainnet (costs tao):
btcli s register --netuid 34 --wallet.name [wallet_name] --wallet.hotkey [wallet.hotkey] --subtensor.network finney
Deploy Miner
Action: Run the following command to begin mining with the base miner.
pm2 start run_neuron.py -- --miner
Attention:
You will notice two pm2 processes running if you have auto update and self healing on (default behavior). You can check with
pm2 listTo stop your miner, run
pm2 delete minerTo turn off auto update and self healing, which respectively watch git for new updates and restarts your miner process every 6 hours, you can use the associated flags when running your initial miner process:
pm2 start run_neuron.py -- --miner --no-auto-update --no-self-heal