Recent advances in machine learning got me thinking about how far we are from replicating the human brain, so I decided to do some research on this.

Brain computing

The human brain has a very complex structure, but its computational substrate is quite easy to understand. For our purposes, we can simplify it to two main elements - neurons and synapses. Knowing some basics of computer science, you can think of neurons as accumulators and synapses as multipliers that allow the brain to perform multiply-accumulate operations.

There are different estimates of the number of neurons and synapses in the brain. In this post, we will use the estimates of 100 billion neurons and 100 trillion synapses, which seems to be the consensus1. Assuming that each synapse is both a weight and a FLOP, that is 100 trillion parameters. For reference, the latest GPT-3 model has 175 billion parameters2, so our brain is more than 500 times larger.

The synaptic firing rate of a single neuron ranges from 1 Hz to 200 Hz3 and depend on many factors, such as what the brain is currently doing. Multiplying, this yields a range of 100 billion to 20 trillion action potentials per second.

This is 20 PFLOPS of compute and 200 TB of float16 weights.

Costs

Latest NVIDIA H100 GPU claims 1 PFLOPS of compute4, but unless you are weight sharing, you’ll run into the RAM bandwidth limit of 3 TB/s4 long before that. With float16, it’s 1.5 TFLOP per GPU.

The brain is always learning, so it needs to be doing forward and backward passes. The backward-forward FLOP ratio in neural networks can vary, but it is typically 2:15, which means that the backward pass takes twice as much compute than the forward pass, so we need another 40 PFLOPS of compute.

We are up to 60 PFLOPS on 40,000 GPUs, needed only for reading and writing weights. Since deep learning is usually done in minibatches, the GPU comes with 80 GB of RAM each4, giving a total of 3.2 PB of RAM, an 16x overprovision.

With RAM bandwidth being the limiting factor, we get:

  • 1x the RAM bandwidth
  • 16x the storage (minibatch size)
  • 667x the compute (weight sharing ratio)

Nvidia will probably sell the GPUs for $36,000 each6, so if you ordered from them, it would be $1,44 billion.

Assuming they draw 700 W each4, that’s 16 MW of power. At $100 per MWh7, it’s $1600 an hour to run.

Total cost: $1,44 billion + $1600 an hour

References

  1. Basic Neural Units of the Brain: Neurons, Synapses and Action Potential - arxiv.org

  2. Model index for researchers - openai.com

  3. Neuron firing rates in humans - aiimpacts.org

  4. NVIDIA H100 Tensor Core GPU Datasheet - nvidia.com

  5. What’s the backward-forward FLOP ratio for Neural Networks? - lesswrong.com

  6. Nvidia Hopper H100 80GB Price Revealed - tomshardware.com

  7. Sectorial electricity price distributtions - iea.org