👉 Check out my new book "OpenAI GPT For Python Developers - The art and science of developing intelligent apps with OpenAI GPT-3, DALL·E 2, CLIP, and Whisper. A comprehensive and example-rich guide suitable for learners of all levels"
ChatGPT may cost up to $3M per month
OpenAI CEO Sam Altman announced on Twitter that ChatGPT, the AI assistant based on the GPT-3.5 architecture for natural language interaction with humans, had reached one million users just a few days after its launch.
The chatbot has been well-received for its high-quality, longer output, and improved ability to follow instructions, surpassing the performance of previous models.
The AI assistant may cost $100K per day, or $3M per month, according to a tweet by Tom Goldstein. That's the reasoning behind it:
- A 3-billion parameter model can generate a token in about 6ms on an A100 GPU (using half precision+tensorRT+activation caching). If we scale that up to the size of ChatGPT, it should take 350ms secs for an A100 GPU to print out a single word.
- Of course, you could never fit ChatGPT on a single GPU. You would need 5 80Gb A100 GPUs just to load the model and text. ChatGPT cranks out about 15-20 words per second. If it uses A100s, that could be done on an 8-GPU server (a likely choice on Azure cloud).
- So what would this cost to host? On Azure cloud, each A100 card costs about $3 an hour. That's $0.0003 per word generated. But it generates a lot of words! The model usually responds to my queries with ~30 words, which adds up to about 1 cent per query.
- ChatGPT acquired 1M users within its first 5 days of operation. If an average user has made 10 queries per day, I think it’s reasonable to estimate that ChatGPT serves ~10M queries per day.
