The Problem With Long System Prompts

If you've built anything with an LLM API, you've probably added a system prompt. At first it's a sentence or two. Then you add instructions. Then examples. Then context about your product. Before long, your system prompt is a few hundred lines and every …