Last spring I spoke with a procurement lead at a mid-sized data center company. She said her job was like trying to buy concert tickets for a show that had already sold out three months ago. She had spreadsheets running across four monitors, vendor relationships that she was burning through like firewood, and a CFO asking daily why the AI infrastructure expansion was stalled. The hardware had simply evaporated from the market.
I have a typical story to recount from last year’s GPU shortage. A few months ago I spoke with a procurement lead from a mid-sized data center company. During the shortage period her work basically consisted of trying to buy concert tickets for a show that already sold out three months ago.
The surge problem isn’t what most people think it is.
It’s common for people to say that, with more foresight into future demand, the correct strategy would have been to place orders for required equipment earlier in time. However, the reality is that, in practice, most teams are organized to operate as much as possible in a reactive manner to best operate data infrastructure to greatest extent possible. The natural function of the team is to react to failures as they occur, to determine the cause of the failure, and then work as quickly as possible to restore service. In this manner, planning for future requirements to be able to function as a commodity trader and predict when needed equipment will disappear from distribution channels is an important part of the function of the team. But it is not the primary function of the team. And in the end, it is a function that is best served by a team that is actively operating data infrastructure in production.
I have been noticing more and more unusual patterns in demand lately. The typical historical data does not accurately model the demand for building out the AI infrastructure. A couple of weeks ago, I heard that a single hyperscaler placed a very large order for a certain part. Within weeks, that part went on 40-week lead times across all of the distribution channels. Typically, the planning horizon for datacenter infrastructure planning is on the order of 6 months or so for large server deployments. Therefore, for the components that make up the AI infrastructure, when one goes on very long lead times, the team planning the build out for that new infrastructure will be planning in the dark.
Build the buffer before you need it
Treating the cost of holding inventory as a negative cost prior to the crisis is a huge problem in the industry today. It’s like not having insurance because it costs money until you need it. When you are trying to buy out the market for a part that has become critical to your operation and you find out that if you had only held a small amount of safety stock it would have only cost a fraction of what it is costing you now is a huge disappointment. Carrying inventory has real costs. Nobody is denying this. But the math during a crunch period almost always favors having held some safety stock. That becomes obvious the moment you are scrambling to do business in the marketplace and the options in front of you look like this:
Even if you are paying to hold inventory the costs are typically dwarfed by the costs of not having a safety stock of critical components during a supply surge. In the worst case scenario you will buy the components on the open market at inflated prices. In the best case scenario you will wait a couple of weeks for additional inventory to arrive. Either way the cost of not having held a safety stock of critical components far outweighs the cost of holding a small buffer of safety stock. Note: holding a small buffer of safety stock for critical components does not equal hoarding. Hoarding is buying and holding a large amount of inventory in anticipation of future supply problems. A small, deliberate buffer of safety stock for critical components is insurance. Treat it as such.
- Premium pricing from secondary market vendors eats your budget fast
- Project timelines slip, which creates downstream costs that rarely show up in the procurement post-mortem
- Engineering teams sit idle waiting on hardware, which is an expensive kind of waiting
- Vendor relationships get strained when you’re suddenly demanding expedited orders you didn’t plan for
A small buffer of stock for critical components is not hoarding, it is insurance. It costs something to hold that stock, to be sure, but when the crunch time comes and you are forced to negotiate in the poor position of needing to buy in a short time, the cost of holding the stock is less than what you would have to pay to buy in a short time.
Vendor Diversification
As mentioned above, having 2-3 suppliers for critical components is not enough. Typically the other suppliers are just vague impressions of people that you know who sell the same parts. It is impossible to get purchase orders from these vague impressions in times of stress.
The way that most teams have managed to deal with component shortages is to have a very wide supplier network for all of the different enterprise computing components that make up their servers and data centers. This is to have dealt with the shortages during the 2021–2023 component shortage (such as the global semiconductor shortage) better than just about everyone else, were not the teams with the deepest pockets. Instead, they were teams that had done a very good job of building up a wide supplier network for all of the parts and pieces that made up their servers and data centers, and had a very clear eyed view of just how single threaded their supply chain actually was. A single threaded supply chain is like a bridge held up by a single beam: it’s perfectly fine until it’s not. Teams that navigated the 2021–2023 component shortages best weren’t the ones with the deepest pockets. They were the ones with the widest supplier networks and the clearest-eyed view of their own vulnerabilities. Which, if you think about it, is a lesson that applies to almost every other kind of organizational resilience too.
Lessons from the component shortages of the last years can be applied to many areas of organizational resilience, not just datacenter provisioning. The team that got through the shortages best was not the one with the deepest pockets, but the team that had a wide supplier network and a good understanding of the weaknesses of their own processes.
How AI infrastructure changes the planning equation
Note the list of demand sensitive components required for AI workloads. The same components that impact other high performance computing (HPC) applications such as scientific simulations and other forms of computing intensive processing. However, whereas the impact to typical server workloads can often be mitigated by purchasing an alternate part or model, AI workloads require very specific parts. The consequence is that when a critical component for an AI deployment is out of stock, the entire deployment comes to a halt.
There is an increase in AI workloads, which also need to be planned with longer lead times for procurement, such that while typical server refreshes have a 12 week or so procurement cycle for inventory that was planned based on historical models, AI cluster builds need to be planned for with 6 to 9 months of lead time for component procurement, in order to account for the component availability, customs, integration testing and other issues that can arise during the procurement process for commodity hardware.
| Infrastructure type | Typical procurement lead time | Risk level during surges |
| Standard server refresh | 8 to 16 weeks | Moderate |
| GPU-based AI cluster | 24 to 40 weeks | Very high |
| Networking upgrades | 10 to 20 weeks | High |
| Storage expansion | 6 to 12 weeks | Low to moderate |
Practical steps that actually move the needle
(Disclaimer: I used to think that “tracking industry signals” was a euphemism for “wasting time reading news”. After seeing teams get caught off guard time and time again by huge demand surges because they ignored fab capacity announcements or hyperscaler earnings announcements, I have come around to viewing this as one of the key things that an infrastructure procurement team does, and something that is worth your time to get right).
- Map your critical components quarterly, not annually. The market moves too fast for annual reviews to stay meaningful by the time anyone reads them.
- Set lead time thresholds that trigger early procurement automatically, without requiring a committee decision each time. Committees are slow. Supply windows are not.
- Build supplier relationships when you don’t need them. Call the account rep during a calm stretch. It changes the entire tenor of that relationship when you eventually call in a crisis.
- Track industry signals — fab capacity announcements, hyperscaler earnings calls, geopolitical developments that touch semiconductor supply chains. Not paranoia. Competitive awareness.
While I have no answers to how to deal with the current imbalances in supply and demand of various components on the market that trigger surges in demand, I do have some thoughts on how one should treat procurement in general, to deal with such crises in the future. In order to deal with imbalances of supply and demand on the market for various components of servers, one needs to treat procurement (or purchasing) as a strategic issue, rather than as an administrative one. The quiet times (low demand for servers) are the times to build up one’s ability to deal with the frenzied times (high demand for servers, due to imbalances of supply and demand on the market).
