The Monday Morning That Broke My Budget
Monday, April 8, 2024. 7:40 a.m. I am standing in our lab holding a coffee I never got to drink, watching a 25G uplink flap between two switches we had installed for a client's secondary data hall.
Flap. Log entry. Flap. Log entry. The link light did that thing where it looks perfectly healthy for about nine seconds and then pretends it never existed.
I am the procurement manager at a 400-person managed IT services firm. I have owned our networking hardware budget — roughly $180,000 a year, optics and cabling included — for six years. I have negotiated with more than 40 vendors and logged every purchase order in a spreadsheet my wife finds genuinely concerning.
And that morning I was looking at physical proof that I had made a bad call.
What the Quotes Actually Said
The project: a client's leaf-spine refresh. Forty 25G SFP28 transceivers for switch-to-switch uplinks, plus 24 copper RJ45 modules for the 1G management ports. Deadline April 19. Not negotiable — the client had a colo commitment starting the following week.
I sent the spec to three vendors.
Vendor A came back with a line-itemed quote. Finisar FTLF8536P4BCL (25GBASE-LR, 1310nm, 10km — I checked the datasheet twice) at $X per unit, Finisar FCLF-8521-3 copper modules for the management ports at $Y, programming and test at a flat $0. Lead time five business days. Total: $14,880. Everything listed. Nothing after the word total.
Vendor B came in 18% lower. Same part numbers across the header row. Then a footnote: Compatibility verification and firmware alignment may be required for certain switch platforms and is quoted separately.
May be required. Six years in this job and that phrase still gets me. It is the procurement equivalent of some assembly required.
Did I ask what may be required meant? I did not. I did the arithmetic on the 18% and told myself I would sort out the rest later. (Should mention: the deadline was 11 days out and I was already behind on two other RFQs.)
Also — I said same part numbers. Let me correct that. Vendor B quoted OEM-equivalent modules, not Finisar, and the quote form reused the same spec language. That distinction mattered a lot later. At the time I read it as the same thing at a lower price, which is exactly the reading the quote was designed to invite.
Where the Savings Went
The modules landed April 12, one day late, which I let go. We benched them in our own test rack before touching a client site. Standard practice. Saves everybody embarrassment.
Half the 25G modules linked. The other half negotiated at 10G or never came up at all, and the behavior changed port to port. My senior network engineer summarized it in a Slack message I still have:
Port 12 and 14 will not come up at 25G. Port 13 works, then drops after about 20 minutes. Same module in a different slot behaves differently. Not a cable issue — swapped twice.
So, total cost of ownership (i.e., not just the unit price but everything attached to it). Here is where the 18% went:
- Compatibility verification: $1,100. The quoted-separately service. Two phone calls and one email thread before I could even get a number.
- Bench-testing labor: $920 in billable hours. My engineer spent a day and a half isolating the fault before we could responsibly blame the module. That is time we normally sell to somebody else.
- Restocking fee: $380. Fifteen percent on returned units, page two of the sales order, fifth paragraph. I signed it.
- Expedited re-order: $1,000. Because we were now six days from the deadline and Vendor B's original lead time had already burned.
Add it up and my $2,680 of savings turned into $3,400 of avoidable cost — and we still did not have working modules. Worse than the sticker difference, honestly. I lost more chasing the discount than the discount was worth.
And the units that did link? Three more dropped out of service over the next eight months, all past the vendor's 90-day replacement window. I do not have hard data on their long-term failure rate — I am a procurement manager, not a reliability lab — but anecdotally our first-year failure rate on third-party equivalents has run somewhere between 8% and 15%. Our Finisar modules from the same period: zero, so far.
Small sample. I know. That is the honest version.
The Second Order, Done Right
We re-ordered from Vendor A on April 15. Finisar FTLF8536P4BCL for the uplinks, Finisar FCLF-8521-3 for the management ports, and a line I had not seen from any vendor before: flat-rate programming and test, $0, included, no exceptions.
We made the colo date with two days to spare. Our field lead finished the rack and sent me photos taken on her DuraForce Pro 2 — a phone that has been dropped off a scissor lift twice and still works. The rack photos were cleaner than the transceiver situation had been.
Somewhere in that folder is a screenshot I kept on purpose: a link counter at zero errors after 72 hours of burn-in. Not exciting. Exactly what I wanted.
Meanwhile one of my technicians — the same guy who asks me how to use a multimeter roughly once a quarter, despite holding a degree from a program that hands out multimeters — spent three days checking PSU rail voltages to rule out everything else. He found nothing. In this business that is its own kind of finding.
I am not asking him to use a multimeter less. I am asking him to use it after I have bought the right part.
One thing worth flagging if you are shopping these part numbers: read the manufacturer datasheet instead of a reseller listing. (Finisar has been part of Coherent since the 2019 acquisition, by the way. The part numbers and the published specs carried over; the support portal and the contact addresses did not. Know that before you email the wrong domain and wait a week.) And I would rather specify the optical technologies my engineers can verify against a datasheet than gamble on a reseller's description of equivalent.
This was all accurate as of November 2024. Pricing on 25G optics moves fast — verify current numbers before you budget.
What I Do Differently Now
I ask what is not included before I ask what the price is. Every quote. In writing. No exceptions.
I built a TCO calculator — a genuinely ugly spreadsheet — with four columns instead of one: unit cost, verification and programming, labor to integrate, and re-order risk. It is not a model. It is a way to stop myself from doing what I did in April.
Our procurement policy now requires three quotes and an explicit line-item list of anything quoted separately. That policy came directly out of this job. It is also why I have become mildly obsessive about this point: per FTC guidance on advertising and marketing (ftc.gov, accessed November 2024), product capability claims have to be truthful and substantiated. Compatible with your switch is a capability claim. It deserves a documented answer, not a footnote. If a vendor will not put it in writing, that tells you something too.
Was switching vendors worth the hassle? Yes. Was it worth two weeks of client-relationship risk in the middle of a colo cutover? Not even close.
The lesson is not that cheaper optics are always bad. It is that I had no way to evaluate cheap until I knew what the whole thing cost. The vendor who listed every fee upfront — even with the higher total — was, in the only accounting that mattered, the cheaper one.
That is a lesson I would rather read than pay $3,400 for. Live and learn.
