Choosing a factory system without letting the demonstration decide it

Replace the demonstration with a scripted scenario

Give each candidate the same package before they present: a handful of your real parts, one bill with the structure that causes you trouble, a routing including a subcontract operation, an order that gets split, and a change that arrives mid-production. Ask them to configure and run it, in front of you, using that data. What you learn is not whether the function exists but how much configuration it takes, how many screens an operator touches, and where the consultant hesitates. A candidate declining to work with your data has told you something worth knowing.

The requirements that genuinely differentiate

Every serious product handles the standard flow: order, plan, buy, make, ship, invoice. Differentiation lives in the exceptions your business runs on. Co-products and by-products. Catch weights. Lot splitting and merging. A rework loop with its own routing and cost. Unit-of-measure conversion between purchase, stock and consumption. Subcontract operations in the middle of a routing. Configured products. Transfers between sites. Backdated transactions after a shutdown. List yours before you meet anyone, weight them heavily, and treat the standard flow as a pass-or-fail gate rather than a scored category.

Why a scoring matrix produces a defensible wrong answer

A long weighted list of requirements gives everyone a similar total, because every candidate covers most ordinary functions. The handful of items that would actually determine success are diluted among hundreds that would not, and the weightings themselves are guesses nobody would defend individually. The matrix then supplies numerical cover for a decision reached another way. A more honest structure uses a short list of knockout criteria, deep hands-on testing of the few differentiating requirements, and a written argument for the recommendation that a sceptical reader could challenge.

Reference calls that repay the effort

Ask for a site of your size, in your process type, live for long enough to be past the honeymoon, and then ask to visit rather than to call. On site, talk to a planner, a supervisor and an operator, not the project sponsor who has to defend the decision. The productive questions are what took longer than expected, what they had to change about how they work, what still runs on a spreadsheet, and what they would specify differently. A vendor unable or unwilling to provide such a reference has answered a different question.

You are also choosing the people who will implement it

Two sites running identical software get very different outcomes depending on who configured it. Ask which specific consultants would be assigned, what else they are committed to, whether they have worked in your process type, and what happens if they are reassigned mid-project. Put named individuals and a replacement clause in the contract where you can. Ask how escalation works and who has authority to change scope. The product decision is usually reversible at a price; the implementation relationship shapes the result far more than the feature comparison suggested.

Frequently asked questions

Should we bring in an independent adviser to run the selection?

It helps where the team has never done one and where internal politics would otherwise decide it. Test independence carefully: many advisers hold implementation relationships with the products they shortlist, and a disclosed one is manageable while an undisclosed one is not. Keep the decision internal regardless. An adviser can structure the process, challenge assumptions and run the scenario testing, but the organisation that has to live with the outcome should own the recommendation.

How detailed should the requirements list be?

Detailed enough to describe your exceptions concretely, and no longer. Enormous line-by-line lists exhaust the team writing them and get answered with blanket compliance claims that mean nothing. Describing scenarios in operational language is more revealing than listing features, because a vendor can claim a feature and cannot easily fake running your scenario. Aim for a short document that an experienced production manager would recognise as their plant rather than a comprehensive catalogue nobody reads.

Can we shortlist on functionality alone?

Functionality gets you a shortlist and never a decision. Once candidates cover your critical exceptions, the remaining differences are implementation capability in your sector, the cost profile over the life of the system, how upgrades are handled, how open the integration surface is, and whether you can get your own data back out. Those factors decide whether the system is still serving you years later, and none of them appear in a feature comparison.