The expensive gap between a successful demonstration and an accepted system
A robot can complete a beautifully filmed demonstration and still fail as a production asset. The demo may use ideal parts, a familiar fixture, a single operator, a warmed-up camera, and an engineer who knows precisely when to intervene. Production introduces the opposite conditions: variable incoming material, changing shifts, maintenance, restart after an interruption, and upstream equipment that does not always behave. A procurement decision needs evidence that a system can handle the agreed operating envelope, not just one favorable cycle.
That distinction is supported by two different public bodies, each addressing a different question. NIST’s research on measuring manufacturing assembly robot performance emphasizes that useful measures depend on the task and application rather than a single universal robot characteristic [S1]. OSHA’s industrial robot guidance treats the robot application as a system—including tooling, controls, sensors and interfaces—and calls for documented risk assessment across design, integration, operation and maintenance [S2]. Neither source provides a one-page commercial acceptance contract. Buyers must translate their needs into measurable tests and responsibilities.
The practical solution is to make acceptance a sequence of evidence gates. The buyer and supplier agree on what is being tested, which conditions are representative, what constitutes a failure, who can witness the test, and how exceptions are resolved. The resulting record is more useful than a promotional claim of high throughput or precision.
First define the application boundary
The purchase order might say ’robotic palletizing cell,’ but a cell is not merely an arm. The relevant boundary may include gripper, conveyor, pallet supply, guarding, scanner, controller, software, compressed air, vision, electrical cabinet, safety interfaces and operator procedures. If an upstream conveyor or downstream packaging station is excluded, document the handoff assumptions. Otherwise a later dispute can be framed as somebody else’s equipment.
OSHA’s technical manual explicitly describes industrial robot systems in terms of manipulators, end effectors, control systems, power sources, sensors and communication interfaces [S2]. That is a useful prompt for commercial scoping, although OSHA’s definition should not be mistaken for a contractual allocation of liability.
Before a factory acceptance test (FAT), capture the version-controlled boundary: drawings, bills of material, controller and software versions, safety settings, selected end effector, sensor configuration, expected part families and environmental limits. Identify the test configuration as a baseline. If hardware or code changes before site acceptance, the buyer should know which results remain applicable and which tests need repeating.
A six-gate acceptance sequence
Gate 1 — Requirements and traceability. Every material requirement should map to an objective test or a documented review. A requirement such as ’high accuracy’ is not testable until the team defines the relevant part, coordinate frame, measurement method, tolerance, sampling plan and operating condition. NIST’s assembly performance work proposes task-oriented measures because the most important characteristics differ by application [S1]. Use that principle to build a requirements-to-evidence matrix. Record not only ’pass’ or ’fail,’ but the evidence file and its version.
Gate 2 — Representative production challenge. Test more than the supplier’s ideal demonstration part. Agree on a part mix covering approved variation: dimensions, surface finishes, weight bands, packaging presentation, lighting or reflectivity, and any known edge cases. The buyer must supply or approve a defensible sample. A test with ten hand-selected pieces should not be marketed as proof of performance on thousands of future SKUs. If the intended process changes after commissioning, record it as a new qualification question rather than silently stretching the original result.
Gate 3 — Measured performance. Choose metrics that connect to the production promise: accepted units per hour, first-pass yield, failed grasps, recovery time, false rejects, placement error, availability during the test window and operator interventions. Define denominators carefully. ’98% successful picks’ can mean 98 successful first attempts out of 100 presentations, or 98 eventual successes after retries; these are different business outcomes. Where force control matters, NIST’s published benchmarking research discusses the need for metrics and test methods to characterize robot force-control capability [S3]. It does not validate any particular vendor’s advertised number.
.jpg){kind=link}
Gate 4 — Safe operating modes and foreseeable interventions. A passing production cycle does not establish safe integration. OSHA’s robot technical manual discusses risk assessment, safeguarding, manual teaching, interlocks, presence-sensing devices, training and procedures [S2]. The safety team should determine which checks and documentation apply to the actual installation and relevant regulations and standards. Test normal operation, expected entry, stop and restart behaviors, and maintenance access under an approved plan. Do not improvise hazardous fault tests to satisfy a checklist. Any safety validation must be performed by qualified personnel using an authorized procedure.
Gate 5 — Fault recovery and operational handover. A cell that works only while its original programmer is present is not fully handed over. Agree on controlled recovery scenarios: a missed pick, blocked sensor, empty feeder, communications loss, incorrect part presentation, and a restart after a safe stop. For each, measure whether operators can identify the condition, follow an approved recovery sequence and resume without bypassing safeguards. Test backup and restore only with a qualified plan and an agreed rollback path. OSHA’s guidance specifically addresses written procedures and training for complex activities and abnormal conditions [S2].
Gate 6 — Documentation, training and residual defects. Acceptance should produce an evidence archive: final configuration, risk-assessment records, training attendance, maintenance instructions, approved operating procedures, measured test results, unresolved issues and a named owner for each issue. A conditional acceptance is not the same as an unconditional pass. Record the severity, containment, fix date, retest requirement and payment consequence of each exception in the commercial agreement. OSHA’s manual calls for documented risk assessments and appropriate worker training; commercial remedies remain a matter for the contracting parties [S2].
FAT and SAT should answer different questions
A factory acceptance test usually occurs before shipment or full deployment and asks whether the supplier-built system meets the agreed requirements under controlled conditions. A site acceptance test (SAT) asks whether the installed application performs as promised with the customer’s real utilities, upstream and downstream equipment, operators, network conditions and production material. These are common industry project labels, not an assertion that OSHA or NIST mandates this particular two-stage commercial process.
Do not copy the same test script into both stages without considering what changed. FAT may establish robot motion, basic vision behavior, safeguarding design reviews and representative cycle performance. SAT should confirm installed interfaces, production material, environmental conditions, safe handoffs, shift training and repeatability in the actual facility. Some tests must be repeated after installation; others may be reviewed by evidence where repetition would be unsafe or impractical. The plan should explain why.
The biggest source of confusion is the word ’capacity.’ A robot’s nominal cycle time is not necessarily the cell’s effective output. Queueing, presentation failures, tooling changes, operator replenishment and downstream blocking can all dominate throughput. The buyer should specify whether the contractual metric is peak instantaneous rate, sustained rate during a defined window, or net good units over a shift. Include planned exclusions and how downtime is categorized before the test begins.
Write pass criteria before seeing the result
A credible acceptance protocol defines the threshold before the supplier runs the trial. Otherwise teams may unconsciously adjust what ’good enough’ means after observing a borderline result. A useful record contains requirement ID, test method, instrument or data source, sample definition, acceptance threshold, actual observation, evidence link, witness, deviation and disposition. Avoid recording a bare green checkmark when the underlying measurement is available.
A practical buyer-created example might say: ’During an agreed two-hour production-material run, record total presentations, first-attempt successful transfers, accepted downstream units, interventions and safety stops. Pass criteria to be agreed by both parties before execution.’ The example deliberately contains no invented threshold. A 95% first-attempt pick rate could be unacceptable in a high-volume line and perfectly adequate in a supervised low-volume application; the economics and hazards differ.
Instrument uncertainty also matters. If a supplier claims a tight placement tolerance, ask how the result was measured, whether the method is repeatable, and whether the test reflects the application. NIST’s assembly performance publication is especially useful as a reminder to connect measurement to task [S1]. It is not a substitute for a calibrated, application-specific measurement plan.
Three procurement traps to avoid
Trap one: treating a component specification as a cell guarantee. The arm’s repeatability does not automatically establish placement quality once the gripper, part compliance, calibration, vision, fixturing and thermal conditions are included. Ask for integrated application evidence. A vendor’s component data sheet can support the design review, but it cannot by itself close the system acceptance record.
Trap two: confusing a safety feature with a validated safe application. A robot may support collaborative functions, but the actual task, tool, workpiece and operating environment still matter. OSHA describes collaborative safeguards and emphasizes verifying safety settings and risk reduction measures [S2]. Avoid claims that a collaborative robot needs no risk assessment or that a named feature guarantees safe deployment.
Trap three: letting open defects disappear into an email chain. The supplier may say the vision false-reject problem will be fixed after shipment. The buyer may assume the final invoice is held until it is resolved. If the acceptance protocol is silent, the commercial outcome becomes uncertain. Maintain a signed exception register and distinguish blocking defects from accepted minor punch-list items.
The minimum evidence packet a buyer should request
For a typical integrated industrial cell, a buyer can request the following as a negotiable procurement checklist, not a regulatory certification: a system boundary drawing; requirements-to-test matrix; test scripts and raw measurements; configuration and software revision list; risk-assessment and safeguarding documentation appropriate to the application; operator and maintenance procedures; training records; defect register; and signed acceptance disposition. If data contain trade secrets, negotiate controlled access rather than dropping the evidence requirement entirely.
The archive should be navigable by someone who was not present at commissioning. A video clip can supplement it, but video alone rarely supplies denominators, sample provenance, software version, or a reliable record of failed attempts. For high-value installations, consider a short independent witness report identifying the observed conditions and any limits on the witness’s scope. Independence should be stated, not implied.
What this means for supplier discovery on GlobleMatch
A directory that lists robot suppliers only by industry and product category makes initial discovery easier but does not solve buyer qualification. The next layer of value is structured evidence readiness. Suppliers could voluntarily declare which documentation they can provide under NDA, whether they support witnessed FAT and SAT, which application metrics they typically validate, and whether local commissioning and training are available. These should be self-reported fields, never converted into unverified compliance badges.
For integrators, distinguish design capability from demonstrated delivery. A buyer might need machine tending with high-mix fixtures, food-compatible end effectors, or vision-guided bin picking. The relevant question is not ’Have you built robots?’ but ’Can you show a comparable acceptance method and explain the limits of the evidence?’ Supplier profiles can link to public case studies, official technical documentation and the company’s own descriptions, with a clear separation between claims and independent verification.
For buyers, a lightweight RFQ worksheet could generate the same six-gate structure across competing suppliers. This improves comparability without pretending that every project has identical metrics. GlobleMatch should not score a supplier as ’safe’ or ’certified’ based solely on a marketing page. Its role can be to make evidence gaps visible before a purchase order, then let qualified engineering and safety professionals complete the application-specific review.
A better closing question
Instead of asking whether a robot can do the task, ask whether the proposed system can demonstrate the task repeatedly, under agreed conditions, with documented safeguards, recovery procedures and accountable owners for exceptions. A successful demo earns attention. A traceable acceptance record earns confidence.
Related directory categories
Browse the categories this analysis points to in the GlobleMatch directory: robotics system integrators, who typically build the cell and run acceptance with the buyer; industrial robot suppliers, whose machines sit at the centre of the test boundary; and machine vision and 3D sensing suppliers, whose measurement systems supply much of the acceptance evidence.
Sources
- [S1] NIST, Measuring and Representing the Performance of Manufacturing Assembly Robots, NISTIR 8090, December 10, 2015: nist.gov — supports task-specific measurement and performance representation; it does not set universal FAT/SAT criteria.
- [S2] OSHA, Technical Manual, Section IV, Chapter 4: Industrial Robot Systems and Industrial Robot System Safety, updated September 1, 2021: osha.gov — supports system boundary, documented risk assessment, safeguarding, procedures and training; guidance and applicable requirements must be distinguished.
- [S3] NIST, Benchmarking Robot Force Control Capabilities: Experimental Results, NISTIR 8097, January 7, 2016: nist.gov — supports the need for application-relevant force-control metrics and test methods; not a vendor certification.
- [S4] OSHA, Robotics — Hazard Recognition (topic page): osha.gov — used as a cross-check for the existence and relevance of the OSHA technical manual; no quantitative safety claims are drawn from it.