The scan checked the recorded source snapshot before narrowing the evidence.
OpenHermes-2.5 contained 1,001,551 source rows. Some rows produced more than one user and assistant exchange, resulting in 1,008,268 training examples. No source rows were skipped.
The result is based on all 1,001,551 rows in the recorded source snapshot, not a sample.
Answers to the same prompt were grouped so disagreements could be inspected together.
The complete responses had to be Yes, No, True, or False after capitalization and punctuation were ignored. Each counted group contained at least one Yes or True answer and at least one No or False answer.
The broad comparison led to a smaller result that could be stated plainly.
Across the recorded dataset snapshot, 4,641 groups contained identical prompts with different answers. A contributing source named UnnaturalInstructions accounted for 145 of those groups. Of those, 68 contained a Yes or True answer in one row and a No or False answer in another for the exact same prompt.
One prompt appeared three times with incompatible answers.
These were three separate training examples from the same contributing source. Each used the identical prompt below.
You are given a description of an event, and you need to determine if it is credible or not by outputting True or False.
A rock was thrown into the air and disappeared.False, Yes, and True cannot all be correct answers to the same True-or-False prompt. Datascreen surfaced the disagreement without pretending to determine the correct answer automatically.
Five copies of one programming prompt included four answers to unrelated questions.
The shared prompt asked for code that sums a list of integers. One response answered that request. The other four responses belonged to different tasks, making the affected examples unsuitable without repair or removal.
Write a function that takes in a list of integers and outputs the sum of all the integers. List: [1, 2, 3].
Python code that adds the list and outputs 6.
['What color was the lamb's fleece?'] - 5
Conflicting examples can teach incompatible outputs for the same input.
Training data is supposed to provide a useful target for each prompt. When identical prompts carry opposite labels or answers from unrelated tasks, the training signal no longer expresses one coherent outcome. Datascreen makes those rows reviewable together instead of leaving the disagreement buried across a million-row file.
A verified dataset conflict with clear limits on the claim.
All 1,001,551 rows in the recorded source snapshot were checked. One named contributing source contained 68 groups where the exact same prompt received a Yes or True answer in one row and a No or False answer in another.
This result does not show that OpenHermes-2.5 is broadly low quality, that the conflicts were intentional, or that a downstream model was harmed.
OpenHermes-2.5 at commit b82037821055c377bed0d495e72e46de3bc72e84
SHA-256 · abe573d17eade4161aac321028027dd5ba614a6d9516d51bde9299d5353e1609