Embedded Files
Measuring Quality So It Can Be Improved
Measuring Quality So It Can Be Improved
Quality metrics are only useful when they accurately represent the work being performed and help people decide what to do next.
At LibreDigital, I inherited a quality process that calculated errors against the total number of pages in a publication. That measurement didn't account for whether those pages were actually part of the converted digital product, and the resulting data was used primarily to identify individual operator errors.
I redesigned the quality framework around a different purpose. Instead of using the data primarily to identify individual mistakes, I wanted it to show us where performance was changing and help us understand why.
That required changing both how quality was measured and what happened after a problem appeared. The resulting framework made performance more comparable, established shared definitions for errors, and created enough lead time to investigate recurring problems before contractual quality was at risk.
| The goal wasn't simply to count errors. It was to create information we could act on.
Measuring the Work That Actually Happened
Measuring the Work That Actually Happened
The existing calculation treated every page in a publication as an opportunity for error, even when some pages, such as advertisements, were intentionally excluded from the digital product.
I changed the measurement to calculate errors against the number of content pages actually being converted.
That distinction made comparisons more meaningful. Ten errors across ten converted pages represented substantially different performance from ten errors across one hundred converted pages, even though the raw error count was identical.
Performance over multiple titles was calculated from the average of the individual title results, allowing weekly reporting to reflect performance across a body of work rather than the size of any one publication.
______________
| Design principle: A useful metric has to reflect the work being measured.
Creating a Shared Definition of Quality
Creating a Shared Definition of Quality
A measurement system only works if the people collecting the data are measuring the same things in the same way.
Turning Error Definitions Into Structured Data
Turning Error Definitions Into Structured Data
I created a standardized taxonomy of error types, categories, severity levels, and definitions for reviewers evaluating converted publications.
Severity reflected the impact and context of the error. Structural problems such as missing content or broken continuation links were high severity. For proofreading errors, severity also depended on context. A typo in prominent content, such as a headline or byline, carried more weight than the same kind of error within ordinary article text.
The taxonomy served two purposes. It gave operators and reviewers a shared reference for what each error meant, and the same controlled vocabulary powered data validation in the review spreadsheet so reviewers selected standardized values when reporting errors.
A standardized taxonomy gave reviewers shared definitions for identifying errors while providing the controlled values used to collect consistent quality data.
______________
| Design principle: Quality can only be measured consistently when people share a definition of what an error is.
Creating Time to Correct Problems
Creating Time to Correct Problems
I didn't want the contractual quality requirement to be the point at which we discovered that performance had deteriorated.
I introduced an internal requirement that teams maintain an average error rate of 10% or lower. This standard was intentionally more stringent than the quality requirements in our retail-channel agreements.
The difference created a proactive quality buffer. Weekly reporting could surface declining team or vendor performance while there was still time to investigate the cause and take corrective action before contractual quality was at risk.
The internal 10% quality threshold created an early-warning buffer, allowing recurring problems to be investigated and corrected before contractual quality requirements were at risk.
______________
| Design principle: Set internal controls early enough to create time for correction, not simply to identify failure after it occurs.
Diagnosing What the Data Revealed
Diagnosing What the Data Revealed
Weekly reports went to internal Operations and external vendor managers, who shared relevant results with team leads responsible for smaller teams and individual operators.
But a recurring error didn't automatically mean someone needed retraining.
I used patterns in the quality data as a starting point for investigation. The response depended on what was actually causing the error rather than on the error alone.
Recurring error patterns were investigated to distinguish system, knowledge, and performance problems, allowing the response to address the underlying cause and subsequent reporting to
confirm whether the intervention worked.
______________
When an “Operator Error” Was a Product Problem
When an “Operator Error” Was a Product Problem
Weekly quality reporting revealed a recurring pattern of errors from one external vendor. Rather than treating the pattern solely as an operator-performance problem, I investigated what was happening.
The vendor had encountered a software defect and developed a workaround to compensate for it without reporting the underlying problem.
I gathered details about the issue, documented the expected and actual system behavior, and brought the problem to Product as a fix request that was routed to Engineering.
The quality data had surfaced the symptom. Investigating the pattern revealed the actual cause.
| The appropriate intervention depended on the cause of the error, not simply the fact that an error had occurred.
Quality as a Feedback System
Quality as a Feedback System
The framework changed quality reporting from a count of individual mistakes into a system for understanding performance.
Once the data became more consistent and the measurements more meaningful, patterns that had previously been difficult to see began to emerge. The internal quality buffer gave us time to investigate those patterns before they threatened contractual performance.
What happened next depended on what we found. A quality problem might lead back to the software, the documentation supporting the work, or the way someone was performing the task. The intervention could then address the underlying cause, and subsequent reporting showed whether it worked.
| Quality measurement shouldn't end with a score. It should tell you where to look next.
_____________________________________________________________________________________
Explore Another Special Project
Explore Another Special Project
Google Sites
Report abuse