Introduction to Utility Rating 5
Important Rating Definitions and Ideas 5
The Purpose of Search Quality Rating 6
Raters Must Represent the User 6
Task Language and Task Location (Locale) 8
Queries with Multiple Meanings 8
Query Meanings Can Change Over Time 10
Classification of User Intent: Action, Information, and Navigation: Do-Know-Go 11
Important Definitions and Ideas 12
Identifying the Purpose of the Page 13
Understanding Web Page Content 13
Utility Rating Scale for Result Blocks 15
Rating Porn (P), Foreign Language (F), and Didn't Load (D) Results 37
Rating Results with Content that Doesn't Load 43
Location and Utility Rating 45
Queries that "Ask" for Nearby Results or Nearby Information 48
User Location and Understanding the Query 51
Queries with an Explicit Location 53
Vital Ratings for Queries with a User Location 54
Rating Queries with User Location and Explicit Location 55
Rating Local Intent Queries 55
Rating Examples with User Location, Explicit Location, and Local Intent 55
Page Quality Rating Guidelines 59
Overview of Page Quality Evaluation 59
Introduction to Page Quality 60
Important Information about the Page Quality Guidelines 60
Your Money or Your Life (YMYL) 61
Landing Page Considerations 62
Identifying the Purpose of the Page 62
Identifying the Main Content, Supplementary Content, and Advertisements 63
Rating the Quality of the Main Content 64
Rating the Quantity of Helpful Main Content 67
Rating the Helpfulness of the Supplementary Content 68
Rating the Layout of the Page/Use of Space on the Page 69
Answering Homepage and Website Questions 72
Finding the Homepage of the Website 72
Is the Purpose of the Page Consistent with the Website? 74
Who is Responsible for the Content of the Website and the Content of the Page? 75
Does the Website Have an Appropriate Amount of Contact Information? 76
What Kind of Reputation Does the Website Have? 77
Is the Homepage of the Website Updated/Maintained? 81
Additional Page Quality Rating Guidance 81
Assigning a Page Quality Rating to "Your Money or Your Life" (YMYL) Pages 82
Assigning a Page Quality Rating to Encyclopedia Pages 82
Assigning a Page Quality Rating to Pages with no Main Content/Error Messages 83
Balancing Page Level and Website Level Questions to Assign an Overall Page Quality Rating ..
How to Check for Copied Content 84
Assigning an Overall Page Quality Rating 86
Webspam: a Special Type of Lowest Page Quality 91
Looking for Technical Signals 91
Overview of Block Utility (BU) Rating 103
The Relationship between PQ and Utility 103
Queries with Multiple Interpretations and Intents 105
Specificity of Queries and Landing Pages 107
Misspelled and Mistyped Queries and Results 117
Misspelled and Mistyped Queries 117
Spelling Suggestion Result Blocks 120
Non-Vital Results for URL Queries 123
Product Queries: Action (Do) vs. Information (Know) Intent 124
Using the Evaluation Platform 128
Accessing the Evaluation Platform (EP) 128
Evaluation Platform Screenshot 128
Block Utility Task Page Screenshot 129
Notes about Using the Block Utility Rating Interface 131
Using the "Report a Problem / Release this Task" Button 131
Reporting Results with Duplicate Landing Pages 132
Using the Page Quality Rating Scale in Block Utility Rating Tasks 134
General Guidelines Overview
Welcome to the Search Quality Rating Program!
As a Search Quality Evaluator, you will work on many different types of rating projects. The General Guidelines primarily cover block utility (BU) rating; however, the concepts are also important for many other types of rating tasks.
Please note:
• For brevity, we refer to "Search Quality Evaluators" as "raters" in these guidelines.
• The General Guidelines contain many rating examples. Because webpage content changes so frequently, we use screenshots created when the rating examples were written. Some screenshots may seem outdated when you read through the guidelines.
