Utility Ratings for Foreign Language Results

You must assign a utility rating for all result blocks in your task, even if the result blocks have a foreign language landing page.
In most cases, pages you flag as F should be rated OT, because they cannot be understood by most users in your locale and are therefore useless. (Recall that if users in your locale can read the language, then you shouldn't be using the F flag.) Occasionally, you will encounter helpful F pages.
If the query is clearly "asking" for a foreign language result, then the utility rating of the foreign language page should

not be OT. For example, please assign the V rating and F flag for baidu.com if the query is [baidu.com], English (US).
Videos are often an example where foreign language pages are helpful and desired. Think about user intent and what pages are good for users. If the query "asks" for a foreign language song, band, film, sporting event, etc., then a video of the song, band, film, sporting event, etc. is helpful since it can probably be understood or enjoyed even though it is in a foreign language. For these types of queries, foreign language results are often expected.

If the video is someone talking *about* the song, band, film, or event, the result probably cannot be understood or enjoyed and should be assigned an OT rating and the F flag.

Query and User

Intent

LP of the Result Block, Link to Video, Result Block, Utility Rating

Explanation


[gangnam style], English (US)

Watch a video of this song.

Link to LP Screenshot
Link to view the video
:

This is a very popular video in the US, even though the song is in Korean. This result is extremely helpful. The language of the LP is mostly English (even though the video is in Korean), so the F flag is not needed.


[video of celine dion singing s'il suffisait d'aimer], English (US)

Watch a video of Celine Dion singing this song.

Link to LP Screenshot

Link to view the video

This video is just what English (US) users are looking for, even though the video is not in English. The language of the LP is mostly English (even though the video is in French), so the F flag is not needed.


[alex c], English (US)

Watch a video by this German artist or find information about her.

Link to LP Screenshot
Link to view the video

The query is for the German artist, Alex C. The landing page has a video sung by her in German. It would be helpful for many English (US) users who type the query, even though it is not in English. The language of the landing page isn't clear.

Use your judgment when deciding whether to assign the F flag.


[Kasal, Kasali, Kasalo], English (US)

Watch a trailer of this Filipino film or find information about it.


Link to LP Screenshot

Link to view the video

The query is for "Kasal, Kasali, Kasalo," a Filipino film. The LP is a clip from the movie in Filipino (Tagalog), the language spoken in the Philippines. It would be helpful for many English (US) users who type the query, even though it is not in English. Use your judgment when deciding whether to assign the F flag.


[the beatles video], English (US)

Watch a video of the Beatles.

Link to LP Screenshot

Link to view the video

The video on the LP documents a visit by the Beatles to Tokyo. Most of the spoken language on the video is in Japanese. This video would be helpful for no or very few English (US) users. Use your judgment when deciding whether to assign the F flag.

English Language Results

The following rating guidance is for raters in non-English locales. You may stop reading this section if the language spoken in your locale is English!
Your utility ratings need to reflect how helpful the result is for users in your locale.
We know that you can read English (you are reading this document!), but you should only give high utility ratings to English results if users in your locale would expect or desire them. Unless requested by the query, English results should be considered useless if most users in the locale can't read them.

Here are some examples using Korean (KR) as the task language. In Korea, most users don't read English. Unless the query "asks for" English results, we will consider them unhelpful or even useless (OT):

Query and User

LP of the Result Block, Result Block, Utility Rating

Explanation


[Barack Obama], Korean (KR)

Find information about Barack Obama.

Landing page link: LP

This Korean language Wikipedia page on Barack Obama is very helpful for Korean (KR) users.

Landing page link:
LP

This English Wikipedia article about Barack Obama has similar content to the Korean Wikipedia page. However, very few users would be able to read this page.

(Note: this query does not "ask for" English language results.)


[Titanic 1997], Korean (KR)

Purchase a DVD or find information about the movie "Titanic," released in 1997.

Landing page link:
LP

Although the query was typed in English, most Korean users would expect to see Korean language reviews or Korean stores for purchasing the movie.

This Korean result on a Korean website is very helpful.

Landing page link: LP

This English language landing page has helpful content. However, very few users would be able to read this page.

(Note: this query does not "ask for" English language results.)

In some locales, English is one of the official languages or a commonly spoken language. In these locales, English websites are easy to use and could be helpful (depending on the query).

For example, the Singapore government recognizes four official languages: English, Malay, Chinese, and Tamil, but English is the first and most dominant language in Singapore.

Query and User Intent

LP of the Result Block, Result Block, Utility Rating

Explanation

[Barack Obama], Chinese_Simplified (SG)

Find information about Barack Obama.

Landing page link: LP


This Wikipedia page in English about Obama would be very helpful to users in Singapore.

Landing page link: LP

This Wikipedia page in Chinese about Obama would also be very helpful to users in Singapore.



« Previous Next »