Machine Translation Accuracy Across Philippine Languages — Measured

Machine translation learns from parallel text — the same content in two languages, aligned. For English and French there are decades of it: parliamentary records, treaties, subtitles, technical manuals. For Ilocano and Tagalog there is very little, and for Kapampangan or Pangasinan there is less still.

The Philippines has over a hundred languages. Tagalog has a national broadcasting industry behind it. Ilocano, with roughly 8.7 million people identifying as Ilocano in the 2020 census, has a fraction of that written output — and almost none of it paired sentence-for-sentence with another language.

This is what “low-resource” means in practice. It is not that the language is simple or that nobody has tried. There is simply not enough aligned text for the usual methods to work well.

Co-official in the Philippines and the bridge language most machine translation systems pivot through — including this one. Ilocano↔Tagalog output is usually routed via English internally, which is a large part of why it drops nuance.

Because there is so little direct Ilocano–Tagalog data, systems translate Ilocano into English, then English into Tagalog. Two translations happen where you asked for one, and each is a chance to lose something.

The failure this produces is distinctive. English is full of ambiguous words, and when the pivot lands on the wrong sense, the output is confidently and completely wrong:

Kannawanis the clearest case. It means “right” as in the direction. Routed through the English word “right”, it returns as Gamit ang karapatan— “using the legal right”. Nothing in the Ilocano suggests entitlement. English introduced it.

The pivot also destroys distinctions English does not make. Both Ilocano and Tagalog separate an inclusive “we” from an exclusive one. English has one word for both, so a distinction present in the source and available in the target is lost in the middle.

A translator that fails visibly is manageable. The problem with Philippine-language machine translation is that it usually fails invisibly.

When the engine does not recognise a word, it returns it unchanged. Nothing marks it as untranslated. In our audit, ten phrases came back identical to the input despite having ordinary equivalents in the target language — including Haan (no), Danom (water) and Ikan (fish).

What makes this genuinely hard to spot is that unchanged output is sometimes correct. Seventeen further phrases in the same test were identical in both languages legitimately — lima, pito, itlog, kape. So “it came back the same” tells you something is worth checking, not that something is broken.

Rather than assert any of this, we tested it. All 411translatable entries in our Ilocano–Tagalog phrasebook went through this site's own translator and were compared against the human-checked Tagalog.

Machine output matched or came close for 59% of them. Reliability tracked how concrete the vocabulary was: colours scored 86% and time expressions 77%, while family terms, introductions and everyday courtesy all came in at 33–44%.

That pattern makes sense. Colour and time words are short, common, and have one-to-one equivalents. Kinship and politeness carry social meaning that differs between cultures and does not survive a round trip through English.

Full results, including every failure, are on the accuracy page, and the method is described on the methodology page.

The category breakdown is the most useful thing to come out of the measurement, because it is predictive. You can tell in advance whether a phrase is likely to survive the trip.

The gradient runs from physical description down to social relationship. A colour is a colour in any culture. A term of address encodes who outranks whom, how well you know them, and how formal the setting is — and none of that survives a round trip through a language that marks it differently.

The failures in that bottom band are rarely nonsense. They are usually the right idea at the wrong level of formality, which is harder to catch and often more damaging.

Notice that Agyamanak gained a po that was never in the source. The engine added deference on its own. In the other direction it stripped warmth from Tatang, returning the formal Ama. Neither is a mistranslation in the dictionary sense; both change the relationship between speaker and listener.

In Philippine languages that is not a stylistic nicety. Getting formality wrong with an elder is a social error, and it is precisely the error a translation tool is least able to warn you about.

Nothing here is a permanent property of the languages. It is a data problem, and data problems are solvable — slowly.

• Direct parallel corpora. Ilocano and Tagalog text aligned sentence by sentence, removing the English pivot. This is the single highest-value fix and the hardest to produce.
• Honest confidence signals.The engine knows when it has failed to recognise a word — it returns the input unchanged. Surfacing that as “not translated” instead of presenting it as output would eliminate the most damaging failure mode overnight.
• Dialect awareness. Regional Ilocano vocabulary is passed through silently rather than flagged as unrecognised.
• Curated phrase data. The least glamorous option and the one available today: for common situations, a human-checked phrasebook outperforms any engine, because a person decided what the phrase means in context.

This section used to say we had not measured Cebuano, Hiligaynon, Waray, Bikol, Kapampangan or Pangasinan and would not put a number on them. We have now measured all of them: 72 directions across eight Philippine languages plus English, 22,788 individual translations, scored the same way as the original Ilocano audit.

Mean agreement across every direction is 54.5%. The best is Bikol → Tagalog at 68.6%; the worst is Cebuano → Ilocano at 41.2%. No other site publishes a like-for-like comparison across these languages, which is why we are publishing ours with its limitations attached rather than waiting until it is beyond criticism.

The original Ilocano audit found that translating out of a low-resource language beat translating into it, by about nine points. We treated that as a general property of low-resource translation. Measured across eight languages, it is not one.

4 of the 9 languages are read better than they are written, 5 are the other way round, and for 6 of them the gap is under five points either way — small enough that it would be reading noise as a finding. The asymmetry is real, but it is a property of the individual language and the data behind it, not of low-resource translation in general.

One language breaks the pattern completely. Hiligaynon is read 59.3% and written 45.9% — 13.4 points apart, roughly double the next largest gap and the only figure here comparable to the original Ilocano finding. It is the worst target language on the list by a clear margin, and that holds across every one of the 8 sources we measured it from.

The top of the “translating into it” column deserves more suspicion than credit. Tagalog leads it at 62.6%, ahead of Tagalog — which is not what the training data would predict, since every language here has far more Tagalog text sitting beside it than anything else. The likelier explanation is our own dataset: Tagalog has one of the thinnest reference columns, so its pairs are measured on fewer and more concrete rows.

That is the caution to carry into the whole table. It measures the engine against our reference translations, and where those are thinner or less certain the score moves for reasons that have nothing to do with the engine. The next section sets out the limits properly.

A ranking invites conclusions it cannot support, so three limits are worth stating before the full table.

• The reference translations are not equally solid. The Ilocano figures are scored against cited, checked data. The other seven languages are scored against our compiled phrase set, which is still awaiting native-speaker review. A low score for a language may partly reflect our reference column rather than the engine — and that possibility is largest exactly where the score is lowest.
• The phrase sets are not identical. Each pair is measured on the rows where both languages have an entry, which ranges from 116 to 138. Kapampangan and Pangasinan are measured on slightly smaller sets because we left gaps rather than inventing forms.
• The grading is automatic. A correct translation worded differently from ours is marked wrong, so every figure here is a floor rather than an estimate. That penalty is not necessarily even across languages.

None of this makes the comparison useless. It makes it a first measurement rather than a settled one, and we would rather publish it that way than not at all.

All 72measured directions, best first. Each links to that pair's own page, where the full category breakdown and its reverse-direction comparison are published. The whole table, and the phrase set behind it, is downloadable as CSV and JSON under CC BY-SA 4.0.

A number is only useful if you know what it excludes. Ours has four limitations worth stating plainly, and two of them cut against us.

• The grading is automatic. A correct translation worded differently from ours is marked down. The true figure is probably somewhat better than 58.9%.
• The test set is short phrases. Our phrasebook is everyday expressions, which is where the engine struggles most because there is no context to disambiguate. Longer prose likely scores higher.
• Only Ilocano rests on reviewed data. All 72 directions are now measured, but the reference translations for the other seven languages are compiled and pending native review. Treat the Ilocano figures as the firmest and the rest as provisional.
• One engine, one point in time. Translation services change without notice. The figure describes what we observed on the date shown, not a permanent property.

We publish the number anyway, with the caveats attached, because the alternative — asserting that results are “fast and accurate” and offering nothing to check — is what every competing site already does.

There is an unfashionable conclusion buried in all of this. For the vocabulary people actually need most — greetings, courtesy, family, food — a curated phrase list compiled by people outperforms a general-purpose engine, and does so by a wide margin.

The reason is not that the engine is badly built. It is that these are precisely the phrases whose meaning is fixed by convention rather than composition. Agyamanakmeans thank you because that is what Ilocano speakers say, not because “I am grateful” decomposes that way. An engine translating the parts gets the parts right and the phrase wrong.

A human compiler settles it once, records the result, and everyone after that gets the right answer. That is why our 428 checked entries sit alongside the translator rather than behind it — for anything in that list, the list is the better answer.

Machine translation earns its place on everything else: the sentence nobody anticipated, the message that has to be understood roughly and now. Knowing which situation you are in is most of the skill.

• Do not assume whole sentences are safer. We measured it: longer input was no better going into Tagalog and clearly worse going into Ilocano.
• Treat unchanged output as a prompt to check, not as proof of either success or failure.
• Watch the length. Output noticeably longer than the input often means something was invented — our engine turned a two-clause greeting into one containing the word “guys”.
• Use checked phrase data where it exists. Our phrasebook covers common situations and was compiled by people.
• For anything legal, medical, official or public, hire a translator. A tool that is right about three times in five is a draft, not a document.