Machine Translation & Post-editing in the Industry

Machine Translation & Post-editing in the Industry
THIS RESEARCH IS CURRENTLY CLOSED TO SUBMISSIONS.

This research is carried out within the framework of [anonymized].

It aims to identify the skills necessary for professional post-editors of machine translation.

Your participation in this survey-based empirical research is voluntary and your data will remain confidential and will only be used for research purposes. Please transfer it to the person in your firm who knows the most about the translation workflow.

Should you have any questions, feel free to contact [anonymized].

Filling out this questionnaire takes 15-20 minutes, depending on your expertise in machine translation post-editing. Required questions are marked with an asterisk.

At the end, you will be invited to leave your email address if you wish to receive a report with the results.
Basic Information
1. Gender*

MaleFemaleN/A
2. Age*
3. What is the highest degree you have completed?*

High schoolTrade or vocational trainingBachelor's degreeMaster's degreeDoctorate degreeOther
4. Job title*
5. Type of company*

Translation agency or Language Service Provider (LSP)Firm, company, corporation or publishing houseOrganization (NGO, public body...)Other
6. Does it have a translation department?*

YesNo
7. Size of company:*

Less than 10 employeesBetween 11 and 50 employeesBetween 51 and 500 employeesMore than 500 employees
8. Does it have office(s) in Europe?*

YesNo
9. Name of company*
10. Country*
10_1. Please, enter the country where your firm/company has an office below:*
MTPE Projects
MTPE (machine translation post-editing) is defined as "editing and correcting machine translation output". Machine translation is defined as "automatic translation of text from one natural language to another using a computer application"

Source: ISO 18587-2017
11. Does your company handle translation projects with post-editing of machine translation (MTPE)?*

YesNo
11_1. For which type of documents do you use MTPE?*

Documents with restricted access (low-visibility content)Publicly available documents (high-visibility content)OtherI do not know

Client-TSP Agreement
12. Do you use an agreement between client and translation service provider (TSP)?*

YesNo
13. Does the client-TSP agreement contain any of the following elements?*

Project specifications: Linguistic compliance requirements (such as style guide, terminology, phraseology, etc.)Additional specifications: Corpora used to train MT engines (alignments to tune or customize the MT system)Additional specifications: Source texts content profile, pre-editing of the ST...Additional specifications: Quality of the MT system for the language pair of the projectAdditional specifications: Type of PE (light or full)Additional specifications: Use of technology (MT system)Additional specifications: Forbidden use of machine translationAdditional specifications: Validation of the target content by the client experts or subsidiariesAdditional specifications: Translation Quality Assessment (TQA) report of the target textOther

Current Trends in MTPE Projects
The next three sections will ask the core questions of this survey.

Your MTPE Background
14. How long have you been handling MTPE projects?*

Less than 1 yearBetween 1 and 5 yearsMore than 5 years
15. What is the percentage of your translation production via post-editing (PE) of machine translation (MT)?*

25% or lessBetween 26% and 50%Between 51% and 75%More than 76%
16. Please select the most frequent language pairs in which you translate content via MTPE.
Source language 1*
Target language 1*
Source language 2
Target language 2
Source language 3
Target language 3
It seems that your language is not in the list. Please specify it below.

Language codes may follow ISO 639-2 or ISO 639-3, and ">" shows direction towards. For instance "EN>FR" would be English into French. Feel free to write the name of the languages in full letters if you are not sure about the ISO code. To show bidirectionality you can use "<>", for instance: English<>French.
16_1. For which language pairs do you use MTPE?*
17. In general, who triggers the decision of translating content via MTPE?*

The client or requester of the translationOur own firm or companyThe end-users of the translated contentOther
18. Does the client ever send an MT output along with the source text?*

Yes, and they inform us of the MT system usedYes, and they do not inform us of the MT system usedNo
19. Are your MTPE projects generally post-edited with the source text (ST) as reference?*

No, the source text is not available (we do monolingual post-editing)Yes, it is bilingual edititing: source + MT outputYes, it is trilingual editing: source + MT output + post-edited output
20. How do you decide if a ST is apt for machine translation and post-editing?*

Content profiling (type of text, repetition rate, visibility, etc.)Current general quality of MT for the pair of languages at handExternal factors play a central role: turnaround time & costsA thorough Source Text Analysis (STA) is carried out to evaluate non-translatability indicatorsOther
21. How do you decide if an MT output is of enough quality to be post-edited?*

Automatic scoring (metrics like BLEU, TER, METEOR, etc.)Human evaluation by error categorizationMachine Translation Quality Estimation (QE) or confidence scores (predictive)Human task-based evaluation: edit distance after post-editingOther
22. Have you ever used MT output without informing the client of such workflow?*

NoYesYes, post-edited
23. What is the level of post-editing (PE) that is most often required?*

Light post-editing or "good enough" qualityFull post-editing or "human" qualityI do not know
24. To what extent are post-edited texts revised or checked by a second translator/reviser?*

They undergo both revision and a final checkThey are revisedThey undergo a final checkThey do not undergo any further revision or checkI do not know
25. Which of the following PE scenarios is most used in your firm or company?*

Improving medium (acceptable) quality raw output to high or publishable qualityImproving poor quality raw output to medium (acceptable) qualityImproving poor quality raw output to high or publishable quality.I do not know
26. Do you keep track of post-editors productivity rate?*

YesNo
27. How many words per hour do you expect a post-editor to be able to post-edit on average?*

Less than 1000 wphBetween 1000 and 3000 wphBetween 3000 and 5000 wphMore than 5000 wph
Controlled Language / Authoring
28. Do you use any controlled language (CL) or controlled authoring (CA) tool to write or pre-edit content that you will translate with MT?*

YesNo
29. Who developed the CL that you use to write or pre-edit content that you will translate using MT?*

Internally at my firm or companyA third partyOther
30. For what language(s) do you use CL or CA with a view of translating the content via MT?*

The Machine Translation Post-Editor Profile
31. How many in-house post-editors work at your company?*
32. How many freelance MT post-editors does your database contain?*
33. Do your in-house post-editors work 100% of their time on MTPE projects?*

Yes, they work full-time on post-editing of MTNo, they also translate or do translation project managementNo, they have a long list of other tasks

34. What workload do these tasks represent for in-house post-editors?*

	Main task	Secondary task	Eventual task	N/A
Customization/Tuning of MT engines
Feedback collection on MT output quality for solution engineers
Management of MTPE projects: outsourcing, etc.
Material preparation for MT engine training (building corpora, alignment, cleaning TM...)
MT output quality evaluation (error categorization...)
PE guidelines design
Post-editing machine translation output
Pre-editing the source text
Proofreading of post-edited output (monolingual)
Quality control & text checking
Revision of post-edited MT output (bilingual)
Support users with CAT/MT tools
Terminology extraction and TB management
Tracking PE productivity

35. What workload do these tasks represent for freelance post-editors?*

	Main task	Secondary task	Eventual task	N/A
Customization/Tuning of MT engines
Feedback collection on MT output quality for solution engineers
Management of MTPE projects: outsourcing, etc.
Material preparation for MT engine training (building corpora, alignment, cleaning TM...)
MT output quality evaluation (error categorization...)
PE guidelines design
Post-editing machine translation output
Pre-editing the source text
Proofreading of post-edited output (monolingual)
Quality control & text checking
Revision of post-edited MT output (bilingual)
Support users with CAT/MT tools
Terminology extraction and TB management
Tracking PE productivity

36. How do you usually pay for MTPE projects?*

Editing distance (post-analysis)Hour rate (time spent is established by the freelance post-editor)Hour rate (time spent is recorded by a programme)Source word rate (pre-analysis)Source page rateTarget word rateTarget page rateOther
Post-editor Recruitment Criteria
37. Which of the following post-editor profiles would you value more for your MTPE projects?*

A professional with a stronger linguist profile (with some subject field knowledge)A professional with a stronger "expert" profile (with knowledge in languages)
Please pay special attention to the next two questions. They are essential to guarantee sound results for this research.

38. Which of the following criteria does your firm/company apply to recruit or select a post-editor of MT?*

	Mandatory	Very important	Important	Not important
Capacity to post-edit into both directions
CAT tool(s) knowledge
Certification in PE by a professional association (ProZ, TAUS, etc.)
Experience in project management
MT system knowledge
Pre-editing or controlled language (CL) skills
Previous experience in post-editing MT output
Productivity (processing speed)
Quality assurance (QA) checking skills
Revision & proofreading skills
Skills using automatic speech recognition (ASR) or touch-screen technology
Specific locale (variant, sublanguage)
Subject field knowledge or specialization
Technical skills: macros, xliff, tmx, Java, RegEx...
Terminology management & information mining skills
Translation Quality Assessment (TQA) skills (scores, metrics, evaluation, etc.)
University degree in Translation or related studies

39. Please rate the following MT post-editing skills & competencies according to the importance you think they have for a professional post-editor. 1 = slightly important | 5 = very important.*

	1	2	3	4	5
Capacity to decide when to edit or discard (translate from scratch) an MT result
Capacity to post-edit according to PE guidelines
Capacity to post-edit up to human quality (full PE)
Capacity to post-edit to a good enough quality (light PE)
Capacity to pre-edit a source text according to CL
Capacity to train & tune an MT engine
Capacity to identify MT output errors
Capacity to apply the right correction strategy
Capacity to advise when MTPE is appropriate for a text or project
Capacity to provide feedback for the MT solution engineers
Capacity to learn about new technologies

40. If you ever receive unsatisfactory deliveries, in what way do they not respect the MTPE assignment?*

It does not happen; our post-edited texts are always fit for purposeThey are under-edited (necessary corrections to the MT output are not applied)They are over-edited (preferential changes that were not required are applied)They are pseudo-edited (the post-editor introduces errors that were not present in the MT output)I do not knowOther
41. Do you have a recruitment test for post-editors, different from the one for translators?*

YesNo

The MT Post-Editing Project
Post-Editing Assignment or Brief
42. Do you use a post-editing assignment or brief?*

Yes, we give detailed information: MT system used, how it was trained, metrics and scores of its quality, PE guidelines, among other itemsYes, we might provide some information: explanation of the workflow or PE guidelinesNo
42_1. Which of the following does the PE assignment/brief include?*

Content profile (type of text, communication channel, STA...)Examples of scenarios indicating when to discard a segment (= translate from scratch)Information on how the MT engine was trained and its most frequent errorsMetrics about MT output qualityMT system architecture information (hybrid, rule-based, statistical, neural)Name of the machine translation (MT) systemPE guidelines: general list of do's and dont's (such as: "fix misspelling errors: YES", "improve the style: NO")PE guidelines: list of examples tailored to the language pairPE level: light or full post-editingOther
Post-Editing Guidelines
43. Which PE guidelines do you send to the post-editor?*

Internally customized and fixed for the whole firm or companyInternally customized and tailored to content type or languageTAUS post-editing guidelinesOnly PE level indication (light or full post-editing)Other
Post-Editor Feedback
44. Do you ask your post-editor for feedback concerning MT output in order to improve the MT system?*

YesNo
44_1. What elements are sent by the post-editor after a MTPE project?*

Examples of recurrent errors: only an informal list of wordsExamples of recurrent errors (in a structured template): source, MT output, and post-edited outputGeneral description of how the MT engine performsType of error: collocation, syntax, grammar, spelling, punctuation, terminology...Severity of the errors: critical, major, minor, neutral...Other
TQA (Translation Quality Assessment)
45. Do you use a Translation Quality Assessment (TQA) system for MTPE projects?*

NoYes, one integrated in the CAT toolYes, one not integrated in the CAT toolOther
45_1. Is your TQA model defined according to a risk management strategy?*

YesNoI do not know
45_2. Which of the following TQA systems do you use?*

ATA (American Translators Association) error categorizationITS 2.0 Localization quality issuesLISA QA MetricMQM (Multidimensional Quality Metrics)MTM LinguaSoft QA ModelSAE J2450SDL TMS Classic ModelTAUS DQF (offline)TAUS DQF Dashboard (online)Other

Training in Machine Translation Post-Editing
46. Does your firm/company offer MTPE training?*

We haven't organized one as of yetWe have organized at least oneTraining on MTPE is organized regularly
47. Who attends the MTPE training?*
48. Does the training change depending on who is attending?*

YesNo
49. Who is the trainer?*

In-house staffTAUS Post-Editing/Reviewing CourseOther
49_1. Who is the trainer for in-house participants?*

In-house staffTAUS Post-Editing/Reviewing CourseOther
49_2. Who is the trainer for freelance participants?*

In-house staffTAUS Post-Editing/Reviewing CourseOther
50. How is the training organized?*

PresentialOnline (videoconference)Online (recorded)Semi-presentialOther
50_1. How is the training organized for in-house participants?*

PresentialOnline (videoconference)Online (recorded)Semi-presentialOther
50_2. How is the training organized for freelance participants?*

PresentialOnline (videoconference)Online (recorded)Semi-presentialOther
51. Do you know what the training provided by in-house staff includes?*

YesNo
52. What elements does the training include?*

Description of content profiles and text types according to MT systemsIntegration between the CAT tool and MT systemMT evaluation: automatic & reference-based (metrics)MT evaluation: human (scoring, ranking, error categorization)MT systems: rule-based, example-based, statistical, hybrid, or neuralPE attitude: decide when to discard and translate from scratchPE guidelines: exhaustive list and examples in the relevant language pairPE levels: light and full post-editingPractical PE exercises in the relevant language pairPE risks: under-editing, over-editing & pseudo-editingPE techniques and strategies (shift, replacement, addition, deletion)PE technology: PE tool, ASR, touch-screen, etc.Pre-editing and controlled languagesProductivity tracking toolsQuality estimation or confidence scores (predictive, without post-edited gold reference)Other

52_1. Which elements does the training include for each type of attendees?*

	For in-house participants	For freelance
Description of content profiles and text types according to MT systems
Integration between CAT tool and MT system
MT evaluation: automatic & reference-based (metrics)
MT evaluation: human (scoring, ranking, error categorization)
MT systems: rule-based, example-based, statistical, hybrid, or neural
PE attitude: decide when to discard and translate from scratch
PE guidelines: exhaustive list and examples in the relevant language pair
PE levels: light and full post-editing
Practical PE exercises in the relevant language pair
PE risks: under-editing, over-editing & pseudo-editing
PE techniques and strategies (shift, replacement, addition, deletion)
PE technology: PE tool, ASR, touch-screen...
Pre-editing and controlled languages
Productivity tracking tools
Quality estimation or confidence scores (predictive, without post-edited gold reference)

Technology & Tools
53. What MT architecture does your MT system have?*

Rule-basedStatisticalNeuralI do not knowOther
54. What MT provider(s) do you use most often for your MTPE projects?
1st most used*
2nd most used (if applicable)
54_1. What MT provider do you use?*
55. Do you use existing translation memories and/or glossaries to train, customize or tune an MT engine?*

YesNoI do not know
56. Is your MT system integrated with any translation productivity tools other than a CAT tool?*

Yes, with localization softwareYes, with subtitling softwareYes, both with localization and subtitling softwareNoOther
57. Are MT output and Translation Memory results combined in the editor?*

YesNo
TM/MT Not integrated
58. In what tool is the MT output post-edited?*

Text-processing toolMT editorOther
Integrated TM/MT
59. In which tool are MT and TM integrated?*

CAT toolMT editorOther
60. What CAT tool(s) do you use most often?
1st most used*
2nd most used
61. Does the post-editor see the origin of the translation result?*

YesNoI do not know
62. Do you offer "repaired" segments? (both TM result and MT output mixed to build the fuzzy to be post-edited)*

YesNoI do not know
63. What tool do you use to track the productivity rate?*

Excel sheetKlokProject management programs like Plunet, XTRF, Lotus, Everwin, Trello, Dapulse, etc.QualitivitySoftware developed internallyTAUS DQFNoneOther
64. Do the post-editors use any of the following tools to perform quality assurance (QA) checks?*

AntidoteCheckmateContentQuoglobalReviewLanguage ToollexiQAProject Open Translation QualityQA DistillerTQAuditorVerifikaXbenchNoneOther
65. What controlled authoring tool do you use most often?*

AcrolinxCLATCongreeCrossAuthorHyperSTEMaxITMindReaderNone, only text processing tools.Other

Feelings & Thoughts
66. Do you think that current PE training courses are adequate for a professional post-editor position?*

YesNoOther
67. Would you like to have access to an online platform that allows your firm/company to send PE assignments to post-editor trainees?*

YesNoOther
68. Are you aware of the publication of or existing work on any of the following standards?*

ISO 18587:2017 Translation services — Post-editing of machine translation outputASTM WK46396 New Practice for Development of Translation Quality MetricsNoOther
69. Please enter here any comments that you might have on the questions of this survey.
Thank you!
Should you agree to be contacted to discuss some of the questions of this survey, please enter your e-mail address and full name below:
example@example.com
Full name:

First nameLast name
If you only wish to receive a report with the results of this survey, please leave your e-mail address below:
example@example.com

Thank you very much indeed for having spent some time filling in this survey.
For this research, we are only considering LSPs, firms or corporations that translate content using machine translation and human post-editing.
For this research, we are only considering LSPs, firms or corporations that have offices in Europe and that have a translation department.
Public organizations such as NGOs or public bodies are not considered for the moment. However, similar research among these institutions might be carried out in the near future.

If you wish to participate in it, please leave your personal details below:
Full name:

First nameLast name
E-mail:
example@example.com
Organization, NGO or institution:
Should be Empty:

Machine Translation & Post-editing in the Industry

Basic Information

MTPE Projects

Client-TSP Agreement

Current Trends in MTPE Projects

Controlled Language / Authoring

Post-editor Recruitment Criteria

Post-Editing Assignment or Brief

Post-Editing Guidelines

Post-Editor Feedback

TQA (Translation Quality Assessment)

Training in Machine Translation Post-Editing

Technology & Tools

TM/MT Not integrated

Integrated TM/MT

Feelings & Thoughts

Thank you!