14.12. Internationalization¶

14.12.1. String locations in CamCOPS¶

Task-specific XML files.

These are loaded by the server (see EXTRA_STRING_FILES) and downloaded by the client. They contain string versions in different languages.

Client calls look like xstring("stringname"), or xstring(STRINGNAME).

Server calls look like req.xstring("stringname") or req.wxstring("stringname"), etc. The w prefix is for functions that “web-safe” their output (escaping HTML special characters).
Client/server shared strings.

Some strings are shared between the client and the server, but are not specific to a given task. They are in the special camcops.xml string file, loaded as above.

Client calls typically look like appstring(appstrings::STRINGNAME); the strings are named by constants listed in appstrings.h.

Server calls typically look like req.wappstring(AS.STRINGNAME) where AS is defined in camcops_server.cc_modules.cc_string.

If a string is “mission critical” for the client, then it should be built into the client core instead (as below).
Client core.

Some text does not require translation (see below).

Text visible to the user should be within a Qt tr("some text") call. Qt provides tools to collect these strings into a translation file, translate them via Qt Linguist, and distribute them – see below.

Client strings that are used only once can live in the source code where they are used.

Client strings that are used in several places should appear in textconst.h.
Server code.

Some text does not require translation (see below).

Text visible to the user should look like _("some text"). The use of _() is standard notation that is picked up by internationalization software that scans the source code. The _ function is aliased to an appropriate translation function, usually via _ = req.gettext. A per-request system is used so that different users of the web site can operate simultaneously in different languages.

Where text is re-used, it is placed in cc_text.py. This is in general preferable because it allows better automatic translation mechanisms than the XML system.

14.12.2. CamCOPS language rules¶

Hard-coded English is OK for the following:

code
Qt debugging stream
command-line text
debugging tests
task short names (typically standardized abbreviations)
database structure (e.g. table names, field names)
config file settings, and things that refer to them

Everything else should be translatable.

14.12.3. Overview of the Qt translation system¶

Begin by editing the Qt project file to add a TRANSLATIONS entry.

Qt suggests looking up languages via QLocale::system::name() (see https://doc.qt.io/qt-5/internationalization.html). This (https://doc.qt.io/qt-5/qlocale.html#name) returns a string of the form language_country where language is a lower-case two-letter ISO-639 language code (i.e. ISO-639-1), and country is an uppercase, “two- or three-letter” ISO 3166 country code. Thus, for example: en_GB (English, United Kingdom of Great Britain and Northern Ireland); probably da_DK (Danish, Denmark).

See
- https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
- https://en.wikipedia.org/wiki/ISO_3166
Though the more common format appears to be language-country, Qt Linguist prefers and auto-recognizes the underscore version.
- https://en.wikipedia.org/wiki/IETF_language_tag
- https://witekio.com/de/blog/qt-internationalization-arabic-chinese-right-left/
Moreover, the underscore is the standard format for locale (rather than language tags); the underscore is also used by POSIX and Python
- https://en.wikipedia.org/wiki/Locale_(computer_software)
- Python: see locale.getlocale()
- Babel insists on it (rejecting hyphens).
We will use the underscore notation throughout.

For example, to add Danish:
```
TRANSLATIONS = translations/camcops_da_DK.ts
```
These files live in the source tree, not the resource file.
Run qmake.
lupdate scans a Qt project and updates a .ts translation file. In Qt Creator, run Tools ‣ External ‣ Linguist ‣ Update Translations (lupdate)
- .ts translation files are XML.
- Pay close attention to error messages from lupdate. In particular, static methods are fine, but classes implementing tr() must both (a) inherit from QObject and (b) use the Q_OBJECT macro.
- However, there is a Qt bug as of 2019-05-03: C++11 raw strings generate the error “Unterminated C++ string”; https://bugreports.qt.io/browse/QTBUG-42736; fixed in Qt 5.12.2.
- To delete obsolete strings, use the -no-obsolete option; e.g.
```
~/dev/qt_local_build/qt_linux_x86_64_install/bin/lupdate -no-obsolete camcops.pro
```
The Qt Linguist tool edits .ts files.
- Class-related strings appear helpfully in against their class.
- Free-standing strings created with QObject::tr() appear against QObject.
- However, lupdate looks like it reads the C++ in a fairly superficial way; specifically, it will NOT find strings defined this way:
```
#define TR(stringname, text) const QString stringname(QObject::tr(text))
TR(SOMESTRING, "some string contents");
```
  but will find this semantically equivalent version:
```
const QString SOMESTRING(QObject::tr("some string contents"));
```
  See also https://doc.qt.io/archives/qq/qq03-swedish-chef.html.

14.12.4. Implementing translatable strings in C++¶

Note also in particular that for obvious initialization order reasons, QObject::tr() doesn’t help with static variables. See e.g. https://stackoverflow.com/questions/3493540/qt-tr-in-static-variable. So, options include:
- if used in one place in the code, just use tr() from within a Q_OBJECT class, e.g.
```
void doSomething()
{
    // ...
    alert(tr("You need to draw on the banana!"));
}
```
- if used multiple in one function, e.g.
```
void doSomething()
{
    const QString mystring(tr("Configure ExpDetThreshold task"));
    // ...
}
```
- if used repeatedly from different places, consider a static member function, e.g.
```
// something.h

class Something
{
    Q_OBJECT
    // ...
private:
    static QString txtAuditory();
}


// something.cpp

QString Something::txtAuditory()
{
    return tr("Auditory");
}
```
  … which appears in the right class in Qt Linguist.
You are likely to need to re-run qmake before lupdate (or, for example, it can fail to pick up on namespaces).

14.12.5. Setting up the translation system¶

See code.

14.12.6. Making the binary translation files¶

Run lrelease, e.g. from within Qt Creator as Tools ‣ External ‣ Linguist ‣ Release Translations (lrelease). This converts .ts files to .qm files.
You need to add the .qm files to your resources.
As always, the :/ prefix in a filename, or qrc:/// for a URL, points to the resources.

14.12.7. Tasks: xstrings¶

Language-specific xstring fetch mechanism implemented.
When the client asks for server strings, all languages are sent.

14.12.8. Determining the language¶

Client: the user chooses the language.
Server: (a) there is a default server language, which also applies when the user is not logged in; (b) users can choose a language.

14.12.9. Server core¶

We need to internationalize (a) Python strings and (b) Mako template strings.
- https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/templates/mako_i18n.html
- https://docs.pylonsproject.org/projects/pylons-webframework/en/latest/i18n.html
- http://danilodellaquila.com/en/blog/pyramid-internationalization-howto
- http://babel.pocoo.org/en/latest/
- https://github.com/wichert/lingua
- https://docs.python.org/3/library/gettext.html#class-based-api; there are editors for gettext systems, like Poedit (this is a pretty good open-source one, with a paid professional upgrade; from Ubuntu, install with sudo snap install poedit)
  - https://wiki.gnome.org/Apps/Gtranslator
  - https://userbase.kde.org/Lokalize
- https://stackoverflow.com/questions/37998300/python-gettext-specify-locale-in
Systems like gettext and Qt’s work on the basis that you write the actual string in the code, then translate it (rather than having an intermediate “lookup string name”) – clearer for developers.
gettext is very much like Qt’s system:
- xgettext or Babel scans your code and extracts message catalogues, producing .po (Portable Object) files.
- .po files are compiled to .mo (Machine Object) files
- gettext loads .mo files and does the translation.
- The convention is to use _("some string") as the notation for the translation function. (Is that what Babel looks for?) Thus, in Mako, the equivalent is ${_("some string")}.
This looks like a well-established framework. Babel supports Mako.
The difficulty is that many have a monolithic context, rather than a request-specific context, in which they translate.

We manage this as follows.

Mako

<% _ = request.gettext %>
## TRANSLATOR: string context described here
<p>${_("Please translate me")}</p>

It would be nice if we could just put the _ definition in base.mako, but that doesn’t come through with <%inherit file=.../>. But we can add it to the system context via camcops_server.cc_pyramid.CamcopsMakoLookupTemplateRenderer. So we do.

Generic Python

_ = request.gettext
# TRANSLATOR: string context described here
mytext = _("Please translate me")

If an appropriate comment tag is used, either in Python or Mako (here, TRANSLATOR:, as defined in build_server_translations.py), the comment appears in the translation files.

Forms

See cc_forms.py; the forms need to be request-aware. This is quite fiddly.

Efficiency

Try e.g.

inotifywait --monitor server/camcops_server/translations/da_DK/LC_MESSAGES/camcops.mo

… from a single-process CherryPy instance (camcops_server .serve_cherrypy), there’s a single read call only.

14.12.10. Updating server strings¶

There is a CamCOPS development tool, build_server_translations.py. Its help is as follows:

USAGE: build_server_translations.py [-h] [--verbose] [--poedit POEDIT]
                                    operation

Create translation files for CamCOPS server. CamCOPS knows about the following
locales:

    ['da_DK']

Operations:

    extract
        Extract strings from code that looks like, for example,
            _("please translate me")
        in Python and Mako files. Write the strings to this .pot file:
            /path/to/camcops/server/translations/camcops_translations.pot

    init_missing
        For any locales that do not have a .po file, create one.

    update
        Updates all .po files from the .pot file.

    [At this stage, edit the .po files with Poedit or similar.]

    poedit
        Launch (spawn) Poedit to edit the .po files.

    compile
        Converts each .po file to an equivalent .mo file.

    all
        Executes all other operations, except poedit, in sequence.

POSITIONAL ARGUMENTS:
  operation        Operation to perform; possibilities are ['extract',
                   'init_missing', 'update', 'poedit', 'compile', 'all']

OPTIONAL ARGUMENTS:
  -h, --help       show this help message and exit
  --verbose        Be verbose (default: False)
  --poedit POEDIT  Path to 'poedit' tool. Default is taken from POEDIT
                   environment variable or 'which poedit'. (default:
                   /path/to/poedit)

14.12.11. Updating client strings¶

Similarly, there is a client-side development tool, build_client_translations.py. Its help is as follows:

USAGE: build_client_translations.py [-h] [--lconvert LCONVERT]
                                    [--lrelease LRELEASE] [--lupdate LUPDATE]
                                    [--poedit POEDIT] [--trim] [--no_trim]
                                    [--verbose]
                                    operation

Create translation files for CamCOPS client.

Operations:

    po2ts
        Special. Converts all .po files to .ts files in the translations
        directory, if and only if the .po file is newer than the .ts file (or
        the .ts file doesn't exist).

    update
        Updates all .ts files (which are XML, one per language) from the .pro
        file and thence the C++ source code.

    [At this stage, you could edit the .ts files with Qt Linguist. If you can't
    find it, use Qt Creator and look within your project in "Other files" /
    "Translations", right-click a .ts file, and then "Open With" / "Qt
    Linguist".]

    ts2po
        Special. Converts all Qt .ts files to .po files in the translations
        directory, if and only if the .ts file is newer than the .po file (or
        the .po file doesn't exist).

    release
        Updates all .qm files (which are binary) from the corresponding .ts
        files (discovered via the .pro file).

    poedit
        Launch (spawn) Poedit to edit the .po files.

    all
        Executes all other operations in sequence, except poedit.
        This should be safe, and allow you to use .po editors like Poedit. Run
        this script before and after editing.

POSITIONAL ARGUMENTS:
  operation            Operation to perform; possibilities are ['po2ts',
                       'update', 'ts2po', 'release', 'poedit', 'all']

OPTIONAL ARGUMENTS:
  -h, --help           show this help message and exit
  --lconvert LCONVERT  Path to 'lconvert' tool (part of Qt Linguist). Default
                       is taken from LCONVERT environment variable or 'which
                       lconvert'. (default: /path/to/lconvert)
  --lrelease LRELEASE  Path to 'lrelease' tool (part of Qt Linguist). Default
                       is taken from LRELEASE environment variable or 'which
                       lrelease'. (default: /path/to/lrelease)
  --lupdate LUPDATE    Path to 'lupdate' tool (part of Qt Linguist). Default
                       is taken from LUPDATE environment variable or 'which
                       lupdate'. (default: /path/to/lupdate)
  --poedit POEDIT      Path to 'poedit' tool. Default is taken from POEDIT
                       environment variable or 'which poedit'. (default:
                       /path/to/poedit)
  --trim               Remove redundant strings. (default: True)
  --no_trim            Do not remove redundant strings. (default: False)
  --verbose            Be verbose (default: False)

14.12.12. Workflow¶

Write code using the _("text") style in Python and the tr("text") style in C++.
Run
./server/tools/build_server_translations.py all ./tablet_qt/tools/build_client_translations.py all
Edit each translation file in turn using Poedit.
Re-run
./server/tools/build_server_translations.py all ./tablet_qt/tools/build_client_translations.py all