18.2. Building the CamCOPS client

The CamCOPS client is written in C++11 using the Qt cross-platform framework.

18.2.1. Prerequisites

Ensure the following prerequisites are met: https://wiki.qt.io/Building_Qt_5_from_Git

18.2.1.1. Linux

  • Linux should come with Python and the necessary build tools.

  • To build Android programs under Linux, you will also need a Java development kit (JDK), such as OpenJDK: sudo apt install openjdk-8-jdk.

  • Tested in Aug 2018 with:

    Ubuntu 16.04
    Ubuntu 18.04 / gcc 7.3.0
    

18.2.1.2. Windows

  • Install a recent version of Python. Make sure it’s on your PATH.

  • Install a Microsoft Visual C++ compiler. A free one is Visual Studio Community. As you install Visual Studio, don’t forget to tick the C++ options.

  • Install these other tools:

  • Add everything to the PATH.

    • In Windows 10, persistent environment variable settings are accessible by searching the Start menu for “environment variables”, or Start ‣ Control Panel ‣ System and Security ‣ System ‣ Advanced System Settings ‣ Environment Variables.

    • You can use either the User or the System settings, as you see fit.

    • PATH elements are separated with semicolons, if you edit the path manually.

    • For example, you may want these:

      C:\cygwin64\bin
      C:\Program Files\NASM
      C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build
      
      -- These are usually added automatically by installers:
      
      C:\Program Files\Git\cmd
      C:\ActiveTcl\bin
      C:\Perl64\bin
      
    • Do make sure that the PATH doesn’t have an unquoted ampersand in; this is technically legal but it causes no end of trouble (see build_qt.py). (The usual culprit is MySQL.) The build_qt.py script will check this.

  • Tested in July 2018 with:

    ActivePerl 5.24.3 build 2404 (64-bit)
    ActiveTcl 8.6.7 build 0 (64-bit)
    CMake 3.12.0 (64-bit)
    Cygwin Setup 2.889 (64-bit)
    Microsoft Visual Studio Community 2017
    NASM 2.13.03 (64-bit)
    Python 3.6
    Qt Creator 4.7.0
    Windows 10 (64-bit)
    

18.2.1.3. macOS (formerly OS X)

  • Tested in July 2018 with:

    macOS 10.13.4
    Xcode 9.4.1 (macOS SDK 10.13; iOS SDK 11.4)
    *** IN PROGRESS
        *** build_qt:
            --build_macos_x86_64
            --build_ios_arm_v7_32
            --build_ios_arm_v8_64
            --build_ios_simulator_x86_32)
            --build_ios_simulator_x86_64 *** can't build SQLCipher
    
        *** redo build_qt command-line help
    

18.2.1.4. All operating systems

  • Install the open-source edition of Qt, with Qt Creator. (You only really need the Tools component. We will fetch Qt separately.)

  • Make sure you have Git installed.

  • Set some environment variables, so we can be consistent in these instructions. Specimen values:

    Environment variable Example value (Linux) Example value (Windows) Notes
    CAMCOPS_QT_BASE_DIR ~/dev/qt_local_build %USERPROFILE%\dev\qt_local_build Read by build_qt.py.
    CAMCOPS_SOURCE_DIR ~/dev/camcops %USERPROFILE%\dev\camcops Used in these instructions and by the Windows Inno Setup script.
    CAMCOPS_VENV ~/dev/camcops_venv %USERPROFILE%\dev\camcops_venv Used in these instructions.
    CAMCOPS_VISUAL_STUDIO_REDIST_ROOT N/A. C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.14.26405 Used by the Windows Inno Setup script.
  • Fetch CamCOPS. For example, for the GitHub version:

    # Linux
    git clone https://github.com/RudolfCardinal/camcops $CAMCOPS_SOURCE_DIR
    
    REM Windows
    git clone https://github.com/RudolfCardinal/camcops %CAMCOPS_SOURCE_DIR%
    
  • Create a virtual environment and install some Python tools:

    # Linux
    python3 -m virtualenv $CAMCOPS_VENV
    . $CAMCOPS_VENV/bin/activate
    pip install cardinal_pythonlib==1.0.23
    
    REM Windows
    python -m virtualenv %CAMCOPS_VENV%
    %CAMCOPS_VENV%\Scripts\activate
    pip install cardinal_pythonlib==1.0.23
    

18.2.2. Build OpenSSL, SQLCipher, Qt

Build a copy of Qt and supporting tools (OpenSSL, SQLCipher) from source using the CamCOPS build_qt.py tool (q.v.). For example:

# Linux
$CAMCOPS_SOURCE_DIR/tablet_qt/tools/build_qt.py --build_all
REM Windows
python %CAMCOPS_SOURCE_DIR%/tablet_qt/tools/build_qt.py --build_all

18.2.2.1. Troubleshooting build_qt

Problem (Windows): fatal error C1041: cannot open program database '...\openssl-1.1.0g\app.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS

… even when -FS is in use via jom.

Solution: just run build_qt.py again; this error usually goes away. Presumably the Qt jom tool doesn’t always get things quite right with Visual C++, and this error reflects parallel compilation processes clashing occasionally. It’s definitely worth persisting, because Jom saves no end of time.

If it fails repeatedly, add the --nparallel 1 option. (It seems to be the OpenSSL build that’s prone to failing; you can always interrupt the program after OpenSSL has finished, and use the full number of CPU cores for the much longer Qt build.)

18.2.3. Run and set up Qt Creator

  • Run Qt Creator.
  • If you are compiling for Android:
    • Install the Android SDK and the Android NDK.
    • Configure your Android SDK/NDK and Java JDK at: Tools ‣ Options ‣ Android, or in newer versions of Qt Creator, Tools ‣ Options ‣ Devices ‣ Android ‣ Android Settings.
  • Proceed with the instructions below.

18.2.4. Qt versions

See Tools ‣ Options ‣ Kits ‣ Qt Versions.

Assuming you set your qt_local_build directory to ~/dev/qt_local_build, the build_qt.py script should have generated a series of qmake (or, under Windows, qmake.exe) files within that directory:

Operating system qmake
Linux 64-bit qt_linux_x86_64_install/bin/qmake
Android (ARM) qt_android_armv7_install/bin/qmake
Android emulator qt_android_x86_32_install/bin/qmake
Mac OS/X 64-bit qt_osx_x86_64_install/bin/qmake
iOS (ARM) qt_ios_armv8_64_install/bin/qmake
iOS Simulator qt_ios_x86_64_install/bin/qmake
Windows 32-bit qt_windows_x86_32_install/bin/qmake
Windows 64-bit qt_windows_x86_64_install/bin/qmake

18.2.5. Qt kits

See Tools ‣ Options ‣ Kits ‣ Kits.

Options last checked against Qt Creator 4.6.2 (built June 2018).

Note

If you did not install a version of Qt with Qt Creator, pick one of your own kits and choose “Make Default”. Otherwise you will get the error Could not find qmake spec 'default'. (e.g. in the General Messages tab when you open your application) and the ..pro (project) file will not parse. See https://stackoverflow.com/questions/27524680.

Custom_Linux_x86_64

Option Setting
Name Custom_Linux_x86_64
File system name  
Device type Desktop
Device Local PC (default for Desktop)
Sysroot  
Compiler: C GCC (C, x86 64bit in /usr/bin)
Compiler: C++ GCC (x86 64bit in /usr/bin)
Environment [not editable: “No changes to apply.”]
Debugger System GDB at /usr/bin/gdb
Qt version THE “LINUX 64-BIT” ONE FROM QT VERSIONS, ABOVE
Qt mkspec  
CMake Tool System CMake at /usr/bin/cmake
CMake Generator CodeBlocks - Unix Makefiles
CMake Configuration [not editable]
Additional Qbs Profile Settings  

Custom_Android_ARM

Option Setting
Name Custom_Android_ARM
File system name  
Device type Android Device
Device Run on Android (default for Android)
Sysroot  
Compiler: C <No compiler>
Compiler: C++ Android GCC (C++, arm-4.9) [1]
Environment [not editable: “No changes to apply.”]
Debugger Android Debugger for Android GCC (C++, arm-4.9) [1]
Qt version THE “ANDROID” ONE FROM QT VERSIONS, ABOVE
Qt mkspec  
CMake Tool System CMake at /usr/bin/cmake
CMake Generator CodeBlocks - Unix Makefiles
CMake Configuration [not editable]
Additional Qbs Profile Settings  

Custom_Android_x86 – NOT FULLY TESTED

Option Setting
Name Custom_Android_x86
File system name  
Device type Android Device
Device Run on Android (default for Android)
Sysroot  
Compiler: C <No compiler>
Compiler: C++ Android GCC (i686-4.9)
Environment [not editable: “No changes to apply.”]
Debugger Android Debugger for Android GCC (i686-4.9)
Qt version THE “ANDROID EMULATOR” ONE FROM QT VERSIONS, ABOVE
Qt mkspec  
CMake Tool System CMake at /usr/bin/cmake
CMake Generator CodeBlocks - Unix Makefiles
CMake Configuration [not editable]
Additional Qbs Profile Settings  

Custom_Windows_x86_64

Option Setting
Name Custom_Windows_x86_64
File system name  
Device type Desktop
Device Local PC (default for Desktop)
Sysroot [...]\qt_local_build\qt_windows_x86_64_install\bin
Compiler: C Microsoft Visual C++ Compiler 14.0 (amd64)
Compiler: C++ Microsoft Visual C++ Compiler 14.0 (amd64)
Environment [not editable: “No changes to apply.”]
Debugger Auto-detected CDB at C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe
Qt version THE “WINDOWS 64-BIT” ONE FROM QT VERSIONS, ABOVE
Qt mkspec  
CMake Tool System CMake at C:\Program Files (x86)\CMake\bin\cmake.exe
CMake Generator CodeBlocks - MinGW Makefiles, Platform: <none>, Toolset: <none>
CMake Configuration CMAKE_CXX_COMPILER:STRING=%{Compiler:Executable:Cxx} CMAKE_C_COMPILER:STRING=%{Compiler:Executable:C} CMAKE_PREFIX_PATH:STRING=%{Qt:QT_INSTALL_PREFIX} QT_QMAKE_EXECUTABLE:STRING=%{Qt:qmakeExecutable}
Additional Qbs Profile Settings  

Custom_Windows_x86_32

Option Setting
Name Custom_Windows_x86_32
File system name  
Device type Desktop
Device Local PC (default for Desktop)
Sysroot [...]\qt_local_build\qt_windows_x86_32_install\bin
Compiler: C Microsoft Visual C++ Compiler 14.0 (amd64_x86)
Compiler: C++ Microsoft Visual C++ Compiler 14.0 (amd64_x86)
Environment [not editable: “No changes to apply.”]
Debugger None
Qt version THE “WINDOWS 32-BIT” ONE FROM QT VERSIONS, ABOVE
Qt mkspec  
CMake Tool System CMake at C:\Program Files (x86)\CMake\bin\cmake.exe
CMake Generator CodeBlocks - MinGW Makefiles, Platform: <none>, Toolset: <none>
CMake Configuration CMAKE_CXX_COMPILER:STRING=%{Compiler:Executable:Cxx} CMAKE_C_COMPILER:STRING=%{Compiler:Executable:C} CMAKE_PREFIX_PATH:STRING=%{Qt:QT_INSTALL_PREFIX} QT_QMAKE_EXECUTABLE:STRING=%{Qt:qmakeExecutable}
Additional Qbs Profile Settings  

Note

For the Microsoft Visual C++ compiler, amd64 means 64-bit and x86 means 32-bit. Then the two-part options are cross-compilers, in which the first part is the type of the host machine (the one running the compiler) and the second part is the type of the destination machine (the one that will run the compiled executable). Therefore, in full, x86 produces 32-bit output using a 32-bit compiler; amd64 produces 64-bit output using a 64-bit compiler (i.e. requiring a 64-bit computer to do the compiling); x86_amd64 produces 64-bit output using a 32-bit compiler (so you can build for 64-bit machines using a 32-bit machine), and amd64_x86 produces 32-bit output using a 64-bit compiler. So, if you have a 64-bit machine, you probably want to use amd64_x86 and amd64; if you have a 32-bit machine, you definitely want to use x86 and x86_amd64.

18.2.6. Build settings

18.2.6.1. Android

Under Project ‣ Build Settings ‣ Build Steps ‣ Build Android APK:

Option Setting
Application ‣ Android build SDK PREVIOUSLY: android-23 [= default]. NOW: android-28 [= default].
Sign package ‣ Keystore ~/Documents/CamCOPS/android_keystore/CAMCOPS_ANDROID_KEYSTORE.keystore [NB not part of published code, obviously!]
Sign package ‣ Sign package Yes (at least for release versions)
Advanced actions ‣ Use Ministro service to install Qt Do NOT tick. (Formerly, before 2018-06-25, this was Qt deployment ‣ Bundle Qt libraries in APK. The objective remains to bundle Qt, not to install it via Ministro.)
Additional libraries ~/dev/qt_local_build/openssl_android_armv7_build/openssl-1.1.0g/libcrypto.so ~/dev/qt_local_build/openssl_android_armv7_build/openssl-1.1.0g/libssl.so

Then in the file AndroidManifest.xml (which Qt Creator has a custom editor for):

Option Setting
Package name org.camcops.camcops
Version code [integer; may as well use consecutive]
Version name [string]
Minimum required SDK API 16: Android 4.1, 4.1.1 [default]
Target SDK WAS: API 23: Android 6.0 [default]. AS OF 2018-06-25: API 26: Android 8.0 [Google Play Store requires this soon]. DOWNGRADED AGAIN 2018-07-16: OpenSSL problems. Probably because you have to rebuild OpenSSL for Android (see DEFAULT_ANDROID_API_NUM in build_qt.py).
Application name CamCOPS
Activity name CamCOPS
Run camcops
Application icon [icon]
Include default permissions for Qt modules [tick]
Include default features for Qt modules [tick]
Boxes for other permissions [no other specific permission requested]

But then you must also edit AndroidManifest.xml manually to include the line:

<meta-data android:name="android.app.load_local_libs" android:value="-- %%INSERT_LOCAL_LIBS%% --:lib/libssl.so:lib/libcrypto.so"/>
Note this bit:                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For versions, see:

If you run this without a keystore, it produces a debug build (e.g. QtApp-debug.apk). If you run it with a keystore/signature, it produces android-build-release-signed.apk (formerly QtApp-release-signed.apk). The APK name is fixed at this point (https://forum.qt.io/topic/43329/qt-5-3-1-qtcreator-rename-qtapp-debug-apk-to-myapp). We can rename the APK if we want, or just upload to Google Play, distribute, etc.

18.2.6.2. Linux

Under Build Settings ‣ Build Environment`, set e.g. LD_LIBRARY_PATH=/home/rudolf/dev/qt_local_build/openssl_linux_x86_64_build/openssl-1.1.0g/

18.2.6.3. General

(I’d like to put general settings in a camcops.pro.shared file, as per http://doc.qt.io/qtcreator/creator-sharing-project-settings.html, but this isn’t working well at present.)

  • Open the camcops.pro project file in Qt Creator.
  • Add your chosen kit(s) to the CamCOPS project.
  • Use defaults, except everywhere you see Build Settings ‣ Build Steps ‣ Make ‣ Make arguments, add -j 8 for an 8-CPU machine to get it compiling in parallel.
  • Build.

18.2.7. Distributing the Whisker client

18.2.7.1. Google Play Store settings

  • Developer URL is https://play.google.com/apps/publish ‣ pick your application ‣ e.g. Release management / App releases

  • App category: “Utility/other”.

  • Content rating: by Google’s definitions, CamCOPS hits criteria for references to illegal drugs (e.g. Deakin1HealthReview, and when strings are available, the various drug abuse scoring scales). Did not meet Google Play’s criteria for sex, violence, etc.

  • Note that “Pending publication” means you’re waiting for Google Play to sort itself out, not that you have to do anything.

  • Note re versions:

    • As above, the AndroidManifest.xml has an INTEGER version, so we may as well use consecutive numbers. See the release history below.

    The Google Developer site will check the version codes. Failed uploads can sometimes block that version number.

  • You upload a new version with App releases ‣ Create Release.

  • Note also that if you try to install the .apk directly to a device that’s had an installation from Google Play Store, you’ll get the error INSTALL_FAILED_UPDATE_INCOMPATIBLE (I think). Or if you mix debug/release versions.

  • Finally, note that there can be a significant delay between uploading a new release and client devices seeing it on Google Play (or even being able to see it at https://play.google.com/store, or via the direct link at https://play.google.com/store/apps/details?id=org.camcops.camcops). Perhaps 10 minutes to the main web site?

18.2.7.2. Google Play Store release history

Google Play Store release name AndroidManifest.xml version code AndroidManifest.xml name To Play Store on Minimum Android API Target Android API
2.0.1 (beta) 2 2.0.1 2017-08-04 16 23
2.0.4 (beta) 3 2.0.4 2017-10-22 16 23
2.2.3 (beta) 5 2.2.3 2018-06-25 16 26
2.2.4 (beta) 6 2.2.4 2018-07-18 23 26
2.2.6 (beta) 7 2.2.6 2018-07-31 23 26
2.2.7 8 2.2.7 2018-08-19 23 26
2.2.8 9 2.2.8 pending 23 26
2.2.9 10 2.2.9 pending 23 26

18.2.7.3. Windows client packaging

This can be done automatically via the camcops_windows_innosetup.iss script, which runs under Inno Setup.

18.2.8. Notes

18.2.8.1. Version constraints for third-party software

18.2.8.2. Android

18.2.8.3. Debugging

  • DON’T FORGET to set up both Debug and Release (+/- Profile) builds.

  • Phone USB debugging negotiation sometimes takes a while. On the Samsung Galaxy phone, the alert light goes red when in Debug mode.

  • If a USB Android device appears not to connect (via adb devices), appears then disappears as you connect it (via lsusb | wc), and gives the dmesg error device descriptor read/64, error -71 or similar, try a different cable (see https://stackoverflow.com/questions/9544557/debian-device-descriptor-read-64-error-71); try also plugging it directly into the computer’s USB ports rather than through a hub.

  • If you lose the debugger windows in Qt Creator midway through a debug session, press Ctrl-4.

  • This error (with a variety of compiler names):

    .../mkspecs/features/toolchain.prf(50): system(execute) requires one or two arguments.
    Project ERROR: Cannot run compiler 'g++'. Maybe you forgot to setup the environment?
    

    means that you need to re-run qmake manually. It usually occurs if you delete your build* directories.

  • For debugging, consider install Valgrind: sudo apt install valgrind

18.2.8.4. Troubleshooting qmake/compilation

  • Sometimes you have to restart Qt creator after creating new build settings; it loses its .pro file and won’t show the project, or complains of a missing .pro file when you try to build.

  • The first build can be very slow as it compiles all the resources; this usually looks like a process stuck compiling qrc_camcops.cpp to qrc_camcops.o

  • If builds are very slow, you may have forgotten to use all your CPU cores; try e.g. -j 8 (for 8 cores) as an argument to make, as above.

  • If an Android build fails for a bizarre reason (like garbage in a .java file that looks like it’s been pre-supplied), delete the whole build directory, which is not always removed by cleaning.

  • This error whilst building CamCOPS:

    /home/rudolf/dev/qt_local_build/qt_linux_x86_64_install/bin/qmlimportscanner: error while loading shared libraries: libicui18n.so.55: cannot open shared object file: No such file or directory
    /home/rudolf/dev/qt_local_build/qt_linux_x86_64_install/mkspecs/features/qt.prf:312: Error parsing JSON at 1:1: illegal value
    Project ERROR: Failed to parse qmlimportscanner output.
    

    … occurred after an upgrade from Ubuntu 16.04 to 18.04; the problem relates to missing OS libraries (libicu); the easiest thing is to rebuild Qt.

18.2.8.5. Troubleshooting running CamCOPS

  • Runtime error, failing to find libssl.so or libcrypto.so:

    Starting /.../camcops...
    /.../camcops: error while loading shared libraries: libssl.so: cannot open shared object file: No such file or directory
    /.../camcops exited with code 127
    

    CamCOPS needs the libssl.so and libcrypto.so that was built by build_qt.py. Until we have a proper Linux client distribution, do this:

    $ export LD_LIBRARY_PATH=~/dev/qt_local_build/openssl_linux_x86_64_build/openssl-1.1.0g/
    

    … or wherever you built those .so libraries. Then re-run the CamCOPS .client.

  • This error whilst running CamCOPS (Ubuntu 18.04):

    Starting /.../camcops...
    /.../camcops: error while loading shared libraries: libOpenVG.so.1: cannot open shared object file: No such file or directory
    /.../camcops exited with code 127
    

    Thoughts:

    # Which files have similar names?
    
    $ find -L / -type f -name "libOpenVG.so*" 2>/dev/null
    /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so.1       # symlink to libOpenVG.so.1.0.0
    /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so.1.0.0   # actual file
    /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so         # symlink to libOpenVG.so.1.0.0
    /usr/lib/x86_64-linux-gnu/libOpenVG.so                  # symlink to mesa-egl/libOpenVG.so
    
    # Which packages provide these files?
    
    $ dpkg --search libOpenVG
    libopenvg1-mesa:amd64: /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so.1.0.0
    libopenvg1-mesa-dev: /usr/lib/x86_64-linux-gnu/libOpenVG.so
    libopenvg1-mesa-dev: /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so
    libopenvg1-mesa:amd64: /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so.1
    
    # Ergo, the problem can be solved with:
    
    $ sudo ln -s /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so.1 /usr/lib/x86_64-linux-gnu/libOpenVG.so.1
    
    # Yup, that fixes it.
    

    Solution:

    sudo ln -s /usr/lib/x86_64-linux-gnu/mesa-egl/libOpenVG.so.1 /usr/lib/x86_64-linux-gnu/libOpenVG.so.1
    

Footnotes

[1](1, 2) Prior to Qt 5.12.0, the compiler was “Android GCC (arm-4.9)”, and the debugger was “Android Debugger for Android GCC (arm-4.9)”.