15. Change log/history

Both client and server changes are described here.

Quick links:

15.2. Original Titanium/Javascript client, Python server with custom MySQL interface (defunct)

15.2.1. 2013

Server v1.0, 2013-08-14

  • First version (1.0).

Client v1.0, 2013-11-13

  • first version
  • requires server version 1.0

Client v1.01, 2013-11-13

  • test of version number increment
  • Bugfix: Executive menu had a duff entry in and crashed.

Server v1.01, 2013-11-20

  • Test of version number increment (1.01).
  • Trivial change: ensure empty “*_SUMMARY_TEMP_current*” views aren’t created for anonymous tasks.

Client 1.02, 2013-11-22 onwards

  • A couple of cosmetic changes.
  • Analytics yes/no option.
  • Changed app “domain” to org.camcops.*, so app is org.camcops.camcops
  • Signed APK file.
  • QuestionTypedVariables improved in a few respects.
  • QuestionDiagnosticCode bugfix (didn’t appear read-only in read-only mode).
  • android:allowBackup explicitly set to false
  • ID description/policy check on upload.
  • Titanium API now 3.2.0.GA
  • Page jump in questionnaires when read-only.
  • Two CECAQ3 fields used the wrong keyboard/type.
  • Text field/cursor colours improved for iOS/Android.
  • Bugfix to QuestionCanvas_webview.

Server v1.02, 2013-11-28

  • Mostly changes on the app side (q.v.).
  • Change to DemoQuestionnaire fields.
  • DOB task filter.
  • Server analytics with yes/no option.
  • Fixed layout on old versions of Internet Explorer.
  • get_id_info command in the database interface.
  • QoL* tasks remain in beta; data structure may change.
  • Changes for CentOS, including Python version check and altered shebang. Using “#!/usr/bin/env python2.7” is perhaps desirable, but Lintian requires e.g. “#!/usr/bin/python2.7”: http://lintian.debian.org/tags/python-script-but-no-python-dep.html
  • Clinical text view.

15.2.2. 2014

Client v1.03, 2014-01-10

  • Requires server version 1.03.
  • Fixed Titanium 3.2.0 multiline TextArea regression.
  • CGI-SCH task, pending permissions.
  • androidtipaint/QuestionCanvas_tipaint improved/fixed for Titanium 3.2.0.
  • Questionnaire scrollview made full height (Titanium now capable of it).
  • Single-tap/double-tap methods in diagnostic coding, now Titanium bug https://jira.appcelerator.org/browse/TIMOB-15540 fixed.
  • Photo rotation bug fixed.
  • ListView for diagnostic code search.
  • QoL-SG phrasing improved.
  • Tested on iOS 7.0.3/7.0.4, Android 4.1.1.

Server v1.03, 2014-01-10

  • CGI-SCH task.

Client v1.04, 2014-01-14

  • First beta version.
  • Bugfix to Patient.js (re address display crash).
  • Changes to SetMenu_Deakin_1.js
  • Confirmation of CGI-SCH permissions.

Client v1.05, 2014-01-14

  • Password entry windows improved: return key now accepts data entry.

Client v1.06, 2014-01-16

  • Requires server version 1.06.
  • CPFT_LPS_Referral, CPFT_LPS_Discharge, CPFT_LPS_ResetStartClock tasks. This are IN BETA.
  • Batch upload empty tables for speed (big improvement).
  • NULL-but-optional indicator in widgets:
    • QuestionDateTime, QuestionPickerInline, QuestionPickerPopup, QuestionSlider, ImageGroupVertical.
  • offerNullButton option in QuestionDateTime, QuestionDiagnosticCode
  • Variable column widths in ContainerTable, plus populateVertically option.
  • Bugfix in QuestionTypedVariables layout for colWidthPrompt.
  • (2014-01-18) Minor layour change in CPFT_LPS_Referral.

Server v1.06, 2014-01-16

  • REQUIRES DATABASE CHANGE BEFORE INSTALLATION: DROP TABLE _dirty_tables;
  • CPFT_LPS_Referral, CPFT_LPS_Discharge, CPFT_LPS_ResetStartClock tasks. IN BETA; MAY CHANGE.
  • QoL* tasks remain in beta; data structure may change.
  • Options in man page.
  • cc_patient.py / get_id_generic and similar: bugfix to use Unicode
  • Clinical text provided by Photo/PhotoSequence.
  • Batch upload empty tables.
  • rnc_db: skips creation of tables that exist already (removes a warning).
  • Joint PK for _dirty_tables, and change from TEXT to VARCHAR(255) for the tablenamefield.
  • Bugfix to database.pl / flag_deleted_where_clientpk_not: wasn’t device-specific! Was used by blob upload on the tablet, i.e. dbupload.sendTableRecordwise()

Server v1.07, 2014-02-14

  • REQUIRES DATABASE CHANGE BEFORE INSTALLATION: DROP TABLE _security_webviewer_sessions;
  • CPFT* tasks remain in beta; data structure may change.
  • QoL* tasks remain in beta; data structure may change.
  • Additional content for clinical text views.
  • Python virtualenv.
  • Dumping/reporting options for suitably privileged users. Additional user permissions: may_dump_data, may_run_reports.
  • Bugfix to Session class to prevent the (incredibly unlikely) event of an IP address hop with an identical session token.
  • Security improvement to Session class: change token upon login.
  • Speedup to Session design (inc. integer PK).
  • Typo in CAPS text, Q24.
  • Speedup to LSTRING XML processor.
  • Speedup via transaction-based database handling in the Python handler.
  • Redirect to destination URL after re-authentication.

Server v1.08, 2014-07-22

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • Automatic version-based database structure upgrade via the –maketables command. (Similarly on the tablet side.)
  • Distinct patient reports.
  • CPFT* tasks remain in beta; data structure may change.
  • QoL* tasks remain in beta; data structure may change.
  • Remote IP addresses stored in audit log (additional field: remote_addr).
  • Auditing of clinical text views.
  • Some string constant code cleanup.
  • Some Perl code cleanup and upload audit simplification.
  • perltidy for Perl code.
  • Trackers/CTVs clearer in their errors when no data found.
  • Ability to apply multiple filters simultaneously.
  • Option to force password changes periodically/ad hoc.
  • PEP8 compliance for core Python code.
  • PEP8 compliance for task code.
  • Proper multiple inheritance handling for diagnosis.py, pcl.py.
  • Disclaimer/acknowledgement recording.
  • Audit all login attempts, plus user addition/deletion.
  • OptionParser to ArgumentParser.
  • Internal URLs for tasks altered slightly.
  • Better internal timezone handling.
  • Commit during menu-driven administration to prevent database locking.
  • Lock user accounts after multiple login failures.
  • HL7 message framework. (Validated internally and against HL7 Inspector.)
  • File export message framework, with post-export script option.
  • Database title, ID descriptions, and policies now have their primary home in the configuration file. Copied to database purely for researcher lookup.
  • File locking for the regeneration of summary tables.
  • XML export (tasks, trackers, CTVs).
  • Unit testing framework (and a couple of bugs fixed).
  • Shift to unsigned ints for PKs.
  • Option to introspect source code.
  • Option to view table definitions from webview.
  • Basic non-modifying anonymisation system.
  • Bugfix: added vignette to ICD10-PD display.
  • Bugfix: HAMD-7 maximum is 26, not 23.
  • Bugfix: CECAQ3 failed to calculate some summary scores with no siblings, and paternal psychological abuse score was sometimes inappropriately blank.
  • Bugfix: filter for incomplete tasks only wasn’t working.
  • Bugfix: logic bugfix in ICD-10 manic, mixed, schizophrenia.
  • Bugfix: categorization text in BMI.
  • Bugfix: clinical text for SLUMS reported incorrect maximum.
  • BMI thresholds refined in the underweight zone and referenced properly.
  • All field comments.
  • Manual erasure of individual tasks.
  • Manual deletion of entire patients/associated tasks.
  • Manual application of special notes.
  • CTV is clearer when tasks are incomplete.
  • More consistent formatting of null values in HTML. (Note that the quick way to view null handling is to specify a nonexistent server PK.) The aim is that all user answers should be proceesed via the answer() function, to apply typographic indications that the field is null.
  • camcopswebview.py renamed to camcops.py.
  • Optimization on compile.
  • Ensure commit/rollback always occurs, even after exceptions.
  • “crash” action to induce a deliberate exception, for testing.
  • Configurable save-as filenames for tasks, trackers, and CTVs.
  • Server-side validation of fields (field_contents_valid).
  • Unit tests prohibit tasks from having summary fields with the same name as a main task field.
  • Option to disable password autocompletion on the login page.
  • Server version number in “office” details.
  • Generator function for task list.
  • Drop-down lists for filters remember state.
  • Basic research dump (likely to be the most useful in practice).

Client v1.08, 2014-07-23

  • Requires server version 1.08.
  • Field renaming within Icd10Schizophrenia to avoid misnomers:
    • tpah_commentary TO hv_commentary
    • tpah_discussing TO hv_discussing
    • tpah_from_body TO hv_from_body
  • CPFT* tasks remain in beta.
  • Chaining of tasks.
  • Page jump within live questionnaires (allowPageJumpDuringEditing).
  • Radio buttons allow double-clicks/taps to unset them (particularly applicable for potentially loaded questions).
  • Bugfix to HAMD-7: Q4 value 4 and Q5 values 3/4 were not offered, and maximum is 26, not 23.
  • Bugfix to SLUMS: Q9a, Q9b were scored as 1 point each; should be 2.
  • Bugfix calling bad afterwardsFunc() after “move” upload.
  • BMI thresholds refined in the underweight zone and referenced properly.
  • Textual annotation to ICD-10 F90.0, as the actual text gives you no clue that it’s a division of hyperkinetic disorders.
  • dbcore.js changed to reflect Titanium bugfix.
  • Android theme changed to light (with consequent changes to questionnaire font size editing screen, etc.).

Client v1.09, 2014-08-02

  • Requires server version 1.09.
  • Sends BLOBs in ways that cannot be confused with (even very bizarre) strings.
  • PANSS stripped down to data collection tool only, for copyright reasons.
  • Not distributed yet.

Server v1.09, 2014-08-02

  • REQUIRES TABLET CLIENT V1.09.
  • Full rewrite of the database upload script to Python.
  • Fix MySQL “morning bug” (“MySQL server has gone away”) from the Perl upload script.
  • Logic change to flag_all_records_deleted(), which was not restricted to _current/era=’NOW’ records, but should have been.
  • Also rolls back preservation flag changes as part of general rollback.
  • BLOB transfer encoding improved; fixes design flaw that was due to the use of the Perl CSV module. (Requires tablet client v1.09 as a result.)
  • Internal code changes: explicit modules in all cases, removing cc_shared.py.
  • PANSS stripped down to data collection tool only, for copyright reasons.

Client v1.10, 2014-08-08

  • Default network timeout changed from 5 s (5000 ms) to 60 s (60000 ms), as shorter timeouts were causing large BLOB uploads to fail.
  • Minor fix to newline decoding for the mobileweb client.
  • Ability to null out dates of birth (for anonymised research use).
  • NULL dates now show in the widget as 01 Jan 1900, not the current date (it’s impossible to show an actual NULL, and the current date is confusing when you have neonates).
  • QuestionDateTime widget wouldn’t successfully NULL itself on Android. (So now it NULLs itself but doesn’t update its pseudo-date; it just displays the NULL icon.)
  • First jshint compliance (except for included third-party libraries)…
  • … then jslint compliance.
  • Unit testing framework.
  • Not distributed yet.

Client v1.12, 2014-09-11

  • Renamed ExpDetThreshold/ExpectationDetection tasks (and tables) to add a “[C/c]ardinal_” prefix, as the names were too vague. THEREFORE requires server version 1.12 as well.
  • Session-based authentication for tablets to improve speed (i.e. no need for bcrypt reauthentication within the same session, as for the web front end).
  • Whisker interface.

Client v1.14, 2014-10-15

  • Requires server version 1.14.
  • Server can enforce a minimum tablet version, and tablet can specify a minimum server version. Version numbers are in common/VERSION.js for the tablet.
  • Bugfix: tablet registration crashed if the Patient table hadn’t been created. And similar subsequent bug when uploading with no tables.
  • CAPE-42 task.

Server v1.10, 2014-08-16

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • Database upload script could fail to insert but not complain to the tablet.
  • Stopped database handler (rnc_db.py) masking any exceptions.
  • Improved exception handling in database.py.
  • Bug: patient table incorrectly had forename/surname/DOB fields as NOT NULL. Sex column also now has that constraint removed (enforced elsewhere but one could envisage not enforcing it).
  • Tablet-side (webclient) minor fix to newline escaping.
  • Removed Unicode from error messages in make_summary_tables(), since they also go to the Apache log.
  • Bugfix: login failures were redirecting to the page for acknowledging terms and conditions. Bug was in login().
  • Bugfix: effective deadlock between the process of a mandatory password change for new users and acknowledging terms/conditions.
  • Make database/username more prominent (bold) in menus. Was easy to ignore.
  • pyflakes compliance.

Server v1.11, 2014-09-06

  • Future necessity to discriminate field types that all use VARCHAR; e.g. (and esp.) ISO-8601 dates versus others. So change sqltype to cctype internally; see cc_db.add_sqltype().
  • Significant simplification of work done in tasks with ancillary tables. New cc_task.Ancillary class; q.v.
  • Export to CRIS staging database and autocreate draft data dictionary.

Server v1.12, 2014-09-11

  • REQUIRES TABLET CLIENT V1.12.
  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • Renamed ExpDetThreshold/ExpectationDetection tasks (and tables) to add a “[C/c]ardinal_” prefix, as the names were too vague. THEREFORE requires tablet version 1.12 as well.
  • Session-based authentication for tablets to improve speed (i.e. no need for bcrypt reauthentication within the same session, as for the web front end).

Server v1.13, 2014-10-02

  • Trivial code changes.

Server v1.14, 2014-10-15

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • REQUIRES TABLET CLIENT V1.14.
  • Server can enforce a minimum tablet version, and tablet can specify a minimum server version. Version numbers are in cc_version.py on the server.
  • CAPE-42 task.

Client v1.15, 2014-10-18

  • Requires server version 1.15.

  • NHS numbers were being corrupted, i.e. very long (10-digit) numbers.

    • Critical error. Stored correctly in database.

    • SQLite maximum integer is 2^63 - 1 = 9,223,372,036,854,775,807.

    • Javascript safe max is 9,007,199,254,740,991.

    • A valid database was read incorrectly by dbsqlite.js / getAllRows().

    • Ah. Titanium bug: https://jira.appcelerator.org/browse/TIMOB-3050

    • Workaround is either

      (a) float, which won’t be quoted by the SQLite quote() function, and which MySQL will happily accept (rounding); and all numbers are floats anyway in Javascript;

      or

      (b) text, with parseInt() when reading from SQLite to Javascript. This will send integer values quoted, but MySQL will convert even e.g. ‘9876543209.999’ (with the quotes) to 9876543210 when inserted into a BIGINT field, so that’s OK. The parseInt() function will truncate, which is also fine.

      I guess float is slightly more logical. Let’s be quite clear: in Javascript, all numbers are floats; they are 64-bit floating point values, the largest safe exact integer is Number.MAX_SAFE_INTEGER, or 9007199254740991.

    • So:

  • QuestionTypedVariables applies +/- Number.MAX_SAFE_INTEGER when no other limits are specified (in getValidatedInt).

  • No negative ID numbers (in Patient.js).

  • Changed columnDefSQL() in dbsqlite.js to use REAL for DBCONSTANTS.TYPE_INTEGER and DBCONSTANTS.TYPE_BLOBID. No value conversion is required.

  • Equivalent change in fieldTypeMatches().

  • Removed AUTOINCREMENT tag from PKs (SQLite behaviour doesn’t require this).

  • Added changeColumnTypes() function.

  • Database upgrade changes type of patient ID numbers in patient table.

  • On the server (MySQL) side, the fields were

    • INT: -2,147,483,648 to 2,147,483,647 or 4,294,967,295 unsigned (4-byte)
    • and need to be
    • BIGINT: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 or 18,446,744,073,709,551,615 unsigned (8-byte)

Server v1.15, 2014-10-20

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • ID number fields become unsigned BIGINT, not unsigned INT. Fixes critical error (inability to represent NHS numbers.) See VERSION_TRACKER.txt for the tablet software.

Client v1.16, 2014-10-26

  • Text-as-button widgets:
    • QuestionBooleanText / props.asTextButton
    • QuestionMultipleResponse / props.asTextButton
    • QuestionMCQ / props.asTextButton
  • Reworking of corresponding underlying widget code.
  • QuestionDateTime supports text entry (including by default).
  • Updated moment.js to 2.8.3
  • Minor other code changes and improvement of demo questionnaire.

Server v1.17, 2014-11-12

  • HAM-D: complained inappropriately about ‘3’ codes (meaning ‘not measured’) for weight questions; maximum score adjusted accordingly from 53 to 52; comment for Q16B was erroneously labelled Q16A.

Client v1.17, 2014-11-13

  • HAM-D scoring was wrong for “weight - not measured” option. Fixed. Maximum changed from 53 to 52 accordingly.

Client v1.2, 2014-11-27

  • Requires server version 1.2.
  • WEMWBS/SWEMWBS scales.
  • QuestionMCQGrid wasn’t centring its buttons properly, because McqGroup wasn’t copying its incoming tipropsArray through properly.
  • Bugfix to webclient database handling, in:
    • dbwebclient.js / convertResponseToRecordList()
    • netcore.js / parseServerReply()
  • Some improvements to MobileWeb, though Titanium bugs remain, e.g.:
  • GAF: applies 0-100 input constraint.
  • GAF: interprets raw score of zero as “unknown” for total-score purposes.

Server v1.2, 2014-11-28

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • WEMWBS/SWEMWBS tasks.
  • GAF: interprets raw score of zero as “unknown” for total-score purposes.
  • CGI: requires full completion for a valid total score.
  • BPRS total score was erroneously including Q19/Q20.
  • Scoring clarity expanded (e.g. BPRS, BPRS-E, CGI).
  • Exclude manually erased tasks from list (unless “include old versions” is selected). See
    • cc_task.get_session_candidate_task_pks_whencreated()
    • cc_task.get_all_current_pks()
  • Bugfix to cc_task.make_extra_summary_tables().

Server v1.21, 2014-12-04

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • Draft support for RiO metadata export (for RiO’s batch document upload facility). Some information pending, e.g. whether UTF-8 is supported in metadata.

Client v1.21, 2014-12-26

  • Fixes bug found in v1.17. Symptom: crash after adding new patient in some circumstances (?when ID check failed). Error of “‘undefined’ is not an object (evaluating ‘this.props.pages[this.currentPage].pageTag’) at Questionnaire.js (line 1)” Added getPageTag() function to check for invalid index effects.

  • Note in passing: to view iPad-based SQLite files, copy them elsewhere with e.g. http://www.macroplant.com/iexplorer/

  • Curious crash on loading on an iPad whereas fine under the iOS simulator. Occurring in

    • dbinit.js
    • storedvars.databaseVersion.setValue(…)
    • this.dbstore()
    • dbcommon.storeRow()
    • dbsqlite.updateByPK()
    • dbsqlite.getFieldValueArgs()

    Segmentation fault (view console with Xcore > Window > Devices > click the tiny up-arrow at the bottom left of the right-hand pane for the device). Titanium SDK: 3.5.0.Alpha http://builds.appcelerator.com.s3.amazonaws.com/index.html

    … upgraded to 3.5.0.RC (install SDK + change tiapp.xml)

    … fixed. So a Titanium bug.

15.2.3. 2015

Server v1.22, 2015-01-07

  • Improved audit search.

Client v1.30, 2015-01-30

  • Requires server version 1.3.
  • IDED3D task.
  • Bug related to serialization of moment() objects from webviews. Probably introduced in v1.16. The moment.js library now includes a moment.toJSON() function, which overrides custom work in my json_encoder_replacer() function. However, moment.js’s version loses information (specifically, time zone, not to mention that it’s hard as the recipient to detect whether the object should be reconverted to a moment() object.) Therefore: preprocess_moments_for_json_stringify() … in conversion.js and taskhtmlcommon.jsx.
  • Alerts with large content no longer scroll under iOS 8. Apparently this is an Apple bug: https://jira.appcelerator.org/browse/TIMOB-17745
  • Raphael.js upgraded from 2.1.0 to 2.1.3.
  • Bugfix: if endUpload() failed, the failure wasn’t processed properly.

Server v1.30, 2015-01-30

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • IDED3D task.
  • Cardinal_ExpectationDetection and Cardinal_ExpDetThreshold: ISO-8601 fields changed from TEXT to (internal) ISO8601 (i.e. SQL VARCHAR).
  • Prohibit manual erasure of non-finalized (live-on-tablet) tasks (for one thing, the tablet might re-upload and surprise the erasing user).
  • Manually erased records become non-current.
  • Fix latent bug by finalizing special notes along with their tasks.
  • Forcible finalization/preservation, with _forcibly_preserved flag.
  • Option to drop superfluous columns when remaking tables.
  • Bugfix: other filters failed if non-current tasks shown (get_session_candidate_task_pks_whencreated).

Client v1.31, 2015-02-10

  • Requires server version 1.31.
  • dbsqlite.renameColumns() and dbsqlite.changeColumnTypes() fail more politely with non-existing columns (remember that not all tables may exist, even if the app has been launched before, so don’t throw an error).
  • IDED3D: Minor config text bugfix.
  • IDED3D: Save stimulus shapes to database as SVG.
  • IDED3D: Occasional missing sounds. Reaches “playsound: filename =” message. I suspect this is a Titanium bug, but am not certain.
  • IDED3D: Correct/incorrect sounds changed to more distinctive chords with more similar subjective volumes.
  • IDED3D: Change colours for the colour-blind? A/w Annette.

Server v1.31, 2015-02-10

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • IDED3D task: extra field to store shapes (ided3d.shape_definitions_svg).
  • Patient deletion reports tasks that will be deleted.
  • Ability to edit patient details, for finalized records.
  • HL7 resending triggered by cancelling, not deleting, existing messages (in cc_task.Task.delete_from_hl7_message_log(), etc.)

Server v1.32, 2015-02-15

  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • Enforces sensible MySQL engine settings.
  • Switches tables to Barracuda format to avoid uploading bug when rows too large.

Server v1.33, 2015-02-19

  • Tweaks to RiO metadata export, based on feedback from Servelec.

Server v1.34, 2015-03-01

  • Long text (e.g. ProgressNote) crashed PDF generator when in a table. Tasks prone to this (ProgressNote, PsychiatricClerking) reworked to avoid tables.
  • Bug in RecipientDefinition.report_error() fixed.

Client v1.32, 2015-03-10 to 2015-04-22

  • setRemoteBackup(false) call, to disable back to Apple iCloud; see dbinit.js
  • Intermittent crash on Android 4.4.4 (build 23.0.1.A.4.30). Relates to database access?
    • Always create all tables at task start. (A crash due to a missing table was still possible, and the kind of thing it’s easy to miss on a development machine that tends to have everything precreated. Mind you, not sure that was the actual bug; see next point.) See ensure_database_ok().
    • Explicitly close all recordsets (cursors) opened on all db.execute() operations.
    • Did not relate to database access in 10k soak test, and crash occurred outside updateByPK function. Maybe relating to visual display. Key error:
      • E/BufferQueue( 292): [org.camcops.camcops/org.appcelerator.titanium.TiActivity] dequeueBuffer: can’t dequeue multiple buffers without setting the buffer count
    • This? https://code.google.com/p/android/issues/detail?id=63738 Android source is: https://android.googlesource.com/platform/frameworks/native/+/jb-dev/libs/gui/BufferQueue.cpp But crash also occurred inside updateByPK function (unless from a different thread).
    • No… relates to setBackgroundImage() calls. - https://jira.appcelerator.org/browse/TC-5369
    • Attempt at change:
      • Get rid of all setBackgroundImage() calls for situations calling for multiple alternative images (e.g. radio buttons). Also setBackgroundSelectedImage().
      • Replace with method of loading all alternative images at the start, and using hide()/show() calls.
      • Affects ValidityIndicator; StateRadio; StateCheck.
      • setImage() calls also removed from ImageGroupVertical.
      • Residual setImage() calls, which may also be suspect if the Android file system is duff:
        • QuestionCanvas_*
        • QuestionImage
        • QuestionPhoto
      • NOT successful. If anything, crashes more frequent. Therefore, most likely a memory problem? E.g. ACE-III “learning address” page: 26 x QuestionBooleanText, each with up to 4 potential images loaded, each ~3k on disk, would give 312k (when image caching would reduce that to 12k); might be larger in memory, and if the “imageref_ashmem create failed” message is showing the size – which it is; see https://code.google.com/p/skia/source/browse/trunk/src/images/SkImageRef_ashmem.cpp?spec=svn11558&r=11558 … then it’s about 36k per image, i.e. we were using 3.7 Mb for that page. That’s then perhaps less surprising.
    • Reverted.
    • New technique
      • imagecache.js
      • Cache cleared from questionnaire.js
      • Applied to ValidityIndicator, StateRadio, StateCheck … except you can’t pass Blobs to Titanium.UI.createButton, only to createImageView … so ImageView used instead of button for now (which loses the “currently being touched” facility). See AS_BUTTONS flag in qcommon.js.
      • However, ImageVerticalGroup goes to preloading method for performance reasons.
  • Allow user to specify the number of lines used for fixed-height multiline text entry: multilineDefaultNLines.

Client v1.33, 2015-04-26

  • Bugfix: CGI didn’t offer all options for Question 3 (drug effects)!

Client v1.34, 2015-04-26

  • Probable bugfix: IDED3D performed its stage failure check before its stage success check at the end of trials (should be the other way round).

Client v1.40, 2015-05-27

  • Requires server version 1.40.
  • FROM-LP framework set menu
  • O’Brien group set menu 1
  • Brief COPE
  • CBI-R
  • ZBI (data collection tool only with option for institution to supply text)
  • HADS (data collection tool only with option for institution to supply text)
  • AUDIT-C
  • CGI-I
  • Patient Satisfaction Scale
  • Referrer Satisfaction Scale (generic + specific)
  • Friends and Family Test
  • IRAC
  • MDS-UPDRS (data collection tool only)
  • GDS-15
  • AUDIT and AUDIT-C corrected to be clinician-colour pages, and instruction page added.
  • extrastrings framework - at registration, the tablet downloads sets of extra strings from its server. This allows the conversion of crippled tasks to fully-functional ones, subject to the hosting institution’s right to offer the strings up to its tablets (which is a matter for the institution, the strings not being distributed with CamCOPS).
  • clinician_service field as part of clinician block (and used for service feedback); corresponding storedvars.defaultClinicianService variable.
  • boldPrompt option to QuestionTypedVariables
  • editing_time_s field as standard on all tasks

15.2.4. 2016

Server v1.40, 2016-01-28

  • From May 2015 to 28 Jan 2016.
  • REQUIRES COMMAND TO UPGRADE EACH DATABASE: camcops –maketables /etc/camcops/MYCONFIGNAME.conf
  • NOTE THAT THE camcops_meta command is now available, e.g. camcops_meta –filespecs /etc/camcops/camcops_*.conf –ccargs maketables
  • Brief COPE Inventory.
  • CBI-R.
  • ZBI (data collection tool only with option for institution to supply text).
  • HADS (data collection tool only with option for institution to supply text).
  • AUDIT-C
  • CGI-I
  • Patient Satisfaction Scale
  • Referrer Satisfaction Scale (generic + specific)
  • Friends and Family Test
  • IRAC
  • MDS-UPDRS (data collection tool only)
  • GDS-15
  • DEMQOL
  • DEMQOL-Proxy
  • Default “respondent” framework, for DEMQOL-Proxy.
  • Bugfix to ProgressNote: get_task_html() crashed because “answer” was not imported.
  • EXTRA_STRING_FILES system, with “get_extra_strings” command to database API.
  • PHQ-9 database comment fixed for Q10.
  • comment_fmt for HADS fields. Note MySQL: SHOW FULL COLUMNS FROM table.
  • IES-R (skeleton only).
  • WSAS (skeleton only).
  • PDSS (skeleton only).
  • PSWQ.
  • Y-BOCS, Y-BOCS-SC (skeleton only).
  • DAD (skeleton only).
  • BADLS (skeleton only).
  • NPI-Q (skeleton only).
  • FRS.
  • INECO Frontal Screening (IFS) (skeleton only).
  • Add clinician to GAF.
  • Diagnosis reports.
  • Device report.
  • update_multiple_databases.py script
  • Unit tests to ensure no overlap for task longnames/shortnames/tables; see cc_task.unit_tests().
  • clinician_service field as part of clinician block
  • xhtml2pdf @page size changed from “a4” to “A4” in cc_html.py to remove “WARNING:xhtml2pdf:Unknown size value for @page”; see https://github.com/chrisglass/xhtml2pdf/issues/71 … however, no effect.
  • Switch from xhtml2pdf, bypassing Weasyprint, to wkhtmltopdf (via pdfkit) as the (default and now only) PDF renderer.
  • Abstract base class for PCL tasks wasn’t inheriting from object; now is.
  • editing_time_s field for all tasks.
  • Indexing of ID number fields in patient table.
  • Python package format internally.
  • Did not implement SVG logos for PDF generation; made files larger not smaller. Stick with PNG.
  • Remove delayed imports; bug-prone. http://stackoverflow.com/questions/744373 Except in cc_hl7, which imports phq9 for testing (which imports specific things from cc_task, which imports cc_hl7). … subsequently revisited; delayed imports now remain only for unit tests, where they are more convenient.
  • Refresh button for tasks (because browsers keep asking you twice if you hit F5).
  • EXTRA_STRING_FILES can use globs (in cc_string.py).
  • Support MPLCONFIGDIR (default: /var/cache/camcops/matplotlib) to speed up matplotlib/pyplot loading.
  • Updated to current pythonlib.
  • Python build toolchain.
  • Moved to Python 3.
    • Of note: comparison of None to int now fails.
  • Supplied with Gunicorn, to enable front-end web servers like Apache to talk to CamCOPS, and have CamCOPS upgrade/restart, without having to (a) restart Apache, or (b) integrate a specific Python version into Apache with mod_wsgi. The new system runs in a virtual environment, entirely separated (in terms of code) from the front-end web server, communicating with it via a private port or Unix socket.
  • Disable HTTP client-side caching for added security. See also http://codebutler.github.io/firesheep
  • Change to relative URL addressing to make that work simply (without having to tell CamCOPS where it’s mounted).
  • ALLOW_INSECURE_COOKIES debugging option.
  • Fix nasty bug in rnc_web using “extraheaders=[]” in function signature, allowing headers (e.g. cookies) to accumulate over multiple calls (and leak across clients). 2016-01-09. (But note what is NOT a bug: multiple Chrome “incognito” tabs share each other’s cookies: https://code.google.com/p/chromium/issues/detail?id=24690)
  • Removed the “Tablet device” filter option for tasks (it generates long complex-looking lists of IDs and isn’t helpful for end users). Removal done simply by taking the option out of the form, in cc_session.get_current_filter_html().
  • New server environment variable options (see instructions.txt): - MPLCONFIGDIR - CAMCOPS_DEBUG_TO_HTTP_CLIENT - CAMCOPS_PROFILE - CAMCOPS_SERVE_STATIC_FILES
  • Changes to config variables: - RESOURCES_DIRECTORY – removed - INTROSPECTION_DIRECTORY – removed - CAMCOPS_LOGO_FILE_ABSOLUTE – added (optional) - MAIN_STRING_FILE – added (optional) - EXTRA_STRING_FILES – added (optional)
  • Added process ID to log output.
  • Task counting report.
  • Restructure Task/Ancillary classes to be more concise.
  • Better unit testing inc. checking for __dict__/fieldname conflicts.
  • camcops_meta.py script for e.g. upgrading multiple databases.
  • PyMySQL==0.7.1 (upgraded from PyMySQL==0.6.7) to eliminate error on inserting BLOBs (“TypeError: can’t use a string pattern on a bytes-like object”).

Server v1.41, 2016-01-29

  • Bugfix to large research data dumps (were timing out due to inefficient SQL). Changed cc_task.get_ancillary_items(), with some back-end functions in rnc_db too (changed fetch_all_objects_from_db_where(); added create_object_from_list() ).

Server v1.50, 2016-07-29

  • Change _device VARCHAR(255) fields to _device_id INT.
  • Change *_user VARCHAR(255) fields to *_user_id INT.
  • Note that this leaves only a few “odd” things from the point of standard RDBMSs:
    • multiple keys on server side (adding _device_id and _era) to reflect multiple devices with write-only access;
    • history on server side (adding _current, and forward/backward PK chain)
    • the “_era” field is textual (ISO-8601), because (a) no database seems to store DATETIME values with milli-/microsecond accuracy and proper timezone information (in the sense that you can recreate the timezone of origin); (b) we can use a non-NULL special value, in our case “NOW”, as it makes things simpler for end users to use “a = b” consistently and not have to do “a = b OR a IS NULL AND b IS NULL”.
    • patient IDs are unchecked and are allowed to be incomplete (to reflect our need to operate with incomplete information, and in anonymous as well as fully-identified environments), and duplicate patient records are allowed (across, but not within, device/era combinations).
  • Static type checking for server Python code.

Known problems and bugs at the end of the Titanium client

  • visually disabled elements not yet implemented (starting point only in qcommon.js)
  • wait class imperfect and may leak
  • ti.imagefactory module does not support x86 architecture, just arm
  • Titanium iOS re-layout is very slow. Visible e.g. when changing questionnaire font sizes, but more important for multiline multiline text areas. Bug report: https://jira.appcelerator.org/browse/TC-3560
  • mobileweb edition not yet working
  • Alerts with large content no longer scroll under iOS 8. Apparently this is an Apple bug: https://jira.appcelerator.org/browse/TIMOB-17745

Where were version numbers stored in the Titanium client?

  1. App version number is stored in tiapp.xml
  2. Tablet’s minimum server version requirement is in Resources/common/VERSION.js
  3. Server version number is stored in server/cc_modules/cc_version.py (as is the server’s minimum tablet version requirement).
  4. Server changelog is stored in server/changelog.debian
  5. The web page also has a record of the most recent version, in download/index.html

Indirectly:

  • Tablet app: Resources/common/VERSION.js reads the app version using Titanium.App.version, which is determined by tiapp.xml. In turn, it exports this as CAMCOPS_VERSION.
  • Tablet build: SHIP_ANDROID reads VERSION.js
  • Server build: MAKE_PACKAGE reads cc_constants.py

Human-readable details are shown in this file.

15.3. Current C++/SQLite client, Python/SQLAlchemy server

15.3.1. 2017

Client v2.0.0 beta

  • Development of C++ version from scratch. Replaces Titanium version.
  • Released as beta to Google Play on 2017-07-17.

Client v2.0.1 beta

  • More const checking.
  • Bugfix to stone/pound/ounce conversion.
  • Bugfix to raw SQL dump.
  • ID numbers generalized so you can have >8 (= table structure change).

Client v2.0.2 beta

  • Cosmetic bug fixes, mainly for phones, including a re-layout of the ACE-III address learning for very small screens.
  • Bugfix: deleting a patient didn’t deselect that patient.
  • Default software keyboard for date entry changed.
  • Bugfix for canvas widget on Android (size was going wrong).
  • Automatic adjustment for high-DPI screens as standard in QuBoolean (its image option), QuCanvas, QuImage, QuThermometer.

Client v2.0.3 beta, 2017-08-07

  • Trivial type fix to patient_wanted_copy_of_letter (String → Bool) in the unused task CPFTLPSDischarge.

Server v2.1.0 beta, 2017-10-17

  • Major changes, including…
  • SQLAlchemy for database work
  • Group concept
  • HOWEVER, HL7 EXPORT AND ANONYMOUS STAGING DATABASE SUPPORT DISABLED; further release pending.

Client v2.0.4 beta, 2017-10-22

  • Bugfix: BLOB FKs weren’t being set properly from BlobFieldRef helper functions.

Client v2.0.5 beta, 2017-10-23

  • Bugfix: if the server’s ID number definitions were consecutively numbered, the client got confused and renumbered them from 1.

Server v2.1.1 beta, 2017-10-23

  • Bugfix: WSAS “is complete?” flag failed to recognize the “retired or work irrelevant for other reasons” flag.

15.3.2. 2018

Client v2.2.0 beta, 2018-01-04 to 2018-02-03

  • To solve the problem of clients and servers being upgraded independently: Reads tables from server during registration (see server v2.2.0). Implemented a “minimum server version” option internally for each task (see contemporary server changelog). Minimum server version increased from v2.0.0 to v2.2.0.
  • Bugfix: adding a new patient from a task list didn’t wipe the task list until the patient was re-changed (failure to call setSelectedPatient from ChoosePatientMenu::addPatient; in fact, the patient name details changed without changing the underlying patient selection).
  • Bugfix: don’t think the patient ID number table was being made routinely (!?).
  • New CIS-R task.
  • Internal fix to DynamicQuestionnaire to defer first-page creation until after constructor.
  • Menu additions for CPFT Affective Disorders Research Database.

Server v2.2.0, 2018-01-04 to 2018-04-24

  • To solve the problem of clients and servers being upgraded independently: Maintains a minimum client (tablet) version per task; during registration, offers the client the list of its tables and the minimum number. This allows a newer client to recognize that the server is older and has ‘missing’ tables, and act accordingly. See camcops_server.cc_modules.client_api.ensure_valid_table_name(). Minimum tablet version remains v1.14.0.
  • An obvious question: with that mechanism in place, is there any merit to the client maintaining a list of minimum server versions for each task? The change to the client’s “minimum server version” to 2.2.0 (for client v2.2.0) means that future clients will always have the “supported versions” information from the server. So, might a client advance mean that it might want to refuse old versions of the server, even though the server might be happy to accept? (That’s the only situation when a client’s per-table minimum server version would come into play.) Well, perhaps it’s possible, even if it’s very unlikely (and would probably indicate bad backwards compatibility on the client’s part! Let’s implement it for symmetry. Actually, thinking further, it might be quite useful: if you upgrade a task and add extra fields, using this would potentially allow the client to work with older servers unless a specific task is used. Implemented; see client changelog above. The default for all tasks is the client-wide minimum server version.
  • New report to find patients by ICD-10 or ICD-9-CM diagnosis (inclusion and exclusion diagnoses) and age.
  • Bugfix where reports would only be produced in HTML format.
  • New CIS-R task.

Server v2.2.1, 2018-04-24 to 2018-06-11

  • Username added to login audit.
  • SQLAlchemy Engine scope fixed (was per-request; that was wrong and caused ‘Too many connections’ errors; now per URL across the application, as it should be; see cc_config.py).
  • Links to de-identified versions of tasks.
  • Group administrators can now change passwords for other users in their group, as long as the other user isn’t a groupadmin or superuser.
  • A released (CPFT) version of 2.2.0 raised a “The resource could not be found” error when using the /view_groups URL, heading to groups_view.mako.
    • Initially: not sure why; development version works fine. No files obviously missing. Only that page not working, of all the main menu pages. This was as the superuser. The problem was an exception being raised from the template.render_unicode() call in CamcopsMakoLookupTemplateRenderer.__call__. Aha – problem may have been a completely full disk. No; disk was completely full, but that wasn’t the problem.
    • v2.2.1 released just in case I’d missed something.
    • No, it was a problem manifesting in groups_table.mako, which used u.username for u in group.users giving AttributeError: 'NoneType' object has no attribute 'username'. Now, that is defined in Group as users = association_proxy("user_group_memberships", "user").
    • The problems looks to be in the data: there was an entry in the _security_user_group table with user_id = NULL (and group_id = 3).
    • Not yet sure where that duff value came from. Template updated to cope with the problem, regardless. (Perhaps the value came from an earlier version of merge_db.py?)

Server v2.2.2, 2018-06-19

Client v2.2.1 beta, 2018-08-06

  • Background striping for the QuMcqGrid* classes.
  • Bugfix: android:allowBackup=”false” added back to AndroidManifest.xml

Client v2.2.3, server v2.2.3, 2018-06-23

  • Khandaker/Insight medical history task.
  • Client requires server v2.2.3. (Was a global requirement; should have been task-specific. REVERTED to minimum server version 2.2.0 in client 2.2.6.)

Client v2.2.4, 2018-07-18

  • Bugfix to Android client for older Android versions.
    • On startup, CamCOPS was crashing with “Unfortunately, CamCOPS has stopped.” on older Android versions (e.g. 4.4.x).
    • The USB debugging stream showed: java.lang.UnsatisfiedLinkError: dlopen failed: could not load library "libcrypto.so.1.1" needed by "libcamcops.so"; caused by library "libcrypto.so.1.1" not found.
    • Thoughts: see comments in changelog.rst.
    • Upshot: Android API 19 (Android 4.4.x) no longer works. API level 23 (Android 6.0.1) is fine; intermediates untested. It’s a little unclear what’s changed (unless I was just behind on testing for old versions of Android and the problem had been there for a while). One possibility was that the shared OpenSSL libraries were being compiled for android-23 (as per build_qt.py) and that was not the same as minSdkVersion in AndroidManifest.xml. The problems are explained well at https://stackoverflow.com/questions/21888052/what-is-the-relation-between-app-platform-androidminsdkversion-and-androidtar/41079462#41079462, where APP_PLATFORM is equivalent to the API version used by build_qt.py to compile OpenSSL etc.
    • The upshot from that article is that libraries compiled with the Android NDK (like OpenSSL in our case) must be compiled with for the same SDK version (APP_PLATFORM) as minSdkVersion.
    • We were using minSdkVersion="16", so I tried setting DEFAULT_ANDROID_API_NUM = 16 in build_qt.py, and recompiling for Android using build_qt.py --build_android_arm_v7_32, continuing to use NDK r11c. I moved targetSdkVersion back to 26 (soon to be the Google Play minimum). This works on Android 6.0.1 (API 23, using debug mode). However, it still crashes (as above) with Android 4.4.x (API 18). As of Feb 2018, about 58% of Android in the wild is API 23 or higher (https://en.wikipedia.org/wiki/Android_version_history), and about 82% is API 21 and higher. It is certainly better to fail to run than to crash, so let’s say that we will set API 23 (Android 6.0) as the minimum for now.

Server v2.2.4, 2018-06-29

  • Update to libraries:
    • alembic from 0.9.6 to 0.9.9
    • cardinal_pythonlib from 1.0.16 to 1.0.18
    • colorlog from 3.1.0 to 3.1.4
    • CherryPy from 11.0.0 to 16.0.2
    • deform from 2.0.4 to 2.0.5
    • distro from 1.0.4 to 1.3.0
    • dogpile.cache from 0.6.4 to 0.6.6
    • gunicorn from 19.7.1 to 19.8.1
    • matplotlib from 2.1.0 to 2.2.0
    • mysqlclient from 1.3.12 to 1.3.13
    • numpy from 1.13.3 to 1.14.5
    • pendulum from 1.3.0 to 2.0.2
    • pyramid from 1.9.1 to 1.9.2
    • pyramid_debugtoolbar from 4.3 to 4.4
    • python-dateutil from 2.6.1 to 2.7.3
    • pytz from 2017.2 to 2018.5
    • scipy from 1.0.0rc1 to 1.1.0
    • sqlalchemy from 1.2.0b2 to 1.2.8
    • typing from 3.6.2 to 3.6.4
  • Bugfix to SQLAlchemy/Alembic handling, such that tables are always created with CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci rather than the erroneous COLLATE utf8mb4_unicode_ci CHARSET utf8mb4. See MySQL: Illegal mix of collations.

Server v2.2.5, 2018-07-23

  • Python package: camcops-server.

Server and client v2.2.6, 2018-07-26

  • Logic bugfix and improved clarity in client Task::isTaskUploadable.
  • Client minimum server version returned to 2.2.0 (from 2.2.3); specific Khandaker1MedicalHistory requirement of 2.2.3 added.
  • Fixed inadvertently broken server: the upgrade_db command defaulted to showing SQL only, not doing the job!
  • BDI shows alert for non-zero suicidality question.
  • BDI shows custom somatic symptom score (Khandaker Insight study) for BDI-II.
  • BDI shows question topics (taken from freely available published work), though no task content is present.
  • Added missing server string camcops/data_collection_only.
  • CssClass constants.
  • CISR client now shows more detail in its summary view.
  • Bugfix to CISR client logic; code fallthrough for CONC3_CONC_PREVENTED_ACTIVITIES.
  • Client returns to maximized mode after returning from fullscreen, if it was maximized before.
  • Client calls ensurePolished() for sizeHint() functions of widgets containing text, which should make initial sizing more accurate.
  • Fix to fullscreen modes under Windows (see compilation_windows.txt).
  • Whisker test task (2018-08-15).
  • Windows distribution (2018-08-16).

Server v2.2.7, 2018-07-31

  • Bugfix relating to offset-naive versus offset-aware datetimes in cc_user.SecurityLoginFailure.clear_dummy_login_failures_if_necessary.

Client v2.2.7, 2018-08-17

  • Bugfix to CISR: field sleep_gain3 was missing from field list.
  • Search facility for all-tasks list.
  • OS information.

Client v2.2.8 to 2.3.0 (from 2018-09-10)

  • Bugfix to CISR client: page colour was clinician, now patient.

  • Bugfix to PHQ9: question 10 was still mandatory in the Questionnaire even if zero score for other questions.

  • Client moved from Qt 5.11.1 to Qt 5.12.0 (2018-09-24).

    • Code changes: tablet_qt/layouts/flowlayout.cpp temporarily switches off -Werror=missing-field-initializer warning which arises from qcborvalue.h when including #include <QtWidgets>; this is https://bugreports.qt.io/browse/QTBUG-68889. The compiler was g++ from GCC 4.9, part of Android NDK r11c. We disable with #pragma GCC diagnostic ignored "-Wmissing-field-initializer"
    • Checked for Linux, Android; Windows checks pending.
  • “Page jump” button only shown in questionnaires if (allowed and) there is more than one page, or the questionnaire is dynamic.

  • New variant on QuBoolean/BooleanWidget to display “false as blank”. Used in FACT-G task.

  • QuPage::indexTitle() for different titles (if desired) on the page jump index to the heading at the top of the page.

  • Markedly improved error messages when you aim the client at a web server but not the CamCOPS client API.

  • Rounding of DPI prior to icon sizing (we were using e.g. 96.0126 DPI, which is probably the system reporting inaccurately).

  • ID number validation system and NHS number validation.

  • Removed all defunct preprocessor references to LIMIT_TO_8_IDNUMS_AND_USE_PATIENT_TABLE, DUPLICATE_ID_DESCRIPTIONS_INTO_PATIENT_TABLE, and ALLOW_SEND_ANALYTICS.

  • ID policy supports “NOT” and other new tokens; see server changelog.

    If an old client is used with a new server, the server may offer “invalid” policies (as seen by the old client); these will refuse uploads, as per IdPolicy::complies(). If a new client is used with an old server, there should be no problem.

  • CardinalExpDetThreshold was missing ancillaryTables() and ancillaryTableFKToTaskFieldname().

  • Turn off patient ID information in debug stream for MenuItem,

  • Add network status messages to debug stream.

  • New task: CORE-10.

  • New task: CESD.

  • New task: CESD-R.

  • New task: PTSD Checklist for DSM-5 (PCL-5).

  • New task: FACT-G.

  • New task: EQ-5D-5L.

  • Client validates patients with the server on upload. This supports future “predefined patient” support. This is a “client asks”, not “server tells” feature at present.

  • Version bumped to 2.3.0. If server is at least 2.3.0, uses the new “validate patients on upload” feature (2018-11-13). Minimum server version remains at 2.2.0.

  • Word wrap on for log box by default (better legibility in upload).

  • Since the server can now report PID when providing error messages (patients that don’t validate), the “upload” function is now restricted to unlocked devices.

  • Databases were not being vacuumed (call was being made after database thread had been shut down). Fixed.

  • Fixed bug: patient was not deselected (in NetworkManager::uploadNext()) with a “copy” upload, but that failed to take account of patients/tasks marked as “individually finished”. Now always deselected (also triggers refresh of anonymous task list).

  • ProgressNote now reports itself as incomplete if the note is empty, in addition to if it is NULL. Corresponding change on the server.

  • Bug found in upload process relating to BLOB upload in the “per-record” fashion. Specifically, when the client set the _move_off_tablet flag on a BLOB (in NetworkManager::applyPatientMoveOffTabletFlagsToTasks()), it then asked the server “which records to send?” via camcops_server.cc_modules.client_api.op_which_keys_to_send(). This took account of actual modifications, but not changes to the _move_off_tablet flag; so the record wasn’t resent; so older client BLOBs that were not being modified in that upload were not correctly marked as preserved. Solution: new TabletParam.MOVE_OFF_TABLET_VALUES parameter to this command. Modifications to NetworkManager::requestRecordwisePkPrune() and, on the server, camcops_server.cc_modules.client_api.op_which_keys_to_send(). To make this safe retrospectively, the server insists on all records being sent if this field is not present in the request.

Server v2.2.8 to 2.3.0 (2018-09-14 to 2018-11-26)

  • GROUP_NAME_MAX_LEN, DEVICE_NAME_MAX_LEN and USER_NAME_MAX_LEN changed from 255 to 191 because MySQL<=5.6 only supports indexing of 191 characters in utf8mb4 mode on InnoDB tables; see https://dev.mysql.com/doc/refman/5.7/en/charset-unicode-conversion.html

  • Shebang changed for build_qt.py

  • SQLAlchemy NAMING_CONVENTION changed in cc_sqlalchemy.py as some fields were yielding index/constraint names that were too long… then reverted and specific changes made for cpft_lps_discharge.management_specialling_behavioural_disturbance.

  • Removed introspection options; replaced with better docs.

  • Documentation now at https://camcops.readthedocs.io/.

  • cardinal_pythonlib to 1.0.38

  • alembic to 1.0.0

  • create_database_migration.py checks the database version is OK first.

  • Make Alembic compare MySQL TINYINT(1) to be equal to Boolean() in the metadata, so its default suggestions are more helpful.

  • CherryPy from 16.0.2 to 18.0.1, but this did not fix https://github.com/cherrypy/cherrypy/issues/1618. However, it is a non-fatal error; just carry on.

  • Better server docstrings.

  • All summary tables report the CamCOPS server version that calculated the summary, in the field camcops_server_version.

  • If no extra string files at all are found, the server aborts.

  • Typo fixed in demo Apache config re Unix domain sockets (inappropriately had “https” and a trailing slash).

  • Upload API: improved camcops_server.cc_modules.client_api.upload_record() to use camcops_server.cc_modules.client_api.upload_record_core(), in common with camcops_server.cc_modules.client_api.upload_table().

  • Bugfix to MOCA server display: trail picture was shown twice, clock picture not at all.

  • Probable bugfix to code that handles very old tablet versions, now moved to camcops_server.cc_modules.client_api.process_upload_record_special(). Code was unlikely to trigger; comparison of a Table to a tablename would have failed.

  • ID number validation system and NHS number validation.

  • ID policy supports NOT, address, gp, otherdetails, and otheridnum; see patient identification.

    This makes it easier for research studies to support a “no PID” rule, as a per-group setting.

  • Bugfix to validation colours for groups_table.mako.

  • Group admin facility to list all users’ e-mail addresses with mailto: URLs.

  • Server renamed from camcops to camcops_server; package, executable, etc. Similarly camcops_meta to camcops_server_meta. Note that this may break automatic launch scripts, e.g. via supervisord; you should review these.

  • Added dependency bcrypt==3.1.4 to setup.py.

  • camcops_server.cc_modules.cc_config.CamcopsConfig.get_dbsession_context() re-raises exceptions.

  • Better upgrade/downgrade database facilities for developers.

  • Task report was claiming to slice by creation date but was slicing by addition (upload) date; fixed (to creation date).

  • Task index. See cc_taskindex.py, with corresponding changes in e.g. cc_taskcollection.py and client_api.py. Significant speedup on the server.

    • Design note: we should not have a client-side index that gets uploaded. This would be a bit risky (trusting clients with the server’s index); also, the client’s index couldn’t use server PKs (which we’d want); etc.
  • Upload speedup by optimizing existing upload method and via new one-step upload.

  • Fixed bug where predecessor records of individually-preserved client records were not themselves preserved properly.

  • Documentation of structured upload testing method in client_api.py (q.v.).

  • Improvements to Debian/RPM packaging, including use of venv from the Python 3.3+ standard library rather than virtualenv.

15.3.3. 2019

Server v2.3.1 and client v2.3.1 (2018-11-27 to 2019-03-24)

  • cardinal_pythonlib to 1.0.49.

    • Fixes misconversion of previous 24-hour filter times to their morning equivalents, in the task filter view. To test, set e.g. a start time of 01:30 and an end time of 23:30; save the filter; re-edit the filter and re-save it; check the end time stays correct.
    • Improved e-mail handling, pro tem.
    • For build_qt.py under Windows, implement a directory change via Python and not tar for the “untar” operation.
    • Request-logging middleware.
  • Fixed trivial bugs and added clarity about item sequencing.

    • The bug was: PhotoSequence used zero-based numbering for the seqnum field until something was re-ordered, at which point it went to one-based numbering (renumberPhotos() versus addPhoto()). The server assumed zero-based numbering. Similarly in the diagnosis tasks (renumberItems() versus addItem()).
    • Regardless of the mathematical or computing merits, our experience of research users is that they are more far comfortable with one-based numbering. (Both 0-based and 1-based approaches are clearly possible. A nice essay by van Glabeek, 1999, in support of 1-based numbering is “Do we count from 0 or from 1? The ordinal use of cardinal expressions” at http://kilby.stanford.edu/~rvg/ordinal.html. Part of his point is that the ambiguity arises when we move from an ordinal, “1st”, to a cardinal, “1”. Dijkstra’s more famous 1982 argument for 0-based numbering, at http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html, is pretty weak for this purpose.)
    • So, we will use one-based numbering for sequence numbers of database objects for all future tasks and as the decision for previously inconsistent tasks. We will allow zero-based numbering to persist for older specialist tasks. Changes therefore as follows:
    • photo.py: clarified column comment to make 1-based numbering explicit; HTML display now uses seqnum not seqnum + 1.
    • photosequence.cpp: fixed bug in addPhoto() so it uses 1-based numbering
    • diagnosis.py: clarified column comment to make 1-based numbering explicit; HTML display now uses seqnum not seqnum + 1.
    • diagnosistaskbase.cpp: fixed bug in addItem() so it uses 1-based numbering
    • cardinal_expdetthreshold.py: no change except cosmetically to clarify zero-based trial numbering; not worth changing
    • cardinalexpdetthreshold.cpp: no change; continues with zero-based trial numbering; not worth changing
    • cardinal_expectationdetection.py: no change except cosmetically to clarify zero-based trial numbering; not worth changing
    • cardinalexpectationdetection.cpp: no change; continues with zero-based trial numbering; not worth changing
    • ided3d.py: was already happily using 1-based numbering with clear database/XML comments
    • ided3d.cpp: was already happily using 1-based numbering in the database, and maintaining clearly labelled 0- and 1-based numbering for internal purposes (e.g. ided3dtrial.h; ided3dstage.h).
  • SQL Server support.

  • SNOMED-CT support.

    • For copyright reasons, SNOMED-CT codes for tasks held in a separate file and cross-referenced by arbitrary strings (not the codes themselves).
    • For ICD-9-CM and ICD-10 codes, we preconvert them from an Athena OHDSI data set (if the user is permitted to use that).
    • Command-line options for the client to print its ICD diagnostic codes. These are then added to the server, hugely reducing the number of codes we need to cache (e.g. ICD-9-CM: from 7,911 to 573; ICD-10: from 199,611 to 3,318).
      • Internal design note: creating a DiagnosticCodeSet requires xstrings from the CamcopsApp for its descriptions. Options are therefore (1) defer code printing until the database is open (but that means security checks required!); (2) have all calls to CamcopsApp::xstring() return blanks until the database is open (but then an overhead for everything); (3) have DiagnosticCodeSet not ask for xstrings if it’s being created in “no xstring” mode. Went with (3).
  • Config file documentation moved from demo file to docs.

  • cherrypy from 18.0.1 to 18.1.0, to fix https://github.com/cherrypy/cherrypy/issues/1618; nope, still not fixed. Must be soon…

  • Pygments==2.2.0 to Pygments==2.3.1, in the hope that it fixes some C++ lexing failures that work in the online Pygments demo (http://pygments.org/). Nope…

  • Removed relative imports, as per PEP8 and https://stackoverflow.com/questions/4209641/absolute-vs-explicit-relative-import-of-python-module.

  • Log tidy-up (for delayed evaluation via BraceStyleAdapter).

  • Suppress wkhtmltopdf output with --quiet option; see camcops_server.cc_modules.cc_constants.WKHTMLTOPDF_OPTIONS.

  • CardinalExpDet task complains less when drawing graphs with missing data.

  • Shift to Python csv module for generating TSVs, using the excel-tab dialect. This works well.

  • Bugfix: newlines were not being unescaped properly on receipt from the client (they were remaining in the database as escaped two-character \n strings). Call to unescape_newlines() added to camcops_server.cc_modules.cc_convert.decode_single_value(). This function reverses convert::toSqlLiteral() in the client.

  • Substantially improved export facilities, including whole-database export, push exports, and e-mail exports.

    • Design decision: keep details in config file, or shift to web-based configuration?
      • Unimportant: need to launch from command line. (Could launch via a named database record.) Not a factor.
      • Less important: more work in writing/managing web-based configuration. Fractional point for config file.
      • Relevant: configuration on the fly? This is dangerous but could be useful. However, the only thing likely to be edited is a destination e-mail address. In general, export configuration feels like a slightly high-risk thing to have editable only, even by a web-based superuser; for example, it allows the specification of arbitrary shell scripts, and if this were done online, the editing user might be unable to check that filename. (Moreover, an editing user who is using ssh might find it inconvenient to use a web interface.) Point for config file.
      • Relevant: audit trail. We want the export log to be able to refer to an export config. We probably want a true record of the export config used. However, we don’t want to duplicate thousands of config records (e.g. the same config being run once per minute for years).
        • Solution for database: create a new record when an export config is changed; have run records refer to them by PK.
        • Solution for config file: create a new record when an export config is changed… have run records refer to them by PK…
      • Decision: config file, with snapshot copied to database for auditing.
    • Breaking changes:
      • [server] config file section renamed to [site].
      • Python web server options moved from command-line to config file, in a new [server] section.
      • [recipients] config file section renamed [export]
      • HL7_LOCKFILE changed to a broader EXPORT_LOCKDIR system and moved from the [server] to the [export] section
      • Then other changes to the actual export definitions (see docs for the server config file), each in sections named [recipient:XXX] in the config file.
      • Database drops old HL7-specific tables and adds a new set of export tables (also: more extensible for future methods).
  • QuSlider takes a new setSymmetric() option to remove the colour to the left of (horizontal) or below (vertical) the slider “handle”.

  • Questionnaire takes QuPage* as well as QuPagePtr as arguments to its constructor.

  • TickSlider and QuSlider allow their labels to overspill the edges and therefore work much better.

  • QuElement supports an alignment parameter and all layouts (e.g. QuPage, QuFlowContainer) respect this and sometimes add additional options.

  • Bugfix regarding Alembic.

    • alembic==1.0.0 to alembic==1.0.7, in the hope it gets constraint names right. Made no difference, so onwards:
    • The problem materializes when MySQL’s 64-character limit on constraints (the same as as for other identifiers) is exceeded.
    • A prototypical problem is the table cpft_lps_discharge and its field management_specialling_behavioural_disturbance, defined as a CamCOPS BoolColumn("management_specialling_behavioural_disturbance", constraint_name="ck_cpft_lps_discharge_msbd").
    • The camcops_server.cc_modules.cc_sqla_coltypes.BoolColumn class sets its type_ parameter, effectively, to Boolean(name=conv(NAME_PASSED)).
    • The conv() function, which is sqlalchemy.sql.elements.conv(), is meant to mark the string as already having been converted via a naming convention. It’s documented at https://docs.sqlalchemy.org/en/latest/core/constraints.html#sqlalchemy.schema.conv.
    • When we ask SQLAlchemy to make the table directly, via camcops_server create_db, it issues the constraint as CONSTRAINT ck_cpft_lps_discharge_msbd CHECK (management_specialling_behavioural_disturbance IN (0, 1)).
    • So far, so good.
    • Alembic is aware of the metadata and its naming convention via CamCOPS’s env.py.
    • When Alembic is called via camcops_server upgrade_db, it sees this column as Column('management_specialling_behavioural_disturbance', Boolean(name='ck_cpft_lps_discharge_msbd'), table=None).
      • We established this by temporarily editing alembic.operations.ops.CreateTableOp.create_table().
    • The resulting SQL constraint is CONSTRAINT ck_cpft_lps_discharge_management_specialling_behavioural_disturbance CHECK (management_specialling_behavioural_disturbance IN (0, 1)).
    • Note that our naming convention, camcops_server.cc_modules.cc_sqlalchemy.NAMING_CONVENTION, contains "ck": "ck_%(table_name)s_%(column_0_name)s".
    • So the Alembic-generated SQL uses our naming convention, and “neither Alembic nor SQLAlchemy currently create names for constraint objects where the name is otherwise unspecified” (https://docs.sqlalchemy.org/en/latest/core/constraints.html#configuring-constraint-naming-conventions), so it’s not likely to be coming from anywhere else.
    • The bug looks like Alembic is ignoring the conv() indicator.
    • This is with alembic==1.0.0 or alembic==1.0.7.
    • Searching the Alembic code for conv and then if conv leads to operations/base.py which contains op.f. This appears to be what we want: https://alembic.sqlalchemy.org/en/latest/ops.html#alembic.operations.Operations.f.
    • It is likely that the file of interest, 0001_start.py, was created before SQLAlchemy 0.9.4, when op.f became part of autogenerated output (according to the Alembic docs).
    • So the solution: add op.f to relevant parts of 0001_start.py. Find relevant columns in the source by searching for constraint_name=. Yup! That fixes it. When all are fixed, there should be an equal number of sa.Boolean(name=op.f( lines.
    • An example in the correct format from 0001_start.py is therefore sa.Column('management_specialling_behavioural_disturbance', sa.Boolean(name=op.f("ck_cpft_lps_discharge_msbd")), nullable=True).
    • Also renamed the constraint on deakin_1_healthreview.willing_to_participate_in_further_studies from wtpifs to ck_deakin_1_healthreview_wtpifs to match our convention. (It’s OK to rename these; they will affect new creation, but even if this were not part of the first Alembic revision, downgrading is by dropping a whole table, not dropping it constraint by constraint.)
  • When running an older version of CamCOPS (e.g. 2.2.7) on a Surface Book 2 / Windows 10: in no-keyboard Tablet mode, touches are not detected in the camera mode. Trackpad works fine. This was fixed by recompiling on this machine.

    Todo

    Does this mean that a QML onClicked event behaves differently with respect to touch events depending on whether it’s compiled on a touch-aware or a touch-unaware computer? That might represent a Qt bug; investigate and report if so. In the meantime, compile for Windows on a Surface Book 2 or similar.

    Also relevant:

  • New task: Perinatal POEM (Patient-rated Outcome and Experience Measure).

  • New task: Goal-Based Outcomes – Goal Record Sheet (GBO-GReS).

  • New task: Goal-Based Outcomes – Goal Progress Chart (GBO-GPC).

  • New task: Assessment Patient Experience Questionnaire for Psychological Therapies (APEQPT).

  • New task: Outcome Rating Scale (ORS).

  • New task: Session Rating Scale (SRS).

  • Bugfixes 2019-03-01: upload from very old tablets (e.g. v1.33) was broken. Errors included Unknown 'idnum1' field when uploading patient table. Also placed a size limit on an audit entry (one was >0.5 Mb).

    Note that this apparent bug isn’t really a bug (noticed when uploading from the old Titanium client):

    1 subject failed against an upload policy of
        forename AND surname AND dob AND sex
    145 subjects failed against an upload policy of
        sex AND ((forename AND surname AND dob) OR anyidnum)
    

    The Titanium client did not recognize anyidnum – and all patients will fail against an invalid policy. So that makes sense.

  • merge_db function made much more conservative about importing groups, ID number types – user must specify the mapping manually to avoid inadvertent errors.

  • Updated trackers to cope with blank (None) values, e.g. from GBO-GRaS task.

  • Menu header functions updated so that anonymous tasks show the “anonymous” icon properly.

  • Report: tasks by month/username.

  • check_index command.

  • Removed support for Python 3.5 since we want typing.Collection. Minimum is now Python 3.6. (That also allows f-strings.)

  • Bugfix: when password change frequency was >0, got “TypeError: can’t compare offset-naive and offset-aware datetimes” from camcops_server.cc_modules.cc_user.User.set_password_change_flag_if_necessary(). Added camcops_server.cc_modules.cc_request.CamcopsRequest.now_utc_no_tzinfo().

  • Changes to session management, to

  • Bugfix: policy validation used a combinatorial approach that became extremely slow when lots of ID numbers were in use (looking like a crash and sometimes causing database timeouts and follow-on errors). Rewritten 2019-03-23.

  • New task: Edinburgh Postnatal Depression Scale (EPDS). (Database revision 0019.)

  • ID number fields made mandatory in patient editing questionnaire on the client. (Reduces the chance of uploading a blank ID number, which wouldn’t help anyone.)

  • f-strings

  • Server released to CPFT on 2019-03-24.

Client and server v2.3.2 (2018-03-25 to 2018-04-04)

  • Session information in ORS and SRS summaries.
  • EPDS provides a CTV summary.
  • EPDS moved within CPFT Perinatal Service menu from “generic measures” to “specific conditions”.
  • Bugfix to C++ scoring function Icd10Depressive::main_complete(); some combinations were being labelled as “unknown” when more accuracy was possible.
  • Fixed CORE-10 alignment problem.
  • That was a more general problem of different name/value pairings sharing the same names. Fixable on that basis. See QuMcqGrid::setAlternateNameValueOptions.
  • New task: Postpartum Bonding Questionnaire (PBQ).
  • On the server, group administrators can change passwords and upload groups for users that they manage (meaning those users who are a member of one of their groups, and who are not a group administrator or superuser).
  • User deletion failed if the audit trail referred to the user (but no other checks failed). Was failing at a low (database) level with a foreign key constraint. Now performs a check for audit trail in camcops_server.cc_modules.webview.any_records_use_user().
  • Facility to hide individual special/sticky notes (with audit trail), so they’re not shown in HTML (+ PDF) and XML views. See e-mail RNC/JK/RE, 2018-10-12.
  • EMAIL_HOST_USERNAME no longer mandatory – surprisingly, some servers accept e-mails without a username.
  • Bugfix to camcops_server.cc_modules.cc_session.CamcopsSession.n_sessions_active_since(), which wasn’t converting to UTC properly.

Client and server v2.3.3

  • Windows service.

  • Bump from cardinal_pythonlib==1.0.49 to 'cardinal_pythonlib==1.0.53 for a bugfix (1.0.51) then SQL Server custom functions (1.0.52), then new MIME types (1.0.53).

  • Improvement to default behaviour of tools/create_database_migration.py: modified camcops_server.alembic.env.filter_column_ops() to skip modifications where modify_type is None. I’m not sure why these are now coming in droves from Alembic (it might be that this is what happens when a comment is changed).

  • In the process, indexed _exported_tasks.start_at_utc (somehow missed from 0014_new_export_mechanism.py).

  • Database revision to add all column comments. Note also:

    • Alembic misses out existing_nullable=False for fields with autoincrement=True
    • Manual checks are required for mysql.VARCHAR(...) as these can either be sa.String(length=...) or sa.Unicode(length=...).
  • Attempted fix for DELETE takes forever bug under SQL Server when reindexing as part of the upgrade_db command.

    • Search for if_sqlserver_disable_constraints_triggers.

    Todo

    check this fixes the SQL Server “DELETE” bug +++

  • Excel XLSX and OpenOffice/LibreOffice ODS formats supported for basic download.

  • Internationalization.

    • Client
    • MenuWindow changes to permit dynamic language change
    • Server
    • Client strings, core server to Danish
  • Discovered camera bug (on Ubuntu system): opening camera system crashed client with error fatal: unknown(0): Failed to create OpenGL context for format QSurfaceFormat; see https://bugreports.qt.io/browse/QTBUG-47191; this is with Qt 5.12.0.

    • I think this was because I’d upgraded the OS but not rebooted.
    • Multiple attempts to fix this, but it applied to all OpenGL programs (e.g. the Qt 3D bar graph demo) and the right thing to do was to make CamCOPS check for OpenGL rather than just assume its presence. See QuPhoto and openglfunc.cpp.
  • Bugfix to SingleTaskMenu: if you had a patient unselected, then locked the app, the task list wasn’t appropriately refreshed.

  • Bugfix for SQL DATETIME columns when used via database URLs like mysql+mysqldb://.. rather than mysql+pymysql://.

  • Caching for Task.isComplete() on the client.

  • SQLAlchemy upgraded from 1.2.8 to 1.3.0 in response to security vulnerabilities.

  • New option Settings ‣ Fetch all server info.

  • Ensure dialogs (e.g. initial password prompt) show the title in full, and don’t clip text that they contain (e.g. good: upload dialogue; bad: ?some of the cancellation dialogs). See uifunc::minimumSizeForTitle(). Not perfect (some guesswork), but better.

  • Client option to drop unknown tables.

  • cc_text.py and better server string framework for internationalization; see String locations in CamCOPS.

  • QuPage and corresponding questionnaire updates to provide a method for more complex validation when the user clicks “Finish” or tries to navigate away from a page. Initial use: preventing forward slashes in the server hostname.

  • make_xml_skeleton.py development tool

  • New task: Lynall M-E — IAM study — medical history

  • New task: Kirby Monetary Choice Questionnaire (MCQ)

  • New task: Assessment Patient Experience Questionnaire for CPFT Perinatal Services (APEQ-CPFT-Perinatal).

  • New task: Maternal Antenatal Attachment Scale (MAAS).

  • General release. But Android bug; see 2.3.4.

Client 2.3.4 (released 2019-06-20)

  • The Google Play Store will soon require 64-bit builds (https://android-developers.googleblog.com/2019/01/get-your-apps-ready-for-64-bit.html). In order to get 64-bit ARM compilation working for Android:

    • lots of work to build_qt.py
    • Default Android NDK from r11c to r20, which means moving from gcc to clang
    • OpenSSL from 1.1.0g to 1.1.1c to cope with clang
    • SQLCipher from 3.4.2 to 4.2.0 to cope with OpenSSL 1.1.1
    • Qt from 5.12.0 to 5.12.3 plus some Git tweaks to get upstream support for Android NDK r20.
    • Qt not compiling; bug raised at https://bugreports.qt.io/browse/QTBUG-76445.
  • V2.3.3 for Android was crashing on startup. From debugging views, error was “dlsym failed: undefined symbol main”; “Could not find main method”; subsequently “SIGSEGV” and “backtrace”.

    • objdump -t libcamcops.so | grep main gave

      001d4144 l     F .text  00000178              .hidden main
      

      whereas in a basic test app, objdump -t libbasic_qt_app.so | grep main gave

      00002e70 g     F .text  00000118              main
      

      Can also use the nm tool, or readelf -a, which is very clear (and probably others too).

      So why is main() hidden?

      The problem was -fvisibility=hidden in camcops.pro; fixed with VISIBLE_SYMBOL macro in preprocessor_aid.h.

Client and server v2.3.5, released 16 Sep 2019

  • Target Android API from 26 to 28 as now required by Google.

  • Bugfix to trackers, which were ignoring zero values; see camcops_server.cc_modules.cc_tracker.Tracker.get_single_plot_html().

  • Slightly hacky bugfix to sizehelpers::labelExtraSizeRequired(), to mitigate odd bug in which questions in a QuMcqGrid were over-word-wrapped. Debugging sequence:

    • commented sizehelpers.h

    • tried a size policy of expandingFixedHFWPolicy() in mcqfunc::addQuestion() – no joy, reverted

    • uncommented “#define OFFER_LAYOUT_DEBUG_BUTTON” in questionnaire.cpp

    • dump the layout from a PHQ9 (to the debug console)

    • The first question (“1. Little interest or pleasure in doing things”) is:

      LabelWordWrapWide<0x000056045feb4eb0 'question'>, visible, pos[DOWN] (0, 129), size[DOWN] (407 x 48), hasHeightForWidth()[UP] true, heightForWidth(407[DOWN])[UP] 58, minimumSize (0 x 0), maximumSize (16777215 x 16777215), sizeHint[UP] (407 x 29), minimumSizeHint[UP] (109 x 29), sizePolicy[UP] (Expanding, Fixed) [hasHeightForWidth=true], stylesheet: false, [WARNING: geometry().height() < heightForWidth(geometry().width())] [alignment from layout: AlignLeft | AlignVCenter]
      
      • … where “[DOWN]” means “imposed from above, i.e. from the layout” and “[UP]” means “determined by the widget or its contents and told to the layout”
      • … so this indicates the widget is saying “I’d like to be 407 x 29” (sizeHint) and “(horizontal expanding) 407 is a reasonable size; you can enlarge or shrink me if you want, but I’d like to be as large as possible; (HFW set) my height depends on my width; (vertical fixed) once my height is set from my width, that is fixed”
      • … on a screen in which [via GIMP] the label is 407 x 48 within a cell that should be about 899 x 48
      • … so, the widget is not asking for enough horizontal space.
    • This, therefore, points the finger at LabelWordWrapWide::sizeHint().

    • This is already ensuring the CSS etc. is applied, via ensurePolished(). That should deal with CSS-defined margins and the like.

    • uncomment “#define DEBUG_CALCULATIONS” in labelwordwrapwide.cpp

    • “1. Little interest or pleasure in doing” = 331 wide; “things” = 56 wide; another space = 9 wide – that’s suggest about 396 for the text. Then there is a left border of 12 pixels (ish) and probably the right one is identical – so the width should perhaps be about 420, and it’s asking for 407.

    • So where’s the deficit? Could either be the margins or the text itself.

    • LabelWordWrapWide::sizeOfTextWithoutWrap() should provide the text size itself. This may not be spot on; but running a grep on the output gives (in the last version – font sizes etc. may change as Qt lays stuff out) 397 wide. Which sounds OK.

    • grep “text_size” calcs.txt | grep “1. Little” gives:

      camcops[18336]: 2019-07-05T23:36:18.533: debug: ../tablet_qt/widgets/labelwordwrapwide.cpp(432): virtual QSize LabelWordWrapWide::sizeHint() const - text_size QSize(397, 19) -> QSize(407, 29) ... text: "1. Little interest or pleasure in doing things"
      

      so that suggests that 10 width is being added for margins, and that is too small; so the bug looks like it is in LabelWordWrapWide::extraSizeForCssOrLayout(), which, via a similar grep, is returning QSize(10, 10).

    • In turn that suggests the problem is in sizehelpers::labelExtraSizeRequired(), or the cache system.

    • Disabling “#define LWWW_USE_STYLE_CACHE”… no difference. So probably not the cache system there.

    • However, LabelWordWrapWide::extraSizeForCssOrLayout() is returning QSize(10, 10), which looks too small.

    • uncomment “#define DEBUG_WIDGET_MARGINS” in sizehelpers.cpp

    • looks like the margins are coming from the stylesheet, not the layout:

      camcops[22749]: 2019-07-06T00:12:30.198: debug: ../tablet_qt/lib/sizehelpers.cpp(217): QSize sizehelpers::widgetExtraSizeForCssOrLayout(const QWidget*, const QStyleOption*, const QSize&, bool, QStyle::ContentsType)widget "LabelWordWrapWide<0x000056458065d9c0 'question'>"; child_size QSize(0, 0); stylesheet_extra_size QSize(10, 10); extra_for_layout_margins QSize(0, 0) => total_extra QSize(10, 10)
      

      … which is right because QuMcqGrid::makeWidget() does “grid->setContentsMargins(uiconst::NO_MARGINS);”

    • So we’re looking at https://doc.qt.io/archives/qt-4.8/qstyle.html#sizeFromContents

    • The actual extra comes from the CSS: “#mcq_grid #question { padding: 5px; }”

    • As a test, doubling the width in sizehelpers::labelExtraSizeRequired() works. Reverted that.

    • comment out “#define ADD_EXTRA_FOR_LAYOUT_OR_CSS” in labelwordwrapwide.cpp – and see comments there; previous problems with QLabel. Nope, didn’t help; reverted.

    • set “show_widget_stylesheets = true” in layoutdumper::DumperConfig header and re-dump – but the widget doesn’t have its own stylesheet (just the global one, via uiconst::CSS_CAMCOPS_QUESTIONNAIRE) so that doesn’t help.

    • The odd thing is that I think the CSS is setting 5px padding, but 10px is coming out.

    • Check by changing that bit of CSS to 0 – yes.

    • So… the calculated extra size reflects the CSS, but the actual painting doesn’t! padding 0 -> 0 displayed on left; padding 5px -> 10px displayed on left; padding 10px -> 15px displayed on left; padding 15px -> 20 px displayed on left. So consistently +5. Odd. Similarly larger on the other sides.

    • Is it because QPushButton has padding: 5px, and sizehelpers::labelExtraSizeRequired() uses QStyle::CT_PushButton? No, setting QPushButton padding to 0 made no difference.

    • Anyway, back to doubling the width in sizehelpers::labelExtraSizeRequired(). A hack… And reverted other debugging options.

  • Client asks for information to be re-fetched (not the client to be re-registered – which is a privileged operation) when the server information doesn’t match stored copies.

  • Better SNOMED coding for the clinical tasks Progress note and Psychiatric clerking.

  • Bugfix: server group editing page crashed if no ID numbers defined. Changed in camcops_server.cc_modules.TokenizedPolicy.set_valid_idnums().

  • Client: For ACE-III and similar: when pages don’t scroll, offer facility to zoom widgets. See ZoomableWidget. Used in ACE-III for letters, picture naming, etc.

  • Server: restore autogeneration of CRIS and CRATE data dictionaries. See cc_anon.py etc.

  • HTML and PDF titles for tasks.

  • User list shows, for group administrators, which groups they administer.

  • Bugfix: client C++ functions mathfunc::countWhere() and mathfunc::countWhereNot() now respect NULL values, via mathfunc::containsRespectingNull(). This behaviour now matches camcops_server.cc_modules.cc_task.Task.count_where() and camcops_server.cc_modules.cc_task.Task.count_wherenot(). Applicable to the SHAPS task.

  • New task: Elixhauser Comorbidity Index (ElixhauserCI). (Database revision 0029.)

  • New task: Cambridge-Chicago Compulsivity Trait Scale (CHI-T). (Database revision 0030.)

  • New task: Short UPPS-P Impulsive Behaviour Scale (SUPPS-P). (Database revision 0031.)

  • New task: EULAR Sjögren’s Syndrome Patient Reported Index (ESSPRI). (Database revision 0032.)

  • New task: Ankylosing Spondylitis Disease Activity Score (ASDAS). (Database revision 0033.)

  • New task: Multidimensional Fatigue Inventory (MFI-20). (Database revision 0034.)

  • New task: Short-Form McGill Pain Questionnaire (SF-MPQ2). (Database revision 0035.)

  • New task: Disease Activity Score-28 (DAS28). (Database revision 0036.)

  • New task: Snaith–Hamilton Pleasure Scale (SHAPS). (Database revision 0037.)

  • Add optional waist circumference to BMI. (Database revision 0038.)

  • Add setMinimiumDate() and setMaximumDate() to QuDateTime. This also fixes the broken default minimum date of 1st January 1880.

  • Set strict_undefined=True for Mako template lookups, so they crash immediately on typos.

    • … nope; reverted. Tricky to get e.g. not_found.mako to inherit generic_failure.mako and override msg and extra_html without having unknown variables being handled (as undefined) rather than raising an error.
  • New tasks: Khandaker GM — MOJO study. (Database revisions 0039-0041.)

  • More consistent numbering/naming convention for custom tasks:

    • Numbering may be used in code (filenames, class names), if desired, which helps the programmer (it groups lots of files relating to the same task together quickly when searching).
    • Numbering not used in menus or task names, because it’s slightly confusing for the user (are we numbering tasks overall? Studies? Tasks within studies?).
    • Unnamed studies may be named “S1”, “S2”, …
    • Board format remains, overall: PI, study name, task name.
    • Historical table names not changed.
    • Future table names: try to avoid numbers.
  • Regression: crash in creating SVG figures from cardinal_expdetthreshold.py and cardinal_expectationdetection.py. Details in comments here. Likely due to a matplotlib change.

Client and server v2.3.6, released 31 Oct 2019

Client and server v2.3.7, IN PROGRESS

  • pyexcel-ods3 and pyexcel-xlsx for spreadsheet export; faster and much smaller for ODS files. See cc_tsv.py.