15.2.180. camcops_server.cc_modules.merge_db

camcops_server/cc_modules/merge_db.py

Copyright (C) 2012, University of Cambridge, Department of Psychiatry. Created by Rudolf Cardinal (rnc1001@cam.ac.uk).

This file is part of CamCOPS.

CamCOPS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

CamCOPS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.

Tool to merge data from one CamCOPS database into another.

Has special code to deal with old databases.

camcops_server.cc_modules.merge_db.ensure_default_group_id(trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext) → None[source]

Ensure that the TranslationContext has a default_group_id key in its info dictionary. This is the ID, in the destination database, of the group to put records in where those records come from an older, pre-group-based CamCOPS database.

The user may have specified that default_group_id` on the command line. Otherwise, they may have specified a ``default_group_name, so we’ll use the ID of that group (creating it if necessary). If they specified neither, we will raise an AssertionError, because we have come to a situation where we need one or the other.

Parameters

trcon – the TranslationContext

camcops_server.cc_modules.merge_db.ensure_dest_iddef_exists(which_idnum: int, dst_session: sqlalchemy.orm.session.Session) → camcops_server.cc_modules.cc_idnumdef.IdNumDefinition[source]

Ensures that the specified ID number type exists in the destination database.

Parameters

  • which_idnum – ID number type

  • dst_session – SQLAlchemy session for the destination database

Raises

ValueError –

camcops_server.cc_modules.merge_db.ensure_no_iddef_clash(src_iddef: camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, dst_iddef: camcops_server.cc_modules.cc_idnumdef.IdNumDefinition) → None[source]

Ensure that a given source and destination pair of ID number definitions, which must match on which_idnum, have the same description and short description, or raise ValueError.

Parameters
camcops_server.cc_modules.merge_db.fetch_group_id_by_name(group_name: str, dst_session: sqlalchemy.orm.session.Session) → int[source]

Returns the group ID of the group with the specified name, in the destination session.

If there are multiple such groups, that’s a bug, and MultipleResultsFound will be raised.

If there’s no such group in the destination database with that name, one will be created, and its ID returned.

Parameters

  • group_name – group name

  • dst_session – destination SQLAlchemy Session

Returns

group ID in the destination database

camcops_server.cc_modules.merge_db.flush_session(dst_session: sqlalchemy.orm.session.Session) → None[source]

Flushes the destination SQLAlchemy session.

camcops_server.cc_modules.merge_db.get_dest_groupnum(src_groupnum: int, trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext, oldobj: Any) → int[source]

For a given source group number, returns the corresponding destination group number (validating en route).

Parameters

  • src_groupnum – the group number in the source database

  • trcon – the TranslationContext

  • oldobj – the source object

Returns

the corresponding which_idnum in the destination database

Raises

ValueError –

camcops_server.cc_modules.merge_db.get_dest_which_idnum(src_which_idnum: int, trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext, oldobj: Any) → int[source]

For a given source ID number type, returns the corresponding destination ID number type (validating en route).

Parameters

  • src_which_idnum – which_idnum in the source database

  • trcon – the TranslationContext

  • oldobj – the source object

Returns

the corresponding which_idnum in the destination database

Raises

ValueError –

camcops_server.cc_modules.merge_db.get_dst_group(dest_groupnum: int, dst_session: sqlalchemy.orm.session.Session) → camcops_server.cc_modules.cc_group.Group[source]

Ensures that the specified group number exists in the destination database and returns the corresponding group.

Parameters

  • dest_groupnum – group number

  • dst_session – SQLAlchemy session for the destination database

Returns

the group

Raises

ValueError –

camcops_server.cc_modules.merge_db.get_dst_iddef(dst_session: sqlalchemy.orm.session.Session, which_idnum: int) → Optional[camcops_server.cc_modules.cc_idnumdef.IdNumDefinition][source]

Fetches an ID number definition from the destination database, ensuring it exists.

Parameters

  • dst_session – destination SQLAlchemy Session

  • which_idnum – integer expressing which ID number type to look up

Returns

an camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, or None if none was found

camcops_server.cc_modules.merge_db.get_skip_tables(src_tables: List[str]) → List[cardinal_pythonlib.sqlalchemy.table_identity.TableIdentity][source]

From the list of source table names provided, return details of tables in the metadata to skip because they are not in the source database.

Also checks that some core CamCOPS tables are present in the source, or raises ValueError.

Parameters

src_tables – list of all table names in the source database

Returns

list of cardinal_pythonlib.sqlalchemy.table_identity.TableIdentity objects representing tables to skip

Note that other tables to skip are defined in merge_camcops_db().

camcops_server.cc_modules.merge_db.get_src_iddefs(src_engine: sqlalchemy.engine.base.Engine, src_tables: List[str]) → Dict[int, camcops_server.cc_modules.cc_idnumdef.IdNumDefinition][source]

Get information about all the ID number definitions in the source database.

Parameters

  • src_engine – source SQLAlchemy Engine

  • src_tables – list of all table names in the source database

Returns

{which_idnum: idnumdef} mappings, where each idnumdef is a camcops_server.cc_modules.cc_idnumdef.IdNumDefinition not attached to any database session

Return type

dictionary

camcops_server.cc_modules.merge_db.group_exists(group_id: int, dst_session: sqlalchemy.orm.session.Session) → bool[source]

Does a group exist in the destination session with the specified group ID?

Parameters

  • group_id – integer group ID

  • dst_session – destination SQLAlchemy Session

camcops_server.cc_modules.merge_db.log_warning_srcobj(srcobj: Any) → None[source]

Prints a source (old) object to the log.

Parameters

srcobj – the source object

camcops_server.cc_modules.merge_db.merge_camcops_db(src: str, echo: bool, report_every: int, dummy_run: bool, info_only: bool, default_group_id: Optional[int], default_group_name: Optional[str], groupnum_map: Dict[int, int], whichidnum_map: Dict[int, int], skip_export_logs: bool = True, skip_audit_logs: bool = True) → None[source]

Merge an existing database (with a pre-v2 or later structure) into a comtemporary CamCOPS database.

Parameters

  • src – source database SQLAlchemy URL

  • echo – echo the SQL that is produced?

  • report_every – provide a progress report every n records

  • dummy_run – don’t alter the destination database

  • info_only – show info, then stop

  • default_group_id – integer group ID (in the destination database) to use for source records that have no group (because they come from a very old source database) but need one

  • default_group_name – group name (in the destination database) to use for source records that have no group (because they come from a very old source database) but need one

  • groupnum_map – dictionary mapping group ID values from the source database to the destination database

  • whichidnum_map – dictionary mapping which_idnum values from the source database to the destination database

  • skip_export_logs – skip export log tables

  • skip_audit_logs – skip audit log table

camcops_server.cc_modules.merge_db.postprocess(src_engine: sqlalchemy.engine.base.Engine, dst_session: sqlalchemy.orm.session.Session) → None[source]

Implement any extra processing after merge_db() has been called.

  • Reindexes tasks.

  • Warns you about things that need to be done manually.

Parameters

  • src_engine – source database SQLAlchemy engine

  • dst_session – destination database SQLAlchemy session

camcops_server.cc_modules.merge_db.translate_fn(trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext) → None[source]

Function to translate source objects to their destination counterparts, where special processing is required. Called as a callback from cardinal_pythonlib.sqlalchemy.merge_db.merge_db().

Parameters

trcon – the TranslationContext; all the relevant information is in here, and our function modifies its members.

This function does the following things:

  • For any records uploaded from tablets: set _group_id, if it’s blank.

  • For camcops_server.cc_modules.cc_user.User objects: if an identical user is found in the destination database, merge on it rather than creating a new one. Users with matching usernames are considered to be identical.

  • For Device objects: if an identical device is found, merge on it rather than creating a new one. Devices with matching names are considered to be identical.

  • For camcops_server.cc_modules.cc_group.Group objects: if an identical group is found, merge on it rather than creating a new one. Groups with matching names are considered to be identical.

  • For camcops_server.cc_modules.cc_patient.Patient objects: if any have ID numbers in the old format (as columns in the Patient table), convert them to the PatientIdNum system.

  • If we’re inserting a PatientIdNum, make sure there is a corresponding camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, and that it’s valid.

  • If we’re merging from a more modern database with the camcops_server.cc_modules.cc_idnumdef.IdNumDefinition table, check our ID number definitions don’t conflict.

  • Check we’re not creating duplicates for anything uploaded.