14.2.91. camcops_server.cc_modules.merge_db

camcops_server/cc_modules/merge_db.py


Copyright (C) 2012-2019 Rudolf Cardinal (rudolf@pobox.com).

This file is part of CamCOPS.

CamCOPS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

CamCOPS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CamCOPS. If not, see <http://www.gnu.org/licenses/>.


Tool to merge data from one CamCOPS database into another.

Has special code to deal with old databases.

camcops_server.cc_modules.merge_db.ensure_default_group_id(trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext) → None[source]

Ensure that the TranslationContext has a default_group_id key in its info dictionary. This is the ID, in the destination database, of the group to put records in where those records come from an older, pre-group-based CamCOPS database.

The user may have specified that default_group_id` on the command line. Otherwise, they may have specified a ``default_group_name, so we’ll use the ID of that group (creating it if necessary). If they specified neither, we will raise an AssertionError, because we have come to a situation where we need one or the other.

Parameters:trcon – the TranslationContext
camcops_server.cc_modules.merge_db.ensure_dest_iddef_exists(which_idnum: int, dst_session: sqlalchemy.orm.session.Session) → camcops_server.cc_modules.cc_idnumdef.IdNumDefinition[source]

Ensures that the specified ID number type exists in the destination database.

Parameters:
  • which_idnum – ID number type
  • dst_session – SQLAlchemy session for the destination database
Raises:

ValueError upon failure

camcops_server.cc_modules.merge_db.ensure_no_iddef_clash(src_iddef: camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, dst_iddef: camcops_server.cc_modules.cc_idnumdef.IdNumDefinition) → None[source]

Ensure that a given source and destination pair of ID number definitions, which must match on which_idnum, have the same description and short description, or raise ValueError.

Parameters:
camcops_server.cc_modules.merge_db.fetch_group_id_by_name(group_name: str, dst_session: sqlalchemy.orm.session.Session) → int[source]

Returns the group ID of the group with the specified name, in the destination session.

If there are multiple such groups, that’s a bug, and MultipleResultsFound will be raised.

If there’s no such group in the destination database with that name, one will be created, and its ID returned.

Parameters:
  • group_name – group name
  • dst_session – destination SQLAlchemy Session
Returns:

group ID in the destination database

camcops_server.cc_modules.merge_db.flush_session(dst_session: sqlalchemy.orm.session.Session) → None[source]

Flushes the destination SQLAlchemy session.

camcops_server.cc_modules.merge_db.get_dest_groupnum(src_groupnum: int, trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext, oldobj: Any) → int[source]

For a given source group number, returns the corresponding destination group number (validating en route).

Parameters:
  • src_groupnum – the group number in the source database
  • trcon – the TranslationContext
  • oldobj – the source object
Returns:

the corresponding which_idnum in the destination database

Raises:

ValueError if bad

camcops_server.cc_modules.merge_db.get_dest_which_idnum(src_which_idnum: int, trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext, oldobj: Any) → int[source]

For a given source ID number type, returns the corresponding destination ID number type (validating en route).

Parameters:
  • src_which_idnum – which_idnum in the source database
  • trcon – the TranslationContext
  • oldobj – the source object
Returns:

the corresponding which_idnum in the destination database

Raises:

ValueError if bad

camcops_server.cc_modules.merge_db.get_dst_group(dest_groupnum: int, dst_session: sqlalchemy.orm.session.Session) → camcops_server.cc_modules.cc_group.Group[source]

Ensures that the specified group number exists in the destination database and returns the corresponding group.

Parameters:
  • dest_groupnum – group number
  • dst_session – SQLAlchemy session for the destination database
Returns:

the group

Raises:

ValueError upon failure

camcops_server.cc_modules.merge_db.get_dst_iddef(dst_session: sqlalchemy.orm.session.Session, which_idnum: int) → Union[camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, NoneType][source]

Fetches an ID number definition from the destination database, ensuring it exists.

Parameters:
  • dst_session – destination SQLAlchemy Session
  • which_idnum – integer expressing which ID number type to look up
Returns:

an camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, or None if none was found

camcops_server.cc_modules.merge_db.get_skip_tables(src_tables: List[str]) → List[cardinal_pythonlib.sqlalchemy.table_identity.TableIdentity][source]

From the list of source table names provided, return details of tables in the metadata to skip because they are not in the source database.

Also checks that some core CamCOPS tables are present in the source, or raises ValueError.

Parameters:src_tables – list of all table names in the source database
Returns:list of cardinal_pythonlib.sqlalchemy.table_identity.TableIdentity objects representing tables to skip

Note that other tables to skip are defined in merge_camcops_db().

camcops_server.cc_modules.merge_db.get_src_iddefs(src_engine: sqlalchemy.engine.base.Engine, src_tables: List[str]) → Dict[int, camcops_server.cc_modules.cc_idnumdef.IdNumDefinition][source]

Get information about all the ID number definitions in the source database.

Parameters:
  • src_engine – source SQLAlchemy Engine
  • src_tables – list of all table names in the source database
Returns:

{which_idnum: idnumdef} mappings, where each idnumdef is a camcops_server.cc_modules.cc_idnumdef.IdNumDefinition not attached to any database session

Return type:

dictionary

camcops_server.cc_modules.merge_db.group_exists(group_id: int, dst_session: sqlalchemy.orm.session.Session) → bool[source]

Does a group exist in the destination session with the specified group ID?

Parameters:
  • group_id – integer group ID
  • dst_session – destination SQLAlchemy Session
camcops_server.cc_modules.merge_db.log_warning_srcobj(srcobj: Any) → None[source]

Prints a source (old) object to the log.

Parameters:srcobj – the source object
camcops_server.cc_modules.merge_db.merge_camcops_db(src: str, echo: bool, report_every: int, dummy_run: bool, info_only: bool, default_group_id: Union[int, NoneType], default_group_name: Union[str, NoneType], groupnum_map: Dict[int, int], whichidnum_map: Dict[int, int], skip_export_logs: bool = True, skip_audit_logs: bool = True) → None[source]

Merge an existing database (with a pre-v2 or later structure) into a comtemporary CamCOPS database.

Parameters:
  • src – source database SQLAlchemy URL
  • echo – echo the SQL that is produced?
  • report_every – provide a progress report every n records
  • dummy_run – don’t alter the destination database
  • info_only – show info, then stop
  • default_group_id – integer group ID (in the destination database) to use for source records that have no group (because they come from a very old source database) but need one
  • default_group_name – group name (in the destination database) to use for source records that have no group (because they come from a very old source database) but need one
  • groupnum_map – dictionary mapping group ID values from the source database to the destination database
  • whichidnum_map – dictionary mapping which_idnum values from the source database to the destination database
  • skip_export_logs – skip export log tables
  • skip_audit_logs – skip audit log table
camcops_server.cc_modules.merge_db.postprocess(src_engine: sqlalchemy.engine.base.Engine, dst_session: sqlalchemy.orm.session.Session) → None[source]

Implement any extra processing after merge_db() has been called.

  • Reindexes tasks.
  • Warns you about things that need to be done manually.
Parameters:
  • src_engine – source database SQLAlchemy engine
  • dst_session – destination database SQLAlchemy session
camcops_server.cc_modules.merge_db.translate_fn(trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext) → None[source]

Function to translate source objects to their destination counterparts, where special processing is required. Called as a callback from cardinal_pythonlib.sqlalchemy.merge_db.merge_db().

Parameters:trcon – the TranslationContext; all the relevant information is in here, and our function modifies its members.

This function does the following things:

  • For any records uploaded from tablets: set _group_id, if it’s blank.
  • For camcops_server.cc_modules.cc_user.User objects: if an identical user is found in the destination database, merge on it rather than creating a new one. Users with matching usernames are considered to be identical.
  • For Device objects: if an identical device is found, merge on it rather than creating a new one. Devices with matching names are considered to be identical.
  • For camcops_server.cc_modules.cc_group.Group objects: if an identical group is found, merge on it rather than creating a new one. Groups with matching names are considered to be identical.
  • For camcops_server.cc_modules.cc_patient.Patient objects: if any have ID numbers in the old format (as columns in the Patient table), convert them to the PatientIdNum system.
  • If we’re inserting a PatientIdNum, make sure there is a corresponding camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, and that it’s valid.
  • If we’re merging from a more modern database with the camcops_server.cc_modules.cc_idnumdef.IdNumDefinition table, check our ID number definitions don’t conflict.
  • Check we’re not creating duplicates for anything uploaded.