15.2.180. camcops_server.cc_modules.merge_db¶
camcops_server/cc_modules/merge_db.py
Copyright (C) 2012, University of Cambridge, Department of Psychiatry. Created by Rudolf Cardinal (rnc1001@cam.ac.uk).
This file is part of CamCOPS.
CamCOPS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
CamCOPS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.
Tool to merge data from one CamCOPS database into another.
Has special code to deal with old databases.
- camcops_server.cc_modules.merge_db.ensure_default_group_id(trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext) None [source]¶
Ensure that the
TranslationContext
has adefault_group_id
key in itsinfo
dictionary. This is the ID, in the destination database, of the group to put records in where those records come from an older, pre-group-based CamCOPS database.The user may have specified that
default_group_id` on the command line. Otherwise, they may have specified a ``default_group_name
, so we’ll use the ID of that group (creating it if necessary). If they specified neither, we will raise anAssertionError
, because we have come to a situation where we need one or the other.- Parameters
trcon – the
TranslationContext
- camcops_server.cc_modules.merge_db.ensure_dest_iddef_exists(which_idnum: int, dst_session: sqlalchemy.orm.session.Session) camcops_server.cc_modules.cc_idnumdef.IdNumDefinition [source]¶
Ensures that the specified ID number type exists in the destination database.
- Parameters
which_idnum – ID number type
dst_session – SQLAlchemy session for the destination database
- Raises
ValueError –
- camcops_server.cc_modules.merge_db.ensure_no_iddef_clash(src_iddef: camcops_server.cc_modules.cc_idnumdef.IdNumDefinition, dst_iddef: camcops_server.cc_modules.cc_idnumdef.IdNumDefinition) None [source]¶
Ensure that a given source and destination pair of ID number definitions, which must match on
which_idnum
, have the same description and short description, or raiseValueError
.- Parameters
src_iddef – source
camcops_server.cc_modules.cc_idnumdef.IdNumDefinition
dst_iddef – destination
camcops_server.cc_modules.cc_idnumdef.IdNumDefinition
- camcops_server.cc_modules.merge_db.fetch_group_id_by_name(group_name: str, dst_session: sqlalchemy.orm.session.Session) int [source]¶
Returns the group ID of the group with the specified name, in the destination session.
If there are multiple such groups, that’s a bug, and
MultipleResultsFound
will be raised.If there’s no such group in the destination database with that name, one will be created, and its ID returned.
- Parameters
group_name – group name
dst_session – destination SQLAlchemy
Session
- Returns
group ID in the destination database
- camcops_server.cc_modules.merge_db.flush_session(dst_session: sqlalchemy.orm.session.Session) None [source]¶
Flushes the destination SQLAlchemy session.
- camcops_server.cc_modules.merge_db.get_dest_groupnum(src_groupnum: int, trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext, oldobj: Any) int [source]¶
For a given source group number, returns the corresponding destination group number (validating en route).
- Parameters
src_groupnum – the group number in the source database
trcon – the
TranslationContext
oldobj – the source object
- Returns
the corresponding which_idnum in the destination database
- Raises
ValueError –
- camcops_server.cc_modules.merge_db.get_dest_which_idnum(src_which_idnum: int, trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext, oldobj: Any) int [source]¶
For a given source ID number type, returns the corresponding destination ID number type (validating en route).
- Parameters
src_which_idnum – which_idnum in the source database
trcon – the
TranslationContext
oldobj – the source object
- Returns
the corresponding which_idnum in the destination database
- Raises
ValueError –
- camcops_server.cc_modules.merge_db.get_dst_group(dest_groupnum: int, dst_session: sqlalchemy.orm.session.Session) camcops_server.cc_modules.cc_group.Group [source]¶
Ensures that the specified group number exists in the destination database and returns the corresponding group.
- Parameters
dest_groupnum – group number
dst_session – SQLAlchemy session for the destination database
- Returns
the group
- Raises
ValueError –
- camcops_server.cc_modules.merge_db.get_dst_iddef(dst_session: sqlalchemy.orm.session.Session, which_idnum: int) Optional[camcops_server.cc_modules.cc_idnumdef.IdNumDefinition] [source]¶
Fetches an ID number definition from the destination database, ensuring it exists.
- Parameters
dst_session – destination SQLAlchemy
Session
which_idnum – integer expressing which ID number type to look up
- Returns
an
camcops_server.cc_modules.cc_idnumdef.IdNumDefinition
, orNone
if none was found
- camcops_server.cc_modules.merge_db.get_skip_tables(src_tables: List[str]) List[cardinal_pythonlib.sqlalchemy.table_identity.TableIdentity] [source]¶
From the list of source table names provided, return details of tables in the metadata to skip because they are not in the source database.
Also checks that some core CamCOPS tables are present in the source, or raises
ValueError
.- Parameters
src_tables – list of all table names in the source database
- Returns
list of
cardinal_pythonlib.sqlalchemy.table_identity.TableIdentity
objects representing tables to skip
Note that other tables to skip are defined in
merge_camcops_db()
.
- camcops_server.cc_modules.merge_db.get_src_iddefs(src_engine: sqlalchemy.engine.base.Engine, src_tables: List[str]) Dict[int, camcops_server.cc_modules.cc_idnumdef.IdNumDefinition] [source]¶
Get information about all the ID number definitions in the source database.
- Parameters
src_engine – source SQLAlchemy
Engine
src_tables – list of all table names in the source database
- Returns
{which_idnum: idnumdef}
mappings, where eachidnumdef
is acamcops_server.cc_modules.cc_idnumdef.IdNumDefinition
not attached to any database session- Return type
dictionary
- camcops_server.cc_modules.merge_db.group_exists(group_id: int, dst_session: sqlalchemy.orm.session.Session) bool [source]¶
Does a group exist in the destination session with the specified group ID?
- Parameters
group_id – integer group ID
dst_session – destination SQLAlchemy
Session
- camcops_server.cc_modules.merge_db.log_warning_srcobj(srcobj: Any) None [source]¶
Prints a source (old) object to the log.
- Parameters
srcobj – the source object
- camcops_server.cc_modules.merge_db.merge_camcops_db(src: str, echo: bool, report_every: int, dummy_run: bool, info_only: bool, default_group_id: Optional[int], default_group_name: Optional[str], groupnum_map: Dict[int, int], whichidnum_map: Dict[int, int], skip_export_logs: bool = True, skip_audit_logs: bool = True) None [source]¶
Merge an existing database (with a pre-v2 or later structure) into a comtemporary CamCOPS database.
- Parameters
src – source database SQLAlchemy URL
echo – echo the SQL that is produced?
report_every – provide a progress report every n records
dummy_run – don’t alter the destination database
info_only – show info, then stop
default_group_id – integer group ID (in the destination database) to use for source records that have no group (because they come from a very old source database) but need one
default_group_name – group name (in the destination database) to use for source records that have no group (because they come from a very old source database) but need one
groupnum_map – dictionary mapping group ID values from the source database to the destination database
whichidnum_map – dictionary mapping
which_idnum
values from the source database to the destination databaseskip_export_logs – skip export log tables
skip_audit_logs – skip audit log table
- camcops_server.cc_modules.merge_db.postprocess(src_engine: sqlalchemy.engine.base.Engine, dst_session: sqlalchemy.orm.session.Session) None [source]¶
Implement any extra processing after
merge_db()
has been called.Reindexes tasks.
Warns you about things that need to be done manually.
- Parameters
src_engine – source database SQLAlchemy engine
dst_session – destination database SQLAlchemy session
- camcops_server.cc_modules.merge_db.translate_fn(trcon: cardinal_pythonlib.sqlalchemy.merge_db.TranslationContext) None [source]¶
Function to translate source objects to their destination counterparts, where special processing is required. Called as a callback from
cardinal_pythonlib.sqlalchemy.merge_db.merge_db()
.- Parameters
trcon – the
TranslationContext
; all the relevant information is in here, and our function modifies its members.
This function does the following things:
For any records uploaded from tablets: set
_group_id
, if it’s blank.For
camcops_server.cc_modules.cc_user.User
objects: if an identical user is found in the destination database, merge on it rather than creating a new one. Users with matching usernames are considered to be identical.For
Device
objects: if an identical device is found, merge on it rather than creating a new one. Devices with matching names are considered to be identical.For
camcops_server.cc_modules.cc_group.Group
objects: if an identical group is found, merge on it rather than creating a new one. Groups with matching names are considered to be identical.For
camcops_server.cc_modules.cc_patient.Patient
objects: if any have ID numbers in the old format (as columns in the Patient table), convert them to thePatientIdNum
system.If we’re inserting a
PatientIdNum
, make sure there is a correspondingcamcops_server.cc_modules.cc_idnumdef.IdNumDefinition
, and that it’s valid.If we’re merging from a more modern database with the
camcops_server.cc_modules.cc_idnumdef.IdNumDefinition
table, check our ID number definitions don’t conflict.Check we’re not creating duplicates for anything uploaded.