The purpose of SMR

Problem

When running ETL for your datawarehouse, changes to source databases often have an impact. The degree of impact of course depends on the nature and extent of the changes, but also on the "intelligence" of the ETL, or better: the extent to which the ETL can handle such changes. This intelligence is often lacking in which case it is up to the administrators to take timely action and adapt the ETL to the changed source database.

In practice however, one is not always timely aware of any changes or what those changes are. In that case often the consequence is that the ETL processes simply "crashes" in the production environment.

Cause

Sometimes the supplier (external or internal) of the source database informs the administrators about changes but that is certainly not always the case. The danger here is that the database change has already passed the test in the acceptance environment and is being used for production for some time without any problems, then the ETL suddenly crashes.

Example: a column definition changes from 50 to 100 characters, only when more than 50 characters are actually used an error occurs in the ETL because 'something' no longer fits, in many cases the error situation will first occur in the production environment.
To prevent production downtime, it is important that every change is detected and reported as soon as possible - that is, preferably even before testing - so that this can be taken into account and problems can be prevented.

Solution

The solution to the above problem is relatively simple; By keeping a repository of metadata for the relevant source databases, it is possible to periodically compare this with the source and report any changes to it.

This is exactly what SMR does.

How it works

See the video...