forked from Informatica-EIC/REST-API-Samples
-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathsrc_dbSchemaReplicationLineage_py.html
More file actions
426 lines (426 loc) · 91.2 KB
/
src_dbSchemaReplicationLineage_py.html
File metadata and controls
426 lines (426 loc) · 91.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=emulateIE7" />
<title>Coverage for src\dbSchemaReplicationLineage.py: 0%</title>
<link rel="stylesheet" href="style.css" type="text/css">
<script type="text/javascript" src="jquery.min.js"></script>
<script type="text/javascript" src="jquery.hotkeys.js"></script>
<script type="text/javascript" src="jquery.isonscreen.js"></script>
<script type="text/javascript" src="coverage_html.js"></script>
<script type="text/javascript">
jQuery(document).ready(coverage.pyfile_ready);
</script>
</head>
<body class="pyfile">
<div id="header">
<div class="content">
<h1>Coverage for <b>src\dbSchemaReplicationLineage.py</b> :
<span class="pc_cov">0%</span>
</h1>
<img id="keyboard_icon" src="keybd_closed.png" alt="Show keyboard shortcuts" />
<h2 class="stats">
122 statements
<button type="button" class="run shortkey_r button_toggle_run" title="Toggle lines run">0 run</button>
<button type="button" class="mis show_mis shortkey_m button_toggle_mis" title="Toggle lines missing">122 missing</button>
<button type="button" class="exc show_exc shortkey_x button_toggle_exc" title="Toggle lines excluded">0 excluded</button>
</h2>
</div>
</div>
<div class="help_panel">
<img id="panel_icon" src="keybd_open.png" alt="Hide keyboard shortcuts" />
<p class="legend">Hot-keys on this page</p>
<div>
<p class="keyhelp">
<span class="key">r</span>
<span class="key">m</span>
<span class="key">x</span>
<span class="key">p</span> toggle line displays
</p>
<p class="keyhelp">
<span class="key">j</span>
<span class="key">k</span> next/prev highlighted chunk
</p>
<p class="keyhelp">
<span class="key">0</span> (zero) top of page
</p>
<p class="keyhelp">
<span class="key">1</span> (one) first highlighted chunk
</p>
</div>
</div>
<div id="source">
<p id="t1" class="pln"><span class="n"><a href="#t1">1</a></span><span class="t"><span class="str">"""</span> </span><span class="r"></span></p>
<p id="t2" class="pln"><span class="n"><a href="#t2">2</a></span><span class="t"><span class="str">Created on Jun 26, 2018</span> </span><span class="r"></span></p>
<p id="t3" class="pln"><span class="n"><a href="#t3">3</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t4" class="pln"><span class="n"><a href="#t4">4</a></span><span class="t"><span class="str">@author: dwrigley</span> </span><span class="r"></span></p>
<p id="t5" class="pln"><span class="n"><a href="#t5">5</a></span><span class="t"><span class="str">***************************************************************************************</span> </span><span class="r"></span></p>
<p id="t6" class="pln"><span class="n"><a href="#t6">6</a></span><span class="t"><span class="str">DB schema replication custom lineage generator</span> </span><span class="r"></span></p>
<p id="t7" class="pln"><span class="n"><a href="#t7">7</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t8" class="pln"><span class="n"><a href="#t8">8</a></span><span class="t"><span class="str">process:-</span> </span><span class="r"></span></p>
<p id="t9" class="pln"><span class="n"><a href="#t9">9</a></span><span class="t"><span class="str"> scenario: the tables in 2 different schemas</span> </span><span class="r"></span></p>
<p id="t10" class="pln"><span class="n"><a href="#t10">10</a></span><span class="t"><span class="str"> (perhaps in different databases/resources) are replicated</span> </span><span class="r"></span></p>
<p id="t11" class="pln"><span class="n"><a href="#t11">11</a></span><span class="t"><span class="str"> in EDC - we have no way to automatically know that there is lineage</span> </span><span class="r"></span></p>
<p id="t12" class="pln"><span class="n"><a href="#t12">12</a></span><span class="t"><span class="str"> between the schema contents (tables & columns)</span> </span><span class="r"></span></p>
<p id="t13" class="pln"><span class="n"><a href="#t13">13</a></span><span class="t"><span class="str"> this utility will generate the custom lineage import to create the links</span> </span><span class="r"></span></p>
<p id="t14" class="pln"><span class="n"><a href="#t14">14</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t15" class="pln"><span class="n"><a href="#t15">15</a></span><span class="t"><span class="str"> given two schemas - leftSchema and rightSchema</span> </span><span class="r"></span></p>
<p id="t16" class="pln"><span class="n"><a href="#t16">16</a></span><span class="t"><span class="str"> find the 2 schemas objects in the catalog (GET /2/catalog/data/objects)</span> </span><span class="r"></span></p>
<p id="t17" class="pln"><span class="n"><a href="#t17">17</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t18" class="pln"><span class="n"><a href="#t18">18</a></span><span class="t"><span class="str"> for each schema</span> </span><span class="r"></span></p>
<p id="t19" class="pln"><span class="n"><a href="#t19">19</a></span><span class="t"><span class="str"> execute /2/catalog/data/relationships (2 levels schema->table->column)</span> </span><span class="r"></span></p>
<p id="t20" class="pln"><span class="n"><a href="#t20">20</a></span><span class="t"><span class="str"> for each table & column - store the id & name (names converted to</span> </span><span class="r"></span></p>
<p id="t21" class="pln"><span class="n"><a href="#t21">21</a></span><span class="t"><span class="str"> lower-case for case-insensitive match)</span> </span><span class="r"></span></p>
<p id="t22" class="pln"><span class="n"><a href="#t22">22</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t23" class="pln"><span class="n"><a href="#t23">23</a></span><span class="t"><span class="str"> for the stored objects (tables/columns) left side...</span> </span><span class="r"></span></p>
<p id="t24" class="pln"><span class="n"><a href="#t24">24</a></span><span class="t"><span class="str"> find the same table/column in the right side</span> </span><span class="r"></span></p>
<p id="t25" class="pln"><span class="n"><a href="#t25">25</a></span><span class="t"><span class="str"> if found - write a custom lineage link to csv</span> </span><span class="r"></span></p>
<p id="t26" class="pln"><span class="n"><a href="#t26">26</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t27" class="pln"><span class="n"><a href="#t27">27</a></span><span class="t"><span class="str"> Note: the custom lineage format used is:-</span> </span><span class="r"></span></p>
<p id="t28" class="pln"><span class="n"><a href="#t28">28</a></span><span class="t"><span class="str"> Association,From Connection,To Connection,From Object,To Object</span> </span><span class="r"></span></p>
<p id="t29" class="pln"><span class="n"><a href="#t29">29</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t30" class="pln"><span class="n"><a href="#t30">30</a></span><span class="t"><span class="str"> where: From Connection and To Connection will be empty</span> </span><span class="r"></span></p>
<p id="t31" class="pln"><span class="n"><a href="#t31">31</a></span><span class="t"><span class="str"> Association will be either core.DirectionalDataFlow</span> </span><span class="r"></span></p>
<p id="t32" class="pln"><span class="n"><a href="#t32">32</a></span><span class="t"><span class="str"> or core.DataSetDataFlow</span> </span><span class="r"></span></p>
<p id="t33" class="pln"><span class="n"><a href="#t33">33</a></span><span class="t"><span class="str"> the From and To Object will be the full object id</span> </span><span class="r"></span></p>
<p id="t34" class="pln"><span class="n"><a href="#t34">34</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t35" class="pln"><span class="n"><a href="#t35">35</a></span><span class="t"><span class="str"> when importing - there is no need for auto connection assignment,</span> </span><span class="r"></span></p>
<p id="t36" class="pln"><span class="n"><a href="#t36">36</a></span><span class="t"><span class="str"> since the full id's are provided this happens automatically</span> </span><span class="r"></span></p>
<p id="t37" class="pln"><span class="n"><a href="#t37">37</a></span><span class="t"><span class="str"> this is possible using v10.2.0 with a patch,</span> </span><span class="r"></span></p>
<p id="t38" class="pln"><span class="n"><a href="#t38">38</a></span><span class="t"><span class="str"> and works native in v10.2.1+</span> </span><span class="r"></span></p>
<p id="t39" class="pln"><span class="n"><a href="#t39">39</a></span><span class="t"><span class="str">"""</span> </span><span class="r"></span></p>
<p id="t40" class="pln"><span class="n"><a href="#t40">40</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t41" class="mis show_mis"><span class="n"><a href="#t41">41</a></span><span class="t"><span class="key">import</span> <span class="nam">argparse</span> </span><span class="r"></span></p>
<p id="t42" class="mis show_mis"><span class="n"><a href="#t42">42</a></span><span class="t"><span class="key">import</span> <span class="nam">csv</span> </span><span class="r"></span></p>
<p id="t43" class="mis show_mis"><span class="n"><a href="#t43">43</a></span><span class="t"><span class="key">import</span> <span class="nam">json</span> </span><span class="r"></span></p>
<p id="t44" class="mis show_mis"><span class="n"><a href="#t44">44</a></span><span class="t"><span class="key">import</span> <span class="nam">time</span> </span><span class="r"></span></p>
<p id="t45" class="pln"><span class="n"><a href="#t45">45</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t46" class="mis show_mis"><span class="n"><a href="#t46">46</a></span><span class="t"><span class="key">from</span> <span class="nam">src</span><span class="op">.</span><span class="nam">edc_utilities</span> <span class="key">import</span> <span class="nam">edcSessionHelper</span><span class="op">,</span> <span class="nam">edcutils</span> </span><span class="r"></span></p>
<p id="t47" class="pln"><span class="n"><a href="#t47">47</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t48" class="pln"><span class="n"><a href="#t48">48</a></span><span class="t"><span class="com"># set edc helper session + variables (easy/re-usable connection to edc api)</span> </span><span class="r"></span></p>
<p id="t49" class="mis show_mis"><span class="n"><a href="#t49">49</a></span><span class="t"><span class="nam">edcHelper</span> <span class="op">=</span> <span class="nam">edcSessionHelper</span><span class="op">.</span><span class="nam">EDCSession</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t50" class="pln"><span class="n"><a href="#t50">50</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t51" class="pln"><span class="n"><a href="#t51">51</a></span><span class="t"><span class="com"># define script command-line parameters (in global scope for gooey/wooey)</span> </span><span class="r"></span></p>
<p id="t52" class="mis show_mis"><span class="n"><a href="#t52">52</a></span><span class="t"><span class="nam">parser</span> <span class="op">=</span> <span class="nam">argparse</span><span class="op">.</span><span class="nam">ArgumentParser</span><span class="op">(</span><span class="nam">parents</span><span class="op">=</span><span class="op">[</span><span class="nam">edcHelper</span><span class="op">.</span><span class="nam">argparser</span><span class="op">]</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t53" class="pln"><span class="n"><a href="#t53">53</a></span><span class="t"><span class="com"># add args specific to this utility (left/right resource, schema, classtype...)</span> </span><span class="r"></span></p>
<p id="t54" class="mis show_mis"><span class="n"><a href="#t54">54</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t55" class="pln"><span class="n"><a href="#t55">55</a></span><span class="t"> <span class="str">"-lr"</span><span class="op">,</span> <span class="str">"--leftresource"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">True</span><span class="op">,</span> <span class="nam">help</span><span class="op">=</span><span class="str">"name of the left resource to find objects"</span> </span><span class="r"></span></p>
<p id="t56" class="pln"><span class="n"><a href="#t56">56</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t57" class="mis show_mis"><span class="n"><a href="#t57">57</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t58" class="pln"><span class="n"><a href="#t58">58</a></span><span class="t"> <span class="str">"-ls"</span><span class="op">,</span> <span class="str">"--leftschema"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">True</span><span class="op">,</span> <span class="nam">help</span><span class="op">=</span><span class="str">"name of the left schema/container object"</span> </span><span class="r"></span></p>
<p id="t59" class="pln"><span class="n"><a href="#t59">59</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t60" class="mis show_mis"><span class="n"><a href="#t60">60</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t61" class="pln"><span class="n"><a href="#t61">61</a></span><span class="t"> <span class="str">"-lt"</span><span class="op">,</span> <span class="str">"--lefttype"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">False</span><span class="op">,</span> <span class="nam">default</span><span class="op">=</span><span class="str">"com.infa.ldm.relational.Schema"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t62" class="pln"><span class="n"><a href="#t62">62</a></span><span class="t"> <span class="nam">help</span><span class="op">=</span><span class="str">"class type for the schema level object"</span> </span><span class="r"></span></p>
<p id="t63" class="pln"><span class="n"><a href="#t63">63</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t64" class="mis show_mis"><span class="n"><a href="#t64">64</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t65" class="pln"><span class="n"><a href="#t65">65</a></span><span class="t"> <span class="str">"-rr"</span><span class="op">,</span> <span class="str">"--rightresource"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">True</span><span class="op">,</span> <span class="nam">help</span><span class="op">=</span><span class="str">"name of the right resource to find objects"</span> </span><span class="r"></span></p>
<p id="t66" class="pln"><span class="n"><a href="#t66">66</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t67" class="mis show_mis"><span class="n"><a href="#t67">67</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t68" class="pln"><span class="n"><a href="#t68">68</a></span><span class="t"> <span class="str">"-rs"</span><span class="op">,</span> <span class="str">"--rightschema"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">True</span><span class="op">,</span> <span class="nam">help</span><span class="op">=</span><span class="str">"name of the right schema/container object"</span> </span><span class="r"></span></p>
<p id="t69" class="pln"><span class="n"><a href="#t69">69</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t70" class="mis show_mis"><span class="n"><a href="#t70">70</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t71" class="pln"><span class="n"><a href="#t71">71</a></span><span class="t"> <span class="str">"-rt"</span><span class="op">,</span> <span class="str">"--righttype"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">False</span><span class="op">,</span> <span class="nam">default</span><span class="op">=</span><span class="str">"com.infa.ldm.relational.Schema"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t72" class="pln"><span class="n"><a href="#t72">72</a></span><span class="t"> <span class="nam">help</span><span class="op">=</span><span class="str">"class type for the right schema level object"</span> </span><span class="r"></span></p>
<p id="t73" class="pln"><span class="n"><a href="#t73">73</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t74" class="mis show_mis"><span class="n"><a href="#t74">74</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t75" class="pln"><span class="n"><a href="#t75">75</a></span><span class="t"> <span class="str">"-pfx"</span><span class="op">,</span> <span class="str">"--csvprefix"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">False</span><span class="op">,</span> <span class="nam">default</span><span class="op">=</span><span class="str">"schemaLineage"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t76" class="pln"><span class="n"><a href="#t76">76</a></span><span class="t"> <span class="nam">help</span><span class="op">=</span><span class="str">"prefix to use when creating the output csv file"</span> </span><span class="r"></span></p>
<p id="t77" class="pln"><span class="n"><a href="#t77">77</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t78" class="mis show_mis"><span class="n"><a href="#t78">78</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t79" class="pln"><span class="n"><a href="#t79">79</a></span><span class="t"> <span class="str">"-rtp"</span><span class="op">,</span> <span class="str">"--righttableprefix"</span><span class="op">,</span> <span class="nam">required</span><span class="op">=</span><span class="key">False</span><span class="op">,</span> <span class="nam">default</span><span class="op">=</span><span class="str">""</span><span class="op">,</span> <span class="nam">help</span><span class="op">=</span><span class="str">"table prefix for right datasets"</span> </span><span class="r"></span></p>
<p id="t80" class="pln"><span class="n"><a href="#t80">80</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t81" class="mis show_mis"><span class="n"><a href="#t81">81</a></span><span class="t"><span class="nam">parser</span><span class="op">.</span><span class="nam">add_argument</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t82" class="pln"><span class="n"><a href="#t82">82</a></span><span class="t"> <span class="str">"-o"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t83" class="pln"><span class="n"><a href="#t83">83</a></span><span class="t"> <span class="str">"--outDir"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t84" class="pln"><span class="n"><a href="#t84">84</a></span><span class="t"> <span class="nam">default</span><span class="op">=</span><span class="str">"out"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t85" class="pln"><span class="n"><a href="#t85">85</a></span><span class="t"> <span class="nam">required</span><span class="op">=</span><span class="key">False</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t86" class="pln"><span class="n"><a href="#t86">86</a></span><span class="t"> <span class="nam">help</span><span class="op">=</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t87" class="pln"><span class="n"><a href="#t87">87</a></span><span class="t"> <span class="str">"output folder to write results - default = ./out "</span> </span><span class="r"></span></p>
<p id="t88" class="pln"><span class="n"><a href="#t88">88</a></span><span class="t"> <span class="str">" - will create folder if it does not exist"</span> </span><span class="r"></span></p>
<p id="t89" class="pln"><span class="n"><a href="#t89">89</a></span><span class="t"> <span class="op">)</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t90" class="pln"><span class="n"><a href="#t90">90</a></span><span class="t"><span class="op">)</span> </span><span class="r"></span></p>
<p id="t91" class="pln"><span class="n"><a href="#t91">91</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t92" class="pln"><span class="n"><a href="#t92">92</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t93" class="mis show_mis"><span class="n"><a href="#t93">93</a></span><span class="t"><span class="key">def</span> <span class="nam">get_schema_objects</span><span class="op">(</span><span class="nam">schema_name</span><span class="op">,</span> <span class="nam">schema_type</span><span class="op">,</span> <span class="nam">resource_name</span><span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t94" class="pln"><span class="n"><a href="#t94">94</a></span><span class="t"> <span class="str">"""</span> </span><span class="r"></span></p>
<p id="t95" class="pln"><span class="n"><a href="#t95">95</a></span><span class="t"><span class="str"> given a schema name, schema class type (e.g. hanadb is different)</span> </span><span class="r"></span></p>
<p id="t96" class="pln"><span class="n"><a href="#t96">96</a></span><span class="t"><span class="str"> and resource name, find the schema object</span> </span><span class="r"></span></p>
<p id="t97" class="pln"><span class="n"><a href="#t97">97</a></span><span class="t"><span class="str"> then</span> </span><span class="r"></span></p>
<p id="t98" class="pln"><span class="n"><a href="#t98">98</a></span><span class="t"><span class="str"> execute a relationships call to get the schema tables & columns</span> </span><span class="r"></span></p>
<p id="t99" class="pln"><span class="n"><a href="#t99">99</a></span><span class="t"><span class="str"> (parent/child links)</span> </span><span class="r"></span></p>
<p id="t100" class="pln"><span class="n"><a href="#t100">100</a></span><span class="t"><span class="str"> note: some models separate primary key columns from regular columns</span> </span><span class="r"></span></p>
<p id="t101" class="pln"><span class="n"><a href="#t101">101</a></span><span class="t"><span class="str"> note: some models have different relationships (e.g. sap hana db)</span> </span><span class="r"></span></p>
<p id="t102" class="pln"><span class="n"><a href="#t102">102</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t103" class="pln"><span class="n"><a href="#t103">103</a></span><span class="t"><span class="str"> returns a dictionary of all tables & columns for the schema & the id of</span> </span><span class="r"></span></p>
<p id="t104" class="pln"><span class="n"><a href="#t104">104</a></span><span class="t"><span class="str"> the schema object</span> </span><span class="r"></span></p>
<p id="t105" class="pln"><span class="n"><a href="#t105">105</a></span><span class="t"><span class="str"> key=table val=tableid</span> </span><span class="r"></span></p>
<p id="t106" class="pln"><span class="n"><a href="#t106">106</a></span><span class="t"><span class="str"> key=table.column val=columnid</span> </span><span class="r"></span></p>
<p id="t107" class="pln"><span class="n"><a href="#t107">107</a></span><span class="t"><span class="str"> """</span> </span><span class="r"></span></p>
<p id="t108" class="mis show_mis"><span class="n"><a href="#t108">108</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\tget_schema_objects for:"</span> <span class="op">+</span> <span class="nam">schema_name</span> <span class="op">+</span> <span class="str">" resource="</span> <span class="op">+</span> <span class="nam">resource_name</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t109" class="pln"><span class="n"><a href="#t109">109</a></span><span class="t"> <span class="com"># schemaDict returned key=TABLE.COLUMN value=column id</span> </span><span class="r"></span></p>
<p id="t110" class="mis show_mis"><span class="n"><a href="#t110">110</a></span><span class="t"> <span class="nam">schema_dict</span> <span class="op">=</span> <span class="op">{</span><span class="op">}</span> </span><span class="r"></span></p>
<p id="t111" class="mis show_mis"><span class="n"><a href="#t111">111</a></span><span class="t"> <span class="nam">table_names</span> <span class="op">=</span> <span class="op">{</span><span class="op">}</span> </span><span class="r"></span></p>
<p id="t112" class="pln"><span class="n"><a href="#t112">112</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t113" class="pln"><span class="n"><a href="#t113">113</a></span><span class="t"> <span class="com"># url = catalogServer + "/access/2/catalog/data/objects"</span> </span><span class="r"></span></p>
<p id="t114" class="mis show_mis"><span class="n"><a href="#t114">114</a></span><span class="t"> <span class="nam">url</span> <span class="op">=</span> <span class="nam">edcHelper</span><span class="op">.</span><span class="nam">baseUrl</span> <span class="op">+</span> <span class="str">"/access/2/catalog/data/objects"</span> </span><span class="r"></span></p>
<p id="t115" class="mis show_mis"><span class="n"><a href="#t115">115</a></span><span class="t"> <span class="nam">query</span> <span class="op">=</span> <span class="op">(</span><span class="str">f'+core.resourceName:"{resource_name}"'</span> </span><span class="r"></span></p>
<p id="t116" class="pln"><span class="n"><a href="#t116">116</a></span><span class="t"> <span class="op">+</span> <span class="str">f' +core.classType:"{schema_type}"'</span> </span><span class="r"></span></p>
<p id="t117" class="pln"><span class="n"><a href="#t117">117</a></span><span class="t"> <span class="op">+</span> <span class="str">f' +core.name:"{schema_name}"'</span> </span><span class="r"></span></p>
<p id="t118" class="pln"><span class="n"><a href="#t118">118</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t119" class="mis show_mis"><span class="n"><a href="#t119">119</a></span><span class="t"> <span class="nam">parameters</span> <span class="op">=</span> <span class="op">{</span><span class="str">"q"</span><span class="op">:</span> <span class="nam">query</span><span class="op">,</span> <span class="str">"offset"</span><span class="op">:</span> <span class="num">0</span><span class="op">,</span> <span class="str">"pageSize"</span><span class="op">:</span> <span class="num">1</span><span class="op">}</span> </span><span class="r"></span></p>
<p id="t120" class="mis show_mis"><span class="n"><a href="#t120">120</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\tquery="</span> <span class="op">+</span> <span class="nam">query</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t121" class="pln"><span class="n"><a href="#t121">121</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t122" class="mis show_mis"><span class="n"><a href="#t122">122</a></span><span class="t"> <span class="nam">schema_id</span> <span class="op">=</span> <span class="key">None</span> </span><span class="r"></span></p>
<p id="t123" class="mis show_mis"><span class="n"><a href="#t123">123</a></span><span class="t"> <span class="nam">table_count</span> <span class="op">=</span> <span class="num">0</span> </span><span class="r"></span></p>
<p id="t124" class="mis show_mis"><span class="n"><a href="#t124">124</a></span><span class="t"> <span class="nam">column_count</span> <span class="op">=</span> <span class="num">0</span> </span><span class="r"></span></p>
<p id="t125" class="pln"><span class="n"><a href="#t125">125</a></span><span class="t"> <span class="com"># make the call to find the schema object</span> </span><span class="r"></span></p>
<p id="t126" class="mis show_mis"><span class="n"><a href="#t126">126</a></span><span class="t"> <span class="nam">response</span> <span class="op">=</span> <span class="nam">edcHelper</span><span class="op">.</span><span class="nam">session</span><span class="op">.</span><span class="nam">get</span><span class="op">(</span><span class="nam">url</span><span class="op">,</span> <span class="nam">params</span><span class="op">=</span><span class="nam">parameters</span><span class="op">,</span> <span class="nam">timeout</span><span class="op">=</span><span class="num">3</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t127" class="mis show_mis"><span class="n"><a href="#t127">127</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"session get finished: {response.status_code}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t128" class="mis show_mis"><span class="n"><a href="#t128">128</a></span><span class="t"> <span class="nam">rc</span> <span class="op">=</span> <span class="nam">response</span><span class="op">.</span><span class="nam">status_code</span> </span><span class="r"></span></p>
<p id="t129" class="pln"><span class="n"><a href="#t129">129</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t130" class="mis show_mis"><span class="n"><a href="#t130">130</a></span><span class="t"> <span class="key">if</span> <span class="nam">rc</span> <span class="op">!=</span> <span class="num">200</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t131" class="mis show_mis"><span class="n"><a href="#t131">131</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"error reading object: rc="</span> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">rc</span><span class="op">)</span> <span class="op">+</span> <span class="str">" response:"</span> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">response</span><span class="op">.</span><span class="nam">json</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t132" class="mis show_mis"><span class="n"><a href="#t132">132</a></span><span class="t"> <span class="key">if</span> <span class="nam">rc</span> <span class="op">==</span> <span class="num">401</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t133" class="mis show_mis"><span class="n"><a href="#t133">133</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t134" class="pln"><span class="n"><a href="#t134">134</a></span><span class="t"> <span class="str">"\t401:Possible Missing/bad credentials"</span> </span><span class="r"></span></p>
<p id="t135" class="pln"><span class="n"><a href="#t135">135</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t136" class="mis show_mis"><span class="n"><a href="#t136">136</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="nam">str</span><span class="op">(</span><span class="nam">response</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t137" class="mis show_mis"><span class="n"><a href="#t137">137</a></span><span class="t"> <span class="key">return</span> </span><span class="r"></span></p>
<p id="t138" class="pln"><span class="n"><a href="#t138">138</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t139" class="pln"><span class="n"><a href="#t139">139</a></span><span class="t"> <span class="com"># get the total # of objects returned (first part of the json resultset)</span> </span><span class="r"></span></p>
<p id="t140" class="mis show_mis"><span class="n"><a href="#t140">140</a></span><span class="t"> <span class="nam">total_objects</span> <span class="op">=</span> <span class="nam">response</span><span class="op">.</span><span class="nam">json</span><span class="op">(</span><span class="op">)</span><span class="op">[</span><span class="str">"metadata"</span><span class="op">]</span><span class="op">[</span><span class="str">"totalCount"</span><span class="op">]</span> </span><span class="r"></span></p>
<p id="t141" class="mis show_mis"><span class="n"><a href="#t141">141</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\tobjects returned: "</span> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">total_objects</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t142" class="pln"><span class="n"><a href="#t142">142</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t143" class="mis show_mis"><span class="n"><a href="#t143">143</a></span><span class="t"> <span class="key">for</span> <span class="nam">item</span> <span class="key">in</span> <span class="nam">response</span><span class="op">.</span><span class="nam">json</span><span class="op">(</span><span class="op">)</span><span class="op">[</span><span class="str">"items"</span><span class="op">]</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t144" class="mis show_mis"><span class="n"><a href="#t144">144</a></span><span class="t"> <span class="nam">schema_id</span> <span class="op">=</span> <span class="nam">item</span><span class="op">[</span><span class="str">"id"</span><span class="op">]</span> </span><span class="r"></span></p>
<p id="t145" class="mis show_mis"><span class="n"><a href="#t145">145</a></span><span class="t"> <span class="nam">schema_name</span> <span class="op">=</span> <span class="nam">edcutils</span><span class="op">.</span><span class="nam">get_fact_value</span><span class="op">(</span><span class="nam">item</span><span class="op">,</span> <span class="str">"core.name"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t146" class="pln"><span class="n"><a href="#t146">146</a></span><span class="t"> <span class="com"># get the tables & columns</span> </span><span class="r"></span></p>
<p id="t147" class="mis show_mis"><span class="n"><a href="#t147">147</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\tfound schema: "</span> <span class="op">+</span> <span class="nam">schema_name</span> <span class="op">+</span> <span class="str">" id="</span> <span class="op">+</span> <span class="nam">schema_id</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t148" class="pln"><span class="n"><a href="#t148">148</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t149" class="mis show_mis"><span class="n"><a href="#t149">149</a></span><span class="t"> <span class="nam">lineage_url</span> <span class="op">=</span> <span class="nam">edcHelper</span><span class="op">.</span><span class="nam">baseUrl</span> <span class="op">+</span> <span class="str">"/access/2/catalog/data/relationships"</span> </span><span class="r"></span></p>
<p id="t150" class="mis show_mis"><span class="n"><a href="#t150">150</a></span><span class="t"> <span class="nam">lineage_parms</span> <span class="op">=</span> <span class="op">{</span> </span><span class="r"></span></p>
<p id="t151" class="pln"><span class="n"><a href="#t151">151</a></span><span class="t"> <span class="str">"seed"</span><span class="op">:</span> <span class="nam">schema_id</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t152" class="pln"><span class="n"><a href="#t152">152</a></span><span class="t"> <span class="str">"association"</span><span class="op">:</span> <span class="str">"core.ParentChild"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t153" class="pln"><span class="n"><a href="#t153">153</a></span><span class="t"> <span class="str">"depth"</span><span class="op">:</span> <span class="str">"2"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t154" class="pln"><span class="n"><a href="#t154">154</a></span><span class="t"> <span class="str">"direction"</span><span class="op">:</span> <span class="str">"OUT"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t155" class="pln"><span class="n"><a href="#t155">155</a></span><span class="t"> <span class="str">"includeAttribute"</span><span class="op">:</span> <span class="op">{</span><span class="str">"core.name"</span><span class="op">,</span> <span class="str">"core.classType"</span><span class="op">}</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t156" class="pln"><span class="n"><a href="#t156">156</a></span><span class="t"> <span class="str">"includeTerms"</span><span class="op">:</span> <span class="str">"false"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t157" class="pln"><span class="n"><a href="#t157">157</a></span><span class="t"> <span class="str">"removeDuplicateAggregateLinks"</span><span class="op">:</span> <span class="str">"false"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t158" class="pln"><span class="n"><a href="#t158">158</a></span><span class="t"> <span class="op">}</span> </span><span class="r"></span></p>
<p id="t159" class="mis show_mis"><span class="n"><a href="#t159">159</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t160" class="pln"><span class="n"><a href="#t160">160</a></span><span class="t"> <span class="str">"\tGET child rels for schema: "</span> <span class="op">+</span> <span class="nam">lineage_url</span> <span class="op">+</span> <span class="str">" parms="</span> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">lineage_parms</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t161" class="pln"><span class="n"><a href="#t161">161</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t162" class="pln"><span class="n"><a href="#t162">162</a></span><span class="t"> <span class="com"># get using uid/pwd</span> </span><span class="r"></span></p>
<p id="t163" class="mis show_mis"><span class="n"><a href="#t163">163</a></span><span class="t"> <span class="nam">lineage_response</span> <span class="op">=</span> <span class="nam">edcHelper</span><span class="op">.</span><span class="nam">session</span><span class="op">.</span><span class="nam">get</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t164" class="pln"><span class="n"><a href="#t164">164</a></span><span class="t"> <span class="nam">lineage_url</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t165" class="pln"><span class="n"><a href="#t165">165</a></span><span class="t"> <span class="nam">params</span><span class="op">=</span><span class="nam">lineage_parms</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t166" class="pln"><span class="n"><a href="#t166">166</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t167" class="mis show_mis"><span class="n"><a href="#t167">167</a></span><span class="t"> <span class="nam">lineage_status</span> <span class="op">=</span> <span class="nam">lineage_response</span><span class="op">.</span><span class="nam">status_code</span> </span><span class="r"></span></p>
<p id="t168" class="mis show_mis"><span class="n"><a href="#t168">168</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\tlineage resp="</span> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">lineage_status</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t169" class="mis show_mis"><span class="n"><a href="#t169">169</a></span><span class="t"> <span class="key">if</span> <span class="nam">lineage_status</span> <span class="op">!=</span> <span class="num">200</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t170" class="mis show_mis"><span class="n"><a href="#t170">170</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t171" class="pln"><span class="n"><a href="#t171">171</a></span><span class="t"> <span class="str">f"error getting schema contents (tables) rc={rc}"</span> </span><span class="r"></span></p>
<p id="t172" class="pln"><span class="n"><a href="#t172">172</a></span><span class="t"> <span class="str">f" response:{response.json}"</span> </span><span class="r"></span></p>
<p id="t173" class="pln"><span class="n"><a href="#t173">173</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t174" class="mis show_mis"><span class="n"><a href="#t174">174</a></span><span class="t"> <span class="key">if</span> <span class="nam">rc</span> <span class="op">==</span> <span class="num">401</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t175" class="mis show_mis"><span class="n"><a href="#t175">175</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t176" class="pln"><span class="n"><a href="#t176">176</a></span><span class="t"> <span class="str">"\t401:Possible Missing/bad credentials"</span> </span><span class="r"></span></p>
<p id="t177" class="pln"><span class="n"><a href="#t177">177</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t178" class="mis show_mis"><span class="n"><a href="#t178">178</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="nam">str</span><span class="op">(</span><span class="nam">response</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t179" class="mis show_mis"><span class="n"><a href="#t179">179</a></span><span class="t"> <span class="key">return</span> </span><span class="r"></span></p>
<p id="t180" class="pln"><span class="n"><a href="#t180">180</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t181" class="mis show_mis"><span class="n"><a href="#t181">181</a></span><span class="t"> <span class="key">if</span> <span class="nam">lineage_response</span><span class="op">.</span><span class="nam">text</span><span class="op">.</span><span class="nam">startswith</span><span class="op">(</span><span class="str">"{items:"</span><span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t182" class="pln"><span class="n"><a href="#t182">182</a></span><span class="t"> <span class="com"># bug (10.2.0 & 10.2.1) - the items collection should be "items"</span> </span><span class="r"></span></p>
<p id="t183" class="mis show_mis"><span class="n"><a href="#t183">183</a></span><span class="t"> <span class="nam">lineage_json</span> <span class="op">=</span> <span class="nam">lineage_response</span><span class="op">.</span><span class="nam">text</span><span class="op">.</span><span class="nam">replace</span><span class="op">(</span><span class="str">"items"</span><span class="op">,</span> <span class="str">'"items"'</span><span class="op">,</span> <span class="num">1</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t184" class="pln"><span class="n"><a href="#t184">184</a></span><span class="t"> <span class="key">else</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t185" class="mis show_mis"><span class="n"><a href="#t185">185</a></span><span class="t"> <span class="nam">lineage_json</span> <span class="op">=</span> <span class="nam">lineage_response</span><span class="op">.</span><span class="nam">text</span> </span><span class="r"></span></p>
<p id="t186" class="pln"><span class="n"><a href="#t186">186</a></span><span class="t"> <span class="com"># relsJson = json.loads(lineageJson.replace('items', '"items"'))</span> </span><span class="r"></span></p>
<p id="t187" class="mis show_mis"><span class="n"><a href="#t187">187</a></span><span class="t"> <span class="nam">rels_json</span> <span class="op">=</span> <span class="nam">json</span><span class="op">.</span><span class="nam">loads</span><span class="op">(</span><span class="nam">lineage_json</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t188" class="pln"><span class="n"><a href="#t188">188</a></span><span class="t"> <span class="com"># print(len(relsJson))</span> </span><span class="r"></span></p>
<p id="t189" class="pln"><span class="n"><a href="#t189">189</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t190" class="mis show_mis"><span class="n"><a href="#t190">190</a></span><span class="t"> <span class="key">for</span> <span class="nam">lineageItem</span> <span class="key">in</span> <span class="nam">rels_json</span><span class="op">[</span><span class="str">"items"</span><span class="op">]</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t191" class="pln"><span class="n"><a href="#t191">191</a></span><span class="t"> <span class="com"># print('\t\t' + str(lineageItem))</span> </span><span class="r"></span></p>
<p id="t192" class="mis show_mis"><span class="n"><a href="#t192">192</a></span><span class="t"> <span class="nam">in_id</span> <span class="op">=</span> <span class="nam">lineageItem</span><span class="op">.</span><span class="nam">get</span><span class="op">(</span><span class="str">"inId"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t193" class="mis show_mis"><span class="n"><a href="#t193">193</a></span><span class="t"> <span class="nam">out_id</span> <span class="op">=</span> <span class="nam">lineageItem</span><span class="op">.</span><span class="nam">get</span><span class="op">(</span><span class="str">"outId"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t194" class="pln"><span class="n"><a href="#t194">194</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t195" class="pln"><span class="n"><a href="#t195">195</a></span><span class="t"> <span class="com"># print('new inId===' + inId + " outId=" + outId)</span> </span><span class="r"></span></p>
<p id="t196" class="pln"><span class="n"><a href="#t196">196</a></span><span class="t"> <span class="com"># print(edcutils.get_fact_value(lineageItem["inEmbedded"], "core.name"))</span> </span><span class="r"></span></p>
<p id="t197" class="mis show_mis"><span class="n"><a href="#t197">197</a></span><span class="t"> <span class="nam">assoc_id</span> <span class="op">=</span> <span class="nam">lineageItem</span><span class="op">.</span><span class="nam">get</span><span class="op">(</span><span class="str">"associationId"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t198" class="pln"><span class="n"><a href="#t198">198</a></span><span class="t"> <span class="com"># print("\t\t" + inId + " assoc=" + assocId)</span> </span><span class="r"></span></p>
<p id="t199" class="pln"><span class="n"><a href="#t199">199</a></span><span class="t"> <span class="com"># if assocId=='com.infa.ldm.relational.SchemaTable':</span> </span><span class="r"></span></p>
<p id="t200" class="mis show_mis"><span class="n"><a href="#t200">200</a></span><span class="t"> <span class="key">if</span> <span class="nam">assoc_id</span><span class="op">.</span><span class="nam">endswith</span><span class="op">(</span><span class="str">".SchemaTable"</span><span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t201" class="pln"><span class="n"><a href="#t201">201</a></span><span class="t"> <span class="com"># note - custom lineage does not need table and column</span> </span><span class="r"></span></p>
<p id="t202" class="pln"><span class="n"><a href="#t202">202</a></span><span class="t"> <span class="com"># count the tables & store table names</span> </span><span class="r"></span></p>
<p id="t203" class="mis show_mis"><span class="n"><a href="#t203">203</a></span><span class="t"> <span class="nam">table_count</span> <span class="op">+=</span> <span class="num">1</span> </span><span class="r"></span></p>
<p id="t204" class="pln"><span class="n"><a href="#t204">204</a></span><span class="t"> <span class="com"># tableName = inId.split('/')[-1]</span> </span><span class="r"></span></p>
<p id="t205" class="mis show_mis"><span class="n"><a href="#t205">205</a></span><span class="t"> <span class="nam">table_name</span> <span class="op">=</span> <span class="nam">edcutils</span><span class="op">.</span><span class="nam">get_fact_value</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t206" class="pln"><span class="n"><a href="#t206">206</a></span><span class="t"> <span class="nam">lineageItem</span><span class="op">[</span><span class="str">"inEmbedded"</span><span class="op">]</span><span class="op">,</span> <span class="str">"core.name"</span> </span><span class="r"></span></p>
<p id="t207" class="pln"><span class="n"><a href="#t207">207</a></span><span class="t"> <span class="op">)</span><span class="op">.</span><span class="nam">lower</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t208" class="pln"><span class="n"><a href="#t208">208</a></span><span class="t"> <span class="com"># store the table name (for lookup when processing the columns)</span> </span><span class="r"></span></p>
<p id="t209" class="pln"><span class="n"><a href="#t209">209</a></span><span class="t"> <span class="com"># key=id, val=name</span> </span><span class="r"></span></p>
<p id="t210" class="mis show_mis"><span class="n"><a href="#t210">210</a></span><span class="t"> <span class="nam">table_names</span><span class="op">[</span><span class="nam">in_id</span><span class="op">]</span> <span class="op">=</span> <span class="nam">table_name</span> </span><span class="r"></span></p>
<p id="t211" class="mis show_mis"><span class="n"><a href="#t211">211</a></span><span class="t"> <span class="nam">schema_dict</span><span class="op">[</span><span class="nam">table_name</span><span class="op">]</span> <span class="op">=</span> <span class="nam">in_id</span> </span><span class="r"></span></p>
<p id="t212" class="pln"><span class="n"><a href="#t212">212</a></span><span class="t"> <span class="com"># if assocId=='com.infa.ldm.relational.TableColumn':</span> </span><span class="r"></span></p>
<p id="t213" class="mis show_mis"><span class="n"><a href="#t213">213</a></span><span class="t"> <span class="key">if</span> <span class="nam">assoc_id</span><span class="op">.</span><span class="nam">endswith</span><span class="op">(</span><span class="str">".TableColumn"</span><span class="op">)</span> <span class="key">or</span> <span class="nam">assoc_id</span><span class="op">.</span><span class="nam">endswith</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t214" class="pln"><span class="n"><a href="#t214">214</a></span><span class="t"> <span class="str">".TablePrimaryKeyColumn"</span> </span><span class="r"></span></p>
<p id="t215" class="pln"><span class="n"><a href="#t215">215</a></span><span class="t"> <span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t216" class="pln"><span class="n"><a href="#t216">216</a></span><span class="t"> <span class="com"># columnName = inId.split('/')[-1]</span> </span><span class="r"></span></p>
<p id="t217" class="mis show_mis"><span class="n"><a href="#t217">217</a></span><span class="t"> <span class="nam">column_count</span> <span class="op">+=</span> <span class="num">1</span> </span><span class="r"></span></p>
<p id="t218" class="mis show_mis"><span class="n"><a href="#t218">218</a></span><span class="t"> <span class="nam">column_name</span> <span class="op">=</span> <span class="nam">edcutils</span><span class="op">.</span><span class="nam">get_fact_value</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t219" class="pln"><span class="n"><a href="#t219">219</a></span><span class="t"> <span class="nam">lineageItem</span><span class="op">[</span><span class="str">"inEmbedded"</span><span class="op">]</span><span class="op">,</span> <span class="str">"core.name"</span> </span><span class="r"></span></p>
<p id="t220" class="pln"><span class="n"><a href="#t220">220</a></span><span class="t"> <span class="op">)</span><span class="op">.</span><span class="nam">lower</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t221" class="mis show_mis"><span class="n"><a href="#t221">221</a></span><span class="t"> <span class="nam">table_name</span> <span class="op">=</span> <span class="nam">table_names</span><span class="op">[</span><span class="nam">out_id</span><span class="op">]</span><span class="op">.</span><span class="nam">lower</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t222" class="pln"><span class="n"><a href="#t222">222</a></span><span class="t"> <span class="com"># print("column=" + tableName + "." + columnName)</span> </span><span class="r"></span></p>
<p id="t223" class="mis show_mis"><span class="n"><a href="#t223">223</a></span><span class="t"> <span class="nam">schema_dict</span><span class="op">[</span><span class="nam">table_name</span> <span class="op">+</span> <span class="str">"."</span> <span class="op">+</span> <span class="nam">column_name</span><span class="op">]</span> <span class="op">=</span> <span class="nam">in_id</span> </span><span class="r"></span></p>
<p id="t224" class="pln"><span class="n"><a href="#t224">224</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t225" class="mis show_mis"><span class="n"><a href="#t225">225</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t226" class="pln"><span class="n"><a href="#t226">226</a></span><span class="t"> <span class="str">"\tgetSchema: returning "</span> </span><span class="r"></span></p>
<p id="t227" class="pln"><span class="n"><a href="#t227">227</a></span><span class="t"> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">column_count</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t228" class="pln"><span class="n"><a href="#t228">228</a></span><span class="t"> <span class="op">+</span> <span class="str">" columns, in "</span> </span><span class="r"></span></p>
<p id="t229" class="pln"><span class="n"><a href="#t229">229</a></span><span class="t"> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">table_count</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t230" class="pln"><span class="n"><a href="#t230">230</a></span><span class="t"> <span class="op">+</span> <span class="str">" tables"</span> </span><span class="r"></span></p>
<p id="t231" class="pln"><span class="n"><a href="#t231">231</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t232" class="mis show_mis"><span class="n"><a href="#t232">232</a></span><span class="t"> <span class="key">return</span> <span class="nam">schema_dict</span><span class="op">,</span> <span class="nam">schema_id</span> </span><span class="r"></span></p>
<p id="t233" class="pln"><span class="n"><a href="#t233">233</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t234" class="pln"><span class="n"><a href="#t234">234</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t235" class="mis show_mis"><span class="n"><a href="#t235">235</a></span><span class="t"><span class="key">def</span> <span class="nam">main</span><span class="op">(</span><span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t236" class="pln"><span class="n"><a href="#t236">236</a></span><span class="t"> <span class="str">"""</span> </span><span class="r"></span></p>
<p id="t237" class="pln"><span class="n"><a href="#t237">237</a></span><span class="t"><span class="str"> initialise the csv file(s) to write</span> </span><span class="r"></span></p>
<p id="t238" class="pln"><span class="n"><a href="#t238">238</a></span><span class="t"><span class="str"> call get_schema_objects for both left and right schema objects</span> </span><span class="r"></span></p>
<p id="t239" class="pln"><span class="n"><a href="#t239">239</a></span><span class="t"><span class="str"> match the tables/columns from the left schema to the right</span> </span><span class="r"></span></p>
<p id="t240" class="pln"><span class="n"><a href="#t240">240</a></span><span class="t"><span class="str"> when matched</span> </span><span class="r"></span></p>
<p id="t241" class="pln"><span class="n"><a href="#t241">241</a></span><span class="t"><span class="str"> write a lineage link - table and column level</span> </span><span class="r"></span></p>
<p id="t242" class="pln"><span class="n"><a href="#t242">242</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t243" class="pln"><span class="n"><a href="#t243">243</a></span><span class="t"><span class="str"> Note: this script generates the newer lineage format using complete</span> </span><span class="r"></span></p>
<p id="t244" class="pln"><span class="n"><a href="#t244">244</a></span><span class="t"><span class="str"> object id's and relationship types</span> </span><span class="r"></span></p>
<p id="t245" class="pln"><span class="n"><a href="#t245">245</a></span><span class="t"><span class="str"> connection assignment will not be necessary</span> </span><span class="r"></span></p>
<p id="t246" class="pln"><span class="n"><a href="#t246">246</a></span><span class="t"><span class="str"> works with v10.2.1+</span> </span><span class="r"></span></p>
<p id="t247" class="pln"><span class="n"><a href="#t247">247</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t248" class="pln"><span class="n"><a href="#t248">248</a></span><span class="t"><span class="str"> """</span> </span><span class="r"></span></p>
<p id="t249" class="mis show_mis"><span class="n"><a href="#t249">249</a></span><span class="t"> <span class="nam">args</span> <span class="op">=</span> <span class="nam">args</span><span class="op">,</span> <span class="nam">unknown</span> <span class="op">=</span> <span class="nam">parser</span><span class="op">.</span><span class="nam">parse_known_args</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t250" class="pln"><span class="n"><a href="#t250">250</a></span><span class="t"> <span class="com"># setup edc session and catalog url - with auth in the session header,</span> </span><span class="r"></span></p>
<p id="t251" class="pln"><span class="n"><a href="#t251">251</a></span><span class="t"> <span class="com"># by using system vars or command-line args</span> </span><span class="r"></span></p>
<p id="t252" class="mis show_mis"><span class="n"><a href="#t252">252</a></span><span class="t"> <span class="nam">edcHelper</span><span class="op">.</span><span class="nam">init_edc_session</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t253" class="pln"><span class="n"><a href="#t253">253</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t254" class="mis show_mis"><span class="n"><a href="#t254">254</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"command-line args parsed = {args} "</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t255" class="pln"><span class="n"><a href="#t255">255</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t256" class="mis show_mis"><span class="n"><a href="#t256">256</a></span><span class="t"> <span class="nam">start_time</span> <span class="op">=</span> <span class="nam">time</span><span class="op">.</span><span class="nam">time</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t257" class="pln"><span class="n"><a href="#t257">257</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t258" class="mis show_mis"><span class="n"><a href="#t258">258</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"dbSchemaReplicationLineage:start"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t259" class="mis show_mis"><span class="n"><a href="#t259">259</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"Catalog={edcHelper.baseUrl}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t260" class="mis show_mis"><span class="n"><a href="#t260">260</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"left: resource={args.leftresource}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t261" class="mis show_mis"><span class="n"><a href="#t261">261</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"left: schema={args.leftschema}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t262" class="mis show_mis"><span class="n"><a href="#t262">262</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"left: type={args.lefttype}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t263" class="mis show_mis"><span class="n"><a href="#t263">263</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"right: resource={args.rightresource}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t264" class="mis show_mis"><span class="n"><a href="#t264">264</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"right: schema={args.rightschema}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t265" class="mis show_mis"><span class="n"><a href="#t265">265</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"right: type={args.righttype}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t266" class="mis show_mis"><span class="n"><a href="#t266">266</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"output folder:{args.outDir}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t267" class="mis show_mis"><span class="n"><a href="#t267">267</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"output file prefix:{args.csvprefix}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t268" class="mis show_mis"><span class="n"><a href="#t268">268</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">f"right table prefix:{args.righttableprefix}"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t269" class="pln"><span class="n"><a href="#t269">269</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t270" class="pln"><span class="n"><a href="#t270">270</a></span><span class="t"> <span class="com"># initialize csv output file</span> </span><span class="r"></span></p>
<p id="t271" class="mis show_mis"><span class="n"><a href="#t271">271</a></span><span class="t"> <span class="nam">column_header</span> <span class="op">=</span> <span class="op">[</span> </span><span class="r"></span></p>
<p id="t272" class="pln"><span class="n"><a href="#t272">272</a></span><span class="t"> <span class="str">"Association"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t273" class="pln"><span class="n"><a href="#t273">273</a></span><span class="t"> <span class="str">"From Connection"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t274" class="pln"><span class="n"><a href="#t274">274</a></span><span class="t"> <span class="str">"To Connection"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t275" class="pln"><span class="n"><a href="#t275">275</a></span><span class="t"> <span class="str">"From Object"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t276" class="pln"><span class="n"><a href="#t276">276</a></span><span class="t"> <span class="str">"To Object"</span><span class="op">,</span> </span><span class="r"></span></p>
<p id="t277" class="pln"><span class="n"><a href="#t277">277</a></span><span class="t"> <span class="op">]</span> </span><span class="r"></span></p>
<p id="t278" class="pln"><span class="n"><a href="#t278">278</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t279" class="pln"><span class="n"><a href="#t279">279</a></span><span class="t"> <span class="com"># set the csv fileName</span> </span><span class="r"></span></p>
<p id="t280" class="mis show_mis"><span class="n"><a href="#t280">280</a></span><span class="t"> <span class="nam">csv_file_name</span> <span class="op">=</span> <span class="op">(</span> </span><span class="r"></span></p>
<p id="t281" class="pln"><span class="n"><a href="#t281">281</a></span><span class="t"> <span class="str">f"{args.outDir}/{args.csvprefix}_{args.leftschema.lower()}"</span> </span><span class="r"></span></p>
<p id="t282" class="pln"><span class="n"><a href="#t282">282</a></span><span class="t"> <span class="str">f"_{args.rightschema.lower()}.csv"</span> </span><span class="r"></span></p>
<p id="t283" class="pln"><span class="n"><a href="#t283">283</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t284" class="pln"><span class="n"><a href="#t284">284</a></span><span class="t"> <span class="com"># python 3 & 2.7 use different methods</span> </span><span class="r"></span></p>
<p id="t285" class="mis show_mis"><span class="n"><a href="#t285">285</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"initializing file: "</span> <span class="op">+</span> <span class="nam">csv_file_name</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t286" class="mis show_mis"><span class="n"><a href="#t286">286</a></span><span class="t"> <span class="nam">f_csv_file</span> <span class="op">=</span> <span class="nam">open</span><span class="op">(</span><span class="nam">csv_file_name</span><span class="op">,</span> <span class="str">"w"</span><span class="op">,</span> <span class="nam">newline</span><span class="op">=</span><span class="str">""</span><span class="op">,</span> <span class="nam">encoding</span><span class="op">=</span><span class="str">"utf-8"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t287" class="mis show_mis"><span class="n"><a href="#t287">287</a></span><span class="t"> <span class="nam">col_writer</span> <span class="op">=</span> <span class="nam">csv</span><span class="op">.</span><span class="nam">writer</span><span class="op">(</span><span class="nam">f_csv_file</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t288" class="mis show_mis"><span class="n"><a href="#t288">288</a></span><span class="t"> <span class="nam">col_writer</span><span class="op">.</span><span class="nam">writerow</span><span class="op">(</span><span class="nam">column_header</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t289" class="pln"><span class="n"><a href="#t289">289</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t290" class="pln"><span class="n"><a href="#t290">290</a></span><span class="t"> <span class="com"># get the objects from the left schema into memory</span> </span><span class="r"></span></p>
<p id="t291" class="mis show_mis"><span class="n"><a href="#t291">291</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t292" class="pln"><span class="n"><a href="#t292">292</a></span><span class="t"> <span class="str">f"get left schema: name={args.leftschema}"</span> </span><span class="r"></span></p>
<p id="t293" class="pln"><span class="n"><a href="#t293">293</a></span><span class="t"> <span class="str">f" resource={args.leftresource}"</span> </span><span class="r"></span></p>
<p id="t294" class="pln"><span class="n"><a href="#t294">294</a></span><span class="t"> <span class="str">f" type={args.lefttype}"</span> </span><span class="r"></span></p>
<p id="t295" class="pln"><span class="n"><a href="#t295">295</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t296" class="mis show_mis"><span class="n"><a href="#t296">296</a></span><span class="t"> <span class="nam">left_objects</span><span class="op">,</span> <span class="nam">left_schema_id</span> <span class="op">=</span> <span class="nam">get_schema_objects</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t297" class="pln"><span class="n"><a href="#t297">297</a></span><span class="t"> <span class="nam">args</span><span class="op">.</span><span class="nam">leftschema</span><span class="op">,</span> <span class="nam">args</span><span class="op">.</span><span class="nam">lefttype</span><span class="op">,</span> <span class="nam">args</span><span class="op">.</span><span class="nam">leftresource</span> </span><span class="r"></span></p>
<p id="t298" class="pln"><span class="n"><a href="#t298">298</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t299" class="pln"><span class="n"><a href="#t299">299</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t300" class="pln"><span class="n"><a href="#t300">300</a></span><span class="t"> <span class="com"># get the objects from the right schema into memory</span> </span><span class="r"></span></p>
<p id="t301" class="mis show_mis"><span class="n"><a href="#t301">301</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t302" class="pln"><span class="n"><a href="#t302">302</a></span><span class="t"> <span class="str">f"get right schema: name={args.rightschema}"</span> </span><span class="r"></span></p>
<p id="t303" class="pln"><span class="n"><a href="#t303">303</a></span><span class="t"> <span class="str">f" resource={args.rightresource}"</span> </span><span class="r"></span></p>
<p id="t304" class="pln"><span class="n"><a href="#t304">304</a></span><span class="t"> <span class="str">f" type={args.righttype}"</span> </span><span class="r"></span></p>
<p id="t305" class="pln"><span class="n"><a href="#t305">305</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t306" class="mis show_mis"><span class="n"><a href="#t306">306</a></span><span class="t"> <span class="nam">right_objects</span><span class="op">,</span> <span class="nam">right_schema_id</span> <span class="op">=</span> <span class="nam">get_schema_objects</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t307" class="pln"><span class="n"><a href="#t307">307</a></span><span class="t"> <span class="nam">args</span><span class="op">.</span><span class="nam">rightschema</span><span class="op">,</span> <span class="nam">args</span><span class="op">.</span><span class="nam">righttype</span><span class="op">,</span> <span class="nam">args</span><span class="op">.</span><span class="nam">rightresource</span> </span><span class="r"></span></p>
<p id="t308" class="pln"><span class="n"><a href="#t308">308</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t309" class="pln"><span class="n"><a href="#t309">309</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t310" class="mis show_mis"><span class="n"><a href="#t310">310</a></span><span class="t"> <span class="nam">matches</span> <span class="op">=</span> <span class="num">0</span> </span><span class="r"></span></p>
<p id="t311" class="mis show_mis"><span class="n"><a href="#t311">311</a></span><span class="t"> <span class="nam">missing</span> <span class="op">=</span> <span class="num">0</span> </span><span class="r"></span></p>
<p id="t312" class="pln"><span class="n"><a href="#t312">312</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t313" class="mis show_mis"><span class="n"><a href="#t313">313</a></span><span class="t"> <span class="key">if</span> <span class="nam">len</span><span class="op">(</span><span class="nam">left_objects</span><span class="op">)</span> <span class="op">></span> <span class="num">0</span> <span class="key">and</span> <span class="nam">len</span><span class="op">(</span><span class="nam">right_objects</span><span class="op">)</span> <span class="op">></span> <span class="num">0</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t314" class="pln"><span class="n"><a href="#t314">314</a></span><span class="t"> <span class="com"># create the lineage file</span> </span><span class="r"></span></p>
<p id="t315" class="mis show_mis"><span class="n"><a href="#t315">315</a></span><span class="t"> <span class="nam">col_writer</span><span class="op">.</span><span class="nam">writerow</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t316" class="pln"><span class="n"><a href="#t316">316</a></span><span class="t"> <span class="op">[</span><span class="str">"core.DataSourceDataFlow"</span><span class="op">,</span> <span class="str">""</span><span class="op">,</span> <span class="str">""</span><span class="op">,</span> <span class="nam">left_schema_id</span><span class="op">,</span> <span class="nam">right_schema_id</span><span class="op">]</span> </span><span class="r"></span></p>
<p id="t317" class="pln"><span class="n"><a href="#t317">317</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t318" class="pln"><span class="n"><a href="#t318">318</a></span><span class="t"> <span class="com"># iterate over all left objects - looking for matching right ones</span> </span><span class="r"></span></p>
<p id="t319" class="mis show_mis"><span class="n"><a href="#t319">319</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\nprocessing: "</span> <span class="op">+</span> <span class="nam">str</span><span class="op">(</span><span class="nam">len</span><span class="op">(</span><span class="nam">left_objects</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span> <span class="str">" objects (left side)"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t320" class="mis show_mis"><span class="n"><a href="#t320">320</a></span><span class="t"> <span class="key">for</span> <span class="nam">left_name</span><span class="op">,</span> <span class="nam">leftVal</span> <span class="key">in</span> <span class="nam">left_objects</span><span class="op">.</span><span class="nam">items</span><span class="op">(</span><span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t321" class="pln"><span class="n"><a href="#t321">321</a></span><span class="t"> <span class="com"># if the target is using a prefix - add it to leftName</span> </span><span class="r"></span></p>
<p id="t322" class="mis show_mis"><span class="n"><a href="#t322">322</a></span><span class="t"> <span class="key">if</span> <span class="nam">len</span><span class="op">(</span><span class="nam">args</span><span class="op">.</span><span class="nam">righttableprefix</span><span class="op">)</span> <span class="op">></span> <span class="num">0</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t323" class="mis show_mis"><span class="n"><a href="#t323">323</a></span><span class="t"> <span class="nam">left_name</span> <span class="op">=</span> <span class="nam">args</span><span class="op">.</span><span class="nam">righttableprefix</span><span class="op">.</span><span class="nam">lower</span><span class="op">(</span><span class="op">)</span> <span class="op">+</span> <span class="nam">left_name</span> </span><span class="r"></span></p>
<p id="t324" class="pln"><span class="n"><a href="#t324">324</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t325" class="pln"><span class="n"><a href="#t325">325</a></span><span class="t"> <span class="com"># print("key=" + left_name + " " + leftVal + " " + str(left_name.count('.')))</span> </span><span class="r"></span></p>
<p id="t326" class="mis show_mis"><span class="n"><a href="#t326">326</a></span><span class="t"> <span class="key">if</span> <span class="nam">left_name</span> <span class="key">in</span> <span class="nam">right_objects</span><span class="op">.</span><span class="nam">keys</span><span class="op">(</span><span class="op">)</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t327" class="pln"><span class="n"><a href="#t327">327</a></span><span class="t"> <span class="com"># match</span> </span><span class="r"></span></p>
<p id="t328" class="mis show_mis"><span class="n"><a href="#t328">328</a></span><span class="t"> <span class="nam">right_val</span> <span class="op">=</span> <span class="nam">right_objects</span><span class="op">.</span><span class="nam">get</span><span class="op">(</span><span class="nam">left_name</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t329" class="mis show_mis"><span class="n"><a href="#t329">329</a></span><span class="t"> <span class="nam">matches</span> <span class="op">+=</span> <span class="num">1</span> </span><span class="r"></span></p>
<p id="t330" class="pln"><span class="n"><a href="#t330">330</a></span><span class="t"> <span class="com"># print("\t" + right_val)</span> </span><span class="r"></span></p>
<p id="t331" class="pln"><span class="n"><a href="#t331">331</a></span><span class="t"> <span class="com"># check if it is formatted as table.column or just table</span> </span><span class="r"></span></p>
<p id="t332" class="mis show_mis"><span class="n"><a href="#t332">332</a></span><span class="t"> <span class="key">if</span> <span class="nam">left_name</span><span class="op">.</span><span class="nam">count</span><span class="op">(</span><span class="str">"."</span><span class="op">)</span> <span class="op">==</span> <span class="num">1</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t333" class="pln"><span class="n"><a href="#t333">333</a></span><span class="t"> <span class="com"># column lineage - using DirectionalDataFlow</span> </span><span class="r"></span></p>
<p id="t334" class="mis show_mis"><span class="n"><a href="#t334">334</a></span><span class="t"> <span class="nam">col_writer</span><span class="op">.</span><span class="nam">writerow</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t335" class="pln"><span class="n"><a href="#t335">335</a></span><span class="t"> <span class="op">[</span><span class="str">"core.DirectionalDataFlow"</span><span class="op">,</span> <span class="str">""</span><span class="op">,</span> <span class="str">""</span><span class="op">,</span> <span class="nam">leftVal</span><span class="op">,</span> <span class="nam">right_val</span><span class="op">]</span> </span><span class="r"></span></p>
<p id="t336" class="pln"><span class="n"><a href="#t336">336</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t337" class="pln"><span class="n"><a href="#t337">337</a></span><span class="t"> <span class="key">else</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t338" class="pln"><span class="n"><a href="#t338">338</a></span><span class="t"> <span class="com"># table level - using core.DataSetDataFlow</span> </span><span class="r"></span></p>
<p id="t339" class="mis show_mis"><span class="n"><a href="#t339">339</a></span><span class="t"> <span class="nam">col_writer</span><span class="op">.</span><span class="nam">writerow</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t340" class="pln"><span class="n"><a href="#t340">340</a></span><span class="t"> <span class="op">[</span><span class="str">"core.DataSetDataFlow"</span><span class="op">,</span> <span class="str">""</span><span class="op">,</span> <span class="str">""</span><span class="op">,</span> <span class="nam">leftVal</span><span class="op">,</span> <span class="nam">right_val</span><span class="op">]</span> </span><span class="r"></span></p>
<p id="t341" class="pln"><span class="n"><a href="#t341">341</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t342" class="pln"><span class="n"><a href="#t342">342</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t343" class="pln"><span class="n"><a href="#t343">343</a></span><span class="t"> <span class="com"># write a line to the custom lineage csv file (connection assignment)</span> </span><span class="r"></span></p>
<p id="t344" class="pln"><span class="n"><a href="#t344">344</a></span><span class="t"> <span class="com"># colWriter.writerow([leftResource,rightResource,leftRef,rightRef])</span> </span><span class="r"></span></p>
<p id="t345" class="pln"><span class="n"><a href="#t345">345</a></span><span class="t"> <span class="key">else</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t346" class="mis show_mis"><span class="n"><a href="#t346">346</a></span><span class="t"> <span class="nam">missing</span> <span class="op">+=</span> <span class="num">1</span> </span><span class="r"></span></p>
<p id="t347" class="mis show_mis"><span class="n"><a href="#t347">347</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"\t no match on right side for key="</span> <span class="op">+</span> <span class="nam">left_name</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t348" class="pln"><span class="n"><a href="#t348">348</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t349" class="pln"><span class="n"><a href="#t349">349</a></span><span class="t"> <span class="key">else</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t350" class="mis show_mis"><span class="n"><a href="#t350">350</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"error getting schema info... - no linking/lineage created"</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t351" class="pln"><span class="n"><a href="#t351">351</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t352" class="mis show_mis"><span class="n"><a href="#t352">352</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span> </span><span class="r"></span></p>
<p id="t353" class="pln"><span class="n"><a href="#t353">353</a></span><span class="t"> <span class="str">f"dbSchemaLineageGen:finished. {matches} links created, {missing} missing (found in left, no match on right)"</span> </span><span class="r"></span></p>
<p id="t354" class="pln"><span class="n"><a href="#t354">354</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
<p id="t355" class="mis show_mis"><span class="n"><a href="#t355">355</a></span><span class="t"> <span class="nam">print</span><span class="op">(</span><span class="str">"run time = %s seconds ---"</span> <span class="op">%</span> <span class="op">(</span><span class="nam">time</span><span class="op">.</span><span class="nam">time</span><span class="op">(</span><span class="op">)</span> <span class="op">-</span> <span class="nam">start_time</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t356" class="pln"><span class="n"><a href="#t356">356</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t357" class="mis show_mis"><span class="n"><a href="#t357">357</a></span><span class="t"> <span class="nam">f_csv_file</span><span class="op">.</span><span class="nam">close</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
<p id="t358" class="pln"><span class="n"><a href="#t358">358</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t359" class="pln"><span class="n"><a href="#t359">359</a></span><span class="t"> </span><span class="r"></span></p>
<p id="t360" class="pln"><span class="n"><a href="#t360">360</a></span><span class="t"><span class="com"># call main - if not already called or used by another script</span> </span><span class="r"></span></p>
<p id="t361" class="mis show_mis"><span class="n"><a href="#t361">361</a></span><span class="t"><span class="key">if</span> <span class="nam">__name__</span> <span class="op">==</span> <span class="str">"__main__"</span><span class="op">:</span> </span><span class="r"></span></p>
<p id="t362" class="mis show_mis"><span class="n"><a href="#t362">362</a></span><span class="t"> <span class="nam">main</span><span class="op">(</span><span class="op">)</span> </span><span class="r"></span></p>
</div>
<div id="footer">
<div class="content">
<p>
<a class="nav" href="index.html">« index</a> <a class="nav" href="https://coverage.readthedocs.io">coverage.py v5.3</a>,
created at 2020-12-08 16:27 +0100
</p>
</div>
</div>
</body>
</html>