Skip to content

Commit

Permalink
[Backport 2.6][#9270][backup] Fixed compatibility issue in yb_backup.
Browse files Browse the repository at this point in the history
Summary:
`yb_backup` script must correctly parse output of `yb-admin list_snapshots show_details` command for older releases.

Old format:
```
{"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
{"type":"TABLE","id":"d9603c2cab0b48ec807936496ac0e70e","data":{"name":"t2",...,"namespace_id"=...}}}}
{"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
{"type":"TABLE","id":"28b5cebe9b0c4cdaa70ce9ceab31b1e5","data":{\
       "name":"t2idx","indexed_table_id":"d9603c2cab0b48ec807936496ac0e70e",...,"namespace_id"=...}}}}
```
Plus the old format (before Aug 2020) does not have 'namespace_name' field, which was added later.
New format:
```
{"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
{"type":"TABLE","id":"d9603c2cab0b48ec807936496ac0e70e","data":{"name":"t2",...,"namespace_id"=...}}
{"type":"TABLE","id":"28b5cebe9b0c4cdaa70ce9ceab31b1e5","data":{\
       "name":"t2idx","indexed_table_id":"d9603c2cab0b48ec807936496ac0e70e",...,"namespace_id"=...}}}}
```
Original diff: https://phabricator.dev.yugabyte.com/D11420
Original commit: e8c72f3

Test Plan:
Jenkins: rebase: 2.6, hot

Existing backup tests:

ybd --cxx-test tools_yb-backup-test_ent

ybd --java-test org.yb.pgsql.TestYbBackup
ybd --java-test org.yb.cql.TestYbBackup --tp 1

Reviewers: bogdan, sergei

Reviewed By: sergei

Subscribers: jenkins-bot, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D12873
  • Loading branch information
OlegLoginov committed Sep 3, 2021
1 parent 3b3d7e9 commit e898207
Showing 1 changed file with 7 additions and 5 deletions.
12 changes: 7 additions & 5 deletions managed/devops/bin/yb_backup.py
Original file line number Diff line number Diff line change
Expand Up @@ -1045,11 +1045,11 @@ def wait_for_snapshot(self, snapshot_id, op, timeout_sec, update_table_list,
# 0436035d-c4c5-40c6-b45b-19538849b0d9 COMPLETE
# {"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
# {"type":"TABLE","id":"d9603c2cab0b48ec807936496ac0e70e","data":{"name":"t2",...}}
# {"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
# {"type":"TABLE","id":"28b5cebe9b0c4cdaa70ce9ceab31b1e5","data":{\
# "name":"t2idx","indexed_table_id":"d9603c2cab0b48ec807936496ac0e70e",...}}
# c1ad61bf-a42b-4bbb-94f9-28516985c2c5 COMPLETE
# ...
keyspaces = {}
for line in output.splitlines():
if not snapshot_done:
if line.find(snapshot_id) == 0:
Expand All @@ -1065,10 +1065,12 @@ def wait_for_snapshot(self, snapshot_id, op, timeout_sec, update_table_list,
object_type = loaded_json['type']
object_id = loaded_json['id']
data = loaded_json['data']
if object_type == 'TABLE':
keyspace_prefix = 'ysql.' if data['table_type'] == 'PGSQL_TABLE_TYPE' \
else ''
snapshot_keyspaces.append(keyspace_prefix + data['namespace_name'])
if object_type == 'NAMESPACE' and object_id not in keyspaces:
keyspace_prefix = 'ysql.' \
if data['database_type'] == 'YQL_DATABASE_PGSQL' else ''
keyspaces[object_id] = keyspace_prefix + data['name']
elif object_type == 'TABLE':
snapshot_keyspaces.append(keyspaces[data['namespace_id']])
snapshot_tables.append(data['name'])
snapshot_table_uuids.append(object_id)

Expand Down

0 comments on commit e898207

Please sign in to comment.