The Wayback Machine - https://web.archive.org/web/20220527100807/https://github.com/dgraph-io/dgraph/pull/7942
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(restore): Introduce incremental restore #7942

Merged
merged 22 commits into from Jul 7, 2021

Conversation

ahsanbarkati
Copy link
Contributor

@ahsanbarkati ahsanbarkati commented Jul 2, 2021

Incremental restore can be used to restore a set of incremental backups on a cluster with a part of the backup already restored.
No mutations should be made between a normal restore and an incremental restore.


This change is Reviewable

@ahsanbarkati ahsanbarkati requested a review from manishrjain as a code owner Jul 2, 2021
@github-actions github-actions bot added the area/graphql label Jul 2, 2021
Copy link
Contributor

@NamanJain8 NamanJain8 left a comment

Looks good to me. Nice change. 🎉

One issue that we discussed regarding the stale predicates not getting deleted bcoz of predicate move.

schema/schema.go Show resolved Hide resolved
Copy link
Contributor

@manishrjain manishrjain left a comment

:lgtm: Also compare the current manifest with the last manifest that was applied, and drop any predicates that have moved etc.

Secondly, use the incremental stream writer.

Reviewed 9 of 9 files at r1.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @ahsanbarkati)


worker/draft.go, line 677 at r1 (raw file):

		defer closer.Done()

		glog.Infof("Got restore proposal at Index:%d, ReadTs:%d",

Index: <space> %d


worker/online_restore.go, line 245 at r1 (raw file):

}

type IncrementalWriter struct {

Don't need this. We should ideally use incremental stream.


worker/online_restore.go, line 350 at r1 (raw file):

		if mapRes.shouldDropAll {
			if err := pstore.DropAll(); err != nil {
				return errors.Wrap(err, "failed to reduce incremental restore map")

failed to drop all ...

Copy link
Contributor

@NamanJain8 NamanJain8 left a comment

Few nit picks. Rest looks good.

worker/online_restore.go Outdated Show resolved Hide resolved
worker/online_restore.go Outdated Show resolved Hide resolved
worker/online_restore.go Show resolved Hide resolved
worker/restore_map.go Outdated Show resolved Hide resolved
worker/online_restore.go Outdated Show resolved Hide resolved
Copy link
Contributor Author

@ahsanbarkati ahsanbarkati left a comment

Reviewable status: 1 of 48 files reviewed, 4 unresolved discussions (waiting on @manishrjain and @NamanJain8)


worker/draft.go, line 677 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Index: <space> %d

Done.


worker/online_restore.go, line 245 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Don't need this. We should ideally use incremental stream.

Done.


worker/online_restore.go, line 350 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

failed to drop all ...

Done.

@ahsanbarkati ahsanbarkati merged commit dfa5dae into release/v21.03-slash Jul 7, 2021
3 of 4 checks passed
@ahsanbarkati ahsanbarkati deleted the ahsan/incremental-restore branch Jul 7, 2021
ahsanbarkati added a commit that referenced this issue Aug 2, 2021
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.

(cherry picked from commit dfa5dae)
ahsanbarkati added a commit that referenced this issue Aug 2, 2021
* feat(restore): Introduce incremental restore (#7942)

This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.

(cherry picked from commit dfa5dae)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/graphql
3 participants