Sync Data Trim & Maintenance - Trouble Completing Large Sync
-
Is there anything that can/should be done to trim the sync data between devices & the server? For some reason one of my Vivaldi Android instances got out of sync with the server and now thinks it has to do an initial sync. Well, this is the kind of thing that requires somewhere on the order of 16 hours to complete. Unfortunately, during those 16 hours it usually hiccups somehow and then restarts from the beginning. I had just been babying the sync for 14 hours when I foolishly loaded a few webpages and caused it to restart again.
When I had it set to sync all data, in the Events tab of vivaldi://sync looking at the data in one of the first events it shows there are over 886,000 changes to sync. Changing my settings to only sync bookmarks merely reduced this number to over 840,000. I have a large bookmarks library, and while uncommon I don't think I'm unique.
I have somewhere near 90,000 bookmarks, so I'm not sure what has made this number so large. I'm guessing that it is partly due to the recovery operation I did recently on my desktop. I had accidentally deleted a large portion of my bookmarks and they weren't in the trash. To recover them I imported a recent backup that included them, which imported the entire library in that backup. So I then had to dedupe those bookmarks to restore my library to its normal state.
The dedupe process involved deleting at least 60,000 bookmarks, but there was an issue where the deduplication deletions got restored by the sync I believe. So I disabled sync while I finished removing all the duplicates, then enabled sync to push the correct library the server. I assume each one of those imports and deletes is a change to be synced, but that still doesn't get up to ~10x my library. Surely it's not trying to sync every change I've ever made since the creation of my sync account?
This synced fine to the Snapshot installation on my phone, but the Beta installation is in this funky state now. I cannot just blow away the profile and sync as if it's a new installation; I have a lot of bookmarks that were created in the Beta instance and have not been pushed to the server. If I destroy that data, I won't be able to rebuild it -- there are far too many new bookmarks for me to able to reproduce them with any accuracy.
So I'm curious if there is any kind if maintenance I could do to reduce the amount of changes that have to be synced to restore the sync connection. It would be nice if the sync could be segmented even, where it synced 100,000 changes and then wrote them to the system so they persisted. Then I could sync the next 100,000 and so on until I have finally synced all the changes.
As it works now, the sync is all or nothing. If the sync fails for whatever reason, then it doesn't hold onto any of the changes it successfully downloaded and restarts from the beginning. Perhaps there is the possibility for a change in the design of how sync works. Successfully downloaded changes could be written to the system on the fly. Changes are downloaded in sets of 100, however it seems that writing each set would incur a significant performance penalty. Maybe something like every 1,000, 2,500, 5,000 or 10,000 would be a good cutoff. Probably on the higher end, it doesn't take long to download 1,000 changes, perhaps even as high as 25,000 would be the right balance of performance and stability.
I may file a bug to this effect. This issue is hampering my productivity over the course of days as I try to work through it, and a quick fix of resetting the profile would lead to data loss. I have some data to collect before I can do that though.
Thanks for any help resolving this.
-
Omg, please shoot me. I got it to download all changes this time... the log showed 11 changes received in its final update rather than 100 changes received, more still to download (or whatever the exact language is). I just stared at it holding my breath, then it crashed.
Loaded the browser back up again, and it's starting over with the 840,000+ changes to download. Ugh... I'm going to go through and save off any tabs I want to keep and make the sync internals the only tab that exists I guess.
-
Finally! After a few days of running these long syncs and crashing them out shy some point, I babysat three sync process once again, and kept it going without delays so it went much faster. This time it didn't crash the app at the very end and my bookmarks successfully downloaded & uploaded as you can see in the screenshot below.
Now it's time to turn the other sync data back on and have that sync. This should be much easier however, only ~50k changes I think, 800k fewer than this was.
I am still interested in figuring out if there is anything I should do to trim down that size, and what the 886,000 changes were when that's >8 times the number of bookmarks. I also think that splitting the sync into smaller units that get persisted on the device would alleviate this problem to a significant degree. I'll dig into this more tomorrow or over the weekend.
-