Import history from Firefox without duplicates

[auipga]

Hi there, I did some search but didn't find any related topic.

When importing the history from Firefox, no deduplication occurs, which is indeed bad.
From time to time I want to reimport my history to catch the history of my synced Firefox on Android. But now all old history entries, which have been already imported a few weeks ago are dups.
After a few repetitions my history looks like this:

Also half a million page views take some time to load (each time I focus the already loaded hostory tab)
Do you know a solution? Thanks you.

[auipga]

My temporary solution:
Make a backup of vivaldi/Default/History!
History is a SQLite file.
Run this statement to delete all entries in main.visits table which are duplicates by url (which is an ID to urls table) and visit_time, but keep the first (min(duplicates.id)).

delete from visits
where exists
          (select 1 from visits as duplicates
           where visits.url = duplicates.url
             and visits.visit_time = duplicates.visit_time
           group by duplicates.url, duplicates.visit_time
           having count(1) > 1 and min(duplicates.id) <> visits.id);

Do this on your own risk. Don't blame me for a broken profile

I got some inspiration from https://stackoverflow.com/revisions/6471575/2

[auipga]

If you just want to see your dups then run this:

select v.id, v.url
from visits v
         join (select min(id) minid, url, visit_time, count(*) as qty
               from visits
               group by url, visit_time
               having count(1) > 1) duplicates
              on v.url = duplicates.url and v.visit_time = duplicates.visit_time
                  and v.id <> minid;

As Chrome uses 1601/01/01 as zero in timestamps, here are some useful statements if you want to play with them (e.g. to filter by date).

-- convert date to chrome timestamp
select (strftime('%s', datetime('2015-10-23 14:27:56')) + 11644473600) * 1000000;
-- convert chrome timestamp to readable date
select date(datetime(13090084076000000 / 1000000 - 11644473600, 'unixepoch'));

[greybeard]

@auipga can one use something like this for duplicate Bookmarks?

[auipga]

@greybeard No, bookmarks are stored as json file. Maybe you want to try some tool with an UI: https://chrome.google.com/webstore/detail/bookmarks-clean-up/oncbjlgldmiagjophlhobkogeladjijl/r (untested)

[greybeard]

@auipga I my wee mind, I'd think you'd do the deduplication in FF (or whatever browser), then perform the Export/Import.
That way you'd only have to deal with dupes caused buy bookmarking the same site in multiple browsers.