Solved Any info or docs on how notes attachments are stored in LevelDB?

darkmuck

I'm not sure if this is the correct forum for this question but I figured I would at least ask here.

I have created a C# utility that can convert the Notes JSON file to individual markdown files while preserving the folder hierarchy and such. I have all of that working but where I am stuck is not knowing how to view data in Vivaldi's sync store (LevelDB). I have tried to connect to the ldb files programmatically but they have a lock on them. If I were to find a way I still wouldn't know what to look for to understand how Notes attachments would be stored (how to query for them).

Does anyone know if there is any knowledge or documentation out there on any of this? I could dig through the Vivaldi source code but that would take some time. My end goal is to be able to export the attachments with the markdown but also implement an import mechanism so the tool could do both import and export to automate a sort of synchronization with markdown files. The only thing left is to get the attachments included from LevelDB and maybe have an option to convert any base64 in-line images to files.

Pathduck

@darkmuck Hi

LevelDB is basically an opaque box unless you're a (Chromium?) developer...
There are tools to read these DBs but AFAIK no simple good free ones (last I searched). Implementation might also differ making it very specific to the program using it.

BUT - I don't think you need to think about LDB here. From a quick test adding some attachments to a note:

      "children": [ {
         "content": "OKCE3E7BYB43N74KFK2MYMXZEBZTG5XSJVF5F3OVWJNC3TAROVSQ.5437",
         "date_added": "13331825790931332",
         "guid": "97595165-38dc-4a0c-8095-cb17054a71d6",
         "id": "5",
         "subject": "anton-100x.jpg",
         "type": "attachment",
         "url": ""
      }, {
         "content": "LACP2BNYDWUFEYRPNM2FRSCKRHRP7ZFVAVB6QM56LUAMKS3MO2KQ.1132544",
         "date_added": "13331825924772010",
         "guid": "99bdbe91-ceba-4edc-8e32-a8627957f821",
         "id": "6",
         "subject": "03 - Jetset - Afrika.mp3",
         "type": "attachment",
         "url": ""
      }, {
         "content": "XWKRHNWRZVIKAIQTKSNLI3Y3D5TMCPDQLMHAKFPUKAD5PMXEPWMA.3281368",
         "date_added": "13331826014123632",
         "guid": "1209baa5-29fe-4c61-9ccc-1db62c9b68e5",
         "id": "7",
         "subject": "ReShade_Setup_5.8.0.exe",
         "type": "attachment",
         "url": ""
      }, {
         "content": "2MW5NVYRFFU4ANKXATUPXQUCKWEHJYED7BJX3S4SPN3ROYK7M3ZA.832",
         "date_added": "13331826049456106",
         "guid": "5012cb66-09e2-41ae-b09b-4a54ea4563a3",
         "id": "8",
         "subject": "Notes.zip",
         "type": "attachment",
         "url": ""
      } ],
      "content": "TEST\nTEST\nTEST",
      "date_added": "13331825755853389",
      "guid": "a4d00902-e19a-4b2e-bd1c-bb99bba049b9",
      "id": "4",
      "subject": "",
      "type": "note",
      "url": ""
   }

So that's an image, mp3, exe and zip file. The attachment dialog only accepts images but there are simple tricks around that in Windows

The clue here is the content value:
XWKRHNWRZVIKAIQTKSNLI3Y3D5TMCPDQLMHAKFPUKAD5PMXEPWMA.3281368
This value is referenced in SyncedFiles\SyncedFilesData - also a JSON:

      "XWKRHNWRZVIKAIQTKSNLI3Y3D5TMCPDQLMHAKFPUKAD5PMXEPWMA.3281368": {
         "has_content_locally": true,
         "local_references": {
            "229171": [ "1209baa5-29fe-4c61-9ccc-1db62c9b68e5" ]
         },
         "mimetype": "application/octet-stream",
         "sync_refrences": {
         }

So the string refers to a renamed file located in the same folder.

2MW5NVYRFFU4ANKXATUPXQUCKWEHJYED7BJX3S4SPN3ROYK7M3ZA.832
LACP2BNYDWUFEYRPNM2FRSCKRHRP7ZFVAVB6QM56LUAMKS3MO2KQ.1132544
OKCE3E7BYB43N74KFK2MYMXZEBZTG5XSJVF5F3OVWJNC3TAROVSQ.5437
SyncedFilesData
SyncedFilesData.bak
XWKRHNWRZVIKAIQTKSNLI3Y3D5TMCPDQLMHAKFPUKAD5PMXEPWMA.3281368

Interestingly, the "extension" used here seems to be the file size which is a strange way of doing things...

Unfortunately, these files are not actually synced... Maybe understandable given the potential for taking up a lot of disk space and network capacity on Vivaldi's servers. Especially if people abuse it like I just did and start attaching EXE and ZIP files

darkmuck

I saw the cryptic naming of the attachments but hadn't thought they would be mapped to something on the file system. This would make it a bit easier for me though as I may not even need to care about LevelDB. I could use the subject property value as the filename when I do the conversion to markdown then vice-vesa on import but i'd need to have a way to preserve the original metadata (maybe a JSON file alongside the exported markdown files?). I want my tool to be able to do full import and export so I can run it as a scheduled task to "sync" with flat files in my OneDrive folder. I couldn't find anyone else doing this in a way that included attachments (unless I didn't look hard enough). But this will provides a way to synchronize everything related to notes including attachments.

Thanks for your help! This seems to point me in the right direction.

Pathduck

@darkmuck Sure!

A good tip for figuring out where stuff is referenced is simply search through all profile files containing the string. I use Total Commander, a great tool for such tasks.

Also, I know other users are simply keeping the "SyncedFiles" folder in sync between devices using their own sync services and it should work at least in theory... unless the user does a lot of constant adding/removal of notes and Speed Dial thumbnails.

So shouldn't really be necessary to write code to do this unless you really enjoy a challenge

darkmuck

@Pathduck

I enjoy the challenge but then again I do dev for a living anyways. A big pain point this will solve for me is giving me a way to view the notes as rendered markdown with attachments on my Android in OneDrive. Right now Vivaldi on Android only provides plain text markdown and nothing else. Also if I get the import working with attachments I can edit them with another tool on Android as well (WYSIWYG would be nice to have).

darkmuck

If Vivaldi was fully open sourced I would just contribute to the notes feature directly on the desktop and mobile versions of it. But I don't think this is possible.

daniel

The latest snapshot release introduces import and export for Vivaldi Notes in the user-interface. The new export format is Markdown with Front Matter containing the metadata. The notes folder structure is preserved during import and export.

Attachments are not, yet, part of the export or import. Support for exporting and importing attachments is pending on future support for attachments in Vivaldi Sync.

Pathduck

@daniel It's great, and also that it exports as .md files now and in a separate folder

One minor detail - many md viewers will interpret ---- as a H2 making it really large, see:
https://markdownlivepreview.com
https://codebeautify.org/markdown-viewer

This one seems to understand it's supposed to be HR though:
https://dillinger.io

If the intention is for it to be a HR, it should look like:


----
title: The Title

date: 2023-06-26T13:27:56.132Z

----

Also Markdown doesn't really "do" line breaks, so the heading might be all in one line:
title: The Title date: 2023-06-26T13:27:56.132Z
And using <br> or \ is nasty so better to separate lines.

Ref:
https://commonmark.org/help/
https://www.markdownguide.org/basic-syntax/

daniel

@Pathduck This is a bug. It was supposed to be three dashes (horizontal rule) and not four. Nice catch.

The metadata preamble uses Front Matter syntax. It’s intended to be both machine- and human-readable to allow for easier conversion between Vivaldi Notes and other formats. The current format (with three dashes) is compatible with Joplin, and other open-source Markdown-focused notes editors.

ldexterldesign

@daniel,

Thanks for feature

Will import/export attachments function without Vivaldi Sync (i.e. simply export image alongside .md file)?: if no then may I ask why not and suggest it's the default with sync an opt in

Aside, I invite Vivaldi to adopt [local first] development

Hope to hear back

Cheers

[local first]: https://localfirstweb.dev < https://syntax.fm/show/739/the-lofi-movement-building-local-first-apps

daniel

@ldexterldesign It will work without Sync, yes. The import/export of attachments is pending on Sync support because the implementation of attachments internally is expected to change. Along with that change, we expect to add some limitations on file types and file sizes. Time spent right now on import/export for attachments would most likely need to be reinvested right after the upcoming changes to Sync.

We’re doing incremental improvements.

antor124

This post is deleted!