POST search engine

sedrossetti

I configured StartPage search engine to use POST as it is more secure and private.
I found that if you want to stop using POST, the configuration is not updated after you uncheck the checkbox. I had to delete the search engine, an add it again.

Pathduck

@sedrossetti Hi - please always give your Vivaldi version from Help > About when posting about issues/bugs. I am assuming this is for Snapshot 5.3 releases., with the new search engine implementation?

I can confirm this happens in 5.3 Snapshot (not in Stable), when trying to change a search engine from POST to GET.

Workaround: remove POST parameter before changing method.

Please read:

carefully and report the bug to the Vivaldi bugtracker

sedrossetti

@Pathduck sorry.
I'm having this error in Vivaldi Snapshot
5.3.2679.25 (Build oficial) (arm64)

Pathduck

@sedrossetti Please report this bug and post the VB-number you get here.

sedrossetti

@Pathduck said in POST search engine:

How to report a bug for Vivaldi

This is the bug report:
VB-89553

pafflick

@sedrossetti We were unable to reproduce your issue (tested in Snapshot 5.3.2679.30 and in a newer, internal build). Could you please clarify if you have updated the URL field as well, or did you just uncheck the "Use POST" checkbox?

Pathduck

@pafflick Hi, there's definitely an issue here, and relatively easy to reproduce in latest Stable (just released):

Create a new search engine for Startpage using POST:
URL: https://www.startpage.com/do/search
POST param: q=%s
Use POST
Test it and see it uses POST
Uncheck Use POST and change URL to regular:
https://www.startpage.com/do/search?q=%s
Test again
Notice that address bar now reads:
https://www.startpage.com/do/search?q=test+this
So one would think this was a GET request but it's not.
Press Ctrl+R to reload. "Confirm form resubmission" pops up as it will for POST requests.
Open Devtools and reload again. Look at headers for request, it still uses POST:

Request URL: https://www.startpage.com/do/search?q=test+this
Request Method: POST
Status Code: 200 OK
Remote Address: 37.0.87.39:443
Referrer Policy: strict-origin-when-cross-origin

Notice also that request contains both query string arguments and form data. Startpage seems to take the query string first, but the request is still POST.

The user will expect that unchecking "Use POST" will make the search use GET without having to first remove the POST arguments.

pafflick

@Pathduck Startpage might not be the best example, as it uses POST by default - meaning that this scenario (searching via GET, eg. https://www.startpage.com/do/search?q=test+this and the request being sent via POST regardless) is reproducible in every single browser. To use GET, you need to go to Startpage settings and change the HTTP request method under Privacy and Safety.

I have checked this with another search engine and I can confirm that the POST parameter is being used when it probably shouldn't - but that's not what the bug report I was replying to seems to be about.

Pathduck

@pafflick Well, the setting in Startpage is only for searching from the page itself. Obviously when using a browser search engine it just send a GET/POST request directly to the site.

If you want I can open a new one with the info I gave above to make sure it's reproducible?

There's also issues with encoding when using POST that I've reported in VB-88767, would be nice to get a confirmation of that one

pafflick

@Pathduck said in POST search engine:

the setting in Startpage is only for searching from the page itself. Obviously when using a browser search engine it just send a GET/POST request directly to the site.

That doesn't seem to be the case. If you change the settings to use GET (Startpage settings) and send a query using POST (Vivaldi settings), it will change to GET automatically, and you'll see your query in the URL. Similarly, if you set up Vivaldi to use GET but don't change the Startpage settings to also use GET, it will change to POST automatically. That's what happens in Vivaldi and other browsers. It's controlled by the website, not the browser.

@Pathduck said in POST search engine:

If you want I can open a new one with the info I gave above to make sure it's reproducible?

You can do that, of course - but I'm not sure if that should be fixed, actually. If we "fix" it and an inexperienced user deselects the "Use POST" checkbox (without adjusting the URL parameter accordingly), it'll entirely break their search engine, while with the current "bug" it'll remain operational (though the behaviour will be unexpected).

Not sure if that's the case brought up in this thread - we have yet to hear from @sedrossetti to clarify it - otherwise, the bug report will have to be closed as unreproducible.

@Pathduck said in POST search engine:

VB-88767

I've confirmed it internally yesterday.

Pathduck

@pafflick said in POST search engine:

That doesn't seem to be the case. If you change the settings to use GET (Startpage settings) and send a query using POST (Vivaldi settings), it will change to GET automatically, and you'll see your query in the URL.

Hmm, not so sure

See here, changing SP setting to use GET and then calling SP from an engine: https://ttm.sh/bV8.mp4

If we "fix" it and an inexperienced user deselects the "Use POST" checkbox (without adjusting the URL parameter accordingly), it'll entirely break their search engine

That's true, but I don't think we can expect inexperienced users to even understand what unchecking POST does I think still sending POST requests even if the option is unselected is very inconsistent.

I've confirmed it internally yesterday.

Thanks

BTW I reported as:
VB-89665

PS: Ignore the PornBB engine I have there, it's for testing another case , it's true I swear!

pafflick

@Pathduck said in POST search engine:

See here, changing SP setting to use GET and then calling SP from an engine: https://ttm.sh/bV8.mp4

I have tried this like 8 times now, in Stable, Snapshot and in an internal build, and I cannot reproduce it. For me, there's always GET in the header, even though the search engine in the browser is set to use POST.
Perhaps the culprit is that I'm using q=%s,segment=startpage.vivaldi in the POST parameter, as mere q=%s redirects me to some CAPTCHA form and refuses to perform any searches even after resolving it multiple times, so I have no way of testing it otherwise.

sedrossetti

Hello. I apologize for my delay.
After trying more, I found this:

For GET searches, you should use this URL for Startpage: https://www.startpage.com/sp/search?q=%s. If you open the Developer Tools, and go to Network and perform a search, you will find that a GET request was done. If you use https://www.startpage.com/do/search it will perform a POST request.
Try the Startpage Privacy Tools extension which adds Startpage as a search engine, and look for the configuration.
If I want to move to GET requests, instead of POST, in Vivaldi Preferences, I have to delete the query parameters in order to have the modification saved. If I don't delete the query parameters, the change to use GET doesn't get saved.

sedrossetti

@pafflick I'am being redirected to CAPTCHA form, but if I am in my work VPN.

sedrossetti

Other thing I found is that, even Startpage works with POST for searching, it uses GET for suggestions. So, if you configure to use POST, you are not able to set GET for suggestions. Even if you leave the params blank for the suggestions and add the query params in the URL