[VB-95824] Emails with invalid date disrupt order of sorting by date

EricJH

As the title states. I have emails sorted descending by date. This gets broken when I receive emails with an invalid date according to Vivaldi mail client.

With tinkering I can temporarily get the order sorted it out. It takes fiddling with changing the sorting order back and forth, go to the top and bottom of the list vice versa and enabling showing and hiding emails that are in trash vice versa. All in random order. I don't have exact steps to reproduce the temporary fixing of sorting.

Steps to reproduce:

Sign up for one or more newsleteters of NRC news paper: https://www.nrc.nl/nieuwsbrieven/
After receiving the emails try finding them using the search function of mail. They will not be found
Restart Vivaldi
Open the mail client and use search to find them to make sure they are processed
End search and make sure Vivaldi mail is set to sort by Date descending
The sorting order will look broken
Find the offending emails and notice they break the sorting
Find the offending emails in a separate folder C:\Users%Username%\AppData\Local\Vivaldi\User Data\Profile 1\Mail######\NaN\NaN\NaN
To get the emails sorted properly temporarily it takes fiddling with: changing the sorting order back and forth, go to the top and bottom of the list and vice versa and enabling showing and hiding emails that are in trash etc. I don't have exact steps to reproduce the fixing of the sorting.

On both the Snapshot and Vivaldi stable builds on Windows 10 (19.45.2604) I made a new profile and the problem happens with both.

Builds:
Vivaldi Snapshot build v5.8.2950.3
Vivaldi stable v5.7.2921.60

I have witnessed this behavior on three different installs of Vivaldi Snapshots on three different Windows installations.

The offending emails are stored in a separate folder:

C:\Vivaldi Snapshot\User Data\Profile 1\Mail######\NaN\NaN\NaN
C:\Users%Username%\AppData\Local\Vivaldi\User Data\Profile 1\Mail######\NaN\NaN\NaN

Vivaldi invalid date.png

Vivaldi invalid date bottom.png

Vivaldi invalid date middle.png

Vivaldi invalid date single.png

mib2berlin

@EricJH
Hi, I test this at moment, if I can reproduce it I can confirm your report.
There was a lot of mail fixes in the last two snapshots (Beta) builds, may you check this with a standalone install.

https://help.vivaldi.com/desktop/install-update/standalone-version-of-vivaldi/

Cheers, mib

yngve

We need more details to be able to debug this: Examples of the dates in the emails, which you can find in the raw version (right click on the email, choose raw, look for the Date header).

Please reply to the confirmation email you got for the bug you filed with the information.

Usually, though, all such invalid dates are in spams.

EricJH

@mib2berlin I made a separate installation of the latest snapshot, v5.8.26950.3, and the problem reproduces as in the attached photos.

After installing and fetching the emails all shows fine. After restarting Vivaldi things start happening. Initially the emails from NRC show up as listed but when selected they jump from view within two seconds. I have noticed that before when the problem started to happen.

After I had selected the email they showed up at the bottom. After restarting Vivaldi one more time they showed up exactly as in the attached snapshots; at the same place in time.

EricJH

@yngve Thank you for getting back to me.

The problem started to emerge around February 28 or March 1. I have been receiving the newsletters of NRC news paper for several years so the emails are by no means spam mails.

My guess is that there is a configuration error on the side of NRC which in turn trips out the Vivaldi mail client.

By email I have sent a zip archive with 8 offending emails. I hope that provides the requested information.

Let me know if you need any more information.

mib2berlin

@EricJH
I sign up for a newsletter but don´t get the verification mail, I will check tomorrow again.

Cheers, mib

yngve

@EricJH Thanks.

Oh, .... My .... that's ... one bad date (according to the ISO format):

Date: wo, 01 mrt 2023 17:17:22 +0100

Best guess: Somebody is using localized month and day names.

EricJH

@yngve said in [VB-95824] Emails with invalid date disrupt order of sorting by date:

@EricJH Thanks.

Oh, .... My .... that's ... one bad date (according to the ISO format):
Date: wo, 01 mrt 2023 17:17:22 +0100
Best guess: Somebody is using localized month and day names.

There's the naughty. That's Dutch right there. How does the ISO format prescribe the format to be? I guess it would be in English.

That's the first part. Now for the second part. Why does Vivaldi mail client mess up the sorting order? It's annoying.

yngve

@EricJH No idea about that (I don't normally dig around in the mail code); my guess is that there is some kind of interaction between the information for the code processing the date, and the sorting. It might be a case of GIGO; bad inputs cause bad outputs. One possibiliity is that the sorting algorithm does not handle a NaN (Not A Number) "properly" (Assuming there is a "proper" way to handle them) as one of the dates being compared, and as a result the entries get badly sorted. (The sorting algorithm most commonly used is called QuickSort, and unless I am mistaken it could easily cause this kind of mess due to how it works)

EricJH

@yngve I will await further instructions through email in case QA wants more information.

Are you 100% certain that the date format if not according to the mentioned ISO standards? I could connect with NRC and notify them of possible error on their end. If they stop sending emails with invalid date it will mitigate the problem on my end.

yngve

@EricJH A correction; the date is not an ISO date (that is YYYY-MM-DD), sorry, but the format of the date string in email has been fixed for more than 40 years.

From RFC 822 (from 1982) one of the original email format specifications:

     5.  DATE AND TIME SPECIFICATION
     
     5.1.  SYNTAX
     
     date-time   =  [ day "," ] date time        ; dd mm yy
                                                 ;  hh:mm:ss zzz
     
     day         =  "Mon"  / "Tue" /  "Wed"  / "Thu"
                 /  "Fri"  / "Sat" /  "Sun"
     
     date        =  1*2DIGIT month 2DIGIT        ; day month year
                                                 ;  e.g. 20 Jun 82
     
     month       =  "Jan"  /  "Feb" /  "Mar"  /  "Apr"
                 /  "May"  /  "Jun" /  "Jul"  /  "Aug"
                 /  "Sep"  /  "Oct" /  "Nov"  /  "Dec"
     
     time        =  hour zone                    ; ANSI and Military
     
     hour        =  2DIGIT ":" 2DIGIT [":" 2DIGIT]
                                                 ; 00:00:00 - 23:59:59
     
     zone        =  "UT"  / "GMT"                ; Universal Time
                                                 ; North American : UT
                 /  "EST" / "EDT"                ;  Eastern:  - 5/ - 4
                 /  "CST" / "CDT"                ;  Central:  - 6/ - 5
                 /  "MST" / "MDT"                ;  Mountain: - 7/ - 6
                 /  "PST" / "PDT"                ;  Pacific:  - 8/ - 7
                 /  1ALPHA                       ; Military: Z = UT;
                                                 ;  A:-1; (J not used)
                                                 ;  M:-12; N:+1; Y:+12
                 / ( ("+" / "-") 4DIGIT )        ; Local differential
                                                 ;  hours+min. (HHMM)

from the current version, RFC 2822 (published 2001):

3.3. Date and Time Specification

   Date and time occur in several header fields.  This section specifies
   the syntax for a full date and time specification.  Though folding
   white space is permitted throughout the date-time specification, it
   is RECOMMENDED that a single space be used in each place that FWS
   appears (whether it is required or optional); some older
   implementations may not interpret other occurrences of folding white
   space correctly.

date-time       =       [ day-of-week "," ] date FWS time [CFWS]

day-of-week     =       ([FWS] day-name) / obs-day-of-week

day-name        =       "Mon" / "Tue" / "Wed" / "Thu" /
                        "Fri" / "Sat" / "Sun"

date            =       day month year

year            =       4*DIGIT / obs-year

month           =       (FWS month-name FWS) / obs-month

month-name      =       "Jan" / "Feb" / "Mar" / "Apr" /
                        "May" / "Jun" / "Jul" / "Aug" /
                        "Sep" / "Oct" / "Nov" / "Dec"

day             =       ([FWS] 1*2DIGIT) / obs-day

time            =       time-of-day FWS zone

time-of-day     =       hour ":" minute [ ":" second ]

hour            =       2DIGIT / obs-hour

minute          =       2DIGIT / obs-minute

second          =       2DIGIT / obs-second

zone            =       (( "+" / "-" ) 4DIGIT) / obs-zone

From RFC 5322 (pub 2008):

3.3.  Date and Time Specification

   Date and time values occur in several header fields.  This section
   specifies the syntax for a full date and time specification.  Though
   folding white space is permitted throughout the date-time
   specification, it is RECOMMENDED that a single space be used in each
   place that FWS appears (whether it is required or optional); some
   older implementations will not interpret longer sequences of folding
   white space correctly.

  date-time       =   [ day-of-week "," ] date time [CFWS]

   day-of-week     =   ([FWS] day-name) / obs-day-of-week

   day-name        =   "Mon" / "Tue" / "Wed" / "Thu" /
                       "Fri" / "Sat" / "Sun"

   date            =   day month year

   day             =   ([FWS] 1*2DIGIT FWS) / obs-day

   month           =   "Jan" / "Feb" / "Mar" / "Apr" /
                       "May" / "Jun" / "Jul" / "Aug" /
                       "Sep" / "Oct" / "Nov" / "Dec"

   year            =   (FWS 4*DIGIT FWS) / obs-year

   time            =   time-of-day zone

   time-of-day     =   hour ":" minute [ ":" second ]

   hour            =   2DIGIT / obs-hour

   minute          =   2DIGIT / obs-minute

   second          =   2DIGIT / obs-second

   zone            =   (FWS ( "+" / "-" ) 4DIGIT) / obs-zone

I think you will agree that the specification is VERY clear on this point.

This format is also the default one used in HTTP.

EricJH

I have reported the issue with NRC customer services referring to this topic. I hope it gets fixed on their end.

WildImpresario

This is still happening in the current version of Vivaldi mail client. What's worse, trying to delete the message seems to cause a different message to be deleted (the cursor jumps to a different place on the list of headers when the "delete" option is picked).

In this case, the invalid date is:

Date: Sat, 21 Sep 2024 24:21:03 +0100

(i.e., > 24 hours)

Note: I say "seems" because after closing and reopening Vivaldi, the message that appeared to have been (incorrectly) deleted seems to be back. But whenever the "invalid" message is deleted, you get a mismatch between the message list and the message content, and after that point (unless you close and reopen Vivaldi) it's very hard to know what you're actually deleting.

It seems that Vivaldi mail is using the actual message ID (file reference?) for some operations, but using its position on the sorted list for other operations.

This is a problem with Vivaldi; it needs to be able to handle malformed data.

edwardp

@WildImpresario The bug report referenced above, is closed, won't fix.

If the data coming from the mail provider is malformed, then in my opinion, it's not Vivaldi's problem to fix. The mail provider needs to address it.

WildImpresario

The "mail provider" in this case was a spammer sending phishing e-mails, and possibly deliberately sending malformed data to try to exploit mail clients that can't handle it properly.

It very much is Vivaldi's problem.

No other mail client that I've used has this issue (including the old Opera), they all handle it fine.

Trusting internet data to always be well formed is security suicide.

yojimbo274064400

@WildImpresario,

~~FWIW~~ ~~I could not reproduce the invalid date issue. For example following shows 3 messages where:~~

~~Date: header value of Sat, 21 Sep 2024 24:21:03 +0100 is invalid~~
~~Date: header does not exist~~
~~Date: header value is Sat, 21 Sep 2024 23:21:03 +0100~~

~~The former two do not show as invalid date but instead the receipt time and date.~~

WildImpresario

So you're saying that even when a message has an invalid date, Vivaldi doesn't warn you about it?

Are you using POP3 or IMAP?

Using POP3 (all mail stored locally), Vivaldi 6.9.3447.46 (Stable channel) (64-bit), Windows:

yojimbo274064400

@WildImpresario said in [VB-95824] Emails with invalid date disrupt order of sorting by date:

So you're saying that even when a message has an invalid date, Vivaldi doesn't warn you about it?
⋮

I was but no longer after reviewing the process used for the invalid date; sorry.

I will re-examine when time allows

yojimbo274064400

@WildImpresario,

When using POP3 do you see errors similar to the following
and then find correlation between it and the appearance of messages with invalid Date: headers after restarting Vivaldi?

For example, I saw one such error and on restarting Vivaldi a new message appeared in Inbox folder without further errors in log.

The jumping of selected message issue seems to affect Unseen, Read, and Unread messages which have an invalid date and change of state. For example, following shows three emails with invalid date. Toggling last message in list's read status results in it jumping to top of list as shown below:

When using IMAP there is no indication of invalid date and whilst the jumping of message issue does not occur resync with server results in duplicate message. For example marking message 2 of 3 as read and then resync results in replication of existing message (1 to 3) despite server mailbox contain just 3 messages as shown below:
.

Have you raised a bug report for the issue of jumping messages when using POP3?

yojimbo274064400

FYI following bug reports raised for issue discussed in this thread:

(VB-110006) IMAP option Advanced > Re-sync folder with server results in duplication of messages with invalid Date: header on client
(VB-110007) POP Inbox folder's list of message re-ordered when status of message with invalid Date: header changes