Parse data with written Date

[Wick]

Hello!
I was trying to add https://bondblox.com/api/fetchGraphData?requestSource=Mobile&requestApplication=SEO&requestDevice=Desktop&isin=XS2187711572&showLatest=false values to PP, using $.price[*][0] for date and $.price[*][1] for price as I do usually, but this date I retrieve no data.

I guess that’s due to the written date format that can’t be understand: “Wed Jun 03 2020 00:00:00 GMT+0000 (Coordinated Universal Time)”

It looks like it’s most similar with the format: EEE MMM dd yyyy HH:mm:ss ‘GMT’Z ‘(’‘Coordinated Universal Time’’)’ DateTimeFormatter (Java Platform SE 8 ) but it’s not working

Is there still a way to add this using a particular format date?

Thanks a lot

[ProgFriese]

@Wick

Is there still a way to add this using a particular format date?

Fascinating problem

I am in locale “de_DE”

When I try to test your problem, I get the same failure

“could not be parsed at index 0”

DateFormat EEE MMM dd yyyy HH:mm:ss 'GMT'Z (zzzz) was not my idea, it is from gemini.

Thinking about daynames and monthnames in my locale and in your string, I try the following DateFormat EEE MMM dd yyyy HH:mm:ss 'GMT'Z (zzzz)[en], but nothing changed.

Changing the PP settings to “en_GB”

Try again

Now “could not be parsed at index 4” (this means IMHO that “Tue Jun 09 2020 00:00:00 …” was parsed with success?)

Maybe the used java_locale have “Sept” instead of “Sep” for September?

Maybe @AndreasB has an idea?

[ProgFriese]

@ProgFriese

java_locale

Fascinating.

@AndreasB Is it possible to override java_locale (from preferences) by specifying a date format for the JSON provider, e.g.

EEE MMM dd yyyy HH:mm:ss 'GMT'Z (zzzz)

use the origin java_locale from preferences

EEE MMM dd yyyy HH:mm:ss 'GMT'Z (zzzz)[en_US]

use a special wanted java_locale

[Wick]

Interesting !

If I change my language to put that then go back to another language will it still work or not?

[ProgFriese]

@Wick

will it still work or not?

or not, I suppose

[Wick]

Then specifying a date format would be a great option indeed

[ProgFriese]

I have looked around in the code, seems to be
portfolio-0.80.3/name.abuchen.portfolio/src/name/abuchen/portfolio/online/impl/GenericJSONQuoteFeed.java

For that code I asked gemini

Answer: It has to work with the code of 0.80.3

But it does not work for me. The AI is talking nonsense again.

The idea to use MMMM insted of MMM was already tried, without sucess.

AI is a total asset, says my boss. He’s never tried it himself.

[AndreasB]

@ProgFriese I believe line number is 253 is the relevant one. The date format is created using the locale that the program is used to run. In so far Gemini is correct…

I think we would need a new property, locale, which is used to create the data formatting pattern. If it is not set, then we fall back to the current default locale

@Wick

If I change my language to put that then go back to another language will it still work or not?

No. The date pattern is created at the time when the data is downloaded. That locale is used.

[ProgFriese]

@AndreasB

I think we would need a new property, locale, which is used to create the data formatting pattern. If it is not set, then we fall back to the current default locale

That sounds as if it wouldn’t be difficult for you to implement.

[AndreasB]

I found Claude Code not to be so bad in implementing this:

With the next version:

[Wick]

Thanks for the great solution from both of you ! I’m eager to download the new update

[hporten]

I am a bit puzzled by the fact that the user has to specify a date format let alone a locale. JSON dates can be parsed in a locale-agnostic way, afair. This was the case when I authored a date parser over a decade ago at least. The code ended up in Apples Safari so it was working well enough for lots of users.

See the parseDate() function in /kf5-qt5/frameworks/kjs/src/kjs/date_object.cpp The only human-readable month specification that we hardcoded was the English one (see findMonth()).

I do lack the historic knowledge of the needs in PP however. There possibly was a need for the introduction of the Date Format field? Due to the limitations of the Java date parsing code maybe?

[AndreasB]

@hporten

I am a bit puzzled by the fact that the user has to specify a date format let alone a locale. JSON dates can be parsed in a locale-agnostic way, afair.

Dates come in all shapes and forms apparently. Sometimes as numbers - in milliseconds, or seconds, or days - and sometimes as strings - as ISO dates, with time or without, with timezone or without - with an assumed timezone or as UTC.

Most of those cases, PP attempts to derive a reasonable way to parse the dates. To handle, edge cases, the (optional) configuration was added.

In this particular case, it was in the form of “Wed Jun 03 2020 00:00:00 GMT+0000 (Coordinated Universal Time)”.

Thinking about it, I agree that it can make sense to always use the en_US locale to parse dates (instead of the default locale). Only if the dates are then provided in - say German - locale the user could override. But I admit that is less likely for an API that is less likely.

[AndreasB]

One more things comes to mind: We should change the UI so that the mandatory fields are marked (URL, JSON path to date and price). All other things are optional.

[hporten]

Thanks for the quick reply. And sorry for the late input. I got the IDE working only two days ago.

The lack of real date format standardization is a big pita indeed. Getting the cited C++ -based parser compatible with real world string representations took ages and required many exceptions and fault tolerance.

The format encountered by the reporter of this issue is trivial in comparison though. It should not require any extra configuration options. Here is what my Web browser makes out of the string:

new Date(“Wed Jun 03 2020 00:00:00 GMT+0000 (Coordinated Universal Time)”)

Date Wed Jun 03 2020 02:00:00 GMT+0200 (Central European Summer Time)

Except of really buggy implementations it should be safe to assume that JSON as emitted by a provider can be parsed by a Web browser or any other JSON client without any extra format and locale specification. Otherwise, a lot of other client software would already be in trouble.

Oh, well. It seems to be too late for the next release. But I will keep an eye on this to see whether the user experience can be simplified. As you suggested, the marking of mandatory fields would be a first step.

[ProgFriese]

@AndreasB

Implementation completed

Wow. You solve problems faster than Lucky Luke shoots his shadow.

[kimmerin]

@ProgFriese

The idea to use MMMM insted of MMM was already tried, without sucess.

Even with the new feature in PP allowing you to specify a Locale you might still have parsing issues, if the abbreviated month is german. The template-character M means the “context sensitive” month (see SimpleDateFormat (Java Platform SE 8 ) while there is another template - L - for the “standalone” month. The examples don’t help because there is no difference between those two with en_US (I’ve already complained about this in i18n-dev but to no avail). When using de_DE as Locale you e.g. get Sep. when using the context sensitive version. This is also expected when parsing a date which might explain why parsing fails even with months that share the same abbreviated form.