If you still want to use morningstar, an alternative is to use selenium to load the pages and extract the data.
Thanks for the hint.
I don’t have much time for this at the moment, but I am thinking in the following direction:
- Abandon the Direct Web Services and return to the old SAL service interface (which is also able to provide json). Originally, I turned to Direct Web Services, mainly because they were well documented. With the SAL service interface, it is not so easy, but earlier versions of the script give a lot of hints on what to find where. I have seen for example that
https://api-global.morningstar.com/sal-service/v1/etf/portfolio/v2/sector/<secid>/dataprovides stock sector allocation for an ETF with Morningstar secid (if one has the right authorization token). - Try to find a way to map an ISIN to security type and Morningstar secid. In worst case, user has to manually maintain a mapping table. But ideally, I can find some way to automate it. Manually, the information can be obtained from:
https://global.morningstar.com/de/search/securities?query=<isin>. - Most tricky part is to get a valid authorization token for step 1. If I open a page like
https://global.morningstar.com/en-eu/investments/etfs/<secid>/portfolio, I can find a valid token in the html of the page (<sal-components data-v-aadb39c8="" access-token="eyJhbGci . . . . . . ."). Manually copying it to the script for temporary usage would be an option, though not a very user-friendly one. Therefore, I am hoping to find a way to extract it automatically like in the old version of the script from fbuchinger. But my scraping competences are very limited :-).
Any hints on any of these three topics, might be helpful.
Just to let you all know:
I have now modified the “old” main branch. It is now again capabale of retrieving etf, fund and stock information from Morningstar pages. However, it is not very user friendly (see points 2 and 3 above). User has to manually put a valid authentication token into the code and has to manually maintain isin2secid.json mapping. It is a start and I wanted to share it as soon as possible.
The main branch is older and has different features than new-api-branch. I haven’t fixed the new-api-branch yet, but plan to do so.
Thanks for this. I hope that it’ll be updated to be noob friendly. lol
:-)) Thanks @JackOff. Actually, I think I will leave it as it is for now. I got used to it and I even like that I have better control over isin2secid mapping.
Hallo Alfons,
ich habe mal basierend auf deinem ursprünglichen Programm ein automatisierte Lösung in Phyton gebaut (das läuft so weit automatisch):
- Playwright besucht eine Morningstar ETF-Portfolio-Seite und extrahiert den JWT-Token aus dem HTML.
- Neuer API-Endpunkt: api-global.morningstar.com/sal-service/v1/… (direkte Requests-Aufrufe funktionieren mit dem Token).
- ISIN > SecID Mapping: Suche auf global.morningstar.com/de/search/securities?query={ISIN}, dann Validierung jedes Kandidaten via Metadaten-API
(prüft isin Feld).
Einschränkungen, gegenüber der bisherigen Morningstar-API:
- Holdings auf 25 begrenzt (Morningstar-Free-Version) — Top 50/100/etc. nicht mehr möglich
- Es wird ein Browserfenster geöffnet um den Token zu holen
- Industry-Taxonomie: Kein Endpunkt in der neuen API. Für mich das grösste Problem. Hat jemand eine Idee wo wir die Insustrieklassifikation bekommen können?
Ich hatte Andreas schon mal vorgeschlagen, den Code in PP direkt einzubinden, denn, wenn alle ständig das ISIN > SecID Mapping abfragen, dann wird Mornigstar diesen Zugang bald wieder blockieren. Das selbe gilt für die Datenabfrage bei global.morningstar.com. Wenn wir die Daten im Cache bei PP ablegen würden und nur einmal im Monat aktualisieren, dann könnten die Morningstarabfragen deutlich reduziert werden.
Hi @Carsten,
A few thoughts/comments:
Currently, my version of the script does not request the isin to secid mapping at all. The user has to manage it more or less manually and it is permanently stored in isin2secid.json. Manageing/storing it directly in PP would indeed be nice to have.
To avoid too many unnecessary request, I think your automatic version could also store the obtained mapping in isin2secid.json and update it only occassionally.
Not sure, if I understand your industry taxonomy issue, but for stocks, you can get sector and also industry from: https://api-global.morningstar.com/sal-service/v1/stock/equityOverview/{SECID}/data.
Hi Alfons,
Storing the mapping: in my version I am storing the Isin2SecID mapping locally, but if all user doing that regularly, we will see quite a number of API-calls. I don’t know how often this is changed by Morningstar. Is there a need to update this on regular basis?
Industry taxonomy: you are right about stocks, but my problem are not the stocks I am more interested in ETFs and funds. Those “look through data” were availible on the old API for ETFs and funds. Based on my research this data is only available behind a pay wall, eg. EOHD.com. Will probably integrate this as an option.
Hi Alfons,
getting now all the relevant Holdings per ETF/Fund from the API (no 25 limit anymore). With this I have integrated a calculation for the Industry allocation. no need to use EOHD anymore.
will do some testing in the next days
Hi @Carsten ,
Please share the details how you are using the API to get these results.
Hi Mate, sorry for being a pain, but is there a way that your update be available to noobs like me 
?
Hi @JackOff, in the meantime, I have understood how @Carsten retrieved large number of holdings per etf/fund. It is already integrated into main branch of my version of the script (command line option -top_holdings). In addition to that, he has expanded the script to also fetch industry mapping per holding and to derive industry mapping for etf/fund as a whole from that, if I understand him correctly. This is not part of my version of the script, but maybe I will experiment with that in future. (Generally, I would expect that people should be able to extend the script easily by themselves for such special purposes with (or without) the help of some LLM. Maybe even noobs ;-).