Automatic import of classifications

If you still want to use morningstar, an alternative is to use selenium to load the pages and extract the data.

Thanks for the hint.

I don’t have much time for this at the moment, but I am thinking in the following direction:

  1. Abandon the Direct Web Services and return to the old SAL service interface (which is also able to provide json). Originally, I turned to Direct Web Services, mainly because they were well documented. With the SAL service interface, it is not so easy, but earlier versions of the script give a lot of hints on what to find where. I have seen for example that https://api-global.morningstar.com/sal-service/v1/etf/portfolio/v2/sector/<secid>/data provides stock sector allocation for an ETF with Morningstar secid (if one has the right authorization token).
  2. Try to find a way to map an ISIN to security type and Morningstar secid. In worst case, user has to manually maintain a mapping table. But ideally, I can find some way to automate it. Manually, the information can be obtained from: https://global.morningstar.com/de/search/securities?query=<isin> .
  3. Most tricky part is to get a valid authorization token for step 1. If I open a page like https://global.morningstar.com/en-eu/investments/etfs/<secid>/portfolio, I can find a valid token in the html of the page (<sal-components data-v-aadb39c8="" access-token="eyJhbGci . . . . . . ."). Manually copying it to the script for temporary usage would be an option, though not a very user-friendly one. Therefore, I am hoping to find a way to extract it automatically like in the old version of the script from fbuchinger. But my scraping competences are very limited :-).

Any hints on any of these three topics, might be helpful.

Just to let you all know:

I have now modified the “old” main branch. It is now again capabale of retrieving etf, fund and stock information from Morningstar pages. However, it is not very user friendly (see points 2 and 3 above). User has to manually put a valid authentication token into the code and has to manually maintain isin2secid.json mapping. It is a start and I wanted to share it as soon as possible.

The main branch is older and has different features than new-api-branch. I haven’t fixed the new-api-branch yet, but plan to do so.

Thanks for this. I hope that it’ll be updated to be noob friendly. lol

:-)) Thanks @JackOff. Actually, I think I will leave it as it is for now. I got used to it and I even like that I have better control over isin2secid mapping.

Hallo Alfons,

ich habe mal basierend auf deinem ursprünglichen Programm ein automatisierte Lösung in Phyton gebaut (das läuft so weit automatisch):

  1. Playwright besucht eine Morningstar ETF-Portfolio-Seite und extrahiert den JWT-Token aus dem HTML.
  2. Neuer API-Endpunkt: api-global.morningstar.com/sal-service/v1/… (direkte Requests-Aufrufe funktionieren mit dem Token).
  3. ISIN > SecID Mapping: Suche auf global.morningstar.com/de/search/securities?query={ISIN}, dann Validierung jedes Kandidaten via Metadaten-API
    (prüft isin Feld).

Einschränkungen, gegenüber der bisherigen Morningstar-API:

  • Holdings auf 25 begrenzt (Morningstar-Free-Version) — Top 50/100/etc. nicht mehr möglich
  • Es wird ein Browserfenster geöffnet um den Token zu holen
  • Industry-Taxonomie: Kein Endpunkt in der neuen API. Für mich das grösste Problem. Hat jemand eine Idee wo wir die Insustrieklassifikation bekommen können?

Ich hatte Andreas schon mal vorgeschlagen, den Code in PP direkt einzubinden, denn, wenn alle ständig das ISIN > SecID Mapping abfragen, dann wird Mornigstar diesen Zugang bald wieder blockieren. Das selbe gilt für die Datenabfrage bei global.morningstar.com. Wenn wir die Daten im Cache bei PP ablegen würden und nur einmal im Monat aktualisieren, dann könnten die Morningstarabfragen deutlich reduziert werden.

Hi @Carsten,

A few thoughts/comments:

Currently, my version of the script does not request the isin to secid mapping at all. The user has to manage it more or less manually and it is permanently stored in isin2secid.json. Manageing/storing it directly in PP would indeed be nice to have.

To avoid too many unnecessary request, I think your automatic version could also store the obtained mapping in isin2secid.json and update it only occassionally.

Not sure, if I understand your industry taxonomy issue, but for stocks, you can get sector and also industry from: https://api-global.morningstar.com/sal-service/v1/stock/equityOverview/{SECID}/data.

Hi Alfons,

Storing the mapping: in my version I am storing the Isin2SecID mapping locally, but if all user doing that regularly, we will see quite a number of API-calls. I don’t know how often this is changed by Morningstar. Is there a need to update this on regular basis?

Industry taxonomy: you are right about stocks, but my problem are not the stocks I am more interested in ETFs and funds. Those “look through data” were availible on the old API for ETFs and funds. Based on my research this data is only available behind a pay wall, eg. EOHD.com. Will probably integrate this as an option.

Hi Alfons,

getting now all the relevant Holdings per ETF/Fund from the API (no 25 limit anymore). With this I have integrated a calculation for the Industry allocation. no need to use EOHD anymore.

will do some testing in the next days

Hi @Carsten ,

Please share the details how you are using the API to get these results.

Hi Mate, sorry for being a pain, but is there a way that your update be available to noobs like me :slight_smile: ?

Hi @JackOff, in the meantime, I have understood how @Carsten retrieved large number of holdings per etf/fund. It is already integrated into main branch of my version of the script (command line option -top_holdings). In addition to that, he has expanded the script to also fetch industry mapping per holding and to derive industry mapping for etf/fund as a whole from that, if I understand him correctly. This is not part of my version of the script, but maybe I will experiment with that in future. (Generally, I would expect that people should be able to extend the script easily by themselves for such special purposes with (or without) the help of some LLM. Maybe even noobs ;-).

I am currently experimenting with the script. I get a 206 error code on most mutual funds and ETFs. Anyone have encountered the problem and solved it? For example:

@ Retrieving data for fund CA92205P1045 (0P0000ZAH3) …
[Vanguard FTSE Canada All Cap Index ETF]

Return Response [206] with resp.text: “adf57ec0-8b58-43da-b669-3289142c1cc0\n202102 Can’t get SecurityInfo with 0P0000ZAH3”.

I tried switching locale to en-CA, but it does not help. I do find the ETF on morningstar.ca.

I just went to validate my access token in the browser DevTools, and I noticed that in that case the URL in ends with “&secId=F00000QA9G”, while the broswer URL still shows 0P0000ZAH3. Morningstar doing some internal conversion?

Edit: I performed further testing using F00000QA9G. It suppresses the 206 error, but no data is returned.

After further testing, using the F… secid instead of the 0P… works for most ETFs and Mutual funds. I still have a few ETFs/MFs where I get no errors, but just a message that there is no % for sectors, regions, etc. while I can find the full info on Morningstar, for example Vanguard FTSE Canada All Cap Index ETF.

Hi,

I have the same experience. Sometimes a SecId becomes useless and another one pops up (including internal ones not directly visible in search). In such case, I replace it in isin2secid.json and it usually works well (at least for some time). This manual maintenance of SecIds has the advantage that user can easily adapt and that the results are better determined, but of course, it is extra effort.

About missing information for some ETFs/funds: #PPC:ISIN2 functionality might be useful for you. It fetches the information from another ISIN (which should be similar and should ideally follow the same index).

(I don’t see CA92205P1045 on Morningstar’s Web site).

@PierreT: I found CA92205P1045 under en-ca site. From there, I retrieved SecId F00000QA9G. I think it should work with that one.

Indeed, it works but not quite. The script finds the fund, extract data. In debug, I see that it extract numbers for sectors, country, region and holdings. But the script then turn all these numbers to 0. Deep diving into sectors, I found out that it is multiplying by long_equity, which is equal to zero for that fund.

Looking at that fund’s morningstar page, there is only 2 columns for asset allocation. So probably why it fails. Will try to fix it later today, by falling back on net if long is zero.

Very good point. In new-api-branch I had switched to net, but when I revived the main branch, I forgot and kept the former long. I think I found net to be more often available. So maybe I should switch also main to net. (Honestly, I was never really sure which one is more appropriate.)

Problem is not with longEquity. It is that Canadian ETFs and Mutual funds use a different Asset Type classification. I fixed it as follow:

long_equity = (float(value.get('assetAllocEquity',{}).get('longAllocation',0)) +
    float(value.get('AssetAllocNonUSEquity',{}).get('longAllocation',0)) +
    float(value.get('AssetAllocUSEquity',{}).get('longAllocation',0)) +
    float(value.get('CANAssetAllocCanEquity',{}).get('longAllocation',0)) +
    float(value.get('CANAssetAllocUSEquity',{}).get('longAllocation',0)) +
    float(value.get('CANAssetAllocInternationalEquity'{}).get('longAllocation',0)))/100

Ok. Will add it to my version.