consequently, the command line option -equity_only is discontinued
new command line option -bonds_in_funds adds retrieval of bond related information (to Bond Style, Bond Sector, Country, Region and Holding)
command line option to create own categories for bonds in Country and Region is now called -seg_bonds (was: -bond_seg)
additional checks provide warnings about negative net values and about net values larger than 100%
for simple, small cases, the script adapts the net values by adding the negative value of an allocation in the range of 0% to -0,25% to another allocation which has a value that is larger than 100% (if there is such allocation)
for other cases, the script suggests manual setting of the values
a new token in the notes field of a security in PP instructs the script to skip the retrieval of data for that security (the token is #PPC:SKIP)
@Alfons1Qvor12 Awesome, that worked like a charme! I just tried out the new options and also tagged two of my equities with the skip-flag in PP after modiying their classifications. Lo and behold, those two values were skipped as planned!
Thanks again for all your efforts, much appreciated!
Just a heads-up. Morningstar is planning some changes in their free service offerings. See announcement (in German).
I don’t know, if this will negatively impact the functionality of the script (either main branch or new-api-brach or both), but the risk looks very high to me as services like instant x-ray will not be offered anymore.
I will wait and see what happens and might also make efforts to adapt as far as possible, but cannot give any guarantees or time lines for that.
There is an AI generated python script that downloads name,TER,fundsize,category for selected ISIN’s from Morningstar and displays it. You need to modify this line in the script:
isins_to_process = [“IE00BK5BQT80”, “IE00BMFKG444”]
Morningstar has now made the announced changes. Impact is not yet fully clear to me, but let me try to summarise my current understanding:
Original versions of the script (fizban99, Alfons1Qvor12-main ) don’t work properly anymore. (It seems that only data for “funds” can be retrieved, but no data for “etfs”).
My experimental new API branch (Alfons1Qvor12-new-api-branch) is still able to download classifications for funds and etfs including bond-etfs.
Everything in my versions that relies on Instant X-Ray does not work anymore, i.e. options -xr and -stocks in main and new-api-branch do not work anymore.
So for now, there is no way to retrieve classification of stocks. For funds and etfs, I recommend using Alfons1Qvor12-new-api-branch (but without option -stocks).
Or we update the script so it starts pulling data from the new website… (the advantage is that it works with JSON, so there’s no need to scrape HTML anymore—JSON is easier).
Now the site’s search works with a different ID (before it was the secureID, now it uses the fundID):
If you go to the network tab in Chrome, you’ll see all the calls and you’ll notice that everything now returns JSON. Of course, you still have the option of reading the HTML if you prefer.
The new API branch already works with JSON and uses ISIN as id for retrieval. For all kind of funds, it works fine. However, I had issues with getting good classification information for individual stocks. Therefore, I fell back to the old Instant X-Ray for stocks.
Your hint about differrent IDs is very useful. I see from the JSON that there is a “companyID” as well. I might be able to use it for getting classification data of individual stocks.
UPDATE: Actually the security ID still works fine for stock. I tested with some Polish values using your links (in Spanish but could be another language). Example: https://global.morningstar.com/es/inversiones/acciones/0P0001DJHF/cotizacion. It provides some classification information. However, I struggling with getting Country and Region for such stock.
I have now restored support of -stocks option on Alfons1Qvor12-new-api-branch. Thanks to hint from @fizban, it now uses lt.morningstar.com/3y3wd9echv/xray/ for stocks. The solution is not ideal, because it does not have as many stocks as the other sites and it might not last for long, but for now it is a quick fix.
Thank you for providing this great piece of work to the community! It has been very useful for me until now. Unfortunately with the newest version I get an error message like this example for multifaktortest.xml. Any suggestions?
[Franklin FTSE India UCITS ETF ]:
Traceback (most recent call last):
File “/Users/user34/pp-portfolio-classifier-new-api-branch/portfolio-classifier.py”, line 1737, in
pp_file.add_taxonomy(taxonomy)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File “/Users/user34/pp-portfolio-classifier-new-api-branch/portfolio-classifier.py”, line 1382, in add_taxonomy
securities = self.get_securities()
File “/Users/user34/pp-portfolio-classifier-new-api-branch/portfolio-classifier.py”, line 1661, in get_securities
security_h = security.load_holdings()
File “/Users/user34/pp-portfolio-classifier-new-api-branch/portfolio-classifier.py”, line 843, in load_holdings
self.holdings.load(isin = self.ISIN, name = self.name, isRetired = self.isRetired)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/Users/user34/pp-portfolio-classifier-new-api-branch/portfolio-classifier.py”, line 917, in load
bearer_token = self.get_bearer_token(domain)
File “/Users/user34/pp-portfolio-classifier-new-api-branch/portfolio-classifier.py”, line 879, in get_bearer_token
resultstringtoken = re.findall(token_regex, response.text)[0]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
I can not really reproduce this issue. Maybe it was a temporary outage of https://www.morningstar.{domain} or it depends on the domain that you are using. Please try again and/or try with another domain.
(I can reproduce the issue with using -d com as command line parameter. But the retrieval does not work with the domain .com. It should be a country domain).
Thanks for the quick reply - I wasn’t aware of the -d command line parameter. I just tried again but unfortunately I get the same error without the parameter or with “-d de” or any other domain I tried.
Hmm. So it always goes wrong for all domains and there is never any success, right? I have no idea why it does not work on your machine. It does on mine. I have now added a check to the script, but it won’t solve your issue. It just orderly terminates the script without IndexError.
Maybe just two ideas of what else you could try:
Delete the filecache.sqlite in the folder of the script.
Make sure that you are not logged in to Morningstar site with any browser on your machine. (I don’t really think that this will help, but I cannot come up with anything better).
BTW: The retrieval is always the same. It does not depend on the security for which it is fetched.
If this doesn’t help, could you maybe try with the “main” version of the script? It will not produce useful results (because of the changes at Morningstar), but it might help to better understand what is going on when the IndexError does not occur with that version of the script. (And maybe trying with fizban’s version could also give a hint to a solution).
Good news: Alfons1Qto12/pp-portfolio-classifier now supports -stocks option again. I moved it to the new API (where it already was a few months ago).
https://www.emea-api.morningstar.com/ecint/v1/securities/{isin} does not have data on as many stocks as Instant X-Ray or the web service. But there is still a decent coverage.
If I find the time, I will integrate retrieval from https://global.morningstar.com/de/investments/aktien/{msid}/ for cases like BLOOBER which is not on the new API. (Still I am struggling to find information on the country of the company on such a page. Except for the phone number in contacts, I haven’t found anything yet).
