Yes, you just have to scroll a little bit to the top, than wasting my time and making the thread unreadable…
Hello,
the -stocks option is only supported in my fork (Alfons1Qto12/pp-portfolio-classifier).
Currently, my fork has two branches:
main: This uses the same Morningstar API as the original script (fbuchinger and fizban99) in addition to the original script, it supports a few more features like e.g. the-stocksoption and most importantly, it modifies existing xml files rather than creating the xml from scratch (which preserves colour, balancing weights, sequence of categories, …)new-api-branch: This is a temporary branch for moving the script to another (better documented) Morningstar API and to add a few more features which are only possible with the new API. That one also supports-stocksoption and it is even already ported to the new API. I would be happy to get feedback, if that branch is providing correct and stable results. Currently I am working on support for funds which contain bonds (bond style, bond sector, regional exposure,… .). I also plan to integrate a feautre for estimating the “Vorabpauschale” for users in Germany.
Link to main is in:
Link to new-api-branch is in the post mentioned by Jo92 above.
2 Likes
Thanks Alfons1Qvor12! The problem was that I used fizban repo (like 4th post in this thread), which has info about -stocks in README however fails with that option. After getting your fork of course it works as expected ;). I found this thread in Google search, it’s hard to follow all 200 posts from multiple threads when searching for a single working solution ;).
I spend some time on running and comparing output results from both branches on US, NL, PL, UK stocks and ETFs. New API seems to be much faster, but lacks some data, details below.
- Let’s start with US, new API results:
Old one:
Categorization of Stock-style is a bit different. Which one is better - hard to determine. New API failed to found IGLN.UK (IE00B4ND3602), old one struggles with NU (KYG6683N1034).
- PL stocks. Here new API is unusable - over half of the stocks were not found, where old API finds almost all, except single IFI.WA (PLPWRMD00011). Sample tickers for debug purposes that work in old API only: PLBUDMX00013, PLBSK0000017
old API PL sector - almost complete:
new API PL sector - half of a data:
- EUR ETFs ASWC.DE (IE000OJ5TQP4), NUKL.DE (IE000M7V94E1) and InPost.NL (LU2290522684) stock. For sure holdings are more detailed in old API (edit: I used default settings without changing
-top_holdings, after changing that to 50 I get very similar output). InPost.NL was not found by both APIs.
Old API holdings - more detailed data:
New API holdings:
For my purposes old API presents better results and I will use new one only for US stocks.
IMHO, it would be nice to have:
a) integration of both APIs - when new API won’t find a ticker than it should call old one in order to get the data
b) possibility to use old / new API by using a flag
@wjtkport: Thanks a lot. This is the type of information I am looking for. A lot for me to chew. (It is definitely an option to keep stock retrieval as it is as it is anyway a different solution than the one for funds on the old API. But maybe I can find a way to get similar information also from the new API. After all, it is all from the same source).
UPDATE:
Some initial findings:
-
BABA.US is Large Blend. So new API implementation (Large Value) seems to be faulty. I will check. (Also the other US stocks are only correctly mapped to stock-style using the old method).
-
IE00B4ND3602 is an ETC for Gold. This is classified as a fund of type “Commodities”. I haven’t address this type yet on the new API. I only have “Equity”, “Fixed Income” (= bonds), “Allocation” (= multi assest) yet. In any case, categories for Equity don’t make much sense in context of an ETC and the old script retrieves only “Asset-Type = Other”, nothing else. I will bring this also to the new API.
-
KYG6683N1034 now is a stock and indeed works well on the new API and is not suported by the previous stock retrieval mechanism. +1 for the new API.
(Also InPost.NL using LU2290522684 works fine with the new API). -
Retrieval of holdings is better determined on new API, because old API only gets what it gets (typically 10, but sometimes more) whereas with new API we get a defined number.
-
About Polish stocks: To be more precise, on the new API, a lot of information is also retrieved for them, but stock-style is missing. As with US stocks, there seems to be an issue with the way how I try to retrieve stock-style. (For funds, everything works fine on the new API).
UPDATE2: For stocks, I plan to fall back to the old solution. With the new API, I don’t get reliable information on stock style. Maybe I am using the wrong information in JSON (QuantitativeFairValue.InvestmentStyle), but I don’t find anything better. (Drawback is that there are a few stocks for which the old xray method does not provide results, like for Nu (KYG6683N1034) mentioned above). For funds (equity and bonds and mixed) on the other hand, I will continue to explore the new API. It is very promising and gives reliable and well structured results.
2 Likes
Nice analysys!
I think you can include a scenario where you use a second api when the preferred one doesn’t find information (or important details) about a given asset. The speed of new API is tempting 
Actually, with the help of the ISINs from your Polish stock, I found out why the current Instant X-Ray method for stocks misses some of them. I am confident that I will be able to close the gaps in the Instant X-Ray method. So this will be the method of choice for stocks. I just don’t manage to get reliable information on stock-style for stocks via the new API. So it is also not a backup solution.
For funds on the other hand, the new API is really great. It still needs a bit of fine tuning to well distinguish between different types of funds (including for bonds and maybe even for ETCs), but then it will be much more reliable and easy to maintain than the one with the old API (“PortfolioSAL”). (And hopefully also faster ;-).
1 Like
I found another case that lacks categorization in new API: Ethereum ETN DE000A3GPSP7
MAJOR UPDATE of my temporary development branch new-api-branch (not yet on main)
- funds with bonds
- significantly improved coverage of stocks retrieval (for @wjtkport ;-).
-xrcommand line option removed- !!! Names of taxonomies have changed. I recommend to rename in PP first before running the script (or delete them altogether) !!!
New names are:
- Asset Type (was: Asset-Type)
- Stock Style (was: Stock-style)
- Stock Sector (was: Sector)
- Bond Style (is new)
- Bond Sector (is new)
- Region
- Country
- Holding
Please also note that countries in ‘Country’ now come in two forms:
“France” and “France (Bonds)” to distinguish country percentage from equity allocations from country percentage from bond allocations. I am thinking about a command line flag to enable/disable this distinction, but for now, it is in.
NOTE THAT THIS new-api-branch IS STILL EXPERIMENTAL
For link, see:
PLEASE TEST!
4 Likes
Thanks for the update! I checked this on few of my portfolios. EU and US stocks looks good in terms of categorization, but still a lot of PL stocks are not found in the new branch.
List for debug purposes:
FRO.WA PLFERRO00016
SHO.WA PLSHPR000021
KTY.WA PLKETY000011
ING.WA PLBSK0000017
SNG.WA PLCMPLD00016
4MS.WA PL4MASS00011
IFI.WA PLPWRMD00011
BFT.WA PLBNFTS00018
NEU.WA PLTRFRM00018
CBF.WA PLR220000018
SNT.WA PLSNKTK00019
ABS.WA PLABS0000018
PCO.WA NL0015000AU7
KLE.WA PLMRTIN00011
ANR.WA PLANSWR00019
BLO.WA PLBLOBR00014
[SGN.PL]:
@ No matching information for ISIN PLCMPLD00016 found on Morningstar, skipping it...
[SHO.PL]:
@ No matching information for ISIN PLSHPR000021 found on Morningstar, skipping it...
[NOVOB.DK]:
@ Retrieving data for stock DK0062498333 from MS Instant X-Ray
(Name: "Novo Nordisk AS Class B")
(MSIN for Instant X-Ray: "0P0000BDH4" | Country: "Denmark")
[PGE.PL]:
@ Retrieving data for stock PLPGER000010 from MS Instant X-Ray
(Name: "Polska Grupa Energetyczna SA")
(MSIN for Instant X-Ray: "0P0001C8XH" | Country: "Poland")
[NEU.PL]:
@ No matching information for ISIN PLTRFRM00018 found on Morningstar, skipping it...
[SNT.PL]:
@ No matching information for ISIN PLSNKTK00019 found on Morningstar, skipping it...
[PKO.PL]:
@ Retrieving data for stock PLPKO0000016 from MS Instant X-Ray
(Name: "PKO Bank Polski SA")
(MSIN for Instant X-Ray: "0P0000JJJM" | Country: "Poland")
[ANR.PL]:
@ No matching information for ISIN PLANSWR00019 found on Morningstar, skipping it...
[PZU.PL]:
@ Retrieving data for stock PLPZU0000011 from MS Instant X-Ray
(Name: "Powszechny Zaklad Ubezpieczen SA")
(MSIN for Instant X-Ray: "0P00015A4H" | Country: "Poland")
[BFT.PL]:
@ No matching information for ISIN PLBNFTS00018 found on Morningstar, skipping it...
[BDX.PL]:
@ Retrieving data for stock PLBUDMX00013 from MS Instant X-Ray
(Name: "Budimex SA")
(MSIN for Instant X-Ray: "0P0000C6QW" | Country: "Poland")
[ING.PL]:
@ No matching information for ISIN PLBSK0000017 found on Morningstar, skipping it...
[BLO.PL]:
@ No matching information for ISIN PLBLOBR00014 found on Morningstar, skipping it...
[CDR.PL]:
@ Retrieving data for stock PLOPTTC00011 from MS Instant X-Ray
(Name: "CD Projekt SA")
(MSIN for Instant X-Ray: "0P00017UZH" | Country: "Poland")
[ACP.PL]:
@ Retrieving data for stock PLSOFTB00016 from MS Instant X-Ray
(Name: "Asseco Poland SA")
(MSIN for Instant X-Ray: "0P0000TMOY" | Country: "Poland")
[KLE.PL]:
@ No matching information for ISIN PLMRTIN00011 found on Morningstar, skipping it...
[PCO.PL]:
@ No matching information for ISIN NL0015000AU7 found on Morningstar, skipping it...
[IFI.PL]:
@ No matching information for ISIN PLPWRMD00011 found on Morningstar, skipping it...
[CBF.PL]:
@ No matching information for ISIN PLR220000018 found on Morningstar, skipping it...
[4MS.PL]:
@ No matching information for ISIN PL4MASS00011 found on Morningstar, skipping it...
[ABS.PL]:
@ No matching information for ISIN PLABS0000018 found on Morningstar, skipping it...
[KGH.PL]:
@ Retrieving data for stock PLKGHM000017 from MS Instant X-Ray
(Name: "KGHM Polska Miedz SA")
(MSIN for Instant X-Ray: "0P0000TMOX" | Country: "Poland")
[KTY.PL]:
@ No matching information for ISIN PLKETY000011 found on Morningstar, skipping it...
Old API categorization:
New API:
Thanks a lot for the feedback. The examples are extremely useful. I think I see the root cause of the issue. The new API does not know anything about those ISINs, not even that it is stock, but the Instant X-Ray might still provide the information as soon as I have a useful MSIN (and there is usually more than one and not all are useful for this purpose). So I still need to tweak the logic a bit to also get those ones.
1 Like
@wjtkport: Please try now with the latest version. I guess I am close to 100%. (I try to get useful MSINs from .de, .fr, .nl and .at hoping that at least one provides the full information).
2 Likes
Success! All my PL stocks are properly categorized with the recent update ;). Thanks for your work on this matter!
Would it be possible for you to add a flag to optionally replace stock/etf name in portfolio with the one gathered by your script?
[PKO.PL]:
@ Retrieving data for stock PLPKO0000016 from MS Instant X-Ray
(Name: "PKO Bank Polski SA")
(MSIN for Instant X-Ray: "0P0000JJJM" | Country: "Poland")
In this sample stock name is “PKO.PL” and I would like to change it automatically to “PKO Bank Polski SA”
Hi to the devs of the classification extensions of PP!
Is it correct that those work only with xml version of pp files?
I ask because I also use the mobile version and that only works with binary file format…
Kind of a dead lock…
Just want to get sure.
@wjtkport: Thanks a lot for the testing.
About the automatic update of other securities data: I will put it on my todo list, but I am a bit hesitant to touch other parts of the xml file outside the taxonomy section. Also, it is overlapping a lot with the add new investment instrument functionality already provided by PP.
BTW: Is there a tool/way to fill the Additional Attributes in PP (Logo, Gesamtkostenquote (TER), Fondsgröße, Anbieter, …)? In principle, some of them could also be retrieved from Morningstar.
Well, the script requires xml as input and generates xml as output. But from PP you can save it in binary format and then the information is also available in the mobile version.
I perform the following steps:
- Open PP (which contains a tab with the binary version).
- Use “Save as” for the binary version to store it as xml (and the tab changes to xml version).
- Close PP
- Run the script with xml file as input and output (and don’t forget to keep a copy of the original file either as binary or as xml as backup).
- Open PP (which now contains a tab with the xml version).
- Use “Save As” to store the xml as binary again (overwriting the old binary and turning the tab back to binary version).
Done.
Ok, thanks. thought so…
It would be cool if PP offered a plugin architecture that (configurably) triggers such updates. And of course it would be cool if your plugin could read the binary format. 
But, ok, there is at least a workaround.
I tried it out. The test as described worked well. But the first test with my real pp (a copy of course) ended this way:
@Alfons1Qvor12 cool work.
By the way, the “binary” format is protobuf. And there is an also a protobuf binding for python available.
Please also drop me an email to portfolio dot performance dot help at gmail dot com. Maybe we have an integration that PP calls a script with the XML and then gets an XML or JSON of the updated taxonomy and it merges it back. Then we could add this as menu entry.
5 Likes
@ppulli: I assume that you saved your to ‘XML with “id” attributes’. The script only works with xml that is saved as (pure) ‘XML’.
(‘XML with “id” attributes’ is relatively new. I don’t know the difference, but obviously it doesn’t work with the references created by the script. I guess the references might need to contain an id or something similar).
I will add this limitation to the README when I re-write it for the new API.
@AndreasB Thanks for the great offer. Xml in and xml out would be the easier solution to update taxonomies (though the data from Morningstar comes in JSON). The biggest challenge is to get the references to the securities in xml right. I cannot promise to work much on this topic, but I think it is worth a try. I will contact you in the next days.
4 Likes