Automatic import of classifications

English Import/export
371

Hello portfolio-b2,

3/ this feature is now implemented in pp-portfolio-classifier@0.0.22 thanks to Benjamin’s contribution. Use the maxItems config option as described in the readme file.

Regards

372

Hello Thomas,

it should be fixed in 0.0.21

let me know if you still have a problem.

Regards

373

Thank you @LuAp for taking the time to investigate and share your findings!

These are indeed very interesting observations that I had not identified:

  1. The fixTotal parameter - You’re right, this parameter is currently unused in the code. I will remove it from the config files.

  2. The separate bond/equity country exposures - This is a great catch! I was not aware that the API returned these separately. Being able to extract them individually would indeed provide more accurate classifications.

  3. The NotClassified field in Breakdowns - This is a smart approach to correctly weight the exposures. I’ll analyze your suggestion and see if I can implement it in a configurable way.

Thank you also for providing the sample API response - this will help me better understand the data structure and test any potential changes.

I’ll analyze all these elements and see how I can incorporate them into the classifier. Your feedback is highly appreciated!

374

just tried it wit version 0.0.23 → works perfect! Thanks.

375

Hello LuAp,

I implemented the third point as you suggested by multiplying each of the Breakdown values with (1-NotClassified/100). The feature is available in release 0.0.25 but it’s not active in the default configuration. You have to enable it in your config/local.json file with:

"country": {
  ...
  "notClassifiedField": "NotClassified",
  "multigroup": true,
  ...
}

You can look at the example in the README file.

I think it’s an interesting feature that provides more accurate data. The point that surprises me is that when I look at the Morningstar website for LU2872292243, the country breakdown shows for example Japan at 6.38%, which is the value returned by the classifier before the implementation (releases before 0.0.25). With the new implementation, the result is different and does not reflect what is displayed on the Morningstar website. For this reason, I did not activate it in the default configuration.

I am interested in your feedback, as well as feedback from other users and the opinion of the community, to know if it should be the new default configuration.

Regards,

rylorin

376

@rylorin Thanks for your response. Contrary to my expectation I was able to fork your code and implement a new flag inclNotClassified that allows the user to configure the calculation to include the NotClassified bit of the API response. Check it out here:
https://github.com/LuAp-maker/pp-portfolio-classifier.
If you’re interested, I can create a pull request to merge into your code (Edit: Since I don’t know how to publish the compiled version as you did, merging into your code would be probably be most useful to other users). So far, I’ve only been using it for myself. I also added some classification mappings, but didn’t change any existing ones, so that users of those mappings shouldn’t be affected.

377

Hi @rylorin,

I’ve updated the script, and the local.json as in the README.md
“holding” : {
“active”: true,
“maxItems”: 100,
“name”: “Actions > Avoirs”
}

But… Everything in the config.json is consumed, except this parameter : all the holdings are back in my taxonomy and no “Console log” of Truncating to 100 items. What I’ve missed ?

378

Hello LuAp,

Yes please, could you submit a PR and we will see how we can manage the merge, as it won’t merge automatically.

Thanks

379

Hello portfolio-b2,

I’m afraid that you didn’t miss anything and my last change broke the maxItems limit.
I will have a look on how to fix this.
Stay tuned!

Regards

380

Hello portfolio-b2,

fixed in v0.1.0 and NotClassified feature temporarily disabled. Will reintroduce it when LuAp submit his PR.

holding enabled and limited to 100 items in default config.

Regards

381

Hello all,

It seems that the script does not work anymore. Morningstar/CloudFront returns x-amzn-waf-action “challenge” when the script tries to fetch the authentication token. I have no idea how to fix this. Any help would be greatly appreciated @rylorin , @fizban or anybody.

382

Looks like web application firewall is blocking script access.

Depending on your script (haven‘t checked it) two possible ways forward come to my mind.

  • Header contains a real user agent.
  • Script in a real browser engine, eg. Selenium, Chromium, etc.
383

Maybe this could a solution:

https://thetechlogger.medium.com/how-to-solve-aws-waf-captcha-using-python-6267d9b25500

384

@Sn1kk3r5 @Rafa

Thanks for the hints. Let me summarise the situation, because I think that it is slightly more complex:

  • The script uses the information provided by Morningstar’s Direct Web Services (and fetched from URLs like “www.emea-api.morningstar. com /ecint/v1/securities/”)
  • The authentication token used by the script is however retrieved from “www.morningstar.{domain}/Common/funds/snapshot/PortfolioSAL.aspx”. This has been in the script by fbuchinger from the very beginning.
  • Some time ago, Morningstar discontinued those snapshot services as far as I understand. Still, the authentication token was provided via the request above. I suspect that now they have discontinued also this service even if the 202 CloudFront message suggest waf-action and captcha.
  • I suspect that the solution to solve this is to integrate a user name and password for a Morningstar account into the script as described at: authentication-api overview. (Actually, I guess that that is what Morningstar wants from the users now).
  • This might be some piece of work and I don’t know when I will find the time for it. But if anybody has another solution or is able to spend the time to code this (maybe with the help of some AI), this would of course be very helpful for all.
385

The error message refer a “temporary” failure therefore we may expect it to work again in a few days unless this was done on purpose.

In the meanwhile I’ve implemented authentification with credentials in v0.1.2 as it seems quite easy, but I don’t have an active subscription for testing.

@LuAp I just noticed your PR, I will work on it in the next few days.

386

Hmm. Unfortunately, it is not as simple. I retrieved a proper token using my free Morningstar account but when I tried to use it in the script, the server replied that authorisation failed. (I read this that I am authenticated but not authorised).

387

Let’s hope for the best :slight_smile:

388

Also “https:// global.morningstar. com/api/v1/de/search/securities” seems to be gone. The script uses it to get the mapping from isin to secid. Does anybody know an alternative?

In the long run, we might need to switch to pages like “https:// global.morningstar. com/en-eu/investments/etfs/0P0000XRVP/portfolio”, but also for those the secid (0P0000XRVP in this case for ISIN JP5841350001) would be useful.

UPDATE:

As the service is not back and as even a normal Morningstar subscription does not work with the Morningstar API, I have to assume that Morningstar does not want to offer this servcie to ordinary internet users anymore (but only to certain corporate customers). So the script doesn’t work anymore and I have added the following to the README:


pp-portfolio-classifier

DOES NOT WORK ANYMORE. MORNINGSTAR HAS DISCONTINUED THE PUBLIC SERVICES AND/OR MOVED THEM BEHIND CORPORATE SUBSCRIPTION WALLS.
389

Is yfinance useful (see code and output examples in following messages) ?

390

Thanks for the hint.

I have been looking at yfinance and yahooquery.

As far as I can see, they can provide asset allocation, sector weightings and top 10 holdings, but neither of them reliably provides ETF country and region allocations as far as I understand. Also stock style information seems to be very limited.

Therefore, I am still looking for alternatives.