I’ve the same problem too.
Hi @Fabio85, did you use the latest version of the script? This issue should not occur anymore. If it does nevertheless or if you see the error message “Error: No xpath found for UUID ‘{uuid}’”, please let me know.
Hello all,
Does anybody know, if the X-Ray part of the script still works properly? I have the impression that it does not. I regularly get the message “… will be retrieved from x-ray…”, but I don’t see any results of that.
Thanks
Hi! I tried the script now and i got two error:
331: SyntaxWarning: invalid escape sequence ‘{’
secid = re.search(‘{“i”:“([^”]+)"’, response).group(1)
TypeError: expected string or bytes-like object, got ‘NoneType’
I will check, but this part of the code was not to touched by me.
Could it be that you have changed the content of the script (e.g. by copying it or by editing it in some text procession application like MS Word)?
The original code is:
secid = re.search('\{"i":"([^"]+)"', response).group(1)
Your code is missing the “\” and is having a ‘ instead of a ’ character.
So this is not a problem of the script, but a problem of the copy which you created on your system.
It’s looks correct to me (line 331). I downloaded a fresh copy of your repo.
But I dont know why I get this error:
Ok. So the characters were probably changed when you copied them to this chat. I cannot explain the issue. This is legacy code. Do you also have it also when you use other versions (e.g. fizban99’s)?
1 Like
Yes. Also with the fizban one i got the error with the line 331 (I see also another people got this error in this thread). The second error instead it’s gone away and it’s generate the output file.
If I edit the script with this line the first error disappear (I asked to chatGPT)
The second error appears near the end of the script. It seems to process everything correctly but then eventually fails and does not create the output file.
I do have something similar now as well @Alfons1Qvor12
Warning: security 'Rubis' does not have isin, skipping it...
Traceback (most recent call last):
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 916, in <module>
pp_file.add_taxonomy(taxonomy)
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 629, in add_taxonomy
securities = self.get_securities()
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 869, in get_securities
security = self.get_security(sec_xpath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 590, in get_security
match = re.search(token_pattern,note)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\re\__init__.py", line 176, in search
return _compile(pattern, flags).search(string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'
Wanted to try the new option you mentioned “Maybe you could try it out to see how well it works. (I use #PPC:[ISIN2=IE00BFMXXD54] to fetch holdings for LU0496786574).”
cheers
Ok. I updated the legacy code and turned the search pattern into a raw string according to the wisdom of ChatGPT. Hope that helps.
Thanks @Fabio85 and @spartok. This is the piece of code which tries to parse the new ISIN2. I will check and try to make it more robust. (But this one is already a raw string. Not sure why it goes wrong nevertheless).
@Alfons1Qvor12 : same (I think) error – FYI I just updated the .py file
I do see in the logs that it’s retrieving from the alternative isin
[MSCI World UCITS ETF - EUR (C)]
Warning: percentages not found for Holding for F0000101SV
@ Retrieving data for etf LU1781541179 (F0000109UZ) using domain 'de'...
[Alternative ISIN for LU1681043599]
@ Retrieving data for etf IE00BK5BQT80 (F000013XH8) using domain 'de'...
[VWCE (IBKR)]
Warning: security 'DYDX' does not have isin, skipping it...
Warning: security 'total actions pme' does not have isin, skipping it...
@ isin FR0000121014 is a stock, skipping it...
[LVMH]
@ isin PLPKN0000018 is a stock, skipping it...
[PKN Orlen]
@ isin FR0010177378 not found in Morningstar for domain 'de', skipping it... Try another domain with -d <domain>
[Réserve Ecureuil]
@ isin PLMNRTR00012 is a stock, skipping it...
[Monnari Trade]
@ isin IT0003132476 is a stock, skipping it...
[ENI]
@ isin FR0012819381 is a stock, skipping it...
[Groupe Guillin]
@ isin FR0013214145 is a stock, skipping it...
[SMCP]
@ isin AT0000A00Y78 not found in Morningstar for domain 'de', skipping it... Try another domain with -d <domain>
[PETRO WELT]
@ Retrieving data for etf IE00BSPLC298 (F00000VD44) using domain 'de'...
[ZPRX (Europe Small Cap Value)]
Warning: security 'Innelec Multimédia SA' does not have isin, skipping it...
@ Retrieving data for etf LU1327051279 (F00000WSTK) using domain 'de'...
[Lyxor S&P 500 Daily (-2x) Inverse UCITS ETF]
Warning: percentages not found for Holding for F00000WSTK
Warning: percentages not found for Country for F00000WSTK
@ isin FR0013258662 is a stock, skipping it...
[ALD]
Warning: security 'total actions long terme' does not have isin, skipping it...
@ isin IE00BDC5DG00 is a stock, skipping it...
[KENMARE RES]
@ isin CY0103562118 is a stock, skipping it...
[Tharisa plc]
@ Retrieving data for etf FR0010592014 (F000001ZQF) using domain 'de'...
[Lyxor CAC 40 Daily (2x) Leveraged UCITS ETF]
Warning: percentages not found for Holding for F000001ZQF
Warning: percentages not found for Country for F000001ZQF
@ isin FR0013079092 is a stock, skipping it...
[We.Connect]
@ isin DK0060083210 is a stock, skipping it...
[Dampskibsselskabet Norden A/S]
@ isin FR0000033888 is a stock, skipping it...
[Gevelot]
@ Retrieving data for etf LU1829221024 (F000011UJM) using domain 'de'...
[Amundi Nasdaq-100 II UCITS ETF Acc]
Warning: percentages not found for Holding for F000011UJM
@ Retrieving data for etf IE00B53SZB19 (F000005PU2) using domain 'de'...
[Alternative ISIN for LU1829221024]
Warning: security 'Palantir' does not have isin, skipping it...
Warning: security 'ICX' does not have isin, skipping it...
@ isin GB00BYX7JT74 not found in Morningstar for domain 'de', skipping it... Try another domain with -d <domain>
[Diversified Energy Company PLC]
Warning: security 'Rubis' does not have isin, skipping it...
Warning: security 'Serum' does not have isin, skipping it...
@ isin NL0010273215 is a stock, skipping it...
[ASML Holding]
@ isin FR0010654087 is a stock, skipping it...
[TXCOM]
@ isin FR0000038606 is a stock, skipping it...
[Manitou]
@ isin FR0000130809 is a stock, skipping it...
[Société Générale]
Warning: security 'BTC/USDT' does not have isin, skipping it...
Warning: security 'Poshmark' does not have isin, skipping it...
@ isin FR0000051070 is a stock, skipping it...
[Maurel & Prom]
Warning: security 'Solana' does not have isin, skipping it...
@ isin PLJSW0000015 is a stock, skipping it...
[JASTRZEBSKA]
@ isin FR0004155000 is a stock, skipping it...
[SFPI]
@ Retrieving data for etf FR0011550185 (F00000QGL6) using domain 'de'...
[S&P500]
Warning: percentages not found for Holding for F00000QGL6
@ Retrieving data for etf IE00BFMXXD54 (F000013LQL) using domain 'de'...
[Alternative ISIN for FR0011550185]
@ isin None not found in Morningstar for domain 'de', skipping it... Try another domain with -d <domain>
[ASSY Index]
Warning: security 'Ethereum EUR' does not have isin, skipping it...
@ isin GA0000121459 is a stock, skipping it...
[Total Gabon]
Traceback (most recent call last):
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 916, in <module>
pp_file.add_taxonomy(taxonomy)
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 629, in add_taxonomy
securities = self.get_securities()
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 869, in get_securities
security = self.get_security(sec_xpath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Romain\Desktop 4\testscriptclassifier\update15oct\pp-portfolio-classifier-main\portfolio-classifier.py", line 590, in get_security
match = re.search(token_pattern,note)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\re\__init__.py", line 176, in search
return _compile(pattern, flags).search(string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'
@Alfons1Qvor12 ok I just retried by extracting the whole zip again
It went through without any error, but some ETF’s are now not being processed, like nasdad & ZPRV, see below :
Retrieving data for etf IE00BSPLC413 (F00000VD46) using domain 'de'...
[ZPRV (USA Small Cap Value)]
Asset-Type for secid F00000VD46 will be retrieved from x-ray...
Stock-style for secid F00000VD46 will be retrieved from x-ray...
Sector for secid F00000VD46 will be retrieved from x-ray...
Holding for secid F00000VD46 will be retrieved from x-ray...
Region for secid F00000VD46 will be retrieved from x-ray...
Country for secid F00000VD46 will be retrieved from x-ray...
@ Retrieving data for etf IE00BSPLC298 (F00000VD44) using domain 'de'...
[ZPRX (Europe Small Cap Value)]
Warning: security 'Innelec Multimédia SA' does not have isin, skipping it...
Warning: security 'IOTA' does not have isin, skipping it...
Warning: security 'Swipe' does not have isin, skipping it...
@ isin PLWTCHN00030 is a stock, skipping it...
[WITTCHEN S.A.]
Warning: security 'Palantir' does not have isin, skipping it...
@ isin GB0006449366 is a stock, skipping it...
[Anglo Pacific Group]
Warning: security 'Ethereum EUR' does not have isin, skipping it...
@ isin FR0010202606 is a stock, skipping it...
[Sidetrade SA]
@ Retrieving data for etf LU1327051279 (F00000WSTK) using domain 'de'...
[Lyxor S&P 500 Daily (-2x) Inverse UCITS ETF]
Warning: percentages not found for Holding for F00000WSTK
Warning: percentages not found for Country for F00000WSTK
@ Retrieving data for etf LU1829221024 (F000011UJM) using domain 'de'...
[Amundi Nasdaq-100 II UCITS ETF Acc]
Asset-Type for secid F000011UJM will be retrieved from x-ray...
Stock-style for secid F000011UJM will be retrieved from x-ray...
Sector for secid F000011UJM will be retrieved from x-ray...
Holding for secid F000011UJM will be retrieved from x-ray...
Region for secid F000011UJM will be retrieved from x-ray...
Country for secid F000011UJM will be retrieved from x-ray...
@ Retrieving data for etf IE00B53SZB19 (F000005PU2) using domain 'de'...
[Alternative ISIN for LU1829221024]
Asset-Type for secid F000005PU2 will be retrieved from x-ray...
Stock-style for secid F000005PU2 will be retrieved from x-ray...
Sector for secid F000005PU2 will be retrieved from x-ray...
Holding for secid F000005PU2 will be retrieved from x-ray...
Region for secid F000005PU2 will be retrieved from x-ray...
Country for secid F000005PU2 will be retrieved from x-ray...
@ isin FR0000120669 is a stock, skipping it...
Hmmm. Of course, it is great that it runs through without error. However, the issue which you experience with IE00BSPLC413 is still strange.
In my environment, I get the following result:
\@ Retrieving data for etf IE00BSPLC413 (F00000VD46) using domain 'de'...
[SPDR MSCI USA Small Cap Value Weighted UCITS ETF]
...
Info: Entry for 'Lumen Technologies Inc Ordinary Shares' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Lumen Technologies Inc Ordinary Shares' created
Info: Entry for 'Jackson Financial Inc' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Jackson Financial Inc' created
Info: Entry for 'United States Steel Corp' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'United States Steel Corp' created
Info: Entry for 'Lithia Motors Inc Class A' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Lithia Motors Inc Class A' created
Info: Entry for 'Performance Food Group Co' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Performance Food Group Co' created
Info: Entry for 'Arrow Electronics Inc' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Arrow Electronics Inc' created
Info: Entry for 'TD Synnex Corp' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'TD Synnex Corp' created
Info: Entry for 'World Kinect Corp' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'World Kinect Corp' created
Info: Entry for 'Plains GP Holdings LP Class A' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Plains GP Holdings LP Class A' created
Info: Entry for 'Unum Group' in 'Holding' created
Info: Entry for 'SPDR MSCI USA Small Cap Value Weighted UCITS ETF' in 'Unum Group' created
None of the issue that you experience are happening in my environment. So there seems to be something fundamentally different. I am not at all experienced in Python, so I don’t know what it could be. (Python version? Library versions? Other Libraries which I don’t have? Operation system?) But in any case, it seems to be related to the handling of strings and that it is kind of sporadic. I suspect that this issue with strings is somehow also affecting the web scraping function. I have not changed anything in that part and I am not sure, if I understand what it is doing, but I will check, if I can do some changes similar to the ones which I did for the previous fixes already.
(I assume that you have run pip3 install -r requirements.txt).
By the way: I don’t get anything for IE00B53SZB19 either. So the site doesn’t deliver anything useful.
mmh, did I miss something then ? I’ve checked several isin, following your advice, and went with one that is giving information on morningstar (and it’s physical)
looking good now! I’ve just switched the domain to fr (where I live)
btw, the holdings that are retrieved are just a small part of the total
curious to know how you use that info ?
also curious more generally on the taxonomy obtained via the script, as it’s a first for me
do you then use them in custom performance dashboard ? something else ?
if you don’t mind sharing some screenshots, I’d love to see!
cheers
Good hint. I think we are onto something: I deleted isin2secid.json and cache.sqlite. And then it ran through for all of them. So maybe that is the same effect as your change of domain.