Investing.com is not allowing anymore to scrape historical prices

I’ve got this ISIN in portfolio - it’s italian bond (BTP) - IT0005547390. I’m used to get historical quotes from investing.com via the “table on website” provider, by inserting the URL: https://www.investing.com/rates-bonds/it0005547390-historical-data

I do the same for all other bonds I have in portfolio, as I cannot find their quotes anywhere else.

But since June 26th, investing.com is returning an HTTP 403 error:

Any tricks I can use to get the quotes ? Or does the embedded scraper need to be updated so it doesn’t get banned by investing.com ?

Thanks for your support

https://www.investing.com/rates-bonds/it0005547390-historical-data works for me

Maybe the 403-return is just for your IP - too much request from you?

Hey, thanks for your feedback

It seems the issue is still there.

This time I’ve logged my IP and tried to cURL the same URL from the command line and I still get a 403. But I can browse to the very same URL with a browser and don’t get any error!

So likely the issue is in one of the following:

  1. PP calls investing.com URLs in too short a timespan for investing to allow it (I’ve got a few bonds in my pf)

  2. investing.com doesn’t like my IP. But would there any reason why ?

I will try with a different IP and see if I get blocked again!

Try another URL too: https://de.investing.com/rates-bonds/it0005547390-historical-data

Hey @ProgFriese thanks for the hint, this time before opening up PP I’ve made sure my IP address had changed and cURL-ed both the original URL and yours (the one with the .de subdomain): in both cases, I get a 403

So honestly I have no idea why this is not working

Maybe I can try to open up a VPN and see if that changes somehow


Since July 15th, it hasn’t been downloading data for any asset from investing.com for me either

it’s a disaster

I try

on Btp Valore 13giu27 - Quotazioni in tempo reale | MilanoFinanza.it

and find the JSON-Request

https://www.milanofinanza.it/Grafici/GraficoStorico?stockCodeRT=56c3&codiceStrumento=1ac&nome=

The Response is a little bit tricky (bad structur)

[["Btp Val Gn27"],["1784592000000","100.9789"],["1784505600000","100.9883"],["1784246400000","101.006"],["1784160000000","101.0277"],["1784073600000","101.0319"],["1783987200000","101.0189"],["1783900800000","101.0782"],["1783641600000","101.097"],["1783555200000","101.09"],["1783468800000","101.1185"],["1783382400000","101.1873"],
..... and so on ....
["1687737600000","100.352"],["1687478400000","100.31"],["1687392000000","100.239"],["1687305600000","100.3224"],["1687219200000","100.244"],["1687132800000","100.2089"],["1686873600000","100.2267"],["1686787200000","100.0897"],["1686700800000","100.1953"],["1686614400000","100.0765"]]

because of the “Btp Val Gn27” in the first field, it is not possible to use the normal asterisk ($[*]).

URL: https://www.milanofinanza.it/Grafici/GraficoStorico?stockCodeRT=56c3&codiceStrumento=1ac&nome=
Path to date: $[1:][0]
Path to quote: $[1:][1]

If you want to try that stuff for another ISIN (Btp | Milano Finanza), use CTRL+Shift+i in Firefox and filter under networkanalysis for stockcode:

This worked like a charm @ProgFriese

:heart:

thank you!

@AndreasB
When you create a new security as a blank security, the current price is set by default to ‘same as historical prices’. As this is the usual way to create bonds – which are generally configured using only historical prices – I believe there is a small hidden risk of a DDoS attack here.
If, for example, you enter a URL from investing.com in the historical prices field, is that URL then queried several times a day (unless you actively set the current price to ‘no download’)? If so, the template for ‘blank security’ should be amended accordingly.

If the exact same request is made within (I think) 5 minutes, then PP is not actually making a second call but using the caches response from the previous result. So there is not necessarily a problem of doing too many requests - in this case the call for historical prices is identical to the one for latest prices.

I agree it is not intuitive when to make a second call (Yahoo has different API endpoints) and when not (the web site has one table from which to extract the prices).

What exactly did you do to use this for gold?

https://www.edelkurs.de/ did you now this?