Portfolio Performance database – ppxml2db import/export

English
22

Thanks, that worked and was easy. I already had installed the precompiled binaries for Sqlite from sqlite-dll-win-x64-3460000.zip, which perhaps is what you meant. The batch file was excellent – it quickly produced the db file without any issues.

23

The projects makes strides in supporting more and more of Portfolio Performance XML functionality. Recently released is version 1.7: GitHub - pfalcon/ppxml2db: Scripts to import PortfolioPerformance (https://github.com/portfolio-performance/portfolio) XML into a SQLite DB and export back

3 Likes
Bulk update quote feed for all my securities
Bulk update quote feed for all my securities
24

ppxml2db scripts package test

https://github.com/pfalcon/ppxml2db is fairly primitive so I’ve packaged it so the scripts run more like normal utilities.

Test it from https://github.com/flywire/ppxml2db.

25

None of my securities are read without omissions. It is difficult to identify parts of the original PP data file that are dropped when running it through ppxml2db and db2ppxml.

issue 18 comment:

So, bottom line: with 1.7.1, if you

  1. take your original Testfile_Selfwealth.xml (from the attached zip archive),
  2. round-trip it through ppxml2db.py/db2ppxml.py,
  3. then open in the latest version of PP, then save, you get “canonical” XML,
  4. and if you try to round-trip it again, there’re no further changes

That’s the intended behaviour

While this is not not described in the README.md the process makes it nearly impossible to trace omissions and changes to the original data, especially with real user files because no trace is left of attributes stripped from every security. The development is also linux focused so things like newline characters in notes can be corrupted for Windows users.

pfalcon writes:

[ppxml2db] potentially opens a possibility to integrate database backend directly into PP (by making at least a first step - providing a realisitic database schema for that).

26

@flywire , @pfalcon . When doing the round trip test on my own file, I noticed the following errors:

  • a user-defined bookmark attribute attached to a security is not replicated in the output.
  • on an exchange-rate-type security the target currency field is not replicated, so that it is interpreted as a standard security after the round trip.
  • when using the rebalancing feature in the taxonomy, the rebalancing-included field is returned as a string, not a boolean.
  • When using the expected return feature in the taxonomy, the expected-return:in-use and expected-return:value fields are both returned as strings. The first should be boolean, the second an int.

I’ve attached a kommer.xml file modified to demonstrate these errors.
kommer_modified.xml (1,4 MB)

27

@LuAp, thanks for the report. Feel free to open a ticket at GitHub · Where software is built so it’s not forgotten.

28

Hi all,

last holiday weekend ahead I had a little time and joy to give it another try with my complex portfolio.

System as follows:
MacBook Pro
Apple M4 Pro
macOS Tahoe 26.1
ppxml2db version: 1.7.3

Initially I got the following error:

Traceback (most recent call last): File "/Users/xxx/Privat/ppxml2db/ppxml2db.py", line 10, in <module> import lxml.etree as ET ModuleNotFoundError: No module named 'lxml'

Caused by the missing module LXML.

After installing, I faced the following error:

Traceback (most recent call last): File "/Users/xxx/Privat/ppxml2db/ppxml2db.py", line 637, in <module> conv.iterparse() ~~~~~~~~~~~~~~^^ File "/Users/xxx/Privat/ppxml2db/ppxml2db.py", line 556, in iterparse assert self.uuid2ctr_map[uuid].startswith(what), self.uuid2ctr_map[uuid] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ AssertionError: referenceAccount

So I checked ppxml2db.py and replaced line 556:

assert self.uuid2ctr_map[uuid].startswith(what), self.uuid2ctr_map[uuid]

with:

if what == "account":
    assert self.is_account_tag(self.uuid2ctr_map[uuid]), self.uuid2ctr_map[uuid]
else:
    assert self.uuid2ctr_map[uuid].startswith(what), self.uuid2ctr_map[uuid]

Which did the trick.

Haven’t tried the round trip as an analytical view is all it needs.

Hope it helps anyone facing the same obsticle.

2 Likes
29

Bug reports should be made in the GitHub repo including the Version (even though version is not tagged) and testing the round trip.

The first time I was able to round trip (except for Notes formatting) my fairly complex file in Windows I was with Version 1.7.2.

30

My answer is not considered as a bug report, but an experience report.

32

Version is updated in my original statement.

33

@flywire

since @pfalcon wasn’t active for quite a while and you seem to be involved in the project (sorry if I’m mistaken) I’d have a question.

Looking at the Makefile and dbhelper.py, it seems DivyDiary dividend info isn’t included. Was this left out on purpose, or did it just fly under the radar?

34

TL;DR No, I’m just an enthusiastic user. @pfalcon has made it clear development is guided by his own needs, round tripping the kommer.xml sample file, and issues raised in the GitHub repo. “It seems DivyDiary dividend info isn’t included.”

@pfalcon and I both contributed to the same issues calling for database access to PP and @pfalcon went on to release https://github.com/pfalcon/ppxml2db as a comprehensive but limited solution. @AndreasB even provided an alternate PP datafile format to support the utility.

Ideally PP would directly support xml and SQL formats like say GnuCash but the SQL format is not part of PP, it is generated by https://github.com/pfalcon/ppxml2db. By design it will always be half-baked because rather than implement all PP features it implements features based on: the developer’s own needs, round tripping the kommer.xml sample file, and issues raised in the GitHub repo.

As you pointed out, it also includes a makefile which Windows can’t run. There is a fairly trivial workaround to support Windows via a batch file or preferably by including the functionality directly in the code which I provided but the developer is not inclined to support it.

Those raising an issue in the repo are requested to provide a diff file of ppxml2db and a PP minimal working example. This is problematic because a user might not be able to isolate the cause of the problem from their production file, and if ppxml2db parses the file, diff files aren’t a standard concept in Windows. (Neither of these would be an issue for the instance you raise.)

I run a fork of a fork including the Widows process. Releases aren’t tagged in the repo so I’m working on an automated process that would patch the original repo as commits are released.

Development of FOSS software as a hobby can be slow and intermittent, or even cease.

35

Copy that, thanks.
I’m not complaining in any direction, everyone is free to spend his time the way one wants!

So I’m going to create my own local version which handles DiviDiary data.
Just to play around with confirmed future dividend payments and grow rates.

Grafana on top of the DB is a little overdrawn but fun and powerful to play with.

36

perfect! that’s what I’m looking for. I have exactly the same task. It would be nice If you have smth./want to share your progress.

Currently I’m investigating the data model and this tool from the task perspective.

Thanks

37

I enhanced the handler script and the respective security_events table.
All DiviDiary data is now in the db file.

Runs like a charm.

Currently I‘m developing some dashboard in Grafana.