Yes, it use a database backend, but it use Python pickles (or what it is called) to store serialized data in BLOBS (its built for Berkley, but they have start using sqlite), they are talking about changing the serialized data to json strings, they also have a experimental mongodb backend that don’t use blobs or “pickles”…
Using sql to query the data directly is not possible because of thatbut there is a sqlite export… some use that…
You are right, the file format of Gramps even seems to be version dependent. For the update to version 5, it even looks like the database has to be migrated manually via exchange format.
Your project appears to me to be extremely challenging. Is an application like Gramps, which is actually intended for a different use case and moreover uses a proprietary file format, the right one? Basically you just need a rather simple data structure, but longevity, high performance and flexible evaluation methods, don’t you?
If it’s not a one-man project, a client-server architecture would be advantageous, so that the data can be kept somewhere central and generally accessible, independent of the individual evaluation.
Just some thoughts from a systems engineer …
Apart from that, I’ve made a little progress with unraveling .aeon files.
first:
You don’t need to update the Berkley database in version 5, and it do upgrade it from earlier versions. The version dependencies is based on the Python version used…
It is also easy to convert to sqlite.
The move via export/exchange formats is just recommended due to clean ups, but for really old databases you will need to go via the xml…
It is actually a very clean way of ding it, instead of risking corruption and crashes…
Upgrading from 4 to 5 was not a big deal for me, not even on Windows, converting from Berkley to sqlite, postgresql or mongodb was neither a big deal… but of course there is always some people that have trouble…
Yes, you are right, it would be better with a database system, but most server solutions do not have any kind of good visualization… Therefore I am thinking about using Excel as an input, and export to markdown, because manually adding rows of data in Excel is really simple, and I have already found a script that can convert a csv to markdown notes, one note for each row, with features to chose columns as YAML frontmatter keys etc. There is also Excel addons that creates graphviz graphs, and even the NodeXL addin for network graphs if needed…
And if that doesn’t work, even I can create a simple VBA that can export to markdown as I need it formatted.
Another benefit with Excel is that it can easily import nearly every xml with minimal programming (but some work), and it can actually export to the same formatted xml also (with a little more work).
I can also use Excel as a front for a full featured database if needed… so it might be possible to combine even more usage as the time goes…
Last thing about Excel is that I actually can get thing done in Excel…
I am also using Openrefine.
I actually find csv more and more usefull, as long as it is not a csv from Freeplane… haha
I am thinking about trying to use Arches, I have also looked at Openmappr, Omeka, Heurists etc.
I don’t use Gramps for this project at the moment, but if the feature I mentioned is added, it will be a good software for this type of projects to, because of its hierarchal place registration and the way you can relate people to other people even though they are not in family… and it is possible to add Events without first adding a person for that event…
but as I mentioned, it is some limitations still…
Another thing is that Gramps has experimental database backends for both mongodb and postgresql, the postgresql is more or less adopted now, and when they move away from the blobs, it will be easy to utilize from 3rd party systems…
It is no doubt that Gramps is the most advanced genealogy software, but the learning curve is steep…
Multi-user will most likely come in a future version , but they need time to turn around changing out some older libraries… and maybe understand what potential the software actually have…
Personally I think that if they did the changes I mentioned, they would get more researchers (not only hobby genealogy researchers) coming to use the software, and with researchers in i.e. humanity, historical social research etc. there is a lot of people with Python experience, and maybe they will get more research knowledgeable programmers to help with the development of the software and new features…
Regarding timelines in genealogy software, its used a lot, but most software only have some kind of table format, listing the events from start to end…
Progeny Genealogy have Genlines, but it is based on family relations and only read gedcom or databases from the “top 3-4” genealogy software, no easy way to add events… and no way to add events not related to a person…
Sidetrack: Have you ever tested Causality
or Constellation
or Running Reality (maybe not that much for fiction?)
Just a few of the thousands of open source free software I have found on my way trying to find “Utopia” in the “research software world”…
Oooh, and I do know that I could get anything I would with R and Shiny 
or Plotly and Dash, but I have just not managed to wrap my head around it…
PS.
@matt has already answered this post, so I am not afraid of any off topic discussion as long as it is about workflows regarding how to visualize research…
But if he wants to move this part of the thread to a more suited group, he is most welcome to do so
Well, sure, it’s your show. Besides, it’s certainly fun to dig up all the software treasures.
As for database visualization, my idea would be this: for example, export a small subset as a CSV-formed report that can be imported into Aeon Timeline. The point of this is that the Aeon project is a disposable product for pure visualization, so to speak, and that its data export capabilities don’t matter at all (if I recall correctly, the complaint about Aeon’s limited export capabilities was somewhere at the beginning of our discussion). The same applies to the other visualization software.
I looked at Causality a long time ago; I liked it a lot, but it is heavily specialized in screenwriting. Its basic idea of visualizing setup and payoff may now also be achieved with Aeon Timeline 3.
In the meantime I have completely changed my Aeon 2 to yWriter converter from csv to .aeonzip format. That means yWriter projects can now be created without the detour via Aeon’s CSV export. Before the release, however, I want to do a little more testing.
The processing of the TL3 .aeon format is done in the same way, except that there is an additional layer, and accordingly more UIDs have to be collected. It is only a question of time …
Anyway, November is approaching, and with it the end of toying around and the beginning of real writing.
It will absolutely be possible to create an addon (gramplet) for Gramps that can utilize the xml from The Timeline Project, It should also be possible to make a gramplet that can read/write to the AT3 format, like you do for your yWriter project.
But I do not have the knowledge and patient to sit down and learn Python, even though I know the theory around programming and the logics, I just can’t manage to focus my thoughts on it… even though I can sit for hours searching for things in newspapers… it’s strange how our brain works…
(Maybe I should write a novel about it, I am sure it could have been a great science action real dark story at the end… hahaha, but actually, that is one of the reasons why I want to have my research in the same timeline as my research object, and linked together, to record the parallell stories, when did I work on what object, how much time, what did I do etc., so I am actually creating a AT3 template for that… if I get it working as intended, i will share it…)
I will look at your code and see if I can learn something when you publish it… and maybe I can ask for som help/advice, if I see that I maybe can figure out something…
I also used to use Clooz, and are testing Clooz4 beta, Clooz is a document focused research and register software for genealogy research, but I think the developer wants to do “more” with it as soon as he have has finished version 4… again the problem of using multiple software appear, different really good and helpful feature but in different software…
This is the code that iterates through Aeon 2’s JSON data and builds a yWriter novel structure from it.
In the meantime, it seems feasible to me to create an .aeonzip file from a yWriter project the other way around. But now NaNoWriMo has priority.
Funny, with me it’s the other way around: Even when I’m feeling so bad and all creativity fails, I can still program all day. That’s what’s kept me going for a living.
Go for it! Here’s the place to be in November.
Cheers,
Peter