I would like to try and link/sync an Aeon timeline directly with an external data source (a bit like the Scrivener sync feature). I noticed that the timeline data is stored in a binary file in what looks like JSON format.
I would like to create timelines directly from DB data and have changes made in the timeline written back to the DB, without going through an Import/Export procedure. Is this possible, and what would be the right format to read open/read/write the binary *.aeon file?
As far as I know, aeon files are just compressed json files. So, you just uncompress, edit and compress again.
Cheers,
Matt
If you know Python, you might wish to have a look at my JSON extractor script.
A more sophisticated application example is my Aeon3 to yWriter exporter. Based on its code, there’s also an Aeon 3 importer plugin for LibreOffice.
Please note that my code handles the file format as it looked around the time of the Aeon 3 beta test. Whether this has changed in the meantime, I can’t say.
Cheers,
Peter
2 Likes
A few more technical details:
The Aeon3 file, as I know it, is composed of a JSON part and a binary part. My extraction script separates the JSON part, formats it and saves it in a text file.
The file format is not easy to figure out, because the flexibility of the software is reflected in the data structure. That is, it is quite an extensive tree full of cross-references, which in turn are defined in other sub-trees. By the way, so far this is also true for the Aeon2 file format.
However, the Aeon3 format adds an extensive amount of redundancy, which, as far as I understand it, is supposed to speed up loading, because it takes over various program states directly.
Thus the Aeon 3 format can be read by third-party software, but hardly written. By the way, this was the decisive reason for me to stay with Aeon 2, whose format I can read and write completely, which I use for synchronization with yWriter.
If you want to send data to Aeon 3 with your own program, the csv import might be the easier way. Apart from that, there was also once talk of an API for synchronization, but I don’t know if that has been realized yet.
1 Like
Thanks for the pointers. Glad to see I am not the only one with “funny” ideas 
The binary part seems to be the JPEG image of a jellyfish (go figure…)
The rest of the binary is JSON text. I am going to try your scripts and if I have to change something to make it work for my use case, I’ll send a pull request to your repo. Stay tuned…
1 Like
There is no need for that. Just “clone and own”, if you like. 
Have fun!
1 Like
FYI, I just updated the JSON extractor script, fixing broken library dependencies and making it run with the latest Python version.
It’s been so long since I built the last version … 
2 Likes
Your updated script is working very well! I like the level up/down trick - haven’t thought of that.
I also learned that the binary part of an *.aeon file are attachment(s) to an event. The jellyfish above is actually from an entry in the “Feature Overview.aeon” example that comes with Aeon TL.
Reading the JSON of an Aeon file into an external DB is solved. But writing back to it looks like a hard problem. Every event has a hashed identifier of length 21 which will need to be added to a (new) event/entry. I haven’t figured which hashing algo is being used, and honestly, I don’t feel like spending time on this.
So plan B: I’ll customise a new, blank Aeon file and then see if I can get some mapping automation between the Export/Import working ok. An earlier import test produced several error messages because the data types were missing from the blank Aeon target file (instead of creating them).
If that doesn’t work (like you, I am struggeling a bit with the Aeon data structure), I’ll bite the bullet and go over to plan C and use the Open Source Timeline and its “wonderful” XML 
BTW, I saw that you have done some work with it and yWriter already. I just wish Aeon made integration a bit easier…
1 Like
I’m glad to hear that you like my tool. If you can convert the Aeon3 project to database format, you have already done most of the lookup work. You can see the Python class that does this for me in the json_timeline3.py module here:
Creating UIDs isn’t rocket science. For Aeon 2, I generate them with a standard hash algorithm, deriving the keys from the node type and label. I guess, this works also for Aeon 3. You can find the basic algorithm in the uid_helper.py module here:
By the way, in the aeon3yw repository, there is also a (still unused) uid_helper.py module, which creates UIDs based on random numbers. However, I prefer a non-random algorithm in order to faciliate testing.
I have the open source “Timeline” software in view in case I switch to Linux later; however, I find Aeon Timeline 2 more pleasant to work with under Windows.
Timeline’s XML file format is somewhat easier to read and write, but it also has its pitfalls. Most importantly, the application itself is extremely sensitive to deviations from the (implicit) specification, for example, regarding the order of XML elements. Undocumented file format extensions can quickly become a showstopper. And some other concepts are also implemented in a somewhat tricky way. In any case, an exciting area for the programmer who likes to experiment. Have fun!
1 Like
Could you share your code? a link to your github or something similar?
I am not a developer, but I really would like a way to utilize my Aeon 3 data for other software and a database format that I might be able to import to mongodb/neo4j or use with cytoscape/gephi or Tulip Network Graph Software would be of interest for me…
I guess the easiest way to do this is to use the csv export from Aeon 3, where everything is sorted and correctly labeled. For csv there are (e.g. via spreadsheets) proven ways into the databases.
When I wrote my file converters for Aeon 2 and Aeon 3, I got a csv importer to work first each time. Once that worked, I went the rocky road of JSON format to speed things up.
However, the csv export of Aeon Timeline 3 still looked a bit incomplete at the time of beta testing. If I am not mistaken, short names were missing. I hope that something has been done here in the meantime.
Yeh, we have talked a little about this before, my hope is to be able to work without importing and transform data in Excel or other spreadsheet tools before using the data in e.g., Gephi/Cytoscape or other graph network software or neo4j… or maybe some online timeline enabled tool.
Sure, but you won’t be able to get around analyzing and filtering the data before it gets into the other app, will you? With a spreadsheet, even users without programming skills, or at least with the widespread know-how in macro programming, might be able to do that.
Once it works through the spreadsheet, and a proven workflow has emerged, it’s all the easier to specify a dedicated converter software that digs the right stuff out of the aeon file’s complex data structures.
At the moment we can agree about that… BUT, and it is a huge BUT… If we, the users don’t start to ask for open data and open standards, nothing will ever change.
BUT, if we are active in asking for and discussion open standards and stop use software that doesn’t conform to open data stadards, developers that want to earn money on their products need to start listen…
And for data in e.g., Aeon, there is not a problem to make a full export to a format like JSON-ND, GraphML or other open standard interchangeable file formats that will be easier to use for those that would like to use their data in multiple applications, it will also be better for the developer that do not have to bloat their software with a lot of extra features.
Just one example that is a digression here…
If there was a full export of Aeon entities and object to Markdown with YAML keywords, those Notes could be used as a “hub” for other research tools, as it is possible now by using CSV and a transcoder script for CSV to MD or by writing a VBA script in Excel or doing the same with a Python script.
The problem is that most users do not know how to, but if they could export to e.g., JSON-LD, the data could with little effort been imported and utilized in multiple other research tools, in addition to web publishing tools…
But, when users just silently accept anything a developer do, and keep on paying them money for “nothing”, nothing will ever change…
Just see how Microsoft have changed their attitude toward open source and open data the last 2 decades.
I agree with you about the requirement for open standards for data exchange.
However, experience shows that while text-based formats such as XML or JSON represent a major advance over binary formats, they do not solve the problem per se. A prominent example is Microsoft’s OXML format, which is a pro forma ISO standard, but with its huge specification has never been fully implemented. I wrote a yWriter converter for it as well, and it was quite ugly compared to the ODF files, which are almost as easy to read and write as HTML.
As far as Aeon Timeline is concerned, first of all it must be clearly stated that .aeon cannot be an interchange format in any way. It reflects all the complexity and particularly flexibility of the application, and perhaps also the shortcomings of the programming framework used.
As the application gets new features, the data format has to be extended as well, which in my opinion pretty much rules out standard compliance.
There is probably some open file format that can also be used for timelines, but for practical reasons that would only be considered as an additional export format. I can’t even say whether it brings any added value compared to the csv files. (*) csv could also be called an open format (as long as it is not to be read with Microsoft products – Excel has its own rules again). Anyway, it is easy to read and write.
Finally, an example of how open file formats can limit an application: When LibreOffice got a new rendering engine, a number of nifty features were implemented right away to make use of the typographical possibilities of the latest Open Type Font standard. Unfortunately, this is more DTP than word processing, and there wasn’t a way to store the necessary parameters in the ODF document format. So they invented a notation in which the typographical parameters were pasted to the font names in a somewhat backward compatible way. If you search for a while you will find the specification and the only way to turn off ligatures for example. Mind you: This solution requires you to change the font names manually and append codes from the respective font. You cannot access the options via the user interface.
(*) Edit:
Yes, it does add value because the format probably contains semantics that are missing in csv files.
Having gone into general thoughts earlier, I want to briefly address your suggestions:
This is close to what my JSON extractor mentioned above does. But as I said, not much is gained yet with JSON or NDJSON, because the organization of the data still needs to be figured out. And the actual interesting semantics of it all are largely user-defined, unless one of the default templates is assumed.
Well, this could represent some of the new views of Aeon Timeline 3, but would be hard to relate to the actual timeline. And it would not fit with the JSON-based solutions.
I must confess that until now I didn’t know that at all. But if I understand it correctly, JSON-LD is used to link data on the Internet. As a simplification compared to XML with XPATH? At the moment I don’t see the relation to Aeon Timeline, apart from the fact that there the data is linked in a complex way internally and the parsing requires a lot of lookup. But that’s hardly what JSON-LD is about, is it?
I’ll stick with this: at the lowest level, it’s not hard to use standard formats like XML, JSON, CSV, or YAML. This at least makes the data readable. But the semantics, that is, the standardized organization of the data, the defined meaning of the elements, that is all still open and requires a lot of detailed work.
And setting standards here means limiting the range of features.
A good example of how difficult something like this can be is the XML format of the open source project Timeline. At first it was straightforward and quite elegant. But when they started grouping events, they implemented it with a hack by adding prefixes to the event labels. And there you go, the whole thing was awkward and error-prone to parse. And, of course, there was no idea at all that third-party software should parse the whole thing. Instead, there is at least an API for custom export plugins. But that in turn did not fit with my concept at all. So much for open software.
Well, if people always look for problems, they will never find a solution… It’s the same as in real life… if you only look for all the problems stopping you, you will never get out of the “Devils spiral”.
I do understand your point. To avoid any misunderstandings: Problems don’t have to stop you, but you do have to recognize them in time and take them seriously in order to be able to solve them.
At least, that’s how an engineer thinks. Building something that works is often easy. But building it in a way that ensures there will be no trouble requires experience and can end in tedious day-to-day business.
What I ment with the json-ld, json-nd, graphml etc. or even the Cytoscape json format, is that any newer version of a Network Graph or just “graph” file format, holds everything you can put into a timeline in AEON 3, even the “complex” relations…
Remember that a network graph can have thousands of edges to a node, and it can have thousands of nodes, the right file format will be able to have clusters of nodes (don’t remember the correct term atm.), those file formats can also have time/date periods, multiple start and end dates etc. AND those newer “standards” also support time series, they also support custom attributes fields… all this will allow a user use e.g., Cytoscape or Gephi to analyze their data in different ways than possible with AEON… for those of us using AEON as a tool for e.g., historical research or other type of projects that is not “writers project”, I do not know how much use it is for a writer to import their data into Gephi to make e.g., a directed network graph, but for a project with historical data, with maybe 50-60000 possible linked entities, it might be a way to find new connections in the data, based on attributes not linked in the Aeon Timeline.
Yes, it is possible to do that in Excel, but if you want a visual way of presenting the links/relations, a network graph would be helpful, especially if it was a network graph plotted on a map with a dynamic timeline… And yes, it can be done by importing multiple CSV files to the same software, but you will usually need some way to split the Edges (links) from the Nodes for an import to a graph software…
Not everything will be “solved” with an open standard graph format, but it will be a lot easier to interchange data between different type of software, especially for researchers…
Yes, that sounds interesting, although as a writer I don’t see a need for me here. My problem is more the notorious “world-building disease” that you get when you accumulate too much metadata.
Anyway, a more accessible file format would really be a plus and might also make me consider switching from Aeon 2 to Aeon 3.
Although you could get a lot out of an .aeon file when it comes to visualization only, with scripting. However, it is work after all.
Can anyone actually say something about the file format of the current beta version? It is supposed to have been changed, isn’t it?
1 Like