Exporting Aeon Timeline 3 to Obsidian

Templates and Showcase
#30

A general Markdown export is something I have been requesting from the very early days of alpha testing Aeon Timeline. I want to be able to take an outline from AT3 and put it into Ulysses. But an export to Markdown would also work as the basis for an Obsidian integration, no doubt. There has been some teasing of adding further ways to export in future beta versions of AT3. But whether Markdown is one of those formats I have no idea. With Ulysses I can take word files and convert them easily and accurately into Markdown. So I have a few options for possible Ulysses integrations from AT3 in the future.

On a personal note I have no need of Obsidian. I have a copy of the app and hold an insider license, but I find the whole back link approach too much for my fiction workflow. Obsidian is a hugely impressive piece of software but now Ulysses has its own powerful internal linking with backwards and forwards navigation, I can’t ever see me needing Obsidian’s feature set.

I do hope those seeking Obsidian integration get what they want from AT3 though. Especially if it means we all get a solid, flexible markdown export.

4 Likes
#31

For whatever it is worth: I think it is very difficult to write an export or sync script as every need is very different. It would be nice if Aeon either confirmed that the file format is frozen and won’t change, so no effort is wasted, or if they even would provide documentation to help write a generic module (probably in Python) to read the files. I did some research earlier this summer but it was simply too much for me to spend time on.

I use Aeon to plan a roleplaying campaign and in that I have a lot of events which I want to refer to in my text. I write in Markdown, which I process using Python and GPP to generate HTML. I export the timeline to CSV and have a Python class parsing that. I then write things like

Frodo left Bag End on the @aeon(Frodo Leaves Bag End)(start:dth) of @aeon(Frodo Leaves Bag End)(start:mmm), the @aeon(Frodo Leaves Bag End)(start:mth) month.

and in the HTML I will get

Frodo left Bag End on the 23rd of September, the 9th month.

The code is quite simple but extensive since it adopts the output to if I write in Swedish or English (@aeon(Frodo Leaves Bag End)(start:dth) would render in English as “23rd” but in Swedish as “23:e”).

#32

What a great idea! I’ve got to try this because Obsidian is my canon repository of knowledge for my saga and I’m just now getting started with AT3.

However, I’ve run into a snag when trying to use your script.
Could you please have a peek at this Grok conversation and let me know what I’m doing wrong?

Might the .aeon file format have changed? Is this script still compatible with the current format?

1 Like
#33

There nothing you’re doing wrong, except ignoring the Requirements section on the aeon3obsidian home page. As stated there in bold, as well as in a recent post in this thread, the AT3 file format has changed since I figured it out during the Beta test phase. Because I don’t have an AT3 licence, I wouldn’t do any further development on the script. However, it’s Open Source, waiting for someone to adopt the code.

Anyway, LLM’s response provides some useful hints on where to start when adapting the code to the new file format. The first step is to extract and analyze the JSON structure of your .aeon file. Then adapt the read() method of the Aeon3File class in the aeon3_file.py module so that it finds the elements in the right place in the data tree.

If we are lucky, the required JSON elements will all still exist, but they will have been moved to other hierarchy levels in the structure. However, the file format can be silently changed again at any time.

1 Like
#34

Thanks for the reply! So the short of it is that the script is currently broken until someone can pick apart the AT3 project file format and teach the script how to find the relevant information.

I might be able to do it if I put a lot of time into it. I’ve done some crazy stuff with Python in the past, but it’s been a while. LLM might be able to refresh my memory, but it’d be a time commitment. The script does sound like a fantastic tool though! I could really use the ability to export Aeon data into Obsidian. Thanks for your work with it. I’ll have to do without for now.

#35

“Broken” is too harsh a word. I’d rather call it “outdated”.

Crazy stuff with Python is not necessary at all, and I would never rely on LLM-generated code. How to deal with JSON files is very well explained in the Python documentation and is not rocket science. Probably the biggest learning achievement is to fork the GitHub repository, commit the changes in a properly documented way, build the release package and create a pull request.

1 Like
#36

That’s cool. I might give it a shot.

I do like to use LLM to point me in the right direction when dealing with unfamiliar techniques. Like I used it at work last year to create a JSON parser that extracts our company’s sale data from our ERP’s API. It was quite a learning experience since I had zero experience with any of that.

At any rate, I just created a sample AT3 file with a single test event and gave it to Grok to see if it could find the event and point out anything notable about the structure. Here’s what it found.

#37

You can of course do it the way you are used to, but be aware that there is no intelligence behind the LLM. The answer I saw merely reflects the complexity and redundancy of the .aeon file format. Better to start from the Python code, where you can immediately see what the required tags are called and in which parts of the structure to look for them.

#38

It looks like it’s fun to play around with LLM-generated code. Maybe you’ll even manage to publish your own tool.

To give you a reference, I just uploaded the zipped Obsidian vault created with Aeon3obsidian from the “Murder on the Orient Express” sample project to my GitHub repository:

Good luck!

#39

A look at the LLM answer shows that it’s not quite that simple. In the new AT3 file format, the JSON elements of interest have been both restructured and renamed.

For instance, the old item dictionary in aeon3obsidian.py is

jsonData['data']['items']['byId']

whereas the new one in the LLM-generated script is

json_data["data"]["itemsById"]

But since it seems to follow a pattern, it could be quite easy to update aeon3obsidian.py. However, this should be done from a proper understanding of the JSON data.

#40

You’re correct that LLM doesn’t have any true intelligence, but it’s pretty good at generating code if you provide iterative feedback. Check out this conversation where I work with Grok to fix the script for the new AT3 project format and also enhance it to handle multiple eons rather than limit to the Gregorian A.D. calendar. Also working on enhancing it to provide item types and custom properties.

And to be clear, I haven’t written a single line of code here. It’s just me giving Grok the debug results of the script run. Going to put it on the shelf for the moment as I have some more pressing things to work on, but this was a pretty impressive test to see if LLM could solve the issue all by itself. :joy:

#41

But you can, if you add Scrivener to your workflow. Just sync AT3 metadata to Scrivener, then use Scrivener to compile the metadata into markdown files. You can even write the metadata to YAML if you want.

In disclosure, I’ve only confirmed this with the Scrivener folks, but haven’t done this step because I came up with another approach that I’ve already written about: AT3, Scrivener, and Obsidian - Q & A - Aeon Timeline

#42

Yes, it is interesting to see how such an LLM generates code. However, I found some unmotivated refactoring, as well as scattered changes where, for example, instance variables were replaced by parameters, which is a difficult source of errors to find.

Also, an obvious hallucination that wouldn’t go unnoticed with a decent IDE.
Original:

self._write_file(f'{self.folderPath}/__index.md', text)

LLM:

self._write_file(f'{selfaveringfolderPath}/__index.md', text)

Then the LLM contradicts itself: In the first iteration, it claims that the problem is the scan function and replaces it with a regex function. However, in the follow-up comment, it recommends replacing regex with a scanner. In the last iteration, I then see a scanner, which is a cumbersome variation of my code. I also see a test for Zipfile, which seems completely unnecessary to me.

Something like this shows me that with the LLM we have a counterpart who pretends to have superior knowledge, but in reality understands nothing.

The final version of the code is quite pimped out and a bit opaque and may or may not work for a particular example, but I don’t want to have to look for bugs in it. Instead of fiddling around with the code until it somehow works, I still stick with the good old programming skills.

Whatever the case, can you use it to convert the “Murder on the Orient Express” example satisfactorily?

1 Like
#43

Everything you’re saying is completely true. I’ve had LLMs (like Cody) take me down some pretty insane rabbit holes when trying to learn a new technology. It seems to get worse the longer the conversation goes on. Often just starting fresh will make it think clearly again.

One time it tried building for me some truly absurd generic JSON parsing methods in C# when I explained that our API endpoints all returned vastly different fields. It got the bright idea that we could make a generic import that would try to determine field names and set up a series of rules on what to do with them. Ended up wasting hours before I gave up and made my own solution.

But despite its flaws, I have to say it’s invaluable in specific cases. It’s superb at writing quick SQL queries to give you a custom result set you’re after if you just give it some example relationships and say “now make it do this”. It’s also really good at giving a summary of a new tech you’ve never used just to get you up to speed.

One Python script it made for me from scratch via me explaining a really customized video editing need and going through 10 versions to get it perfected is a script I’ve used hundreds of times since. It just works.

But ultimately, if you don’t have a technical brain that can take over for it once it starts going off the rails, it’s going to just make a huge mess of everything.

The same is true for writing fiction, for what it’s worth.
LLMs are awful at generating story prose. Like don’t even bother. But what it’s actually quite good at is giving honest feedback (if you ask it to be honest) on your story text. The prompts have to be worded in specific ways or you’ll get nothing but garbage flattery of course. But I’ve had some incredibly helpful conversations with LLMs to give feedback on my writing techniques.

#44

As far as story feedback is concerned, it often helps to read the story out loud to yourself. I also have a screen reader plugin in LibreOffice that reads the text to you in a rather monotonous way, which can lead to surprising insights.

Anyway, by now I’ve spent almost the whole Sunday with aeon3obsidian, so I almost feel like programming the necessary changes for the new .timeline version. It won’t be that much, but I’m missing a valid reference file.
Would you mind saving the “Murder on the Orient Express” sample project and putting it in the attachment, or emailing it to me offlist? .aeon files should be allowed in the forum, unlike other file types.

1 Like
#45

Sure, no problem!
Clicky.

If you’re looking for a good free text-to-speech solution, try loading up Microsoft Edge (yeah the one we all disable when we install Windows).

Then go to the following URL:

data:text/html,<html contenteditable>

Paste text into the blank page it gives you.
Then press CTRL-SHIFT-U
On the upper right corner, you’ll find Voice Options where you can pick from a bunch of different voices. I prefer Microsoft Brian Online.

#46

Thanks for the file and the information about the screen reader. Anyway, the LibreOffice plugin connects to the Windows system text reader.

I will now take a closer look at the new file format and return as soon as there are satisfying results.

Cheers,
Peter

2 Likes
#47

Fantastic! I’d be happy to provide feedback on the results as I will probably have a lot of ideas on the usefulness of the Obsidian notes. I’ve got a pretty complex use case that will no doubt stretch the limits.

#48

Most of the work was done quickly, but the relationships are still missing, which now have to be unraveled anew. I also still have to work on the narrative, which is no longer a complete tree but a kind of multi-linked list. But it’s just a bit of busywork that I want to do on the side over the next few days.

Just for the record: Which AT3 version did you use to save the reference project?
The file format version is indicated as 15, and the AT3 version 3.5 has already been announced, which should bring extensive innovations, which will probably also be reflected in the file format.

#49

I did some investigation last year (as you might remember @Peter_T ), and the conclusion then was that the file format might change without warning (it is not stable, for better or worse) and there are quite a few cross references. It was the latter that made me give up because it is a lot of reverse engineering is needed. I think deducing the cross references without support from the Aeon team would be painful.

Having some kind of python library to read the files would be beneficial to Aeon IMHO and something the Aeon team should help support. This would greatly help integration with any other application and in the long run grow Aeon’s customer base. All IMHO of course. :relaxed::laughing:

1 Like