Tree cache for timetable loading - #283
Open
Jonas-July wants to merge 86 commits into
Open
Conversation
To efficiently store bitfields, they are used and reused across timetables. A hash_map maintains this mapping to allow quick checking of existing bitfields. After loading, this hash_map isn't needed anymore, so the timetable only stores the mapping from indices to bitfields (as a vector). From the timetable, it is possible to reconstruct the hash_map. This eliminates another global variable. For now, only assert that the reconstruction and the original are equal.
Since both hash_maps are equal, use the reconstructed one to eliminate the global variable.
Intermediate timetables can later be reused for caching purposes. Thus save them inside a cache directory. The naming is chosen such that each timetable gets its own directory; this may be expanded in the future.
Useful for hot restart/caching
The change detector is supposed to identify sources that have changed or rather have not changed since the last run. To be able to make that determination, it checks that the filepath and config are the same, and the file hasn't been written to since the last run. This then implies that the cache file is still valid. As such, save that information for later use. If saving fails, emit an error in the log, but don't abort the program since it still functions, just without the cache. The timestamp returned by fs::last_write_time is relative to some filesystem clock which may or may not change at any time. In order to reliably compare those timestamps, they are converted to the system time which represents time as the Unix epoch (since C++20, previously not defined but very likely). To then store the value, it is converted into a 64bit int representing the nanoseconds since the Unix epoch. The configs are hashed and then compared. Hash collisions are negligible.
Since the intermediate data necessary for caching is stored, we can use it to load from a previous checkpoint when restarting an import. To find out which sources we can skip, we load the previous change detector and compare it with the current one. Due to the linear dependency of importing, we can just check that each source in the list is unchanged in the order they are presented. The source arrays must be the same size, otherwise the caches may overwrite each other, making them invalid. If a changed source is found (path and/or last write time don't match), all successive sources will also need recomputing, thus break. In order to restart from cache, we need to load the cached result. Due to linearity, again, we can just load the result of the previous source which hasn't changed and thus would produce equivalent timetable and shape data. There could still be issues though. If the cached timetable cannot be loaded for some reason, we can't use it and have to recompute it anyway. The same is true if the date_range doesn't match. In that case, the then previous one has to be loaded (and the one before that, etc.). If the very first source has to be recomputed, there is no cache left, and the initial timetable with the special stations is created and provided. The timetable is always finalised at the end, so changing finalize_opt don't have to be considered. If the assistance_times change, the cache may be invalid and has to be deleted. The code doesn't check for that. It's probably possible to reorder the input sources to potentially get more in common with the cached state, but that would make equivalence a bit harder to prove. The order usually doesn't change anyway or at least it can be prevented on the caller side.
The shape_idx and shape_offset_idx are relative to the shape storage, so adjust them when merging. If the indices are invalid, simply adding them together leads to an overflow which then produces a valid index. The value however is wrong and can lead to crashes or silent issues. If a value is invalid, the adjusted value is also invalid.
The bitfield indices directly impact the timetable in various ways. This means that the resulting indices need to be adjusted and the references corrected. To that end, affected vectors of the original timetable are saved, before the bitfields_ member is cleared. We can then load the file with an empty hashmap. When the file has been loaded, the new traffic_days are calculated. This is necessary as the old ones use the correct indices, the new ones however don't and thus need to be adjusted. Later on, this won't be necessary as such when an empty timetable is passed to the loader. Similarly, the old data is restored where relevant to better show the merge. Finally, merge the new bitfields into the existing timetable and store the adjusted indices as they might be non-linear. This mapping is then used to correct the bitfield indices for the traffic days. No behaviour should change due to this commit.
The location data is grouped by type, thus each vector needs to be merged separately. A few variables don't get used during loading, only during finalising, and thus don't need merging (they'll always be empty). The routes contain sequences of stops which reference the location_idx. The stops however only use 28 of the 32 bits which the location_idx_t has, in order to bitpack the other 4 boolean properties into a single 32 bit type. This means that the packed value needs to be turned into a stop to then map the location_idx(). The location index of stops isn't just the location_ variable, because the location could be invalid. Since the type is shorter, simply comparing against location_idx_t::invalid() doesn't work. Instead, create a stop from an invalid value and check if the location_ is the same. Then return location_idx_t::invalid() to satisfy the interface. Otherwise just recast the location_. Correspondingly adjusted the indices.
bitvec doesn't have an "append"-like operator, so instead loop over indices for merging. route_bikes_allowed_per_section_ and route_cars_allowed_per_section_ are manually resized during loading to correspond in size to route_idx_t. However, if no routes are added, that resizing doesn't occur and hence, the vectors might be empty. So prepare them to the correct size after clearing. The empty values don't get used.
The merge logic doesn't have much to do with loading timetables in general and is heavily dependent on the structure of timetables. Thus, move the merge logic into a function inside the timetable class. There is no change in behaviour expected. For completeness, a description of the structure of the new method is provided: 1. Some fields don't get used during loading and always remain empty. They don't need to be merged and aren't handled. This precondition is checked in case this assumption changes in the future. 2. date_range must be the same between the timetables. It is set from the outside and it is assumed that at least one timetable has the correct date_range (and then all of them have if no errors occur). 3. This isn't strictly necessary, but does provide a quick overview of which fields are handled. They are all read-only of course. The order is based on the order of definition in timetable.h 4. bitfields and string_idx map have to be merged first to determine the new indices. They are then passed to the index mapping. 5. The index mapping is a read-only structure which determines the mapping from the (relative) indices in other_tt to the new absolute indexing. 6. The other fields get merged independently from each other. The mapping indices only depend on some offsets that can be statically determined beforehand. The merging is done structurally, with some exceptions. Each field gets merged according to its type, until objects are reached which don't need merging as they stand alone. However, they might contain or be references which need redirecting to new targets since the merge invalidates most references within other_tt. This too can be done mostly structurally by mapping the members of composite types and mapping affected reference types. Care must be taking with possibly invalid references as they must remain invalid after mapping.
Small refactoring towards a tree like structure
To avoid issues with imports, use explicit namespace notation and explicitly import the relevant header files.
This becomes useful when only wanting to load changed sources
The new struct `loading_source_node` takes on the responsibility of loading the timetable from source and returning it. The node contains all needed information to load the source file if necessary. The loading, however, can be postponed, enabling possibly lazy evaluation.
The `loading_tree` structure manages the binary cache tree. This
design allows arbitrary binary trees to be loaded from cache. The
loading is done via recursion over both children.
The nodes are referenced via indices and their relationships are
described by the parents_, left_, and right_ vectors.
The first node (idx=0) is always the default timetable. After that
follow the other source based nodes, and then the intermediate merging
nodes. In total, there are always 2(n+1)-1 nodes, where n is the number
of timetable sources.
Each node has a corresponding cache location to which it will save its
result. By default, it is the node index, but that can be changed if
necessary. To avoid issues between the automatic and the manual indices,
the manual indices can't be set to an automatic value.
Building up the tree has 5 steps:
- Insert the default node at index 0
- Insert source nodes up to n+1
- Determine the updated sources by comparing with the previous cache
- Insert intermediate nodes. They are updated if either child is updated
- Write the cache structure to disk. From now on, it is constant.
Loading from the tree-like cache is done via tree traversal. There are
3 types of nodes:
- The default timetable node at index 0,
- the loading nodes up to l.source_paths_.size(),
- the merging nodes after that.
1. The default timetable gets constructed on the fly.
2. The latter two will check whether they need to be recomputed. If they
don't, they'll try to read the timetable from the cache.
3. If the node needs to be recomputed (either due to update or cache
failure), we need to determine which kind of node it is:
3.1. Merging nodes call both children recursively. If that fails and they
return nothing, then they get skipped. If both have been loaded
successfully, their timetables are merged and written. The shapes
only get saved if both are available; Usually either both are or both
are not, as that is dependent on the global setting of whether to
save the shapes or not. If they get saved, the left gets added first,
then the right one.
3.2. If it is a source based node, then it will try to load from source.
On success, it writes the resulting timetable into the cache
location; The shapes are automatically written to there during the
loading from source.
In case the loading fails, this is propagated upwards, where it
can be skipped.
Finally, the timetable gets finalized. This is unchanged.
Some nodes are source based and thus need access to their underlying source. Previously, this was handled by fixing the positions relative to each other. Since the nodes may need to be reordered to facilitate the tree creation, storing the mapping explicitly makes that easier. The addition of source based nodes is also now handled inside the loading_tree, only requiring a simple function call to add a source.
The current tree structure is close to a degenerate binary tree. Using a complete binary tree is usually more efficient for evenly spread out updates and is easily computable. The initial slots are filled with the source based nodes. After that, the merging nodes are created in a similar manner to binary heaps. The most obvious difference is the reversed ordering due to insertion order (starting at the leaves instead of the root). This algorithm produces a complete binary tree. The algorithm chooses the two left-most nodes without a parent to set as children for the next node. This is done until the next node would be only node left. This means that the merge order is not necessarily the same as the ordering inside the array if the source nodes are at different depths (always the case unless the number of leafs is a power of 2). However, the default timetable node is required to be the left-most merged node due to the special stations. With this tree creation algorithm, it suffices to rotate the sources by the difference to the next power of 2 (that is the number of leafs missing on the last level).
Currently, the cache strategy assumes that the cached tree structure is identical to the current loading tree. Any deviation leads to cache invalidation and wrong results. This puts severe restrictions not only on the cache structures that can be accepted, but also on the number of nodes contained within them. In particular, adding or removing sources invalidates the cache even if the same algorithm is used, leading to extreme performance degradation. The dynamic matching of nodes during the tree building checks that the assumptions are correct and attempts to make good use of the existing cache. The approach here is not optimal for all existing tree structures, and doesn't try to adjust the new cache structure to the existing one. It does, however, produce correct results with all valid cache tree structures. --- The matching works on this algorithm: A merge node can be reused if the underlying sources are unchanged. The ordering of the sources doesn't need to be checked because any valid reordering produces essentially the same timetable. Thus, the sources can be sorted for easier matching. Due to the tree structure, the sets of children are unique for each node. Source based nodes are the same if the source data is unchanged. Thus, the path can be used to quickly determine a match candidate. The default timetable node has the reserved cache index of 0. When a match has been found, it's cache index is transferred to the new node. That index might already be used by another node, in which case they swap them. Since each cache index is unique, this swap doesn't mess with the matching. --- In summary, the loading from source is only done with new sources, while cached merging nodes may or may not be useful depending on the differences in structure. Best performance is achieved when prepending sources on the left.
felixguendling
force-pushed
the
master
branch
2 times, most recently
from
December 11, 2025 14:31
a4efa76 to
116082c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Currently, loading a timetable means reloading every timetable, which can be very time-consuming.
Using caches and an efficient tree cache structure, this can be avoided, improving loading times considerably at the cost of an initial penalty during cache creation. Subsequent updates can make use of the cached timetables to save on recomputations.
Features
In pursuit of this goal, it is necessary to make the timetable merging explicit (that's currently inplace). This is done granularly to ensure correctness and make debugging easier. These commits consequently make up a large portion of the commits in this PR.
Another feature is the dynamic matching of timetables, whereby the given sources are not just mapped to caches based on their position, but their cache location is chosen depending on the current cache state. This is particularly useful when rearranging sources as is done e.g. when adding and/or removing sources.
The loading tree is implemented as a complete tree (stored in breadth-last order). This is an easy data structure with a logarithmic number of merges when updating.
Performance
TODO
Full transitous data set took 45 min to init the tree, and 8 min on a subsequent update, on my system
The cache takes up 305GiB (roughly 11.3x final size) on disk
Possible improvements
The nodes are still computed sequentially. The tree computation itself can be parallelised to save on time during initial construction; later updates have less impact. There, the merge function can also be parallelised to speed up.
Reading and writing the caches from disk also takes up some time. This can be improved with faster disks, and a fast merge function could allow foregoing the cache (almost) entirely, eliminating the disk writes.
Further information
Further information on the rationale and implementation for the interested reader can be found in the commit messages.
Important milestones are: