This adds helper functions to determine the first or last directory separator
in a string and to check if a given character is a separator.
Paths starting with a separator are now also considered absolute on
Windows as these are rooted at the current drive.
Note that it's fine to use DIRECTORY_SEPARATOR when combining strings as
Windows API calls accept both forward and backward slashes as separators.
Co-authored-by: Michał Skalski <[email protected]>
References #3684.
This reduces the clustering problem (primary clustering) but is not
completely free of it (secondary clustering) it still reduces the maximum
and average probing lengths.
With the previous approach we'd require at least an additional pointer
per item to store them in a list (15-18% increase in the overhead per
item). Instead we switch from handling collisions with overflow lists to
an open addressing scheme and store the actual table as variable-sized
indices pointing into an array of all inserted items in their original
order.
This can reduce the memory overhead even compared to the previous
implementation (especially for smaller tables), but because the array for
items is preallocated whenever the table is resized, it can be worse for
certain numbers of items. However, avoiding all the allocations required
by the previous design is actually a big advantage.
Depending on the usage pattern, the performance can improve quite a bit (in
particular when inserting many items). The raw lookup performance is a bit
slower as probing lengths increase with open addressing, but there are some
caching benefits due to the compact storage. So for general usage the
performance should be better. For instance, one test I did was counting the
occurrences of words in a list of 1'000'000 randomly selected words from a
dictionary of ~58'000 words (i.e. using a counter stored under each word as
key). The new implementation was ~8% faster on average while requiring
10% less memory.
Since we can't remove items from the array (would change the indices of all
items that follow it) we just mark them as removed and remove them once the
hash table is resized/rehashed (the cells in the hash table for these may
be reused). Due to this the latter may also happen if the number of stored
items does not increase e.g. after a series of remove/put operations (each
insertion requires storage in the array, no matter if items were removed).
So if the capacity is exhausted, the table is resized/rehashed (after lots
of removals the size may even be reduced) and all items marked as removed
are simply skipped.
Compared to the previous implementation the load factor/capacity is
lowered to reduce chances of collisions and to avoid primary clustering to
some degree. However, the latter in particular, but the open addressing
scheme in general, make this implementation completely unsuited for the
get_match() functionality (purposefully hashing to the same value and,
therefore, increasing the probing length and clustering). And keeping the
keys optionally sorted would complicate the code significantly. So we just
keep the existing hashlist_t implementation without adding code to maintain
the overall insertion order (we could add that feature optionally later, but
with the mentioned overhead for one or two pointers).
The maximum size is currently not changed. With the new implementation
this translates to a hard limit for the maximum number of items that can be
held in the table (=CAPACITY(MAX_SIZE)). Since this equals 715'827'882
items with the current settings, this shouldn't be a problem in practice,
the table alone would require 20 GiB in memory for that many items. The
hashlist_t implementation doesn't have that limitation due to the overflow
lists (it can store beyond it's capacity) but it itself would require over
29 GiB of memory to hold that many items.
The main intention here is that we can change the hashtable_t
implementation without being impeded by the special requirements imposed
by get_match() and sorting the keys/items in buckets.
This can improve negative lookups, but is mostly intended to be used
with get_match() so keys/items can be matched/enumerated in a specific
order. It's like storing sorted linked lists under a shared key but
with less memory overhead.
While it's unlikely that so many (large) items are allocated, this is
technically more correct. The result previously could overflow an
unsigned int (the conversion to size_t happened afterwards).
This was quite confusing previously: While calling insert_before()
and then remove_at() properly replaced the current item, calling them the
other way around inserted the new item before the previous item because
remove_at() changed the enumerator's position to the previous item.
The behavior in corner cases (calling the methods before or after
enumeration) is also changed slightly.
This will allow us to implement e.g. enumerator_cleaner without having to
use that unportable 5 pointer forwarding or having to define a callback for
each instance.
A generic implementation for enumerate() is provided so only venumerate()
has to be implemented, which may be simplified by using the VA_ARGS_VGET()
macro.
Because enumerate() for value based arrays returns a pointer directly to
the internal array elements and because array_remove_at() or rather the
called array_remove() may move elements over the element at the currently
enumerated position, the pointer passed to enumerate() will point to a
different array element after the array_remove_at() call. The caller
will thus operate on the wrong element if that pointer is accessed again
before calling enumerate().
For performance reasons we currently don't change the implementation to copy
each array element during enumeration to a private member of the enumerator and
return a pointer to that. Similarly, due to the danger of subtle bugs we don't
remember the pointer passed to enumerate() to later redirect it to a copy
created during the array_remove_at() call.
While glob should return a NULL terminated gl_pathv when having no matches,
at least on OS X this is not true when using GLOB_DOOFFS. Rely on the
number of matches returned in gl_pathc, which seems to be more reliable in
error cases.
This enumerator is a wrapper around glob(3). If that function is not
supported NULL is returned. If no files match or an error occurs during
the pattern expansion an error is logged and the enumerator simply returns
no items.
RFC: if GLOB_ERR is not supplied glob returns GLOB_NOMATCH if e.g. the
base directory of the pattern does not exist, which would otherwise
result in an error. This way there is at least a clear error message in
case of a typo.
Otherwise, array->esize is promoted to int and if array->esize * num
results in a value > 0x7fffffff the return value would be incorrect due
the implicit sign extension when getting cast to size_t.
This required some refactoring when handling encrypted payloads.
Also changed log messages so that "encrypted payload" is logged instead
of "encryption payload" (even if we internally still call it that) as
that's the name used in RFC 5996.
Its functionality can be replicated by calling insert_before() followed
by remove_at(). Not the other way around, though, because remove_at()
changes the enumerator position.
Currently we use the very versatile linked-list collection to store elements
with variable count. This is fine, but very inefficient: Due to the many
methods in the linked list, on 64-bit platforms an empty list alone is more
than 200 bytes. As we currently have about 50 lists per IKE_SA/CHILD_SA pair,
this takes up to 10KB just for managing the empty lists. This is about the
half of memory used by an IKE_SA/CHILD_SA pair, and obviously way too much.
The new array type is not an object, but a collection of functions on an
abstract type.
The following lists are per IKE_SA and should be considered for a replacement
with more efficient arrays (this uses load-testers on-demand created dynamic
configurations, other scenarios have different lists):
14 -> ike_sa_create() @ src/libcharon/sa/ike_sa.c:2198
10 -> auth_cfg_create() @ src/libstrongswan/credentials/auth_cfg.c:1088
6 -> task_manager_v2_create() @ src/libcharon/sa/ikev2/task_manager_v2.c:1505
6 -> proposal_create() @ src/libcharon/config/proposal.c:592
5 -> peer_cfg_create() @ src/libcharon/config/peer_cfg.c:657
4 -> child_sa_create() @ src/libcharon/sa/child_sa.c:1090
2 -> child_cfg_create() @ src/libcharon/config/child_cfg.c:536
1 -> ike_cfg_create() @ src/libcharon/config/ike_cfg.c:330
1 -> put_connected_peers() @ src/libcharon/sa/ike_sa_manager.c:854
Since we always search for the nearest separator (and strip them from
the front of the next token) there can't be any separators left at the
end of a token.
Tobias Brunner