If it takes a while to start one of the threads, another thread might already
have passed the usleep() call previously used and re-enabled cancelability
so that the loop that checked for it would never terminate.
This reduces the clustering problem (primary clustering) but is not
completely free of it (secondary clustering) it still reduces the maximum
and average probing lengths.
With the previous approach we'd require at least an additional pointer
per item to store them in a list (15-18% increase in the overhead per
item). Instead we switch from handling collisions with overflow lists to
an open addressing scheme and store the actual table as variable-sized
indices pointing into an array of all inserted items in their original
order.
This can reduce the memory overhead even compared to the previous
implementation (especially for smaller tables), but because the array for
items is preallocated whenever the table is resized, it can be worse for
certain numbers of items. However, avoiding all the allocations required
by the previous design is actually a big advantage.
Depending on the usage pattern, the performance can improve quite a bit (in
particular when inserting many items). The raw lookup performance is a bit
slower as probing lengths increase with open addressing, but there are some
caching benefits due to the compact storage. So for general usage the
performance should be better. For instance, one test I did was counting the
occurrences of words in a list of 1'000'000 randomly selected words from a
dictionary of ~58'000 words (i.e. using a counter stored under each word as
key). The new implementation was ~8% faster on average while requiring
10% less memory.
Since we can't remove items from the array (would change the indices of all
items that follow it) we just mark them as removed and remove them once the
hash table is resized/rehashed (the cells in the hash table for these may
be reused). Due to this the latter may also happen if the number of stored
items does not increase e.g. after a series of remove/put operations (each
insertion requires storage in the array, no matter if items were removed).
So if the capacity is exhausted, the table is resized/rehashed (after lots
of removals the size may even be reduced) and all items marked as removed
are simply skipped.
Compared to the previous implementation the load factor/capacity is
lowered to reduce chances of collisions and to avoid primary clustering to
some degree. However, the latter in particular, but the open addressing
scheme in general, make this implementation completely unsuited for the
get_match() functionality (purposefully hashing to the same value and,
therefore, increasing the probing length and clustering). And keeping the
keys optionally sorted would complicate the code significantly. So we just
keep the existing hashlist_t implementation without adding code to maintain
the overall insertion order (we could add that feature optionally later, but
with the mentioned overhead for one or two pointers).
The maximum size is currently not changed. With the new implementation
this translates to a hard limit for the maximum number of items that can be
held in the table (=CAPACITY(MAX_SIZE)). Since this equals 715'827'882
items with the current settings, this shouldn't be a problem in practice,
the table alone would require 20 GiB in memory for that many items. The
hashlist_t implementation doesn't have that limitation due to the overflow
lists (it can store beyond it's capacity) but it itself would require over
29 GiB of memory to hold that many items.
The main intention here is that we can change the hashtable_t
implementation without being impeded by the special requirements imposed
by get_match() and sorting the keys/items in buckets.
This can improve negative lookups, but is mostly intended to be used
with get_match() so keys/items can be matched/enumerated in a specific
order. It's like storing sorted linked lists under a shared key but
with less memory overhead.
This allows measuring the delay between events more accurately if a
device is often suspended.
While CLOCK_BOOTTIME would be preferable, Android's bionic C library
does not support it for condvars.
On some systems it might be preferable to use e.g. CLOCK_BOOTTIME
instead of CLOCK_MONOTONIC, which is also not affected by time
adjustments but includes times when the system was suspended.
If cards/libraries don't support signature mechanisms with hashing, we fall
back to do it ourselves in software and pass the PKCS#1 digestInfo ASN.1
structure to sign via CKM_RSA_PKCS mechanism.
Closesstrongswan/strongswan#168.
OpenSSL currently doesn't support squeezing bytes out of an XOF multiple
times. Unfortunately, EVP_DigestFinalXOF() completely resets the context
and later calls not simply fail, they cause a null-pointer dereference in
libcrypto. This fixes the crash at the cost of repeating initializing
the whole state and allocating too much data for subsequent calls.
There is an open issue and PR that might add a function that allows
squeezing more data from an XOF in a future version of OpenSSL.
strtol(3) accepts values in the range of [LONG_MIN;LONG_MAX]. Based
on the architecture (32 or 64 bits), these values expand to either
0x8000000000000000/0x7fffffffffffffff for 64-bit builds, or
0x80000000/0x7fffffff for 32-bit builds.
The behavior when retrieving non-default values for charon.spi_min or
charon.spi_max, for example, depends on the architecture of the target
platform. While 0xC000001/0xCFFFFFFE work fine on a 64-bit build, on a
32-bit build, due to the use of strtol(3), an ERANGE causes get_int()
to return the default values.
By using strtoul(3) the default is only returned if the input value
exceeds 32 or 64 bits, based on the platform. Negative values are still
parsed correctly.
Signed-off-by: Thomas Egerer <[email protected]>
Doing this for the self-signed check also (i.e. if this and issuer are
the same) is particularly useful if the issuer uses a different key type.
Otherwise, we'd try to verify the signature with an incompatible key
that would result in a log message.
Fixes#3357.
There are a ton of libsoup/GLib-related "leaks" that we can't whitelist
and with leak detective active there is a delay that interestingly doesn't
happen with soup_session_sync_new(), so tests failed with a timeout (actually
they hung due to the lock in the fetcher manager).
On Travis, the curl plugin is used for the tests, so that's not an issue
there (and without LD the tests complete quickly and successfully).
As noted in 8ea13bbc5c newer compilers might optimize out the
assignment leading to invalid values in the keyUsage extension (as the
length was still set, the extension was encoded, just not with the
intended values).
Fixes#3249.
While it's unlikely that so many (large) items are allocated, this is
technically more correct. The result previously could overflow an
unsigned int (the conversion to size_t happened afterwards).
Tobias Brunner