[m-rev.] for review: Wait for work-stealing engine threads to terminate with pthread_join.
Peter Wang
novalazy at gmail.com
Mon Apr 13 17:24:27 AEST 2026
Previously, we created _detached_ threads to run work-stealing engines.
The only reason for using detached threads instead of joinable threads
was because the code for thread creation was originally designed for
creating Mercury threads (the interface exported by the thread.m module
expects detached threads).
When the program is about to end, the main thread notifies the engines
to shut down, then waits on a semaphore that is incremented when an
engine is shut down. But an engine can only increment the semaphore
BEFORE its thread terminates. That is, while the semaphore indicates
that the engine has shut down (no longer responding), the thread that
the engine was running on may continue for an indetermine amount of time
before it is terminated. The main thread may think that it is safe to
proceed, even while some of the engine threads are still running.
I found that that on a Linux/glibc system, with a statically linked
binary, this setup could sometimes cause an "Aborted" error message at
program exits (after Mercury main/2). From backtraces, I believe the
problem is as described: the main thread is already in a exit() call
while engine threads are still performing their own cleanup, leading to
an abort() call.
The solution is to do what we should have done to begin with: run
work-stealing engines in non-detached threads, and call pthread_join()
to wait for engine threads to terminate before allowing the main thread
to continue with program termination.
runtime/mercury_context.c:
Delete references to shutdown_ws_semaphore.
runtime/mercury_thread.c:
runtime/mercury_thread.h:
Make MR_create_worksteal_thread create a non-detached thread.
runtime/mercury_wrapper.c:
In mercury_runtime_init, record the IDs of the threads created for
running work-stealing engines in an array.
In mercury_runtime_terminate, after notifying each work-stealing
engine to shut down, wait for the engine threads to terminate
by calling pthread_join().
Sample backtrace:
Thread 1 (Thread 0x7f6dcafb46c0 (LWP 11122) (Exiting)):
#0 0x000000000093c40c in pthread_kill ()
#1 0x00000000009219ce in raise ()
#2 0x00000000004013b2 in abort ()
#3 0x0000000000401bd4 in uw_init_context_1[cold] ()
#4 0x00000000009cebda in _Unwind_ForcedUnwind ()
#5 0x0000000000940044 in __pthread_unwind ()
#6 0x000000000093b9e0 in pthread_exit ()
#7 0x00000000009022ad in GC_pthread_exit ()
#8 0x00000000008bd062 in action_shutdown_ws_engine ()
#9 0x00000000008be0ea in scheduler_module_idle_sleep ()
#10 0x00000000008ca58d in MR_call_engine ()
#11 0x00000000008e5d05 in MR_init_thread_inner ()
#12 0x00000000008e5e8c in MR_create_worksteal_thread_2 ()
#13 0x00000000009048d9 in GC_inner_start_routine ()
#14 0x00000000008f46fe in GC_call_with_stack_base ()
#15 0x000000000093a85c in start_thread ()
#16 0x000000000096875c in clone3 ()
Thread 2 (Thread 0x1c533c0 (LWP 11092)):
#0 0x000000000096656d in write ()
#1 0x0000000000933f35 in _IO_new_file_write ()
#2 0x00000000009321c1 in _IO_new_do_write ()
#3 0x0000000000936906 in _IO_flush_all ()
#4 0x0000000000936f2d in _IO_cleanup ()
#5 0x000000000092236e in __run_exit_handlers ()
#6 0x00000000009223be in exit ()
#7 0x0000000000917d0f in __libc_start_call_main ()
#8 0x0000000000919ed0 in __libc_start_main_impl ()
#9 0x0000000000401d85 in _start ()
Thread 3 (Thread 0x7f6dd97d16c0 (LWP 11093) (Exiting)):
#0 0x0000000000941d19 in alloc_new_heap ()
#1 0x00000000009421f2 in arena_get2.part ()
#2 0x0000000000943fb9 in tcache_init.part ()
#3 0x00000000009448c4 in malloc ()
#4 0x00000000009d2141 in _Unwind_Find_FDE ()
#5 0x00000000009cda7d in uw_frame_state_for ()
#6 0x00000000009ce0d3 in uw_init_context_1 ()
#7 0x00000000009cebda in _Unwind_ForcedUnwind ()
#8 0x0000000000940044 in __pthread_unwind ()
#9 0x000000000093b9e0 in pthread_exit ()
#10 0x00000000009022ad in GC_pthread_exit ()
#11 0x00000000008bd062 in action_shutdown_ws_engine ()
#12 0x00000000008be0ea in scheduler_module_idle_sleep ()
#13 0x00000000008ca58d in MR_call_engine ()
#14 0x00000000008e5d05 in MR_init_thread_inner ()
#15 0x00000000008e5e8c in MR_create_worksteal_thread_2 ()
#16 0x00000000009048d9 in GC_inner_start_routine ()
#17 0x00000000008f46fe in GC_call_with_stack_base ()
#18 0x000000000093a85c in start_thread ()
#19 0x000000000096875c in clone3 ()
---
runtime/mercury_context.c | 16 +---------------
runtime/mercury_thread.c | 17 ++++-------------
runtime/mercury_thread.h | 4 ++--
runtime/mercury_wrapper.c | 24 +++++++++++++++++++++---
4 files changed, 28 insertions(+), 33 deletions(-)
diff --git a/runtime/mercury_context.c b/runtime/mercury_context.c
index 2e9694486..e432ba839 100644
--- a/runtime/mercury_context.c
+++ b/runtime/mercury_context.c
@@ -1,7 +1,7 @@
// vim: ts=4 sw=4 expandtab ft=c
// Copyright (C) 1995-2007, 2009-2011 The University of Melbourne.
-// Copyright (C) 2014, 2016-2018, 2020-2021, 2024 The Mercury team.
+// Copyright (C) 2014, 2016-2018, 2020-2021, 2024, 2026 The Mercury team.
// This file is distributed under the terms specified in COPYING.LIB.
// mercury_context.c - handles multithreading stuff.
@@ -271,7 +271,6 @@ static MR_Context *free_small_context_list = NULL;
#ifdef MR_LL_PARALLEL_CONJ
MR_Integer volatile MR_num_idle_ws_engines = 0;
static MR_Integer volatile MR_num_outstanding_contexts = 0;
-static MercurySem shutdown_ws_semaphore;
static MercuryLock MR_par_cond_stats_lock;
@@ -362,7 +361,6 @@ MR_init_context_stuff(void)
#ifdef MR_DEBUG_RUNTIME_GRANULARITY_CONTROL
pthread_mutex_init(&MR_par_cond_stats_lock, MR_MUTEX_ATTR);
#endif
- MR_sem_init(&shutdown_ws_semaphore, 0);
#endif
pthread_mutex_init(&MR_STM_lock, MR_MUTEX_ATTR);
@@ -880,9 +878,6 @@ MR_finalize_context_stuff(void)
#ifdef MR_THREAD_SAFE
pthread_mutex_destroy(&MR_runqueue_lock);
pthread_mutex_destroy(&free_context_list_lock);
- #ifdef MR_LL_PARALLEL_CONJ
- MR_sem_destroy(&shutdown_ws_semaphore);
- #endif
#endif
#ifdef MR_PROFILE_PARALLEL_EXECUTION_SUPPORT
@@ -2133,14 +2128,6 @@ MR_shutdown_ws_engines(void)
MR_sched_yield();
}
}
-
- for (i = 0; i < (MR_num_ws_engines - 1); i++) {
- int err;
-
- do {
- err = MR_SEM_WAIT(&shutdown_ws_semaphore, "MR_shutdown_ws_engines");
- } while (err == -1 && MR_SEM_IS_EINTR(errno));
- }
}
#endif // MR_LL_PARALLEL_CONJ
@@ -2571,7 +2558,6 @@ action_shutdown_ws_engine(void)
assert(engine_id != 0);
assert(MR_ENGINE(MR_eng_type) == MR_ENGINE_TYPE_SHARED);
MR_finalize_thread_engine();
- MR_SEM_POST(&shutdown_ws_semaphore, "MR_do_sleep shutdown_sem");
pthread_exit(0);
}
diff --git a/runtime/mercury_thread.c b/runtime/mercury_thread.c
index ac4cfbebe..f333263cd 100644
--- a/runtime/mercury_thread.c
+++ b/runtime/mercury_thread.c
@@ -1,7 +1,7 @@
// vim: ts=4 sw=4 expandtab ft=c
// Copyright (C) 1997-2001, 2003, 2005-2007, 2009-2011 The University of Melbourne.
-// Copyright (C) 2014, 2016, 2018 The Mercury team.
+// Copyright (C) 2014, 2016, 2018, 2026 The Mercury team.
// This file is distributed under the terms specified in COPYING.LIB.
#include "mercury_imp.h"
@@ -60,25 +60,16 @@ MR_shutdown_engine_for_threads(MercuryEngine *eng);
static void *
MR_create_worksteal_thread_2(void *goal);
-MercuryThread *
+MercuryThread
MR_create_worksteal_thread(void)
{
- MercuryThread *thread;
- pthread_attr_t attrs;
+ MercuryThread thread;
int err;
char errbuf[MR_STRERROR_BUF_SIZE];
assert(!MR_thread_equal(MR_primordial_thread, MR_null_thread()));
- // Create threads in the detached state so that resources will be
- // automatically freed when threads terminate (we don't call
- // pthread_join() anywhere).
-
- thread = MR_GC_NEW_ATTRIB(MercuryThread, MR_ALLOC_SITE_RUNTIME);
- pthread_attr_init(&attrs);
- pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED);
- err = pthread_create(thread, &attrs, MR_create_worksteal_thread_2, NULL);
- pthread_attr_destroy(&attrs);
+ err = pthread_create(&thread, NULL, MR_create_worksteal_thread_2, NULL);
#if 0
fprintf(stderr, "pthread_create returned %d (errno = %d)\n", err, errno);
diff --git a/runtime/mercury_thread.h b/runtime/mercury_thread.h
index 3481b32d0..28180deb0 100644
--- a/runtime/mercury_thread.h
+++ b/runtime/mercury_thread.h
@@ -1,7 +1,7 @@
// vim: ts=4 sw=4 expandtab ft=c
// Copyright (C) 1997-1998, 2000, 2003, 2005-2007, 2009-2011 The University of Melbourne.
-// Copyright (C) 2014-2018 The Mercury team.
+// Copyright (C) 2014-2018, 2026 The Mercury team.
// This file is distributed under the terms specified in COPYING.LIB.
#ifndef MERCURY_THREAD_H
@@ -208,7 +208,7 @@
// create_worksteal_thread() creates a new POSIX thread, and creates and
// initializes a work-stealing Mercury engine to run in that thread.
- extern MercuryThread *MR_create_worksteal_thread(void);
+ extern MercuryThread MR_create_worksteal_thread(void);
// The primordial thread. Currently used for debugging.
diff --git a/runtime/mercury_wrapper.c b/runtime/mercury_wrapper.c
index dfab7c26f..b4ddf3a91 100644
--- a/runtime/mercury_wrapper.c
+++ b/runtime/mercury_wrapper.c
@@ -299,6 +299,8 @@ static int MR_num_output_args = 0;
#ifdef MR_LL_PARALLEL_CONJ
MR_Unsigned MR_num_ws_engines = 0;
+static MercuryThread *MR_ws_engine_threads; // Will be initialised to an array
+ // of length MR_num_ws_engines - 1.
MR_Unsigned MR_max_engines = 1024;
MR_Unsigned MR_granularity_wsdeque_length_factor = 8;
@@ -639,12 +641,16 @@ mercury_runtime_init(int argc, char **argv)
MR_create_thread_local_mutables(MR_MAX_THREAD_LOCAL_MUTABLES));
// Start up additional work-stealing Mercury engines.
+ // The current thread will be running the first engine.
#ifdef MR_LL_PARALLEL_CONJ
{
int i;
- for (i = 1; i < MR_num_ws_engines; i++) {
- MR_create_worksteal_thread();
+ MR_ws_engine_threads = MR_GC_NEW_ARRAY(MercuryThread,
+ MR_num_ws_engines - 1);
+
+ for (i = 0; i < MR_num_ws_engines - 1; i++) {
+ MR_ws_engine_threads[i] = MR_create_worksteal_thread();
}
#ifdef MR_THREADSCOPE
@@ -655,7 +661,7 @@ mercury_runtime_init(int argc, char **argv)
}
#endif
- while (MR_num_idle_ws_engines < MR_num_ws_engines-1) {
+ while (MR_num_idle_ws_engines < MR_num_ws_engines - 1) {
// busy wait until the worker threads are ready
MR_ATOMIC_PAUSE;
}
@@ -3000,6 +3006,18 @@ mercury_runtime_terminate(void)
#if !defined(MR_HIGHLEVEL_CODE) && defined(MR_THREAD_SAFE)
MR_shutdown_ws_engines();
+ // Each work-stealing engine has been notified to shut down.
+ // Wait for the threads that they are running on to terminate before
+ // continuing.
+ {
+ int i;
+
+ for (i = 0; i < MR_num_ws_engines - 1; i++) {
+ pthread_join(MR_ws_engine_threads[i], NULL);
+ MR_ws_engine_threads[i] = MR_null_thread();
+ }
+ }
+
#ifdef MR_THREADSCOPE
if (MR_ENGINE(MR_eng_ts_buffer)) {
MR_threadscope_finalize_engine(MR_thread_engine_base);
--
2.51.0
More information about the reviews
mailing list