[m-rev.] Parallel runtime thread pinning.
Peter Wang
novalazy at gmail.com
Mon Aug 24 09:33:27 AEST 2009
On 2009-08-24, Paul Bone <pbone at csse.unimelb.edu.au> wrote:
>
> For post-commit review.
>
> A later changeset will address Julien's review comments on my other
> recent changes to the parallel runtime.
>
>
> Parallel runtime thread pinning.
>
> This change introduces two new features in the mercury runtime;
> pinning of threads to CPU cores/threads and runtime detection of the number of
> CPU cores/threads available.
>
> If MR_num_threads has not been specified in the runtime options with the -P
> flag we use the sysconf(_SC_NPROCESSORS_ONLN) call if available to detect the
> number of CPUs online and set MR_num_threads available. As before this
> defaults to 1.
Excellent.
> Index: configure.in
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/configure.in,v
> retrieving revision 1.541
> diff -u -p -b -r1.541 configure.in
> --- configure.in 18 Aug 2009 05:10:39 -0000 1.541
> +++ configure.in 19 Aug 2009 06:43:37 -0000
> @@ -1138,6 +1139,24 @@ MERCURY_CHECK_FOR_FENV_FUNC([fesetround]
>
> #-----------------------------------------------------------------------------#
> #
> +# Check for declarations.
> +#
> +
> +mercury_check_for_declarations () {
> + for mercury_cv_decl in "$@"
> + do
> + mercury_cv_decl_define="MR_HAVE_`echo $mercury_cv_decl | \
> + tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ`"
> + AC_CHECK_DECL($mercury_cv_decl, [
> + AC_DEFINE_UNQUOTED($mercury_cv_decl_define)
> + ])
> + done
> +}
> +
> +mercury_check_for_declarations "_SC_NPROCESSORS_ONLN"
Can't you just use #ifdef _SC_NPROCESSORS_ONLN in the source code?
> Index: doc/user_guide.texi
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/doc/user_guide.texi,v
> retrieving revision 1.590
> diff -u -p -b -r1.590 user_guide.texi
> --- doc/user_guide.texi 19 Aug 2009 07:45:11 -0000 1.590
> +++ doc/user_guide.texi 23 Aug 2009 22:27:28 -0000
> @@ -9587,7 +9587,10 @@ This only has an effect if the executabl
> @item -P @var{num}
> @findex -P (runtime option)
> Tells the runtime system to use @var{num} threads
> -if the program was built in a parallel grade.
> +if the program was built in a parallel low-level C grade.
> +The mercury runtime attempts to automatically determine this value if support
Mercury
> +is available from the operating system.
> +If it cannot or support is unavailable it defaults to @samp{1}.
>
> @sp 1
> @item --max-contexts-per-thread @var{num}
> @@ -9608,6 +9611,17 @@ grade.
> @c and the @samp{MR_PROFILE_PARALLEL_EXECUTION_SUPPORT} C macro was
> @c defined when the runtime system was compiled.
>
> + at sp 1
> + at item --thread-pinning
> + at findex --thread-pinning
> +Request that the runtime system attempts to pin mercury engines (POSIX threads)
> +to CPU cores/hardware threads.
Mercury
> +This only has an effect if the executable was build in a parallel low-level C
> +grade.
built
> +It is disabled by default unless @samp{-P @var{num}} is not specified and the
> +runtime system is able to detect the number of processors enabled by the
> +operating system.
I think it should just be disabled unless explicitly enabled. You can't assume
that the machine is dedicated to your process.
> Index: runtime/mercury_conf.h.in
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_conf.h.in,v
> retrieving revision 1.64
> diff -u -p -b -r1.64 mercury_conf.h.in
> --- runtime/mercury_conf.h.in 30 Jul 2009 04:02:56 -0000 1.64
> +++ runtime/mercury_conf.h.in 19 Aug 2009 07:13:41 -0000
> @@ -136,6 +136,7 @@
> ** MR_HAVE_FENV_H we have <fenv.h>
> ** MR_HAVE_SYS_MMAN_H we have <sys/mman.h>
> ** MR_HAVE_SYS_SEM_H we have <sys/sem.h>
> +** MR_HAVE_SCHED_H we have <sched.h>
Indentation.
> @@ -358,6 +362,14 @@
> #undef MR_HAVE_SIGCONTEXT_STRUCT_2ARG
>
> /*
> +** These specify weather the given C macros are defined.
whether
> +**
> +** MR_HAVE__SC_NPROCESSORS_ONLN, This is defined as a parameter for sysconf to
> +** determine the number of processors online.
> +*/
> +#undef MR_HAVE__SC_NPROCESSORS_ONLN
> +
> +/*
> ** For debugging purposes, if we get a fatal signal, we print out the
> ** program counter (PC) at which the signal occurred.
> **
> Index: runtime/mercury_context.c
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_context.c,v
> retrieving revision 1.67
> diff -u -p -b -r1.67 mercury_context.c
> --- runtime/mercury_context.c 17 Aug 2009 08:12:17 -0000 1.67
> +++ runtime/mercury_context.c 20 Aug 2009 02:31:36 -0000
> @@ -31,6 +31,10 @@ ENDINIT
> #include <math.h> /* for sqrt and pow */
> #endif
>
> +#ifdef MR_HAVE_SCHED_H
> +#include <sched.h>
> +#endif
> +
> #include "mercury_memory_handlers.h"
> #include "mercury_context.h"
> #include "mercury_engine.h" /* for `MR_memdebug' */
> @@ -62,6 +66,9 @@ MR_Context *MR_runqueue_tai
> #ifdef MR_LL_PARALLEL_CONJ
> MR_SparkDeque MR_spark_queue;
> MercuryLock MR_sync_term_lock;
> + MR_bool MR_thread_pinning = MR_FALSE;
> + static MercuryLock MR_next_cpu_lock;
> + static MR_Unsigned MR_next_cpu = 0;
> #endif
>
> MR_PendingContext *MR_pending_contexts;
> @@ -134,6 +141,7 @@ MR_init_thread_stuff(void)
> #ifdef MR_LL_PARALLEL_CONJ
> MR_init_wsdeque(&MR_spark_queue, MR_INITIAL_GLOBAL_SPARK_QUEUE_SIZE);
> pthread_mutex_init(&MR_sync_term_lock, MR_MUTEX_ATTR);
> + pthread_mutex_init(&MR_next_cpu_lock, MR_MUTEX_ATTR);
> #ifdef MR_DEBUG_RUNTIME_GRANULARITY_CONTROL
> pthread_mutex_init(&MR_par_cond_stats_lock, MR_MUTEX_ATTR);
> #endif
> @@ -155,6 +163,66 @@ MR_init_thread_stuff(void)
> pthread_cond_init(&MR_thread_barrier_cond, MR_COND_ATTR);
> #endif
>
> + /*
> + * Configure MR_num_threads if unset to match number of processors on the
> + * system, If we do this then we prepare to set processor affinities later
> + * on.
> + */
> + if (MR_num_threads == 0)
> + {
> +#if defined(MR_HAVE_SYSCONF) && defined(MR_HAVE__SC_NPROCESSORS_ONLN)
> + long result;
> +
> + result = sysconf(_SC_NPROCESSORS_ONLN);
> + if (result < 1) {
> + /* We couldn't determine the number of processors. */
> + MR_num_threads = 1;
> + } else {
> + MR_num_threads = result;
> + /* On systems that don't support sched_setaffinity we don't try to
> + ** automatically enable thread pinning. This prevents a runtime
> + ** warning that could unnecessarily confuse the user.
> + **/
> +#if defined(MR_LL_PARALLEL_CONJ) && defined(MR_HAVE_SCHED_SETAFFINITY)
> + MR_thread_pinning = MR_TRUE;
> +#endif
> + }
> +#else
> + MR_num_threads = 1;
> +#endif
> + }
> +#endif
> +}
> +
> +void
> +MR_pin_thread(void)
> +{
> +#if defined(MR_THREAD_SAFE) && defined(MR_LL_PARALLEL_CONJ) && \
> + defined(MR_HAVE_SCHED_SETAFFINITY)
> + MR_LOCK(&MR_next_cpu_lock, "MR_pin_thread");
> + if (MR_thread_pinning == MR_TRUE) {
You should just write `if (var)' or `if (!var)' when testing booleans.
> + cpu_set_t cpus;
> +
> + if (MR_next_cpu < CPU_SETSIZE) {
> + CPU_ZERO(&cpus);
> + CPU_SET(MR_next_cpu, &cpus);
> + if (sched_setaffinity(0, sizeof(cpu_set_t), &cpus) == 0)
> + {
> + MR_next_cpu++;
> + } else {
> + perror("Warning: Couldn't set CPU affinity");
> + /* if this failed once it will probably fail again so disable
> + ** it.
> + */
Formatting.
> + MR_thread_pinning = MR_FALSE;
> + }
> + } else {
> + perror("Warning: Couldn't set CPU affinity due to a static " \
> + "system limit");
Don't need the backslash.
> Index: runtime/mercury_context.h
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_context.h,v
> retrieving revision 1.52
> diff -u -p -b -r1.52 mercury_context.h
> --- runtime/mercury_context.h 16 Aug 2009 10:47:55 -0000 1.52
> +++ runtime/mercury_context.h 20 Aug 2009 02:31:43 -0000
> @@ -344,6 +344,7 @@ extern MR_Context *MR_runqueue_tai
> #ifdef MR_LL_PARALLEL_CONJ
> extern MR_SparkDeque MR_spark_queue;
> extern MercuryLock MR_sync_term_lock;
> + extern MR_bool MR_thread_pinning;
> #endif
>
> #if defined(MR_THREAD_SAFE) && defined(MR_PROFILE_PARALLEL_EXECUTION_SUPPORT)
> @@ -458,13 +459,20 @@ extern MR_Context *MR_create_context(c
> extern void MR_destroy_context(MR_Context *context);
>
> /*
> -** MR_init_thread_stuff() initializes the lock structures for the runqueue.
> +** MR_init_thread_stuff() initializes the lock structures for the runqueue,
> +** and detect the number of threads to use on the LLC backend.
> */
> extern void MR_init_thread_stuff(void);
detects
> Index: runtime/mercury_wrapper.c
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_wrapper.c,v
> retrieving revision 1.198
> diff -u -p -b -r1.198 mercury_wrapper.c
> --- runtime/mercury_wrapper.c 16 Aug 2009 10:18:36 -0000 1.198
> +++ runtime/mercury_wrapper.c 19 Aug 2009 07:11:18 -0000
> @@ -289,7 +289,13 @@ static char *MR_mem_usage_report
>
> static int MR_num_output_args = 0;
>
> -MR_Unsigned MR_num_threads = 1;
> +/*
> +** This is initialized to zero, if it is still zero after configuration of the
full stop
> +** runtime but before threads are started then the number of processors on the
> +** system is detected and used if support is available. Otherwise we fall back
> +** to 1
full stop
> +*/
> +MR_Unsigned MR_num_threads = 0;
>
> static MR_bool MR_print_table_statistics = MR_FALSE;
>
> @@ -1718,6 +1728,11 @@ MR_process_options(int argc, char **argv
> MR_max_contexts_per_thread = size;
> break;
>
> + case MR_THREAD_PINNING:
> +#if defined(MR_THREAD_SAFE) && defined(MR_LL_PARALLEL_CONJ)
> + MR_thread_pinning = MR_TRUE;
> +#endif
> +
break;
Peter
--------------------------------------------------------------------------
mercury-reviews mailing list
Post messages to: mercury-reviews at csse.unimelb.edu.au
Administrative Queries: owner-mercury-reviews at csse.unimelb.edu.au
Subscriptions: mercury-reviews-request at csse.unimelb.edu.au
--------------------------------------------------------------------------
More information about the reviews
mailing list