[m-rev.] Parallel runtime thread pinning.

Peter Wang novalazy at gmail.com
Mon Aug 24 09:33:27 AEST 2009


On 2009-08-24, Paul Bone <pbone at csse.unimelb.edu.au> wrote:
> 
> For post-commit review.
> 
> A later changeset will address Julien's review comments on my other
> recent changes to the parallel runtime.
> 
> 
> Parallel runtime thread pinning.
> 
> This change introduces two new features in the mercury runtime;
> pinning of threads to CPU cores/threads and runtime detection of the number of
> CPU cores/threads available.
> 
> If MR_num_threads has not been specified in the runtime options with the -P
> flag we use the sysconf(_SC_NPROCESSORS_ONLN) call if available to detect the
> number of CPUs online and set MR_num_threads available.  As before this
> defaults to 1.

Excellent.

> Index: configure.in
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/configure.in,v
> retrieving revision 1.541
> diff -u -p -b -r1.541 configure.in
> --- configure.in	18 Aug 2009 05:10:39 -0000	1.541
> +++ configure.in	19 Aug 2009 06:43:37 -0000

> @@ -1138,6 +1139,24 @@ MERCURY_CHECK_FOR_FENV_FUNC([fesetround]
>  
>  #-----------------------------------------------------------------------------#
>  #
> +# Check for declarations. 
> +#
> +
> +mercury_check_for_declarations () {
> +    for mercury_cv_decl in "$@"
> +    do
> +        mercury_cv_decl_define="MR_HAVE_`echo $mercury_cv_decl | \
> +            tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ`"
> +        AC_CHECK_DECL($mercury_cv_decl, [
> +            AC_DEFINE_UNQUOTED($mercury_cv_decl_define)
> +        ])
> +    done
> +}
> +
> +mercury_check_for_declarations "_SC_NPROCESSORS_ONLN"

Can't you just use #ifdef _SC_NPROCESSORS_ONLN in the source code?

> Index: doc/user_guide.texi
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/doc/user_guide.texi,v
> retrieving revision 1.590
> diff -u -p -b -r1.590 user_guide.texi
> --- doc/user_guide.texi	19 Aug 2009 07:45:11 -0000	1.590
> +++ doc/user_guide.texi	23 Aug 2009 22:27:28 -0000
> @@ -9587,7 +9587,10 @@ This only has an effect if the executabl
>  @item -P @var{num}
>  @findex -P (runtime option)
>  Tells the runtime system to use @var{num} threads
> -if the program was built in a parallel grade.
> +if the program was built in a parallel low-level C grade.
> +The mercury runtime attempts to automatically determine this value if support

Mercury

> +is available from the operating system.
> +If it cannot or support is unavailable it defaults to @samp{1}.
>  
>  @sp 1
>  @item --max-contexts-per-thread @var{num}
> @@ -9608,6 +9611,17 @@ grade.
>  @c and the @samp{MR_PROFILE_PARALLEL_EXECUTION_SUPPORT} C macro was
>  @c defined when the runtime system was compiled.
>  
> + at sp 1
> + at item --thread-pinning
> + at findex --thread-pinning
> +Request that the runtime system attempts to pin mercury engines (POSIX threads)
> +to CPU cores/hardware threads.

Mercury

> +This only has an effect if the executable was build in a parallel low-level C 
> +grade.

built

> +It is disabled by default unless @samp{-P @var{num}} is not specified and the
> +runtime system is able to detect the number of processors enabled by the
> +operating system.

I think it should just be disabled unless explicitly enabled.  You can't assume
that the machine is dedicated to your process.

> Index: runtime/mercury_conf.h.in
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_conf.h.in,v
> retrieving revision 1.64
> diff -u -p -b -r1.64 mercury_conf.h.in
> --- runtime/mercury_conf.h.in	30 Jul 2009 04:02:56 -0000	1.64
> +++ runtime/mercury_conf.h.in	19 Aug 2009 07:13:41 -0000
> @@ -136,6 +136,7 @@
>  **	MR_HAVE_FENV_H		we have <fenv.h>
>  **	MR_HAVE_SYS_MMAN_H	we have <sys/mman.h>
>  **	MR_HAVE_SYS_SEM_H 	we have <sys/sem.h>
> +**	MR_HAVE_SCHED_H     we have <sched.h>

Indentation.

> @@ -358,6 +362,14 @@
>  #undef	MR_HAVE_SIGCONTEXT_STRUCT_2ARG
>  
>  /*
> +** These specify weather the given C macros are defined.

whether

> +**
> +** MR_HAVE__SC_NPROCESSORS_ONLN, This is defined as a parameter for sysconf to
> +** determine the number of processors online.
> +*/
> +#undef	MR_HAVE__SC_NPROCESSORS_ONLN
> +
> +/*
>  ** For debugging purposes, if we get a fatal signal, we print out the
>  ** program counter (PC) at which the signal occurred.
>  **

> Index: runtime/mercury_context.c
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_context.c,v
> retrieving revision 1.67
> diff -u -p -b -r1.67 mercury_context.c
> --- runtime/mercury_context.c	17 Aug 2009 08:12:17 -0000	1.67
> +++ runtime/mercury_context.c	20 Aug 2009 02:31:36 -0000
> @@ -31,6 +31,10 @@ ENDINIT
>    #include <math.h> /* for sqrt and pow */
>  #endif
>  
> +#ifdef MR_HAVE_SCHED_H 
> +#include <sched.h>
> +#endif
> +
>  #include "mercury_memory_handlers.h"
>  #include "mercury_context.h"
>  #include "mercury_engine.h"             /* for `MR_memdebug' */
> @@ -62,6 +66,9 @@ MR_Context              *MR_runqueue_tai
>  #ifdef  MR_LL_PARALLEL_CONJ
>    MR_SparkDeque         MR_spark_queue;
>    MercuryLock           MR_sync_term_lock;
> +  MR_bool               MR_thread_pinning = MR_FALSE;
> +  static MercuryLock    MR_next_cpu_lock;
> +  static MR_Unsigned    MR_next_cpu = 0;
>  #endif
>  
>  MR_PendingContext       *MR_pending_contexts;
> @@ -134,6 +141,7 @@ MR_init_thread_stuff(void)
>    #ifdef MR_LL_PARALLEL_CONJ
>      MR_init_wsdeque(&MR_spark_queue, MR_INITIAL_GLOBAL_SPARK_QUEUE_SIZE);
>      pthread_mutex_init(&MR_sync_term_lock, MR_MUTEX_ATTR);
> +    pthread_mutex_init(&MR_next_cpu_lock, MR_MUTEX_ATTR);
>    #ifdef MR_DEBUG_RUNTIME_GRANULARITY_CONTROL
>      pthread_mutex_init(&MR_par_cond_stats_lock, MR_MUTEX_ATTR);
>    #endif
> @@ -155,6 +163,66 @@ MR_init_thread_stuff(void)
>      pthread_cond_init(&MR_thread_barrier_cond, MR_COND_ATTR);
>    #endif
>  
> +    /* 
> +     * Configure MR_num_threads if unset to match number of processors on the
> +     * system, If we do this then we prepare to set processor affinities later
> +     * on.
> +     */
> +    if (MR_num_threads == 0)
> +    {
> +#if defined(MR_HAVE_SYSCONF) && defined(MR_HAVE__SC_NPROCESSORS_ONLN) 
> +        long result;
> +
> +        result = sysconf(_SC_NPROCESSORS_ONLN);
> +        if (result < 1) {
> +            /* We couldn't determine the number of processors. */
> +            MR_num_threads = 1;
> +        } else {
> +            MR_num_threads = result;
> +            /* On systems that don't support sched_setaffinity we don't try to
> +            ** automatically enable thread pinning.  This prevents a runtime
> +            ** warning that could unnecessarily confuse the user.
> +            **/
> +#if defined(MR_LL_PARALLEL_CONJ) && defined(MR_HAVE_SCHED_SETAFFINITY) 
> +            MR_thread_pinning = MR_TRUE;
> +#endif
> +        }
> +#else
> +        MR_num_threads = 1;
> +#endif
> +    }
> +#endif
> +}
> +

> +void
> +MR_pin_thread(void)
> +{
> +#if defined(MR_THREAD_SAFE) && defined(MR_LL_PARALLEL_CONJ) && \
> +        defined(MR_HAVE_SCHED_SETAFFINITY) 
> +    MR_LOCK(&MR_next_cpu_lock, "MR_pin_thread");
> +    if (MR_thread_pinning == MR_TRUE) {

You should just write `if (var)' or `if (!var)' when testing booleans.

> +        cpu_set_t   cpus;
> +
> +        if (MR_next_cpu < CPU_SETSIZE) {
> +            CPU_ZERO(&cpus);
> +            CPU_SET(MR_next_cpu, &cpus);
> +            if (sched_setaffinity(0, sizeof(cpu_set_t), &cpus) == 0)
> +            {
> +                MR_next_cpu++;
> +            } else {
> +                perror("Warning: Couldn't set CPU affinity");
> +                /* if this failed once it will probably fail again so disable
> +                ** it. 
> +                */

Formatting.

> +                MR_thread_pinning = MR_FALSE;
> +            }
> +        } else {
> +            perror("Warning: Couldn't set CPU affinity due to a static " \
> +                "system limit");

Don't need the backslash.

> Index: runtime/mercury_context.h
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_context.h,v
> retrieving revision 1.52
> diff -u -p -b -r1.52 mercury_context.h
> --- runtime/mercury_context.h	16 Aug 2009 10:47:55 -0000	1.52
> +++ runtime/mercury_context.h	20 Aug 2009 02:31:43 -0000
> @@ -344,6 +344,7 @@ extern      MR_Context  *MR_runqueue_tai
>  #ifdef  MR_LL_PARALLEL_CONJ
>    extern    MR_SparkDeque   MR_spark_queue;
>    extern    MercuryLock     MR_sync_term_lock;
> +  extern    MR_bool         MR_thread_pinning;
>  #endif
>  
>  #if defined(MR_THREAD_SAFE) && defined(MR_PROFILE_PARALLEL_EXECUTION_SUPPORT) 
> @@ -458,13 +459,20 @@ extern  MR_Context  *MR_create_context(c
>  extern  void        MR_destroy_context(MR_Context *context);
>  
>  /*
> -** MR_init_thread_stuff() initializes the lock structures for the runqueue.
> +** MR_init_thread_stuff() initializes the lock structures for the runqueue, 
> +** and detect the number of threads to use on the LLC backend.
>  */
>  extern  void        MR_init_thread_stuff(void);

detects

> Index: runtime/mercury_wrapper.c
> ===================================================================
> RCS file: /home/mercury1/repository/mercury/runtime/mercury_wrapper.c,v
> retrieving revision 1.198
> diff -u -p -b -r1.198 mercury_wrapper.c
> --- runtime/mercury_wrapper.c	16 Aug 2009 10:18:36 -0000	1.198
> +++ runtime/mercury_wrapper.c	19 Aug 2009 07:11:18 -0000
> @@ -289,7 +289,13 @@ static  char        *MR_mem_usage_report
>  
>  static  int         MR_num_output_args = 0;
>  
> -MR_Unsigned         MR_num_threads = 1;
> +/*
> +** This is initialized to zero, if it is still zero after configuration of the

full stop

> +** runtime but before threads are started then the number of processors on the
> +** system is detected and used if support is available.  Otherwise we fall back
> +** to 1 

full stop

> +*/
> +MR_Unsigned         MR_num_threads = 0;
>  
>  static  MR_bool     MR_print_table_statistics = MR_FALSE;
>  

> @@ -1718,6 +1728,11 @@ MR_process_options(int argc, char **argv
>                  MR_max_contexts_per_thread = size;
>                  break;
>  
> +            case MR_THREAD_PINNING:
> +#if defined(MR_THREAD_SAFE) && defined(MR_LL_PARALLEL_CONJ)
> +                MR_thread_pinning = MR_TRUE;
> +#endif
> +

break;

Peter
--------------------------------------------------------------------------
mercury-reviews mailing list
Post messages to:       mercury-reviews at csse.unimelb.edu.au
Administrative Queries: owner-mercury-reviews at csse.unimelb.edu.au
Subscriptions:          mercury-reviews-request at csse.unimelb.edu.au
--------------------------------------------------------------------------



More information about the reviews mailing list