[m-rev.] for review: Clarify some code relating to redzones.

[Julien Fischer]

Hi Zoltan,

On Sat, 31 Oct 2026 at 20:47, Zoltan Somogyi <zoltan.somogyi at runbox.com> wrote:
>
> On Sat, 31 Oct 2026 08:56:13 +1100 (AEDT), "Zoltan Somogyi" <zoltan.somogyi at runbox.com> wrote:
> > For me, this just replaces one pair of confusing names with another.
> > The name MR_USE_PAGE_PROTECTION is not all that useful
> > because it does say *what purpose* we should use page protection for.
> > And MR_USE_REDZONES is not clear because it assumes that the
> > reader knows what redzones ARE, and earlier we established that
> > there are two possible meanings: the last page of a zone that is never
> > unprotected (provided the platform can protect pages), or a larger
> > set of pages, some of which can become unprotected.
>
> Upon further reflection, I propose the following definition,
> which I think you are already using, but is not yet explicitly written down,
> and is not (yet) reflected in structure field names and variable names.
>
> - The redzone is the area of memory at the end of a memory zone
>   (det/nondet stack, heap, etc) that we ever explicitly deny access to,
>   via mprotect or similar functionality. Both it, and its components
>   the hard and soft redzones, must have sizes that are exact multiples
>   of the page size, and their address must be page aligned,
>   which must be ensured by us, because we cannot rely Boehm doing it.
>
> - The hard redzone is the last page of the redzone, which we never
>   unprotect.

Right; we also refer to this using the term "hardzone" in a few spots.

> - The soft redzone is the rest of the redzone. It starts out with no access
>   being permitted (or as close to it as mprotect/equivalents can get),
>   but we undo this protection on access.
>
> - The purpose of the hard redzone is definitely to guarantee that we catch
>   about-to-happen overflows.

This suggests that for some uses of memory zones we don't need hard redzones.
(The zones used for stack and trail segments are the obvious examples.)

> This requires the hard rezone to be at least as big
>  as the largest stack frame.
>
> - The possible purposes of the soft redzone are: (a) keeping track of how much
>   of a memory zone is actually used, both for plain curiosity and to inform
>   decisions about how big we should make memory zones, and (b) to help
>   Boehm gc do its job faster, by ... (fill in the blank, because I don't know
>   the mechanism by which this would work).
>   I don't think (a) is relevant anymore, because we can get the same info
>   in a more useful manner using stack segments. I have no idea whether
>   (b) is still relevant. If it is not, then we can stop using soft redzones,
>   though keeping its code around wouldn't hurt, *provided* that the code
>   is *documented* as being unused.

I don't think (b) is a thing.  MR_default_handler() is the only function that
handles access in redzones.  The only interaction it has with a garbage
collector is in low-level agc grades, where it calls MR_schedule_agc().
(Presumably, that only really matters when the zone in question is a heap
zone.)

> Do you guys agree?

In principle, sure. I would also offer the following observation:  using names
like  "redzone" and "hardzone" inside something called a "memory zone" is
fairly confusing.

Julien.