[m-rev.] diff: disable string_format_o on taifun

[Fergus Henderson]

On 21-Nov-2002, Peter Ross <pro at missioncriticalit.com> wrote:
> fjh wrote:
> > On 22-Nov-2002, Peter Ross <pro at missioncriticalit.com> wrote:
> > > Estimated hours taken: 0.5
> > > Branches: main
> > >
> > > tests/general/string_format/Mmakefile:
> > >     Disable string_format_o on taifun as the C library sprintf is buggy.
> > ...
> > > +ifneq "$(findstring taifun,$(HOST))" ""
> >
> > Pete, you seem to very often attack the symptom, not the cause.
> >
> > The symptom is that the test case is failing on taifun.
> > The cause is a bug in the Solaris 2.8 C library.
> > The right fix is to disable this test case for Solaris 2.8
> > (or maybe for all Solaris), not just for taifun.
>
> I thought about that.  My reasoning is that a user of our test-suite would
> rather see that there is a problem with printing floats using the type
> specifier o, so we should only not test this case when testing on our
> machines.  The best way to do this was to check for the host name that we
> test on.

Hmm.  At any rate your patch isn't complete -- we're also running tests
on mundroo, which is another Solaris system that has the same problem.

But more generally, I think it would be better to only report unexpected
failures.  Firstly, that would be more consistent with the approach that
we have taken with other test cases up to this point.  Secondly,
doing otherwise makes it harder to start testing on a new host,
since you need to manually disable the expected failures for that new host.
Also it makes it harder for users who have installed on a new machine
to distinguish between things which they have screwed up, and things
which are not their fault.  I would like people to be able to run the
test suite from a release version on a new machine, and report any
failures as bugs.

Currently, if users of the test suite want to know all the failures,
including expected failures, then they need to grep for "XXX"
in all the Mmakefiles in the tests directory and its subdirectories.
Changing some but not all of the test cases to use hostname-based
checks would not remove this need.

In the long run it would be nicest to have test cases which are expected
to fail put on a separate list, and have the names of such test cases
be output by the test framework, e.g. as "EXPECTED FAIL for test <blah>
in grade <blah>".

-- 
Fergus Henderson <fjh at cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------