[m-dev.] .NET back-end test results

[Fergus Henderson]

I've run the tests in the `il' grade and analyzed the results.
Here are the results of my analysis.

First, the number of distinct bug symptoms detected:

	Bugs in the test suite: 5
	Bugs in the .NET back-end: 8
	Unimplemented features: 7

So, at one patch per bug symptom or unimplemented feature,
we are just 20 patches away from passing all the tests ;-)
Of course, it may be the case that the same symptoms are being
generated by more than one bug, in which case more bug-fixes
will be needed.  Also, there may be some additional bugs which
show up once we've fixed all of those.

Second, the number of test case failures.
Note that each bug can in general cause many test case failures.

Passed test cases: 280
Failed test cases: 459 total
   Spurious failures: 291 subtotal
	    152 FAILED.VALID
	     80 FAILED.INVALID
	     59 FAILED.NO_CLAUSES
   Failures as a result of genuine bugs: 117 subtotal
	     65 FAILED.UNDEFINED_CLASS
	     23 FAILED.INVALID_CAST
	     29 FAILED.MISC
   Failures due to features that are not yet implemented,
   for which we output an appropriate "sorry" message: 51
	     51 FAILED.SORRY

Inclued below is a detailed analysis.

SPURIOUS FAILURES

S1
- test/valid/* report spurious failures due to Mmakefile making
  incorrect assumption that we generate `.o' files (FAILED.VALID)

S2
- test/invalid/* report spurious failures due to the Mmakefile not catering
  for --il implying --intermodule-optimization (FAILED.INVALID)

S3
- lots of tests fail due to the use of C foreign clauses (FAILED.NO_CLAUSES)
  tests/general/impure_foreign is a special case of this;
  it fails at run-time because the Mercury fall-back clauses just call error/1.
  tests/hard_coded/pragma_import and tests/hard_coded/pragma_export are
  similar.

S4
- tests/hard_coded/*.split (not supported for il grade)

S5
- tests/hard_coded/special_character fails, due to '\0321' in the
  source code getting converted to octal 245 in the output file.
  This is a bug in the test case; it assumes that the internal
  representation is the same as the file representation, which is
  not the case for .NET.

BUGS

B1
- tests/hard_coded/term_io_test fails:
  gets syntax errors at run-time, cause unknown.

B2
- tests/general/commutative fails:
  tests/hard_coded/agg fails:
  invalid cast
  Likewise for lots of others (FAILED.INVALID_CAST)

B3
- tests/general/interpreter fails, due to problem with equivalence types;
  the argument types for the unify & compare predicates for equivalence types
  are invalid (undefined)
  Likewise for lots of others (FAILED.UNDEFINED_CLASS)

B4
- tests/general/string_to_float fails,
  because string__to_float throws an exception rather than failing

B5
- tests/general/string_format_test fails, due to various formatting errors
  (e.g. no left justification, missing 0x prefixes, not truncating).
  Likewise for
	  tests/hard_coded/float_rounding_bug
	  tests/hard_coded/float_reg
	  tests/hard_coded/arbitrary_constraint_class
	  tests/hard_coded/arbitrary_constraint_pred_1
	  tests/hard_coded/arbitrary_constraint_pred_2
	  tests/hard_coded/module_test

B6
- tests/hard_coded/*foreign_type* fail, due to not finding `mercury.dll'.
  (XXX probably due to me not installing it properly)

B7
- tests/hard_coded/no_fully_strict fails,
  because uncaught exceptions do not result in a non-zero exit status
  Likewise for
  	tests/hard_coded/uncaught_exception

B8
- tests/hard_coded/no_warn_singleton fails,
  due to determinism warning for table_io_copy_io_state from table_builtin.m
  Likewise for
  tests/warnings/arg_order_rearrangement.

NOT YET IMPLEMENTED (FAILED.SORRY):

N1
- lots of tests fail due to
  type_ctor_name_and_args NYI

N2
- tests/general/double_match fails:
  io__command_line_arguments NYI

N3
- tests/general/environment fails:
  io__set_environment_variable NYI

N4
- io__read_line_as_string_2 NYI

N5
- tests/general/map_corresponding fails:
  Software Error: rtti_implementation: unimplemented: tuple comparisons
  Likewise for tests/hard_coded/existential_tuples
  tests/hard_coded/higher_order_type_manip
  tests/hard_coded/pretty_printing
  tests/hard_coded/tuple_test
  tests/hard_coded/type_ctor_desc

N6
- tests/general/io_regression fails:
  io__alloc_buffer NYI

N7
- tests/hard_coded/random_permutation random_simple fail:
  MissingMethodException due to copy/2 not being implemented
  Likewise for
  	tests/hard_coded/stable_sort
	tests/hard_coded/random_permutation
	tests/hard_coded/existential_rtti

--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------