[m-dev.] .NET back-end test results

[Fergus Henderson]

On 20-Oct-2002, Fergus Henderson <fjh at cs.mu.OZ.AU> wrote:
> I've run the tests in the `il' grade and analyzed the results.
> Here are the results of my analysis.

Here are the updated results, after a week of serious effort
directed towards squashing bugs.

I've attached a list of all the test cases which currently fail,
so that people can test changes which affect the .NET back-end and
make sure that they do not introduce any new regressions.

I haven't fully reanalyzed all the test case failures,
so some of the stuff below is overly optimistic;
some new bugs will show up once I've analyzed the reasons
for all the test case failures.

> First, the number of distinct bug symptoms detected:
> 
> 	Bugs in the test suite: 5

	Fixed: 2
	Newly discovered/analyzed: 3
	----------------------------
	Updated total: 6

> 	Bugs in the .NET back-end: 8

	Fixed: 6
	Newly discovered/analyzed: 1
	----------------------------
	Updated total: 3

> 	Unimplemented features: 7

	Fixed: 1
	----------------------------
	Updated total: 6

Putting that all together:

 	Bugs in the test suite: 6
 	Bugs in the .NET back-end: 3
 	Unimplemented features: 6
	
> Passed test cases: 280
> Failed test cases: 459 total

	Passed test cases: 381
	Failed test cases: 383 total


SPURIOUS FAILURES

S1
- test/valid/* report spurious failures due to Mmakefile making
  incorrect assumption that we generate `.o' files (FAILED.VALID)

S1b
- many tests in test/valid/* report spurious failures due to
  the use of procedures declared with `:- external' and not defined.

S1c
- many tests in test/valid/* report spurious failures due to the use of Aditi

S1d
- many tests in test/valid/* report spurious failures due to the use of
  --gc accurate

S2
- test/invalid/* report spurious failures due to the Mmakefile not catering
  for --il implying --intermodule-optimization (FAILED.INVALID)

S3
- lots of tests fail due to the use of C foreign clauses (FAILED.NO_CLAUSES)
  tests/general/impure_foreign is a special case of this;
  it fails at run-time because the Mercury fall-back clauses just call error/1.
  tests/hard_coded/pragma_import and tests/hard_coded/pragma_export are
  similar.

S4 [fixed]
- tests/hard_coded/*.split (not supported for il grade)

S5 [fixed]
- tests/hard_coded/special_character fails, due to '\0321' in the
  source code getting converted to octal 245 in the output file.
  This is a bug in the test case; it assumes that the internal
  representation is the same as the file representation, which is
  not the case for .NET.

BUGS

B1 [fixed]
- tests/hard_coded/term_io_test fails:
  gets syntax errors at run-time, due to problems with putback_char.

B2 [fixed]
- tests/general/commutative fails:
  tests/hard_coded/agg fails:
  invalid cast, due to RTTI support assuming low-level data representation
  Likewise for lots of others (FAILED.INVALID_CAST)

B3
- tests/general/interpreter fails, due to problem with equivalence types;
  the argument types for the unify & compare predicates for equivalence types
  are invalid (undefined)
  Likewise for lots of others (FAILED.UNDEFINED_CLASS)

B4 [fixed]
- tests/general/string_to_float fails,
  because string__to_float throws an exception rather than failing

B5
- tests/general/string_format_test fails, due to various formatting errors
  (e.g. no left justification, missing 0x prefixes, not truncating).
  Likewise for
	  tests/hard_coded/float_rounding_bug
	  tests/hard_coded/float_reg
	  tests/hard_coded/arbitrary_constraint_class
	  tests/hard_coded/arbitrary_constraint_pred_1
	  tests/hard_coded/arbitrary_constraint_pred_2
	  tests/hard_coded/module_test
	  tests/hard_coded/string_to_float

B6 [fixed]
- tests/hard_coded/*foreign_type* fail, due to not finding `mercury.dll'.

B7 [fixed]
- tests/hard_coded/no_fully_strict fails,
  because uncaught exceptions do not result in a non-zero exit status
  Likewise for
  	tests/hard_coded/uncaught_exception

B8 [fixed]
- tests/hard_coded/no_warn_singleton fails,
  due to determinism warning for table_io_copy_io_state from table_builtin.m
  Likewise for
  tests/warnings/arg_order_rearrangement
  and lots of tests in tests/valid.

B9
- tests/valid/soln_context dies with "stack underflow"
  due to problem with "free->free" modes

NOT YET IMPLEMENTED (FAILED.SORRY):

N1 [fixed]
- lots of tests fail due to
  type_ctor_name_and_args NYI

N2
- tests/general/double_match fails:
  io__command_line_arguments NYI

N3
- tests/general/environment fails:
  io__set_environment_variable NYI

N4
- io__read_line_as_string_2 NYI

N5
- tests/general/map_corresponding fails:
  Software Error: rtti_implementation: unimplemented: tuple comparisons
  Likewise for tests/hard_coded/existential_tuples
  tests/hard_coded/higher_order_type_manip
  tests/hard_coded/pretty_printing
  tests/hard_coded/tuple_test
  tests/hard_coded/type_ctor_desc

N6
- tests/general/io_regression fails:
  io__alloc_buffer NYI

N7
- tests/hard_coded/random_permutation random_simple fail:
  MissingMethodException due to copy/2 not being implemented
  Likewise for
  	tests/hard_coded/stable_sort
	tests/hard_coded/random_permutation
	tests/hard_coded/existential_rtti

-- 
Fergus Henderson <fjh at cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
-------------- next part --------------
debugger/breakpoints
debugger/browse_pretty
debugger/cmd_quote
debugger/debugger_regs
debugger/exception_cmd
debugger/exception_value
debugger/exception_vars
debugger/existential_type_classes
debugger/field_names
debugger/higher_order
debugger/implied_instance
debugger/interpreter
debugger/loopcheck
debugger/lval_desc_array
debugger/multi_parameter
debugger/polymorphic_output
debugger/print_goal
debugger/queens_rep
debugger/resume_typeinfos
debugger/type_desc_test
debugger/completion
debugger/nondet_stack
debugger/shallow
debugger/all_solutions
debugger/browser_test
debugger/queens
debugger/retry
debugger/tabled_read
debugger/tabled_read_unitize
debugger/tabled_read_decl
debugger/interactive
debugger/declarative/aadebug
debugger/declarative/app
debugger/declarative/args
debugger/declarative/backtrack
debugger/declarative/big
debugger/declarative/browse_arg
debugger/declarative/catch
debugger/declarative/comp_gen
debugger/declarative/deep_warning
debugger/declarative/dependency
debugger/declarative/dependency2
debugger/declarative/family
debugger/declarative/filter
debugger/declarative/func_call
debugger/declarative/gcf
debugger/declarative/higher_order
debugger/declarative/ho2
debugger/declarative/ho3
debugger/declarative/ho4
debugger/declarative/ho5
debugger/declarative/if_then_else
debugger/declarative/input_term_dep
debugger/declarative/ite_2
debugger/declarative/lpe_example
debugger/declarative/neg_conj
debugger/declarative/negation
debugger/declarative/oracle_db
debugger/declarative/output_term_dep
debugger/declarative/pd
debugger/declarative/propositional
debugger/declarative/queens
debugger/declarative/shallow
debugger/declarative/small
debugger/declarative/solutions
debugger/declarative/special_term_dep
debugger/declarative/tabled_read_decl
debugger/declarative/throw
debugger/declarative/untraced_subgoal
general/double_error
general/environment
general/intermod_type
general/interpreter
general/io_regression
general/read_line_as_string
general/string_format_test
general/string_to_float
general/map_corresponding
hard_coded/bigtest
hard_coded/c_write_string
hard_coded/common_type_cast
hard_coded/constraint_order
hard_coded/construct_test
hard_coded/deconstruct_arg
hard_coded/deep_copy
hard_coded/deep_copy_exist
hard_coded/det_in_semidet_cntxt
hard_coded/dupcall_impurity
hard_coded/eqv_type_bug
hard_coded/erroneous_liveness
hard_coded/error_func
hard_coded/existential_float
hard_coded/existential_types_test
hard_coded/expand
hard_coded/export_test
hard_coded/factt
hard_coded/float_field
hard_coded/float_reg
hard_coded/float_rounding_bug
hard_coded/foreign_import_module
hard_coded/foreign_type3
hard_coded/frameopt_pragma_redirect
hard_coded/getopt_test
hard_coded/ground_dd
hard_coded/higher_order_type_manip
hard_coded/ho_func_default_inst
hard_coded/ho_func_reg
hard_coded/ho_solns
hard_coded/ho_univ_to_type
hard_coded/impure_foreign
hard_coded/impure_prune
hard_coded/intermod_c_code
hard_coded/intermod_foreign_type
hard_coded/intermod_multimode_main
hard_coded/mode_choice
hard_coded/multimode
hard_coded/name_mangling
hard_coded/no_inline
hard_coded/no_warn_singleton
hard_coded/nullary_ho_func
hard_coded/pprint_test
hard_coded/pragma_c_code
hard_coded/pragma_export
hard_coded/pragma_import
hard_coded/pragma_inline
hard_coded/pretty_printing
hard_coded/random_permutation
hard_coded/random_simple
hard_coded/redoip_clobber
hard_coded/remove_file
hard_coded/rnd
hard_coded/space
hard_coded/special_char
hard_coded/stable_sort
hard_coded/string_alignment_bug
hard_coded/target_mlobjs
hard_coded/term_to_univ_test
hard_coded/test_bitset
hard_coded/time_test
hard_coded/tuple_test
hard_coded/type_ctor_desc
hard_coded/type_ctor_desc_manip
hard_coded/type_qual
hard_coded/type_to_term_bug
hard_coded/unused_float_box_test
hard_coded/write
hard_coded/write_reg1
hard_coded/write_reg2
hard_coded/copy_pred
hard_coded/copy_pred_2
hard_coded/factt_non
hard_coded/compare_representation
hard_coded/compare_rep_usereq
hard_coded/type_tables
hard_coded/inline_nondet_pragma_c
hard_coded/nondet_c
hard_coded/mapped_module.split
hard_coded/split_c_files.split
hard_coded/typeclasses/arbitrary_constraint_class
hard_coded/typeclasses/arbitrary_constraint_pred_2
hard_coded/typeclasses/arbitrary_constraint_pred_1
hard_coded/typeclasses/complicated_constraint
hard_coded/typeclasses/exist_disjunction
hard_coded/typeclasses/existential_type_classes
hard_coded/typeclasses/existential_rtti
hard_coded/typeclasses/func_default_mode_bug
hard_coded/typeclasses/ground_constraint
hard_coded/typeclasses/impure_methods
hard_coded/typeclasses/instance_unconstrained_tvar_type_spec
hard_coded/typeclasses/mode_decl_order_bug
hard_coded/typeclasses/module_test
hard_coded/typeclasses/record_syntax
hard_coded/typeclasses/superclass_bug
hard_coded/typeclasses/superclass_call
hard_coded/typeclasses/typeclass_exist_method
hard_coded/sub-modules/accessibility
hard_coded/sub-modules/accessibility2
hard_coded/sub-modules/parent
hard_coded/sub-modules/parent2
hard_coded/sub-modules/nested
hard_coded/sub-modules/nested2
hard_coded/sub-modules/nested3
hard_coded/sub-modules/class
hard_coded/exceptions/test_try_all
hard_coded/exceptions/tricky_try_store
hard_coded/purity/purity
hard_coded/purity/impure_func_t1
hard_coded/purity/impure_func_t6
invalid/aditi_errors
invalid/aditi_state_errors
invalid/aditi_update_derived_relation
invalid/aditi_update_errors
invalid/imported_mode
invalid/missing_parent_import
invalid/sub_c
invalid/test_nested
invalid/transitive_import
invalid/transitive_import_class
invalid/undef_mod_qual
invalid/foreign_singleton-nodepend
invalid/foreign_type_2-nodepend
invalid/impure_method_impl-nodepend
invalid/multimode_missing_impure-nodepend
invalid/multimode_syntax-nodepend
invalid/overloading-nodepend
invalid/pragma_c_code_dup_var-nodepend
invalid/pragma_c_code_no_det-nodepend
invalid/undef_symbol-nodepend
invalid/uniq_neg-nodepend
invalid/purity/impure_func_t2
invalid/purity/impure_func_t3
invalid/purity/impure_func_t4
invalid/purity/impure_func_t5
invalid/purity/impure_func_t7
invalid/purity/impure_pred_t1
invalid/purity/impure_pred_t2
invalid/purity/purity
invalid/purity/purity_nonsense
invalid/purity/purity_type_error
tabling/boyer
tabling/expand
tabling/expand_float
tabling/expand_poly
tabling/expand_tuple
tabling/fib
tabling/fib_float
tabling/fib_list
tabling/fib_string
tabling/loopcheck
tabling/unused_args
valid/unify_typeinfo_bug
valid/agc_graph
valid/agc_ho_pred
valid/agc_ite
valid/agc_unbound_typevars
valid/agc_unbound_typevars2
valid/agc_unused_in
valid/abstract_typeclass
valid/constraint_proof_bug
valid/func_method
valid/instance_superclass
valid/instance_unconstrained_tvar
valid/repeated_class_constraint
valid/typeclass_constraint_no_var
valid/typeclass_det_warning
valid/any_inst_merge
valid/common_struct_bug
valid/compl_unify_bug
valid/complicated_unify
valid/constrained_poly_bound_arg
valid/constrained_poly_insts
valid/constructor_arg_names
valid/dcg_test
valid/deforest_bug
valid/deforest_loop
valid/deforest_rerun_det
valid/det_condition
valid/det_inference
valid/det_switch
valid/double_vn
valid/easy_nondet_test
valid/easy_nondet_test_2
valid/empty_bound_inst_list
valid/empty_switch
valid/error
valid/existential_cons
valid/explicit_quant
valid/fail_ite
valid/followcode_det_problem
valid/foreign_underscore_var
valid/func_int_bug_main
valid/func_default_modes
valid/headvar_not_found
valid/higher_order
valid/higher_order2
valid/higher_order3
valid/higher_order_implied_mode
valid/ho_func_call
valid/ho_inst
valid/ho_unify
valid/implied_mode
valid/indexing
valid/inhibit_warn_test
valid/inlining_bug
valid/inst_perf_bug_1
valid/int64
valid/intermod_dcg_bug
valid/intermod_impure
valid/intermod_lambda
valid/intermod_nested_module
valid/intermod_nested_module_bug
valid/intermod_nested_uniq
valid/intermod_pragma_import
valid/intermod_quote
valid/intermod_record
valid/intermod_test
valid/intermod_typeclass
valid/intermod_type_spec
valid/intermod_user_equality
valid/intermod_user_equality_nested
valid/lambda_inference
valid/lambda_instmap_bug
valid/lambda_output
valid/lambda_quant
valid/lambda_quant_bug
valid/lambda_recompute
valid/lambda_struct_bug
valid/lambda_switch
valid/lambda_type
valid/lazy_list
valid/liveness_nonlocals
valid/livevals_seq
valid/loop
valid/loop_in_disj
valid/merge_ground_any
valid/middle_rec_labels
valid/modes_bug
valid/mode_syntax
valid/module_a
valid/module_b
valid/module_c
valid/module_d
valid/module_e
valid/mostly_uniq_bug
valid/mostly_uniq_neg
valid/mostly_uniq_mode_inf
valid/multidet_prune1
valid/multidet_test
valid/nasty_func_test
valid/nested_mod_type_bug
valid/nondet_live
valid/parsing_bug_main
valid/pred_with_no_modes
valid/qualified_cons_id
valid/quantifier_warning
valid/record_syntax_bug
valid/record_syntax_bug_2
valid/record_syntax_bug_3
valid/record_syntax_bug_4
valid/record_syntax_bug_5
valid/recursive_no_tag_type
valid/same_length_2
valid/semidet_disj
valid/shape_type
valid/simplify_bug
valid/simplify_bug2
valid/soln_context
valid/some_switch
valid/spurious_purity_warning
valid/stack_alloc
valid/static
valid/subtype_switch
valid/switch_detection_bug
valid/switch_detection_bug2
valid/switches
valid/tabled_io
valid/tabled_for_io
valid/transitive_instance
valid/tricky_assert2
valid/tricky_ite
valid/tuple_eqv
valid/two_pragma_c_codes
valid/two_way_unif
valid/typeclass_inlining_bug
valid/type_inf_ambig_test
valid/unbound_inst_var
valid/unbound_tvar_in_lambda
valid/undead_proc
valid/uniq_unify
valid/uniq_mode_inf_bug
valid/unreachable_code
valid/unused_args_test2
valid/vn_float
valid/zero_arity
valid/complex_failure
valid/semi_fail_in_non_ite
valid/aditi_error_bug
valid/aditi_update
valid/base_relation
valid/base_relation2
valid/context_anc
valid/ite_to_disj
warnings/singleton_test