null string not generated by default

[mathias-brandewinder]

Consider the following:

let longEnough (s:string) = s.Length > 10
let check (s:string) =
s.Length <= 10 ==> (longEnough s = false)
Check.Quick check

let (bad:string) = null
longEnough (bad)

I would have expected FsCheck to propose a null string as a counter-example. Would love to hear if this is by design (and why), or by accident.

[kurtschelfthout]

It's by accidental design ;). Originally, FsCheck targeted F# exclusively (it still does, largely...the C# interface is a bit of a mess) and so I didn't really think to include nulls. I am open to changing this though for reference types like string. I think it would make sense since the F# code participates in the .NET ecosystem, despite the backwards incompatibility it would introduce.

[kurtschelfthout]

Also see this recent question on SO:

http://stackoverflow.com/questions/23225948/how-to-generate-null-strings-for-fscheck-tests

[Andrea]

I have to say it cough me too :D

[mathias-brandewinder]

My vote is to include nulls; it's such a common issue that I would expect FsCheck to catch it. I remember Pex did it (but then, it targeted C# from the beginning). It might be worth considering adding a friendly syntax for preconditions, along the lines of (notNull s) ==> whatever?

[kurtschelfthout]

Added in 17af4ae

[ploeh]

While I do realize that I'm late to the game, I'm sad to see the inclusion of null as a possible generated value.

One of the great benefits of F# is that it allows me to focus on the important stuff, and forget about nulls. As long as you stay within F#, nulls are impossible, so I find it counter-productive that FsCheck by default now forces me to deal with them anyway.

It's true that when interfacing with the rest of the .NET ecosystem, you'll need to deal with the fact that objects can be null, but you'd almost always need to write a special branch in your code to deal with nulls.

The reason I find FsCheck useful is that it enables me to automatically generate lots of values that all fall within the same Equivalence Class.

As I see it, a Property is essentially a description of a relationship between input and output; a statement that given input drawn from this particular set of equivalent values, the output will have that particular deterministic relationship with the input.

Another way to view it is as a manifestations of Bertrand Meyer's concept of Design by Contract. An operation (function, method, whatever) has a certain set of preconditions. If those preconditions are satisfied, then the outcome guarantees a certain set of postconditions.

The reason I bore you with the above is that I'm trying to explain that null rarely falls in the same Equivalence Class as anything else. Null is useless: you can't do anything with it, because if you try, you'll get a NullReferenceException. The only thing you really can do with null is to check for it:

if x = null then
    raise (ArgumentNullException("x"))

Notice that this is a branch in the code, for only that purpose. A particular test function or method should only exercise a single branch though the System Under Test. There's only one test case that exercises that code branch, and that is when x is null.

Therefore, I would argue that null is a special test case in the vast majority of cases, and shouldn't be included as a standard value in the the set of strings, objects, etc. Null isn't equivalent to anything else.

The current (new) behaviour makes FsCheck much less useful for testing proper F# code, because I'll now need to filter away null values in most of my test cases; and it doesn't provide much value when I need to deal with null (for interop reasons), because null handling is a special branch anyway, and should be tested as a special test case.

[kurtschelfthout]

I appreciate your annoyance at this change. I have probably not done a good job of explaining the rationale in the above, and thus it looks like this was changed on a whim just for C# compatibility. It was not.

The overarching criterion I think should be applied for default generators is that the default generator for a type should generate all the possible values for it (bearing in mind size and other limitations). This is to make the default generators/shrinkers more easily amenable to customization (it's easier to filter out values than to add them, esp. for shrinking) on the one hand, and to force the user to make the assumption that a string will not be null or any other class of values, very explicit in the specification. I.e. not generating nulls by default can catch a user out after the fact with a bug in production; generating nulls forces their hand in thinking about whether or not it's allowed.

Also, the use of null, or lack thereof, in F# is essentially just a convention for strings specifically. For record, union types e.g. I agree with you. But you can write let s : string = null and the compiler does not give a peep. Also strings are often input or read from an outside source like a db, where they very much can be null.

Finally, given String.IsNullOrEmpty and IsNullOrWhiteSpace I don't think your argument that null isn't equivalent to anything else is as universal as you claim.

With all this in mind, I don't think I'll be reverting this change, sorry.

[ploeh]

Well, I can understand the argument of producing as many 'valid' values as possible.

However, this new behaviour becomes rather troublesome when I ask FsCheck to generate records, e.g. of

type Foo = {
    Bar : string
    Baz : string
    Qux : string }

This means that in each and every test case I write, I'll need to explicitly write a boolean expression to filter away all values where either of these strings are null. That's not practical.

Is there a way FsCheck can be configured to turn off that null behaviour for an entire test suite?

When it comes to IsNullOrEmpty and IsNullOrWhitespace, I believe that it's a logical fallacy to deduce from these methods that null is equivalent to white space.

[kurtschelfthout]

Sure, just override the default string generator with one that does not generate nulls. With xunit, you can currently do this on a class/module level or individual test level using the Arbitrary argument on PropertyAttribute.

On 21 Nov 2014, at 19:17, Mark Seemann notifications@github.com wrote:

Well, I can understand the argument of producing as many 'valid' values as possible.

However, this new behaviour becomes rather troublesome when I ask FsCheck to generate records, e.g. of

type Foo = {
Bar : string
Baz : string
Qux : string }
This means that in each and every test case I write, I'll need to explicitly write a boolean expression to filter away all values where either of these strings are null. That's not practical.

Is there a way FsCheck can be configured to turn off that null behaviour for an entire test suite?

When it comes to IsNullOrEmpty and IsNullOrWhitespace, I believe that it's a logical fallacy to deduce from these methods that null is equivalent to white space.

—
Reply to this email directly or view it on GitHub.

[ploeh]

Is there an example of that?

If not, I'll probably figure it out, and be sure to blog about it, or add it to the documentation, but no need to do that if there's already an example :)

[kurtschelfthout]

Yes, it's described in the Using FsCheck.Xunit part of https://fsharp.github.io/FsCheck/RunningTests.html. Well, the example is to generate non-negative floats, but that example is pretty much a blueprint for what you want.

[ploeh]

Thanks; I should have considered that before asking. That looks viable.

[haf]

I also got stuck here while upgrading to get the support of DateTimeOffset, but I'm not using XUnit (who wants attributes all over the place?).

Would you consider taking in Fuchu.FsCheck as a core supported test runner and write some docs on how to use it?

I'm really not interested in nulls for this code; can I override it globally? I can only find C# examples on StackOverflow and I can't find the override stuff in the docs either...

Here's the possibility of giving a Config...