fix: Uniformize bigint construction

[gwhitney]

Adds options to the bigint constructor function math.bigint() to
control whether/how non-integer inputs are rounded or errors thrown,
and whether "unsafe" values outside the range where integers can be
uniquely represented are converted. Changes math.numeric(x, 'bigint')
to call the bigint constructor configured to round and to allow unsafe
values.

Also restores documentation generation for constructor functions.
and initiates History sections in function doc pages

Resolves #3366.
Resolves #3368.
Resolves #3341.

[gwhitney]

OK, I have marked this as draft for two reasons:

1. In light of Inconsistencies in definitions of bigint operations and Fraction operations #3374, there is no need to for the behavior of math.bigint to depend on the predictable configuration, since it always returns a bigint (or throws) in any case. However, it would be most in line with the philosophy adopted in where we ended up on Inconsistencies in definitions of bigint operations and Fraction operations #3374 to not have the round conversion option, but always to throw if the input is not an integer; if you want to make a bigint from a non-integer, call math.round (or floor or ceil or fix) yourself. However, not having the round configuration would make this a breaking change, as the current behavior of math.bigint is to round, whereas of math.numeric(x, 'bigint') is to throw. I think either way we should have the safe conversion option, since without it there is currently no way for the conversion to check whether the conversion might be creating spurious accuracy, which is something mathjs generally tries to do. In fact, if we are going to go for a breaking change anyway, I would suggest making {safe: true} the default conversion option, as more in the spirit of mathjs in general. Please let me know whether you would like to keep the round conversion option to make this a non-breaking change, or for me to eliminate the round conversion option to align better with Inconsistencies in definitions of bigint operations and Fraction operations #3374 but make this a breaking change, and if the latter, what default value you want for the safe option.
2. I need to add support for Complex arguments, since it certainly might be the case that a Complex number could represent an integer (if its imaginary part is zero). But I am waiting on your decision on the previous point to implement this, because if we don't have round options then this can throw whenever the imaginary part is different from 0 to current epsilon configuration, whereas if we do have round options it needs to have logic to see if the imaginary part rounds to 0 under the supplied round option.

Looking forward to your call here, then I can complete this PR to be ready for review.

[josdejong]

Thanks for putting together this draft PR.

I think in general each factory function (number, complex, bignumber, fraction, bigint, ...) is a way to explicitely change the type of a numeric value. You can use that to convert strings, or when the auto conversion of mathjs does complain (like with bignumber(2) + 2/3). I think the bigint factory function should not throw when the input is a non-integer and should always "work" except like when passing a string with invalid contents or an unknown input data type. So I think we should not implement an option safe.

Right now, the rounding behavior of bigint is basically using function round. Making the rounding configurable makes sense I think. How about the option "round": "round" | "floor" | "ceil" | "fix"? What do you think should be the default? Thinking about it, most programming languages use "fix", maybe that is a better default?

About the History description in the comments: I think it will be hard to maintain this by hand. Can we automate this? I think we can expand the script that extracts the documentation from the code comments to also collect all relevant git commits and add that to the generated web page of each function.

The improvements in the examples are nice. I'm not sure though why you have added indentation to some of the inline comments // ...?

[gwhitney]

So I think we should not implement an option safe.

What facilities should we then provide to support a use that needs a bignumber but needs to avoid the creation of illusory precision where there is none? That is exactly our use case, and mathjs is generally supportive of avoiding spurious precision. So I am fine to change this particular signature, but we need a safety mechanism, and then I need to put it in this PR, since it was created and filed precisely to allow us to adopt v14 of mathjs, and our adoption is waiting on such a mechanism. Thanks for your advice/design here.

[gwhitney]

I am fine of course with keeping the rounding config. The current behavior is round, any other default will be breaking. If we are going to break, floor would definitely be the mathematicians' choice.

[gwhitney]

I think History in the source file is much more maintainable than central history: you must edit the function's source file to make the change anyway, so you add a line to the History. Once every function has such a section, I think it will become very natural, and the reviewer can easily check it was done just by glancing at the diffs. However, if I need to augment this with an autoparser to get it merged, let me know and I will do it -- but I would recommend inserting the results in the source file in the History section so you can see them there and possibly tweak them; otherwise I think they are not going to come out very readable.

[gwhitney]

The improvements in the examples are nice. I'm not sure though why you have added indentation to some of the inline comments // ...?

It is because and only because the doc test ignores indented comments, and so a cheap way to fix a doc test that is failing because of free form comments that confuse its attempts to grab the expected return value of an expression is to just indent the comment. This behavior came from a number of pre-existing doc sections that used indented comments as sort of section headers. So the "ignoring indented comments" exception was already there, and I was using it to get more tests to pass. The goal of course is to get to all doc tests passing, at which point they can finally switch to being enforced.

[josdejong]

1. About safe conversion: interesting idea. We can create a separate function like numberIfSafe(value), or extend the existing function number(value, { safe: boolean }), so you can explicitely choose to "blindly" convert bigint to number, or only do so if it doesn't lead to loss of information. What do you think?

2. About history: I really have to think through the various options that we have. Can we discuss this in a separate issue/PR please? I think this has less priority than improving bigint support, at at least I would like to get that finished asap.

3. About the example indentation: would it be possible to solve this in a more neat by having the docgenerator ignore example lines that only contain a comment?

[gwhitney]

1. About safe conversion: interesting idea. We can create a separate function like numberIfSafe(value), or extend the existing function number(value, { safe: boolean }), so you can explicitely choose to "blindly" convert bigint to number, or only do so if it doesn't lead to loss of information. What do you think?

I personally would be happier with options on the conversion functions than new functions -- there are a lot of functions, which gets a bit unwieldy. This PR currently makes a {safe: boolean} option on the bigint(something) function, which you asked me to remove. Would it actually be OK to keep it as is?

Note there is a need for some safety mechanism for number => bigint as well as for bigint => number, because if someone tries to convert a number bigger than 2^54 to a bigint, it creates something that purports to be an exact large integer, but that number most likely did not really represent an exact integer, it likely had roundoff error. Un unchecked conversion of this type creates a false sense of more precision than there really was. It is exactly these kinds of number => bigint conversion that are messing up our modular arithmetic calculations, so we need a mechanism for stopping them.

2. About history: I really have to think through the various options that we have. Can we discuss this in a separate issue/PR please? I think this has less priority than improving bigint support, at at least I would like to get that finished asap.

Yes, I get it. Happy to discuss in the history issue #3341. Does that mean you would like me to remove the History sections I added and the code that supported them from this PR, or should I leave them in as not doing any harm, and to keep around the information I generated, and they can always be massaged to the decided-on scheme later? Fine either way, just want to know what to do to get this PR in shape in the most direct way possible.

3. About the example indentation: would it be possible to solve this in a more neat by having the docgenerator ignore example lines that only contain a comment?

I will give it a try and let you know.

[gwhitney]

5. About the example indentation: would it be possible to solve this in a more neat by having the docgenerator ignore example lines that only contain a comment?

I will give it a try and let you know.

OK, got through this. Unsurprisingly (with the free form nature of doc comments) it was a bit more convoluted than
expected, but now the convention is that any line with only a comment is ignored, unless that comment starts with 'returns' or 'throws'. This change actually shook out a few bugs, including one in logical "and" (and([1,1], 0) was returning [0, 0] rather than [false, false] as documented). All should be well now, and I the only things left I know of before this PR is ready for review are:

• Decision on safe conversion options and the api for them
• Should I remove the History work done so far?
• Need to implement bigint(complex(3, 0)) etc, now that rounding options are set.

I will work on the last one ASAP and look forward to your feedback on the first two.

[josdejong]

(!) Ok yes let's go for a solution with {safe: boolean} option then, so keep it as it is in the PR. I think the default value should be {safe: false} when explictly converting a numeric type. Sorry for the back and forth. I see you implemented this for the bigint() function. I think we have indeed more similar cases for with a {safe: boolean} option is interesting, like you say, not only bigint => number but also number => BigNumber for example. What do you think?

(2) Thanks. Can you please remove the History sections from this PR? (maybe you can create a new temporary feature branch containing the current History sections for later reuse, so we don't throw away your work)

(5) Nice that you managed to get the issue with example lines with comments solved 👍

[gwhitney]

Sure, I can add other safe options. Bigint => number safety is clear, as is BigNumber => number. But number => bignumber is much murkier. Nominally, every number can be exactly represented as a bignumber. But I think the "safety" you have in mind is not wanting 1.20000000001 to become a bignumber because it will make what seems to be roundoff error seem precise. I think we have discussed this before -- there is an algorithm for guessing whether a number comes from a precise rational number, based on continued fractions. But we didn't seem to reach consensus on this before. So my preference would be not to address that type of safety in this pr lest it becomes bogged down.

[gwhitney]

OK I have moved the History work to a branch feat/function_history in the main repo.

[josdejong]

Thanks!

So my preference would be not to address that type of safety in this pr lest it becomes bogged down.

Yes agree, let's adress those ideas in an other PR 👍

[gwhitney]

So my preference would be not to address that type of safety in this pr lest it becomes bogged down.

Yes agree, let's adress those ideas in an other PR 👍

The relevant discussion is in #1485.

[gwhitney]

Slightly confused: am implementing the {safe: true} option as uniformly as I can across different types, and I have gotten to the string => number case. The current code throws various errors if the string is not of the proper format or has too many significant digits or etc. But since this is an explicit request for a conversion to number, shouldn't the default string conversion just call parseFloat (i.e. do the best it can to return a number)? And should the existing logic actually be the logic when {safe: true} is specified? Please advise, thanks. Will hold off committing until I hear back.

[josdejong]

Hm. Let me think. Would we do anyone a favor by returning NaN when trying to convert a string like "foo"? I can't come up with a case where this is the desired behavior, and think we do want to always throw an exception like it does right now?

So then the description of the option "safe" would be something like: When true, it will only convert when this will not lead to a loss of information, like when converting a number with digits into a bigint. When false, it will convert also when this will lead to a loss of information. In both cases, an exception will be thrown when the conversion is not possible at all, like when trying to convert the string "foo" into a number.

Does that make sense?

[gwhitney]

So then the description of the option "safe" would be something like: When true, it will only convert when this will not lead to a loss of information, like when converting a number with digits into a bigint. When false, it will convert also when this will lead to a loss of information. In both cases, an exception will be thrown when the conversion is not possible at all, like when trying to convert the string "foo" into a number.

You could alternatively take the perspective that any string can be converted into a number without throwing when safe is false, just sometimes that number will be NaN, and if there are any characters in the string that are not reflected in the result (like an extraneous "foo" in there, like if "3.14foo" converted to 3.14, or too many digits to fit in the destination type) then that condition constitutes "loss of information" that would cause an error when safe is true. That would be more consistent with the behavior of other types besides 'string' converting to number.

However, the conversion from 'string' to every other scalar type does currently take the point of view that there are some strings that represent values of the type and other strings that simply are not representations of that type and throw errors, even when safe is false. So the behavior you advocate above is clearly more consistent with the current behavior of converting from strings to other types.

It's unfortunate that the two ways of looking at it (consistency with other types to number, vs. consistency with string to other types) lead to different conclusions. There are a couple other considerations:

1. It would be clean if we could define all conversions in these various "constructor" functions in the src/type directory and then use their "safe: true" invocations for the implicit conversions. That methodology would minimize code/logic duplication and provide a clear story to the user: implicit conversions, when allowed, are always the explicit conversions with "safe: true" specified. You will see this point come up in the next set of examples. So please also provide guidance on whether you want to try to make the code literally call the constructor functions with safe set to true when doing implicit conversions, either in this PR or in a follow-up PR.

2. mathjs currently seems rather inconsistent as to which of the two main alternatives for string conversion to number it is trying to implement. Here are some illustrative examples:

math.number('Happy 3.1416 Day!') // throws an error
math.number('Happy 0x3.243F Day!') // returns 3.14158...

math.evaluate(' "3.14"  < 3.1416 ') // returns true
math.evaluate(' "0x3.243F" < 3.1416 ') // throws an error, so
// clearly implicit conversion and explicit conversion using different code

math.evaluate('NaN < 7') // returns false (no conversion, just numbers)
math.evaluate(' "NaN" < 7 ') // throws an error
math.evaluate(' "-Infinity" < 7 ') // returns true

3. mathjs is currently missing any explicit conversion from Complex to number, and the choice of convention we make here will have implications for what the behavior of that conversion should be when I implement it.

--

The main upshot is that whichever of the below alternatives you pick, some cases of some conversions are going to have to change (but I don't think to the point of this PR being a breaking change; I think whichever way you change, all of the behavior changes can be looked at as either extensions or bugfixes to current behavior rather than breaking changes).

So in light of all the above, please let me know if you would like me to:

(A) When safe is false, make all explicit conversions from one scalar type to another always succeed. In this world, we would make, e.g., fraction("foo") return fraction(0). In the case of converting from strings, if safe is true, then if there are any characters that are not reflected in the result, throw an error. This option is the most globally consistent. If we go this way, I will add an explicit conversion of Complex to number that simply takes the real part when safe is false; and when safe is true, it returns the real part when the imaginary part is 0 and throws an error otherwise.

(B) Make type conversions always throw on those elements of the source type that "just don't represent" any element of the destination type. So all conversions from string to other scalar types would have some strings which "just don't represent that type" and throw an error regardless of the "safe" setting; when "safe" is true, even if the overall format of the string is OK, if there are any characters (e.g. too many digits) that aren't reflected in the value, also throw an error. If we go this way, I will add a conversion of Complex to number that always throws when the imaginary part is nonzero, and returns the real part when the imaginary part is zero, under the concept that a Complex number with nontrivial imaginary part "just doesn't represent" a (real) number. Moreover, the consistent behavior under this option for converting 24.5 to bigint would be to always throw (regardless of whether safe is true or false), because 24.5 "just doesn't represent an integer". The advantage of that behavior for the "bigint" constructor is that it would remove the "round" option altogether; if you want to convert a potentially non-integer number to bigint, you just have to call your desired rounding function on it first. Similarly converting fraction(5, 3) to bigint would always throw, as 5/3 is a rational number that "just isn't an integer". (The mental model is that ℤ ⊂ ℚ ⊂ ℝ ⊂ ℂ, represented by bigint, Fraction, number/BigNumber, and Complex, respectively, and all the inclusions are strict, so that elements at one level that aren't in a prior level cannot just be "converted" to the prior type.)

(C) Make type conversions from string always throw on certain strings that "don't represent" the definition type, and all other scalar conversions always return something when safe is false. In this option, number("32foo") and fraction("foo") etc. would all throw, whereas bigint(complex(37.6, -19.7)) would return 38 (since this conversion is between two non-string scalar types, it has to return something). This option is less globally consistent than (A) or (B) (the string type is handled differently from all other scalar types) but likely the smallest total set of changes to current behavior. In this version, the missing Complex -> number conversion would behave as in (A), since both Complex and number are non-string scalars.

(D) Maybe there's some other option I missed that you would prefer to any of the above?

Oh and please also let me know your thoughts on whether the implicit conversions (generated by typed-function rules) should be changed to call these constructor functions with safe set to true. That change should avoid the sort of inconsistencies noted in item (2) above and reduce some code duplication.

[josdejong]

Maybe I should rephrase

"When true, it will only convert when this will not lead to a loss of information"

Into:

"When false it wil convert also when this will lead to a loss of precision"

I think converting a string to a number is really different from converting another numeric data type to a number, because a string can contain anything, whereas a numeric type can only contain numeric information, and, so is in the same "realm" as a number already.

1. Yes that makes sense, using the {safe: true} version of the constructor functions for implicit conversions to ensure consistency. I'm not sure how easy that is to implement, right now these conversions happen in typed.js, but we should not create circular dependencies. Maybe we can extract the logic in helper functions, and use these in both typed.js and the constructor functions if needed.
2. To me, math.number('Happy 0x3.243F Day!') should throw an error since the string doesn't just contain a numeric value. I would say this is a bug and unintentional.
3. To convert a Complex to number, you can use function re. I think we do not need to let the constructor function number() support Complex numbers?

About the ideas (A) to (D): I think it is important to look at it from the user perspective and come up with a pragmatic approach here. Maybe it's easiest to look at the individual cases. What do you think about the following approach?

Example	{safe: false} (default)	{safe: true}
number("foo")	Error	Error
number("3.14foo")	Error	Error
number('Happy 0x3.243F Day!')	Error	Error
number(12345678901234567890n)	12345678901234567000	Error
number(24n)	24	24
number("12345678901234567890")	12345678901234567000	Error
number(bignumber("12345678901234567890"))	12345678901234567000	Error
number(complex(1,2))	Error (Use function re(...) instead)	Error
number(complex(4,0))	4	4
number(fraction(1,3))	0.3333333333333333	0.3333333333333333
bigint(24)	24n	24n
bigint(3.14)	3n	Error
bigint(complex(1,2))	Error (Use bigint(re(...)) instead)	Error
bigint(complex(4,0))	4n	4n
bignumber(3.141592653589793)	bignumber('3.141592653589793')	Error (because the number has >15 significant digits)

[gwhitney]

I feel strongly that number(complex()) should be implemented, for two reasons:

(a) when the complex number happens to have an imaginary part, then it obviously corresponds to a specific number -- in fact, == with respect to that number returns true!. So how could we logically justify/explain that it throws an error to "convert" such a complex number to something it is purportedly equal to?, and

(b) in our project, we have MaJE formulas that users can freely enter. Right now, there is not a way to check what is the data type that their formula produces (that should come with a mathjs core overhaul, but in the meantime...). But the TypeScript code that receives the value needs an entity of a particular type, say number. So there needs to be a uniform way to coerce the result of the formula into the type the code needs. The fewer exceptions there are to that process, and the fewer opportunities there are for it to throw, the better (cleaner, simpler) it is for our code. (In this vein please see also #3451 which I just discovered.)

I also feel that conversions should follow a disciplined principle or philosophy, as they are a notoriously touchy part of any software system that needs to do them, and there needs to be guidance for implementing future ones; I think lack of such principle/guidance led to the concerns this PR is addressing.

But which principle? Clearly you're not interested in (A) above, and that's totally fine. You also don't seem to align with (B), because you recommend bigint(3.14) to be 3n, even though, unequivocally, 3.14 cannot be construed to represent an integer. (In this vein, note that in plain JavaScript, BigInt(3) returns 3n but BigInt(3.14) throws a RangeError, so the JavaScript philosophy appears basically to be (B). )

So whether we could adopt (C) depends on how far you are willing to go with conversions from complex. If it's maintained that all conversions from complex throw, or even just that an unsafe conversion from complex(1,1) to number (or other real type) throws, then (C) is out the window as well. On the other hand, if you were to allow an unsafe conversion from complex(1,1) to any of the other number types to return 1 converted to that type, and only have a safe conversion throw, then it seems like (C) would be a good fit.

If (C) is untenable, then it becomes less clear how to proceed with deciding on the behavior of all of the conversions. They are idiosyncratic and have bugs now, as some of the examples in this conversation show, and those things are causing problems in our project, which is after all why I opened this PR. Here is another possible, somewhat more complicated, position that could be adopted:

(E) All unsafe conversions between types that represent real numbers (in the mathematical sense), including strings that are printed representations of real numbers in any of the most common conventions for writing out such numbers, should succeed as best as possible, and never throw. Safe conversions throw if there is loss of precision, or introduction of illusory precision. Conversions from strings that aren't (entirely) customary representations of numbers, to any number type, throw. Conversions (even unsafe ones) from other number systems besides the real numbers to any type representing real numbers throw when [two sub-alternatives here: (i) the value to be converted does not directly correspond to a real number; or (ii) always]. Conversions from other data types that cannot be characterized as number systems, to any number type, should always throw. (The current situation with Unit in this regard is pretty painful, see #3451, but I wouldn't propose dealing with that concern in this PR, as it would be a breaking change. ) Conversions from number types to other non-numeric data types are up to the design/semantics of that other data type.

At least, (E) would pretty much provide a blueprint for how all conversions should behave. And obviously I would argue strongly for sub-option (E.i) rather than (E.ii) as it will definitely make my life easier. But even (E.ii) would be a specification that could be documented and would direct the completion of this PR and future conversions. Looking forward to your thoughts.

[josdejong]

I'm not sure there is a JavaScripts philosophy, JavaScript can be a mess ;). Anyway, let's focus on coming up with a clear description of the philosophy for mathjs:

• Strings: convert a string to a numeric type only when the string contains a valid numeric value.
• Real numeric types (number, bigint, BigNumber, Fraction):
  • Safe conversion: convert when this will not lead to loss of precision, throw otherwise.
  • Unsafe conversion: convert numeric values also when leading to a loss of precision.
• Complex types:
  • Safe conversion: convert Complex to real only when the imaginary part is zero.
  • Unsafe conversion: this is a point of discussion. I think it would be best to throw.

I'm not sure on your preference regarding complex conversions. Do you prefer to let number(2+4i, {safe: false}) return the real part 2 and silently drop the imaginary part? Or do you only propose to do that when the imaginary part is zero? I'm totally fine the with converting a complex number with zero imaginary part to a number, but I'm strongly in doubt on whether silently dropping an imaginary part would be a good idea.

I've added a case number(complex(4,0)) to the table here #3375 (comment) to get a more complete picture.

[gwhitney]

OK, how about something like this: unsafe conversions throw when there is a complete loss of any information (parts of a string that don't encode a number, the nonzero imaginary part of a complex); safe conversions throw in addition when there is either loss of precision or creation of illusory precision. That seems fairly simple and principled, and seems to align well with both the status quo and your preferences. In this scenario, number(complex) would always throw on nonzero imaginary part, and bignumber(complex, {safe: true}) would in addition throw when there is danger that the real part has roundoff error.

[josdejong]

That is a clear description, thanks.

Would this approach also work for the project that you're currently working on? And just to be sure: would you like to have a complex number throw (that's my preference) or not?

[gwhitney]

That is a clear description, thanks.

Would this approach also work for the project that you're currently working on? And just to be sure: would you like to have a complex number throw (that's my preference) or not?

Yes, this approach will be fine. I think it means that number(3+0i) succeeds and number(3 + 2i) throws, which is also fine for us. I am assuming that is OK with you. I will proceed on this basis as soon as I am able. I think it's higher priority than nanomath, correct, since it's not good to have this PR "hanging out there"?

[josdejong]

I think it means that number(3+0i) succeeds and number(3 + 2i) throws, which is also fine for us

Yes indeed, sounds good! I'm glad we have a satisfying way ahead now.

I think it's higher priority than nanomath, correct, since it's not good to have this PR "hanging out there"?

Feel free to pick up either, depending on how your mood is :). Since this is not a hotfix I think there is no time pressure there.

[gwhitney]

OK I rebased (except for the logo commit you performed just a short while ago), and updated the top-level datatypes page in the docs, in an effort to reflect everything we've discussed and agreed on. Please could you review the new doc page and let me know if you agree with everything I've written? If so, I will go ahead and complete the PR by making sure that all of the conversion functions conform to the new documentation. Note that conforming effort will include implementing a function array(...) since Array is currently the only data type that does not have an explicit converter/constructor function. Let me know if that's OK, or if you definitely feel that instead I should change the docs to note that one exception to the converter/constructors.

[gwhitney]

P.S. On array, note also the recent Discussion question #3435, which is perhaps some evidence that the lack of an array() converter/constructor was felt at least by someone out there.

[josdejong]

OK I rebased (except for the logo commit you performed just a short while ago), and updated the top-level datatypes page in the docs, in an effort to reflect everything we've discussed and agreed on.

I've read up on docs/datatypes/index.md in the PR. I like the diagram, that helps getting an overview! Maybe some day we can replace it with an SVG image. You description with the sections "implicit" and "explicit" is really clear, thanks!

Maybe we can try to rephrase the following sentence:

However, you may supply an options object as a final argument to the conversion, and if it includes the key "safe" with a true value, a safe conversion, equivalent to that used in implicit conversion, will be performed instead.

Into something like:

To explicitly convert a value in a "safe" way, you can pass an object { safe: true } as second argument.

On a side note: wow, there where quite some errors in the "Example usage" section, thanks for improving on that 😅.

Note that conforming effort will include implementing a function array(...) since Array is currently the only data type that does not have an explicit converter/constructor function. Let me know if that's OK

Adding a function array makes sense indeed 👍.

[gwhitney]

Excellent. Will proceed, including the doc wording improvements, as soon as I can.