[erlang-questions] Bit syntax matching gotchas

[Richard A. O'Keefe]

On 3/02/16 7:17 pm, Björn Gustavsson wrote:
> There are some gotchas in the bit syntax that comes up now and then on 
> the mailing list, and also as a bug report at the end of last year: 
> http://bugs.erlang.org/browse/ERL-44

I am sorry to be so late replying to this, but things have been (a) 
frantic and
(b) horrible.
> BACKGROUND ABOUT BIT SYNTAX CONSTRUCTION When constructing binaries, 
> there is an implicit masking of the values. All of the following 
> constructions give the same result: <<255>> <<16#FF>> <<16#FFFF>> <<-1>>

That always felt so WRONG to me.
If I *want* masking, I can write <<X band 16#FF>>.

I really do not understand why this kind of thing was ever
considered acceptable:

3> <<X>> = <<X>>.
** exception error: no match of right hand side value <<"ÿ">>

If you stop doing the masking (except where there is an explicit
'band' licensing it) then
(a) some modules will break
(b) the fix will be easy
(c) the fixed code will be backwards portable
> There have been complaints about the implicit masking behaviour, but 
> there is a lot of code that depends on it, so it would be unwise to 
> change it.

I don't buy that, sorry.

Make a new option:

-bit_syntax(safe|dangerous).

allowed once per module.  dangerous is the current behaviour,
safe raises an exception for overflow.

In R19, introduce it, with dangerous as the default.
In R20, make the compiler issue a warning if a module has a bit syntax
form that would be ambiguous and doesn't state the desired behaviour.
In R21, make safe the default.
In R22, make the compiler issue a warning if a module has
-bit_syntax(dangerous).
In R23, remove the option.

This process *FIXES* the problem.

Instead of safe/dangerous, consistent/inconsistent might be better.
> POSSIBLE SOLUTION #1 The most obvious solution is probably to let the 
> compiler warn for the above cases. The matching would still fail. The 
> developer will need to fix their code. For example: <<-1/signed>> = 
> <<-1>> POSSIBLE SOLUTION #2 There is one problem with the solution #1. 
> It is not possible to produce a warning for the following example: 
> f(Val) -> <<Val:8>> = <<Val:8>>, Val.

Then solution #1 is not a solution.  But why isn't it possible?
The problem arises whenever you have <<...X:N...>> and
the compiler has no proof that X fits in N bits.  This is just
such a case.
> The proble So in addition to warning when possible, another solution 
> is to mask values also when matching. Internally, the compiler could 
> rewrite the function to something like: f(Val) -> <<NewVar:8>> = 
> <<Val:8>>, Val = NewVar band 16#FF, Val. Similar rewriting should be 
> done for literal integer, so the following expression would now match: 
> <<-1>> = <<-1>>

What exactly is the rewriting?
0> <<NuVar>> = <<-1>>,
10> -1 = NuVar band 255.
** exception error: no match of right hand side value 255

That's not a match.
> Solution #1 would point all cases when literal integers could not 
> possibly match and force the developer to fix them. Therefore I choose 
> solution #1.

The problem is not a problem with literal integers.
It is a problem with ANYTHING that doesn't fit being quietly
truncated in one context but not the other.

Anything that leaves matching and building contexts
inconsistent is not a solution.  If a match cannot
*return* a value in a certain place, the corresponding
build should not *accept* that value in that place.


The idea that this can't be fixed because there is code
depending on the old behaviour reminds me of two famous
bugs.
(C) C's switch statement is a botch, with 'break' being
     used both for "exit this loop" and "exit this switch".
     This was fixed in C's grandparent BCPL where 'break'
     was used for "exit this loop" only and 'endcase' was
     used for "exit this switch only".  C's designers were
     aware of that correction to BCPL, but felt they could
     not adopt it because they "already had 100 users".
(X) Excel notoriously gets its leap year calculations wrong
     but Microsoft have announced that this feature will not
     be changed in case there are users who depend on the bug.
I'm also reminded of a bug that was squashed.
(B) Burroughs Algol has pointers.  That made it possible for
     you to write
         EBCDIC POINTER P;
         BEGIN
            EBCDIC ARRAY A[0:79];
            P := A;
         END;
     and then use P.  They didn't notice the problem for a while.
     They then made a temporary fix for the operating system so
     that when you exited a block that declared an array, the OS
     would sweep your stack looking for pointers to it.  They also
     made an addition to the language, so that you could write
         EBCDIC ARRAY B[0:131];
         EBCDIC POINTER P FOR B;
         BEGIN
            EBCDIC ARRAY A[0:79];
            P := A;  %% This is a syntax error!
         END;
     Customers were given about a year to fix their programs,
     then warnings were turned into hard errors and the OS patch
     was removed, and presto chango, absence of dangling pointers
     proved by the compilers.  Burroughs Algol is still available
     -- I have a personal copy of the MCP and software that runs
     in a VM on a Windows box; now all I need is a Windows box --
     and for 30 years that class of bugs has been GONE.

The C bug has been perpetuated into C++, Java,  and JavaScript and
people are *still* making 'break' mistakes, all because people chose
to perpetuate old bugs rather than eliminate them.

The longer any match/build inconsistency is allowed to remain
in bit syntax, the more pain it will cause.