[erlang-questions] Are binaries relocatable?

[Hynek Vychodil]

I think I have found what causes deadlock. I have made sort of cache for
binaries and use global lock as first testing solution to prove concept.
There is a scenario when I return enif_make_resource_binary and then make
sub-binary of this binary (in erlang code) and then store this code again
in cache. Then reference sub-binary reaches zero, it calls resource
destructor, locks mutex and removes from cache which calls enif_free_env()
witch causes calling destructor of its resource which tries lock mutex
again. It can't happen when I have used fresh new binary in mine previous
version.
I have tested to remove the only possible place in erlang where any
sub-binary of existing binary can be stored in cache and it works. So now I
have to wrap head around how call enif_free_env() outside mutex-locked code
part.
Anyway thanks a lot for your help.
    Hynek


On Mon, Sep 1, 2014 at 7:26 PM, Sverker Eriksson <
sverker.eriksson@REDACTED> wrote:

> Looks like that should work if you're doing it right.
>
> * Do you update shared data (like your reference counters) in a thread
> safe manner?
>
> * Another way is to store the copied binary term and then use
> enif_make_copy(env, entry->content.bin.term)
> to return the binary. You can also combine it with enif_make_sub_binary if
> you want to return part of the binary.
>
> * Use debug built VM to ease trouble shooting:
>
> cd $ERL_TOP/erts/emulator && make TYPE=debug smp plain
> $ERL_TOP/bin/cerl -debug
>
>
> /Sverker, Erlang/OTP
>
>
>
> On 08/30/2014 07:07 AM, Hynek Vychodil wrote:
>
>> Thanks for your answer very much. If I understand it correctly this way it
>> should be safe:
>>
>>       ErlNifBinary bin;
>>       if (!enif_is_binary(env, argv[0]))
>>           return enif_make_badarg(env);
>>
>>       ErlNifEnv *priv_env = enif_alloc_env();
>>       ERL_NIF_TERM term = enif_make_copy(priv_env, argv[0]);
>>       ErlNifBinary bin;
>>       enif_inspect_binary(priv_env, term, &bin);
>>       entry->content.bin = (struct my_bin){.env = priv_env, .size =
>> bin.size, .data = bin.data};
>>
>> And now mine entry->content.bin.data should be safe but unfortunately it
>> still eventually hangs whole VM after few thousands cycles of referencing
>> this data
>>
>>          CacheEntryRes_t *res = enif_alloc_resource(dc_entry_type,
>> sizeof(CacheEntryRes_t));
>>          unless (res) return insufficient_memory(env);
>>          res->entry = entry;
>>          ERL_NIF_TERM result = enif_make_resource_binary(env, res,
>>                  entry->content.bin.data, entry->content.bin.size);
>>          enif_release_resource(res);
>>          return result;
>>
>> I do mine own reference count and release by
>>
>>         enif_free_env(entry->content.bin.env);
>>
>> It hangs only if I replace enif_alloc() + memcpy() and enif_free() version
>> with above one :-(
>>
>> Thanks
>>       Hynek
>>
>>
>> On Fri, Aug 29, 2014 at 3:02 PM, Sverker Eriksson <
>> sverker.eriksson@REDACTED> wrote:
>>
>>  On 08/28/2014 03:09 PM, Hynek Vychodil wrote:
>>>
>>>  Hi all,
>>>> I would like to know if binaries are relocatable. If I have binary
>>>> parameter
>>>>
>>>>       ErlNifBinary bin;
>>>>       if (!enif_inspect_binary(env, argv[0], &bin))
>>>>           return enif_make_badarg(env);
>>>>
>>>> I store binary in mine own environment
>>>>
>>>>       ErlNifEnv *priv_env = enif_alloc_env();
>>>>       enif_make_copy(priv_env, argv[0]);
>>>>
>>>> Can I then rely on bin.data pointing into same memory until I free
>>>> enif_free_env(priv_env) ?
>>>>
>>>>  No.
>>> Your bin.data is refering to the original term argv[0] in 'env' which is
>>> only valid until the NIF returns.
>>> You must do enif_inspect_binary on the copy in order to get a safe
>>> pointer
>>> that is valid until you do enif_free_env(priv_env).
>>>
>>>
>>>
>>>     I'm asking because I would like to avoid copying
>>>
>>>> and share data or even return it using enif_make_resource_binary to
>>>> catch
>>>> and
>>>> count references.
>>>>
>>>>  Whether a copied binary share data with its original is implementation
>>> dependent. Today big binaries (>64 bytes) are shared
>>> and small binaries are copied between environments.
>>>
>>>
>>>   I have code which makes copy of binary data but when I switch to
>>> reference
>>>
>>>> bin.data it hangs whole VM. I would like to know if I make some mistake
>>>> in
>>>> mine code and it should work or it is wrong idea at all.
>>>>
>>>>
>>>>   There is a known bug when calling enif_make_copy after
>>>>
>>> enif_inspect_binary
>>> that can cause the binary to get reallocated and thus the bin.data
>>> pointer
>>> to get invalid
>>> before the NIF returns. To detect this bug, you can call
>>> enif_inspect_binary(env,argv[0],&bin2)
>>> after enif_make_copy and check if bin2.data != bin.data.
>>>
>>> This bug has been lingering as I thought it was an exotic scenario
>>> (apparently not) and I have
>>> been a bit unsure how to fix it without ruin performance.
>>>
>>>
>>> /Sverker, Erlang/OTP
>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140901/a74c0ed4/attachment.htm>