mac os wheel broken for 2.2.0rc2

[sylviawhoa]

I'm getting an error running tokenize with tensorflow-text==2.2.0rc2 that I can only reproduce on macs.
(same error on rc1, and possibly earlier versions)

Steps to reproduce:

1. Setup:

python3 -m venv .test_venv 
source .test_venv/bin/activate
pip install --upgrade pip
pip install tensorflow==2.2.0rc3
pip install tensorflow-text==2.2.0rc2

2. Download vocab.txt into the dir you plan to run the test:
   aws s3 cp s3://models.huggingface.co/bert/bert-base-uncased-vocab.txt ./vocab.txt

3. And then run these 5 lines in python

import tensorflow as tf
from tensorflow_text import BertTokenizer
tokenizer = BertTokenizer('./vocab.txt')
test2 = tf.convert_to_tensor(
    'Hello', dtype=tf.string
)
tokenizer.tokenize(test2)

Works on linux, (returns <tf.RaggedTensor [[[100]]]>)
On Mac, it throws an error. I've run on two separate macs (one with all totally fresh installs)

2020-04-16 13:18:07.892934: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at wordpiece_kernel.cc:204 : Invalid argument: Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/bert_tokenizer.py", line 222, in tokenize
    return self._wordpiece_tokenizer.tokenize(tokens)
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 100, in tokenize
    subword, _, _ = self.tokenize_with_offsets(input)
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 156, in tokenize_with_offsets
    tokens.flat_values)
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 182, in tokenize_with_offsets
    **kwargs))
  File "<string>", line 141, in wordpiece_tokenize_with_offsets
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE [Op:WordpieceTokenizeWithOffsets]

running on python 3.7.6

[tgsmith61591]

I'm getting the exact same error in Python 3.6.8 - any ideas how to work around? Here's some more reproducible code:

import tensorflow as tf
import tensorflow_text as tf_text

with open("./vocab.txt", "r") as v:
    vocab = v.read().split("\n")

init = tf.lookup.KeyValueTensorInitializer(
    vocab,
    tf.range(tf.size(vocab, out_type=tf.int64), dtype=tf.int64),
    key_dtype=tf.string,
    value_dtype=tf.int64)

table = tf.lookup.StaticVocabularyTable(init, 1, lookup_key_dtype=tf.string)
tokenizer = tf_text.BertTokenizer(
    table,
    lower_case=False,
    normalization_form=None)

tokenizer.tokenize(
    tf.convert_to_tensor(["this is a rad sentence", "i wish i could tokenize it"], dtype=tf.string)
)

Getting:

Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/bert_tokenizer.py", line 222, in tokenize
    return self._wordpiece_tokenizer.tokenize(tokens)
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 100, in tokenize
    subword, _, _ = self.tokenize_with_offsets(input)
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 156, in tokenize_with_offsets
    tokens.flat_values)
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 182, in tokenize_with_offsets
    **kwargs))
  File "<string>", line 141, in wordpiece_tokenize_with_offsets
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE [Op:WordpieceTokenizeWithOffsets]

The same exact code works on my EC2 instance, the only difference being the python version is 3.6.10:

<tf.RaggedTensor [[[2023], [2003], [1037], [10958, 2094], [6251]], [[1045], [4299], [1045], [2071], [19204, 4697], [2009]]]>

[sylviawhoa]

@tgsmith61591 Pretty sure you'll need to run on

tensorflow==2.1.0
tensorflow-text==2.1.1

until they fix the mac os wheel. You can also run on a non mac machine (EC2 works cause it's Linux)

[tgsmith61591]

Yeah. Disappointing since the custom code I've written depends on the train_step code present in the tf.keras.Model class, which means I'm stuck on 2.2.0. Hopefully this is fixed soon

[sylviawhoa]

I haven't tried tensorflow==2.2.0, tensorflow-text==2.1.1 - there's a pretty good chance that will work

[tgsmith61591]

This still appears to be busted in the 2.2.0 release

[johnmoberg]

I have the same issue.

[tgsmith61591]

Any word on this @thuang513 ? Any chance we can expect a patch soon?

[thuang513]

@broken can you take a look?

[broken]

This actually looks like a problem with the validation of LookupResource in core TF. I see a similar bug opened up on their end. tensorflow/tensorflow#39524
I'll see what I can dig up.

[liyinhgqw]

I have this issue when I am using 1.15.3.
I suspect that it is due to abi compatible issues.
FYI @broken , I compiled tensorflow using MacMini of clang version 11.0.0 and our customised op using Apple clang version 11.0.3. Then this issue happened.

[r-wheeler]

Confirmed this happens on my mac with tensorflow 2.2.rc3 running python 3.6.8 on Catalina.