(Written by @Kiikurage) Original question is in Japanese, but answer is also written in English.

(英語のほうがよければ英語で書きますのでその旨お知らせください)

Kerasには入力の各点に学習可能なバイアスを加えるレイヤーがないようなので、カスタムレイヤー(Bias)を作りました。 このレイヤーを含むKerasモデルをWebDNNで利用するにはどうしたらよいでしょうか？

ちなみにBiasは以下のような簡単なものです。

from keras.engine.topology import Layer
from keras import initializers

class Bias(Layer):
    """
    Custom keras layer that simply adds a scalar bias to each location in the input
    """
    
    def __init__(self, initializer='uniform', **kwargs):
        super(Bias, self).__init__(**kwargs)
        self.initializer = initializers.get(initializer)
    
    def build(self, input_shape):
        self.bias = self.add_weight(name='{}_W'.format(self.name), shape=(input_shape[-1],), initializer=self.initializer)
    
    def call(self, x):
        return x + self.bias

I try to run our model with WebGL backend but I find that the result is wrong.

Our generator model looks like

I try to print calculation results from both WASM and WebGL backends.

I find that two vectors(64 * 128 * 128 feature maps) only differ 0.08 in total before the last conv layer, but after the last conv layer, the total difference becomes 54474...

I upload some debug files here model_before_last_conv is our model without the last conv layer(also without tanh) model_after_last_conv is our model with the last conv layer(without tanh) model_last_conv only contains the last conv layer, I do not know how to debug it independently

Let me know if you need any further information

I tried to use pix2pix implementation. https://github.com/knok/vess2ret

The main network architecture is U-Net, concatinating each layers outputs in encoder side to same level layers in decoder side.

WebDNN seems not supported this feature. I tried to implement _convert_conv2d_transpose, itself seems work but input shape is not matched. Maybe Concatenate layers are ignored?

 % python ~/drive/webdnn/bin/convert_keras.py --input_shape '(1, 512, 512, 3)' --out webdnn_output atob_unet.h5
Using TensorFlow backend.
[convert_keras.py] Generating feedforward graph
(snip)
/opt/p36/lib/python3.6/site-packages/webdnn-1.1.0-py3.6.egg/webdnn/util/console.
py:27: Warning: [KerasConverter] omitting dropout
  warnings.warn(message, category)
Traceback (most recent call last):
  File "/home/knok/drive/webdnn/bin/convert_keras.py", line 113, in <module>
    main()
  File "/home/knok/drive/webdnn/bin/convert_keras.py", line 63, in main
    graph = converter.convert(model)
  File "/opt/p36/lib/python3.6/site-packages/webdnn-1.1.0-py3.6.egg/webdnn/frontend/keras/converter.py", line 108, in convert
    self._convert_operator(node.outbound_layer)
  File "/opt/p36/lib/python3.6/site-packages/webdnn-1.1.0-py3.6.egg/webdnn/frontend/keras/converter.py", line 141, in _convert_operator
    return super(KerasConverter, self)._convert_operator(k_op)
  File "/opt/p36/lib/python3.6/site-packages/webdnn-1.1.0-py3.6.egg/webdnn/frontend/converter.py", line 108, in _convert_operator
    self._handler_map[self.__class__.__name__][operator_key](self, operator)
  File "/opt/p36/lib/python3.6/site-packages/webdnn-1.1.0-py3.6.egg/webdnn/frontend/keras/layers/convolutional.py", line 92, in _convert_conv2d_transpose
    y, = Deconvolution2D(None, ksize=ksize, stride=stride, padding=padding)(x, w) 
  File "/opt/p36/lib/python3.6/site-packages/webdnn-1.1.0-py3.6.egg/webdnn/graph/operators/deconvolution2d.py", line 62, in __call__
    "Input and Kernel variables of Deconvolution2D must be same channel size: " \
AssertionError: Input and Kernel variables of Deconvolution2D must be same channel size: x.shape_dict[Axis.C]=1024, w.shape_dict[Axis.C]=512

The following is the patch for Conv2DTranspose:

--- a/src/graph_transpiler/webdnn/frontend/keras/layers/convolutional.py
+++ b/src/graph_transpiler/webdnn/frontend/keras/layers/convolutional.py
@@ -6,6 +6,7 @@ except ImportError as e:
 from webdnn.frontend.keras.converter import KerasConverter
 from webdnn.frontend.keras.layers.util import do_activation
 from webdnn.graph.operators.convolution2d import Convolution2D
+from webdnn.graph.operators.deconvolution2d import Deconvolution2D
 from webdnn.graph.operators.zero_padding_1d import ZeroPadding1D
 from webdnn.graph.operators.zero_padding_2d import ZeroPadding2D
 from webdnn.graph.order import OrderC, OrderNCHW, OrderNHWC, OrderHWCN, OrderNT
C
@@ -65,8 +66,36 @@ def _convert_separable_conv2d(converter: KerasConverter, k_op
: "keras.layers.Sep
 # noinspection PyUnusedLocal
 @KerasConverter.register_handler("Conv2DTranspose")
 def _convert_conv2d_transpose(converter: KerasConverter, k_op: "keras.layers.Co
nv2DTranspose"):
-    # TODO
-    raise NotImplementedError('[KerasConverter] keras.layers.Conv2DTranspose is
 not supported')
+    x = converter.get_variable(converter.get_input_tensor(k_op)[0])
+    if k_op.data_format == "channels_first":
+        assert x.order == OrderNCHW
+
+    elif k_op.data_format == "channels_last":
+        assert x.order == OrderNHWC
+
+    else:
+        raise ValueError(f"[KerasConverter] Unknown data format is detected: {k
_op.data_format}")
+    
+    w = converter.convert_to_constant_variable(k_op.kernel, OrderHWCN)
+
+    ksize = tuple(k_op.kernel_size)
+    stride = tuple(k_op.strides)
+    if k_op.padding == "valid":
+        padding = (0, 0)
+
+    elif k_op.padding == "same":
+        padding = (ksize[0] // 2, ksize[1] // 2)
+
+    w = converter.convert_to_constant_variable(k_op.kernel, OrderHWCN)
+
+    ksize = tuple(k_op.kernel_size)
+    stride = tuple(k_op.strides)
+    if k_op.padding == "valid":
+        padding = (0, 0)
+
+    elif k_op.padding == "same":
+        padding = (ksize[0] // 2, ksize[1] // 2)
+
+    else:
+        raise ValueError(f"[KerasConverter] Unknown padding: {k_op.padding}")
+
+    y, = Deconvolution2D(None, ksize=ksize, stride=stride, padding=padding)(x, 
w)
+    if k_op.use_bias:
+        b = converter.convert_to_constant_variable(k_op.bias, OrderC)
+        y = y + b
+
+    y = do_activation(k_op.activation, y)
+    converter.set_variable(converter.get_output_tensor(k_op)[0], y)
 
 
 # noinspection PyUnusedLocal

I am following the instructions to convert a resnet50 model word by word. And I get this:

D:\repos\webdnn\resnet>python ..\bin\convert_keras.py resnet50.h5 --input_shape "(1,224,224,3)" --out output --backend webgl
C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
[convert_keras.py] Generating feedforward graph
2018-09-22 07:53:38.725911: I C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
Traceback (most recent call last):
  File "..\bin\convert_keras.py", line 114, in <module>
    main()
  File "..\bin\convert_keras.py", line 61, in main
    model = keras.models.load_model(args.kerasmodel, custom_objects=custom_objects, compile=False)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\saving.py", line 260, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\saving.py", line 334, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\layers\__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\utils\generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\network.py", line 1027, in from_config
    process_node(layer, node_data)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\network.py", line 986, in process_node
    layer(unpack_singleton(input_tensors), **kwargs)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\layers\core.py", line 878, in call
    output = self.activation(output)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\activations.py", line 29, in softmax
    return K.softmax(x)
  File "C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 3154, in softmax
    return tf.nn.softmax(x, axis=axis)
TypeError: softmax() got an unexpected keyword argument 'axis'

The model was saved using these instructions:

(base) D:\repos\webdnn\resnet>python
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from keras.applications import resnet50
C:\Users\jeff\AppData\Local\Continuum\anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
>>> model = resnet50.ResNet50(include_top=True, weights='imagenet')
2018-09-22 07:27:17.054697: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.2/resnet50_weights_tf_dim_ordering_tf_kernels.h5
102858752/102853048 [==============================] - 7s 0us/step
>>> model.save("resnet50.h5")
>>> exit()

Hi.

In my understanding, WebDNN model can be feeded several inputs at once if GPU has enough capacity. Is that right?

And if so, I want to know how many inputs I can feed efficiently on the target GPU. Is it possible?

Thanks.

Using TensorFlow backend.
Here To import all
[convert_keras.py] Generating feedforward graph
2017-11-28 22:22:25.563469: W C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-28 22:22:25.563555: W C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
  File "./webdnn/bin/convert_keras.py", line 117, in <module>
    main()
  File "./webdnn/bin/convert_keras.py", line 67, in main
    graph = converter.convert(model)
  File "E:\CI-Cor-Ready\ai\face-demo-work\webdnn\webdnn\src\graph_transpiler\webdnn\frontend\keras\converter.py", line 105, in convert
    return self._convert_fallback(model)
  File "E:\CI-Cor-Ready\ai\face-demo-work\webdnn\webdnn\src\graph_transpiler\webdnn\frontend\keras\converter.py", line 115, in _convert_fallback
    v.shape[0].value = self._batch_size
  File "E:\CI-Cor-Ready\ai\face-demo-work\webdnn\webdnn\src\graph_transpiler\webdnn\graph\placeholder.py", line 443, in value
    raise TypeError(f"Placeholder#value must be a int, not '{type(new_v)}'")
TypeError: Placeholder#value must be a int, not '<class 'webdnn.graph.placeholder.Placeholder'>'

When I run the command to generate the graph from Resnet example (python bin/convert_keras.py resnet50.h5 --input_shape '(1,224,224,3)' --out output) I get the following error:

Can you help me resolve this error?

I have a very simple very standard LeNet in pytorch and I can't convert it to webdnn. The pooling layer give an error with dilation attribute. even after removing them and leaving just convolution I get a similar error but this time from attribute transA.

File "", line 1, in runfile('/Users/alirezagoudarzi/github/Aya-s-Brain/WebDNN_Test/pytorch/train_model.py', wdir='/Users/alirezagoudarzi/github/Aya-s-Brain/WebDNN_Test/pytorch')

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile execfile(filename, namespace)

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "/Users/alirezagoudarzi/github/Aya-s-Brain/WebDNN_Test/pytorch/train_model.py", line 142, in graph = PyTorchConverter().convert(net, dummy_input)

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/webdnn-1.2.3-py3.6.egg/webdnn/frontend/pytorch/converter.py", line 92, in convert graph = ONNXConverter().convert(onnx.load(proto_path))

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/webdnn-1.2.3-py3.6.egg/webdnn/frontend/onnx/converter.py", line 94, in convert self._convert_operator(onnx_op)

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/webdnn-1.2.3-py3.6.egg/webdnn/frontend/onnx/converter.py", line 114, in _convert_operator super(ONNXConverter, self)._convert_operator(proto)

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/webdnn-1.2.3-py3.6.egg/webdnn/frontend/converter.py", line 117, in _convert_operator self._handler_map[self.class.name][operator_key](self, operator)

File "/Users/alirezagoudarzi/anaconda/envs/py36/lib/python3.6/site-packages/webdnn-1.2.3-py3.6.egg/webdnn/frontend/onnx/defs/math.py", line 370, in _convert_gemm y, = Tensordot(None, axes=(A.order.axes[0 if attrs["transA"].i else 1], B.order.axes[1 if attrs["transB"].i else 0]))(A, B)

KeyError: 'transA'

I'd like to know how to solve the following problem. I've tried a tutorial "Use with Keras Model" as shell command and python script, but both would result in an error like:

AttributeError: 'Model' object has no attribute 'nodes_by_depth'

I really appreciate your help.

I'm trying to convert the yolo_v2 model to webdnn but ran into an error with a Lambda layer - it seems the '_convert_lambda' function has not yet been implemented:

@KerasConverter.register_handler("Lambda")
def _convert_lambda(converter: KerasConverter, k_op: "keras.layers.Lambda"):
    # TODO
    raise NotImplementedError('[KerasConverter] keras.layers.Lambda is not supported')

Would you be able to implement this or suggest how we can implement it ourselves?

We're working on a project to showcase NN with JS using WedDNN for NodeConf Argentina on October 26th so would greatly appreciate a speedy response!

First, thanks for the amazing toolkit! :)

I have a two-class classifier on small grayscale images, trained with Keras and converted to run on WebGL with WebDNN. When I evaluate the model in Keras, it gives me the output [ 0.16141078 0.83858919]. But when I evaluate the model on the same data with WebDNN it gives me [-309.3228759765625, 169.71974182128906]. The values are ordered correctly (argmax still works), but the values are incorrect. I thought maybe they were pre-softmax values, but that doesn't seem correct.

Next, I thought that maybe the problem was that I trained my network on images with pixels in the range 0-1, but maybe WebDNN expects pixels in the range 0-255. So I tried adding the scale: [255] option when setting the input for the network. Changing this gives me [-294116294656, 666073825280], and now the predictions are no longer correct.

How can I interpret the output of the network, and convert these values to the same predictions I get from Keras?

Thanks!

Bumps json5 from 2.2.0 to 2.2.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps loader-utils from 2.0.0 to 2.0.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps minimatch from 3.0.4 to 3.1.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps terser from 5.7.0 to 5.14.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps protobufjs from 6.11.2 to 6.11.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps minimist from 1.2.5 to 1.2.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.