turgut090 / bert-from-r Goto Github PK
View Code? Open in Web Editor NEWBidirectional Encoder Representations from Transformers
bert-from-r's People
Contributors
Stargazers
Watchers
bert-from-r's Issues
Bert R Session Aborted - encountered a fatal error
Hi @henry090 ,
I've managed to progress right up to the end of your script, and it ended with a fatal error on both my laptop and desktop. the R-application had to be restarted. The last bit of the script is as below before it ended with an error.
model %>% compile(
+ k_bert$AdamWarmup(decay_steps=decay_steps,
+ warmup_steps=warmup_steps, lr=learning_rate),
+ loss = 'binary_crossentropy',
+ metrics = 'accuracy'
+ )
>
> model %>% fit(
+ concat,
+ targets,
+ epochs=epochs,
+ batch_size=bch_size, validation_split=0.2)
2020-02-18 09:05:12.027095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-02-18 09:05:22.722361: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 41.02MiB (rounded to 43008000). Current allocation summary follows.
2020-02-18 09:05:22.722808: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256): Total Chunks: 1325, Chunks in use: 1313. 331.3KiB allocated for chunks. 328.3KiB in use in bin. 5.9KiB client-requested in use in bin.
2020-02-18 09:05:22.723130: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512): Total Chunks: 10, Chunks in use: 1. 5.3KiB allocated for chunks. 512B in use in bin. 280B client-requested in use in bin.
2020-02-18 09:05:22.723450: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.3KiB allocated for chunks. 1.3KiB in use in bin. 1.0KiB client-requested in use in bin.
2020-02-18 09:05:22.723760: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048): Total Chunks: 577, Chunks in use: 577. 1.70MiB allocated for chunks. 1.70MiB in use in bin. 1.70MiB client-requested in use in bin.
2020-02-18 09:05:22.724149: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096): Total Chunks: 3, Chunks in use: 3. 17.0KiB allocated for chunks. 17.0KiB in use in bin. 12.0KiB client-requested in use in bin.
2020-02-18 09:05:22.724604: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192): Total Chunks: 67, Chunks in use: 67. 816.3KiB allocated for chunks. 816.3KiB in use in bin. 815.7KiB client-requested in use in bin.
2020-02-18 09:05:22.725115: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384): Total Chunks: 2, Chunks in use: 1. 45.5KiB allocated for chunks. 24.0KiB in use in bin. 13.7KiB client-requested in use in bin.
2020-02-18 09:05:22.725549: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.725952: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.726346: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072): Total Chunks: 22, Chunks in use: 22. 3.46MiB allocated for chunks. 3.46MiB in use in bin. 3.41MiB client-requested in use in bin.
2020-02-18 09:05:22.726827: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.727121: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.727633: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.728034: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152): Total Chunks: 247, Chunks in use: 245. 555.40MiB allocated for chunks. 551.25MiB in use in bin. 551.25MiB client-requested in use in bin.
2020-02-18 09:05:22.728483: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304): Total Chunks: 1, Chunks in use: 1. 4.04MiB allocated for chunks. 4.04MiB in use in bin. 2.25MiB client-requested in use in bin.
2020-02-18 09:05:22.728857: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608): Total Chunks: 193, Chunks in use: 192. 1.77GiB allocated for chunks. 1.76GiB in use in bin. 1.74GiB client-requested in use in bin.
2020-02-18 09:05:22.729147: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.729440: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432): Total Chunks: 14, Chunks in use: 14. 574.22MiB allocated for chunks. 574.22MiB in use in bin. 574.22MiB client-requested in use in bin.
2020-02-18 09:05:22.729660: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864): Total Chunks: 1, Chunks in use: 1. 76.85MiB allocated for chunks. 76.85MiB in use in bin. 41.02MiB client-requested in use in bin.
2020-02-18 09:05:22.729865: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.730065: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-02-18 09:05:22.730301: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 41.02MiB was 32.00MiB, Chunk State:
2020-02-18 09:05:22.730435: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 3172515840
2020-02-18 09:05:22.730541: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02600000 next 1 of size 1280
2020-02-18 09:05:22.730657: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02600500 next 5 of size 3328
2020-02-18 09:05:22.730774: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02601200 next 4 of size 3072
2020-02-18 09:05:22.731356: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02601E00 next 7 of size 3072
2020-02-18 09:05:22.731539: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02602A00 next 10 of size 3072
2020-02-18 09:05:22.731710: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02603600 next 12 of size 3072
2020-02-18 09:05:22.731908: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02604200 next 14 of size 3072
2020-02-18 09:05:22.732090: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02604E00 next 16 of size 3072
2020-02-18 09:05:22.732314: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02605A00 next 23 of size 3072
2020-02-18 09:05:22.732507: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02606600 next 24 of size 3072
2020-02-18 09:05:22.732692: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000C02607200 next 25 of size 3072
2020-02-18 09:05:22.732918: I tensorflow/core/common_ru
The full script which i ran prior to the above
> knitr::opts_chunk$set(echo = TRUE, eval = FALSE)
> Sys.setenv(TF_KERAS=1)
> # to see python version
> reticulate::py_config()
python: C:/Users/jtseng/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython: C:/Users/jtseng/AppData/Local/r-miniconda/envs/r-reticulate/python36.dll
pythonhome: C:/Users/jtseng/AppData/Local/r-miniconda/envs/r-reticulate
version: 3.6.10 |Anaconda, Inc.| (default, Jan 7 2020, 15:18:16) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/Users/jtseng/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version: 1.17.5
> reticulate::py_module_available('keras_bert')
2020-02-18 09:08:54.469027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
[1] TRUE
> tensorflow::tf_version()
[1] ‘2.0’
>
> pretrained_path = './data/uncased_L-12_H-768_A-12'
> config_path = file.path(pretrained_path, 'bert_config.json')
> checkpoint_path = file.path(pretrained_path, 'bert_model.ckpt')
> vocab_path = file.path(pretrained_path, 'vocab.txt')
>
> library(reticulate)
Warning message:
package ‘reticulate’ was built under R version 3.6.2
> k_bert = import('keras_bert')
> token_dict = k_bert$load_vocabulary(vocab_path)
> tokenizer = k_bert$Tokenizer(token_dict)
>
> seq_length = 50L
> bch_size = 70
> epochs = 1
> learning_rate = 1e-4
> DATA_COLUMN = 'comment_text'
> LABEL_COLUMN = 'target'
> model = k_bert$load_trained_model_from_checkpoint(
+ config_path,
+ checkpoint_path,
+ training=T,
+ trainable=T,
+ seq_len=seq_length)
2020-02-18 09:09:03.108824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-02-18 09:09:03.139396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Quadro M2200 major: 5 minor: 2 memoryClockRate(GHz): 1.036
pciBusID: 0000:01:00.0
2020-02-18 09:09:03.139785: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-02-18 09:09:03.140463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-02-18 09:09:03.141118: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-02-18 09:09:03.143718: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Quadro M2200 major: 5 minor: 2 memoryClockRate(GHz): 1.036
pciBusID: 0000:01:00.0
2020-02-18 09:09:03.144277: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-02-18 09:09:03.145173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-02-18 09:09:03.718561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-18 09:09:03.718792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-02-18 09:09:03.718932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-02-18 09:09:03.719707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3025 MB memory) -> physical GPU (device: 0, name: Quadro M2200, pci bus id: 0000:01:00.0, compute capability: 5.2)
> # tokenize text
> tokenize_fun = function(dataset) {
+ c(indices, target, segments) %<-% list(list(),list(),list())
+ for ( i in 1:nrow(dataset)) {
+ c(indices_tok, segments_tok) %<-% tokenizer$encode(dataset[[DATA_COLUMN]][i],
+ max_len=seq_length)
+ indices = indices %>% append(list(as.matrix(indices_tok)))
+ target = target %>% append(dataset[[LABEL_COLUMN]][i])
+ segments = segments %>% append(list(as.matrix(segments_tok)))
+ }
+ return(list(indices,segments, target))
+ }
> # read data
> dt_data = function(dir, rows_to_read){
+ data = data.table::fread(dir, nrows=rows_to_read)
+ c(x_train, x_segment, y_train) %<-% tokenize_fun(data)
+ return(list(x_train, x_segment, y_train))
+ }
> library(keras)
> c(x_train,x_segment, y_train) %<-%
+ dt_data('./data/jigsaw-unintended-bias-in-toxicity-classification/train.csv',2000)
> train = do.call(cbind,x_train) %>% t()
> segments = do.call(cbind,x_segment) %>% t()
> targets = do.call(cbind,y_train) %>% t()
> concat = c(list(train ),list(segments))
> c(decay_steps, warmup_steps) %<-% k_bert$calc_train_steps(
+ targets %>% length(),
+ batch_size=bch_size,
+ epochs=epochs
+ )
> library(keras)
> input_1 = get_layer(model,name = 'Input-Token')$input
> input_2 = get_layer(model,name = 'Input-Segment')$input
> inputs = list(input_1,input_2)
> dense = get_layer(model,name = 'NSP-Dense')$output
> outputs = dense %>% layer_dense(units=1L, activation='sigmoid',
+ kernel_initializer=initializer_truncated_normal(stddev = 0.02),
+ name = 'output')
> model = keras_model(inputs = inputs,outputs = outputs)
> model
Model
Model: "model_1"
________________________________________________________________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
========================================================================================================================================================
Input-Token (InputLayer) [(None, 50)] 0
________________________________________________________________________________________________________________________________________________________
Input-Segment (InputLayer) [(None, 50)] 0
________________________________________________________________________________________________________________________________________________________
Embedding-Token (TokenEmbedding) [(None, 50, 768), (30522, 768)] 23440896 Input-Token[0][0]
________________________________________________________________________________________________________________________________________________________
Embedding-Segment (Embedding) (None, 50, 768) 1536 Input-Segment[0][0]
________________________________________________________________________________________________________________________________________________________
Embedding-Token-Segment (Add) (None, 50, 768) 0 Embedding-Token[0][0]
Embedding-Segment[0][0]
________________________________________________________________________________________________________________________________________________________
Embedding-Position (PositionEmbedding) (None, 50, 768) 38400 Embedding-Token-Segment[0][0]
________________________________________________________________________________________________________________________________________________________
Embedding-Dropout (Dropout) (None, 50, 768) 0 Embedding-Position[0][0]
________________________________________________________________________________________________________________________________________________________
Embedding-Norm (LayerNormalization) (None, 50, 768) 1536 Embedding-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Embedding-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-1-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Embedding-Norm[0][0]
Encoder-1-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-1-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-1-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-1-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-1-MultiHeadSelfAttention-Norm[0][0]
Encoder-1-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-1-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-1-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-1-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-2-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-1-FeedForward-Norm[0][0]
Encoder-2-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-2-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-2-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-2-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-2-MultiHeadSelfAttention-Norm[0][0]
Encoder-2-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-2-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-2-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-2-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-3-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-2-FeedForward-Norm[0][0]
Encoder-3-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-3-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-3-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-3-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-3-MultiHeadSelfAttention-Norm[0][0]
Encoder-3-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-3-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-3-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-3-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-4-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-3-FeedForward-Norm[0][0]
Encoder-4-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-4-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-4-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-4-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-4-MultiHeadSelfAttention-Norm[0][0]
Encoder-4-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-4-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-4-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-4-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-5-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-4-FeedForward-Norm[0][0]
Encoder-5-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-5-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-5-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-5-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-5-MultiHeadSelfAttention-Norm[0][0]
Encoder-5-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-5-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-5-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-5-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-6-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-5-FeedForward-Norm[0][0]
Encoder-6-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-6-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-6-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-6-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-6-MultiHeadSelfAttention-Norm[0][0]
Encoder-6-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-6-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-6-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-6-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-7-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-6-FeedForward-Norm[0][0]
Encoder-7-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-7-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-7-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-7-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-7-MultiHeadSelfAttention-Norm[0][0]
Encoder-7-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-7-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-7-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-7-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-8-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-7-FeedForward-Norm[0][0]
Encoder-8-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-8-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-8-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-8-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-8-MultiHeadSelfAttention-Norm[0][0]
Encoder-8-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-8-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-8-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttention (MultiHeadAttent (None, 50, 768) 2362368 Encoder-8-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttention-Dropout (Dropout (None, 50, 768) 0 Encoder-9-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-8-FeedForward-Norm[0][0]
Encoder-9-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttention-Norm (LayerNorma (None, 50, 768) 1536 Encoder-9-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-9-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-9-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-9-MultiHeadSelfAttention-Norm[0][0]
Encoder-9-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-9-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-9-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttention (MultiHeadAtten (None, 50, 768) 2362368 Encoder-9-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttention-Dropout (Dropou (None, 50, 768) 0 Encoder-10-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-9-FeedForward-Norm[0][0]
Encoder-10-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttention-Norm (LayerNorm (None, 50, 768) 1536 Encoder-10-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-10-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-10-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-10-MultiHeadSelfAttention-Norm[0][0]
Encoder-10-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-10-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-10-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttention (MultiHeadAtten (None, 50, 768) 2362368 Encoder-10-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttention-Dropout (Dropou (None, 50, 768) 0 Encoder-11-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-10-FeedForward-Norm[0][0]
Encoder-11-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttention-Norm (LayerNorm (None, 50, 768) 1536 Encoder-11-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-11-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-11-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-11-MultiHeadSelfAttention-Norm[0][0]
Encoder-11-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-11-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-11-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttention (MultiHeadAtten (None, 50, 768) 2362368 Encoder-11-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttention-Dropout (Dropou (None, 50, 768) 0 Encoder-12-MultiHeadSelfAttention[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttention-Add (Add) (None, 50, 768) 0 Encoder-11-FeedForward-Norm[0][0]
Encoder-12-MultiHeadSelfAttention-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttention-Norm (LayerNorm (None, 50, 768) 1536 Encoder-12-MultiHeadSelfAttention-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-FeedForward (FeedForward) (None, 50, 768) 4722432 Encoder-12-MultiHeadSelfAttention-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-FeedForward-Dropout (Dropout) (None, 50, 768) 0 Encoder-12-FeedForward[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-FeedForward-Add (Add) (None, 50, 768) 0 Encoder-12-MultiHeadSelfAttention-Norm[0][0]
Encoder-12-FeedForward-Dropout[0][0]
________________________________________________________________________________________________________________________________________________________
Encoder-12-FeedForward-Norm (LayerNormalization) (None, 50, 768) 1536 Encoder-12-FeedForward-Add[0][0]
________________________________________________________________________________________________________________________________________________________
Extract (Extract) (None, 768) 0 Encoder-12-FeedForward-Norm[0][0]
________________________________________________________________________________________________________________________________________________________
NSP-Dense (Dense) (None, 768) 590592 Extract[0][0]
________________________________________________________________________________________________________________________________________________________
output (Dense) (None, 1) 769 NSP-Dense[0][0]
========================================================================================================================================================
Total params: 109,128,193
Trainable params: 109,128,193
Non-trainable params: 0
________________________________________________________________________________________________________________________________________________________
>
> model %>% compile(
+ k_bert$AdamWarmup(decay_steps=decay_steps,
+ warmup_steps=warmup_steps, lr=learning_rate),
+ loss = 'binary_crossentropy',
+ metrics = 'accuracy'
+ )
Library for %<-% needs to be called
Script needs to load
library(zeallot)
In order for multiple assignment operator (%<-%) used in script to work
multi-label classification
Hi there, thanks for the tutorial here: https://blogs.rstudio.com/ai/posts/2019-09-30-bert-r/ It's very useful! I wonder if you have one on multi-label classification? (For essentially the same dataset?) Or some code that helps me do that? Thank you very much in advance!
Error in object$get_layer(name = name, index = index)
Hi,
Thank you very much for this tutorial! I have managed to proceed to the step "Determine inputs and outputs, then concatenate them" where I receive the error below. Any suggestions on what is wrong?
Cheers in advance!
Josef
library(keras)
input_1 = get_layer(model,name = 'Input-Token')$input
input_2 = get_layer(model,name = 'Input-Segment')$input
inputs = list(input_1,input_2)
dense = get_layer(model,name = 'NSP-Dense')$output
outputs = dense %>% layer_dense(units=1L, activation='sigmoid',
kernel_initializer=initializer_truncated_normal(stddev = 0.02),
name = 'output')
model = keras_model(inputs = inputs,outputs = outputs)
> input_1 = get_layer(model,name = 'Input-Token')$input
Error in object$get_layer(name = name, index = index) :
attempt to apply non-function
Can't seem to install keras-bert
Ran this in mini-conda prompt. What am I missing?
(base) C:\Users\james>conda install keras-bert
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- keras-bert
Current channels:
- https://repo.anaconda.com/pkgs/main/win-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/win-64
- https://repo.anaconda.com/pkgs/r/noarch
- https://repo.anaconda.com/pkgs/msys2/win-64
- https://repo.anaconda.com/pkgs/msys2/noarch
- https://conda.anaconda.org/conda-forge/win-64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Combining BERT and structured data
Hi Turgut, thank you very much for your great blog article (https://blogs.rstudio.com/ai/posts/2019-09-30-bert-r/). It is very interesting and helpful! I was wondering how it is possible to use the output of the BERT model (e.g. the dense vector/ linear layer before the softmax) as a predictor for another model (i.e. Random Forest). The goal is that the random forest can learn from both the BERT output and the structured data in the given dataset (e.g. Likes, Disagree, etc.). Could you show how to get this output instead of probabilities from BERT?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.