Unified model predicting same vector for unseen

[momalekabid]

Hello,
I trained the unified model with the following parameters on the WantWords/Hill train dataset which I preprocessed by vectorizing each word with the same word2vec model cited in the paper (example command):
python3 train_unified.py --do_train --train_file ../data/wwdata/train_w2v.json --dev_file ../data/wwdata/dev_w2v.json --device cuda:0 --target_arch sgns --save_dir unifiedsave

and then attempted to generate predictions from the trained model on the unseen dataset:
python3 train_unified.py --do_pred --test_file ../data/wwdata/unseen.json --pred_file w2vpreds/unifiedw2v_unseen.json --pred_direction embedding --save_dir unifiedsave --device cuda:0 --embedding_arch sgns

After scoring using the provided scoring function, I attempted to test the predicted vectors for reverse dictionary, and found that the vectors predicted are almost exactly the same for every word:


and thus querying an mse-based vector search for the predicted vector results in the same top ~10 words every time. Any idea where I've gone wrong/how to improve these results?

[PinzhenChen]

Hi, generally many things can happen. I think the easiest thing to check is your trained model's predictions on the "seen" split. While it is not a good benchmark to compare deep learning models as discussed in our paper, I think it is a good subset for sanity check because the test samples have been directly trained on.

Just for your information, here are the numbers from our run's result on the "seen" test split (which we omitted from our paper):

Model	median rank	accuracy@1/10/100	std
Transformer	68	.02/.15/.59	122
Unified	17	.12/.42/.84	86
+ share embed	19	.10/.39/.80	92