SNLP2019

Transcript

1. Probing for Semantic Classes: Diagnosing the Meaning Content of Word

   Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP  2019/09/28 
      <[email protected]>

2. Outline 2 H9P=O8 O8 "!PC   
    
   

    
    $@K !.S%,'3 F5T&
   *'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L  >#".S(rare senses)P=O8APM0 " 4D('% UG

3. Background 3 # " !Word2VecGrove  #"
   NLPIR* - ELMoBERT 

   $'+
   ) %" full-space , % &   L  "$( "   #
    &!

4. Background 4 # 
    "@!Semantic class
   >*( S-class , "@=6
   :-&C?4=6 

   B $8 SEMCAT, HyperLex
   … =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !

5. Background 5 #   Apple Apple
    Apple
   

   + Word embeddingsense embedding  Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSE
   sense embedding  Word embedding Sense embedding Sense embedding

6. WIKI-PSE Resource 6 
   Wikipedia .!2,$#  .! ()  

   S-class 113 FIGER types%+134"10 person/author
   person/politician… à person
   -S-class  
    
   
    *'.!/.! & - S-class0

7. WIKI-PSE Resource 7
    "$%  %2& +:! 343,000-6.-S-class#
   5'5000;-!% 75,.6,.83

   +*1! /4   44,250- 
    
   -S-class
     - S-class )0organization (9food @apple@ – food @apple@ – organization 
    

8.  
    8 Word embedding (word) @apple@ WIKI-PSE  

   word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization  @  word/S-class Uniform sum (unifΣ) !"
    Weighted sum (wghtΣ) !"
     # = % " !" #&" #
   Aggregated word embedding #&"
   Sense (word/S-class) embeddings word, unifΣ, wghtΣ

9. Experiments 9 Problem settings   SkipGram Structured SkipGram 
   

   WIKI-PSE    (LR) # 
   (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding  " 3. wghtΣ sense embedding ! $'& (" Word embedding

10. Experiments 10 Probing Task 1
    S-class Prediction  S-class  

    @apple@ +food ∩ +organization ∩ -event S-class  

11. Experiments 11 MLP > LR
    KNN 8&!S-class ."7;5   

     KNN )#, *:3,
    <(3-/ unifΣ > word > wghtΣ 'rare sense 0$  97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction
    F11 4

12. Experiments 12 Probing Task 1S-class Prediction  
     

     unifΣ  $
        #rare sense    (13,000)
    F1! "

13. Experiments 13 Probing Task 1
    S-class Prediction * #4(!  Recall

    ) S-class&  sense embedding+%3 Rare S-class/"12 %"0, .$- Recall     Dominance S-class'
    Recall 
    

14. Experiments 14 Probing Task 1
    S-class Prediction /&)5. !"$'  ,(20-

       3+1(#41(       Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality
    Recall
    

15. Experiments 15  "    Probing Task 2
    Ambiguity

    Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN    à    %
    FREQUENCY(Baseline)  # &LR

16. NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <--

    Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ  ( !+  entity mention  MC ), CR
    MR $)& SUBJ 
    %*' MRPC "   S-class 

17. Summary & Conclusion 17 * $2"(3 WIKI-PSE'.
    1)/& 0+$2/&

    1, 1)/&%  # 0+$2- a a e Rare sense e harder NLP Rare sense !# e