Key difference between Statistical/ML-based NLP and Knowledge-based/linguistics-based NLP
Stat NLP: speed and robustness are the main concerns KB NLP: Phenomena based Example: Boys, Toys, Toes To get the root remove “s” How about foxes, boxes, ladies Understand phenomena: go deeper Slower processing
Perspective on Statistical MT
What is a good translation? Faithful to source Fluent in target fluency faithfulness
Word-alignment example
(1) (2) (3) (4) Ram has an apple राम के पास एक सेब है (1) (2)(3) (4) (5) (6) Ram of near an apple is
Kinds of MT Systems (point of entry from source to the target text) fwd
Why is MT difficult? Classical NLP problems
Ambiguity Lexical: Went to the bank to withdraw money Structural: Saw the boy with a telescope Ellipsis: I wanted a book and John a pen Co-reference Anaphora: John said he likes music Hypernymic: John’s house is a robust structure
Why is MT Difficult Language Divergence
Lexico-Semantic Divergence Structural Divergence
Language Divergence (English Hindi: Noun to Adjective)
The demands on sportsmen today can lead to burnout at an early age. (noun – the state of being extremely tired or ill, either physically or mentally, because you have worked too hard) खिलाड़यों से जो आज अपेक्षाएं हैं, वे उन्हें कम उम्र में अक्रियाशील कर सकती हैं Sportsmen-from, which today demands exist, that (correlative) them early age in inactive do can (aspectual) V-AUX.
Language Divergence (English Hindi: Noun to Verb)
Every concert they gave us was a sell-out. (an event for which all the tickets have been sold) उनके हर संगीत-कार्यक्रम के सभी टिकट बिक गए थे। Their every concert-of all ticket sell-past-passive-plural (were sold out).
Language Divergence (English Hindi: Adjective to Adverb)
The children were watching in wide-eyed amazement. (with eyes fully open because of fear, great surprise, etc) बच्चे आश्चर्य से आँखें फाड़े देख रहे थे। Children amazement-with eyes opening widely seeing were.
Language Divergence (English Hindi: Adjective to Verb)
He was in a bad mood at breakfast and wasn't very communicative. (able and willing to talk and give information to other people) नाश्ते के समय वह खराब मूड में था और ज्यादा बात-चीत नहीं कर रहा था। Breakfast-of time he bad mood-in was and much conversation not do-past-progressive-sing (was doing).
Language Divergence (English Hindi: Preposition to Adverb)
It gets cooler toward evening. (near a point in time) शाम होते-होते ठंडक बढ़ जाती है। Evening happening-happening (reduplication; typical Indian language phenomenon) cold increase-goes (verb compound; polar vector).
Language Divergence (English Hindi: idiomatic usage)
Given her interest in children, teaching seems the right job for her. (when you consider sth) बच्चों के प्रति (में) उसकी दिलचस्पी देखते हुए, अध्यापन उसके लिए उचित लगता है। Children-towards her interest having seen, teaching for her appropriate seems.
Language Divergence is ubiquitous (Marathi-Hindi-English: case marking and postpositions transfer: works!)
Not only for languages from distant families, but also within close cousins प्रथम ताख्यात वर्तमान(simple present) तो जातो. वह जाता है। He goes. स्थिरसत्य(universal truth) पृथ्वी सूर्याभोवती फिरते. पृथ्वी सूर्य के चारों ओर घूमती है। The earth revolves round the sun.
Language Divergence (Marathi-Hindi-English: case marking and postpositions: works again!)
ऐतिहासिक सत्य (historical truth) कृष्ण अर्जुनास सांगतो... कृष्ण अर्जुन से कहते हैं... Krushna says to Arjuna… अवतरण (quoting) दामले म्हणतात, ... दामले कहते हैं, ... Damle says,...
Language Divergence (Marathi-Hindi-English: case marking and postpositions: does not work!)
संनिहित भूत (immediate past) कधी आलास? हा येतो इतकाच ! कब आये? बस अभी आया । When did you come? Just now (I came). निःसंशय भविष्य (certainty in future) आता तो मार खातो खास ! अब वह मार खायगा ही ! He is in for a thrashing. आश्वासन (assurance) मी तुम्हाला उद्या भेटतो. मैं आप से कल मिलता हूँ। I will see you tomorrow.
Language Divergence Theory: Lexico-Semantic Divergences (ref: Dave, Parikh, Bhattacharyya, Journal of MT, 2002)
Conflational divergence E: stab; H: churaa se maaranaa (knife-with hit) S: Utrymningsplan; E: escape plan Structural divergence E: SVO; H: SOV Categorial divergence Change is in POS category (many examples discussed) Head swapping divergence E: Prime Minister of India; H: bhaarat ke pradhaan mantrii (India-of Prime Minister) Lexical divergence E: advise; H: paraamarsh denaa (advice give): Noun Incorporation- very common Indian Language Phenomenon
Language Divergence Theory: Syntactic Divergences
Constituent Order divergence E: Singh, the PM of India, will address the nation today; H: bhaarat ke pradhaan mantrii, singh, … (India-of PM, Singh…) Adjunction Divergence E: She will visit here in the summer; H: vah yahaa garmii meM aayegii (she here summer-in will come) Preposition-Stranding divergence E: Who do you want to go with?; H: kisake saath aap jaanaa chaahate ho? (who with…) Null Subject Divergence E: I will go; H: jaauMgaa (subject dropped) Pleonastic Divergence E: It is raining; H: baarish ho rahii haai (rain happening is: no translation of it)
Comments