训练word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基于给定的语料库训练词向量\n",
    "在此项目中，我们训练的词向量算法是 skip-gram + negative sampling，并且使用开源的工具来训练模型。 训练词向量的工具很多，但比较高效的词向量底层一般使用c,或者c++来编写。在这里推荐几个训练词向量的工具：\n",
    "\n",
    "https://github.com/danielfrg/word2vec\n",
    "\n",
    "https://github.com/dav/word2vec\n",
    "\n",
    "https://radimrehurek.com/gensim/models/word2vec.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 阅读源代码：建议阅读原版的c代码（上面头两个链接）, 对照视频中的推导来看，重点关注：\n",
    "   - 正样本/负样本如何获取？\n",
    "   - 梯度下降法\n",
    "   - negative sampling怎么采样负样本的？ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 在本机训练词向量，并把结果存入\"embedding.txt\"文件里。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 利用tsne来plot一下词向量，看看是否学出来的词向量比较合理 （https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 给定一个单词，找出跟它意思相近的单词； （similarity measure)，看看是否结果合理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import word2vec\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_list = pd.read_csv('data/essays_and_scores.csv', encoding = 'latin-1')\n",
    "f = open('essay.txt','a')\n",
    "for data in data_list['essay']:\n",
    "    data=data.replace(',','')\n",
    "    data=data.replace('.','')\n",
    "    data=data.replace(':','')\n",
    "    data=data.replace('!','')\n",
    "    data=data.replace('?','')\n",
    "    data=data.replace('/','')\n",
    "    data=data.replace(';','')\n",
    "    data=data.replace(\"'\",'')\n",
    "    data=data.replace('\"','')\n",
    "\n",
    "    f.write(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting training using file essay.txt\n",
      "Words processed: 2800K     Vocab size: 615K  \n",
      "Vocab size (unigrams + bigrams): 352958\n",
      "Words in train file: 2875230\n",
      "Words written: 2800K\r"
     ]
    }
   ],
   "source": [
    "# 连词处理\n",
    "word2vec.word2phrase('essay.txt', 'essay-phrases.txt', verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting training using file essay-phrases.txt\n",
      "Vocab size: 13515\n",
      "Words in train file: 2697512\n",
      "Alpha: 0.000002  Progress: 100.21%  Words/thread/sec: 307.77k   1.68%  Words/thread/sec: 260.21k  c: 285.72k   288.16k  ds/thread/sec: 296.22k  .44k  s: 20.85%  Words/thread/sec: 305.58k  0.018983  Progress: 24.15%  Words/thread/sec: 304.40k  ad/sec: 305.19k  31.80%  Words/thread/sec: 305.68k   0.016131  Progress: 35.56%  Words/thread/sec: 305.57k  ogress: 40.09%  Words/thread/sec: 306.35k  s: 44.04%  Words/thread/sec: 304.66k  : 47.47%  Words/thread/sec: 304.96k  k  21%  Words/thread/sec: 305.51k  gress: 59.48%  Words/thread/sec: 305.34k  read/sec: 305.88k  ogress: 69.69%  Words/thread/sec: 306.08k  06340  Progress: 74.72%  Words/thread/sec: 306.04k  36k  78  Progress: 84.57%  Words/thread/sec: 305.98k  02642  Progress: 89.51%  Words/thread/sec: 306.01k  ad/sec: 306.05k  "
     ]
    }
   ],
   "source": [
    "# 训练模型\n",
    "word2vec.word2vec('essay-phrases.txt', 'essay.bin', size=100, verbose=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting training using file essay.txt\n",
      "Vocab size: 11322\n",
      "Words in train file: 2811899\n",
      "Alpha: 0.000002  Progress: 100.16%  Words/thread/sec: 312.84k  1  Progress: 1.59%  Words/thread/sec: 278.57k  ead/sec: 296.45k  ds/thread/sec: 299.51k  21173  Progress: 15.38%  Words/thread/sec: 302.00k    Words/thread/sec: 309.09k  ad/sec: 307.10k  pha: 0.018440  Progress: 26.32%  Words/thread/sec: 307.68k  c: 308.32k  7.94k  : 0.015916  Progress: 36.41%  Words/thread/sec: 308.79k  Words/thread/sec: 309.33k  ead/sec: 309.90k  ss: 47.66%  Words/thread/sec: 310.24k  lpha: 0.012220  Progress: 51.19%  Words/thread/sec: 310.77k  54.74%  Words/thread/sec: 309.99k  58.20%  Words/thread/sec: 309.63k  lpha: 0.009224  Progress: 63.18%  Words/thread/sec: 310.73k  0.008056  Progress: 67.85%  Words/thread/sec: 311.56k  d/sec: 311.57k  ords/thread/sec: 311.18k  8  Progress: 85.29%  Words/thread/sec: 312.07k   Words/thread/sec: 312.22k  : 312.09k    Progress: 100.24%  Words/thread/sec: 312.87k  "
     ]
    }
   ],
   "source": [
    "word2vec.word2clusters('essay.txt', 'embedding.txt', 100, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('sister', 0.9415132170786729),\n",
       " ('dad', 0.924006436570739),\n",
       " ('cousin', 0.9070727644861944),\n",
       " ('grandma', 0.9004683119666668),\n",
       " ('grandpa', 0.8807705976404732),\n",
       " ('mom', 0.8736161047641924),\n",
       " ('uncle', 0.8646622601922797),\n",
       " ('little_brother', 0.8562046505526084),\n",
       " ('brothers', 0.853756428416747),\n",
       " ('sisters', 0.8482749834046517)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = word2vec.load('essay.bin')\n",
    "indexes, metrics = model.similar(\"brother\")\n",
    "model.generate_response(indexes, metrics).tolist()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}