06-data-cleaning-occupy-central-news.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "# 对占中新闻进行数据清洗\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:32:01.554170Z",
     "start_time": "2023-11-17T07:31:59.763984Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 使用with open读取每一行数据\n",
    "with open(\"./data/occupycentral/zz-hk-2014-10.rtf\", encoding = 'gb18030') as f:\n",
    "    news = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:32:22.211628Z",
     "start_time": "2023-11-17T07:32:22.208079Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16541"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看总共有多少行\n",
    "len(news)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-16T02:28:17.979124Z",
     "start_time": "2021-05-16T02:28:17.968422Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~~~~~~~~~~~~~~~~  #1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "am730 | 2014-10-31 \\par\n",
      "\n",
      "A16| NEWS| C观点| By 施永青 \\par \n",
      "\n",
      "\n",
      "\n",
      "法治有整合社会功能 \\par\n",
      "\n",
      "\\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\\par\\par 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能，今天续谈Integration(整合)的功能。人类社会是一个复杂系统，大系统内还有很多次系统，这些系统既独立，又牵连；既矛盾，又互适；有时互相促进，有时互相制约。正如一个人，既是社会的成员，又是公司的雇员；既是别人的儿子，又是朋辈中的「大哥」；其言行会同时受所处的不同系统所影响。\\par\\par 这些错综复杂的系统之间不可能没有冲突，为了避免社会因纠纷得不到恰当的处理而分崩离析，社会必须发展出一套整合矛盾的方式，这就需要有司法制度。\\par\\par 按帕森斯的说法，法治的基础是先要界定产权。这样才能避免因争夺资源而产生无休止的冲突。再者，交易亦需要在产权获得界定后才能进行。有交易才有市场，才能透过市场机制进行公平竞争，推动经济发展。\\par\\par 另一方面，社会亦需要为人权下定义，这样，政府才能在有认受性的情况下组成，才能有效地去处理公众事务。此外，社会还需要有一套合乎公义的会议程序，以决定如何汇聚众人的意愿。\\par\\par 有了这些基础之后，社会就可以发展出一整套司法制度，让成员知所行止，令社会的矛盾不会恶化。\\par\\par 英国人为香港留下的，可不只是一套可以依据的律例，而是一整套法治的理念与司法程序。香港的回归能进行得相对平稳，与特区政府基本上原封不动地承继了原有的司法系统有莫大的关系。\\par\\par 回归后，虽有人危言耸听，说香港的法治已死，但世人仍公认香港的法治达国际一流水准，而港人亦可以如常在香港生活与做生意，不觉有失去法治的实质威胁。直到\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 运动的出现，香港人才真正感受到失去法治的害处。\\par\\par \\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 运动挑战的可不只是个别「恶法」，而是侵犯了整个法治的根基——产权、人权与政府的执法权。\\par\\par 占领区的物业，地契上列明有Right of way，但现在占领者却不容停车场的车辆出入。这分明损害了这些物业的产权。现在政府却无法加以维护；法庭出了禁制令，\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 者却一样藐视。这样发展下去，谁敢在香港置业？\\par\\par 其实，损害产权等同损害人权，因为人权的一项重要内容，就是个人的财产应获保障。此外，人人都应有追求幸福的权利，但现在占领区生意难做，怎会不妨碍别人追求幸福？\\par\\par \\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 者把自己的行为说成是公民抗命，但公民抗命只是个人行为在道德上的解释，用来拒绝遵守某些个人不认同的法令还讲得通，但绝不可以借此损害他人的产权与人权。\\par\\par 再者，\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 者现时在争取的是宪政改革，本应获社会上绝大多数人赞同才有条件实施，不宜用占领交通要津的方式去逼其他人就范。如果祭起公民抗命的旗帜就可以欲所欲为，只会天下大乱，令法治失去协调与整合作用。 \\par\n",
      "\n",
      " \\par\n",
      "\n",
      "\n",
      "\n",
      "文章编号: 201410315301186 \\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "本内容之版权由相关传媒机构 / 版权持有人拥有。除非获得明确授权，否则严禁复制、改编、分发或发布本内容。版权持有人保留一切权利。 本内容经慧科的电子服务提供。\n",
      "\n",
      "\\page\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~  #2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "大公报 | 2014-10-31 \\par\n",
      "\n",
      "A08| 要闻 \\par \n",
      "\n",
      "\n",
      "\n",
      "扛工仔料加薪4.5% 建筑金融好景 \\par\n",
      "\n",
      "\\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\\par\\par 【大公报讯】记者彩雯报道：上班族明年加薪预测调查结果陆续揭盅，香港人力资源管理学会昨日公布调查预计，明年平均加薪4.5%，高过今年的4.3%加幅，也高过雇主联会建议的4%平均加薪幅度，年尾花红金额平均相当于一至两个月薪金，预料建筑业及金融服务业雇员明年「最好景」，预测加薪幅度最高。不过， 「\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 」持续逾一个月以及沪港通押后的影响，尚未反映在今次调查结果，有待今年底的跟进调查才能确定。\\par\\par 人力资源管理学会向101间公司调查，涉及聘用雇员总数达13.7万人。当中84间公司于今年有整体加薪，两间冻薪，今年平均加薪4.3%。展望明年，80间公司打算明年一至四月将调整基本薪金，其中52间更已落实明年预算调整幅度，当中，50间公司预期会整体加薪，两间公司冻薪，没公司打算减薪，预计明年整体加薪4.5%。调整薪酬主要考虑公司业绩、个人工作表现及通胀等因素。\\par\\par 至于年底花红，设有固定花红制度的47间公司，近40%雇员符合发放固定花红的资格，平均金额为1.01个月基本薪金。设有非固定花红的71间公司中，68%雇员获发非固定花红，平均金额为1.62个月薪金，去年有超过70%雇员获发非固定花红，平均金额为1.58个月。\\par\\par 各行业当中，金融服务业雇员可望「最好景」，调查预测今年底的非固定花红金额相当于6.3个月薪金，明年可望加薪5.9%。建筑业雇员明年加薪也可望达5.9%，但预测今年底非固定花红金额只有大约1.4个月薪金。零售业今年底非固定花红金额可望达4.72个月薪金，但明年加薪幅度预测只有4.2%，低过整体4.5%的加幅。\\par\\par 学会：调查未计\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 \\par\\par 人力资源管理学会前任会长莫家麟表示，受访公司只计算今年一至九月份的薪金调整，当时未发生「\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 」行动， 「\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 」行动已踏入第二个月，影响情况陆续浮现，但未知行动何时结束，难以预计影响的深度，他亦不便揣测。他指，沪港通会对香港带来好处，但现时通车无期，可能会将好处延后，而香港未来亦受很多不明朗因素影响。\\par\\par 香港人才管理协会与浸会大学工商管理学院早前公布薪酬调查显示，过去一年香港地区的薪酬加幅为4.2%至4.5%，由于经济情况未明朗，明年加薪幅度或放缓为4%至4.5%。雇主联会近日向逾500家企业发出加薪指引，建议明年平均加薪4%，其中建筑、医疗保健行业加幅较高。 \\par\n",
      "\n",
      " \\par\n",
      "\n",
      "\n",
      "\n",
      "文章编号: 201410315308873 \\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "本内容之版权由相关传媒机构 / 版权持有人拥有。除非获得明确授权，否则严禁复制、改编、分发或发布本内容。版权持有人保留一切权利。 本内容经慧科的电子服务提供。\n",
      "\n",
      "\\page\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~  #3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "头条日报 | 2014-10-31 \\par\n",
      "\n",
      "P60| 港闻| 「虎」学英语| By 汤锦标 \\par \n",
      "\n",
      "\n",
      "\n",
      "「大嘴巴」英揆闯祸记 \\par\n",
      "\n",
      "\\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\\par\\par 本栏写过苏格兰公投，结果有惊无险，大英帝国完好无缺，英国首相卡梅伦肯定饮得杯落，原来连女皇殿下亦喜上眉梢，甚至喜极而泣。\\par\\par 　　苏公投独立失败，女皇喜极而泣，听落几夸张，我不是女皇肚里条虫，我讲你可以不信，但如泄密者是卡梅伦，情况自然不同。\\par\\par 　　话说「大嘴巴」卡梅伦在一次公众场合，同来访的前纽约市长彭博（Michael Bloomberg）咬耳仔，怎料搞出个大头佛，对话被麦克风收录，卡向彭披露女皇得悉公投结果后的一刻，直情开心到喊──Microphones picked up him saying that the Queen 'purred' a noise of contentment and had even wept for joy when Cameron called her with the news.\\par\\par 　　就这段公投小插曲，《英文虎报》写了一篇八卦新闻，题为 Cameron sorry for letting cat out of the bag，讲到卡梅伦「不小心」泄了女皇机密，公投一事，女皇人前扮中立，人后却喊住想保留威士忌国，卡因此要向女皇道歉──the revelation was a breach of protocol which saw a red-faced Cameron apologizing to the monarch for his big mouth.\\par\\par 　　To let the cat out of the bag （让猫走了出袋子） 是成语，不小心泄漏别人秘密的意思，一般指无恶意，属无心之失──the idiom means to reveal a secret, usually unwittingly, and suggests thoughtlessness rather than malice.\\par\\par 　　除卡梅伦失礼事件外，亦可举一例。有一大班人正准备为朋友举办惊喜派对（surprise party），当事人不知情正在回家途中，给巧遇的友人抛出一句──「我会准时到你的派对」──惊喜派对马上玩完。当英国工党人士知道卡犯错后，即作出批评──It's disrespectful to let the cat out of the bag. 这句话仍有多一层意思，就是指卡不应将一些内幕消息向外界披露，亦是对女皇的不尊重。\\par\\par 　　英语世界中，同猫咪相关的惯用语多如牛毛，简单如raining cats and dogs, fighting like cats and dogs, when the cat's away, the mice will play 等，或者复杂少少如 a catty remark （不饶人的骂人语，多指出自女人） 、put the cat among the pigeons （刻意做出事情引起骚动）、Has the cat got your tongue?（指被人追问时保持沉默）等。\\par\\par 　　另一英文流行语──open a can of worms，或比 let the cat out of the bag更加广泛使用。打开罐子发现满罐都是虫子，比喻做了某件事把问题更加复杂化，举一例子（\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 运动支持者可能不爱听）──当\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 三子宣布启动\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 占中\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 行动那刻钟，a can of worms has been opened，并引致一个困难并且失控的局面。\\par\\par 　　欲回看《虎报》相关文章，请用以下连结。http︰//goo.gl/ul3OLJ\\par\\par 汤锦标\\par\\par 《英文虎报》The Standard总编辑逢周五见报 \\par\n",
      "\n",
      " \\par\n",
      "\n",
      "\n",
      "\n",
      "文章编号: 201410315308918 \\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "本内容之版权由相关传媒机构 / 版权持有人拥有。除非获得明确授权，否则严禁复制、改编、分发或发布本内容。版权持有人保留一切权利。 本内容经慧科的电子服务提供。\n",
      "\n",
      "\\page\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~  #4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "都市日报 | 2014-10-31 \\par\n",
      "\n",
      "P50| 娱乐 \\par \n",
      "\n",
      "\n",
      "\n",
      "《华尔街日报》指周润发击败成龙 \\par\n",
      "\n",
      "\\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 注意：标题和版面之间存在一个空行！所以title是block的第4个元素。\n",
    "for i in range(1, 80):\n",
    "    print(news[i]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:34:27.974857Z",
     "start_time": "2023-11-17T07:34:27.971330Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\par\\par 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能，今天续谈Integration(整合)的功能。人类社会是一个复杂系统，大系统内还有很多次系统，这些系统既独立，又牵连；既矛盾，又互适；有时互相促进，有时互相制约。正如一个人，既是社会的成员，又是公司的雇员；既是别人的儿子，又是朋辈中的「大哥」；其言行会同时受所处的不同系统所影响。\\par\\par 这些错综复杂的系统之间不可能没有冲突，为了避免社会因纠纷得不到恰当的处理而分崩离析，社会必须发展出一套整合矛盾的方式，这就需要有司法制度。\\par\\par 按帕森斯的说法，法治的基础是先要界定产权。这样才能避免因争夺资源而产生无休止的冲突。再者，交易亦需要在产权获得界定后才能进行。有交易才有市场，才能透过市场机制进行公平竞争，推动经济发展。\\par\\par 另一方面，社会亦需要为人权下定义，这样，政府才能在有认受性的情况下组成，才能有效地去处理公众事务。此外，社会还需要有一套合乎公义的会议程序，以决定如何汇聚众人的意愿。\\par\\par 有了这些基础之\n"
     ]
    }
   ],
   "source": [
    "# 需要对中文编码的对象使用中文的方式进行解码\n",
    "print(news[17][:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:34:52.188666Z",
     "start_time": "2023-11-17T07:34:52.185388Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 定义一个函数：实现解码、编码、清洗效果\n",
    "def stringclean(s):\n",
    "    #s = s.decode('gb18030').encode('utf8')\n",
    "    s = s.replace(r'\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 ', '')\n",
    "    s = s.replace(r'\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 ', '')\n",
    "    s = s.replace('\\par', '').replace('\\n', '')\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:34:55.996726Z",
     "start_time": "2023-11-17T07:34:55.992369Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'bbccaadd ee '"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'aabbccaadd ee aa'.strip('a')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:34:58.212237Z",
     "start_time": "2023-11-17T07:34:58.207769Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ccdd ee'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'aabbccdd ee'.strip('ab')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:00.516378Z",
     "start_time": "2023-11-17T07:35:00.511781Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'abccdd ee'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'aabbccdd ee'.replace('ab', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:05.620246Z",
     "start_time": "2023-11-17T07:35:05.615998Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能，今天续谈Integration(整合)的功能。人类社会是一个复杂系统，大系统内还有很多次系统，这些系统既独立，又牵连；既矛盾，又互适；有时互相促进，有时互相制约。正如一个人，既是社会的成员，又是公司的雇员；既是别人的儿子，又是朋辈中的「大哥」；其言行会同时受所处的不同系统所影响。 这些错综复杂的系统之间不可能没有冲突，为了避免社会因纠纷得不到恰当的处理而分崩离析，社会必须发展出一套整合矛盾的方式，这就需要有司法制度。 按帕森斯的说法，法治的基础是先要界定产权。这样才能避免因争夺资源而产生无休止的冲突。再者，交易亦需要在产权获得界定后才能进行。有交易才有市场，才能透过市场机制进行公平竞争，推动经济发展。 另一方面，社会亦需要为人权下定义，这样，政府才能在有认受性的情况下组成，才能有效地去处理公众事务。此外，社会还需要有一套合乎公义的会议程序，以决定如何汇聚众人的意愿。 有了这些基础之后，社会就可以发展出一整套司法制度，让成员知所行止，令社会的矛盾不会恶化。 英国人为香港留下的，可不只是一套可以依据的律例，而是一整套法治的理念与司法程序。香港的回归能进行得相对平稳，与特区政府基本上原封不动地承继了原有的司法系统有莫大的关系。 回归后，虽有人危言耸听，说香港的法治已死，但世人仍公认香港的法治达国际一流水准，而港人亦可以如常在香港生活与做生意，不觉有失去法治的实质威胁。直到占中运动的出现，香港人才真正感受到失去法治的害处。 占中运动挑战的可不只是个别「恶法」，而是侵犯了整个法治的根基——产权、人权与政府的执法权。 占领区的物业，地契上列明有Right of way，但现在占领者却不容停车场的车辆出入。这分明损害了这些物业的产权。现在政府却无法加以维护；法庭出了禁制令，占中者却一样藐视。这样发展下去，谁敢在香港置业？ 其实，损害产权等同损害人权，因为人权的一项重要内容，就是个人的财产应获保障。此外，人人都应有追求幸福的权利，但现在占领区生意难做，怎会不妨碍别人追求幸福？ 占中者把自己的行为说成是公民抗命，但公民抗命只是个人行为在道德上的解释，用来拒绝遵守某些个人不认同的法令还讲得通，但绝不可以借此损害他人的产权与人权。 再者，占中者现时在争取的是宪政改革，本应获社会上绝大多数人赞同才有条件实施，不宜用占领交通要津的方式去逼其他人就范。如果祭起公民抗命的旗帜就可以欲所欲为，只会天下大乱，令法治失去协调与整合作用。 '"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 调用stringclean函数\n",
    "stringclean(news[17]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:16.660041Z",
     "start_time": "2023-11-17T07:35:16.639790Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16541"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 列表内的for循环\n",
    "news_clean = [stringclean(n) for n in news]\n",
    "len(news_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:24.459465Z",
     "start_time": "2023-11-17T07:35:24.455321Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能，今天续谈Integration(整合)的功能。人类社会是一个复杂系统，大系统内还有很多次系统，这些系统既独立，又'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news_clean[17][:120]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:33.100222Z",
     "start_time": "2023-11-17T07:35:33.096491Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 定义两个函数\n",
    "def deletetab(s):\n",
    "    return s.replace('\\t', '')\n",
    "\n",
    "import sys\n",
    "def flushPrint(s):\n",
    "    sys.stdout.write('\\r')\n",
    "    sys.stdout.write('%s' % s)\n",
    "    sys.stdout.flush() # 清洗掉 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-29T01:28:47.013097Z",
     "start_time": "2018-04-29T01:28:47.002696Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on OutStream in module ipykernel.iostream object:\n",
      "\n",
      "class OutStream(io.TextIOBase)\n",
      " |  A file like object that publishes the stream to a 0MQ PUB socket.\n",
      " |  \n",
      " |  Output is handed off to an IO Thread\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      OutStream\n",
      " |      io.TextIOBase\n",
      " |      _io._TextIOBase\n",
      " |      io.IOBase\n",
      " |      _io._IOBase\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, session, pub_thread, name, pipe=None)\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  close(self)\n",
      " |      Flush and close the IO object.\n",
      " |      \n",
      " |      This method has no effect if the file is already closed.\n",
      " |  \n",
      " |  flush(self)\n",
      " |      trigger actual zmq send\n",
      " |      \n",
      " |      send will happen in the background thread\n",
      " |  \n",
      " |  set_parent(self, parent)\n",
      " |  \n",
      " |  write(self, string)\n",
      " |      Write string to stream.\n",
      " |      Returns the number of characters written (which is always equal to\n",
      " |      the length of the string).\n",
      " |  \n",
      " |  writelines(self, sequence)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors defined here:\n",
      " |  \n",
      " |  closed\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |  \n",
      " |  __abstractmethods__ = frozenset()\n",
      " |  \n",
      " |  encoding = 'UTF-8'\n",
      " |  \n",
      " |  flush_interval = 0.2\n",
      " |  \n",
      " |  flush_timeout = 10\n",
      " |  \n",
      " |  topic = None\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from _io._TextIOBase:\n",
      " |  \n",
      " |  detach(...)\n",
      " |      Separate the underlying buffer from the TextIOBase and return it.\n",
      " |      \n",
      " |      After the underlying buffer has been detached, the TextIO is in an\n",
      " |      unusable state.\n",
      " |  \n",
      " |  read(...)\n",
      " |      Read at most n characters from stream.\n",
      " |      \n",
      " |      Read from underlying buffer until we have n characters or we hit EOF.\n",
      " |      If n is negative or omitted, read until EOF.\n",
      " |  \n",
      " |  readline(...)\n",
      " |      Read until newline or EOF.\n",
      " |      \n",
      " |      Returns an empty string if EOF is hit immediately.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from _io._TextIOBase:\n",
      " |  \n",
      " |  errors\n",
      " |      The error setting of the decoder or encoder.\n",
      " |      \n",
      " |      Subclasses should override.\n",
      " |  \n",
      " |  newlines\n",
      " |      Line endings translated so far.\n",
      " |      \n",
      " |      Only line endings translated during reading are considered.\n",
      " |      \n",
      " |      Subclasses should override.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from _io._IOBase:\n",
      " |  \n",
      " |  __del__(...)\n",
      " |  \n",
      " |  __enter__(...)\n",
      " |  \n",
      " |  __exit__(...)\n",
      " |  \n",
      " |  __iter__(self, /)\n",
      " |      Implement iter(self).\n",
      " |  \n",
      " |  __new__(*args, **kwargs) from builtins.type\n",
      " |      Create and return a new object.  See help(type) for accurate signature.\n",
      " |  \n",
      " |  __next__(self, /)\n",
      " |      Implement next(self).\n",
      " |  \n",
      " |  fileno(self, /)\n",
      " |      Returns underlying file descriptor if one exists.\n",
      " |      \n",
      " |      OSError is raised if the IO object does not use a file descriptor.\n",
      " |  \n",
      " |  isatty(self, /)\n",
      " |      Return whether this is an 'interactive' stream.\n",
      " |      \n",
      " |      Return False if it can't be determined.\n",
      " |  \n",
      " |  readable(self, /)\n",
      " |      Return whether object was opened for reading.\n",
      " |      \n",
      " |      If False, read() will raise OSError.\n",
      " |  \n",
      " |  readlines(self, hint=-1, /)\n",
      " |      Return a list of lines from the stream.\n",
      " |      \n",
      " |      hint can be specified to control the number of lines read: no more\n",
      " |      lines will be read if the total size (in bytes/characters) of all\n",
      " |      lines so far exceeds hint.\n",
      " |  \n",
      " |  seek(...)\n",
      " |      Change stream position.\n",
      " |      \n",
      " |      Change the stream position to the given byte offset. The offset is\n",
      " |      interpreted relative to the position indicated by whence.  Values\n",
      " |      for whence are:\n",
      " |      \n",
      " |      * 0 -- start of stream (the default); offset should be zero or positive\n",
      " |      * 1 -- current stream position; offset may be negative\n",
      " |      * 2 -- end of stream; offset is usually negative\n",
      " |      \n",
      " |      Return the new absolute position.\n",
      " |  \n",
      " |  seekable(self, /)\n",
      " |      Return whether object supports random access.\n",
      " |      \n",
      " |      If False, seek(), tell() and truncate() will raise OSError.\n",
      " |      This method may need to do a test seek().\n",
      " |  \n",
      " |  tell(self, /)\n",
      " |      Return current stream position.\n",
      " |  \n",
      " |  truncate(...)\n",
      " |      Truncate file to size bytes.\n",
      " |      \n",
      " |      File pointer is left unchanged.  Size defaults to the current IO\n",
      " |      position as reported by tell().  Returns the new size.\n",
      " |  \n",
      " |  writable(self, /)\n",
      " |      Return whether object was opened for writing.\n",
      " |      \n",
      " |      If False, write() will raise OSError.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from _io._IOBase:\n",
      " |  \n",
      " |  __dict__\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(sys.stdout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:39.721833Z",
     "start_time": "2023-11-17T07:35:39.718373Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 调用deletetab\n",
    "deletetab('\\ta')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:35:52.109775Z",
     "start_time": "2023-11-17T07:35:47.319014Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9"
     ]
    }
   ],
   "source": [
    "# 演示：flushPrint\n",
    "import time, random\n",
    "for i in range(10):\n",
    "    time.sleep(random.random())\n",
    "    flushPrint(i) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:39:35.247136Z",
     "start_time": "2023-11-17T07:39:35.239710Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "def readblocks(data):\n",
    "    copy = False\n",
    "    n = 0\n",
    "    block = []\n",
    "    chunk = defaultdict(lambda:[])\n",
    "    for i in data:\n",
    "        try:\n",
    "            if \"~~~~~~~~~~~~~~~~~~~~~~~~~~  #\" in i:\n",
    "                copy = True\n",
    "            elif \"文章编号:\" in i:\n",
    "                ids = i.replace('文章编号: ', '')\n",
    "                source = block[0].split('|')[0]\n",
    "                info = block[1]\n",
    "                title = deletetab(block[3]) # \n",
    "                body = [j for j in block[6:] if j != '\\n']\n",
    "                body = ' '.join(body)\n",
    "                body = deletetab(body)\n",
    "                body = '\"' + body  + '\"'\n",
    "                line = '\\t'.join([ids, source, info, title, body])\n",
    "                chunk[ids] = line\n",
    "                block = []\n",
    "                n += 1\n",
    "                if n%10 == 0:\n",
    "                    flushPrint(n)\n",
    "                copy = False\n",
    "            # copy must be here.\n",
    "            elif copy:\n",
    "                block.append(i)\n",
    "        except Exception as e:\n",
    "            print(i, e)\n",
    "            pass\n",
    "    return chunk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:40:13.652018Z",
     "start_time": "2023-11-17T07:40:13.648140Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~~~~~~~~~~~~~~~~  #1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n",
      "am730 | 2014-10-31 \\par\n",
      "\n",
      "A16| NEWS| C观点| By 施永青 \\par \n",
      "\n",
      "\n",
      "\n",
      "法治有整合社会功能 \\par\n",
      "\n",
      "\\par\n",
      "\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 注意：标题和版面之间存在一个空行！所以title是block的第4个元素。\n",
    "for i in range(1, 8):\n",
    "    print(news[i][:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:40:17.029845Z",
     "start_time": "2023-11-17T07:40:16.958831Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "710"
     ]
    }
   ],
   "source": [
    "# 按block清洗新闻报道\n",
    "news_result = readblocks(news_clean) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:40:32.978079Z",
     "start_time": "2023-11-17T07:40:32.974072Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "719"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 新闻的数量\n",
    "len(news_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:40:35.825386Z",
     "start_time": "2023-11-17T07:40:35.821135Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['201410315301186 ',\n",
       " '201410315308873 ',\n",
       " '201410315308918 ',\n",
       " '201410315301765 ',\n",
       " '201410310050132 ']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看字典的keys\n",
    "list(news_result.keys())[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:40:39.025449Z",
     "start_time": "2023-11-17T07:40:39.021195Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'201410270050110 \\t文汇报 \\tB04| 投资理财| 专家分析| By 邓声兴  \\t强泰环保水务毛利率高 \\t\"          AMTD证券及财富管理业务部总经理 邓声兴 港股上周走势窄幅上落，虽然内地第三季通胀数据及汇丰中国10月制造业采购经理指数较预期佳，及美股造好，但因市场观望占中局势及沪港通通车仍未有确实日期，令港股升幅有限。恒指全周升278点，每日平均成交减至530亿元，料恒指本周于23,100至23,500区间上落。 股份推介：强泰环保(1395)自9月上市以来股价表现强劲，集团于江苏省拥有三座污水处理设施，并以BOT形式经营，未来盈利增长动力主要来自设施改造后提升收益率，以及重金属污水处理业务。集团去年营业额增加38.8%至6,509万元，盈利增长26.6%至3,226万元，毛利率达76%，虽然早前集团发盈警，预期今年首6个月溢利会大减，但主要是由于上市产生的一次性费用所致。集团毛利率较其他水务股优胜，加上内地积极推进环保强泰环保发展，相信公司未来可保持优势。目标1.50元(1.38元)，止蚀1.20元。 比亚迪(1211)。中国工信部大力推动京津冀公文的公共服务领域的新能源汽车应用，提出2014至2015年，在京津地区公共交通服务领域推广逾2万辆新能源汽车，至2015年底，京津冀地区公交车中新能源汽车比例不低于16%。政策反映中央持续支持新能源汽车发展。比亚迪于9月推出新能源汽车，第四季度将有更多K9电动巴士交付，而电动车电池产量于9月明显增加，相信有助今年的收入增长。比亚迪将于本周公布第三季度盈利及展望，料可成为股价催化剂。目标57元(上周五收市52.15元)，止蚀49元。(笔者为证监会持牌人，未持有上述股份)     \"'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看字典的values\n",
    "list(news_result.values())[90]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:40:43.636053Z",
     "start_time": "2023-11-17T07:40:43.129267Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ids</th>\n",
       "      <th>source</th>\n",
       "      <th>info</th>\n",
       "      <th>title</th>\n",
       "      <th>body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201410315301186</td>\n",
       "      <td>am730</td>\n",
       "      <td>A16| NEWS| C观点| By 施永青</td>\n",
       "      <td>法治有整合社会功能</td>\n",
       "      <td>\"          前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201410315308873</td>\n",
       "      <td>大公报</td>\n",
       "      <td>A08| 要闻</td>\n",
       "      <td>扛工仔料加薪4.5% 建筑金融好景</td>\n",
       "      <td>\"          【大公报讯】记者彩雯报道：上班族明年加薪预测调查结果陆续揭盅，香港人力...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201410315308918</td>\n",
       "      <td>头条日报</td>\n",
       "      <td>P60| 港闻| 「虎」学英语| By 汤锦标</td>\n",
       "      <td>「大嘴巴」英揆闯祸记</td>\n",
       "      <td>\"          本栏写过苏格兰公投，结果有惊无险，大英帝国完好无缺，英国首相卡梅伦肯定...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ids  source                       info               title  \\\n",
       "0  201410315301186   am730    A16| NEWS| C观点| By 施永青            法治有整合社会功能    \n",
       "1  201410315308873     大公报                   A08| 要闻    扛工仔料加薪4.5% 建筑金融好景    \n",
       "2  201410315308918    头条日报   P60| 港闻| 「虎」学英语| By 汤锦标           「大嘴巴」英揆闯祸记    \n",
       "\n",
       "                                                body  \n",
       "0  \"          前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适...  \n",
       "1  \"          【大公报讯】记者彩雯报道：上班族明年加薪预测调查结果陆续揭盅，香港人力...  \n",
       "2  \"          本栏写过苏格兰公投，结果有惊无险，大英帝国完好无缺，英国首相卡梅伦肯定...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "news_list = [i.split('\\t') for i in news_result.values()]\n",
    "df_news = pd.DataFrame(news_list,\n",
    "                       columns= ['ids', 'source',\\\n",
    "                                 'info', 'title', 'body'])\n",
    "df_news.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-24T07:19:54.085577Z",
     "start_time": "2018-04-24T07:19:54.076753Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 保存数据：将数据写入硬盘\n",
    "with open('../data/zz-hk-2014-9-clean.txt','a') as p:\n",
    "     for record in news_result.values():\n",
    "            p.write(record+\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-29T01:43:26.571011Z",
     "start_time": "2018-04-29T01:43:26.541176Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ids</th>\n",
       "      <th>source</th>\n",
       "      <th>info</th>\n",
       "      <th>title</th>\n",
       "      <th>body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201409015325665</td>\n",
       "      <td>南华早报</td>\n",
       "      <td>EDT13| EDT</td>\n",
       "      <td>Pain of stasis</td>\n",
       "      <td>没有文字档。</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201409055305251</td>\n",
       "      <td>大公报</td>\n",
       "      <td>A20| 专版</td>\n",
       "      <td>坚决支持和拥护 全国人大常委会对香港政改的决定</td>\n",
       "      <td>一、本会坚决支持全国人大常委会关于香港特别行政区行政长官普选问题和201...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201409165304250</td>\n",
       "      <td>大公报</td>\n",
       "      <td>B02| 经济．航运</td>\n",
       "      <td>粤投1.8亿入股中超</td>\n",
       "      <td>【大公报记者毛丽娟深圳十五日电】中国水业（01129）与粤海投资（002...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               ids source          info                     title  \\\n",
       "0  201409015325665  南华早报   EDT13| EDT             Pain of stasis    \n",
       "1  201409055305251   大公报      A20| 专版    坚决支持和拥护 全国人大常委会对香港政改的决定    \n",
       "2  201409165304250   大公报   B02| 经济．航运                 粤投1.8亿入股中超    \n",
       "\n",
       "                                                body  \n",
       "0                                         没有文字档。      \n",
       "1            一、本会坚决支持全国人大常委会关于香港特别行政区行政长官普选问题和201...  \n",
       "2            【大公报记者毛丽娟深圳十五日电】中国水业（01129）与粤海投资（002...  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用pandas读取数据，并查看。\n",
    "import pandas as pd\n",
    "df = pd.read_csv( '../data/zz-hk-2014-9-clean.txt', \n",
    "                 sep = \"\\t\", names=['ids', 'source','info', 'title', 'body'])\n",
    "df[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-17T07:42:11.696078Z",
     "start_time": "2023-11-17T07:42:11.691837Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./data/occupycentral/zz-hk-2014-10.rtf',\n",
       " './data/occupycentral/zz-hk-2014-9.rtf']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用os改变默认的工作路径\n",
    "#import os\n",
    "#os.chdir('../data/occupycentral/')\n",
    "\n",
    "# 使用glob读取某一类文件的所有名称\n",
    "import glob\n",
    "filenames = glob.glob('./data/occupycentral/*.rtf')\n",
    "filenames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-29T01:56:25.617281Z",
     "start_time": "2018-04-29T01:56:25.387004Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "../data/occupycentral/zz-hk-2014-10.rtf\n",
      "710../data/occupycentral/zz-hk-2014-9.rtf\n",
      "410"
     ]
    }
   ],
   "source": [
    "for i in filenames:\n",
    "    print(i)\n",
    "    with open(i, encoding = 'gb18030', errors = 'ignore') as f:  # for windows users: errors = 'ignore'\n",
    "        news = f.readlines()\n",
    "        news = [stringclean(n) for n in news]\n",
    "        news_result = readblocks(news)\n",
    "        with open('../data/zz-hk-all-clean2018.txt','a') as p:\n",
    "            for record in news_result.values():\n",
    "                p.write(record+\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## 自学Pandas使用\n",
    "\n",
    "《Python Data Science Handbook》第三章 by Jake VanderPlas\n",
    "\n",
    "\n",
    "\n",
    "https://github.com/computational-class/datascience/blob/gh-pages/4.datasci/notebooks/03.00-Introduction-to-Pandas.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](./images/end.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "48px",
    "left": "1374px",
    "top": "44.375px",
    "width": "226px"
   },
   "toc_section_display": false,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}