ChatGPT

ChatGPT

ChatGPT 简介

ChatGPT 是由 OpenAI 开发的一种自然语言处理模型,它可以自动生成对话和回答问题。ChatGPT 基于 GPT(Generative Pre-trained Transformer)模型,通过预训练模型来学习自然语言处理任务。

ChatGPT 历史

OpenAI 于 2019 年发布了 GPT-2 模型,该模型引起了广泛关注,因为它可以生成高度逼真的自然语言文本。在 2020 年初,OpenAI 推出了 ChatGPT,它是 GPT-2 的一个变体,特别设计用于生成对话和回答问题。

2018年6月,OpenAI 首次发布了 GPT 模型(Generative Pre-trained Transformer)。该模型使用了非监督学习的方法,对大量文本进行预训练,以便在后续任务中获得更好的性能。该模型基于 Transformer 模型结构,这是一种用于自然语言处理的深度学习架构。

2019年2月,OpenAI 推出了 GPT-2 模型,这是 GPT 模型的一个更强大的版本。GPT-2 可以生成高度逼真的自然语言文本,因此 OpenAI 将其作为实验性模型进行发布,并限制了其访问。在 GPT-2 发布后不久,OpenAI 宣布将其作为一项商业化服务推向市场。

2020年5月,OpenAI 推出了 ChatGPT,这是 GPT-2 的一个变体,专门设计用于生成对话和回答问题。与 GPT-2 类似,ChatGPT 也使用了预训练的方法,从大量的文本数据中学习。通过这种方式,ChatGPT 可以生成高质量的自然语言文本,并且可以用于各种对话和问题回答任务。

2020年6月,OpenAI 发布了 GPT-3 模型,这是一个比 GPT-2 更大更强大的模型。GPT-3 拥有 1750 亿个参数,比 GPT-2 的 15 亿个参数多了一个数量级,这使得它能够生成更加逼真的自然语言文本,并且可以用于更多的自然语言处理任务。GPT-3 也是一个预训练模型,使用了与 GPT-2 相似的训练方法。

虽然 GPT-3 与 ChatGPT 并不是同一个模型,但它们都是基于相同的技术和架构。因此,GPT-3 的发布对于 ChatGPT 的发展也有着重要的影响。在 GPT-3 发布后不久,OpenAI 开始使用 GPT-3 的技术和经验来改进 ChatGPT,以提高其质量和效率,并将其推广到更广泛的市场中。

在 ChatGPT 发布后不久,OpenAI 将其作为一项商业化服务推向市场,并推出了不同的订阅方案,以适应不同的使用场景和需求。OpenAI 还在不断改进 ChatGPT,以提高其质量和效率,并将其推广到更广泛的市场中。

ChatGPT 注册

要使用 ChatGPT,您可以在 OpenAI 的网站上注册,然后可以访问他们的 API 来与 ChatGPT 进行交互。OpenAI 提供了一些不同的方案,以适应各种不同的使用场景和需求。

ChatGPT 用法

使用 ChatGPT 的方法非常简单。您可以向 ChatGPT 提出一个问题或问题的一部分,然后 ChatGPT 会生成一个与之相对应的回答。您也可以使用 ChatGPT 生成对话,包括与虚拟人物进行对话等等。以下是一些使用ChatGPT的常见用法: 对话生成:使用ChatGPT生成自然流畅的对话文本,可以用于聊天机器人、语音助手等应用程序。

问答系统:使用ChatGPT将用户的问题转化为自然语言文本,再根据文本生成答案,可以用于智能客服系统、搜索引擎等应用程序。

文本摘要:使用ChatGPT从一篇长文本中提取出最重要的内容,生成简洁的摘要,可以用于新闻聚合、文档归档等应用程序。

文本纠错:使用ChatGPT自动检测和纠正文本中的语法、拼写错误,可以用于自然语言处理、文本编辑等应用程序。

情感分析:使用ChatGPT分析文本的情感色彩,生成文本的情感标签,可以用于社交媒体、市场调研等应用程序。

总之,ChatGPT是一种功能强大的自然语言处理工具,可以帮助开发者和研究人员快速构建自然语言应用程序,提高应用程序的智能化程度和用户体验。

ChatGPT Hacking

由于 ChatGPT 的开放性,有人会试图滥用它,包括用于制作虚假信息、欺诈等。OpenAI 已经采取了措施来确保 ChatGPT 不被滥用,包括限制访问和过滤措施等。

ChatGPT 未来

ChatGPT 的未来非常令人兴奋。它将成为智能对话和聊天机器人的关键技术,这将为人类带来更多的方便和娱乐。ChatGPT 还将继续发展和改进,以提高其生成自然语言的质量和效率。

ChatGPT CheetSheet

如果您想要了解更多有关 ChatGPT 的信息,OpenAI 提供了一份“CheatSheet”,其中包含有关如何使用 ChatGPT 的详细信息和示例。这份 CheatSheet 可以帮助您更好地了解如何使用 ChatGPT,并提供一些实用的技巧和建议。您可以在 OpenAI 的网站上找到这份 CheatSheet.

创意的ChatGPT用法。

创作助手:有些作家使用ChatGPT作为创作助手,来帮助他们生成故事情节、角色和对话等。他们可以输入一些关键词或句子,让ChatGPT继续写下去,或者使用ChatGPT作为灵感来源,以创造出更有趣和富有想象力的故事。

AI情人:有些人使用ChatGPT作为自己的虚拟“AI情人”。他们可以和ChatGPT聊天,分享他们的想法和感受,或者让ChatGPT给他们提供一些安慰和建议。虽然这种使用方法有些奇怪,但是它说明了ChatGPT可以被用来满足人类的各种需求。

AI游戏角色:有些游戏设计师使用ChatGPT来设计AI游戏角色的对话和行为。他们可以将ChatGPT训练成一个特定角色的AI,然后让它在游戏中与玩家进行交互。这种使用方法可以增强游戏的真实感和互动性,使游戏体验更加丰富和有趣。

这些只是一些有趣的ChatGPT用法的例子,展示了ChatGPT的多种可能性。随着技术的不断发展和创新,我们相信ChatGPT将会被用在更多的领域和场景中。

2022 这个世界

注: 晚上翻阅Evernote 里的日记,发现, 2023年初写的这篇 2022 年的总结,还是有点意思, 发上来给大家看看

短时间来看,这个世界在 2022 好像没发生什么特别的, 长时间来看, 2022 它又产生了很多的转折奇点。

区块链

从 2009 年 Bitcoin 论文开始算, 区块链出现 13 年了, 17-18 年出现高潮后的低谷,宛如 2000 年的互联网泡沫, 以为 2020 年后随着 NFT 的泡沫兴起,元宇宙等概念加持, 区块链生态会迎来大发展, 实际并没有。 比特币一跌再跌, 以及回到 5 年前的水平, 其他币,包括以太坊也同样处境艰难。 2022 年整体没有太多的技术创新, 也没有商业上的创新,还是在炒作, 只是少了原来的热度。 唯一不变的是, 随着年轻原生一代人的慢慢增多, 占比越来越多的人对区块链产生与生俱来的信心, 将继续投入巨大的热情去创新, 相信在未来 3 年内必将出现革命性的商业应用,而且多半是 00 后创业的项目。

元宇宙

Meta 股价打骨折, 元宇宙被媒体吹出来的未来幻想在 2022年毛都没看到。 我买了 Occulus Quest 2, 第一次还挺新鲜,之后一直落灰,每次承受着巨大的心理压力从箱子里拿出来看看有没有新鲜玩意, 都被繁琐的糟糕的用户体验折腾到崩溃。 估计火箭航天旅游平民化, 这玩意还不一定能平民化。 三年内是不能碰 Meta 股票了。

电动车

2023 年国家正式取消了电动车补贴, 以为一大波电动车要涨价, 年底的时候我还着急找 TSLA 销售看看能不能提前订货好享受到补贴优惠。没想,2023 开年没几天, 特斯拉 宣布大幅度降价, 最便宜的Model 3 直接感到 23 万以下。 从 16 年我就一直看好 TSLA 这个家公司, 倒不是觉得他家车做的怎么样, 而是在 Youtube 上看了一个他们家工厂的视频, 简直颠覆我对车间的认知。 一家伟大的商业公司, 并不简单的是发明创造了一个好用的用户产品。 在背后,必然革新了一些列的生产关系。福特的成功不在于造成便宜的车, 而是实践了现代化的流水线。同样,丰田也不是表面上早出高质量低成本的车, 而是它的管理和质量理念在制造业中的实践。 每个时代的每个伟大公司, 必然创造了一些远远超越同时代的其他公司的东西。 特斯拉的后现代工厂就是它的天然竞争力。但是资本市场却忽略了这些。 自动驾驶提了那么多年,还是在纸上谈兵, 想必真正应用到实际生活中, 可能不是现在能够预测的事情。 没有根本性的方法的创新, 用现在的憨憨的 所谓 基于简单粗鲁的概率论的 AI 模型, 能真的搞出实用性的自动驾驶, 也是见鬼了。 但这个领域可能更像是 PC 电脑, 一直修修补补慢慢发展, 也别指望有啥终极大 boss 出现。 PC 30 年了, 还是一直不够用, 到今天还困扰在 CPU 、 内存、电池、硬盘空间这些问题上,虽然同 30 年前比, 量上有很多提升, 但之于质上, 并没有太大的变化。 想必自动驾驶在未来很长的时间内,也是一直修修补补, 生意照常做,但不要希望是 iPhone 这种万亿市值的生意。

COVIP-19 疫情

三年了, 20 年初1 月份,农历年前几天, 还在上海办公室加班。等过年回到老家后, 立马宣布全国大规模封控,当时想,操蛋,又要过三个月的非典生活了,2003 年是我至今的灰色记忆,印象中一直抑郁的, 天空也一直是灰色的。 没曾想, 居然整整三年, 在 2022 年的年底最后几天,宣布了 long-covid, 要共存, 甚至还要快速过峰,应阳尽阳。 整个北京城居然两周阳了 80% 以上, 一下子,非常快的速度, 在2022 年的最后几天, 北京城又恢复了堵车,或者叫, 又恢复了生机。 饭店吃饭要排队了, 昨天去三里屯吃饭, 排了一个半小时。 也许是看过 加缪的鼠疫,以及石黑一雄的被掩埋的巨人这类书, 思想上早已经做好长期压抑的锻炼。对 long-covid 倒是没有啥特别抵触对抗的情绪。 对比起身边一个女性朋友天天被病毒性抑郁和政治性抑郁搞得不要不要的, 我倒是乐于观察不同人在这种特殊环境中的反应。 也许很冷血, 但是却能看出来不同的人是如何思考和面对这个世界的。 疫情三年, 奇怪的是, 居然没有真实生活中认识的人得了新冠(除了那些朋友圈中的所谓朋友)。一切恐惧和担心好像又都是刻意营造出来的。 2022 放开了, 并不一定说 2023 年就完全不会被 COVID 困扰了。 看过一篇报道全球有近千种Omicron 变种, 隔几个月来一次的概率也不是很低。 3个奇怪的现象,

  1. 之前新闻号称 80%的无症状, 在这次的 Omicron 大感染中, 并没有看到哪怕一例身边的无症状, 几乎都是轻症。
  2. 上面提到的, 大流行之前, 居然没有真实生活中认识的人得了新冠(除了那些朋友圈中的所谓朋友)。一切恐惧和担心好像又都是刻意营造出来的。
  3. 一切糟糕的事情,都出现在新闻当中。加重了对世界的虚幻的认知。 时常怀疑自己是不是如同 抑郁的 金凯瑞饰演的楚门。 谁知道呢。

金融市场

中国经济数字糟糕到两辈子可能都难看到的数字。 8%,10%,12% 这些青春时期的 GDP增长率, 将可能是如同日本 90 年代初的霓虹灯一样, 一去不复返。 下半辈子,可能就是像日本人一样, 面对着失去的 30 年。 有时我对这句话本身很疑惑, 北京现在的各种条件, 也没比得上日本 十几年前的条件呀, 人家所谓失去了 30 年, 那岂不是以为了, 原来超越了我们可能 50 年。 对未来的年轻人是煎熬, 但对我这中年人, 只是日子罢了, 苦也过, 甜也过,哲学虚无孔乙己一下, 又有啥所谓呢。 2022 年的美股, 简单来说, 就是和 2021 年的 疯狂放水抵消了。 那些在 2021 年凭运气挣来的, 必定在 2022 年凭本事还回去。 搞了几万美刀的期权, 基本清零一分不剩。干干净净,和贾宝玉一样赤条条来赤条条去。

image

在最后几天, 坚守的波音和美航还是回来了,价值投资的概念在美股上还是有一定的生效周期的, 也许在我 大 A 股,可能是 1 万年吧 。 靠着这点仓位, 差不多又回到 21 年的水位了。 非要装模作样总结下经验的话, 那就是, 不要碰 A 股。 宁愿买美股,当定期存着,也比买 A 股好。

AI

AI 终究还是会迎来第三次寒冬。 被炒的火热的各种 transformer 和各种 deep learning 技术, 终于是遇到商业上瓶颈了。 看过各种 AI 公司, 比如思必驰等 语音技术为代表的, 讲真, 和 Meta 家的 VR 技术, 真的是 50 步笑 100 步,10 个人工, 一个智能,都不能算是智能。 以粗糙概率模型为基础的sigma 函数, 加上层层的网络梯度, 引入了太多的噪音,也去掉了太多的变量,之前在无关紧要的各种识别任务中号称打过人类, 但是在开车这种对人来说(是个人都能学会的技能)简单任务, 却一直得不到突破,还是打着商业的旗号, 分成 L1 / L2 / L3 / L4 blabla, 只是换种无耻的方式去骗钱罢了。 IBM 几十年前就干过这种生意, 也还真是有冤大头去付钱。 任然是年末, 理论上没啥大的进展, 但是工程上, 却出现了一个表现看似非常优异的 AI 产品, 那就是chatGPT, 能够一本正经似是而非的给出非常符合专业模式的回答。 还是很 Amazing 的。 至于会不会像当初的 Google 对用户信息获取模式产生革命性的影响, 现在很难回答, 个人觉得,概率还是比较低的。 如果把这个世界上所有的文字信息当做一个系统, Google 的系统是非封闭的, 一直有人类输入信息。从信息论的角度来看, Entropy 是一直减小的。 但是 ChatGPT 是每天基于已有信息二次生成新的信息, 但是从信息论角度来看, entropy 是不变的。 当然你可能也会说, 人类本身每天产出来的新的信息也是基于现有信息加工的, 但我不认同。 诚然, 有很大一部分是已有的信息再加工, 但是在加工过程中, 会引入很多随机的系统外信息,也就是各种天马行空的创意, 这也是艺术家,文学家和哲学家存在 的理由,他们一直给这个信息系统提供新的信息,降低 entropy。 预测下, 2023 年,会出现一些 chatGPT 为基础的各种应用型创业公司, 在垂直领域拿到一些投资, 提升一下生产效率,但并不会带来什么革命性的影响。 Bing 靠着引入 chatGPT 也不会打败 Google。 AI 真的能解决前面说是自动驾驶这类问题, 还却一个图灵奖级别的创新,光是 Deep Learning 显然是不够的。

情感

如果加缪和石黑一雄构建的世界中描述的, 人们的情感在经历了长时间的疫情之后, 会出现很大的变化。 从而潜移默化的改变着未来的行为模式。 比如,从数据统计上来看, 避孕套卖的比原来少很多, 但是怀孕生小孩的更少了。 直接结论是, 人们不愿意做爱了。 疫情让人们的荷尔蒙分泌大大降低,对未来的激情也燃烧殆尽。 (今天去便利蜂买棉签, 在避孕套那排看了下, 发现卖的比较贵的冈本, 有很多大盒装的,但是里面只有两个,而 durex 这种平民型的, 基本都是 10 个装。 可以假想下, 10 个装的,主要针对的是固定伴侣的用户, 而两个装的, 针对的是荷尔蒙分泌更旺盛的用户, 不知道大数据分析是不是支持这样的结论) 。

文艺活动

年轻人还是一茬一茬的, 过完疫情, 大家解放了, 文艺活动还是会多起来的。 电影会繁荣起来的, 文学也是一样。 不过怎么没有疫情文学呢, 也很是纳闷。 我们这个时代还不够糟糕到配上文学吗?

一战的茨威格, 二战的格拉斯,革命小将嬉皮士,乔姆斯基,福柯, 我们的时代呢?没人来拯救么?

Algorithm Review

Algorithm Review

Stack

Queue

Heap

SORT

Math for algorithms

排序算法的商减 (entropy decrease)

Math for algorithms

Math basics for algorithm

A.1 求和公式及其性质

  • 等差 \(\sum_{k=1}^n k = \frac { n(n+1)}{2}\)

\(\sum_{k=1}^n k^2 = \frac {n(n+1)(2n+1)}{6}\)  \(\sum_{k=1}^n k^3 = \frac { n^2(n+1)^2}{4}\)

  • 线性性质

\(\sum_{k=1}^n (ca_k+b_k) = c\sum_{k=1}^n a_k + \sum_{k=1}^n b_k\) \(\sum_{k=1}^n \Theta\left(f(k)\right) = \Theta \left( \sum_{k=1}^n f(k) \right)\)

  • 几何级数
\[\sum_{k=0}^n x^k = 1 + x + x^2 + \ldots + x^n = \frac {x^{n+1} -1} {x-1}\] \[\sum_{k=0}^{\infty} x^k = \frac {1} {x-1}\]
  • 调和级数
\[H_n = 1 + \frac{1}{2} + \frac{1}{3} + \frac{1}{4} + \ldots + \frac{1}{n} = \sum_{k=1}^n \frac{1}{n} = \ln{n} + O(1)\]
  • 级数积分与微分
\[\sum_{k=0}^{\infty} kx^k = \frac {x} {(1-x)^2}\]
  • 列项级数 telescopes
\[\sum_{k=1}^{n} (a_k - a_{k-1}) = a_n - a_0\]

\(\sum_{k=0}^{n-1} (a_{k-1} - a_k) = a_0 - a_n\) \(\sum_{k=1}^{n-1} \frac{1}{k(k+1)} = \sum_{k=1}^{n-1} (\frac{1}{k} - \frac{1}{k+1}) = 1- \frac{1}{n}\)

  • 乘积 \(\lg\left(\prod_{k=1}^{n} a_k \right) = \sum_{k=1}^{n} \lg{a_k}\)

A.2 确定求和时间的界

  • 数学归纳法
  • 确定级数中各项的界
  • \[\sum_{k=1}^{\infty} \frac{1}{k} = \lim _{n \to \infty} {\sum_{k=1}^{n} \frac{1}{k} } = \lim _{n \to \infty} {\Theta(\lg n)} = \infty\]
\[H_n = \sum_{k=1}^n \frac{1}{n} = 1 + \frac{1}{2} + \frac{1}{3} + \frac{1}{4} + \ldots + \frac{1}{n} \leq \ln{n} + 1\]
  • 通过积分求和的近似

    • if $f(x)$ 单调递增: \(\int_{m-1}^{n} f(x)\,dx \leq \sum_{k=m}^n f(k) \leq \int_{m}^{n+1} f(x)\,dx\)

    • if $f(x)$ 单调递减: \(\int_{m}^{n+1} f(x)\,dx \leq \sum_{k=m}^n f(k) \leq \int_{m-1}^{n} f(x)\,dx\)

    \(H_n = \sum_{k=1}^n \frac{1}{n} \geq \int_{1}^{n+1} \frac{1}{x}\,dx = \ln{(n+1)}\) \(\sum_{k=2}^n \frac{1}{n} \leq \int_{1}^{n} \frac{1}{x}\,dx = \ln{(n)}\)

    \[H_n = \sum_{k=1}^n \frac{1}{n} \leq \ln{n}+1\]

重点

  • Illustrate function call stack
  • Describe how to convert recursion to iteration
    • one function call
    • 2 or more function calls
    • tail recursion
  • PERMUTATION
PERM(A, k)
  if(k == n)
    PRINT(A)
  for i = 1 to n 
    SWAP(A, i, k)
    PERM(A, k+1)
    SWAP(A, i, k)


PERM(A, 1)

  • DFS and stack
  • BFS and queue

vue-router

  • History

API

核心是使用 HTML window.history API

  • vm
    • $options
      • _parentVnode
      • $router
      • $route

Vue.util.defineReactive(

  • window
    • location
      • ancestorOrigins: DOMStringList {length: 0}
      • assign: ƒ ()
      • hash: “#html”
      • host: “router.vuejs.org”
      • hostname: “router.vuejs.org”
      • href: “https://router.vuejs.org/guide/#html”
      • origin: “https://router.vuejs.org”
      • pathname: “/guide/”
      • port: “”
      • protocol: “https:”
      • reload: ƒ reload()
      • replace: ƒ ()
      • search: “”
      • toString: ƒ toString()
    • history
    • performance
      • Performance Timeline API,
      • the High Resolution Time API,
      • the Navigation Timing API,
      • the User Timing API,
      • the Resource Timing API.

因为安全的问题, 不能获取 history stack 列表, 只能 push pop 操作。

vue-cli

vue-cli

Guide

type

  • app
  • library
  • wc

  • @vue/cli
  • @vue/cli-service
  • @vue/cli-plugin-
  • vue-cli-plugin-
  • Development
    • Browser Compatibility
      • browserslist
      • Polyfills
    • HTML 和静态资源
      • index file public/index.html
      • interpolation with loadash
        • e.g <link rel="icon" href="<%= BASE_URL %>favicon.ico">
        • <%= VALUE %>
        • <%- VALUE %>
        • <% expression %>
      • preload <link rel="preload">
      • prefetch <link rel="prefetch">
      • Disable Index Generation
      • Building a Multi-Page App
      • Static Assets Handling
      • Relative Path Imports
      • URL Transform Rules
        • absolute: /images/foo.png
        • relative: ./
        • module: <img src="~some-npm-package/foo.png">
        • module: <img src="@some-npm-package/foo.png">, interpret to <projectRoot>/src
      • The public Folder
          <link rel="icon" href="<%= BASE_URL %>favicon.ico">
        
          data () {
              return {
                  baseUrl: process.env.BASE_URL
              }
          }       
                    
          <img :src="`${baseUrl}my-image.png`">
        
        
    • CSS 相关
    • webpack 相关
    • Enviroment Variables and Modes
    • Build Targets
    • Deployments

      Config Reference

  • Global Config
    • ~/.vuerc
    • vue config
  • Target Browsers
  • vue.config.js
    • loaded by @vue/cli-service
    • or by vue section in package.json
        module.exports = {
            baseUrl: '/my-app/' or './',
            outputDir: 'dist',
            assetsDir: '',
            indexPath: 'index.html',
            filenameHashing: false,
            pages: undefined,
            lintOnSave: 'true',
            lintOnSave: process.env.NODE_ENV !== 'production',
            runtimeCompiler: 'false',
            transpileDependencies:[], // 转译 node_modules/ 中的依赖
            productionSourceMap: true,
            crossorigin: undefined,
            integrity: false,
            configureWebpack: {
                plugins: [
                new MyAwesomeWebpackPlugin()
                ]
            },
            chainWebpack: '',
            css.modules: false,
            css.extract: false,
            css.sourceMap: false,
            css: {
                loaderOptions: {
                    css: {
                        // 这里的选项会传递给 css-loader
                    },
                    postcss: {
                        // 这里的选项会传递给 postcss-loader
                    }
                }
            },
            devServer: {
                proxy: 'http://localhost:4000'
            },
            parallel: true,
            pwa: {},
            pluginOptions: {
                foo: {
                // 插件可以作为 `options.pluginOptions.foo` 访问这些选项。
                }
            },
            chainWebpack: config => {
                config.plugins.delete('html')
                config.plugins.delete('preload')
                config.plugins.delete('prefetch')
            }
      
      
                  
      
      
      
      
      
      
      
                  
        }
      
  • Babel
    • @vue/babel-preset-app
    • babel.config.js
    • Polyfills
  • ESLint @vue/cli-plugin-eslint

    • .eslintrc
    • eslintConfig in package.json
  • TypeScript @vue/cli-plugin-typescript
    • tsconfig.json
  • Unit Testing
    • Jest
    • Mocha
  • E2E Testing
    • Cypress
    • Nightwatch

Plugin Dev Guide

Plugins

Q & A

  1. what is vue-loader, babel-loader, lint-loader etc. ?
  2. Javascript moduel, moduleA.js? moduleB/index.js?

    See Learn the basics of the JavaScript module system and build your own library

    Telling stories is as basic to human beings as eating. More so, in fact, for while food makes us live, stories are what make our lives worth living - Richard Kearney

    • tools
      • CommonJS
      • AMD
      • UMD
      • ES6 (Harmony)
    • Best Practices
      • Tree shaking: import , export
      • Webpack vs Rollup vs Babel?
      • webpack:
        • code splitting,
        • async loading of bundles,
        • tree shaking,
      • UI Libraries have a dist folder that has the bundled and minified version for ES and UMD/CJS module systems as a target. There is a lib folder that has the transpiled version of the library.
      • Core Packages have just one folder which has the bundled and minified version for CJS or UMD module system as a target.

Cascading Style Sheets

CSS

  • Syntax
  • selectors
    • select by id #id
    • select by element type p {text-align: center; color: red;}
    • select by class: .center
    • combined: p.center
    • group:
        h1 {
            text-align: center;
            color: red;
        }
      
        h2 {
            text-align: center;
            color: red;
        }
      
        p {
            text-align: center;
            color: red;
        }
      
        h1, h2, p {
            text-align: center;
            color: red;
        }
      
    • comments:
        p {
            color: red;
            /* This is a single-line comment */
            text-align: center;
        }
      
        /* This is
        a multi-line
        comment */
      
  • refer to style sheet:
    • External style sheet
        <head>
        <link rel="stylesheet" type="text/css" href="mystyle.css">
        </head>
      
    • Internal style sheet
        <head>
        <style>
        body {
            background-color: linen;
        }
      
        h1 {
            color: maroon;
            margin-left: 40px;
        } 
        </style>
        </head>
      
    • Inline style
        <h1 style="color:blue;margin-left:30px;">This is a heading</h1>
      
    • Cascading Order: all the styles will “cascade” into a new “virtual” style sheet by the following rules:
      • Inline style (inside an HTML element)
      • External and internal style sheets (in the head section)
      • Browser default
  • Colors
    • Background Color: background-color:Tomato;
    • text color: color:MediumSeaGreen;
    • border: border:2px solid Violet;
    • Color values:
      • rgba(255, 99, 71, 0.5), or hex: #ee82ee
      • hsla(9, 100%, 64%, 0.5)
        • Hue is a degree on the color wheel from 0 to 360. 0 is red, 120 is green, and 240 is blue.
        • Saturation is a percentage value, 0% means a shade of gray, and 100% is the full color.
        • Lightness is also a percentage, 0% is black, 50% is neither light or dark, 100% is white
      • Shades of gray are often defined using equal values for all the 3 light sources: rgb(180, 180, 180) hsl(0, 0%, 0%)
  • backgrounds

      body {
          background-color: lightblue;
          background-image: url("paper.gif");
          background-repeat: repeat-x|no-repeat;
          background-attachment: fixed;
          background-position: right top;
          background-size: 300px 100px;
          background-size: auto|length|cover|contain|initial|inherit;
    
      }
    

    short:

      background: #ffffff url("img_tree.png") no-repeat right top;
      background: bg-color bg-image position/bg-size bg-repeat bg-origin bg-clip bg-attachment initial|inherit;
    
    

    When using the shorthand property the order of the property values is:

    • background-color
    • background-image
    • background-repeat
    • background-attachment
    • background-position
  • Borders
      p.one {
          border-style: solid;
          border-top-style: dotted;
          border-right-style: solid;
          border-bottom-style: dotted;
          border-left-style: solid;
          border-width: 5px;
          border-color: red;
          border: 5px solid red;
          border-radius: 5px;
          border-collapse: separate|collapse|initial|inherit;
    
      }
    
    
    
  • Margins
  • Paddings
  • Height/Width
  • Box Model
  • Outline
  • Text
      h1 {
          color: blue;
          text-align: left|right|center|justify|initial|inherit;
          text-decoration: text-decoration-line text-decoration-color text-decoration-style|initial|inherit;
          text-decoration-line: none|underline|overline|line-through|initial|inherit;
          text-transform: uppercase|lowercase|capitalize;
          text-indent: 50px;
          letter-spacing: 3px;
          line-height: 0.8;
          direction: rtl;
          word-spacing: 10px;
          text-shadow: 3px 2px red;
    
    
    
      }
    
  • Fonts
  • Icons
  • Links
  • Lists
  • Tables
  • Display
    • Block
    • inline
    • display: none|inline|block;
  • max-width
  • postion
      position: static|relative|fixed|absolute|sticky
    

Deep Learning

Deep Learning

\[a = \sigma(z) = \frac 1 {1+e^{-z}} \\ \frac {\partial \sigma} {\partial z} = \sigma(1-\sigma)\]

\(z_j^{l+1} = \sum_{k=1} (w_{jk}^la_k^l) + b_j^{l+1}\) \(z= w \cdot a + b\)

Cost for sample $x$ \(C_x= \frac 1 2 \sum_{j=1}^m (y_j-a_j)^2\)

total Cost:

\[C= \frac 1 2 \sum_{x}C_x\]

the objective function: \(min(C), w.t. w_{jk}, b_j\)

\[\frac {\partial C_x} {\partial z_j^L} = ( a_j^L - y_j) \frac {\partial a_j^L} {\partial z_j^L} \\ =(y_j - \sigma)\cdot \sigma \cdot (1-\sigma) \\ \sigma = a_j^L\]

define: \(\delta^l_j \equiv \frac {\partial C} {\partial z_j^l} \\\)

\(z_j^{l+1} = \sum_{k=1} (w_{jk}^l\sigma(z_k^l) + b_j^{l+1} \\ \frac {\partial z_j^{l+1}} {\partial z_k^{l}} = w_{jk} \cdot \sigma^{\prime} = w_{jk}^{l+1} \cdot \sigma(z_k^{l})(1-\sigma(z_k^{l}))\) to z $$ \frac {\partial C_x} {\partial z_k^l} = \sum_j { \frac {\partial C_x} {\partial z_j^{l+1} } \cdot \frac {\partial z_j^{l+1}} {\partial z_k^{l}} }\

\delta^l_k = \sum_j \delta^{l+1}_j \cdot {\frac {\partial z_j^{l+1}} {\partial z_k^{l} }}

\(Output layer:\) \delta^L_j \equiv \frac {\partial C} {\partial z_j^L}
=\frac {\partial C} {\partial a_j^L} \cdot \sigma^{\prime}(z_j^L)
= ( a_j^L - y_j) \cdot \sigma(z_j^L) \cdot (1 - \sigma(z_j^L)) $$

matrix based: \(\delta^L \equiv \frac {\partial C} {\partial z^L} \\ = \nabla_aC \odot {\sigma^{\prime}(z^L)} \\ = ( a^L - y) \odot {\sigma^{\prime}(z^L)}\) back propagation:

$$


\delta^l_k = \sum_j {\delta^{l+1}_k}

\(\) \delta^L = \nabla_aC \odot {\sigma^{\prime}(z^L)} \

\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot {\sigma^{\prime}(z^l)} \

$$

with $w_{jk}$ $$ \nabla_{b^l}C = \delta^l
\nabla_{w^l}C = \delta^l \sigma^{l-1} \

$$

ES6标准入门

ECMAScript 6 Primer


Date: 20180904

第1章 ECMAScript 6 简介

第2章 let 和 const 命令

2.1 let

  • 类似 var,但是又 locality
  • 代码块内有效
      {
          let a = 10; // 超出此代码块无效, 局部变量
          var b = 5;  // 代码块外有效, 作用域: 函数 + 全局
      }
    
  • 变量提升
  • 暂时性死去 temporal dead zone (TDZ)
  • 不允许重复声明
  • block level domain
  • do
      let x = do {
          let t = f();
          t * t + 1;
      }
    

    2.2 const

    定义常量

    const PI = 3.1415
    const a = []
    a.push('Hello')
    
  • 声明变量的6中方法
    • var
    • function
    • let
    • const
    • import
    • class

顶层对象

  • 浏览器: window
  • Node / web workder: global
  • this

第 3 章 变量的解构赋值 Destructuring assignment

模式匹配

3.1 数组的 Destructuring assignment

let [firstName, surname] = arr;
function fibs() {
    let a = 0;
    let b = 1;
    whiile (true) {
        yield a; 
        [a, b] = [b, a+b];
    }
}
let [first, second, third, fourth, fifth, sixth] = fibs();
sixth

3.2 对象的解构赋值


let jsonData ={
    id: 42,
    status: 'OK',
    data: [867, 5309]
};
let {id, status, data: number } = jsonData;

Map


var map = new Map();
map.set('first', 'hello');
map.set('second', 'world');
for(let [key, value] of map){
    console.log(key + ' is ' value);
}

TODO:

  • 第 4 章: 字符串的扩展

Q & A

  • javascript 中 ; 使用
    • Stackoverflow: Function expression ending with ; vs. not

      At it’s root, what you have there is an assignment statement, and according to the Google Javascript Style Guide, all statements should end in a ;. Especially assignment statements. JSLint requires ; or else it won’t pass.

  • javascript prototype
    • 动态向 javascript object 添加属性, 每个函数都有一个prototype属性,这个属性是指向一个对象的引用,这个对象称为原型对象,原型对象包含函数实例共享的方法和属性,也就是说将函数用作构造函数调用(使用new操作符调用)的时候,新创建的对象会从原型对象上继承属性和方法。
      function employee(name,job,born)
      {
          this.name=name;
          this.job=job;
          this.born=born;
      }
    
      var bill=new employee("Bill Gates","Engineer",1985);
    
      employee.prototype.salary=null;
      bill.salary=20000;
    
    

How Music Works

How Music Works

Note

Physics

Frequency Hertz Amplitude Wavelength Speed (1.4) Musical Instrument Tone

There is a huge variety of musical instruments and sounds, as you would already know from your experience with music. Even two instruments playing the same note can sound very different.

This is because a musical instrument produces a sound wave which is a combination of different but related frequencies (known as harmonics) which all mix together to create the distinctive tone or voice of the instrument.

The lowest frequency is usually dominant, and you perceive this one as the pitch. The combination of the other harmonics provides the distinctive shape of the waveform, and thereby the distinctive tone of the instrument.

String instrument frequency determined by the length of the string. Guitar http://hyperphysics.phy-astr.gsu.edu/hbase/Waves/string.html MusicalNoteFrequencies

Mathematics of Music

https://www.lightnote.co/ https://ianring.com/musictheory/scales/#citation2 https://www.ams.jhu.edu/dan-mathofmusic/ https://news.ycombinator.com/item?id=12792063 https://quariety.com/2018/04/28/a-laymans-intro-to-western-classical-music/ https://www.soundslice.com/scores/auld-lang-syne/

MIT Music21 project

Music21 is a set of tools for helping scholars and other active listeners answer questions about music quickly and simply. If you’ve ever asked yourself a question like, “I wonder how often Bach does that” or “I wish I knew which band was the first to use these chords in this order,” or “I’ll bet we’d know more about Renaissance counterpoint (or Indian ragas or post-tonal pitch structures or the form of minuets) if I could write a program to automatically write more of them,” then music21 can help you with your work.

Music Theory

Piano key frequencies

  • scientific notation
      Eb4, C#,,, C4, d#'', Ab2
    
  • Helmholtz

    libraries

    tone.js music synthesis,

    Teoria.js for theory

  • Note
  • Chord
  • Scale
  • Interval

Web Audio API

Event Driven Design

Introduction

Observer Pattern

Centralized Event Management

Decouple event from event handling

Implementation

Queue Single Thread, queue, watch

Operating System, Kernel mode, user mode

select / epoll model

GUI Design

Updating UI in dedicated thread. Single Thread UI / main thread

epoll

Windows

Android

AsyncTask

Vue

vm._events[event] stores an array of listener functions: (vm._events[event] || (vm._events[event] = [])).push(fn)

  • vm.$emit
  • vm.$on
  • vm.$once( event, callback )
  • vm.$off( [event, callback] )
// node_modules/vue/src/core/instance/events.js
export function eventsMixin (Vue: Class<Component>) {
  const hookRE = /^hook:/
  Vue.prototype.$on = function (event: string | Array<string>, fn: Function): Component {
    const vm: Component = this
    if (Array.isArray(event)) {
      for (let i = 0, l = event.length; i < l; i++) {
        this.$on(event[i], fn)
      }
    } else {
      (vm._events[event] || (vm._events[event] = [])).push(fn)
      // optimize hook:event cost by using a boolean flag marked at registration
      // instead of a hash lookup
      if (hookRE.test(event)) {
        vm._hasHookEvent = true
      }
    }
    return vm
  }

Javascript Event

https://developer.mozilla.org/en-US/docs/Web/API/Event

Java Event

In Java, we implement event using interface, See https://docs.oracle.com/javase/tutorial/uiswing/events/intro.html

Event Hierarchy

Centralized management

sub event

distributed management

register

pub/sub

Programming Models and Phylosophy

A Core Pattern for Events Adila A. Krisnadhi1;2 and Pascal Hitzler1 1 Wright State University, OH, USA 2 Universitas Indonesia, Depok, Indonesia

Refernces:

  • https://github.com/mspnp/architecture-center/blob/master/docs/patterns/event-sourcing.md

AMAP高德 地图

适用平台: Web Android iOS Web Service

位置产品

  1. 地图
  2. 定位
  3. 导航
  4. 路线规划
  5. 搜索
  6. 室内地图
  7. 室内定位

自定义地图 位置数据可视化 海外LBS服务

Java SPI

SPI Service Provider Interface

checkout github repo: https://github.com/xianminx/hellospi

Spring mongo data naming

Mongo Naming

JSON Naming

Google JSON Style Guide

https://publicobject.com/2016/01/20/strict-naming-conventions-are-a-liability/ Gson has built-in magic to convert snakes into camels with its LOWER_CASE_WITH_UNDERSCORES FieldNamingPolicy. We make a global configuration change to our Gson instance and the problem is solved.

Java Naming

Spring data mongo

https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#repositories.query-methods.query-property-expressions


8.4.3. Property Expressions
Property expressions can refer only to a direct property of the managed entity, as shown in the preceding example. At query creation time, you already make sure that the parsed property is a property of the managed domain class. However, you can also define constraints by traversing nested properties. Consider the following method signature:

List<Person> findByAddressZipCode(ZipCode zipCode);
Assume a Person has an Address with a ZipCode. In that case, the method creates the property traversal x.address.zipCode. The resolution algorithm starts by interpreting the entire part (AddressZipCode) as the property and checks the domain class for a property with that name (uncapitalized). If the algorithm succeeds, it uses that property. If not, the algorithm splits up the source at the camel case parts from the right side into a head and a tail and tries to find the corresponding property — in our example, AddressZip and Code. If the algorithm finds a property with that head, it takes the tail and continues building the tree down from there, splitting the tail up in the way just described. If the first split does not match, the algorithm moves the split point to the left (Address, ZipCode) and continues.

Although this should work for most cases, it is possible for the algorithm to select the wrong property. Suppose the Person class has an addressZip property as well. The algorithm would match in the first split round already, choose the wrong property, and fail (as the type of addressZip probably has no code property).

To resolve this ambiguity you can use \_ inside your method name to manually define traversal points. So our method name would be as follows:

List<Person> findByAddress_ZipCode(ZipCode zipCode);
Because we treat the underscore character as a reserved character, we strongly advise following standard Java naming conventions (that is, not using underscores in property names but using camel case instead).

Mongo References:

  • manual references
  • DBRef

Manual references are named after the referenced collection. Use the document type (collection name in singular) followed by _id ( _id ). This is the only case where you can use underscore in the middle. For example, in a dogs collection, a document would have manual references to external documents named like this: { name: 'fido', owner_id: '5358e4249611f4a65e3068ab', race_id: '5358ee549611f4a65e3068ac', colour: 'yellow' ... }

https://docs.mongodb.com/manual/reference/database-references/#document-references

Mongo follows javascript naming convention using camelCase

Mongo has no preference on camelCase v.s. underscore_style just stay consistent with the language. But also be consistent with on-wire json/xml format, also consider for Android/ iOS / Javascript web consumer sides.

http://christophermaier.name/2011/05/22/MongoDB-key-names/

different to relational database, in mongoldb, key names are stored in every document, so avoid using long names.

db.collection.stats()

using shorter keys can potentially save a lot of space and even increase performance (after all, more of a smaller database can fit into memory). WiredTiger engine in Mongo 3.0 supports compression, the point is rather moot.

http://arkusnexus.com/2016/09/12/coding-guidelines-mongodb/

GENERAL GUIDELINES

  • Use tabs to indent. This applies to all MongoDB-specific code (chained functions) and objects used by MongoDB (queries, projections, documents).
  • Always have a space after a : colon.
  • Comma-last. If you divide the components of an object/array into various lines, use one line for each component. The } closing curly brace should follow the last component (except for aggregation). GENERAL GUIDELINES
  • No _ underscores in the middle of names (database, collection, fields) except for manual references.
  • Collections database, variables, properties and function names should uselowerCamelCase. They should also be descriptive. Single character variables and uncommon abbreviations should generally be avoided.
  • Place spaces between nested parentheticals and elements in JavaScript examples. For example, prefer { [ a, a, a ] } over {[a,a,a]}.

COLLECTION NAMES

  • The name should be a plural of the types of documents to be saved.
  • Use camelCase. Normally you shouldn’t have to because the collection name will be one word (plural of the types of documents saved).
  • A collection with “” empty string is not a valid collection name.
  • A collection name should not contain the null character because this defines the end of collection name.
  • Collection name should not start with the prefix “system.” as this is reserved for internal collections.
  • It would be good to not contain the character “$” in the collection name as various drivers available for database do not support “$” in collection name. DATABASE NAMES

  • Try to have the database named after the project and one database per project.
  • Use camelCase.
  • A database with “” empty string is not a valid database name.
  • Database name cannot be more than 64 bytes.
  • Database name are case-sensitive, even on non-case-sensitive file systems. Thus it is good to keep name in lower case.
  • A database name cannot contain any of these characters “/, \, ., “, *, <, >, :, |, ?, $,”. It also cannot contain a single space or null character. FIELD NAMES

  • Use camelCase.
  • Don’t use _ underscore as the starting character of the field name. The only field with _ undescore should be _id.
  • Field names cannot contain . dots or null characters and must not start with a $ dollar sign. Manual references are named after the referenced collection. Use the document type (collection name in singular) followed by _id ( _id ). This is the only case where you can use underscore in the middle. For example, in a dogs collection, a document would have manual references to external documents named like this: { name: 'fido', owner_id: '5358e4249611f4a65e3068ab', race_id: '5358ee549611f4a65e3068ac', colour: 'yellow' ... } FUNCTIONS

  • One method per line should be used when chaining methods.
  • You should also indent these methods with a tab so it’s easier to tell they are part of the same chain. If you need to use a long query, projection or options object for a function, assign it to a var and use the var in the function call to improve readability.

vue-element-admin

axios

Promise based HTTP client for the browser and node.js

mock.js

create project and download deps.

vue create hello_mockjs
cd hello_mockjs 
npm install mockjs

npm run serve
  • 数据模板定义规范 DTD ` ‘name|rule’: value `

random

Random is a useful nodejs commandline / library for generating random objects, like url, email, etc.

npm install random -g 
➜  hello_mockjs git:(master) ✗ random

  Usage: random [options] [command]

  Options:

    -V, --version  output the version number
    -h, --help     output usage information

  Commands:

    boolean        Random.boolean( min,  max,  cur )
    bool           Random.bool( min,  max,  cur )
    natural        Random.natural( min,  max )
    integer        Random.integer( min,  max )
    int            Random.int( min,  max )
    float          Random.float( min,  max,  dmin,  dmax )
    character      Random.character( pool )
    char           Random.char( pool )
    string         Random.string( pool,  min,  max )
    str            Random.str()
    range          Random.range( start,  stop,  step )
    date           Random.date( format )
    time           Random.time( format )
    datetime       Random.datetime( format )
    now            Random.now( unit,  format )
    image          Random.image( size,  background,  foreground,  format,  text )
    img            Random.img()
    color          Random.color( name )
    hex            Random.hex()
    rgb            Random.rgb()
    rgba           Random.rgba()
    hsl            Random.hsl()
    paragraph      Random.paragraph( min,  max )
    cparagraph     Random.cparagraph( min,  max )
    sentence       Random.sentence( min,  max )
    csentence      Random.csentence( min,  max )
    word           Random.word( min,  max )
    cword          Random.cword( pool,  min,  max )
    title          Random.title( min,  max )
    ctitle         Random.ctitle( min,  max )
    first          Random.first()
    last           Random.last()
    name           Random.name( middle )
    cfirst         Random.cfirst()
    clast          Random.clast()
    cname          Random.cname()
    url            Random.url( protocol,  host )
    protocol       Random.protocol()
    domain         Random.domain( tld )
    tld            Random.tld()
    email          Random.email( domain )
    ip             Random.ip()
    region         Random.region()
    province       Random.province()
    city           Random.city( prefix )
    county         Random.county( prefix )
    zip            Random.zip( len )
    d4             Random.d4()
    d6             Random.d6()
    d8             Random.d8()
    d12            Random.d12()
    d20            Random.d20()
    d100           Random.d100()
    guid           Random.guid()
    uuid           Random.uuid()
    id             Random.id()
  Examples:

    $ random date yyyy-MM-dd
    $ random time HH:mm:ss

ES2015+

vuex

每一个 Vuex 应用的核心就是 store(仓库)。“store”基本上就是一个容器,它包含着你的应用中大部分的状态 (state)。

  • reactive in store state vars.
  • commit mutation

vue-router

Questions

  • import request from '@/utils/request' what is @?
  • requests interceptor 中 set login token
  • response interceptor 中验证 response 错误
  • arrow functions

    sidebar: state => state.app.sidebar,
    
  • store
    • getters
    • store modules
      const store = new Vuex.Store({
        modules: {
          app,
          user
        },
        getters
      })
      
      export default store
      
  • export function
  • router
    {
      path: '/',
      component: Layout,
      redirect: '/dashboard',
      name: 'Dashboard',
      hidden: true,
      children: [{
        path: 'dashboard',
        component: () => import('@/views/dashboard/index')
      }]
    },
    
  • vuex
    • getters
    • mapGetters
    • mutations

CSS

Box model

</img>


border-width:
border-top-width:
border-bottom-width:
border-left-width: 
border-right-width:

box-sizing: content-box/ border-box

line-width: normal/1.6/30px/0.8em

margin
margin-top
margin-bottom: 2px/3em/10%/auto
margin-left
margin-right

height:
max-height: none/2000px
max-width:
min-height: 0/200px
min-width:

padding
padding-top:
padding-bottom:
padding-left:
padding-right:

width: auto/240px/50%



overflow: auto/hidden/visible
background-cli: border-box/padding-box/content-box
display: block/inline/inline-block/positioned/table/flex
  • You can use unitless values: the line height will be relative to the font size. line-height: 1.6;
  • You can use pixel values. line-height: 30px;
  • You can use em values: like with unitless values, the line height will be relative to the font size. line-height: 0.8em;

Flexbox

Before the Flexbox Layout module, there were four layout modes:

  • Block, for sections in a webpage
  • Inline, for text
  • Table, for two-dimensional table data
  • Positioned, for explicit position of an element
flexbox
align-content: stretch/flex-start/flex-end/center/space-between/space-around/
align-items: flex-start/flex-end/center/baseline/stretch
align-self: auto/flex-start/flex-end/center/baseline/stretch
flex-basis: auto/80px/
flex-direction: row/row-reverse/column/column-reverse
flex-flow
flex-grow: 0/1
flex-shrink: 0/1
flex-wrap: nowrap/wrap/wrap-reverse
justify-conent: flex-start/flex-end/center/space-between/space-around
order: 0/1/-1/9

Grid

Node.js development stack

Node.js development stack

function projects  
package management npm  
CSS preprocessors less, sass  
JS module loader require.js, Browserfy  
JS transpiler coffee loader, ts-loader  
  • ES2015+
  • vue
  • vuex
  • vue-router
  • axios
  • element-ui
  • Mock.js

Node.js

https://www.w3schools.com/nodejs/default.asp

  • Node.js uses asynchronous programming!
  • Node.js can generate dynamic page content
  • Node.js can create, open, read, write, delete, and close files on the server
  • Node.js can collect form data
  • Node.js can add, delete, modify data in your database

Modules

  • What is a Module in Node.js?

    Consider modules to be the same as JavaScript libraries.A set of functions you want to include in your application.

  • built-in modules
Module Description
assert Provides a set of assertion tests
buffer To handle binary data
child_process To run a child process
cluster To split a single Node process into multiple processes
crypto To handle OpenSSL cryptographic functions
dgram Provides implementation of UDP datagram sockets
dns To do DNS lookups and name resolution functions
domain Deprecated. To handle unhandled errors
events To handle events
fs To handle the file system
http To make Node.js act as an HTTP server
https To make Node.js act as an HTTPS server.
net To create servers and clients
os Provides information about the operation system
path To handle file paths
punycode Deprecated. A character encoding scheme
querystring To handle URL query strings
readline To handle readable streams one line at the time
stream To handle streaming data
string_decoder To decode buffer objects into strings
timers To execute a function after a given number of milliseconds
tls To implement TLS and SSL protocols
tty Provides classes used by a text terminal
url To parse URL strings
util To access utility functions
v8 To access information about V8 (the JavaScript engine)
vm To compile JavaScript code in a virtual machine
zlib To compress or decompress files
  • Include Modules

    To include a module, use the require() function with the name of the module:

      var http = require('http');
    
  • Create Your Own Modules
      // save as mymodule.j* 
      exports.myDateTime = function () {
          return Date();
      };
    
      // in another main.js
      var mymodule = require('./mymodule')
      var datetime = mymodule.myDateTime()
    
  • HTTP Module
      var http = require('http');
    
      //create a server object:
      http.createServer(function (req, res) {
      res.write('Hello World!'); //write a response to the client
      res.end(); //end the response
      }).listen(8080);
    
    • Add an HTTP Header

      ` res.writeHead(200, {‘Content-Type’: ‘text/html’});`

    • Read the Query String

      res.write(req.url);

    • Split the Query String
        var q = url.parse(req.url, true).query;
        var txt = q.year + " " + q.month;
      
  • File System Module
    • Node.js as a File Server
        var fs = require('fs');
      
    • Read Files
        fs.readFile('demofile1.html', function(err, data) {
      
    • Create Files
        var fs = require('fs');
      
        fs.appendFile('mynewfile1.txt', 'Hello content!', function (err) {
            if (err) throw err;
            console.log('Saved!');
        });
      
    • Delete Files

        var fs = require('fs');
      
        fs.unlink('mynewfile2.txt', function (err) {
        if (err) throw err;
        console.log('File deleted!');
        });
      
      
    • rename files
        fs.rename('mynewfile1.txt', 'myrenamedfile.txt', function (err) {
      
    • Upload Files
        var formidable = require('formidable');
      
  • Send an Email
    • nodemailer module var nodemailer = require('nodemailer');
    • multiple receivers
    • html
  • Events module
    • addListener()
    • defaultMaxListeners
    • emit()
    • eventNames()
    • getMaxListeners()
    • listenerCount()
    • listeners()
    • on()
    • once()
    • prependListener()
    • prependOnceListener()
    • removeAllListeners()
    • removeListener()
    • setMaxListeners()
      var events = require('events');
      var eventEmitter = new events.EventEmitter();
    
      eventEmitter.on('scream', function() {
      console.log('A scream is detected!');
      });
      eventEmitter.emit('scream');
    
  • NPM
    • download: npm install upper-case --save
    • use: var uc = require('upper-case')
  • URL
      var url = require('url');
      var adr = 'http://localhost:8080/default.htm?year=2017&month=february';
      var q = url.parse(adr, true);
    
      console.log(q.host); //returns 'localhost:8080'
      console.log(q.pathname); //returns '/default.htm'
      console.log(q.search); //returns '?year=2017&month=february'
    
      var qdata = q.query; //returns an object: { year: 2017, month: 'february' }
      console.log(qdata.month); //returns 'february'
    

Node.js and MySQL

Node.js and MongoDB

  • install driver: npm install mongodb --save
  • var mongo = require('mongodb');
  • create collections
      var MongoClient = require('mongodb').MongoClient;
      var url = "mongodb://localhost:27017/";
    
      MongoClient.connect(url, function(err, db) {
          if (err) throw err;
          var dbo = db.db("mydb");
          dbo.createCollection("customers", function(err, res) {
              if (err) throw err;
              console.log("Collection created!");
              db.close();
          });
    
          var myobj = { name: "Company Inc", address: "Highway 37" };
          dbo.collection("customers").insertOne(myobj, function(err, res) {
              if (err) throw err;
              console.log("1 document inserted");
              db.close();
          });
            
      });
    

package.json

  • package.json

    The package.json file is core to the Node.js ecosystem and is a basic part of understanding and working with Node.js, npm, and even modern JavaScript. The package.json is used as what equates to a manifest about applications, modules, packages, and more - it’s a tool to that’s used to make modern development streamlined, modular, and efficient.

    • name
    • version
    • description
    • license
    • keywords
    • dependencies
    • devDependencies
    • “main”: “app.js”,
    • repository
    • scripts

Learn ES2015

https://old.babeljs.io/learn-es2015/

  • Arrows and Lexical This

Vue.js

Hello world

  1. import: <script src="https://unpkg.com/vue"></script>
  2. HTML ```html
3. javascript

```js
var app = new Vue({
  el: '#app',
  data: {
    message: 'hello, world'
  }

})

HTML 中用一个 mount point <div id="app">, 在 javascript 创建一个 Vue 示例。

HTML 渲染

  1. 文本插值: ``
  2. v-bind::title e.g. ` `
  3. v-on: @click 事件处理 ` `
  4. 条件: v-if
  5. 循环: v-for
  6. 用户输入: v-model: ` `

Component

// 定义名为 todo-item 的新组件
Vue.component('todo-item', {
  template: '<li>这是个待办项</li>'
})

Instead, a component’s data option must be a function, so that each instance can maintain an independent copy of the returned data object:

data: function () {
  return {
    count: 0
  }
}

Organizing components

  • registration:
    • global
      • Vue.component('name', opts)
      • Globally registered components can be used in the template of any root Vue instance (new Vue) created afterwards – and even inside all subcomponents of that Vue instance’s component tree.
    • local
  • Props
    • props is like interface to class, and data is used to maitain interal state, as private fields to class.
  • SFC

Vue.
  • Passing Data to Child Components with Props
  • Sending Messages to Parents with Events
  • v-model
  • Content Distribution with Slots
  • Dynamic Components <component v-bind:is="currentTabComponent"></component>

Vue class / instance

var vm = new Vue({

})

Options/*

  1. el: 锚点
  2. data: 要绑定的数据
  3. computed: 复杂属性
  4. props: 对外的属性接口,类似 C# /Android 中 view 等属性。
  5. methods: 方法

API

global config

  • silent
  • optionMergeStrategies
  • devtools
  • errorHandler
  • warnHandler
  • ignoredElements
  • keyCodes
  • performance
  • productionTip

Global API

  • Vue.extend(options)
  • Vue.nextTick([callback, context])
  • Vue.set(target, key, value)
  • Vue.delete(target, key)
  • Vue.directive(id, [definition])
  • Vue.filter(id, [definition])
  • Vue.component(id, [definition])
  • Vue.use(plugin)
  • Vue.mixin(mixin)
  • Vue.compile(template)
  • Vue.version

Options/data

  • data
  • props: A list/hash of attributes that are exposed to accept data from the parent component.
  • propsData: initial / default data for props
  • computed:

events

  • $emit

synthetic events Creating and triggering events

vm.$emit( eventName, […args] )

Questions

  1. what is javascript prototype?

    fdf

  2. what is javascript context this?

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

javascript context/ scope/ this

  • context/ this: function

context means this keyword

Context is most often determined by how a function is invoked. When a function is called as a method of an object, this is set to the object the method is called on:

So for a global function, this refers to ‘window’ in browser

  • scope: variable
    • local
    • Global
    • block: let/ const

    Understanding Scope and Context in JavaScript

  • Execution Context

  • Execution stack

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

Vue-cli

https://cli.vuejs.org/

new 3.x is @vue/cli, the old is vue-cli

  • instant prototyping
      vue serve
    
  • Create a project
      vue create
    
  • Plugins
  • Presets
  • CLI service

UI

img

  • UI template

      vue ui
    

vue-cli-service

➜  hello-vue git:(master) ✗ ./node_modules/.bin/vue-cli-service 

  Usage: vue-cli-service <command> [options]

  Commands:

    serve     start development server
    build     build for production
    inspect   inspect internal webpack config
    lint      lint and fix source files

  run vue-cli-service help [command] for usage of a specific command.
  • Using the Binary
  • vue-cli-service serve
  • vue-cli-service build
  • vue-cli-service inspect
  • Checking All Available Commands
  • Caching and Parallelization
  • Git Hooks
  • Configuration without Ejecting
  • webpack-dev-server

MISCS

  • UMD
  • AMD
  • CommonJS is a project with the goal of specifying an ecosystem for JavaScript outside the browser (for example, on the server or for native desktop applications).

vue-loader

是一个 webpack 的 loader,可以将用下面这个格式编写的 Vue 组件转换为 JavaScript 模块:

AMD v.s. CommonJS v.s. UMD

https://www.davidbcalhoun.com/2014/what-is-amd-commonjs-and-umd/

AMD v.s. CommonJS

Sandboxed

Asynchronous Module Definition

  • AMD <- RequireJS
      //    filename: foo.js
      define(['jquery', 'underscore'], function ($, _) {
          //    methods
          function a(){};    //    private because it's not returned (see below)
          function b(){};    //    public because it's returned
          function c(){};    //    public because it's returned
    
          //    exposed public methods
          return {
              b: b,
              c: c
          }
      });
    
  • CommonJS <- Browserify

      //    filename: foo.js
      var $ = require('jquery');
      var _ = require('underscore');
    
      //    methods
      function a(){};    //    private because it's omitted from module.exports (see below)
      function b(){};    //    public because it's defined in module.exports
      function c(){};    //    public because it's defined in module.exports
    
      //    exposed public methods
      module.exports = {
          b: b,
          c: c
      };
    
  • UMD: The pattern is admittedly ugly, but is both AMD and CommonJS compatible, as well as supporting the old-style “global” variable definition:

      (function (root, factory) {
          if (typeof define === 'function' && define.amd) {
              // AMD
              define(['jquery'], factory);
          } else if (typeof exports === 'object') {
              // Node, CommonJS-like
              module.exports = factory(require('jquery'));
          } else {
              // Browser globals (root is window)
              root.returnExports = factory(root.jQuery);
          }
      }(this, function ($) {
          //    methods
          function myFunc(){};
    
          //    exposed public method
          return myFunc;
      }));
    

MISC

  • js === v.s. ==

##

vuex

vuex

vuex 集中管理应用中的状态。 类似 Java Spring 中的配置中心。 一种管理的哲学思想 commit 的方式相当于 会隐藏了 setOnListener 的调用, 类似 set 方法, 而不是直接访问状态字段

  • state
  • mutations
  • 触发变化 store.commit('increment')
  • 调用状态: 计算属性
     computed: {
      count () {
        return store.state.count
      }
    }
    
  • 单一状态树, 唯一数据源 SSOT, 状态快照

So actions/mutations allow you to:

  • centrally define allowed changes to the state, and by that,
  • how those changes are connected and logically depend on one another (i.e. by reading an action that commits a series of mutations)
  • track these changes in the devtools very easily, as you see all the Mutation names you just read in the list there.
  • writing expressive code that, to an extend, documents your business logic. You can understand a lot about an app’s business logic by just reading its store actions & mutations, without digging through hundreds of components looking where some piece of state is altered under which circumstances.

  • mapGetter 类同 Java Bean 中的自动生成 get 函数
  • mapMutations 类同 Java Bean 中自动生成 set 函数
  • 可以手写 get / set 函数, 以定制化
    • get: getters:
    • set: mutations:
    • mutaions = event + event handler (type + handler)
    • computed: mapState
  • Actions
    • …mapActions

MISC

  • setInterval
  • clearInterval
  • setTimeout clearTimeout

Spring Jackson

Jackson Exceptions – Problems and Solutions

JsonMappingException UnrecognizedPropertyException

Spring Mongo

  • root Spring Data’s MongoDB abstraction.
  • .config Spring XML namespace configuration for MongoDB specific repositories.
  • .core MongoDB core support.
  • .core.aggregation Support for the MongoDB aggregation framework.
  • .core.convert Spring Data MongoDB specific converter infrastructure.
  • .core.geo Support for MongoDB geo-spatial queries.
  • .core.index Support for MongoDB document indexing.
  • .core.mapping Infrastructure for the MongoDB document-to-object mapping subsystem.
  • .core.mapping.event Mapping event callback infrastructure for the MongoDB document-to-object mapping subsystem.
  • .core.mapreduce Support for MongoDB map-reduce operations.
  • .core.query MongoDB specific query and update support.
  • .core.script Abstraction classes javascript function execution within MongoDB Server.
  • .core.spel Support classes to transform SpEL expressions into MongoDB expressions.
  • .crossstore Infrastructure for Spring Data’s MongoDB cross store support.
  • .gridfs Support for MongoDB GridFS feature.
  • .monitor MongoDB specific JMX monitoring support.
  • .repository MongoDB specific repository implementation.
  • .repository.cdi CDI support for MongoDB specific repository implementation.
  • .repository.config Support infrastructure for the configuration of MongoDB specific repositories.
  • .repository.query Query derivation mechanism for MongoDB specific repositories.
  • .repository.support Support infrastructure for query derivation of MongoDB specific repositories.
  • .util MongoDB driver-specific utility classes for Bson and DBObject interaction.

修改错误

{
    "timestamp": "2018-06-08T06:47:54.222+0000",
    "status": 500,
    "error": "Internal Server Error",
    "message": "Could not extract response: no suitable HttpMessageConverter found for response type [class com.google.gson.JsonObject] and content type [text/plain]",
    "path": "/auth/login/wechat"
}

LetsEncrypt

https://certbot.eff.org/lets-encrypt/ubuntutrusty-nginx

Install LetsEncrypt

root@open2:/etc/nginx/sites-enabled# ls
city.opentown.cn  opentown.cn
root@open2:/etc/nginx/sites-enabled# git diff city.opentown.cn
root@open2:/etc/nginx/sites-enabled# vim city.opentown.cn 
root@open2:/etc/nginx/sites-enabled# certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: opentown.cn
2: adunk.opentown.cn
3: api.opentown.cn
4: cityapi.opentown.cn
5: m.opentown.cn
6: www.opentown.cn
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 4
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for cityapi.opentown.cn
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/city.opentown.cn

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://cityapi.opentown.cn

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=cityapi.opentown.cn
-------------------------------------------------------------------------------

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/cityapi.opentown.cn/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/cityapi.opentown.cn/privkey.pem
   Your cert will expire on 2018-09-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Ethereum and Solidity

Tooling

  • Remix
  • npm / Node.js

web3j web3.js

Introduction to Smart Contracts

Solidity 是 Ethereum 上智能合约实现的主要语言之一。 运行在 EVM 上。

  • static typing
  • variadic return types
  • Complex member variables for contracts including arbitrarily hierarchical mappings and structs
  • Contracts support inheritance, including multiple inheritance with C3 linearization.
  • An application binary interface (ABI) facilitating multiple type-safe functions within a single contract was also introduced (and later supported by Serpent).
  • A documentation system for specifying a user-centric description of the ramifications of a method-call was also included in the proposal, known as “Natural Language Specification”.
  • on EVM
  • compile to bytecode
  • designed around ECMAScript

Blockchain Basics

  • Transactions
  • Blocks
  • EVM
    • Accounts: external accounts controlled by public private key pairs
    • contract accounts controlled by the code stored together with the account.
    • Every account has a persistent key-value store mapping 256-bit words to 256-bit words called storage.
  • Storage:a key-value store that maps 256-bit words to 256-bit words. It is not possible to enumerate storage from within a contract and it is comparatively costly to read and even more so, to modify storage. A contract can neither read nor write to any storage apart from its own.
  • Memory: a contract obtains a freshly cleared instance for each message call. Memory is linear and can be addressed at byte level, but reads are limited to a width of 256 bits, while writes can be either 8 bits or 256 bits wide. Memory is expanded by a word (256-bit), when accessing (either reading or writing) a previously untouched memory word (ie. any offset within a word). At the time of expansion, the cost in gas must be paid. Memory is more costly the larger it grows (it scales quadratically).
  • Stack
  • Instruction Set
  • Message Calls
  • Delegatecall / Callcode and Libraries
  • Logs
  • Create
  • self-destruct

Installing the Solidity Compiler

Solidity by Example

Solidity in Depth

Security Considerations

Using the compiler

Contract Metadata

Application Binary Interface Specification

Joyfully Universal Language for (Inline) Assembly

Style Guide

Common Patterns

List of Known Bugs

Contributing

Frequently Asked Questions

##

Gloabal variable

Ether Units

  • wei,
  • finney,
  • szabo or
  • ether

Time Units

  • 1 == 1 seconds
  • 1 minutes == 60 seconds
  • 1 hours == 60 minutes
  • 1 days == 24 hours
  • 1 weeks == 7 days
  • 1 years == 365 days

Globals

  • block.blockhash(uint blockNumber) returns (bytes32): hash of the given block - only works for 256 most recent blocks excluding current
  • block.coinbase (address): current block miner’s address
  • block.difficulty (uint): current block difficulty
  • block.gaslimit (uint): current block gaslimit
  • block.number (uint): current block number
  • block.timestamp (uint): current block timestamp as seconds since unix epoch
  • gasleft() returns (uint256): remaining gas
  • msg.data (bytes): complete calldata
  • msg.gas (uint): remaining gas - deprecated in version 0.4.21 and to be replaced by gasleft()
  • msg.sender (address): sender of the message (current call)
  • msg.sig (bytes4): first four bytes of the calldata (i.e. function identifier)
  • msg.value (uint): number of wei sent with the message
  • now (uint): current block timestamp (alias for block.timestamp)
  • tx.gasprice (uint): gas price of the transaction
  • tx.origin (address): sender of the transaction (full call chain)

Error Handling

  • assert(bool condition): throws if the condition is not met - to be used for internal errors.
  • require(bool condition): throws if the condition is not met - to be used for errors in inputs or external components.
  • revert(): abort execution and revert state changes

Mathematical and Cryptographic Functions

  • addmod(uint x, uint y, uint k) returns (uint):
  • mulmod(uint x, uint y, uint k) returns (uint):
  • keccak256(…) returns (bytes32):
  • sha256(…) returns (bytes32):
  • sha3(…) returns (bytes32):
  • ripemd160(…) returns (bytes20):
  • ecrecover(bytes32 hash, uint8 v, bytes32 r, bytes32 s) returns (address):
  • <address>.balance (uint256):
  • <address>.transfer(uint256 amount):
  • <address>.send(uint256 amount) returns (bool):
  • <address>.call(...) returns (bool):
  • <address>.callcode(...) returns (bool):
  • <address>.delegatecall(...) returns (bool):
  • this
  • selfdestruct(address recipient)
  • suicide(address recipient)

Doxity: Documentation Generator for Solidity.

All identifiers (contract names, function names and variable names) are restricted to the ASCII character set. It is possible to store UTF-8 encoded data in string variables.

  • public: The keyword public automatically generates a function that allows you to access the current value of the state variable from outside of the contract. Without this keyword, other contracts have no way to access the variable.
  • address: 160 bit value
  • mapping: Mappings can be seen as hash tables which are virtually initialized such that every possible key exists and is mapped to a value whose byte-representation is all zeros.
  • progma once
  • Coin.balances.call(arg)

How to develop and debug?

EVM

  • 256bit register stack
  • e-WASM
  • send()
  • transfer()
  • call.value()

ERC20

pragma solidity ^0.4.23;


// https://github.com/ethereum/EIPs/issues/20
interface ERC20 {
    function totalSupply() external view returns (uint supply);
    function balanceOf(address _owner) external view returns (uint balance);
    function transfer(address _to, uint _value) external returns (bool success);
    function transferFrom(address _from, address _to, uint _value) external returns (bool success);
    function approve(address _spender, uint _value) external returns (bool success);
    function allowance(address _owner, address _spender) external view returns (uint remaining);
    function decimals() external view returns(uint digits);
    event Approval(address indexed _owner, address indexed _spender, uint _value);
}
import java.util.HashMap;
import java.util.Map;

public class ERC20Coin {

    static class Address {
        private String val;
        private static final String PATTERN_ERC20_HASH = "0x[0-9A-Fa-f]{64}";

        public Address(String val) {
            assert (val.matches(PATTERN_ERC20_HASH));
            this.val = val;
        }

        public String toString() {
            return this.val;
        }

        @Override
        public boolean equals(Object obj) {
            return (obj instanceof Address)
                    && ((Address) obj).val.equals(this.val);
        }
    }

    static class TransferEvent{
        public TransferEvent(Address _from, Address _to, Integer _value){

        }
    }

    Address sender;


    String name;
    String symbol;
    long totalSupply;
    Integer demicals;

    Map<Address, Integer> balances;
    Map<Address, Map<Address, Integer>> allowances;


    public ERC20Coin(String name, String symbol, long totalSupply, Integer demicals){
        balances = new HashMap<>();
        allowances = new HashMap<>();
        this.name = name;
        this.symbol = symbol;
        this.totalSupply = totalSupply;
        this.demicals = demicals;
    }

    public Integer balanceOf(Address address) {
        return balances.get(address);
    }

    public void transfer(Address from, Address to, Integer val) {
        Integer balanceFrom = balances.getOrDefault(from, 0);
        Integer balanceTo = balances.getOrDefault(to, 0);

        Map<Address, Integer> allowance;
        if(!allowances.containsKey(from)){
            allowance = new HashMap<Address, Integer>();
        }else{
            allowance = allowances.get(from);
        }
        Integer allowed = allowance.get(sender);
        assert  (allowance.get(sender) >= val) &&
                (balanceFrom >= val);

        if(allowed < Integer.MAX_VALUE){
            allowance.put(sender, allowed - val);
            allowances.put(from, allowance);
        }

        balances.put(from, balanceFrom - val);
        balances.put(to, balanceTo + val);
    }

    public void approve(Address from, Address to, Integer val){
        Map<Address, Integer> allowance;
        if(!allowances.containsKey(from)){
            allowance = new HashMap<Address, Integer>();
        }else{
            allowance = allowances.get(from);
        }

        allowance.put(to, val);
        allowances.put(from, allowance);
    }



}

Airdrop 空投

参考: https://medium.com/bitfwd/how-to-airdrop-your-first-token-3ba187ff7cbf

空投地址列表

空投之前需要找到你需要空投地址列表。 如何找到这些地址列表呢?

  1. 在 Slack 或者 telegram 等群里寻找鱼儿。
  2. 扫描 Ethereum 上 balance >0 的那些有效账号地址
parity export state --chain=ropsten -no-storage -no-code -min-balance=1000 -max-balance=500000 -at=2000000 balances.json

节省 gas 费用

如何一次向多个账号send token?

Spring Security

Servelet Interceptor

authenticationManager, accessDecisionManager, securityMetadataSource

UserDetailsService接口和其loadUserByUsername方法

1)首先我们自定义一个过滤器(调度器,这里我们命名为mySecurityInterceptor),这个过滤器继承AbstractSecurityInterceptor类(这里先说明,本文但凡不是自定义的类或接口都是Spring Security提供的,无须深究)。 它至少包含 authenticationManager,accessDecisionManager,securityMetadataSource三个属性,我们的所有控制将在这三个类中实现。

(2)登录验证:自定义类MyUserDetailService实现UserDetailsService接口和其loadUserByUsername方法,这个方法根据用户输入的用户名,从数据库里面获取该用户的所有权限细信息(统称用户信息)。Spring Security的AuthenticationProcessingFilter拦截器调用authenticationManager,类MyUserDetailService拿到用户信息后,authenticationManager对比用户的密码(即验证用户),如果通过了,那么相当于通过了AuthenticationProcessingFilter拦截器,也就是登录验证通过。

(3)资源访问控制:MySecurityInterceptor继承AbstractSecurityInterceptor、实现Filter是必须的。登陆后,每次访问资源都会被MySecurityInterceptor这个拦截器拦截,它首先会调用MyFilterInvocationSecurityMetadataSource类的getAttributes方法获取被拦截url所需的权限,在调用MyAccessDecisionManager类decide方法判断用户是否够权限。

可能文字描述还是比较抽象,通过实例应该能让大家更加清楚其原理。

UserDetailsService在身份认证中的作用:

Spring Security中进行身份验证的是AuthenticationManager接口,ProviderManager是它的一个默认实现,但它并不用来处理身份认证,而是委托给配置好的AuthenticationProvider,每个AuthenticationProvider会轮流检查身份认证。检查后或者返回Authentication对象或者抛出异常。

验证身份就是加载响应的UserDetails,看看是否和用户输入的账号、密码、权限等信息匹配。此步骤由实现AuthenticationProvider的DaoAuthenticationProvider(它利用UserDetailsService验证用户名、密码和授权)处理。

因此,登录认证其实可以不实现UserDetailsService,而是实现AuthenticationProvider,然后在AuthenticationProvider里面获取用户输入的用户名和密码进行校验也是可以的。或者两者一起使用。

UsernamePasswordAuthenticationToken

http://up.2cto.com/kf/201103/20110314161540730.png

Python multitasking

threading

Thread

  • Methods:
    • start()
    • run()
    • is_alive()
    • join()

Threads are executed in their own system-level thread (e.g., a POSIX thread or Windows threads) that is fully managed by the host operating system. Once started, threads run independently until the target function returns.

from threading import Thread
t = Thread(target=countdown, args=(10,))
t.start()

Due to a global interpreter lock (GIL), Python threads are restricted to an execution model that only allows one thread to execute in the interpreter at any given time. For this reason, Python threads should generally not be used for computationally intensive tasks where you are trying to achieve parallelism on multiple CPUs. They are much better suited for I/O handling and handling concurrent execution in code that performs blocking operations (e.g., waiting for I/O, waiting for results from a database, etc.).

  • Event
    • wait()
    • set()
  • Condition
    • wait()
    • notify_all()
  • Semaphore
sema = threading.Semaphore(0)
sema.release()
  • Lock
self._value_lock = threading.Lock()
with self._value_lock:
        self._value += delta
  • RLock

An RLock or re-entrant lock object is a lock that can be acquired multiple times by the same thread. It is primarily used to implement code based locking or synchronization based on a construct known as a “monitor.” With this kind of locking, only one thread is allowed to use an entire function or the methods of a class while the lock is held.

  • threading.local()

Queue

Perhaps the safest way to send data from one thread to another is to use a Queue from the queue library. To do this, you create a Queue instance that is shared by the threads. Threads then use put() or get() operations to add or remove items from the queue.

  • q.put(data)
  • data = in_q.get()
  • in_q.task_done()
  • q.qsize()
  • q.full()
  • q.empty()

Actor

https://www.oreilly.com/learning/python-cookbook-concurrency#actors

In a nutshell, an actor is a concurrently executing task that simply acts upon messages sent to it. In response to these messages, it may decide to send further messages to other actors. Communication with actors is one way and asynchronous. Thus, the sender of a message does not know when a message actually gets delivered, nor does it receive a response or acknowledgment that the message has been processed.

multiprocessing

multiprocessing

  • Process
  • Exchanging objects between processes

Pool

class Pool(object):
    Process = Process

    def __init__(self, processes=None, initializer=None, initargs=(),
                 maxtasksperchild=None):
        self._setup_queues()
        self._taskqueue = Queue.Queue()
        self._cache = {}
        ... # stuff we don't care about
        self._worker_handler = threading.Thread(
            target=Pool._handle_workers,
            args=(self, )
            )
        self._worker_handler.daemon = True
        self._worker_handler._state = RUN 
        self._worker_handler.start()

        self._task_handler = threading.Thread(
            target=Pool._handle_tasks,
            args=(self._taskqueue, self._quick_put, self._outqueue,
                  self._pool, self._cache)
            )
        self._task_handler.daemon = True
        self._task_handler._state = RUN 
        self._task_handler.start()

        self._result_handler = threading.Thread(
            target=Pool._handle_results,
            args=(self._outqueue, self._quick_get, self._cache)
            )
        self._result_handler.daemon = True
        self._result_handler._state = RUN
        self._result_handler.start()
  • queues:
    • self._taskqueue = Queue.Queue(): cache all submited tasks
    • self._inqueue = SimpleQueue(): _task_handler fetch task from _taskqueue and add to _inqueue for process to work on, this is shared among main process and worker process.
    • self._outqueue = SimpleQueue(): store the result.
  • threads:
    • _worker_handler = _handle_workers(pool): maintain the internal state
    • _task_handler = _handle_tasks(taskqueue, put, outqueue, pool, cache): fetch tasks and send to worker process
    • _result_handler = _handle_results(outqueue, get, cache): process result, callback etc.

Note: callback is called in _result_handler thread in caller process.

concurrent.futures

Executor

  • Executor.submit(fn, *args, **kwargs)
  • Executor.map(func, *iterables, timeout=None)
  • Executor.shutdown(wait=True)

Difference

  • Use the ProcessPoolExecutor for CPU intensive tasks.
  • The ThreadPoolExecutor is better suited for network operations or I/O.

ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=2)

ProcessPoolExecutor

executor = concurrent.futures.ProcessPoolExecutor(max_workers=None)

"""Implements ProcessPoolExecutor.

The follow diagram and text describe the data-flow through the system:

|======================= In-process =====================|== Out-of-process ==|

+----------+     +----------+       +--------+     +-----------+    +---------+
|          |  => | Work Ids |    => |        |  => | Call Q    | => |         |
|          |     +----------+       |        |     +-----------+    |         |
|          |     | ...      |       |        |     | ...       |    |         |
|          |     | 6        |       |        |     | 5, call() |    |         |
|          |     | 7        |       |        |     | ...       |    |         |
| Process  |     | ...      |       | Local  |     +-----------+    | Process |
|  Pool    |     +----------+       | Worker |                      |  #1..n  |
| Executor |                        | Thread |                      |         |
|          |     +----------- +     |        |     +-----------+    |         |
|          | <=> | Work Items | <=> |        | <=  | Result Q  | <= |         |
|          |     +------------+     |        |     +-----------+    |         |
|          |     | 6: call()  |     |        |     | ...       |    |         |
|          |     |    future  |     |        |     | 4, result |    |         |
|          |     | ...        |     |        |     | 3, except |    |         |
+----------+     +------------+     +--------+     +-----------+    +---------+

Executor.submit() called:
- creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict
- adds the id of the _WorkItem to the "Work Ids" queue

Local worker thread:
- reads work ids from the "Work Ids" queue and looks up the corresponding
  WorkItem from the "Work Items" dict: if the work item has been cancelled then
  it is simply removed from the dict, otherwise it is repackaged as a
  _CallItem and put in the "Call Q". New _CallItems are put in the "Call Q"
  until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because
  calls placed in the "Call Q" can no longer be cancelled with Future.cancel().
- reads _ResultItems from "Result Q", updates the future stored in the
  "Work Items" dict and deletes the dict entry

Process #1..n:
- reads _CallItems from "Call Q", executes the calls, and puts the resulting
  _ResultItems in "Request Q"
"""

Future

  • Future.cancel()
  • Future.cancelled()
  • Future.running()
  • Future.result(timeout=None)
  • Future.exception(timeout=None)
  • Future.add_done_callback(fn)

Internal methods, meant for use in unit tests and Executor implementations.

  • Future.set_running_or_notify_cancel()
  • Future.set_result(result)
  • Future.set_exception(exception)

module functions

  • concurrent.futures.wait(fs, timeout=None, return_when=ALL_COMPLETED)
  • concurrent.futures.as_completed(fs, timeout=None)

The main difference between the aforementioned map method with as_completed is that map returns the results in the order in which we pass the iterables. That is the first result from the map method is the result for the first item. On the other hand, the first result from the as_completed function is from whichever future completed first.

#Launching a Daemon Process on Unix

Debugging Python Multi-processes Program

Take aways

  • Another subtle aspect of pools is that mixing threads and process pools together can be a good way to make your head explode. If you are going to use both of these features together, it is often best to create the process pool as a singleton at program startup, prior to the creation of any threads. Threads will then use the same process pool for all of their computationally intensive work.

  • Many programmers, when faced with thread performance problems, are quick to blame the GIL for all of their ills. However, doing so is shortsighted and naive. Just as a real-world example, mysterious “stalls” in a multithreaded network program might be caused by something entirely different (e.g., a stalled DNS lookup) rather than anything related to the GIL. The bottom line is that you really need to study your code to know if the GIL is an issue or not. Again, realize that the GIL is mostly concerned with CPU-bound processing, not I/O.

  • Great care should be made when combining process pools and programs that use threads. In particular, you should probably create and launch process pools prior to the creation of any threads (e.g., create the pool in the main thread at program startup).

## References

TypeScript

TypeScript

TypeScript is a superset for javascript, enabling us to write more elegant javascript code.

TypeScript in 5 minutes

Process

TypeScript –> Javascript –> Nodejs

  1. use tsc to transpile TypeScript into JavaScript,
  2. then run js by nodejs or browser.

Install

npm install -g typescript
tsc --version
Version 2.7.2

tsc --help

Project management

structure

See typescript101

➜  typescript101 git:(master) ✗ tree
.
├── HelloWorld.js
├── HelloWorld.js.map
├── HelloWorld.ts
├── README.md
├── greeter.html
├── greeter.js
├── greeter.js.map
├── greeter.ts
└── tsconfig.json

With Visual Studio Code

Warning: Visual Studio for Mac Community Edition does not support TypeScript natively. So if on Mac, use Visual Studio Code instead. or Use Visual Studio on Windows for better IDE support.

References:

  1. Editing TypeScript in VSCode
  2. Simplest TypeScript project using Visual Studio Code
    • File Scope
    • Explicit Project

tsconfig.json

This is the project definition file.

Spring boot

Sping Boot

Dependency Management

The curated list contains all the spring modules that you can use with Spring Boot as well as a refined list of third party libraries. The list is available as a standard Bills of Materials (spring-boot-dependencies) that can be used with both Maven and Gradle.


BUILD FAILED in 4s
➜  NutNetwork git:(master) ✗ ./gradlew dependencies 
/Users/lucas/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.15/1483490805e8f86c21240958629905e733d079eb/log4j-1.2.15.pom:
    'build.plugins.plugin[io.spring.gradle.dependencymanagement.org.apache.maven.plugins:maven-antrun-plugin].dependencies.dependency.scope' for junit:junit:jar must be one of [compile, runtime, system] but is 'test'. in log4j:log4j:1.2.15
+--- org.springframework.boot:spring-boot-starter-data-rest -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot:2.0.0.RELEASE
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE
|    |    |    |    \--- org.springframework:spring-jcl:5.0.4.RELEASE
|    |    |    \--- org.springframework:spring-context:5.0.4.RELEASE
|    |    |         +--- org.springframework:spring-aop:5.0.4.RELEASE
|    |    |         |    +--- org.springframework:spring-beans:5.0.4.RELEASE
|    |    |         |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         \--- org.springframework:spring-expression:5.0.4.RELEASE
|    |    |              \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.0.0.RELEASE
|    |    |    \--- org.springframework.boot:spring-boot:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-logging:2.0.0.RELEASE
|    |    |    +--- ch.qos.logback:logback-classic:1.2.3
|    |    |    |    +--- ch.qos.logback:logback-core:1.2.3
|    |    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    |    +--- org.apache.logging.log4j:log4j-to-slf4j:2.10.0
|    |    |    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    |    |    \--- org.apache.logging.log4j:log4j-api:2.10.0
|    |    |    \--- org.slf4j:jul-to-slf4j:1.7.25
|    |    |         \--- org.slf4j:slf4j-api:1.7.25
|    |    +--- javax.annotation:javax.annotation-api:1.3.2
|    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.yaml:snakeyaml:1.19
|    +--- org.springframework.boot:spring-boot-starter-json:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework:spring-web:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.6.3
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0 -> 2.6.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    |    \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.4
|    |         +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |         \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    +--- org.springframework.boot:spring-boot-starter-web:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-json:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-tomcat:2.0.0.RELEASE
|    |    |    +--- javax.annotation:javax.annotation-api:1.3.2
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-core:8.5.28
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-el:8.5.28
|    |    |    \--- org.apache.tomcat.embed:tomcat-embed-websocket:8.5.28
|    |    |         \--- org.apache.tomcat.embed:tomcat-embed-core:8.5.28
|    |    +--- org.hibernate.validator:hibernate-validator:6.0.7.Final
|    |    |    +--- javax.validation:validation-api:2.0.1.Final
|    |    |    +--- org.jboss.logging:jboss-logging:3.3.0.Final -> 3.3.2.Final
|    |    |    \--- com.fasterxml:classmate:1.3.1 -> 1.3.4
|    |    +--- org.springframework:spring-web:5.0.4.RELEASE (*)
|    |    \--- org.springframework:spring-webmvc:5.0.4.RELEASE
|    |         +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|    |         \--- org.springframework:spring-web:5.0.4.RELEASE (*)
|    \--- org.springframework.data:spring-data-rest-webmvc:3.0.5.RELEASE
|         +--- org.springframework.data:spring-data-rest-core:3.0.5.RELEASE
|         |    +--- org.springframework:spring-tx:5.0.4.RELEASE
|         |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         |    +--- org.springframework.hateoas:spring-hateoas:0.23.0.RELEASE -> 0.24.0.RELEASE
|         |    |    +--- org.springframework:spring-aop:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-beans:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-context:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-core:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-web:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-webmvc:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.25
|         |    +--- org.springframework.data:spring-data-commons:2.0.5.RELEASE
|         |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.25
|         |    +--- org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE
|         |    |    +--- org.springframework:spring-beans:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-context:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-aop:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.10 -> 1.7.25
|         |    +--- org.atteo:evo-inflector:1.2.2
|         |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.4 -> 2.6.3
|         |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.4 (*)
|         |    \--- org.slf4j:slf4j-api:1.7.25
|         +--- org.springframework:spring-webmvc:5.0.4.RELEASE (*)
|         +--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|         +--- com.fasterxml.jackson.core:jackson-annotations:2.9.4 -> 2.6.3
|         \--- org.slf4j:slf4j-api:1.7.25
+--- org.springframework.boot:spring-boot-starter-data-jpa -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-aop:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    \--- org.aspectj:aspectjweaver:1.8.13
|    +--- org.springframework.boot:spring-boot-starter-jdbc:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- com.zaxxer:HikariCP:2.7.8
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    \--- org.springframework:spring-jdbc:5.0.4.RELEASE
|    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |         \--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    +--- org.hibernate:hibernate-core:5.2.14.Final
|    |    +--- org.jboss.logging:jboss-logging:3.3.1.Final -> 3.3.2.Final
|    |    +--- org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final
|    |    +--- org.javassist:javassist:3.22.0-GA
|    |    +--- antlr:antlr:2.7.7
|    |    +--- org.jboss:jandex:2.0.3.Final
|    |    +--- com.fasterxml:classmate:1.3.0 -> 1.3.4
|    |    +--- dom4j:dom4j:1.6.1
|    |    \--- org.hibernate.common:hibernate-commons-annotations:5.0.1.Final
|    |         \--- org.jboss.logging:jboss-logging:3.3.0.Final -> 3.3.2.Final
|    +--- javax.transaction:javax.transaction-api:1.2
|    +--- org.springframework.data:spring-data-jpa:2.0.5.RELEASE
|    |    +--- org.springframework.data:spring-data-commons:2.0.5.RELEASE (*)
|    |    +--- org.springframework:spring-orm:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-jdbc:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- org.springframework:spring-aspects:5.0.4.RELEASE
|         \--- org.aspectj:aspectjweaver:1.8.13
+--- org.springframework.boot:spring-boot-starter-thymeleaf -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.thymeleaf:thymeleaf-spring5:3.0.9.RELEASE
|    |    +--- org.thymeleaf:thymeleaf:3.0.9.RELEASE
|    |    |    +--- org.attoparser:attoparser:2.0.4.RELEASE
|    |    |    +--- org.unbescape:unbescape:1.1.5.RELEASE
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- org.thymeleaf.extras:thymeleaf-extras-java8time:3.0.1.RELEASE
|         +--- org.thymeleaf:thymeleaf:3.0.0.RELEASE -> 3.0.9.RELEASE (*)
|         \--- org.slf4j:slf4j-api:1.6.6 -> 1.7.25
+--- org.springframework.boot:spring-boot-starter-security -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    +--- org.springframework.security:spring-security-config:5.0.3.RELEASE
|    |    +--- org.springframework.security:spring-security-core:5.0.3.RELEASE
|    |    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    \--- org.springframework.security:spring-security-web:5.0.3.RELEASE
|         +--- org.springframework.security:spring-security-core:5.0.3.RELEASE (*)
|         +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|         \--- org.springframework:spring-web:5.0.4.RELEASE (*)
+--- org.springframework.boot:spring-boot-starter-data-redis -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework.data:spring-data-redis:2.0.5.RELEASE
|    |    +--- org.springframework.data:spring-data-keyvalue:2.0.5.RELEASE
|    |    |    +--- org.springframework.data:spring-data-commons:2.0.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-oxm:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context-support:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- io.lettuce:lettuce-core:5.0.2.RELEASE
|         +--- io.projectreactor:reactor-core:3.1.4.RELEASE -> 3.1.5.RELEASE
|         |    \--- org.reactivestreams:reactive-streams:1.0.2
|         +--- io.netty:netty-common:4.1.21.Final -> 4.1.22.Final
|         +--- io.netty:netty-transport:4.1.21.Final -> 4.1.22.Final
|         |    +--- io.netty:netty-buffer:4.1.22.Final
|         |    |    \--- io.netty:netty-common:4.1.22.Final
|         |    \--- io.netty:netty-resolver:4.1.22.Final
|         |         \--- io.netty:netty-common:4.1.22.Final
|         \--- io.netty:netty-handler:4.1.21.Final -> 4.1.22.Final
|              +--- io.netty:netty-buffer:4.1.22.Final (*)
|              +--- io.netty:netty-transport:4.1.22.Final (*)
|              \--- io.netty:netty-codec:4.1.22.Final
|                   \--- io.netty:netty-transport:4.1.22.Final (*)
+--- org.springframework.boot:spring-boot-starter-web -> 2.0.0.RELEASE (*)
+--- com.h2database:h2 -> 1.4.196
+--- org.mybatis:mybatis:3.3.0
+--- org.mybatis:mybatis-spring:1.2.3
+--- com.mchange:c3p0:0.9.5.1
|    \--- com.mchange:mchange-commons-java:0.2.10
+--- mysql:mysql-connector-java:5.1.31

Code Structure

com
 +- example
     +- myapplication
         +- Application.java
         |
         +- customer
         |   +- Customer.java
         |   +- CustomerController.java
         |   +- CustomerService.java
         |   +- CustomerRepository.java
         |
         +- order
             +- Order.java
             +- OrderController.java
             +- OrderService.java
             +- OrderRepository.java

Algorithm

Divide and Conqure

Steps

  1. divide n -> P1, P2, … Pk
  2. 最小子问题直接求解
  3. 递归/ 迭代求解子问题
  4. 合并子问题的解

Examples

Binary search a sorted array a[i]

  1. divide: n -> (0 – n/2) (n/2 – n)
  2. 最小子问题直接求解: n = 0, 1
  3. 递归/ 迭代求解子问题: each step, search either half.
  4. 合并子问题的解: 最后一个子问题的解就是整个问题的解

求解法

递归

递归是最简单的形式

迭代

部分递归可以转换为迭代

子问题类型

f(n) = f(n-1) , f(n-2)

e.g. Fibonacci 数列问题

此类问题依赖前面问题的求解, 如果是迭代求解可以缓存前面几个问题的解结果。 如果递归则不用

比如递归:

public int fibonacci(int n){
    if(n == 0 || n ==1) return n;
    return fibonacci(n-1)+ fibonacci(n-2); 
}

如果是迭代: 缓存中间结果。

public int fibonacci(int n){
    if(n == 0 || n ==1) return n;
    int f1 = 0;
    int f2 = 1;
    for(int i = 2; i<=n; i++){
        int t = f2;
        f2 = f1+t;
        f1 = t;
    }
    return f2;
}

还有一种更加绝妙的解法, 注意

f(n+1) = f(n) + f(n-1)

所以有:

[f(n+1) f(n)  ]       = [f(n)      f(n-1)] * [1  1]    = [1  1]^n
[f(n)   f(n-1)]         [ f(n-1)   f(n-2)]   [1  0]      [1  0]

通过矩阵的 n 次幂, 可以解出 F(n)

这里可以使用 Divide & Conqure 技巧的就是求解幂。

public double pow(x, n){
    if(n==0) return 1;
    int half = pow(x, n/2);
    if(n%2 ==1)
        return half * half * x;
    else 
        return half * half;
}

将上面的 乘法 * 换成矩阵乘法就可以求解 f(n) 了。 复杂度降为 O(logn).

f(n) = f(n/2) or more generally f(n/k)

此类问题经常对应搜索空间减半,从而实现 O(logn) 的时间复杂度。 binary search 是典型此类问题。

**二分查找的形式: **

while(l < r){
    if(==) return found result;
    else if (< ) l++
    else r--
}

Binary Search 的三个模板:

3 Parts of a Binary Search

  1. Pre-processing: Sort if collection is unsorted
  2. Binary Search: Using a loop or recursion to divide search space in half after each comparison
  3. Post-processing: Determine viable candidates in the remaining space.

3 Templates for Binary Search

  1. Template #1 is used to search for an element or condition which can be determined by accessing a single index in the array.
    • Search Condition can be determined without comparing to the element’s neighbors (or use specific elements around it)
    • No post-processing required because at each step, you are checking to see if the element has been found. If you reach the end, then you know the element is not found
    • Initial Condition: left = 0, right = length-1
    • Termination: left > right
int binarySearch(int[] nums, int target){
  if(nums == null || nums.length == 0)
    return -1;

  int left = 0, right = nums.length - 1;
  while(left <= right){
    // Prevent (left + right) overflow
    int mid = left + (right - left) / 2;
    if(nums[mid] == target){ return mid; }
    else if(nums[mid] < target) { left = mid + 1; }
    else { right = mid - 1; }
  }

  // End Condition: left > right
  return -1;
}
  1. It is used to search for an element or condition which requires accessing the current index and its immediate right neighbor’s index in the array.
    • An advanced way to implement Binary Search.
    • Search Condition needs to access element’s immediate right neighbor
    • Use element’s right neighbor to determine if condition is met and decide whether to go left or right
    • Gurantees Search Space is at least 2 in size at each step
    • Post-processing required. Loop/Recursion ends when you have 1 element left. Need to assess if the remaining element meets the condition.
    • Initial Condition: left = 0, right = length
    • Termination: left == right
int binarySearch(int[] nums, int target){
  if(nums == null || nums.length == 0)
    return -1;

  int left = 0, right = nums.length;
  while(left < right){
    // Prevent (left + right) overflow
    int mid = left + (right - left) / 2;
    if(nums[mid] == target){ return mid; }
    else if(nums[mid] < target) { left = mid + 1; }
    else { right = mid; }
  }

  // Post-processing:
  // End Condition: left == right
  if(left != nums.length && nums[left] == target) return left;
  return -1;
}
  1. Template #3 is another unique form of Binary Search. It is used to search for an element or condition which requires accessing the current index and its immediate left and right neighbor’s index in the array.
    • An alternative way to implement Binary Search
    • Search Condition needs to access element’s immediate left and right neighbors
    • Use element’s neighbors to determine if condition is met and decide whether to go left or right
    • Gurantees Search Space is at least 3 in size at each step
    • Post-processing required. Loop/Recursion ends when you have 2 elements left. Need to assess if the remaining elements meet the condition.
    • Initial Condition: left = 0, right = length-1
    • Termination: left + 1 == right
int binarySearch(int[] nums, int target) {
    if (nums == null || nums.length == 0)
        return -1;

    int left = 0, right = nums.length - 1;
    while (left + 1 < right){
        // Prevent (left + right) overflow
        int mid = left + (right - left) / 2;
        if (nums[mid] == target) {
            return mid;
        } else if (nums[mid] < target) {
            left = mid;
        } else {
            right = mid;
        }
    }

    // Post-processing:
    // End Condition: left + 1 == right
    if(nums[left] == target) return left;
    if(nums[right] == target) return right;
    return -1;
}

e.g. median of two sorted array a and b

/** 
 * Find all pairs (i, j) in sorted array `arr` such that `arr[i] + arr[j] > s`
 */
public int countPairs(int[] arr, int s)
    int l = 0; r = arr.length-1;
    int count = 0;
    while(l < r){
        int t = arr[l] + arr[r];
        if(t >s) {
            count += r-l;
            r--;
        }
        else l++;
    }
    return count;
/**
 * find two numbers with sum equal to s in a sorted array.
 */
public int[] countPairs(int[] arr, int s)
    int l = 0; r = arr.length-1;
    int count = 0;
    int[] result = {-1,-1};
    while(l < r){
        int t = arr[l] + arr[r];
        if(t == s) {
            result[0] = l;
            result[1] = r;
            return result;
        }else if(t<s){
            l++;
        }else if(t>s){
            r--;
        }
    }
    return result;

Dynamic Programming

Steps

  1. 建模
  2. 定义子问题
  3. 迭代、递归公式
  4. 边界条件

– Two eggs problem: You have two identical eggs and you are given access to a 100 story building. You would like to know the highest floor for the egg not to break when dropped. The problem is the egg might break on the 1st floor, or even the 100th floor, you just don’t know. Find the maximum number of trials you need to conduct in order to find the answer.

– You have 5 identical jars. 4 of the jars contain balls with identical size weighing 10g each, while 1 of the jar contains balls weighing 9g each. You are given a digital scale, find out the 1 jar that has the 9g balls with only 1 weighing.

Two Pointers

Recursion

To iterate is human, to recurse divine (Anonymous)

solve( problem p )
    if ( p is simple )
        Solve the problem p directly
    else
        Divide p into new sub-problems p1, p2, ...
        r1 = solve(p1)
        r2 = solve(p2)
        Reassemble r1, r2, ... to solve p
    end
end

状态是相关变量的组合状态,一个变量相当于状态的一个分量

一次循环就是一轮状态转移(state transition)

初始状态 -> 按某种规则转移,有些状态属于终止状态,然后状态的某个变量就是所求

当前状态下,各个变量间互动,迁移,达到另一个状态,状态机的一般形式是 while (true) { //转移规则}, for (int i = 0; i < n; ++i ) {}这种是说,i这个状态分量每一轮的转移是固定的(++i),并且i == n 是触发状态机stop的方式之一。 状态机停止的一般形式是在转移的规则里的break或者 return

Recursively-defined functions

  • Factorial: n!
  • Fibonacci numbers
  • Ackermann Function
  • Tower of Hanoi
  • Fractals
  • Tree and data searches

tail recursion

递归带参数

打印目录树结构带缩进

public void print(Node root){
    print(root, "");
}

private void print(Node node, String ident){
    System.out.println(ident+ node.val);
    for(Node child : node.children){
        print(child, ident+" ");
    }
}

DFS

圆形问题 round / circle/ rotation

  • Distributed Hashing Table
  • 数组循环
  • 队列 实现
  • Link list

Tail Recursion

The following section is exerpt from Stackoverflow.

A tail call [tail recursion] is a kind of goto dressed as a call. A tail call happens when a function calls another as its last action, so it has nothing else to do. For instance, in the following code, the call to g is a tail call:

function f (x)
  return g(x)
end

After f calls g, it has nothing else to do. In such situations, the program does not need to return to the calling function when the called function ends. Therefore, after the tail call, the program does not need to keep any information about the calling function in the stack. …

Because a proper tail call uses no stack space, there is no limit on the number of “nested” tail calls that a program can make. For instance, we can call the following function with any number as argument; it will never overflow the stack:

function foo (n)
  if n > 0 then return foo(n - 1) end
end

As I said earlier, a tail call is a kind of goto. As such, a quite useful application of proper tail calls in Lua is for programming state machines. Such applications can represent each state by a function; to change state is to go to (or to call) a specific function.

  • In traditional recursion, the typical model is that you perform your recursive calls first, and then you take the return value of the recursive call and calculate the result. In this manner, you don’t get the result of your calculation until you have returned from every recursive call.

      public int f(int n){
          if(n==0)
              return 0;
          else{
              int pre = f(n-1);
              return g(pre);
          }
      }
    
  • In tail recursion, you perform your calculations first, and then you execute the recursive call, passing the results of your current step to the next recursive step. This results in the last statement being in the form of “(return (recursive-function params))” (I think that’s the syntax for Lisp). Basically, the return value of any given recursive step is the same as the return value of the next recursive call.

      public int f(int n, int result){
          if(n==0)
              return result;
          else{
              result =  g(result, n);
              return f(n-1,result);
          }
      }
    

The consequence of this is that once you are ready to perform your next recursive step, you don’t need the current stack frame any more. This allows for some optimization. In fact, with an appropriately written compiler, you should never have a stack overflow snicker with a tail recursive call. Simply reuse the current stack frame for the next recursive step. I’m pretty sure Lisp does this.

Tail Recursion vs. Non-Tail Recursion

  • In general, (non-tail) recursive function calls put parameters on the stack
    • Every call grows the stack
    • On return, the parameters are needed to compute the result (together with the partial result returned)
  • In tail recursive functions, the parameters from the call before are not needed anymore
    • Instead, the result is directly handed to the parent
    • Hence, no parameters need to be put on the stack
  • Languages that use tail recursion optimization realize this and don’t grow the stack
  • The languages we cover next are optimized in this way So they are much more efficient when using recursion

References:

Divide and Conquer Strategy for Problem Solving - Recursive Functions

  1. If the problem is small enough to be solved directly, do it
  2. If not, find a smaller problem and use its solution to create the solution to the larger problem

juicefs

JuiceFS 服务将 云上的 OSS 存储空间 mount 到本地的目录。 当操作本地目录时, juicefs daemon 进程会监测本地目录文件变化, 实时地将本地目录上传到 OSS 空间上。 这比Dropbox 等要厉害很多。 也可以做协同开发等等。

curl -L juicefs.io/static/juicefs -o juicefs && chmod +x juicefs
sudo ./juicefs mount buyuebuyue /jfs

一个创新的做法是 https://www.douban.com/note/656209469/

在服务器可本地开发机器上分别: 安装 juicefs ,并将同一个 oss 上的 bucket mount 到服务器和本地。

这样服务器上的目录和本地的目录就能够保持实时同步。 同时 server 跑在远端。

做多人协作也很好。

Dynamic programming

Model Definition

components

  • 界定问题
  • 子问题
  • 递推公式
  • 边界条件
  • 复杂度

示例问题分析

n 阶台阶,每步 1或者2阶, 有多少种走法

这是一个 Fibonacci 问题

迭代公式: f(n) = f(n-1) + f(n-2)

边界条件: f(0) = 1 f(1) =1

字符串相似度(编辑距离)

二维 2D 动态规划

sbt

sbt version

➜  /tmp cat /usr/local/bin/sbt
#!/bin/sh
if [ -f "$HOME/.sbtconfig" ]; then
  echo "Use of ~/.sbtconfig is deprecated, please migrate global settings to /usr/local/etc/sbtopts" >&2
  . "$HOME/.sbtconfig"
fi
exec "/usr/local/Cellar/sbt/0.13.15/libexec/bin/sbt" "$@"

While /usr/local/Cellar/sbt/0.13.15/libexec/bin/sbt is also a shell script wrapper. The real sbt exe and options are configured in file /usr/local/etc/sbtopts.

➜  /tmp cat /usr/local/etc/sbtopts
# ------------------------------------------------ #
#  The SBT Configuration file.                     #
# ------------------------------------------------ #


# Disable ANSI color codes
#
#-no-colors

# Starts sbt even if the current directory contains no sbt project.
#
-sbt-create

# Path to global settings/plugins directory (default: ~/.sbt)
#
#-sbt-dir  /etc/sbt

# Path to shared boot directory (default: ~/.sbt/boot in 0.11 series)
#
#-sbt-boot ~/.sbt/boot  

# Path to local Ivy repository (default: ~/.ivy2)
#
#-ivy ~/.ivy2

# set memory options
#
#-mem   <integer>  

# Use local caches for projects, no sharing.
#
#-no-share

# Put SBT in offline mode.
#
#-offline

# Sets the SBT version to use.
#-sbt-version  0.11.3

# Scala version (default: latest release)
#
#-scala-home <path>        
#-scala-version <version>

# java version (default: java from PATH, currently $(java -version |& grep version))
#
#-java-home <path>

-J-Xmx2G
-J-XX:+CMSClassUnloadingEnabled

The real sbt jar/ bin are under path: /usr/local/Cellar/sbt

➜  sbt  tree -L 2 /usr/local/Cellar/sbt
/usr/local/Cellar/sbt
├── 0.13.15
│   ├── INSTALL_RECEIPT.json
│   ├── bin
│   └── libexec
└── 0.13.9
    ├── INSTALL_RECEIPT.json
    ├── bin
    └── libexec

6 directories, 2 files

➜  0.13.9 tree  /usr/local/Cellar/sbt/0.13.9/ 
/usr/local/Cellar/sbt/0.13.9/
├── INSTALL_RECEIPT.json
├── bin
│   └── sbt
└── libexec
    ├── sbt
    ├── sbt-launch-lib.bash
    └── sbt-launch.jar

2 directories, 5 files
➜  0.13.9 

If you want to use a specific sbt version for the project, configure it in project/build.properties file.

➜  opencity-server git:(master)cat project/build.properties 
sbt.version=1.0.4
➜  opencity-server git:(master)cat project/plugins.sbt     
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.6.11")
➜  opencity-server git:(master) ✗ tree -L 2 ~/.sbt/
/Users/lucas/.sbt/
├── 0.13
│   ├── staging
│   └── templates
├── 1.0
│   ├── server
│   ├── staging
│   └── zinc
├── boot
│   ├── sbt.boot.lock
│   ├── scala-2.10.2
│   ├── scala-2.10.4
│   ├── scala-2.10.5
│   ├── scala-2.10.6
│   ├── scala-2.11.5
│   ├── scala-2.11.7
│   ├── scala-2.11.8
│   ├── scala-2.12.4
│   └── update.log
├── launchers
│   └── sbt-launch-57d0f04f4b48b11ef7e764f4cea58dee4e806ffd.jar
├── preloaded
│   ├── com.jcraft
│   ├── com.thoughtworks.paranamer
│   ├── jline
│   ├── org.fusesource.jansi
│   ├── org.json4s
│   ├── org.scala-lang
│   ├── org.scala-lang.modules
│   ├── org.scala-sbt
│   ├── org.scala-sbt.ivy
│   ├── org.scala-tools.sbinary
│   ├── org.scalamacros
│   └── org.spire-math
└── repositories

version and dependency

There are many components of a play-framework project, you should take care of when configuring the versions.

  • play-framework
  • scala
  • sbt
  • sbt-plugin for play-framework

application-releated

Firstly, you choose a play-framework verison, usally, the latest one. (the current latest is 2.6.11). Since play-framework is written in Scala, and you also need to write your application code in Scala, you have to choose a Scala version. The latest is 2.12.3. Configure it in build.sbt file.

➜  opencity-server git:(master) ✗ cat build.sbt 
name := """opencity-server"""
organization := "cn.opencity"

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.12.3"

libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % Test

// Adds additional packages into Twirl
//TwirlKeys.templateImports += "cn.opencity.controllers._"

// Adds additional packages into conf/routes
// play.sbt.routes.RoutesKeys.routesImport += "cn.opencity.binders._"

build-tool versions

sbt and sbt-plugin for play-framework are used to build the project. There versions are specified under ./project/* dir configuring files.

https://github.com/coursier/coursier#sbt-plugin

  • issues to note:
    • https://github.com/coursier/coursier/issues/450

Mill play-scala micro-services framework https://www.lagomframework.com/

Getting Started with Play 2.x with IntelliJ

Mill is a new build tool for Scala: it compiles your Scala code, packages it, runs it, and caches things to avoid doing unnecessary work. Mill aims to be better than Scala’s venerable old SBT build tool, learning from it’s mistakes and building upon ideas from functional programming to come up with a build tool that is fast, flexible, and easy to understand and use. This post will explore what makes Mill interesting to a Scala developer who is likely already using SBT

Scala and Play with Gradle

Create your own Ethereum token

web3j.io

Use local geth client

RPC client mode

  • Infura
  • local
    • RPC
    • IPC

Java Smart Contract Wrapper

Build and compile

solc

ENS

Ethereum Name Service

Hello, world!

https://www.ethereum.org/greeter

  1. Install ethereum command line tools

Clients

  • Geth: Go implementation Ethereum implementation
  • Parity:
  • Eth: The C++ implementation is simply called Eth. If you want added security by running two different implementations in parallel or are serious about GPU mining, then the C++ “Eth” client is for you.
  • Pyethapp: The Python implementation is called Pyethapp. If you are interested in understanding how Ethereum works and how to extend it, the code base of this client is probably the most readable and has a great contract tester library with fast development cycles. It is not meant for high-end usage as performance in this client is not as high priority as clarity and reliability. If you are a Python developer that wants to build decentralized apps or are interested in Ethereum for research or an academic purpose, this is a great client: we invite you to take a look and contribute to it.
  • Parity a Rust implementation by Parity Technologies
  • A Haskell implementation developed by Blockapps
  • If you are interested in developing a light application that will run entirely in a web browser, then we recommend using EthereumJS as a basis.
  • If you want to create a small hardware project, look into the implementation for the Raspberry Pi
  • If you want to install geth for non-ubuntu linux then we recommend you look into building from source
  • If you want more flexibility on the Mac, try Homebrew

Solidity

Solidity is the major language to develop smart contracts on Ethereum blockchain. The documentation is at https://solidity.readthedocs.io/en/develop/ ##

Editors & IDEs

  • Remix: Browser-based IDE with integrated compiler and Solidity runtime environment without server-side components.
  • IntelliJ IDEA plugin: Solidity plugin for IntelliJ IDEA (and all other JetBrains IDEs)

SWARM

decentralized storage service, like ipfs.

The swarm of Swarm is the collection of nodes of the devp2p network each of which run the bzz protocol on the same network id.

ABI

solc compiler

Events

misc

Whisper: decentralized messaging protocol

geth: command line tools

geth is listening on UDP/TCP port 30303

ps aux |grep geth
lucas            19806  35.5  8.0 558645868 1348204 s001  S+    4:24PM   1:45.52 geth console                                 
lucas            20355   0.0  0.0  4267768    892 s004  S+    4:30PM   0:00.00 grep geth                                      
                                                                           
                

lsof -i -a -p 19806
COMMAND   PID  USER   FD   TYPE            DEVICE SIZE/OFF NODE NAME                                                          
geth    19806 lucas   13u  IPv6 0x9295f2087a46ab9      0t0  UDP *:30303                                                       
geth    19806 lucas   16u  IPv4 0x9295f208ea01d21      0t0  TCP 192.168.1.116:64563->39.107.26.140:61910 (ESTABLISHED)        
geth    19806 lucas   21u  IPv6 0x9295f2082d4f599      0t0  TCP *:30303 (LISTEN)                                              
geth    19806 lucas   28u  IPv4 0x9295f208f6f6101      0t0  TCP 192.168.1.116:63606->ns3066492.ip-79-137-70.eu:30303 (ESTABLIS
HED)                                                                                                                          
geth    19806 lucas   39u  IPv4 0x9295f208ec0e101      0t0  TCP 192.168.1.116:64180->hst-46-166-161-114.balticservers.eu:30303
 (ESTABLISHED)                                                                                                                
geth    19806 lucas   66u  IPv4 0x9295f2082332101      0t0  TCP 192.168.1.116:64552->47.74.5.209:30821 (SYN_SENT)             
geth    19806 lucas   75u  IPv4 0x9295f20823fed21      0t0  TCP 192.168.1.116:64070->47.104.15.188:30303 (ESTABLISHED)        
geth    19806 lucas   88u  IPv4 0x9295f20a183d681      0t0  TCP 192.168.1.116:64559->101.207.224.48:40145 (SYN_SENT)          
                                                                                                                              

Ethereum Network Visualization

http://ethviewer.live/ ethviewer.live

https://ethstats.net/

ethstats.net

0x

mempools

Operations on orders:

  • generating,
  • signing,
  • filling and
  • cancelling ,
  • verifying an orders signature,
  • setting or checking a users ERC20 token balance/allowance
  • and much more.

TestRPC

➜  0x.js-starter-project git:(master) ✗ yarn testrpc

yarn run v1.5.1
$ testrpc -p 8545 --networkId 50 --db ./0x_testrpc_snapshot -m "${npm_package_config_mnemonic}"

Upgrade to Android Studio 3.0

After upgrading to Android Studio 3.0, many configs changed. Please check https://developer.android.com/studio/build/gradle-plugin-3-0-0-migration.html for details.

Versions

Android Studio as an IDE often breaks because of the incompatiblity of these three components:

  • Android Studio
  • Gradle
  • Android Plugin for Gradle

Make sure they are compatible.

In latest version as 2018/02/23,

  • Android Studio v3.0.1
  • Gradle v4.1
  • Android Plugin for Gradle v3.0.1

The version for Android Studio and ` Android Plugin for Gradle` should be always the same.

Naming

com.android.feature

implementation, api, compileOnly, and runtimeOnly.

flavor dimensions

Break Bitcoin Pow

Miners are required to do a Proof of Work (PoW) computation to find a x such that SHA256(digest+x) == 000102340... with specified number of leading zeros.

This is a less strict requirement than birthday attack which requires to find x and y such that SHA256(x) == SHA256(Y).

I guess by inspecting the internal state of SHA256 algorithm when doing a hash, one can dramatically reduce the work to find a x to satisfy PoW, which means he can mine the block much faster than others, which could be big economic incentive to optimize mining process, or, in other words, break the current bitcoin PoW mechanism.

By Using a neural network model to quick find x, can be a promising approach.

0xproject

Bitcoin

Lightening Network

## BlockStream
c-lightening
## Lightening Labs # Ethereum # Smart Contract # HOT/ DDEX # 0xproject # 21 wallet

BlockChain

Bitcoin

Lightening Network

## BlockStream
c-lightening
## Lightening Labs # Ethereum # Smart Contract # HOT/ DDEX # 0xproject # 21 wallet

Python3 install

https://passingcuriosity.com/2015/installing-python-from-source/

[root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 Python-3.6.4]# ./configure --enable-optimizations --prefix=/root/workspace/python3
make
make test
make install

[root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 python3]# export PATH=$PATH:/root/workspace/python3/Python-3.6.4:/root/workspace/python3/bin [root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 python3]# pip3 install virtualenv

Hashing

basic operations

insert lookup delete

One-way hash

information theory

Hash chain

Cryptographic Hash

Universal Hash

perfect Hash

Dynamic

Construction

bit matrix

prime number

implementation in Java/ Python

Applications

cryptography

MD5 public key

one time password

digital cash: bitcoin

Distributed hash table

IPFS

IPFS is the Distributed Web

A peer-to-peer hypermedia protocol to make the web faster, safer, and more open.

###CDN

Routing

Consistent Hashing

MERKEL TREE

Python Pandas

model

DataFrame

index Series

Join

Extract Transform Load pc4t

Tabular data Time series data

Python VIRTUAL ENV

Python virtual enviroment 提供一个独立的 python 运行环境, 用来解决类似 Windows 上的 DLL HELL 问题。 将应用的依赖和版本等隔离开来。每个应用有自己的隔离环境。

Python 在用的版本有很多, 2.6, 2.7, 3.5, 3.6 等等。 每一个依赖也可能很多版本,中间很多兼容性问题。 如果将他们全都放到一起, 就会产生 DLL HELL 的问题。

➜  module2 l /usr/local/lib/python2.7/site-packages 
total 4392
drwxr-xr-x  95 lucas  admin   3.0K Nov 24 18:00 .
drwxr-xr-x   4 lucas  admin   128B Nov  1 20:26 ..
drwxr-xr-x   9 lucas  admin   288B Nov  3 23:25 astroid-1.5.3.dist-info
drwxr-xr-x   7 lucas  admin   224B Nov  3 23:25 backports
drwxr-xr-x   9 lucas  admin   288B Nov  3 23:25 backports.functools_lru_cache-1.4.dist-info

Install

$ [sudo] pip install virtualenv

也可以安装源代码到本地,这样不需要 sudo 权限

$ python virtualenv.py myVE
$ ls -lah
total 256K
drwxr-xr-x 10 lucas lucas 4.0K 2017-11-27 16:01 .
drwxr-xr-x  7 lucas lucas 4.0K 2017-11-27 16:01 ..
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 AUTHORS.txt
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 bin
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 docs
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 LICENSE.txt
-rw-r--r--  1 lucas lucas  345 2016-11-16 11:39 MANIFEST.in
drwxr-xr-x  5 lucas lucas 4.0K 2017-11-27 16:01 myVE
-rw-r--r--  1 lucas lucas 3.4K 2016-11-16 11:39 PKG-INFO
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 README.rst
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 scripts
-rw-r--r--  1 lucas lucas   88 2016-11-16 11:39 setup.cfg
-rw-r--r--  1 lucas lucas 4.0K 2016-11-16 11:39 setup.py
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 tests
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv.egg-info
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv_embedded
-rwxr-xr-x  1 lucas lucas  97K 2016-11-16 11:39 virtualenv.py
-rw-r--r--  1 lucas lucas  86K 2017-11-27 16:01 virtualenv.pyc
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv_support

核心文件是virtualenv.py 当然要求 相关的类库存在。

使用

➜  /tmp virtualenv ENV
New python executable in /private/tmp/ENV/bin/python2.7
Also creating executable in /private/tmp/ENV/bin/python
Installing setuptools, pip, wheel...done.

➜  /tmp tree -L 2 ENV
ENV
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── easy_install
│   ├── easy_install-2.7
│   ├── pip
│   ├── pip2
│   ├── pip2.7
│   ├── python -> python2.7
│   ├── python-config
│   ├── python2 -> python2.7
│   ├── python2.7
│   └── wheel
├── include
│   └── python2.7 -> /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7
├── lib
│   └── python2.7
└── pip-selfcheck.json

5 directories, 15 files

  • ENV/lib/ and ENV/include/ are created, containing supporting library files for a new virtualenv python. Packages installed in this environment will live under ENV/lib/pythonX.X/site-packages/.
  • ENV/bin is created, where executables live - noticeably a new python. Thus running a script with #! /path/to/ENV/bin/python would run that script under this virtualenv’s python.
  • The crucial packages pip and setuptools are installed, which allow other packages to be easily installed to the environment. This associated pip can be run from ENV/bin/pip.

激活

$ source bin/activate

``` $ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

$ source bin/activate

$ echo $PATH /home/xx/virtualenv-15.1.0/myVE/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Jupyter

Jupyter Notebook, 一款基于 Ipython Kernel 的 web 版 python 运行工具。 方便写 python 相关的报告。 混合 Python 运行环境和markdown 文档。

架构

architecture

安装

pip install jupyter

运行

jupyter notebook

其他

svg

Headless-chrome

Running headless Chrome with Selenium in Python

Python Text Mining

Coursera Course: Applied Data Science with Python Specialization. Applied Text Mining in Python

Module 1: Working with Text in Python

  • Library name Minimum version
  • scikit-learn 0.18.1
  • scipy 0.19.0
  • numpy 1.12.1
  • pandas 0.20.1
  • matplotlib 2.0.1
  • seaborn 0.7.1
  • graphviz 0.7

  • String Operations
    # find unique words
    split(' ')
    len()
    set()
    captalize()
    startswith(s)
    endswith(s)
    t in s
    isupper()
    islower()
    isTitle()
    isalpha()
    isdigit()
    isalnum()
    
  • Cleaning Text
    stip()
    
  • Changing Text

pandas

DataFrame ###

NLKT

Natural Language Processing Tookkit

Python Packaging

Python Process

Current Architecture

Disutils Setuptools Distribute Pip

Miscs

  • rpm
  • dpkg
  • yum
  • apt
  • easy_install
  • pip

Python Process

Python Process

  • package multiprocessing
  • Pool map

  • Process

Exchanging objects between processes¶

  • Queue
  • Pipe

Synchronization between processes

multiprocessing

  • Pool
  • Process
  • Exchanging Objects
    • Queue
    • Pipe
  • sync
    • Lock
  • Sharing state
    • Value
    • Array
    • Manager

Parsing

Context-free Grammar

Word categories

  • noun
  • pronoun
  • verb
  • adjective
  • adverb
  • conjunction
  • preposition
  • interjection *

##

  • word
  • consituent phrases

CFG = Context-Free Grammar = Phrase Structure Grammar = BNF = Backus-Naur Form

Context-free grammar

G = <T,N, S,R> • T is set of terminals (lexicon) • N is set of non-terminals For NLP, we usually distinguish out a set P  N of preterminals which always rewrite as terminals. • S is start symbol (one of the nonterminals) • R is rules/productions of the form X ! , where X is a nonterminal and is a sequence of terminals and nonterminals (may be empty). • A grammar G generates a language L.

Parser combinator

Parser

  1. Design an algebra
  2. laws
  3. data types

Algebra

algebra =

  • a collection of functions
  • data types
  • a set of laws over functions

miscs

  • YACC
  • ANTLR
  • algebraic design
  • how is regular expression implemented in various languages?

Interlude

余华 『间奏』

余华的文字有种炫技的感觉, 但是看着又很舒服。技巧纯熟,所以不会有太做作的感觉。

记得以前在哪里看过韩寒表示读不下去余华的文字, 说是没有美感,可能他读的是「兄弟」吧。 那本小说和之前的一些小说文字确实不怎么美丽。

看了余华的一些散文, 其实文字功底是相当地强悍。 之前『活着』之类的作品文字上的直白,怕是有意为之。

回到这部作品, 写的是余华对音乐的一些感悟。 而感悟的点呢,当然有作家敏感的人生和生活等,但最主要的比较还是对文学。

比如对苏联作曲家肖斯塔科维奇的『第七交响曲』(列宁格勒交响曲),余华就把它同美国小说家霍桑小说『红字』做了细致的对比。本来是两个领域的两个东西, 没有可比性。 但是在情感上引起的共鸣又是那么的相似。 让我想起大一时老师给看的『Fantasy 2000』系列动画音乐作品。 本是一两百年前的大师古典乐,迪士尼为其配了动画故事。 结果看起来却像是200年前的大师为迪士尼动画配的乐。 这说明很多艺术所触发的情感本是相同。

余华听音乐时,第一层是听音,声音本身的旋律很好听。 第二层听故事, 旋律的跳动,影响这思考的节奏, 古典乐总是在叙说着一段故事。 第三层听人生,对比自己在文学和其他方面的感受, 有时是如此相似。

章节

  • 叙述: 勃拉姆斯
  • 高潮: 肖斯塔科维奇和霍桑
  • 灵感: 莫扎特
  • 否定: 柏辽兹
  • 色彩: 拉赫玛尼诺夫, 绘画
  • 字与音: 门德尔松

文字

这本书里的很多文字都非常有艺术想象力。读起来很美。比如:

然后他(肖斯塔科维奇)发现一个时代找上门来了。一九四二年三月五日,『第七交响曲』在后方城市古比雪夫首演后,立刻成为了这个正在遭受耻辱的民族的抗击之声,另外一个标题『列宁格勒交响曲』也立刻覆盖了原有的标题『第七交响曲』。

这几乎是一切叙述作品的命运,它们需要获得一个时代的青睐,才能使自己得到成功的位置,然后一劳永逸地坐下去。

Python Thread

Python Thread

  • __init__()
  • run()
  • start()

子进程出错了,如何让整个程序退出?

Python Optimize

项目优化

最近使用 Python做了一个日志转换的小项目。原生 Python,没有使用任何的Framework. 写了约两周。 越写越顺手。 发现 Python 的整个设计比较人性化,没有太多的冷知识和需要记忆的。

项目逻辑比较简单。 一句话解释就是对日志格式进行转换。

原始日志(gzip/8.5M/300K lines), 22个/min
                ↓
每行日志进行目标格式的转换,生成新的日志 (需要生成2种格式的日志,即两份)
                ↓
每5分钟或者700M(压缩前大小, 压缩后大概100M) 滚动
                ↓
新的日志 (gzip/100M/5M lines), 

对于单个原始文件, 最终优化到10s/个的处理速度。

最开始是需要12分钟的速度, 经过多次优化,达到现在的速度。

优化1: Python 字符串处理

字符串处理。 Python 本身的字符串处理速度是比较慢的。 每行进行格式转换的算法。

v1: string.split()

最开始是使用字符串空格切分,得到每个字段。由于每一行的字段数目不是完全一致,切分是分多次进行。 非常慢。

v2: regex

花了一个晚上的时间, 使用 regex 一次性获取所有字段,regex 只需要编译一次。 速度提升非常快。

优化2: bitarray

生成新格式其中一个需求是针对的某些符合需求的日志做替换。 如果日志中的 IP 不在给定的 IP 网段集合中, 则需要使用备用库中的 IP 替换原始 IP。 这涉及到一个判定 IP 在不在给定 IP 网段集合的算法。 IP 网段的形式是 (1.1.1.0/24).约有3000个网段。

v1: 直接循环判定

使用 python ipaddress library:


    for line in lines:
        subnet = ipaddress.ip_network(line)
        self.localnet.append(subnet)

    def ip_in_localnet_raw(self, ip):
        ''' deprecated raw method, which is slow. use  ip_in_localnet
        '''
        address = ipaddress.ip_address(unicode(ip))
        for subnet in self.localnet:
            if address in subnet:
                return True
        return False
    

这个算法当中, 每次 IP 都需要循环3000/2次判定在不在给定 IP 网段集合中。时间非常慢。

v2: bloomfilter

对于快速判定某元素在不在指定集合中, bloomfilter 是非常好的结构。 直接使用 bloomfilter, 将 IP 网段集合转换成 IP 集合。 但是3000个网段,对应的 IP 数量非常庞大,bloom filter 也非常大。

v3: bitarray

IP 数据有很好的特征。 IPV4 实际是32bit unsigned integer. 如果直接将一个 IP 在不在用一个 bit 表示,则需要2^32bit =512MB 的内存。 对于服务器来说, 可以接受。 当然可以直接使用一个 int array 。 但 Python 有现成的 library 来使用: bitarray

        self.ips = bitarray(2**32)

    def __load_localnet(self):
        self.localnet = []
        starttime = datetime.datetime.now()  
        count = 0
        with open(self.localnet_iplist_file, 'r') as f:
            for line in f:
                line = unicode(line.strip())
                count +=1
                try:
                    subnet = ipaddress.ip_network(line)
                    self.localnet.append(subnet)
                    start = int(subnet.network_address)
                    end = start + subnet.num_addresses
                    for x in range(start, end):
                        self.ips[x] = 1
                except Exception as e:
                    self.logger.exception(e.message)
        endtime = datetime.datetime.now() - starttime 
        self.logger.info("GroupLog loading %s localnets done, takes %s" % (count, endtime))          

上面这个函数,将所有的 IP 网段转换成 IP 集合, 存到 ips[] bitarray 当中。 加载需要24s 左右。

v4: 加载速度优化


                    # for x in range(start, end):
                    #     self.ips[x] = 1
                    # This is far more faster than add one by one.
                    self.ips[start:end] = 1
GroupLog loading 2128 localnets done, takes 0:00:24.227503 (set one by one)
GroupLog loading 2128 localnets done, takes 0:00:00.214876 (set by [start:end])
benchmark: call 300000 times in 0:00:02.242983

经过多次优化, 加载和判定速度得到极大优化。

Python 多线程并不能利用多核

Threads are useful when the code is blocked by non bytecode execution, such as I/O or external process execution (C code, system calls, etc). If byte code execution is holding things up, the ProcessPool starts multiple interpreters that can execute in parallel. However, there is more overhead in spinning up these interpreters and in them communicating with the main thread through serialized representations (basically pickle or json over a socket if I understand correctly).

multithread

  • threading
  • thread

    multiprocessing

  • Pool
  • Process
  • Exchanging Objects
    • Queue
    • Pipe
  • sync
    • Lock
  • Sharing state
    • Value
    • Array
    • Manager

GIL

Global Interpreter Lock sub-process v.s. thread

concurrent.futures

  • ThreadPoolExecutor
  • ProcessdPoolExecutor

由于 GIL 的限制, python 的 multi-thread 并不能在多核上运行。 Python 的 multi-thread 虽然将运行线程化了,但是只是在单核上运行。 这样, 对 cpu-intensive 的计算没有意义,对于 I/O 类型的运算才有意义。

msic

  • **kwargs
  • How does OS pip work?
  • multiprocessing.Pool v.s. concurrent.futures.ProcessPoolExecutor

GIL

tick

interpreter instructions

import dis
dis.dis(countdown)

str is text representation in bytes, unicode is text representation in characters.

Python Modules, Classes, Packages

Python Modules, Classes, Packages

Module

  • A module is a file containing Python definitions and statements. The file name is the module name with the suffix ‘.py’ appended. Within a module, the module’s name (as a string) is available as the value of the global variable __name__.
__name__
import fibo
from fibo import fib, fib2
from fibo import *
  • symbol table

  • A module can contain executable statements as well as function definitions. These statements are intended to initialize the module. They are executed only the first time the module is imported somewhere.

  • When a module named ‘spam’ is imported, the interpreter searches for a file named ‘spam.py’ in the current directory, and then in the list of directories specified by the environment variable ‘PYTHONPATH’. This has the same syntax as the shell variable ‘PATH’, that is, a list of directory names. When ‘PYTHONPATH’ is not set, or when the file is not found there, the search continues in an installation-dependent default path; on UNIX, this is usually ‘.:/usr/local/lib/python’.
  • sys.path = the directory containing the input script (or the current directory), + ‘PYTHONPATH’ + the installation-dependent default.

  • dir() find out which names a module defines.
  • standard module __builtins__
# -*- coding: utf8 -*-
import os, sys

# demo module usage

def f():
    print "function f in module 'module.py'"
print "__name__ = ", __name__
print "os.environ['PYTHONPATH'] = ", os.environ.get("PYTHONPATH", '')
print "sys.path = ", sys.path
print "dir() = ", dir()
print "__builtins__ = ", __builtins__
print "dir(__builtins__) = ", dir(__builtins__)

outputs:

➜  python101 python /Users/lucas/dev/workspace/python/python101/module.py
__name__ =  __main__
os.environ['PYTHONPATH'] =
sys.path =  ['/Users/lucas/dev/workspace/python/python101', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages', '/usr/local/Cellar/protobuf/2.6.1/libexec/lib/python2.7/site-packages']
dir() =  ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'f', 'os', 'sys']
__builtins__ =  <module '__builtin__' (built-in)>
dir(__builtins__) =  ['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BufferError', 'BytesWarning', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError', 'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs', 'all', 'any', 'apply', 'basestring', 'bin', 'bool', 'buffer', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']

package

  • Packages are a way of structuring Python’s module namespace by using “dotted module names”.
  • The __init__.py files are required to make Python treat the directories as containing packages;
  • ` all = [“echo”, “surround”, “reverse”]`
  • Intra-package References
  • __path__. This is initialized to be a list containing the name of the directory holding the package’s __init__.py before the code in that file is executed.

Class

# -*- coding: utf8 -*-

class Foo:
    val=0
    def __init__(self, val):
        self.val = val
        pass
    
    def printVal(self):
        print(self.val)


foo = Foo(5)
foo.printVal()
  • __init__
      def __init__(self, val): # constructor
          self.val = val
    
  • self this me, etc.
  • methods
  • variables: Python creates variables on the fly, when they are first assigned, and class member variables are no different.
          self.val = val
    
  • instance we can change the definition of a class on the fly, or even create a completely new class at run-time!
      Foo.printVal = printVal
      obj.printVal()
      del Foo.printVal
    
      MyNewClass = type('MyNewClass', (object,), dict())
    
  • standard methods:
    • __del__: Called when an instance is about to be destroyed, which lets you do any clean-up e.g. closing file handles or database connections
    • __repr__ and __str__: Both return a string representation of the object, but __repr__ should return a Python expression that can be used to re-create the object. The more commonly used one is __str__, which can return anything.
    • __cmp__: Called to compare the object with another object. Note that this is only used with Python 2.x. In Python 3.x, only rich comparison methods are used. Such as __lt__.
    • __lt__, __le__, __eq__, __ne__, __gt__ and __ge__: Called to compare the object with another object. These will be called if defined, otherwise Python will fall-back to using __cmp__.
    • __hash__: Called to calculate a hash for the object, which is used for placing objects in data structures such as sets and dictionaries.
    • __call__: Lets an object be “called” e.g. so that you can write things like this: obj(arg1,arg2,…).
    • Python also lets you define methods that let an object act like an array (so you can write things like this: obj[2] = "foo"), or like a numeric type (so you write things like this: print(obj1 + 3*obj2).
    • __dict__ − Dictionary containing the class’s namespace.
    • __doc__ − Class documentation string or none, if undefined.
    • __name__ − Class name.
    • __module__ − Module name in which the class is defined. This attribute is “__main__” in interactive mode.
    • __bases__ − A possibly empty tuple containing the base classes, in the order of their occurrence in the base class list.
  • In Python, methods and member variables are always public i.e. anyone can access them.
  • private methods and variables: Class methods and variable names that start with two underscores are considered to be private to that class, However, this is only a convention

  • Class Inheritance
      class Foo:
          def __init__(self, val):
              self.val = val
          def printVal(self):
              print(self.val)
        
      class DerivedFoo(Foo):
          def negateVal(self):
              self.val = -self.val
    
  • isinstance
  • issubclass
  • Class Iterators and Generators
    
      class Backwards:
          def __init__(self, val):
              self.val = val
              self.pos = len(val)
        
          def __iter__(self):
              return self
        
          def next(self):
              # We're done
              if self.pos <= 0:
                  raise StopIteration
        
              self.pos = self.pos - 1
              return self.val[self.pos]
    
      for x in Backwards([1,2,3]):
          print(x)
    
  • generator yield

  • class variable
  • instance variable
  • function overloading
  • inheritance

  • data memebers
  • documentation
      class Employee:
          'Common base class for all employees'
          empCount = 0
    
          def __init__(self, name, salary):
              self.name = name
              self.salary = salary
              Employee.empCount += 1
            
          def displayCount(self):
              print "Total Employee %d" % Employee.empCount
    
          def displayEmployee(self):
              print "Name : ", self.name,  ", Salary: ", self.salary
              class_suite
    

    The class has a documentation string, which can be accessed via ClassName.doc.

  • The getattr(obj, name[, default]) − to access the attribute of object.

  • The hasattr(obj,name) − to check if an attribute exists or not.

  • The setattr(obj,name,value) − to set an attribute. If attribute does not exist, then it would be created.

  • The delattr(obj, name) − to delete an attribute.

  • Destroying Objects (Garbage Collection)
         def __del__(self):
    

import

how to import module in parent directory?

check How does python find packages?

Python imports work by searching the directories listed in sys.path.

> import sys
> print '\n'.join(sys.path)

/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages

sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

Find module

> import imp
> imp.find_module('numpy')
(None, '/usr/local/lib/python2.7/dist-packages/numpy', ('', '', 5))

site-python

Python logging

Pythong logging module

img

  • Loggers expose the interface that application code directly uses.
  • Handlers send the log records (created by loggers) to the appropriate destination.
  • Filters provide a finer grained facility for determining which log records to output.
  • Formatters specify the layout of log records in the final output.
  • ` LogRecord`

syslog

Rsyslog

Python Logging

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# create a file handler
handler = logging.FileHandler('hello.log')
handler.setLevel(logging.INFO)

# create a logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(handler)

logger.info('Hello baby')

Logger

logger names track the package/module hierarchy, and it’s intuitively obvious where events are logged just from the logger name.

  • Logger.setLevel() specifies the lowest-severity log message a logger will handle, where debug is the lowest built-in severity level and critical is the highest built-in severity. For example, if the severity level is INFO, the logger will handle only INFO, WARNING, ERROR, and CRITICAL messages and will ignore DEBUG messages.
  • Logger.addHandler() and Logger.removeHandler() add and remove handler objects from the logger object. Handlers are covered in more detail in Handlers.
  • Logger.addFilter() and Logger.removeFilter() add and remove filter objects from the logger object. Filters are covered in more detail in Filter Objects.

Config

logging.ini

three ways:

  • Creating loggers, handlers, and formatters explicitly using Python code that calls the configuration methods listed above.

  • Creating a logging config file and reading it using the fileConfig() function.
      logging.config.fileConfig('logging.conf')
    
  • Creating a dictionary of configuration information and passing it to the dictConfig() function.

Use rotating file handler

Level/ Severity

debug(), info(), warning(), error() and critical().

Handler

  • class Handler()
  • class logging.StreamHandler(stream=None)
  • class logging.NullHandler
  • class logging.FileHandler(filename, mode='a', encoding=None, delay=False)
  • class logging.handlers.WatchedFileHandler(filename[, mode[, encoding[, delay]]])
  • class logging.handlers.RotatingFileHandler(filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0)
    • doRollover()
  • class logging.handlers.TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False)
  • class logging.handlers.SocketHandler(host, port)
  • class logging.handlers.DatagramHandler(host, port)
  • class logging.handlers.SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER, socktype=socket.SOCK_DGRAM)
  • class logging.handlers.NTEventLogHandler(appname, dllname=None, logtype='Application')
  • class logging.handlers.SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None)
  • class logging.handlers.BufferingHandler(capacity)
  • class logging.handlers.MemoryHandler(capacity, flushLevel=ERROR, target=None)
  • class logging.handlers.HTTPHandler(host, url, method='GET')

misc

  • Logging variable data:
      logging.warning('%s before you %s', 'Look', 'leap!')
    
      str.format() 
      string.Template
      from string import Template
      s = Template('$who likes $what')
      s.substitute(who='tim', what='kung pao')
    

TODO

  1. what is python module and __name__ variable? if the python interpreter is running that module (the source file) as the main program, it sets the special __name__ variable to have a value “__main__”. If this file is being imported from another module, __name__ will be set to the module’s name. A module’s __name__ is set equal to ‘__main__’ when read from standard input, a script, or from an interactive prompt. A module can discover whether or not it is running in the main scope by checking its own __name__, which allows a common idiom for conditionally executing code in a module when it is run as a script or with python -m but not when it is imported:

For a package, the same effect can be achieved by including a __main__.py module, the contents of which will be executed when the module is run with -m.

Scrapy

Scrapy Learned

Architecture

Architecture

  • engine
  • spider
  • scheduler
  • downloader
  • item pipelines

Terminology

  • crawl: walk the links
  • parser: parse the webpage html into structured data
  • spider: define how crawler and parser work
  • scraper: extract data from within a webpage.

A crawler gets web pages – i.e., given a starting address (or set of starting addresses) and some conditions (e.g., how many links deep to go, types of files to ignore) it downloads whatever is linked to from the starting point(s).

` A scraper ` takes pages that have been downloaded [Edit: or, in a more general sense, data that’s formatted for display], and (attempts to) extract data from those pages, so that it can (for example) be stored in a database and manipulated as desired.

parse(self, response)

return

  • extract dict {}
  • Item
  • Request with parse() defined.

Selectors

grammar: response.css(selector) or response.xpath(selector)

  • property: a ::text
  • child level: div.pre_post > a
  • class: div.pre_post
  • id: div#id

functions:

  • extract_first() css selection results are array, return the first element of the results, even though there are only one element in the results.

Scrapy Commands

  • scrapyd-deploy
  • scrapy.cfg

Global commands

  • startproject
  • genspider
  • settings
  • runspider
  • shell
  • fetch
  • view
  • version

Project commands

  • crawl
  • check
  • list
  • edit
  • parse
  • bench

project structure

scrapy.cfg
myproject/
    __init__.py
    items.py
    pipelines.py
    settings.py
    spiders/
        __init__.py
        spider1.py
        spider2.py
        ...

Scrapy Shell

TODOS:

  • response.follow
  • yield result or page

Spider

Selector

The selection methods (.xpath() or .css()) return a list of selectors of the same type, so you can call the selection methods for those selectors too.

  • .xpath('//title/text()')
  • .css('title::text')
  • .re('a*')

In [20]: type(response.selector)
Out[20]: scrapy.selector.unified.Selector

In [21]: type(response.selector.xpath('//title/text()'))
Out[21]: scrapy.selector.unified.SelectorList

In [22]: type(response.selector.xpath('//title/text()')[0])
Out[22]: scrapy.selector.unified.Selector

In [23]: 
>>> response.xpath('//title/text()')
[<Selector (text) xpath=//title/text()>]
>>> response.css('title::text')
[<Selector (text) xpath=//title/text()>]
response.css('img').xpath('@src').extract()
response.css('img::attr(src)').extract()
  • extract()
  • extract_first()

Relative Xpath

  • .// relative
  • // absolute

Item

ItemLoader

Scrapy Shell

Item pipeline

Feed Exports

Serialization formats Item exporters

  • JSON
  • JSON lines
  • CSV
  • XML

Requests and Responses

Downloader Middleware

IN settings.py

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.CustomDownloaderMiddleware': 543,
}

hook

Engine -> mw1 -> mw2 -> mw3 -> ->   -> downloader

       engine
          ↓
mw1.process_request()
          ↓
mw2.process_request()
          ↓
mw3.process_request()
          ↓
          ↓
      downloader
          ↓
          ↓
mw3.process_response()
          ↓
mw2.process_response()
          ↓
mw1.process_response()
          ↓
       engine
process_request(request, spider)

process_response(request, response, spider)

## Sitemap

```sh
  python curl  http://www.jollychic.com/sitemap.xml
<?xml version="1.0" encoding="UTF-8"?>
	<?xml-stylesheet type="text/xsl" href="gss.xsl"?>
	<sitemapindex xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
	<sitemap>
		<loc>http://www.jollychic.com/sitemap/sitemap_images_1.xml.gz</loc>
		<lastmod>2017-05-08</lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.jollychic.com/sitemap/sitemap_images_9.xml.gz</loc>
		<lastmod>2017-05-08</lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.jollychic.com/sitemap/sitemap_product_1.xml.gz</loc>
		<lastmod>2017-05-08</lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.jollychic.com/sitemap/sitemap_product_2.xml.gz</loc>
		<lastmod>2017-05-08</lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.jollychic.com/sitemap/sitemap_tag_1.xml.gz</loc>
		<lastmod>2017-05-08</lastmod>
	</sitemap>
</sitemapindex>                  
curl http://www.jollychic.com/sitemap/sitemap_tag_1.xml.gz |gzip -d  |less
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0">
                <url>
                        <priority>0.5</priority>
                </url>
                <url>
                        <loc>http://www.jollychic.com/t/long-white-dresses-t5.html</loc>
                        <priority>0.5</priority>
                </url>
</urlset>
 for index, link in enumerate(links):

Vue

Hello world

  1. import: <script src="https://unpkg.com/vue"></script>
  2. HTML ```html
3. javascript

```js
var app = new Vue({
  el: '#app',
  data: {
    message: 'hello, world'
  }

})

HTML 中用一个 mount point <div id="app">, 在 javascript 创建一个 Vue 示例。

HTML 渲染

  1. 文本插值:
  2. v-bind::title e.g. ` `
  3. v-on: @ 事件处理 ` `
  4. 条件: v-if
  5. 循环: v-for
  6. 用户输入: v-model ` `

component

// 定义名为 todo-item 的新组件
Vue.component('todo-item', {
  template: '<li>这是个待办项</li>'
})

##

Vue class / instance

var vm = new Vue({

})

Options/*

  1. el: 锚点
  2. data: 要绑定的数据
  3. computed: 复杂属性
  4. props: 对外的属性接口,类似 C# /Android 中 view 等属性。
  5. methods: 方法

API

global config

  • silent
  • optionMergeStrategies
  • devtools
  • errorHandler
  • warnHandler
  • ignoredElements
  • keyCodes
  • performance
  • productionTip

Global API

  • Vue.extend(options)
  • Vue.nextTick([callback, context])
  • Vue.set(target, key, value)
  • Vue.delete(target, key)
  • Vue.directive(id, [definition])
  • Vue.filter(id, [definition])
  • Vue.component(id, [definition])
  • Vue.use(plugin)
  • Vue.mixin(mixin)
  • Vue.compile(template)
  • Vue.version

Options/data

  • data
  • props: A list/hash of attributes that are exposed to accept data from the parent component.
  • propsData: initial / default data for props
  • computed:

Questions

  1. what is javascript prototype?

    fdf

  2. what is javascript context this?

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

javascript context/ scope/ this

  • context/ this: function

context means this keyword

Context is most often determined by how a function is invoked. When a function is called as a method of an object, this is set to the object the method is called on:

So for a global function, this refers to ‘window’ in browser

  • scope: variable
    • local
    • Global
    • block: let/ const

    Understanding Scope and Context in JavaScript

  • Execution Context

  • Execution stack

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

The event model for GUI application

Compare javascript event model and Android event model.

Tree structure of the GUI elements architecture.

The root window

Draw

Measure

render

Event

Chaining

registration

unregister

fire

Event propagation.

Animation framework

jQuery

  1. $.each() v.s. $(“.cls”).each()
  2. document, window, $, jQuery()
  3. $() $(document).ready(function(){})
  4. $.fn.method, $.method
  5. $( document ).ready(), window.onload()
  6. $().call(“param”, callback), $().call(“param”, function(){ callback(param1, param2)}),
  7. Avoiding conflicting with other libs
     <!-- Putting jQuery into no-conflict mode. -->
     <script src="prototype.js"></script>
     <script src="jquery.js"></script>
     <script>
    
     var $j = jQuery.noConflict();
     // $j is now an alias to the jQuery function; creating the new alias is optional.
    
     $j(document).ready(function() {
         $j( "div" ).hide();
     });
    
     // The $ variable now has the prototype meaning, which is a shortcut for
     // document.getElementById(). mainDiv below is a DOM element, not a jQuery object.
     window.onload = function() {
         var mainDiv = $( "main" );
     }
    
     </script>
    
  8. Attributes
    • setter: .attr(key, value)
    • getter: .attr(key)
  9. Selecting Elements
    • Selecting Elements by ID $( "#myId" ); // Note IDs must be unique per page. * Selecting Elements by Class Name $( “.myClass” ); link Selecting Elements by Attribute 1 $( “input[name=’first_name’]” ); link Selecting Elements by Compound CSS Selector 1 $( “#contents ul.people li” ); link Selecting Elements with a Comma-separated List of Selectors 1 $( “div.myClass, ul.people” );

The jQuery object and the native DOM element. jQuery element is like a wrapper on a native DOM element, so you can use a lot of convinience methods and properties provided by jQuery. This is a bit like design pattern facet.

 var jQElem = $(domElem)
 var element = $(this)

Rapoo Mechanical Keyboard

Rapoo V500s I really cannot bear the awkward of the Mac Pro Keyboard for long time. Yesterday I bought this the Rapoo V500s Mechanical Keyboard. It is pretty shinning and easy to use. The most importantly, it is cost-effectve and simple to perform. The sound of percussion the keyboard is crisp and rhythmic.

Keyboard shortcuts and combo keys

FN+ key Usage
  W exchange the function of WSAD and
  WIN WIN LOCK
  SCROLL Backlight Switch mode
  F1 Homepage
  F2 Email
  F3 Calculator
  F4 Search
  F5 Player
  F6 Previous Music
  F7 Next Music
  F8 Play/Pause Music
  F9 Halt Key
  F10 Mute Key
  F11 Volume-
  F12 Volume+
  Backlights Brightness-
  Backlights Brightness+
  None
  None

Tech Positions in IT companies

Microsoft

Principle Dev Software

Google

Fix Vim

One day I updated my MacOS to Sierra, it broke a lot of toolchain on my system.

And the most importantly, VIM is broken.

➜  _posts git:(master) ✗ vim
dyld: Library not loaded: /usr/local/opt/ruby/lib/libruby.2.3.0.dylib
  Referenced from: /usr/local/bin/vim
  Reason: image not found
[1]    6993 abort      vim

Run brew doctor to pinpoint the issue.

➜  _posts git:(master) ✗ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!
...
.
.
.

Warning: Some installed formula are missing dependencies.
You should `brew install` the missing dependencies:
  brew install libev libsodium udns

Run `brew missing` for more details.
➜  _posts git:(master) ✗ brew install libev libsodium udns
Updating Homebrew...
...
.
.
.

➜  _posts git:(master) ✗ brew upgrade vim
==> Upgrading 1 outdated package, with result:
vim 8.0.0495
...
.
.
.

➜  _posts git:(master) ✗ which vim
/usr/local/bin/vim
➜  _posts git:(master) ✗ vim
➜  _posts git:(master) ✗

Vim got fixed!

SIP System Integration Protection

sudo gem install bundler
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/rbconfig.rb:213: warning: Insecure world writable dir /Users/lucas/dev/android/dex2jar/dex2jar-0.0.9.15 in PATH, mode 040777
Fetching: bundler-1.13.1.gem (100%)
ERROR:  While executing gem ... (Errno::EPERM)
   Operation not permitted - /usr/bin/bundle

Google it find https://github.com/bundler/bundler/issues/4065

After adding option -n /usr/local/bin, it works.

sudo gem install bundler -n /usr/local/bin
Successfully installed bundler-1.13.1
Parsing documentation for bundler-1.13.1
Installing ri documentation for bundler-1.13.1
1 gem installed

http://apple.stackexchange.com/questions/193368/what-is-the-rootless-feature-in-el-capitan-really

  • You can’t modify anything in /System, /bin, /sbin, or /usr (except /usr/local); or any of the built-in apps and utilities. Only Installer and software update can modify these areas, and even they only do it when installing Apple-signed packages. But since normal OS X-style customizations go in /Library (or ~/Library, or /Applications), and unix-style customizations (e.g. Homebrew) go in /usr/local (or sometimes /etc or /opt), this shouldn’t be a big deal. It also prevents block-level writes to the startup disk, so you can’t bypass it that way.

The full list of restricted directories (and exceptions like /usr/local and a few others) is in /System/Library/Sandbox/rootless.conf. Of course, this file is itself in a restricted area.

When you upgrade to El Capitan, it moves any “unauthorized” files from restricted areas to /Library/SystemMigration/History/Migration-(some UUID)/QuarantineRoot/.

  • You can’t attach to system processes (e.g. those running from those system locations) for things like debugging (or changing what dynamic libraries they load, or some other things). Again, not too much of a big deal; developers can still debug their own programs.

This does block some significant things like injecting code into the built-in Apple apps (notably the Finder). It also means that dtrace-based tools for system monitoring (e.g. opensnoop) will not be able to monitor & report on many system processes.

  • You can’t load kernel extensions (kexts) unless they’re properly signed (i.e. by Apple or an Apple-approved developer). Note that this replaces the old system for enforcing kext signing (and the old ways of bypassing it). But since v10.10.4 Apple has had a way to enable trim support for third-party SSDs, the #1 reason to use unsigned kexts has gone away.

字符串匹配算法

Suffix Tree 后缀树

模式匹配问题

字符串模式匹配问题定义: -输入: 一个字符串模式 Pattern, 一个字符串文本 Text. –输出: 模式Pattern 作为子串出现在文本 Text 中的所有位置.

近似模式匹配问题

近似字符串模式匹配问题定义: -输入: 一个字符串模式 Pattern, 一个字符串文本 Text,正整数 d. –输出: 与模式 Pattern 距离最多为 d 的所有子串出现在文本 Text 中的所有位置.

Readings for Year 2016

Previous reading for Year 2013 & 2014: http://xianminx.github.io/2015/01/01/reading-list/

2016年9月在读

  1. 清明上河图密码 : 隐藏在千古名画中的阴谋与杀局 by 冶文彪
  2. 清明上河图密码2 : 隐藏在千古名画中的阴谋与杀局 by 冶文彪
  3. 清明上河图密码3 : 隐藏在千古名画中的阴谋与杀局 by 冶文彪
  4. 禅茶濯剑录 by 诸相非
  5. 北京折叠 by 郝景芳
  6. 边城 by 沈从文
  7. 罗生门 by 〔日〕芥川龙之介 1.巴黎圣母院 by 〔法〕维克多·雨果 1. by 1. by 1. by

Loading resources from other Android applications.

This post tells you how to “steal” data and code from other Android applications at Runtime.

At the core of the trick is the method called “Context.createPackageContext”. Use this method to create a context for the applications you want to retrieve data from.

After building up a Context for the other application, get the “Resources” object by “context.getResources”, then with the new Resources object, you can do whatever you want. Check the following example:

This example shows how to retrieve all string values from the famous app “WeChat”.

private void testCreatePackageContext() {
    try {
        String packageName = "com.tencent.mm";
        Context c = createPackageContext(packageName,
                Context.CONTEXT_INCLUDE_CODE | Context.CONTEXT_IGNORE_SECURITY);
        Resources otherR = c.getResources();

        Log.i(TAG, "classloader=" + this.getClassLoader());

        Log.i(TAG, "classloader=" + c.getClassLoader());

        Class RClass = Class.forName(packageName + ".a", true, c.getClassLoader());
        Class[] RTypes = RClass.getDeclaredClasses();
        for (Class RType : RTypes) {
            Field[] fields = RType.getDeclaredFields();
            for (Field f : fields) {
                try {
                    int id = f.getInt(null);
                    String formatStr = String.format("%s, %s=0x%x", otherR.getResourceName(id), f, id);
                    if ("string".equalsIgnoreCase(otherR.getResourceTypeName(id))) {
                        formatStr = String.format("%s, %s=0x%x, value=%s", otherR.getResourceName(id), f, id, otherR.getString(id));
                        Log.i(TAG, formatStr);
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }
    } catch (PackageManager.NameNotFoundException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }

}

ClassLoader

18906   android.lucas.swip..  I  classloader=dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/android.lucas.swipeback.demo-2/base.apk"],nativeLibraryDirectories=[/vendor/lib, /system/lib]]]
18906   android.lucas.swip..  I  classloader=dalvik.system.PathClassLoader[DexPathList[[zip file "/system/framework/com.google.android.maps.jar", zip file "/data/app/com.facebook.katana-1/base.apk"],nativeLibraryDirectories=[/data/app/com.facebook.katana-1/lib/arm, /vendor/lib, /system/lib]]]

As the above log shows, 2 classloaders were built, one for the current application, and the other for “WeChat” application.

String Resources

The following log prints out all string resources contained in “WeChat” package. Similarly, you can list all other major types of resources, such as layouts, drawables, raw assets, etc.

28058   android.lucas.swip..  I  com.tencent.mm:string/cje, public static final int com.tencent.mm.a$m.dtP=0x7f091187, value=Reminder: You can only link your own bank cards.
28058   android.lucas.swip..  I  com.tencent.mm:string/ci0, public static final int com.tencent.mm.a$m.dtQ=0x7f091153, value=Last 4 digits
28058   android.lucas.swip..  I  com.tencent.mm:string/cjd, public static final int com.tencent.mm.a$m.dtR=0x7f091186, value=Your information will be encrypted
28058   android.lucas.swip..  I  com.tencent.mm:string/ciz, public static final int com.tencent.mm.a$m.dtS=0x7f091177, value=Add Now
28058   android.lucas.swip..  I  com.tencent.mm:string/cjt, public static final int com.tencent.mm.a$m.dtT=0x7f091196, value=You've already linked this card on WeChat Payment
28058   android.lucas.swip..  I  com.tencent.mm:string/ciy, public static final int com.tencent.mm.a$m.dtU=0x7f091176, value=Open Now
28058   android.lucas.swip..  I  com.tencent.mm:string/ci3, public static final int com.tencent.mm.a$m.dtV=0x7f091156, value=Registered with bank
28058   android.lucas.swip..  I  com.tencent.mm:string/cjg, public static final int com.tencent.mm.a$m.dtW=0x7f091189, value=Cardholder Description
28058   android.lucas.swip..  I  com.tencent.mm:string/cht, public static final int com.tencent.mm.a$m.dtX=0x7f09114c, value=Enter bank card number
28058   android.lucas.swip..  I  com.tencent.mm:string/cjk, public static final int com.tencent.mm.a$m.dtY=0x7f09118d, value=Phone Number
28058   android.lucas.swip..  I  com.tencent.mm:string/cjo, public static final int com.tencent.mm.a$m.dtZ=0x7f091191, value=Pay via both payment password and SMS verification
28058   android.lucas.swip..  I  com.tencent.mm:string/crn, public static final int com.tencent.mm.a$m.dta=0x7f0912b7, value=Details
28058   android.lucas.swip..  I  com.tencent.mm:string/cro, public static final int com.tencent.mm.a$m.dtb=0x7f0912b8, value=Top-up successful
28058   android.lucas.swip..  I  com.tencent.mm:string/crm, public static final int com.tencent.mm.a$m.dtc=0x7f0912b6, value=Details
28058   android.lucas.swip..  I  com.tencent.mm:string/cra, public static final int com.tencent.mm.a$m.dtd=0x7f0912aa, value=Amount (%s)
28058   android.lucas.swip..  I  com.tencent.mm:string/crc, public static final int com.tencent.mm.a$m.dte=0x7f0912ac, value=Input a correct amount
28058   android.lucas.swip..  I  com.tencent.mm:string/cr_, public static final int com.tencent.mm.a$m.dtf=0x7f0912a9, value=Balance Top-up
28058   android.lucas.swip..  I  com.tencent.mm:string/crv, public static final int com.tencent.mm.a$m.dtg=0x7f0912bf, value=Select a bank card
28058   android.lucas.swip..  I  com.tencent.mm:string/cmm, public static final int com.tencent.mm.a$m.dth=0x7f0911fe, value=The bank's system is currently undergoing maintena

Publish Android Library

Like you, as an Android developer that has benefitted from Github open source libraries a lot for long time, I start thinking how to repay this community. The best way, of course, is to share my contribution by publishing android library code. This post tells what to follow when you want to publish your Android library project on Github.

Github & Git

The first of the first, you should take some time to get familiar with Git and Github.

Gradle & Android Stuido

Familiarize yourself with Gradle and Android Studio.

Choose a License

Choose a license for your project. If you do not know how to, check this page: http://choosealicense.com/

“Apache License 2.0” is common to Android open source libraries on Github.

Add a readme

Add “readme.md” file to tell people what the project is about and how to use it.

Publish

When you are done with all the library code and Android Studio / gradle project code stuff, it is time to share it.

Android now jcenter uses by default. It is a good idea to share your library by jcenter too so that other Android developers and use your library by just adding gradle dependency, such as:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile 'com.android.support:appcompat-v7:21.0.3'
    compile 'android.lucas:swipeback:0.0.1'
}

There is no direct support in Android Studio or gradle now and it is a bit tricky to set up.

The post [TUT] Locally release an Android Library for JCenter or Maven Central inclusion is a good start to accomplish the task. To simplify the life, you can customize your library project build.gradle file based on the following template:

https://gist.github.com/d860969fdda132b6d892.git

Then publish with command:

./gradew bintrayUpload

Github page

Give it a dedicated website or webpage makes your open source library project look professional. Github provides “Github Pages” feature so that you can create a page for your project very easily.

Project->Setting -> Options -> Github Pages -> Automatic Page Generator

Then follow the instructions to create a webpage for your project.

Share it on Twitter, G+, Facebook, etc

Finally, after all these done, share it to all social platforms to let people find it.

Git 多账号

git 账号

git 用户经常混淆用户和ssh key 等概念。git码农们经常泡在github,bitbucket 等网站,一方面需要给公司写代码,另外一方面不干寂寞的写开源代码,需要两个Email账户。一个是个人的(personal use),用户在github等网站使用, 比如 gitfan@gmail.com, 另一个是工作用公司邮箱, 比如叫 xiaoming@corp.com 在commit 时候如何区分开来呢?

  • 对于个人项目, 可以在项目下,使用 git config user.email gitfan@gmail.com 添加email
  • 对于公司项目, 可以使用 git config user.email xiaoming@corp.com

但是如果每一个项目都这样配置, 又太麻烦了。 如果公司项目比较多, 可以将xiaoming@corp.com 加到git global 配置中

git config —global user.email xiaoming@corp.com

相反, 如果个人项目比较多, 可以

git config —global user.email gitfan@gmail.com

使用git log 可以看到使用哪个账户commit的:

commit 977277ecc9ebccb2ddb93b99e69d71efbce9e7e3
Author: lucas <xuxianming@umeng.com>
Date:   Fri Feb 27 13:47:57 2015 +0800

    inital project setup

commit 1a82f9456f86dca4d284a56024834ebfb6f26c4a
Author: lucas <xianminx@gmail.com>
Date:   Fri Feb 27 13:37:57 2015 +0800

    Initial commit
(END)

同样,email前面的名字也同理可以配置。 对于—global的配置, git 存储在~/.git/config 当中 对于项目的配置, git 存储在 ./.git/config 项目multi下

➜  SwipeBack git:(master) less .git/config

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
        ignorecase = true
        precomposeunicode = true
[remote "origin"]
        url = git@github.com:xianminx/SwipeBack.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
[user]
        email = xianminx@gmail.com
.git/config (END)

这样配置项目使用的user.email和user.name 就能为公司项目和个人项目区分开用户账户了。

关于git 和 ssh key

ssh key 和git 本地使用没有关系。 如果需要同服务器同步代码, 才会用到。 git通信协议底层使用ssh, 故需要ssh key。 公司用的ssh key 多是用公司邮箱生成的, 如xiaoming@corp.com , 同公司代码库通信或者登陆公司服务器,都使用此key。 ssh key 默认存储在~/.ssh/目录下,如果将公司和个人区分开来呢?

简单回答是不需要, 只用将pub key 加到github 上就可以了, 这个pub key 是可以公开的, 故将公司用的ssh key pubkey 放到github.com 也无妨。 SSH key 和git 用户是两个独立的概念, 不应该混淆在一起 但是如果说一定要区分呢? 可以参考 http://stackoverflow.com/questions/3225862/multiple-github-accounts-ssh-config

在~/.ssh/config 中为每一个ssh server 使用IdentityFile 参数配置好private key.

Step 2: ssh config Set up multiple ssh profiles by creating/modifying ~/.ssh/config. Note the slightly differing ‘Host’ values:

# Default GitHub
Host github.com
    HostName github.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa

# Work GitHub
Host work.github.com
    HostName github.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa_work

Android SwipeBack pattern

Intro

It has been a widely used pattern by swiping from the left edge of the screen to return to the previous page or screen. iOS Safari presents such a feature. This navigation pattern is natural and convient, and it would be great to Android developers and users if it can be ported to Android platform. Take a look for the illustration.

The target of this project is to achieve the effect shown by iOS Safari, on Android Activity transition, like native Activity transition animation shown. Fragment transition animation is easy to implement by using ViewPager etc. and thus not discussed here.

Android implementation

how to: If the user swipes into the screen from the outside of the left edge, the previous page slides in from the left edge while the current page slides out to the right edge of the screen.

It looks like this:

Illusiveness

To make it more illusive, we can slide the previous page from 75% X-axis, that is, the previous page has already slided in 75% percent when the finder touches the left edge, while the content covered by the current page.

The pace of the previous page is 25% of the current page. So when the current page slides from the left to the right by 100%, the underlying previous page just 25%, which makes it 100% filling the screen perfectly.

This illusiveness is what Safari on iOS does.

To implement the effect on Android, we have to solve the following obstacles:

  1. Gesture detection
  2. To show part of the Activity / Page on the screen.

To show the animation, we can use the native animation support for the activity but controls it manually.

The most difficult part is to show 2 Activities simultaneously on the screen.

The solutions I found online are all workarounds by using Fragment.

The open source github projects sockeqwe/SwipeBack and ikew0ng/SwipeBackLayout replaces the decor view and show the swipe gesture, but cannot show the previous Activity correctly.

Steps

  1. Use GestureDetector to detect the touch event on Activity and set the root decorView of the Activity to the moved position.
  2. override Activity’s onTouchEvent method.
  3. Set the Activity theme to Translucent. ‘’

API Design

call android.lucas.swipeback.SwipeBack.enable()

Creating a new Activity is a bad idea because users may use different types of Activities and this also obeys the rule of “Prefer composition to inheritence”.

setBackgroundDrawable method takes effect only the parameter is TRANSPARENT. Setting the drawable to any other drawable does not work.

Add a translucent theme to the activity

<!-- Base application theme. -->
<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
  <item name="android:windowIsTranslucent">true</item>
  <item name="android:windowBackground">@android:color/transparent</item>
  <!-- Customize your theme here. -->
</style>

Another Idea: create a layout and insert it into decorview as a child and parent of LinearLayout. Intercept touch event on this layout.

References

  1. Dragging With ViewDragHelper
  2. Each Navigation Drawer Hides a ViewDragHelper
  3. sockeqwe/SwipeBack
  4. ikew0ng/SwipeBackLayout
  5. How to Implement a Floating Activity in an Android App

Readings for Year 2015

Previous reading for Year 2013 & 2014: http://xianminx.github.io/2013/11/07/reading-list/

2015年1月在读

  1. 讲谈社•中国的历史(套装共10册) by (日) 宫本一夫 等
  2. Resource Revolution: How to Capture the Biggest Business Opportunity in a Century by Stefan Heck / Matt Rogers
  3. 算法帝国 by 克里斯托弗•斯坦纳

2015年3月在读

  1. 论人类不平等的起源 by 卢梭
  2. 人类群星闪耀时 by 〔奥〕茨威格
  3. The Glory and the Dream: A Narrative History of America, 1932-1972 (Goodreads) by William Manchester
  4. Design Patterns - Elements of Reusable Object-Oriented Software by Erich Gamma / Richard Helm / Ralph Johnson / John Vlissides
  5. Notes on the Synthesis of Form by Christopher Alexander
  6. Refactoring - Improving the Design of Existing Code by Martin Fowler / Kent Beck / John Brant / William Opdyke / Don Roberts

2015年4月在读

  1. 僧侣与哲学家 by 珍·法蘭可斯雷蒙 / 馬錫·理察
  2. 宝瓶同谋:大数据时代的思想聚变 by [美] 玛丽琳. 弗格森

2015年08月在读

  1. 光荣与梦想 by 威廉·曼彻斯特
  2. 从0到1 by 〔美〕蒂尔〔美〕马斯特斯

2014 Leftover

  1. Predictably Irrational, The Hidden Forces That Shape Our Decisions , Revised and Expanded Edition by Dan Ariely
  2. Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma / Richard Helm / Ralph Johnson / John Vlissides

  3. 动物庄园 by 乔治·奥威尔
  4. 1984 by George Orwell
  5. 数学与猜想(第1卷) by G. 波利亚
  6. 数学与猜想(第2卷) by G. 波利亚
  7. 怎样解题:数学思维的新方法 by G. 波利亚
  8. 你就是极客! - 软件开发人员生存指南 by Michael Lopp
  9. 快速阅读 by Tony Buzan
  10. 你一定爱读的极简欧洲史 by John Hirst

  11. 你一定爱读的极简欧洲史 by John Hirst
  12. Android软件安全与逆向分析 by 丰生强
  13. 咖啡大全 by 田口護
  14. 蒋公的面子 by 温方伊
  15. 中午吃什麼? 一個經濟學家的無星級開胃指南 by Tyler Cowen
  16. 长尾理论2.0 by 安德森

  17. 失控 by 凯文·凯利
  18. 冰与火之歌(卷一) by [美] 乔治·R. R. 马丁

  19. 三体 by 刘慈欣
  20. 信息简史 by 詹姆斯·格雷克
  21. The Singularity Is Near (奇点临近) by Ray Kurzweil
  22. 信息简史 by 詹姆斯·格雷克
  23. 大数据 : 正在到来的数据革命,以及它如何改变政府、商业与我们的生活 by 涂子沛

  24. 失控 by 凯文·凯利
  25. 奇点临近 by Ray Kurzweil
  26. 西藏生死书 by 索甲仁波切
  27. 讲谈社•中国的历史(套装共10册) by (日) 宫本一夫 等
  28. 宝瓶同谋:大数据时代的思想聚变 by [美] 玛丽琳. 弗格森
  29. 定价未来:撼动华尔街的量化金融史[美]乔治G.斯皮罗(George G. Szpiro)
  30. 僧侣与哲学家 by 珍·法蘭可斯雷蒙 / 馬錫·理察

  31. 自控力 by 凯利·麦格尼格尔
  32. 怪诞心理学2 by 理查德·怀斯曼
  33. Android 内核剖析 by 柯元旦
  34. 走出思维的误区
  35. 认知盈余
  36. 游戏改变世界
  37. Java并发编程实战 by Brian Goetz / Tim Peierls / Joshua Bloch / Joseph Bowbeer / David Holmes / Doug Le1.
  38. [走出思维的误区:批判性思维指南(修订第9版)]
  39. 对”伪心理学”说不(第8版)(How to think straight about psychology) by Keith E. Stanovich
  40. [找寻逝去的自我:大脑、心灵和往事的记忆]
  41. HIERARCHICAL TEMPORAL MEMORY including HTM Cortical Learning Algorithms
  42. On Intelligence by Jeff Hawkins.
  43. 竞争战略 by 迈克尔·波特
  44. 编程珠玑 by Jon Bentley
  45. 文明之光 (一、二)by 吴军

散文

  1. 丰子恺
  2. 木心 《圆光》
  3. 弘一法师 (李叔同)
  4. 汪增祺
  5. 陈之藩《剑河倒影》
  6. 简媜,杨绛,汪曾祺,张炜,北岛,野夫,张晓风,周作人,史铁生,朱自清
  7. 陈丹青的《荒废集》《退步集》
  8. 胡兰成的《今生今世》
  9. 章诒和的《伶人往事》
  10. 林清玄《心的菩提》
  11. 迟子建散文
  12. 梁遇春《春醪集》
  13. 木心先生《哥伦比亚的倒影》
  14. 龙应台《目送》刘瑜《送你一颗子弹》
  15. 陈丹青《多余的素材》
  16. 野夫的《乡关何处》
  17. 阿城–汪曾祺–沈从文–周作人–张岱
  18. 周作人的散文合集,《时光阡陌,你一直未曾走远》
  19. 郁达夫《故都的秋》
  20. 沈从文的《湘行散记》
  21. 林清玄《煮雪》

英文散文

  1. 陆谷孙《20篇:英美现当代散文》
  2. 《The Art of Travel》
  3. 《Beg, Borrow, Steal——A Writer’s Life》
  4. 黄源深编:《英语散文选读》,上海外语教育出版社,2007.
  5. 南建翀编:《经典英语美文》,世界图书出版社,2008.
  6. 杨自伍编:《英国散文名篇欣赏》,上海外语教育出版社,2010.
  7. 推荐看一下网易公开课上苏州大学王腊宝教授的公开课视频: “英语经典美文:从培根到伍尔芙

一些有趣的视频节目

2014年自媒体急速发展, 通过互联网的方式做出了很多高水平的节目, 拜托了电视节目的限制。

罗辑思维

晓说 / 晓松奇谈

奇葩说

财经郎眼

The BIG talk

TED

一刻演讲

听道

Think+

一席

极客公园

Try out Android-L Samples with Android studio

1. 导入

image1

image2

2. 更新文件

samples/android-L/ui/views/FloatingActionButton/FloatingActionButtonBasic/build.gradle

此文件为空, 添加内容


// Top-level build file where you can add configuration options common to all sub-projects/modules.

buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:0.12.+'

        // NOTE: Do not place your application dependencies here; they belong
        // in the individual module build.gradle files
    }
}

allprojects {
    repositories {
        jcenter()
    }
}

3. 更新文件内容

samples/android-L/ui/views/FloatingActionButton/FloatingActionButtonBasic/FloatingActionButtonBasicSample/build.gradle

原始内容



buildscript {
    repositories {
        mavenCentral()
    }

    dependencies {
        classpath 'com.android.tools.build:gradle:0.12.+'
    }
}

apply plugin: 'android'


dependencies {

    compile "com.android.support:support-v4:20.+"
    compile "com.android.support:support-v13:20.+"

}

// The sample build uses multiple directories to
// keep boilerplate and common code separate from
// the main sample code.
List<String> dirs = [
    'main',     // main sample code; look here for the interesting stuff.
    'common',   // components that are reused by multiple samples
    'template'] // boilerplate code that is generated by the sample template process

android {
    compileSdkVersion "android-L"

    buildToolsVersion "20.0.0"

    sourceSets {
        main {
            dirs.each { dir ->
                java.srcDirs "src/${dir}/java"
                res.srcDirs "src/${dir}/res"
            }
        }
        androidTest.setRoot('tests')
        androidTest.java.srcDirs = ['tests/src']

    }

}

更新为:

apply plugin: 'com.android.application'



dependencies {

    compile "com.android.support:support-v4:20.+"
    compile "com.android.support:support-v13:20.+"

}

android {
    compileSdkVersion "android-L"

    buildToolsVersion "20.0.0"

    lintOptions {
        abortOnError false
    }
}

错误

错误Unable to locate a Java Runtime to invoke.

可能是gradlew daemon 进程死掉了。 https://code.google.com/p/android/issues/detail?id=60913

解决方案: 手动杀掉gradle daemon 进程

➜  ClippingBasic  ps aux|grep gradle
lucas           14659   0.0  0.0  2432784    612 s006  S+    4:13PM   0:00.00 grep gradle
lucas           14529   0.0  4.3  3966132 359556   ??  S     4:07PM   0:22.98 /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/bin/java -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xmx1024m -Dfile.encoding=UTF-8 -cp /Users/lucas/.gradle/wrapper/dists/gradle-1.12-bin/64p5p2nte80b6rt6bb068pabi6/gradle-1.12/lib/gradle-launcher-1.12.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 1.12 /Users/lucas/.gradle/daemon 10800000 04d22f9b-874b-469b-8011-f7f2819ec409 -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xmx1024m -Dfile.encoding=UTF-8
➜  ClippingBasic  kill 14529
➜  ClippingBasic  

Could not create plugin of type 'AppPlugin'.

Gradle 版本和gradle android plugin 版本不对 修改gradle 版本

./gradlew --version

 vim ./gradle/wrapper/gradle-wrapper.properties
#Wed Apr 10 15:27:10 PDT 2013
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=http\://services.gradle.org/distributions/gradle-1.12-all.zip

修改 distributionUrl 的值 gradle-1.*-all.zip.

关于Gradle 版本和gradle android plugin 版本的对应关系,见 gradle plugin version compatibility 页面映射表: http://tools.android.com/tech-docs/new-build-system/version-compatibility

Gradle

目标:

  1. 初步了解gradle的DSL 语言,能够看懂, 并简单改写。
  2. 了解Android Gradle build 系统
  3. 为Feedback Android SDK 书写gradle 打包系统,发布。
  4. 生成Eclipse 导入环境。 使得在Eclipse 中可用。

导读

Gradle 是一个通用的编译构建工具。 Gradle 编译脚本使用Groovy的DSL 语言编写。 其强调配置 而不是编程

新的Android 工程使用Gradle 来编译, Android 团队提供了一个官方的Gradle plugin 插件来支持Android 工程使用Gradle 来编译。

Android 新的构建系统使得:

  • 重用代码和资源变得更容易(maven, ivy, aar 依赖管理)
  • 创建应用的变种更容易(收费,免费,等)
  • 配置、扩展、自定义构建过程更容易 (Gradle, 插件结构)
  • 和Android Studio IDE 合二为一(新的Android Studio 直接使用Gradle 编译, 使得脚本和IDE 完全使用一套编译系统)

Gradle 特征:

  • DSL
  • Groovy 构建文件
  • Maven / Ivy 的依赖管理
  • 灵活。 允许最佳实践,但是又不强制
  • plugins
  • IDE

基础

一个Gradle 项目使用根目录下的build.gradle 来描述其构建配置。

  • java-only

    apply plugin: 'java'
    
  • Android

    buildscript {
        repositories {
            mavenCentral()
        }
    
        dependencies {
            classpath 'com.android.tools.build:gradle:0.11.1'
        }
    }
    
    apply plugin: 'android'
    
    android {
        compileSdkVersion 19
        buildToolsVersion "19.0.0"
    }
    
    • buildscript { … } 配置了构建过程需要的代码。 需要注意的是此中的dependencies 只是指定了构建过程需要的依赖, 而不是项目的依赖。
    • android 表明使用android 插件来构建此项目。
    • android{ … } 配置了Android 构建的所有参数。 是Android DSL 的入口。

项目结构

Gradle 利用了习惯而不是配置的原则,尽可能的提供一些合理的默认值。

可以使用”sourceSets” 来配置或者改变默认值

  sourceSets {
      main {
          java {
              srcDir 'src/java'
          }
          resources {
              srcDir 'src/resources'
          }
      }
  }

Android:

  android {
      sourceSets {
          main {
              manifest.srcFile 'AndroidManifest.xml'
              java.srcDirs = ['src']
              resources.srcDirs = ['src']
              aidl.srcDirs = ['src']
              renderscript.srcDirs = ['src']
              res.srcDirs = ['res']
              assets.srcDirs = ['assets']
          }

          androidTest.setRoot('tests')
      }
  }

任务

  • assemble
  • check
  • build
  • clean

Android 任务

  • assemble
    • assembleDebug
    • assembleRelease
  • check
    • lint
  • connectedCheck
    • connectedAndroidTest
    • connectedUiAutomatorTest
  • deviceCheck
  • build
  • clean

自定义构建过程

  • Manifest 参数
    android {
        compileSdkVersion 19
        buildToolsVersion "19.0.0"
    
        defaultConfig {
            versionCode 12
            versionName "2.0"
            minSdkVersion 16
            targetSdkVersion 16
        }
    }
    

构建类型

debug, release instances of BuildType.

  android {
      buildTypes {
          debug {
              applicationIdSuffix ".debug"
          }

          jnidebug.initWith(buildTypes.debug)
          jnidebug {
              packageNameSuffix ".jnidebug"
              jnidebugBuild true
          }
      }
  }

ProGuard

依赖,Android 类库项目, 多项目配置

  • dependencies
    • local
      • compile files(‘libs/foo.jar’)
    • remote
      • compile ‘com.google.guava:guava:11.0.2’
    • multi-project
      • compile project(‘:libraries:lib1’)
        MyProject/
         + app/
         + libraries/
            + lib1/
            + lib2/
      
  • java lib project and android lib project
    • apply plugin: ‘android-library’
    • generate a .aar Android archive output
  • Build flaors and types

Library publication

Testing

Lint support

Build variants

  • 同一个应用的不同版本, 专业版(pro) vs. 免费版
  • 多APK: 不同CPU, 不同系统版本, 不同屏幕。 multi-apk

Product flavors

Build Type + Product Flavor = Build Variant

高级自定义

  • java
  • aapt
  • dex

AAR format

The ‘aar’ bundle is the binary distribution of an Android Library Project.

The file extension is .aar, and the maven artifact type should be aar as well, but the file itself a simple zip file with the following entries: /AndroidManifest.xml (mandatory) /classes.jar (mandatory) /res/ (mandatory) /R.txt (mandatory) /assets/ (optional) /libs/*.jar (optional) /jni//*.so (optional) /proguard.txt (optional) /lint.jar (optional) These entries are directly at the root of the zip file.

The R.txt file is the output of aapt with –output-text-symbols.

Java Plugin

  apply plugin: 'java'

Source sets

  • main
  • test

    Tasks

    Project layout

    sourceSets {
      main {
          java {
              srcDir 'src/java'
          }
          resources {
              srcDir 'src/resources'
          }
      }
    }
    

Dependency management

Eclipse Plugin

  • None
  • Java
  • Groovy
  • Scala
  • War
  • Ear

References

  1. Gradle Java Plugin
  2. Gradle Plugin User Guide
  3. Gradle Build Language Reference

Quickly craft Android apps using open source tools

  • otto
  • Dagger
  • StaggeredGridView
  • Bootstrap

appcompat-v7 Theme demystified

support-v4

增加一些工具类, 可以使用Fragment, NotificationCompat, ViewPager等。 作为一个jar包提供, 更多是一个工具类, 通过jar包方式提供高版本系统的一些API。

support-v7

分三个组成部分:

  • appcompat library
  • gridlayout library
  • mediarouter library

其中最重要的就是appcompat, 为API 7-10 系统提供在11上新加的ActionBar的设计模式。 所以如果你的应用只支持API 11+, 就可以不需要这个library。

appcompat 的核心支持是ActionBar。所以如果你决定你的应用要对API 7-10 支持ActionBar, 需要将Activity 继承 support/v7/app/ActionBarActivity, 并且将activity 的theme 设为Theme.AppCompat。

这些theme都定义在appcompat-v7包中

v0 表示定义在values/values.xml 中 v11 表示定义在values-11/values.xml 中 v14 表示定义在values-14/values.xml 中 app 表示应用定义的style

在v11 之后默认holo 风格

还有对应的Light 风格, 在每个Theme 的后面加上.Light

REST API DESIGN

REST 缩写: REpresentation State Transfer 最初想法有Roy Fielding 在他的博士论文 Architectural Styles and the Design of Network-based Software Architectures 中提出来。

REST 有6大必要条件

  • Uniform Interface 统一接口
    统一接口定义了客户端和服务器之间的接口。简化并解耦了架构, 使得双方都可以独立进化。
    • Resource-based 基于资源 URI as resource identifiers
    • Manipulation of Resources Through Representations
    • Self Descriptive Messages
    • Hypermedia as the engine of Application State
  • Stateless 无状态
    服务器端处理请求需要的状态都包含在客户端发送的请求当中。状态可能是URI的一部分,查询字符串参数, body,也可能在Header 中。URI 唯一标识资源,body 包含了资源的状态。 无状态使得架构更具扩展性
  • Cacheable 可缓存
  • Client-Server 客户端-服务器模型
  • Layered System 分层设计的系统
  • Code on Demand (optional) 按需编码
    Java Applets or javascript

REST enables: performance, scalability, simplicity, modifiability, visibility, portability and reliability.

HTTP 状态码

Tips

  • 使用HTTP 动词
    • GET: 读取某个特定资源或者资源集合
    • PUT: 更新某个资源
    • POST: 创建新的资源。如果找不到合适的动词就用这个动词
    • DELETE: 删除某个资源

Provide Sensible Resource Names

设计一个好的API 80% 的艺术 + 20% 的科学

  • URL中使用标识, 而不是查询字符串, 如 /users/12345
  • 巧妙使用URL 的等级结构特性 来标识资源的结构
  • 为客户端设计, 而不是为数据设计
  • 资源名称使用名词, 使用HTTP 的方法来表示请求的动作部分
  • URL 中使用复数名词
  • 避免使用集合词
  • URL 使用小写, 使用’_‘或者’-‘来分割单词
  • 尽量保持URL 短小

Use HTTP Response Codes to Indicate Status

Offer Both JSON and XML

Create Fine-Grained Resources

Consider Connectedness

Idempotence

幂等性

###

  • API 中的page设计

  • API Design

    • http://developer.github.com/v3
    • http://www.restapitutorial.com/
    • http://docs.python-requests.org/en/latest/
    • https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
    • http://www.slideshare.net/nicolaiarocci/developing-restful-web-apis-with-python-flask-and-mongodb
    • http://docs.python-requests.org/en/latest/community/out-there/#articles-talks

Reference:

  • [django-rest-framework](http://www.django-rest-framework.org/)

Python 学习笔记

May 14, 2014

由于项目的需求, 需要使用Django 技术栈。 Django 是基于Python 语言开发 的流行的Web 框架。 没办法,得学习Python。 由于时间关系, 不能花太多的时间去看大部头的书和文档, 需要快速上手。 在Slideshare 上找到一个不错的入门PPT。

Learn 90% of Python in 90 Minutes by Matt Harrison

Learn 90% of Python in 90 Minutes from Matt Harrison

标题看着很吓人, 也很励志, 90分钟搞定, 很适合我的需求。 start quickly and move fast.

确实大概花了100分钟的时间就看完了。 这个Slides 主要讲Python语言的语法。 多数语言的语法都相似,所有只要你很好的掌握了一门编程语言,学习其他的语言是相当简单容易的。 我之前有几年的C和Java 编程经验, 看起Python的语法,简单很多。

几乎所有基于C的编程语言语法大同小异, 介绍的ToC都会差不多。

主要包括几个方面:

  • 命名
  • 数据类型
  • 方法定义
  • 类定义
  • 类继承
  • 集合: List, Set, Dictionary, Map 等
  • 迭代
  • 操作符
  • 文件操作
  • debug

下面的MindMap 就总结了Python语言的这些方面:

不过有一些方面没有讲到:

  • Exception处理
  • 类库
  • 生态

Python 类库

Python for the web. python 是为web 而设计, 使用python 很大程度上为了于web 交互。 从而不可避免的要有一些web 相关的 库。

json

json 由于AJAX的兴起,大受欢迎, 以至于今天多数的web API 都以JSON 格式提供。 使用python 操作json 也成了日常工作中必不可少的技能。 Django 1.7 内置了新的json 类库, 但是1.7之前, 还是需要使用python 的json 类库.

如果熟悉json 格式的话,学习这个类库会很简单。 Python内部使用dictionary来存储json 数据, 没有class JSONObject 这样的类。

In [10]: type(x)
Out[10]: dict

In [11]: x={'__complex__':True, 'real':1, 'imag':2}

In [12]: type(x)
Out[12]: dict

In [13]: json.dumps(x)
Out[13]: '{"real": 1, "imag": 2, "__complex__": true}'

In [14]: str = json.dumps(x)

In [15]: type(str)
Out[15]: str

In [16]: j = json.loads(str)

In [17]: type(j)
Out[17]: dict

In [18]: j
Out[18]: {u'__complex__': True, u'imag': 2, u'real': 1}

json 类库主要有四个函数:

  • json.dumps() 将python 的dictionary dump 成字符串输出
  • json.loads() 将字符串转化成 dictionary
  • json.JSONEncoder.encode()
  • json.JSONDecoder.decode()

Pretty print json

curl https://github.com/timeline.json |python -mjson.tool

Parse HTML: LXML

Django 使用django.utils.simplejson 类库

requests

Django 学习笔记

April 15, 2014

Writing your first Django app, part 1, Template

  1. Django follows MVC architecture pattern
  2. Install
  3. Hello, World
    • create project
       django-admin.py startproject mysite
    
    • project structure

      ➜  python  tree mysite
      mysite
      ├── manage.py
      └── mysite
          ├── __init__.py
          ├── settings.py
          ├── urls.py
          └── wsgi.py
      
    • Run

  4. settings.py
    • DB
    • debug
    • Applications
  5. Project v.s. Apps
  6. Create App
     $ python manage.py startapp polls
    

April 16, 2014

Writing your first Django app, part 2, Model

  1. start server python manage.py runserver
  2. admin site http://127.0.0.1/admin
  3. Make the poll app modifiable in the admin
  4. Customize the admin form
    • reorder fields
    • add field title
    • add html classes
    • add related objects, StackedInline, TabularInline
  5. Customize the admin change list
    • list_display
    • admin_order_field
    • short_description
    • list_filter
    • search_fields
    • change list pagination
  6. Customize the admin look and feel
    • Customize your project’s templates
    • django template file
    • customize the admin index page

April 16, 2014

Writing your first Django app, part 3, View

  1. Django views are web pages. URL pattern, urlconfs: map url patterns to views.
  2. Write your first view url() arguments, regex, view, kwargs, name
  3. write views that actually do something
    • return HttpResponse
    • Raise exception such as Http404
    • Django db API
    • Template System
    • loader
    • render
    • removing hardcoded urls in template by using {% %}
  4. Namespacing URL names

April 16, 2014

Writing your first Django app, part 4, Form

  1. Write a simple form
  2. Use generic views: Less code is better

April 16, 2014

Writing your first Django app, part 5, Testing

  1. automated testing
  2. basic testing strategy
  3. first test
  4. test a view

April 16, 2014

Writing your first Django app, part 6, static files

  1. Customize your app’s look and feel
  2. Adding a background-image

April 22, 2014

Advanced tutorial: How to write reusable apps

  1. package django apps to a tar file
  2. virtualenv
  3. python package index PyPI

May 14, 2014

i18n, l10n

Logging

May 15, 2014

Python 包管理

大部分现代语言都需要有包管理的机制, 使得开发的组件可以重用。

Java使用jar 打包组件, 使用maven来管理组件和组件之间的依赖。

Python 使用pip来安装和管理本地的组件。 其他工具有easy_install, distribute

pipPyPI (Python Package Index) 线上网站查找需要的包,如同maven 从mvncentral 查找jar包。

Python 使用Eggs, 如同Java 使用jar

WSGI

Web Server Gateway interface

Werkzeug

Werkzeug is a WSGI utility for Python

Q

  1. Run server by python manage.py runserver. How to start it as background service?

盐咖啡


图片来自 http://blog.khymos.org/2010/03/21/a-pinch-of-salt-for-your-coffee-sir/

某天在豆瓣上闲逛, 看到一个标题, “我是咖啡豆, 我为自己带盐”。 这本是借鉴陈欧为聚美优品做的广告,“我为自己代言”。 只是网络上的人越来故意的写错字用以卖萌。 听过各种的咖啡,包括诡异的猫屎咖啡, 但好像是从来没见过咖啡加盐。 想必会有一番风味。不过细想一下,这种简单搭配不存在, 说明味道不会怎么样。在网上搜索了一下, 还真有“盐咖啡”的存在。 讲的是一个美丽感人的爱情故事。

用Google英文搜索了一下, 发现真的存在咖啡加盐的喝法。 Life Hacker 上就有一篇讲到, 当咖啡煮坏了, 太苦的时候, 加一点盐有助于减轻苦味。

其实生活本无所定式和模式,尝试不同的组合, 即使没有发现很好的结果,或许尝试的过程本身就是一种享受。 有那么多的配料存在, 糖, 盐, 孜然, 胡椒, 蜂蜜,奶粉,牛奶,姜,各种各样, 和咖啡,茶混起来,或许能够找到适合的口味。 姜和茶就是老家老辈人很享受的组合。蒙古人会把牛奶和茶煮到一起。

附 盐咖啡的故事

http://article.yeeyan.org/view/PopinJay/118840?from=rss_related
盐咖啡
译者: 梁軍
发表时间:2010-07-19浏览量:2685评论数:0挑错数:0
“盐咖啡的味道如何?”她回答说,“甜到心里!”

 在一个聚会上,他邂逅了她。非常出众的她得到许多男孩的追求。他却非常普通,没人注意。
聚会结束后,他邀请她一起去喝咖啡,她有些吃惊,但是出于礼貌,还是答应了他的邀请。他们坐在一家环境不错的咖啡馆,他非常紧张,一句话也说不出,她却觉得很不自在,心想着,“拜托,让我回家吧……”
突然,他对服务员说,“请给我一点盐好吗? 我想把它加在我的咖啡里。”咖啡馆里每个人都惊奇地看着他——真是个怪人。他的脸变红了,但他仍把盐放入他的咖啡中,并喝下了那杯咖啡。她好奇地问他,“你为什么有这样的嗜好?”他回答,“小时候我住得离大海很近,我喜欢在海里玩,我能够感觉大海的味道,就象这杯盐咖啡的味道。现在,每当我喝盐咖啡,我总是想起我的童年、我的故乡,我非常想念我的家乡和依然生活在那里的父母。” 他说到这些,眼泪涌了出来。她深深地被感动了,
那是发自他心底的真情,一个能够说出对家乡的思念的男人,一定是一个爱家,对家人关怀,对家庭有责任感的人……随后,她也开始说起自己遥远的家乡,她的童年,她的家人。
那真是一次非常不错的交谈,也是他们爱情故事的一个美丽开端。从那以后,他们继续约会。她发现事实上他是一个能满足她要求的男人:他有一颗善良的心,能容忍,多情、细心。他真是一个好人,而她差点错过了他! 真要谢谢他的盐咖啡!
接着,正像每一个美丽的爱情故事,公主嫁给了王子。然后,他们过着幸福的生活……并且,每次为他煮咖啡,她都加上盐,因为她认为这是他喜欢的方式。
四十年后,他去世了,留给她一封信,信中写道,“我最亲爱的,请原谅我。原谅我说了一个持续一生的谎言。这是我对你唯一的一次谎言——盐咖啡。记得我们第一次约会吗?当时我太紧张了,其实我想要一些糖,但却说成了盐。当时我觉得难以改口,只好随其发展了,却怎么也没想到那会是我们交往的开始! 在后来有许多次,我都想告诉你事实真相,但是我太担心,因为答应过你绝不说谎。现在我快要离开人世了,我不担心这么做了。所以,我告诉你,我不喜欢盐咖啡,那味道真是又怪、又难喝……但是我却喝它喝了一辈子。自从我认识你之后,为你做的事,我从不后悔。在我漫长的一生中,拥有你是我最大的幸福。如果我再活一次,我仍想与你相知,再次与你结合,过一辈子,即使我必须再喝盐咖啡我也愿意。”
她的泪水把信纸全浸湿了。后来有一天,有人问起她,“盐咖啡的味道如何?”她回答说,“甜到心里!”

Genymotion

Genymotion is based on VirtualBox, most issues come from VirtualBox. If you cannot start a Genymotion emulator, try reinstall VirtualBox, or repair VirtualBox.

Resolving Genymotion problems on Mac

  • genymotion error

sudo /Library/StartupItems/VirtualBox/VirtualBox restart

  • VERR_SUPLIB_WORLD_WRITABLE

open Disk Utility (search Spotlight), try Repair Disk Permission

Restart Genymotion

Building CyanogenMod for Google Nexus 4

Android Derivative ROMs

  • AOSP
  • CyanogenMod
  • Kandroid
  • MIUI
  • Aliyun OS
  • AOKP
  • Baidu Yi
  • LeWa OS
  • OPhone
  • Replicant
  • XobotOS
  • Stock
    • HTC Sense
    • Samsung TouchWiz
    • LG Optimus UI
    • Sony UI

For more, check Wikipedia.

An excellent video tutorial on Youtube

Components

Hardware driver (proprietary)

Using ./extract-files to extract vendor drivers from real device.

/media/android/cm/system/device/lge/mako$ ll extract-files.sh 
-rwxrwxr-x 1 lucas lucas 354 Nov 18 23:19 extract-files.sh*

This pulls data from device’s /system directory to /vendor/lge/mako/proprietary directory.’

lucas@lucas-ubuntu:/media/android/cm/system/vendor/lge/mako/proprietary$ ll
total 24
drwxr-xr-x 6 lucas lucas 4096 Nov 19 11:29 ./
drwxr-xr-x 3 lucas lucas 4096 Nov 19 10:17 ../
drwxr-xr-x 2 lucas lucas 4096 Nov 19 11:29 bin/
drwxr-xr-x 3 lucas lucas 4096 Nov 19 11:29 etc/
drwxr-xr-x 5 lucas lucas 4096 Nov 19 11:29 lib/
drwxr-xr-x 4 lucas lucas 4096 Nov 19 11:29 vendor/

These includes drivers for CPU, NFC fromBroadcom, Camera, Sensors, Audio, DRM, Cryptography from LG, and imgs, GSM, Camera, GPS, Wi-Fi, Bluetooth, Sensors, Media, DRM, DSP, USB from Qualcomm. All these drivers are proprietary.

Kernel

The next thing is to build the kernel source use breakfast mako (code name for Nexus 4) to download kernel source code from github

Android

Building

Enviroment setup

  • Ubuntu 64 bit
  • Android SDK
  • JDK
  • Python

Download source

  • repo
  • mkdir, /media/android/cm/system/

CM source file structure

link: http://wiki.cyanogenmod.org/w/Doc:_the_cm_source

  • bionic
  • build
  • bootable: cwm recovery
  • dalvik
  • art, replacement of dalvik, android runtime
  • device, board support packages and configurations for a particular device. device//, /device/] asus/grouper, device/lge/mako, where to put customized stuff for each device. `breakfast grouper && repo sync`
  • docs: android open source website
  • external, non-android utilities, with Android.mk to work with android build system
  • frameworks, Android core
  • hardware, platform/hardware specific libs
  • kernel, /kernel//
  • ndk
  • out, make clobber
  • packages, Android standalone apps, like Settings, Browser, Launcher, etc
  • prebuilt(s)
  • system: linux shell, etc.
  • vendor: cm, lge, htc, etc

For Nexus 4:

  • vendor:cm
  • hardware: ti
  • kernel: google/msm
  • device: lge/mako

Repo manifest format

top level ./.repo/default.xml XML file.

  • manifest: the root element of the file.
  • remote: git url shared by one or more projects and the gerrit review server those projects upload changes through.
  • name: short git remote name
  • alias: override name if specified in each project’’s .git/config
  • fetch: git url prefix for all projects which use this remote
  • review: gerrit review server used by repo update
  • default: used when a project does not specify remtoe and revision.
  • manifest-server
  • project: single git repo.
  • annotation
  • remove-project
  • include

Local Manifests

.repo/local_manifests/*.xml


/device/lge/mako/extract-files.sh
adb pull /system/* to /vendor/lge/mako/proprietary



The problems encountered during downloading the cm source

Since the source repo is huge, the downloading process can be intermittant




3. build

breakfast
brunch
lunch


build files

.mk
Makefiles
/build

each module/ subproject has a Android.mk file, telling the build system how to build the module, and where to put the output in the Android directory. 


The built files are put in /out/target/project/CODENAME, with zipped flashable recovery*.zip and fastboot*.img files. 




## $OUT directory
$OUT= /out/target/project/CODENAME

* kernel 
* /system, the /system folder on Android
* /root, contains the files that will be turned into ramdisk loaded and run by the kernel, including init, init.rc, init.CODENAME.rc
* /recovery/root, ramdisk contains the recovery mode 


```sh
mm: make modules, mm -B 
adb sync system

adb remount= adb shell mount -o rw,remount /system

Build Success

/media/android/cm/system/out/target/product/mako
➜  mako  ls -la
total 803408
drwxrwxr-x 10 lucas lucas      4096 Nov 19 21:15 .
drwxrwxr-x  3 lucas lucas      4096 Nov 19 18:33 ..
-rw-rw-r--  1 lucas lucas        19 Nov 19 18:34 android-info.txt
-rw-r--r--  1 lucas lucas   6436864 Nov 19 21:02 boot.img
-rw-rw-r--  1 lucas lucas     34552 Nov 19 18:33 clean_steps.mk
-rw-rw-r--  2 lucas lucas 189761342 Nov 19 21:15 cm-10.2-20131119-UNOFFICIAL-mako.zip
-rw-rw-r--  1 lucas lucas       120 Nov 19 21:15 cm-10.2-20131119-UNOFFICIAL-mako.zip.md5sum
-rw-rw-r--  2 lucas lucas 189761342 Nov 19 21:15 cm_mako-ota-3f402468d6.zip
drwxrwxr-x  2 lucas lucas      4096 Nov 19 20:46 data
drwxrwxr-x  4 lucas lucas      4096 Nov 19 18:39 external
drwxrwxr-x  2 lucas lucas      4096 Nov 19 21:10 fake_packages
-rw-rw-r--  1 lucas lucas     71456 Nov 19 21:13 installed-files.txt
-rwxrwxr-x  1 lucas lucas   5997088 Nov 19 20:36 kernel
drwxrwxr-x 18 lucas lucas      4096 Nov 19 21:13 obj
-rw-rw-r--  1 lucas lucas        49 Nov 19 21:14 ota_script_path
-rw-rw-r--  1 lucas lucas       669 Nov 19 18:33 previous_build_config.mk
-rw-rw-r--  1 lucas lucas    434701 Nov 19 21:01 ramdisk.img
-rw-rw-r--  1 lucas lucas   3845120 Nov 19 21:03 ramdisk-recovery.cpio
-rw-rw-r--  1 lucas lucas   2510001 Nov 19 21:04 ramdisk-recovery.img
drwxrwxr-x  3 lucas lucas      4096 Nov 19 21:02 recovery
-rw-r--r--  1 lucas lucas   8511488 Nov 19 21:04 recovery.img
drwxrwxr-x  9 lucas lucas      4096 Nov 19 21:01 root
drwxrwxr-x  5 lucas lucas      4096 Nov 19 20:59 symbols
drwxrwxr-x 15 lucas lucas      4096 Nov 19 21:01 system
-rw-r--r--  1 lucas lucas 314466532 Nov 19 21:14 system.img
-rw-r--r--  1 lucas lucas 100775392 Nov 19 20:46 userdata.img
➜  mako  

Flashing

  1. Flash ClockworkMod recovery img. fastboot flash recovery recovery.img
  2. Flash
fastboot -w flashall

Building Kernel

  1. Building kernel from source https://android.googlesource.com/device/lge/mako/

  2. Or, using prebuilt kernel image https://android.googlesource.com/device/lge/mako-kernel/

Device

Proprietary Drivers

Binaries for Nexus Devices

https://developers.google.com/android/nexus/drivers

extract-files.sh

extract-broadcom-mako.sh

Restoring to factory state

upgrade to Android 4.4 on Nexus 4

https://developers.google.com/android/nexus/images?hl=fr-FR#occamkrt16s

Note: codename for factory image of Nexus 4 is occam

Android product naming rules

Questions

http://source.android.com/source/known-issues.html

References

  1. http://source.android.com/devices/index.html
  2. Youtube //www.youtube-nocookie.com/embed/1_H4AlQaNa0?rel=0

Readings for Year 2013

说明: 此前由读《暗时间》 催生了读书的计划。 这篇博客用来记录2013 年看过的和想看但没看完或者还没开始看的书。 由于记不太清楚, 个别书可能是2012 年或者更早读过。

更新日志

已读

  1. 暗时间 by 刘未鹏
  2. 如丧 : 我们终于老得可以谈谈未来 by 高晓松
  3. 一个村庄里的中国 by 熊培云
  4. 狂热分子 : 群众运动圣经 by 埃里克·霍弗
  5. 人间失格 by 太宰治
  6. by 莫言
  7. 黑客与画家 by Paul Graham
  8. 天才在左 疯子在右 by 高铭
  9. 奇思妙想: 15位计算机天才及其重大发现 by Dennis E. Shasha / Cathy A. Lazere
  10. Java解惑:Traps, Pitfalls, and Corner Cases by (美)布洛克//加夫特
  11. 编写高质量代码:改善Java程序的151个建议 by 秦小波
  12. 乌合之众:大众心理学研究 by (法)古斯塔夫.勒庞
  13. 演讲之禅:一位技术演讲家的自白 by Scott Berkun

11月在读

  1. 美妙的新世界 by 阿道斯·伦纳德·赫胥黎
  2. Predictably Irrational, The Hidden Forces That Shape Our Decisions , Revised and Expanded Edition by Dan Ariely
  3. Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma / Richard Helm / Ralph Johnson / John Vlissides

2014年1月在读

  1. 动物庄园 by 乔治·奥威尔
  2. 1984 by George Orwell
  3. 第七天 by 余华
  4. 可能与不可能的边界 by Lance Fortnow
  5. 人生中最美妙的事都是免费的 by 尼尔•帕斯理查
  6. 本能: 为什么我们管不住自己? Mean Genes by 特里·伯纳姆
  7. 数学与猜想(第1卷) by G. 波利亚
  8. 数学与猜想(第2卷) by G. 波利亚
  9. 怎样解题:数学思维的新方法 by G. 波利亚
  10. 你就是极客! - 软件开发人员生存指南 by Michael Lopp
  11. 快速阅读 by Tony Buzan

2014年2月在读

  1. 看见 by 柴静
  2. 猫力乱步 by 猫力 读完部分
  3. 你一定爱读的极简欧洲史 by John Hirst

2014年3月在读

  1. 猫力乱步 by 猫力
  2. The Performance of Open Source Applications II Secrets of Mobile Network Performance
  3. 你一定爱读的极简欧洲史 by John Hirst
  4. Android软件安全与逆向分析 by 丰生强
  5. 咖啡大全 by 田口護
  6. 蒋公的面子 by 温方伊
  7. 中午吃什麼? 一個經濟學家的無星級開胃指南 by Tyler Cowen
  8. 长尾理论2.0 by 安德森
  9. 快速阅读 by Tony Buzan

2014年4月在读

  1. 大数据变革: 让客户数据驱动利润奔跑 by 比约•布劳卿 / 拉斯•拉克 / 托马斯•拉姆什

    一本大烂书,完全没有任何引起我兴趣的见解,连新知识点都没有。 建议一小时内看完, 看不完也不要看了。

  2. 至关重要的关系(THE START-UP OF YOU) by Reid Hoffman / Ben Casnocha

    又是一本大烂书,适合在机场等飞机半小时看完的那种。 还不如心灵鸡汤类的成功学, 至少会有一些鸡血故事刺激一下。 自从学了“快速阅读” 的方法后, 看这种书真是可以一目十行。这本书是个奇葩, 看完目录,基本就相当于看完了, 剩下的内容, 找些段子,故事,填充下逻辑就好了。 不知道为什么这本书在豆瓣上评分8.3那么高。 这种商人写书, 真是渣渣。

2014年5月在读

  1. 十六周的人生蜕变: 找到意想不到的自己 by 胡蓉蓉 Vvivi Hu

    小资情调, 认真生活。 文字读起来很舒适。 4星

  2. 失控 by 凯文·凯利
  3. 冰与火之歌(卷一) by [美] 乔治·R. R. 马丁

2014年6月在读

  1. 车库咖啡 by 苏菂(口述) / 王海珍(撰稿)
  2. 互联网思维独孤九剑 by 赵大伟
  3. 三体 by 刘慈欣
  4. 罗辑思维 (Youku 视频节目)
  5. 信息简史 by 詹姆斯·格雷克
  6. The Singularity Is Near by Ray Kurzweil
  7. 互联网思维独孤九剑 by 赵大伟
  8. 互联网新思维:未来十年的企业变形计 by David Kerpen

2014年7月在读

  1. 信息简史 by 詹姆斯·格雷克
  2. 数据之巅:大数据革命,历史、现实与未来 by 涂子沛
  3. 大数据 : 正在到来的数据革命,以及它如何改变政府、商业与我们的生活 by 涂子沛
  4. 奇点临近 by Ray Kurzweil
  5. 短暂飞行 by 张向东

2014年8月在读

  1. 国际平面设计基础教程 by 加文·安布罗斯 [等]

2014年9月在读

没有读书

2014年10月在读

没有读书

2014年11月在读

没有读书

2014年12月在读

  1. 失控 by 凯文·凯利
  2. 奇点临近 by Ray Kurzweil
  3. 西藏生死书 by 索甲仁波切
  4. 讲谈社•中国的历史(套装共10册) by (日) 宫本一夫 等
  5. 每天懂一点色彩心理学•实用篇(附色彩杂学练习册) by (日) 原田玲仁
  6. 配色设计原理 by 奥博斯科编辑部
  7. 宝瓶同谋:大数据时代的思想聚变 by [美] 玛丽琳. 弗格森
  8. 定价未来:撼动华尔街的量化金融史[美]乔治G.斯皮罗(George G. Szpiro)
  9. 僧侣与哲学家 by 珍·法蘭可斯雷蒙 / 馬錫·理察

Waiting List

  1. 自控力 by 凯利·麦格尼格尔
  2. 怪诞心理学2 by 理查德·怀斯曼
  3. Android 内核剖析 by 柯元旦
  4. 走出思维的误区
  5. 认知盈余
  6. 游戏改变世界
  7. Java并发编程实战 by Brian Goetz / Tim Peierls / Joshua Bloch / Joseph Bowbeer / David Holmes / Doug Le1.
  8. [走出思维的误区:批判性思维指南(修订第9版)]
  9. 对”伪心理学”说不(第8版)(How to think straight about psychology) by Keith E. Stanovich
  10. [找寻逝去的自我:大脑、心灵和往事的记忆]
  11. HIERARCHICAL TEMPORAL MEMORY including HTM Cortical Learning Algorithms
  12. On Intelligence by Jeff Hawkins.
  13. 竞争战略 by 迈克尔·波特
  14. 编程珠玑 by Jon Bentley
  15. 文明之光 (一、二)by 吴军

Programmer’s must read

  1. 程序员的自我修养 by 俞甲子 / 石凡 / 潘爱民
  2. Computer Systems:A Programmer’s Perspective by Randal E. Bryant / David R. O’Hallaron
  3. Structure and Interpretation of Computer Programs by Harold Abelson and Gerald Jay Sussman
  4. The Art of Unix Programming by Eric S. Raymond.
  5. The Performance of Open Source Applications part I
  6. The Performance of Open Source Applications part II
  7. The Performance of Open Source Applications by AOSA.

Deep Dive to Android Internals (resources)

  1. XDA Developers’ Android Hacker’s Toolkit: The Complete Guide to Rooting, ROMs and Theming by Jason Tyler (Author), Will Verduzco
  2. NewCirle’s Android Training
  3. Android Architecture
  4. Deep Dive into Android IPC/Binder Framework

散文

  1. 丰子恺
  2. 木心 《圆光》
  3. 弘一法师 (李叔同)
  4. 汪增祺
  5. 陈之藩《剑河倒影》
  6. 简媜,杨绛,汪曾祺,张炜,北岛,野夫,张晓风,周作人,史铁生,朱自清
  7. 陈丹青的《荒废集》《退步集》
  8. 胡兰成的《今生今世》
  9. 章诒和的《伶人往事》
  10. 林清玄《心的菩提》
  11. 迟子建散文
  12. 梁遇春《春醪集》
  13. 木心先生《哥伦比亚的倒影》
  14. 龙应台《目送》刘瑜《送你一颗子弹》
  15. 陈丹青《多余的素材》
  16. 野夫的《乡关何处》
  17. 阿城–汪曾祺–沈从文–周作人–张岱
  18. 周作人的散文合集,《时光阡陌,你一直未曾走远》
  19. 郁达夫《故都的秋》
  20. 沈从文的《湘行散记》
  21. 林清玄《煮雪》

英文散文

  1. 陆谷孙《20篇:英美现当代散文》
  2. 《The Art of Travel》
  3. 《Beg, Borrow, Steal——A Writer’s Life》
  4. 黄源深编:《英语散文选读》,上海外语教育出版社,2007.
  5. 南建翀编:《经典英语美文》,世界图书出版社,2008.
  6. 杨自伍编:《英国散文名篇欣赏》,上海外语教育出版社,2010.
  7. 推荐看一下网易公开课上苏州大学王腊宝教授的公开课视频:“英语经典美文:从培根到伍尔芙

Re-implement the works

前两天读到某某教授的一篇文章,作者提出一个“3遍阅读法(3-pass reading)”,最后一遍的要求叫"re-implement"。 这个词特别好。 我们读文章时, 或者因为使用某些知其然, 不知其所以然的类库, 知识时往往开始觉得很困惑, 经常用错。 源于我们对内在的原理不了解。 而“re-implement”要求我们从原作者的思维触发, 重铸作品, 达到知其所以然的目的。 当自己从头开始完全重铸意见作品时, 就能明白其中很多的条件和假设, 以及各种限制。 这些往往在我们使用是会忽略的因素。 大多数情况下, 一个工具,一套理论,一个开发类库, 都由其适用的条件和环境。 或许作者没有说清楚, 或者使用者没有仔细弄明白说明文档, 往往很容易用错。 这点对于程序员来说更甚。 往往拿到一个类库, 不会仔细阅读说明, 而是简单按照函数名称做一个猜测,忽略它的限制条件。 很多bug 就是由此引入的。 这些类库正常情况下使用良好, 然后在某些极端情况下会出问题。 带来各种bug。

由”re-implement” 一次我想到说其实类的设计充满这妥协和限制。 一个好的程序员应该站在类库作者的角度来考虑这些问题。简单说,即使不要求完全重写类库, 也应该在心里过一遍如果是自己来设计, 会如何做。 有条件的情况下, 多读读类库的源码。 一定会有很多收获。

Android 中的AsyncTask 就是这么一个很容易出错的类。 这个类android.os.AsyncTask 位于android.os 包下,虽然其实更应该是一个util类。这个类本是为了方便程序员写异步操作而设计, 然后它的API本省非常长。 而且在说明里告诉说这个类不适合长时间运行的任务,结果给出的例子就叫做DownloadingTask. 这是一个设计很糟糕的类。

对于它的设计, 如果没读过源码,可能会觉得有什么特别的, 因为位于android.os中。 然而它就是Handler和ThreadPool的简单封装而已。

  • <1.6 单线程
  • 1.6-3.0 多线程
  • =3.0 单线程

代码中使用了SerialExecutorThreadPoolExecutor, 有一个sDefaultExecutor静态成员变量来控制, 可以通过setDefaultExecutor函数来控制。

所有的回调函数都是通过Handler 和sendMessage 来实现。 非常简单,是不是?

另外, ThreadPool 中初始化时定义最大128 个, 同时5个thread 执行。

Looper

Looper 是线程见消息同步的机制。 一个线程发生了某件事情, 如何实时通知另外一个线程? 这是一个通用问题。 常见做法是有一个线程一直死循环, 检查一个变量, 另外的线程发生什么事情就写这个变量。 一直死循环在检查变量的线程就是Looper 线程。 Android 中UI线程就有一个Looper, 当把本身的事情都做完之后, 就会开始死循环, 一旦有消息, 在死循环里完成那个消息触发的事件, 继续死循环。

Android 线程相关

  • Handler, Message, Messenger, Process, AIDL, binder AsyncTask Thread Runnable ThreadPool

移动互联网必读

这里列出一些移动互联网从业者应该关注的站点, 大部分为新闻和评论站点。

  • kickstarter Kickstarter is the world’s largest funding platform for creative projects.

读书笔记: 黑客与画家

第九章 设计者的品味

Play Framework

Play around “Play”

  1. Install
  2. Create “Hello, world.”
    • play new todolist
  3. Anatomy of Play application / project structure
  4. Using the play console
    • launch: play
    • run
    • help play
    • compile
    • console: interactive
    • play debug
    • sbt features
    • play run
    • Ctl + D to exit
    • play clean-all
  5. IDE Support
  6. Samples
    • hello world
    • Computer database
    • CometClock
    • Bootstrap
    • WebSocket
  7. databases: CRUD, Ebean
  8. Enumerators, Enumeratees
  9. Akka actors
  10. The Apache Velocity Project
  11. Deploying to Heroku

Document

  1. HTTP programming
  2. Async HTTP
  3. the template engine
  4. HTTP form submission and validation
  5. with Json
  6. With XML
  7. File Upload
  8. SQL DB
  9. Cache
  10. Calling WebServices
  11. Gloabal
  12. Testing

hand by hand introduction: Your first Play application

IO in Java

流,英文Stream. Something that is flowing. 信息流, 照片流,文字流, 数据流, 字节流

Android 自动化

Android 自动化开发流程 在android 社区中有一些列的开源工程帮助andorid 自动化。

Maven

Android maven plugin

Archetype: MAVEN-ANDROID-ARCHETYPES

Robolectric

Robotium

Android Interview

Interview Questions Collection

  1. careercup,There is an accompany book: “Cracking the Coding Interview: 150 Programming Questions and Solutions” by Gayle Laakmann McDowell
  2. Cracking the coding interview–问题与解答
  3. Quora:Java Interview Questions. A collection of java interview questions on quora.

    Java Resources

  4. Qura: What questions are Java Software Engineers seeing the most of on technical interviews?
  5. Qura: [What are good interview questions to ask JAVA developers?] (http://www.quora.com/Java-programming-language/What-are-good-interview-questions-to-ask-JAVA-developers)
  6. Java puzzlers by Joshua Bloch. RECOMMENDED FOR SENIOR JAVA DEVELOPERS.
  7. Effective Java by Joshua Bloch. RECOMMENDED FOR ADVANCED JUNIOR JAVA DEVELOPERS.
  8. Java Concurrency in Practice by Tim Peierls, Brian Goetz, Joshua Bloch, Joseph Bowbeer, Doug Lea, David Holmes. RECOMMENDED FOR SERVER DEVELOPERS.
  9. Inside the Java Virtual Machine by Bill Venners. Online book available.
  10. Books Related to the JVM, a list of jvm books.
  11. Thinking in Java by Bruce Eckel. JAVA 101
  12. The Java Job Interview Companion
  13. Quora: What are good books to test one’s programming skills?
  14. Wiki:List of data structures
  15. Wiki:List of algorithms
  16. Programming Interviews Exposed: Secrets to Landing Your Next Job
  17. Coding Interviews By Harre He . A Press
  18. Cracking the coding interview by Gayle as mentioned in other posts
  19. Algorithms For Interviews (9781453792995): Adnan Aziz, Amit Prakash: Books
  20. Programming Challenges by Steven Skiena
  21. Elements of Programming Interviews.
  22. Cracking the Coding Interview: 150 Programming Questions and Solutions
  23. Cracking The IT Interview 2nd Edition -
  24. Data Structures and Algorithms Made Easy 2nd Edition - I strongly recommend this book
  25. Cracking the C, C++ and Java Interview 1st Edition -
  26. Test your C Skills - Yashwant Kanetkar
  27. Test your C ++ Skills - Yashwant Kanetkar
  28. Exploring C - Yashwant Kanetkar
  29. Test Your Unix skills - Yashwant Kanetkar
  30. Top 15 Java threading interview questions asked in Investment banks
  31. 有道难题

I mostly agree with John Kurlak’s answer. Personally I feel Cracking the Coding Interview: 150 Programming Questions and Solutions is a good choice if you only know Java. Programming Interviews Exposed is a little elementary and the materials actually need to be updated. Elements of Programming Interviews: 300 Questions and Solutions seems for people who actually want to pass all interviews including the hardest one after concluding all reviews on Amazon, and it is mainly written in C++ which makes this as a great choice for people who can understand on C++.

Sample Questions

  1. Card Shuffle

The following shuffle method purports to shuffle its input array fairly. In other words, it purports to generate all permutations with equal likelihood, assuming that the underlying pseudorandom number generator is fair. Does it make good on its promise? If not, how do you fix it?

import java.util.Random;

public class Shuffle {
    private static Random rnd = new Random();

    public static void shuffle(Object[] a) {
        for (int i = 0; i < a.length; i++)
            swap(a, i, rnd.nextInt(a.length));
    }

    private static void swap(Object[] a, int i, int j) {
        Object tmp = a[i];
        a[i] = a[j];
        a[j] = tmp;
    }
}
  1. Java™ Puzzlers: Traps, Pitfalls, and Corner Cases Puzzle 45: Exhausting Workout
This puzzle tests your knowledge of recursion. What does this program do?

public class Workout {
    public static void main(String[] args) {
        workHard();
        System.out.println("It's nap time.");
    }

    private static void workHard() {
        try {
            workHard();
        } finally {
            workHard();
        }
    }
}
  1. Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

  2. Clock puzzles - find the angle between hr,min,sec handles, etc

  3. From Twitter http://qandwhat.apps.runkite.com/i-failed-a-twitter-interview/

"Consider the following picture:"

alt text

"In this picture we have walls of different heights. This picture is represented by an array of integers, where the value at each index is the height of the wall. The picture above is represented with an array as [2,5,1,2,3,4,7,7,6]."

"Now imagine it rains. How much water is going to be accumulated in puddles between walls?"

alt text

"We count volume in square blocks of 1X1. So in the picture above, everything to the left of index 1 spills out. Water to the right of index 7 also spills out. We are left with a puddle between 1 and 6 and the volume is 10."

Online Tests

  1. Test For Geeks, including java and android tests. Extremely challenging.
  2. Fizz Buzz Test
  3. Brainbench test

Fields

  1. Java Collection
  2. Multi Threading
  3. GC
  4. Design Patterns
  5. Algorithms
  6. Data Structure

Discussion

TODOS

  1. @知乎 如何面试android 工程师? 从哪些方面考察?
  2. 面试官如何写面试反馈?
  3. 整理一个PPT 出来, 分topic 出面试题, 做分享
  4. 整理知乎上的相关java / android 面试题

Android 面试题

  1. Android 中线程与线程,进程与进程之间如何通信?
  2. 解释 Binder 原理, 进程间通信
  3. activity,intent 和 service 是什么关系?
  4. handler 机制的原理是什么?
  5. 横竖屏切换时候 activity 的生命周期?
  6. 解释View 绘制过程
  7. 瀑布流如何回收资源
  8. 常用的Android 开源库有哪些, 如何选择?挑一个谈谈优缺点,如何改进?
  9. 什么情况下会出现ANR, 怎么找到原因, 如何解决?
  10. 起了很多线程, UI线程阻塞
  11. 后台线程, Service
  12. 异步线程
  13. listview 优化

READING: Big Data

两本和Big Data 相关的书, 一本是媒体书, 一本是技术书

Big Data 里讲到大数据的核心是预测。  Jeff Hawkins做Neural Networks 也说AI的核心是预测(On Intelligence)
可以参考之前的blog: [由《暗时间》所引起的读书计划](http://xianminx.github.io/2013/02/01/dark-time-reading-list/)
大数据时代, 逐步放弃对精确性的追求。 
可以在google scholar 上查一查big data 对hedge fund, social network, twitter, finance, privacy, 大选, 民调 等领域的影响。大数据时代的隐私所面对的挑战不是单一数据源上的问题, 而是多数据源合并,通过reverse-anonymization 技术带来的挑战。 

大数据和物联网, 物联网时代每一个物体都在产生数据,核心将从功能转化成数据。 比如Fuel Band  就是一个数据收集器。 

大数据不是指数据绝对意义上的多, 而是采用全量不是采样的分析方法。 

预测只需要知道关联关系, 不需要知道因果关系。 
相关关系只是缺少一个人工的解释。 存在即合理。 

Tera data 数据分析公司

分析中国股市和社交网络数据的关系 
检查利用手机Sensor 累计数据的应用, 比如步速

量变引起质变

在物理测量中, 极大极小意义上的测量和普通意义上的测量完全不同。 极小对应量子力学。 不确定原理。 
极大对应相对论,超光速, 黑洞等。 

数据处理也一样, 同样的计数, 在小数据的情况下, 普通的计数器。 
在大数据的情况下, 同步, 一致性, 等诸多问题。 


所以大数据时代产出的理论如同广义相对论, 囊括了各种不同情况下的描述。 而万有引力只是一种近似。 


找一本大数据的书, 讲述亚马逊的推荐系统。 

大数据时代的隐私保护。 privacy preserving in big data era. 

家具制造, 物流, 企业配料, 赌球赔率

Use probability to approach Pi

Once I found this interview question: calculate Pi. Where, all of us have learned calculating Pi in our primary math class by using regular polygon to approach a circle.

Today I am gonna show you a probability way.

Method

Check the following graph. I hope it is self-explainatory.

Code

https://gist.github.com/xianminx/b59eef29b8137827f1f0#file-pi-java

class Pi{ 
        public static void main(String[] args){
                if(args.length < 1)
                                System.exit(0);
 
                long iter = Long.parseLong(args[0]);
                double pi = calculate(iter);
                System.out.println(String.format("after %d iterations, pi is %f", iter, pi ));  
        }   
 
        public static double calculate(long simNum){
                long i =0L;
                long count =0L;
                while(i++<simNum){
                        double x = Math.random();
                        double y = Math.random();
                        double result = x*x+y*y;
                        if(result < 1.0D){
                                count++;
                        }   
                }   
                return 4.0D* count/simNum;
        }   
 
}
 

Experiment

lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 1000
after 1000 iterations, pi is 3.032000
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 10000
after 10000 iterations, pi is 3.137200
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 100000
after 100000 iterations, pi is 3.140240
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 1000000
after 1000000 iterations, pi is 3.144108
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 10000000
after 10000000 iterations, pi is 3.141199
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 100000000
after 100000000 iterations, pi is 3.141486
lucas@lucas-ubuntu:~/dev/workspace/pi$ 

Conclusion

When iteration is set to 100,000,000, it takes about half a minute on my i7 Core Mac Pro with precision only to 3 digits.

  • This algorithm is unacceptable.
  • The precision cannot be guranteed as it is using a probabilistic way.

READING: Pragmatic Programmer

Pragmatic Programming: from journeyman to master.

Android Theme and Style

http://developer.android.com/guide/topics/ui/themes.html#PlatformStyles

http://developer.android.com/design/index.html

Android ListView Advanced

Introduction

ListView is one of the most fundamental and important Android UI components. Due to the long rectangle shape of mobile device, ListView is widely used as the first level navigation to the more detailed content. Popular using scenarios include, but not limited to, news reader, reading list, e-commere product list, things to do list, ranking list, etc. If your applicaiton intends to providing updatable contents to users, you cannot avoid using this super powerful widgets.

ListView can be very simple and also extremely complex. There are many traps and tricks for novice developers. In this article, the author will summerize some typical traps developers may occur during their use of ListView and try to provide solutions or suggestions. In more general, the author will summerize typical advanced usages of ListView, such as pull to refresh ListView, HTTP cache, data binding, loader, etc.

Simple Example

Android ApiDemos Sample provide a bundle of examples for using ListView (under android-sdk-macosx/samples/android-17/ApiDemos/src/com/example/android/apis/view).

The following is a simple example.

  • /Users/lucas/dev/eclipse-juno-workspace/AdvancedListView/src/com/example/android/apis/view/advancedlistview/SimpleListViewActivity.java
package com.example.android.apis.view.advancedlistview;

import android.app.Activity;
import android.os.Bundle;
import android.widget.ArrayAdapter;
import android.widget.ListAdapter;
import android.widget.ListView;

public class SimpleListViewActivity extends Activity {

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_simple_listview);
		ListView lv = (ListView) this.findViewById(R.id.list);

		ListAdapter adapter = new ArrayAdapter<String>(this,
				android.R.layout.simple_list_item_1, COUNTRIES);

		lv.setAdapter(adapter);
	}

	private static String[] COUNTRIES = new String[] { "CHINA", "US", "JAPAN" };
}
  • /Users/lucas/dev/eclipse-juno-workspace/AdvancedListView/res/layout/activity_simple_listview.xml
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    tools:context=".MainActivity" >

    <ListView
        android:id="@+id/list"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content" />

</RelativeLayout>

This is a very simple example of ListView.

Adapter Architecture

From the above simple example, we find 3 components for the ListView: the standard MVC pattern. Model, View, and Controller.

View

ListView is embeded in Activity in layout XML.

Data

To provide data for each item in the ListView, we need a Adapter. Here we use ListAdapter, constructed from a static String array.

Controller

If you want users to interact with your app, you can add Listeners for user actions, for example, click event, touch event. This can be done by setting setOnItemClickListener.

Framework API

Official Tutorials

Non-Official Tutorials

API Explored

Performance

Reusing

ConvertView

ViewHolader

Header & Footer

Load Data Async

Data Binding

HTTP + JSON

Image

GroupList/ SectionList

Resources on Github

问题

  1. Occasionally, you get the IllegalStateException error, stating

    “The content of the adapter has changed but “ + “ListView did not receive a notification. Make sure the content of “ + “your adapter is not modified from a background thread, but only from “ + “the UI thread. Make sure your adapter calls notifyDataSetChanged() “ + “when its content changes. “

By checking the source code, we find that when the ListView’s ‘mItemCount’ is different from the bound adapter’s count, framework will throw the exception.

Here ‘mItemCount’ is written when the adapter is first time set or when ‘onMeasure’ is called (refresh).

<img src=”imgs/listview_itemcount_write.png”.


            // Handle the empty set by removing all views that are visible
            // and calling it a day
            if (mItemCount == 0) {
                resetList();
                invokeOnItemScrollListener();
                return;
            } else if (mItemCount != mAdapter.getCount()) {
                throw new IllegalStateException("The content of the adapter has changed but "
                        + "ListView did not receive a notification. Make sure the content of "
                        + "your adapter is not modified from a background thread, but only from "
                        + "the UI thread. Make sure your adapter calls notifyDataSetChanged() "
                        + "when its content changes. [in ListView(" + getId() + ", " + getClass()
                        + ") with Adapter(" + mAdapter.getClass() + ")]");
            }


  1. Performance Tips for Android’s ListView
  2. [Google I/O 2010 - The world of ListView]<iframe width="560" height="315" src="//www.youtube.com/embed/wDBM6wVEO70" frameborder="0" allowfullscreen></iframe>

MIND MAP

Android Fragment

What is fragment stack?

There can be multiple fragments in a single Activity/ Screen.

What is transacation in fragment operation and why should there be the concept of transacion?

How to go back to the previous Fragment?

Compare fragment stack and activity stack.

  • change waiting progress bar if has already loaded .

W/Trace ( 7306): Unexpected value from nativeGetEnabledTags: 0

Cyanogenmod

Wiki entry

Read all articles on CyanogenMod:qq

vim tutor

Vim provides a great tutorial tool called vim tutor. You can open it by using vim tutor or vimtutor.

By following the tutorial, you wil get a basic idea how to use vim more efficiently in 20-30 minutes.

Baiscally, vim command has the grammar:

operator times motion

Here, operator means the operation you want to do: like delete, and times is a integer that represents how many times your want to repeat the operation. motion is the sementics you want the operation does, like w, which represent a word, and e, end of a word.

so d5w mean delete the following 5 words. If operator is missing, it means moving by default. So 5w means moving the cursor by 5 words. There are 2 special motions: $, representing the end of the line and 0, representing the beginning of the line. Typing $ will take your cursor to the end of the line and d$ will delete from the cursor to the end of the line.

Due to the high frequency of deleting the whole line when editing in vim, deleting the whole line is simply dd. To delete multiple lines, use 5dd. You can use d5d, but that takes you longer time to type.

operater:

  • delete: d dw de
  • change: c ce
  • replace: r rx

Go to to line: 78 G, uppercase G

  • Search: /abc n for next N for previous
  • find matching parenthesis: %

visualize the block: v v, then move the cursor to hightlight the selected text.

Next, execute: operator number motion.

vimrc config: with VI improved features.

vim ~/.vimrc :r $VIMRUNTIME/vimrc_example.vim

Google Oauth 2.0

Flow

Authentication

OpenID

OAuth2.0

Mobile App Authentication

Authorization

API

SDK

Maven with Eclipse

#

  1. 安装Maven
  2. 安装Eclipse
  3. 安装m2e
mvn archetype:generate -DgroupId={project-packaging} -DartifactId={project-name} -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false

To convert a maven project to a eclipse project

mvn eclipse:eclipse -Dwtpversion=2.0

Domestic Ads Players

# 调查国内的广告商 写一篇survey

Android Booting Process

Linux Booting

  • boot loader
  • boot
  • system
  • init.rc

Android Booting

  • Linux
  • init.rc
  • Dalvik
  • System server

References

  1. Android boot process
  2. General Android boot process
    • http://www.androidenea.com/2009/06/android-boot-process-from-power-on.html
    • http://bloggervarun.wordpress.com/2010/01/25/embedded-linux-boot-process/
  3. Boot of Linux on Embedded Devices
    • Linux U-boot: http://www.denx.de/wiki/view/DULG/UBoot
    • OMAP bring-up: http://elinux.org/Android_on_OMAP
    • eLinux boot time: http://elinux.org/Boot_Time#Measuring_Boot-up_Time
  4. The Android boot image: boot.img
    • Unpack, re-pack boot image: http://android-dls.com/wiki/index.php?title=HOWTO:_Unpack%2C_Edit%2C_and_Re-Pack_Boot_Images#Background

Configure ant for Android projects

There are 3 config files generated by android update project or android updatel ib-project.

local.properties

lucas-mac:apf-framework lucas$ cat local.properties 
# This file is automatically generated by Android Tools.
# Do not modify this file -- YOUR CHANGES WILL BE ERASED!
#
# This file must *NOT* be checked into Version Control Systems,
# as it contains information specific to your local configuration.

# location of the SDK. This is only used by Ant
# For customization when using a Version Control System, please read the
# header note.
sdk.dir=/Users/lucas/dev/android/android-sdk-macosx

ant.properties

	The ant.properties file can be created by you. It is only edited by the
     'android' tool to add properties to it.
     This is the place to change some Ant specific build properties.
     Here are some properties you may want to change/update:

     source.dir
         The name of the source directory. Default is 'src'.
     out.dir
         The name of the output directory. Default is 'bin'.

     For other overridable properties, look at the beginning of the rules
     files in the SDK, at tools/ant/build.xml

     Properties related to the SDK location or the project target should
     be updated using the 'android' tool with the 'update' action.

     This file is an integral part of the build system for your
     application and should be checked into Version Control Systems.

project.properties

lucas-mac:apf-framework lucas$ cat project.properties 
# This file is automatically generated by Android Tools.
# Do not modify this file -- YOUR CHANGES WILL BE ERASED!
#
# This file must be checked in Version Control Systems.
#
# To customize properties used by the Ant build system edit
# "ant.properties", and override values to adapt the script to your
# project structure.
#
# To enable ProGuard to shrink and obfuscate your code, uncomment this (available properties: sdk.dir, user.home):
#proguard.config=${sdk.dir}/tools/proguard/proguard-android.txt:proguard-project.txt

# Project target.
target=android-14
android.library=true

Default build.xml generated by android tool

lucas-ma:apf-framework lucas$ cat build.xml 
<?xml version="1.0" encoding="UTF-8"?>
<project name="apf-framework" default="help">

    <!-- The local.properties file is created and updated by the 'android' tool.
         It contains the path to the SDK. It should *NOT* be checked into
         Version Control Systems. -->
    <property file="local.properties" />

    <!-- The ant.properties file can be created by you. It is only edited by the
         'android' tool to add properties to it.
         This is the place to change some Ant specific build properties.
         Here are some properties you may want to change/update:

         source.dir
             The name of the source directory. Default is 'src'.
         out.dir
             The name of the output directory. Default is 'bin'.

         For other overridable properties, look at the beginning of the rules
         files in the SDK, at tools/ant/build.xml

         Properties related to the SDK location or the project target should
         be updated using the 'android' tool with the 'update' action.

         This file is an integral part of the build system for your
         application and should be checked into Version Control Systems.

         -->
    <property file="ant.properties" />

    <!-- if sdk.dir was not set from one of the property file, then
         get it from the ANDROID_HOME env var.
         This must be done before we load project.properties since
         the proguard config can use sdk.dir -->
    <property environment="env" />
    <condition property="sdk.dir" value="${env.ANDROID_HOME}">
        <isset property="env.ANDROID_HOME" />
    </condition>

    <!-- The project.properties file is created and updated by the 'android'
         tool, as well as ADT.

         This contains project specific properties such as project target, and library
         dependencies. Lower level build properties are stored in ant.properties
         (or in .classpath for Eclipse projects).

         This file is an integral part of the build system for your
         application and should be checked into Version Control Systems. -->
    <loadproperties srcFile="project.properties" />

    <!-- quick check on sdk.dir -->
    <fail
            message="sdk.dir is missing. Make sure to generate local.properties using 'android update project' or to inject it through the ANDROID_HOME environment variable."
            unless="sdk.dir"
    />

    <!--
        Import per project custom build rules if present at the root of the project.
        This is the place to put custom intermediary targets such as:
            -pre-build
            -pre-compile
            -post-compile (This is typically used for code obfuscation.
                           Compiled code location: ${out.classes.absolute.dir}
                           If this is not done in place, override ${out.dex.input.absolute.dir})
            -post-package
            -post-build
            -pre-clean
    -->
    <import file="custom_rules.xml" optional="true" />

    <!-- Import the actual build file.

         To customize existing targets, there are two options:
         - Customize only one target:
             - copy/paste the target into this file, *before* the
               <import> task.
             - customize it to your needs.
         - Customize the whole content of build.xml
             - copy/paste the content of the rules files (minus the top node)
               into this file, replacing the <import> task.
             - customize to your needs.

         ***********************
         ****** IMPORTANT ******
         ***********************
         In all cases you must update the value of version-tag below to read 'custom' instead of an integer,
         in order to avoid having your file be overridden by tools such as "android update project"
    -->
    <!-- version-tag: 1 -->
    <import file="${sdk.dir}/tools/ant/build.xml" />

</project>

Mobile Advertising

# Read a series of articles on Wikipedia. Get a introductroy sensation about mobile advertising and its fundamental concepts. Mobile advertising

Build Android Source

On Mac OSX

Prerequisite:

To build the latest source in a Mac OS environment, you will need an Intel/x86 machine running MacOS 10.6 (Snow Leopard) or MacOS 10.7 (Lion), along with Xcode 4.2 (Apple’s Developer Tools). Although Lion does not come with a JDK, it should install automatically when you attempt to build the source. ‘

Steps

  1. Creating a case-sensitive disk image
    # hdiutil create -type SPARSE -fs 'Case-sensitive Journaled HFS+' -size 40g ~/android.dmg
    
  2. add the following to your ~/.bash_profile to mount the image when you execute “mountAndroid”:
    # mount the android file image
    function mountAndroid { hdiutil attach ~/android.dmg -mountpoint /Volumes/android; } 
    mountAndroid
    
  3. Setting a file descriptor limit On MacOS the default limit on the number of simultaneous file descriptors open is too low and a highly parallel build process may exceed this limit. To increase the cap, add the following lines to your ~/.bash_profile:
    # set the number of open files to be 1024
    ulimit -S -n 1024
    
  4. Download the source