Algorithm Review

Algorithm Review





Math for algorithms

排序算法的商减 (entropy decrease)

Math for algorithms

Math basics for algorithm

A.1 求和公式及其性质

  • 等差

  • 线性性质

  • 几何级数
  • 调和级数
  • 级数积分与微分
  • 列项级数 telescopes

  • 乘积

A.2 确定求和时间的界

  • 数学归纳法
  • 确定级数中各项的界
  • 通过积分求和的近似

    • if $f(x)$ 单调递增:

    • if $f(x)$ 单调递减:


  • Illustrate function call stack
  • Describe how to convert recursion to iteration
    • one function call
    • 2 or more function calls
    • tail recursion
PERM(A, k)
  if(k == n)
  for i = 1 to n 
    SWAP(A, i, k)
    PERM(A, k+1)
    SWAP(A, i, k)

PERM(A, 1)

  • DFS and stack
  • BFS and queue


  • History


核心是使用 HTML window.history API

  • vm
    • $options
      • _parentVnode
      • $router
      • $route


  • window
    • location
      • ancestorOrigins: DOMStringList {length: 0}
      • assign: ƒ ()
      • hash: “#html”
      • host: “”
      • hostname: “”
      • href: “”
      • origin: “”
      • pathname: “/guide/”
      • port: “”
      • protocol: “https:”
      • reload: ƒ reload()
      • replace: ƒ ()
      • search: “”
      • toString: ƒ toString()
    • history
    • performance
      • Performance Timeline API,
      • the High Resolution Time API,
      • the Navigation Timing API,
      • the User Timing API,
      • the Resource Timing API.

因为安全的问题, 不能获取 history stack 列表, 只能 push pop 操作。





  • app
  • library
  • wc

  • @vue/cli
  • @vue/cli-service
  • @vue/cli-plugin-
  • vue-cli-plugin-
  • Development
    • Browser Compatibility
      • browserslist
      • Polyfills
    • HTML 和静态资源
      • index file public/index.html
      • interpolation with loadash
        • e.g <link rel="icon" href="<%= BASE_URL %>favicon.ico">
        • <%= VALUE %>
        • <%- VALUE %>
        • <% expression %>
      • preload <link rel="preload">
      • prefetch <link rel="prefetch">
      • Disable Index Generation
      • Building a Multi-Page App
      • Static Assets Handling
      • Relative Path Imports
      • URL Transform Rules
        • absolute: /images/foo.png
        • relative: ./
        • module: <img src="~some-npm-package/foo.png">
        • module: <img src="@some-npm-package/foo.png">, interpret to <projectRoot>/src
      • The public Folder
          <link rel="icon" href="<%= BASE_URL %>favicon.ico">
          data () {
              return {
                  baseUrl: process.env.BASE_URL
          <img :src="`${baseUrl}my-image.png`">
    • CSS 相关
    • webpack 相关
    • Enviroment Variables and Modes
    • Build Targets
    • Deployments

      Config Reference

  • Global Config
    • ~/.vuerc
    • vue config
  • Target Browsers
  • vue.config.js
    • loaded by @vue/cli-service
    • or by vue section in package.json
        module.exports = {
            baseUrl: '/my-app/' or './',
            outputDir: 'dist',
            assetsDir: '',
            indexPath: 'index.html',
            filenameHashing: false,
            pages: undefined,
            lintOnSave: 'true',
            lintOnSave: process.env.NODE_ENV !== 'production',
            runtimeCompiler: 'false',
            transpileDependencies:[], // 转译 node_modules/ 中的依赖
            productionSourceMap: true,
            crossorigin: undefined,
            integrity: false,
            configureWebpack: {
                plugins: [
                new MyAwesomeWebpackPlugin()
            chainWebpack: '',
            css.modules: false,
            css.extract: false,
            css.sourceMap: false,
            css: {
                loaderOptions: {
                    css: {
                        // 这里的选项会传递给 css-loader
                    postcss: {
                        // 这里的选项会传递给 postcss-loader
            devServer: {
                proxy: 'http://localhost:4000'
            parallel: true,
            pwa: {},
            pluginOptions: {
                foo: {
                // 插件可以作为 `` 访问这些选项。
            chainWebpack: config => {
  • Babel
    • @vue/babel-preset-app
    • babel.config.js
    • Polyfills
  • ESLint @vue/cli-plugin-eslint

    • .eslintrc
    • eslintConfig in package.json
  • TypeScript @vue/cli-plugin-typescript
    • tsconfig.json
  • Unit Testing
    • Jest
    • Mocha
  • E2E Testing
    • Cypress
    • Nightwatch

Plugin Dev Guide


Q & A

  1. what is vue-loader, babel-loader, lint-loader etc. ?
  2. Javascript moduel, moduleA.js? moduleB/index.js?

    See Learn the basics of the JavaScript module system and build your own library

    Telling stories is as basic to human beings as eating. More so, in fact, for while food makes us live, stories are what make our lives worth living - Richard Kearney

    • tools
      • CommonJS
      • AMD
      • UMD
      • ES6 (Harmony)
    • Best Practices
      • Tree shaking: import , export
      • Webpack vs Rollup vs Babel?
      • webpack:
        • code splitting,
        • async loading of bundles,
        • tree shaking,
      • UI Libraries have a dist folder that has the bundled and minified version for ES and UMD/CJS module systems as a target. There is a lib folder that has the transpiled version of the library.
      • Core Packages have just one folder which has the bundled and minified version for CJS or UMD module system as a target.

Cascading Style Sheets


  • Syntax
  • selectors
    • select by id #id
    • select by element type p {text-align: center; color: red;}
    • select by class: .center
    • combined:
    • group:
        h1 {
            text-align: center;
            color: red;
        h2 {
            text-align: center;
            color: red;
        p {
            text-align: center;
            color: red;
        h1, h2, p {
            text-align: center;
            color: red;
    • comments:
        p {
            color: red;
            /* This is a single-line comment */
            text-align: center;
        /* This is
        a multi-line
        comment */
  • refer to style sheet:
    • External style sheet
        <link rel="stylesheet" type="text/css" href="mystyle.css">
    • Internal style sheet
        body {
            background-color: linen;
        h1 {
            color: maroon;
            margin-left: 40px;
    • Inline style
        <h1 style="color:blue;margin-left:30px;">This is a heading</h1>
    • Cascading Order: all the styles will “cascade” into a new “virtual” style sheet by the following rules:
      • Inline style (inside an HTML element)
      • External and internal style sheets (in the head section)
      • Browser default
  • Colors
    • Background Color: background-color:Tomato;
    • text color: color:MediumSeaGreen;
    • border: border:2px solid Violet;
    • Color values:
      • rgba(255, 99, 71, 0.5), or hex: #ee82ee
      • hsla(9, 100%, 64%, 0.5)
        • Hue is a degree on the color wheel from 0 to 360. 0 is red, 120 is green, and 240 is blue.
        • Saturation is a percentage value, 0% means a shade of gray, and 100% is the full color.
        • Lightness is also a percentage, 0% is black, 50% is neither light or dark, 100% is white
      • Shades of gray are often defined using equal values for all the 3 light sources: rgb(180, 180, 180) hsl(0, 0%, 0%)
  • backgrounds

      body {
          background-color: lightblue;
          background-image: url("paper.gif");
          background-repeat: repeat-x|no-repeat;
          background-attachment: fixed;
          background-position: right top;
          background-size: 300px 100px;
          background-size: auto|length|cover|contain|initial|inherit;


      background: #ffffff url("img_tree.png") no-repeat right top;
      background: bg-color bg-image position/bg-size bg-repeat bg-origin bg-clip bg-attachment initial|inherit;

    When using the shorthand property the order of the property values is:

    • background-color
    • background-image
    • background-repeat
    • background-attachment
    • background-position
  • Borders {
          border-style: solid;
          border-top-style: dotted;
          border-right-style: solid;
          border-bottom-style: dotted;
          border-left-style: solid;
          border-width: 5px;
          border-color: red;
          border: 5px solid red;
          border-radius: 5px;
          border-collapse: separate|collapse|initial|inherit;
  • Margins
  • Paddings
  • Height/Width
  • Box Model
  • Outline
  • Text
      h1 {
          color: blue;
          text-align: left|right|center|justify|initial|inherit;
          text-decoration: text-decoration-line text-decoration-color text-decoration-style|initial|inherit;
          text-decoration-line: none|underline|overline|line-through|initial|inherit;
          text-transform: uppercase|lowercase|capitalize;
          text-indent: 50px;
          letter-spacing: 3px;
          line-height: 0.8;
          direction: rtl;
          word-spacing: 10px;
          text-shadow: 3px 2px red;
  • Fonts
  • Icons
  • Links
  • Lists
  • Tables
  • Display
    • Block
    • inline
    • display: none|inline|block;
  • max-width
  • postion
      position: static|relative|fixed|absolute|sticky

Deep Learning

Deep Learning

Cost for sample $x$

total Cost:

the objective function:


to z $$ \frac {\partial C_x} {\partial z_k^l} = \sum_j { \frac {\partial C_x} {\partial z_j^{l+1} } \cdot \frac {\partial z_j^{l+1}} {\partial z_k^{l}} }\

\delta^l_k = \sum_j \delta^{l+1}_j \cdot {\frac {\partial z_j^{l+1}} {\partial z_k^{l} }}

\delta^L_j \equiv \frac {\partial C} {\partial z_j^L}
=\frac {\partial C} {\partial a_j^L} \cdot \sigma^{\prime}(z_j^L)
= ( a_j^L - y_j) \cdot \sigma(z_j^L) \cdot (1 - \sigma(z_j^L)) $$

matrix based: back propagation:


\delta^l_k = \sum_j {\delta^{l+1}_k}

\delta^L = \nabla_aC \odot {\sigma^{\prime}(z^L)} \

\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot {\sigma^{\prime}(z^l)} \


with $w_{jk}$ $$ \nabla_{b^l}C = \delta^l
\nabla_{w^l}C = \delta^l \sigma^{l-1} \



ECMAScript 6 Primer

Date: 20180904

第1章 ECMAScript 6 简介

第2章 let 和 const 命令

2.1 let

  • 类似 var,但是又 locality
  • 代码块内有效
          let a = 10; // 超出此代码块无效, 局部变量
          var b = 5;  // 代码块外有效, 作用域: 函数 + 全局
  • 变量提升
  • 暂时性死去 temporal dead zone (TDZ)
  • 不允许重复声明
  • block level domain
  • do
      let x = do {
          let t = f();
          t * t + 1;

    2.2 const


    const PI = 3.1415
    const a = []
  • 声明变量的6中方法
    • var
    • function
    • let
    • const
    • import
    • class


  • 浏览器: window
  • Node / web workder: global
  • this

第 3 章 变量的解构赋值 Destructuring assignment


3.1 数组的 Destructuring assignment

let [firstName, surname] = arr;
function fibs() {
    let a = 0;
    let b = 1;
    whiile (true) {
        yield a; 
        [a, b] = [b, a+b];
let [first, second, third, fourth, fifth, sixth] = fibs();

3.2 对象的解构赋值

let jsonData ={
    id: 42,
    status: 'OK',
    data: [867, 5309]
let {id, status, data: number } = jsonData;


var map = new Map();
map.set('first', 'hello');
map.set('second', 'world');
for(let [key, value] of map){
    console.log(key + ' is ' value);


  • 第 4 章: 字符串的扩展

Q & A

  • javascript 中 ; 使用
    • Stackoverflow: Function expression ending with ; vs. not

      At it’s root, what you have there is an assignment statement, and according to the Google Javascript Style Guide, all statements should end in a ;. Especially assignment statements. JSLint requires ; or else it won’t pass.

  • javascript prototype
    • 动态向 javascript object 添加属性, 每个函数都有一个prototype属性,这个属性是指向一个对象的引用,这个对象称为原型对象,原型对象包含函数实例共享的方法和属性,也就是说将函数用作构造函数调用(使用new操作符调用)的时候,新创建的对象会从原型对象上继承属性和方法。
      function employee(name,job,born)
      var bill=new employee("Bill Gates","Engineer",1985);

How Music Works

How Music Works



Frequency Hertz Amplitude Wavelength Speed (1.4) Musical Instrument Tone

There is a huge variety of musical instruments and sounds, as you would already know from your experience with music. Even two instruments playing the same note can sound very different.

This is because a musical instrument produces a sound wave which is a combination of different but related frequencies (known as harmonics) which all mix together to create the distinctive tone or voice of the instrument.

The lowest frequency is usually dominant, and you perceive this one as the pitch. The combination of the other harmonics provides the distinctive shape of the waveform, and thereby the distinctive tone of the instrument.

String instrument frequency determined by the length of the string. Guitar MusicalNoteFrequencies

Mathematics of Music

MIT Music21 project

Music21 is a set of tools for helping scholars and other active listeners answer questions about music quickly and simply. If you’ve ever asked yourself a question like, “I wonder how often Bach does that” or “I wish I knew which band was the first to use these chords in this order,” or “I’ll bet we’d know more about Renaissance counterpoint (or Indian ragas or post-tonal pitch structures or the form of minuets) if I could write a program to automatically write more of them,” then music21 can help you with your work.

Music Theory

Piano key frequencies

  • scientific notation
      Eb4, C#,,, C4, d#'', Ab2
  • Helmholtz


    tone.js music synthesis,

    Teoria.js for theory

  • Note
  • Chord
  • Scale
  • Interval

Web Audio API

Event Driven Design


Observer Pattern

Centralized Event Management

Decouple event from event handling


Queue Single Thread, queue, watch

Operating System, Kernel mode, user mode

select / epoll model

GUI Design

Updating UI in dedicated thread. Single Thread UI / main thread






vm._events[event] stores an array of listener functions: (vm._events[event] || (vm._events[event] = [])).push(fn)

  • vm.$emit
  • vm.$on
  • vm.$once( event, callback )
  • vm.$off( [event, callback] )
// node_modules/vue/src/core/instance/events.js
export function eventsMixin (Vue: Class<Component>) {
  const hookRE = /^hook:/
  Vue.prototype.$on = function (event: string | Array<string>, fn: Function): Component {
    const vm: Component = this
    if (Array.isArray(event)) {
      for (let i = 0, l = event.length; i < l; i++) {
        this.$on(event[i], fn)
    } else {
      (vm._events[event] || (vm._events[event] = [])).push(fn)
      // optimize hook:event cost by using a boolean flag marked at registration
      // instead of a hash lookup
      if (hookRE.test(event)) {
        vm._hasHookEvent = true
    return vm

Javascript Event

Java Event

In Java, we implement event using interface, See

Event Hierarchy

Centralized management

sub event

distributed management



Programming Models and Phylosophy

A Core Pattern for Events Adila A. Krisnadhi1;2 and Pascal Hitzler1 1 Wright State University, OH, USA 2 Universitas Indonesia, Depok, Indonesia



AMAP高德 地图

适用平台: Web Android iOS Web Service


  1. 地图
  2. 定位
  3. 导航
  4. 路线规划
  5. 搜索
  6. 室内地图
  7. 室内定位

自定义地图 位置数据可视化 海外LBS服务

Java SPI

SPI Service Provider Interface

checkout github repo:

Spring mongo data naming

Mongo Naming

JSON Naming

Google JSON Style Guide Gson has built-in magic to convert snakes into camels with its LOWER_CASE_WITH_UNDERSCORES FieldNamingPolicy. We make a global configuration change to our Gson instance and the problem is solved.

Java Naming

Spring data mongo

8.4.3. Property Expressions
Property expressions can refer only to a direct property of the managed entity, as shown in the preceding example. At query creation time, you already make sure that the parsed property is a property of the managed domain class. However, you can also define constraints by traversing nested properties. Consider the following method signature:

List<Person> findByAddressZipCode(ZipCode zipCode);
Assume a Person has an Address with a ZipCode. In that case, the method creates the property traversal x.address.zipCode. The resolution algorithm starts by interpreting the entire part (AddressZipCode) as the property and checks the domain class for a property with that name (uncapitalized). If the algorithm succeeds, it uses that property. If not, the algorithm splits up the source at the camel case parts from the right side into a head and a tail and tries to find the corresponding property — in our example, AddressZip and Code. If the algorithm finds a property with that head, it takes the tail and continues building the tree down from there, splitting the tail up in the way just described. If the first split does not match, the algorithm moves the split point to the left (Address, ZipCode) and continues.

Although this should work for most cases, it is possible for the algorithm to select the wrong property. Suppose the Person class has an addressZip property as well. The algorithm would match in the first split round already, choose the wrong property, and fail (as the type of addressZip probably has no code property).

To resolve this ambiguity you can use \_ inside your method name to manually define traversal points. So our method name would be as follows:

List<Person> findByAddress_ZipCode(ZipCode zipCode);
Because we treat the underscore character as a reserved character, we strongly advise following standard Java naming conventions (that is, not using underscores in property names but using camel case instead).

Mongo References:

  • manual references
  • DBRef

Manual references are named after the referenced collection. Use the document type (collection name in singular) followed by _id ( _id ). This is the only case where you can use underscore in the middle. For example, in a dogs collection, a document would have manual references to external documents named like this: { name: 'fido', owner_id: '5358e4249611f4a65e3068ab', race_id: '5358ee549611f4a65e3068ac', colour: 'yellow' ... }

Mongo follows javascript naming convention using camelCase

Mongo has no preference on camelCase v.s. underscore_style just stay consistent with the language. But also be consistent with on-wire json/xml format, also consider for Android/ iOS / Javascript web consumer sides.

different to relational database, in mongoldb, key names are stored in every document, so avoid using long names.


using shorter keys can potentially save a lot of space and even increase performance (after all, more of a smaller database can fit into memory). WiredTiger engine in Mongo 3.0 supports compression, the point is rather moot.


  • Use tabs to indent. This applies to all MongoDB-specific code (chained functions) and objects used by MongoDB (queries, projections, documents).
  • Always have a space after a : colon.
  • Comma-last. If you divide the components of an object/array into various lines, use one line for each component. The } closing curly brace should follow the last component (except for aggregation). GENERAL GUIDELINES
  • No _ underscores in the middle of names (database, collection, fields) except for manual references.
  • Collections database, variables, properties and function names should uselowerCamelCase. They should also be descriptive. Single character variables and uncommon abbreviations should generally be avoided.
  • Place spaces between nested parentheticals and elements in JavaScript examples. For example, prefer { [ a, a, a ] } over {[a,a,a]}.


  • The name should be a plural of the types of documents to be saved.
  • Use camelCase. Normally you shouldn’t have to because the collection name will be one word (plural of the types of documents saved).
  • A collection with “” empty string is not a valid collection name.
  • A collection name should not contain the null character because this defines the end of collection name.
  • Collection name should not start with the prefix “system.” as this is reserved for internal collections.
  • It would be good to not contain the character “$” in the collection name as various drivers available for database do not support “$” in collection name. DATABASE NAMES

  • Try to have the database named after the project and one database per project.
  • Use camelCase.
  • A database with “” empty string is not a valid database name.
  • Database name cannot be more than 64 bytes.
  • Database name are case-sensitive, even on non-case-sensitive file systems. Thus it is good to keep name in lower case.
  • A database name cannot contain any of these characters “/, \, ., “, *, <, >, :, |, ?, $,”. It also cannot contain a single space or null character. FIELD NAMES

  • Use camelCase.
  • Don’t use _ underscore as the starting character of the field name. The only field with _ undescore should be _id.
  • Field names cannot contain . dots or null characters and must not start with a $ dollar sign. Manual references are named after the referenced collection. Use the document type (collection name in singular) followed by _id ( _id ). This is the only case where you can use underscore in the middle. For example, in a dogs collection, a document would have manual references to external documents named like this: { name: 'fido', owner_id: '5358e4249611f4a65e3068ab', race_id: '5358ee549611f4a65e3068ac', colour: 'yellow' ... } FUNCTIONS

  • One method per line should be used when chaining methods.
  • You should also indent these methods with a tab so it’s easier to tell they are part of the same chain. If you need to use a long query, projection or options object for a function, assign it to a var and use the var in the function call to improve readability.



Promise based HTTP client for the browser and node.js


create project and download deps.

vue create hello_mockjs
cd hello_mockjs 
npm install mockjs

npm run serve
  • 数据模板定义规范 DTD ` ‘name|rule’: value `


Random is a useful nodejs commandline / library for generating random objects, like url, email, etc.

npm install random -g 
➜  hello_mockjs git:(master) ✗ random

  Usage: random [options] [command]


    -V, --version  output the version number
    -h, --help     output usage information


    boolean        Random.boolean( min,  max,  cur )
    bool           Random.bool( min,  max,  cur )
    natural        Random.natural( min,  max )
    integer        Random.integer( min,  max )
    int   min,  max )
    float          Random.float( min,  max,  dmin,  dmax )
    character      Random.character( pool )
    char           Random.char( pool )
    string         Random.string( pool,  min,  max )
    str            Random.str()
    range          Random.range( start,  stop,  step )
    date  format )
    time           Random.time( format )
    datetime       Random.datetime( format )
    now   unit,  format )
    image          Random.image( size,  background,  foreground,  format,  text )
    img            Random.img()
    color          Random.color( name )
    hex            Random.hex()
    rgb            Random.rgb()
    rgba           Random.rgba()
    hsl            Random.hsl()
    paragraph      Random.paragraph( min,  max )
    cparagraph     Random.cparagraph( min,  max )
    sentence       Random.sentence( min,  max )
    csentence      Random.csentence( min,  max )
    word           Random.word( min,  max )
    cword          Random.cword( pool,  min,  max )
    title          Random.title( min,  max )
    ctitle         Random.ctitle( min,  max )
    first          Random.first()
    last           Random.last()
    name  middle )
    cfirst         Random.cfirst()
    clast          Random.clast()
    cname          Random.cname()
    url            Random.url( protocol,  host )
    protocol       Random.protocol()
    domain         Random.domain( tld )
    tld            Random.tld()
    email domain )
    ip             Random.ip()
    region         Random.region()
    province       Random.province()
    city  prefix )
    county         Random.county( prefix )
    zip   len )
    d4             Random.d4()
    d6             Random.d6()
    d8             Random.d8()
    d12            Random.d12()
    d20            Random.d20()
    d100           Random.d100()
    guid           Random.guid()
    uuid           Random.uuid()

    $ random date yyyy-MM-dd
    $ random time HH:mm:ss



每一个 Vuex 应用的核心就是 store(仓库)。“store”基本上就是一个容器,它包含着你的应用中大部分的状态 (state)。

  • reactive in store state vars.
  • commit mutation



  • import request from '@/utils/request' what is @?
  • requests interceptor 中 set login token
  • response interceptor 中验证 response 错误
  • arrow functions

    sidebar: state =>,
  • store
    • getters
    • store modules
      const store = new Vuex.Store({
        modules: {
      export default store
  • export function
  • router
      path: '/',
      component: Layout,
      redirect: '/dashboard',
      name: 'Dashboard',
      hidden: true,
      children: [{
        path: 'dashboard',
        component: () => import('@/views/dashboard/index')
  • vuex
    • getters
    • mapGetters
    • mutations


Box model



box-sizing: content-box/ border-box

line-width: normal/1.6/30px/0.8em

margin-bottom: 2px/3em/10%/auto

max-height: none/2000px
min-height: 0/200px


width: auto/240px/50%

overflow: auto/hidden/visible
background-cli: border-box/padding-box/content-box
display: block/inline/inline-block/positioned/table/flex
  • You can use unitless values: the line height will be relative to the font size. line-height: 1.6;
  • You can use pixel values. line-height: 30px;
  • You can use em values: like with unitless values, the line height will be relative to the font size. line-height: 0.8em;


Before the Flexbox Layout module, there were four layout modes:

  • Block, for sections in a webpage
  • Inline, for text
  • Table, for two-dimensional table data
  • Positioned, for explicit position of an element
align-content: stretch/flex-start/flex-end/center/space-between/space-around/
align-items: flex-start/flex-end/center/baseline/stretch
align-self: auto/flex-start/flex-end/center/baseline/stretch
flex-basis: auto/80px/
flex-direction: row/row-reverse/column/column-reverse
flex-grow: 0/1
flex-shrink: 0/1
flex-wrap: nowrap/wrap/wrap-reverse
justify-conent: flex-start/flex-end/center/space-between/space-around
order: 0/1/-1/9


Node.js development stack

Node.js development stack

function projects  
package management npm  
CSS preprocessors less, sass  
JS module loader require.js, Browserfy  
JS transpiler coffee loader, ts-loader  
  • ES2015+
  • vue
  • vuex
  • vue-router
  • axios
  • element-ui
  • Mock.js


  • Node.js uses asynchronous programming!
  • Node.js can generate dynamic page content
  • Node.js can create, open, read, write, delete, and close files on the server
  • Node.js can collect form data
  • Node.js can add, delete, modify data in your database


  • What is a Module in Node.js?

    Consider modules to be the same as JavaScript libraries.A set of functions you want to include in your application.

  • built-in modules
Module Description
assert Provides a set of assertion tests
buffer To handle binary data
child_process To run a child process
cluster To split a single Node process into multiple processes
crypto To handle OpenSSL cryptographic functions
dgram Provides implementation of UDP datagram sockets
dns To do DNS lookups and name resolution functions
domain Deprecated. To handle unhandled errors
events To handle events
fs To handle the file system
http To make Node.js act as an HTTP server
https To make Node.js act as an HTTPS server.
net To create servers and clients
os Provides information about the operation system
path To handle file paths
punycode Deprecated. A character encoding scheme
querystring To handle URL query strings
readline To handle readable streams one line at the time
stream To handle streaming data
string_decoder To decode buffer objects into strings
timers To execute a function after a given number of milliseconds
tls To implement TLS and SSL protocols
tty Provides classes used by a text terminal
url To parse URL strings
util To access utility functions
v8 To access information about V8 (the JavaScript engine)
vm To compile JavaScript code in a virtual machine
zlib To compress or decompress files
  • Include Modules

    To include a module, use the require() function with the name of the module:

      var http = require('http');
  • Create Your Own Modules
      // save as mymodule.j* 
      exports.myDateTime = function () {
          return Date();
      // in another main.js
      var mymodule = require('./mymodule')
      var datetime = mymodule.myDateTime()
  • HTTP Module
      var http = require('http');
      //create a server object:
      http.createServer(function (req, res) {
      res.write('Hello World!'); //write a response to the client
      res.end(); //end the response
    • Add an HTTP Header

      res.writeHead(200, {'Content-Type': 'text/html'});

    • Read the Query String


    • Split the Query String
        var q = url.parse(req.url, true).query;
        var txt = q.year + " " + q.month;
  • File System Module
    • Node.js as a File Server
        var fs = require('fs');
    • Read Files
        fs.readFile('demofile1.html', function(err, data) {
    • Create Files
        var fs = require('fs');
        fs.appendFile('mynewfile1.txt', 'Hello content!', function (err) {
            if (err) throw err;
    • Delete Files

        var fs = require('fs');
        fs.unlink('mynewfile2.txt', function (err) {
        if (err) throw err;
        console.log('File deleted!');
    • rename files
        fs.rename('mynewfile1.txt', 'myrenamedfile.txt', function (err) {
    • Upload Files
        var formidable = require('formidable');
  • Send an Email
    • nodemailer module var nodemailer = require('nodemailer');
    • multiple receivers
    • html
  • Events module
    • addListener()
    • defaultMaxListeners
    • emit()
    • eventNames()
    • getMaxListeners()
    • listenerCount()
    • listeners()
    • on()
    • once()
    • prependListener()
    • prependOnceListener()
    • removeAllListeners()
    • removeListener()
    • setMaxListeners()
      var events = require('events');
      var eventEmitter = new events.EventEmitter();
      eventEmitter.on('scream', function() {
      console.log('A scream is detected!');
  • NPM
    • download: npm install upper-case --save
    • use: var uc = require('upper-case')
  • URL
      var url = require('url');
      var adr = 'http://localhost:8080/default.htm?year=2017&month=february';
      var q = url.parse(adr, true);
      console.log(; //returns 'localhost:8080'
      console.log(q.pathname); //returns '/default.htm'
      console.log(; //returns '?year=2017&month=february'
      var qdata = q.query; //returns an object: { year: 2017, month: 'february' }
      console.log(qdata.month); //returns 'february'

Node.js and MySQL

Node.js and MongoDB

  • install driver: npm install mongodb --save
  • var mongo = require('mongodb');
  • create collections
      var MongoClient = require('mongodb').MongoClient;
      var url = "mongodb://localhost:27017/";
      MongoClient.connect(url, function(err, db) {
          if (err) throw err;
          var dbo = db.db("mydb");
          dbo.createCollection("customers", function(err, res) {
              if (err) throw err;
              console.log("Collection created!");
          var myobj = { name: "Company Inc", address: "Highway 37" };
          dbo.collection("customers").insertOne(myobj, function(err, res) {
              if (err) throw err;
              console.log("1 document inserted");


  • package.json

    The package.json file is core to the Node.js ecosystem and is a basic part of understanding and working with Node.js, npm, and even modern JavaScript. The package.json is used as what equates to a manifest about applications, modules, packages, and more - it’s a tool to that’s used to make modern development streamlined, modular, and efficient.

    • name
    • version
    • description
    • license
    • keywords
    • dependencies
    • devDependencies
    • “main”: “app.js”,
    • repository
    • scripts

Learn ES2015

  • Arrows and Lexical This


Hello world

  1. import: <script src=""></script>
  2. HTML ```html
3. javascript

var app = new Vue({
  el: '#app',
  data: {
    message: 'hello, world'


HTML 中用一个 mount point <div id="app">, 在 javascript 创建一个 Vue 示例。


  1. 文本插值: ``
  2. v-bind::title e.g. ` `
  3. v-on: @click 事件处理 ` `
  4. 条件: v-if
  5. 循环: v-for
  6. 用户输入: v-model: ` `


// 定义名为 todo-item 的新组件
Vue.component('todo-item', {
  template: '<li>这是个待办项</li>'

Instead, a component’s data option must be a function, so that each instance can maintain an independent copy of the returned data object:

data: function () {
  return {
    count: 0

Organizing components

  • registration:
    • global
      • Vue.component('name', opts)
      • Globally registered components can be used in the template of any root Vue instance (new Vue) created afterwards – and even inside all subcomponents of that Vue instance’s component tree.
    • local
  • Props
    • props is like interface to class, and data is used to maitain interal state, as private fields to class.
  • SFC

  • Passing Data to Child Components with Props
  • Sending Messages to Parents with Events
  • v-model
  • Content Distribution with Slots
  • Dynamic Components <component v-bind:is="currentTabComponent"></component>

Vue class / instance

var vm = new Vue({



  1. el: 锚点
  2. data: 要绑定的数据
  3. computed: 复杂属性
  4. props: 对外的属性接口,类似 C# /Android 中 view 等属性。
  5. methods: 方法


global config

  • silent
  • optionMergeStrategies
  • devtools
  • errorHandler
  • warnHandler
  • ignoredElements
  • keyCodes
  • performance
  • productionTip

Global API

  • Vue.extend(options)
  • Vue.nextTick([callback, context])
  • Vue.set(target, key, value)
  • Vue.delete(target, key)
  • Vue.directive(id, [definition])
  • Vue.filter(id, [definition])
  • Vue.component(id, [definition])
  • Vue.use(plugin)
  • Vue.mixin(mixin)
  • Vue.compile(template)
  • Vue.version


  • data
  • props: A list/hash of attributes that are exposed to accept data from the parent component.
  • propsData: initial / default data for props
  • computed:


  • $emit

synthetic events Creating and triggering events

vm.$emit( eventName, […args] )


  1. what is javascript prototype?


  2. what is javascript context this?

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

javascript context/ scope/ this

  • context/ this: function

context means this keyword

Context is most often determined by how a function is invoked. When a function is called as a method of an object, this is set to the object the method is called on:

So for a global function, this refers to ‘window’ in browser

  • scope: variable
    • local
    • Global
    • block: let/ const

    Understanding Scope and Context in JavaScript

  • Execution Context

  • Execution stack

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.


new 3.x is @vue/cli, the old is vue-cli

  • instant prototyping
      vue serve
  • Create a project
      vue create
  • Plugins
  • Presets
  • CLI service



  • UI template

      vue ui


➜  hello-vue git:(master) ✗ ./node_modules/.bin/vue-cli-service 

  Usage: vue-cli-service <command> [options]


    serve     start development server
    build     build for production
    inspect   inspect internal webpack config
    lint      lint and fix source files

  run vue-cli-service help [command] for usage of a specific command.
  • Using the Binary
  • vue-cli-service serve
  • vue-cli-service build
  • vue-cli-service inspect
  • Checking All Available Commands
  • Caching and Parallelization
  • Git Hooks
  • Configuration without Ejecting
  • webpack-dev-server


  • UMD
  • AMD
  • CommonJS is a project with the goal of specifying an ecosystem for JavaScript outside the browser (for example, on the server or for native desktop applications).


是一个 webpack 的 loader,可以将用下面这个格式编写的 Vue 组件转换为 JavaScript 模块:

AMD v.s. CommonJS v.s. UMD

AMD v.s. CommonJS


Asynchronous Module Definition

  • AMD <- RequireJS
      //    filename: foo.js
      define(['jquery', 'underscore'], function ($, _) {
          //    methods
          function a(){};    //    private because it's not returned (see below)
          function b(){};    //    public because it's returned
          function c(){};    //    public because it's returned
          //    exposed public methods
          return {
              b: b,
              c: c
  • CommonJS <- Browserify

      //    filename: foo.js
      var $ = require('jquery');
      var _ = require('underscore');
      //    methods
      function a(){};    //    private because it's omitted from module.exports (see below)
      function b(){};    //    public because it's defined in module.exports
      function c(){};    //    public because it's defined in module.exports
      //    exposed public methods
      module.exports = {
          b: b,
          c: c
  • UMD: The pattern is admittedly ugly, but is both AMD and CommonJS compatible, as well as supporting the old-style “global” variable definition:

      (function (root, factory) {
          if (typeof define === 'function' && define.amd) {
              // AMD
              define(['jquery'], factory);
          } else if (typeof exports === 'object') {
              // Node, CommonJS-like
              module.exports = factory(require('jquery'));
          } else {
              // Browser globals (root is window)
              root.returnExports = factory(root.jQuery);
      }(this, function ($) {
          //    methods
          function myFunc(){};
          //    exposed public method
          return myFunc;


  • js === v.s. ==




vuex 集中管理应用中的状态。 类似 Java Spring 中的配置中心。 一种管理的哲学思想 commit 的方式相当于 会隐藏了 setOnListener 的调用, 类似 set 方法, 而不是直接访问状态字段

  • state
  • mutations
  • 触发变化 store.commit('increment')
  • 调用状态: 计算属性
     computed: {
      count () {
        return store.state.count
  • 单一状态树, 唯一数据源 SSOT, 状态快照

So actions/mutations allow you to:

  • centrally define allowed changes to the state, and by that,
  • how those changes are connected and logically depend on one another (i.e. by reading an action that commits a series of mutations)
  • track these changes in the devtools very easily, as you see all the Mutation names you just read in the list there.
  • writing expressive code that, to an extend, documents your business logic. You can understand a lot about an app’s business logic by just reading its store actions & mutations, without digging through hundreds of components looking where some piece of state is altered under which circumstances.

  • mapGetter 类同 Java Bean 中的自动生成 get 函数
  • mapMutations 类同 Java Bean 中自动生成 set 函数
  • 可以手写 get / set 函数, 以定制化
    • get: getters:
    • set: mutations:
    • mutaions = event + event handler (type + handler)
    • computed: mapState
  • Actions
    • …mapActions


  • setInterval
  • clearInterval
  • setTimeout clearTimeout

Spring Jackson

Jackson Exceptions – Problems and Solutions

JsonMappingException UnrecognizedPropertyException

Spring Mongo

  • root Spring Data’s MongoDB abstraction.
  • .config Spring XML namespace configuration for MongoDB specific repositories.
  • .core MongoDB core support.
  • .core.aggregation Support for the MongoDB aggregation framework.
  • .core.convert Spring Data MongoDB specific converter infrastructure.
  • .core.geo Support for MongoDB geo-spatial queries.
  • .core.index Support for MongoDB document indexing.
  • .core.mapping Infrastructure for the MongoDB document-to-object mapping subsystem.
  • .core.mapping.event Mapping event callback infrastructure for the MongoDB document-to-object mapping subsystem.
  • .core.mapreduce Support for MongoDB map-reduce operations.
  • .core.query MongoDB specific query and update support.
  • .core.script Abstraction classes javascript function execution within MongoDB Server.
  • .core.spel Support classes to transform SpEL expressions into MongoDB expressions.
  • .crossstore Infrastructure for Spring Data’s MongoDB cross store support.
  • .gridfs Support for MongoDB GridFS feature.
  • .monitor MongoDB specific JMX monitoring support.
  • .repository MongoDB specific repository implementation.
  • .repository.cdi CDI support for MongoDB specific repository implementation.
  • .repository.config Support infrastructure for the configuration of MongoDB specific repositories.
  • .repository.query Query derivation mechanism for MongoDB specific repositories.
  • Support infrastructure for query derivation of MongoDB specific repositories.
  • .util MongoDB driver-specific utility classes for Bson and DBObject interaction.


    "timestamp": "2018-06-08T06:47:54.222+0000",
    "status": 500,
    "error": "Internal Server Error",
    "message": "Could not extract response: no suitable HttpMessageConverter found for response type [class] and content type [text/plain]",
    "path": "/auth/login/wechat"


Install LetsEncrypt

root@open2:/etc/nginx/sites-enabled# ls
root@open2:/etc/nginx/sites-enabled# git diff
root@open2:/etc/nginx/sites-enabled# vim 
root@open2:/etc/nginx/sites-enabled# certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

Which names would you like to activate HTTPS for?
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 4
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

Congratulations! You have successfully enabled

You should test your configuration at:

 - Congratulations! Your certificate and chain have been saved at:
   Your key file has been saved at:
   Your cert will expire on 2018-09-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:
   Donating to EFF:          

Ethereum and Solidity


  • Remix
  • npm / Node.js

web3j web3.js

Introduction to Smart Contracts

Solidity 是 Ethereum 上智能合约实现的主要语言之一。 运行在 EVM 上。

  • static typing
  • variadic return types
  • Complex member variables for contracts including arbitrarily hierarchical mappings and structs
  • Contracts support inheritance, including multiple inheritance with C3 linearization.
  • An application binary interface (ABI) facilitating multiple type-safe functions within a single contract was also introduced (and later supported by Serpent).
  • A documentation system for specifying a user-centric description of the ramifications of a method-call was also included in the proposal, known as “Natural Language Specification”.
  • on EVM
  • compile to bytecode
  • designed around ECMAScript

Blockchain Basics

  • Transactions
  • Blocks
  • EVM
    • Accounts: external accounts controlled by public private key pairs
    • contract accounts controlled by the code stored together with the account.
    • Every account has a persistent key-value store mapping 256-bit words to 256-bit words called storage.
  • Storage:a key-value store that maps 256-bit words to 256-bit words. It is not possible to enumerate storage from within a contract and it is comparatively costly to read and even more so, to modify storage. A contract can neither read nor write to any storage apart from its own.
  • Memory: a contract obtains a freshly cleared instance for each message call. Memory is linear and can be addressed at byte level, but reads are limited to a width of 256 bits, while writes can be either 8 bits or 256 bits wide. Memory is expanded by a word (256-bit), when accessing (either reading or writing) a previously untouched memory word (ie. any offset within a word). At the time of expansion, the cost in gas must be paid. Memory is more costly the larger it grows (it scales quadratically).
  • Stack
  • Instruction Set
  • Message Calls
  • Delegatecall / Callcode and Libraries
  • Logs
  • Create
  • self-destruct

Installing the Solidity Compiler

Solidity by Example

Solidity in Depth

Security Considerations

Using the compiler

Contract Metadata

Application Binary Interface Specification

Joyfully Universal Language for (Inline) Assembly

Style Guide

Common Patterns

List of Known Bugs


Frequently Asked Questions


Gloabal variable

Ether Units

  • wei,
  • finney,
  • szabo or
  • ether

Time Units

  • 1 == 1 seconds
  • 1 minutes == 60 seconds
  • 1 hours == 60 minutes
  • 1 days == 24 hours
  • 1 weeks == 7 days
  • 1 years == 365 days


  • block.blockhash(uint blockNumber) returns (bytes32): hash of the given block - only works for 256 most recent blocks excluding current
  • block.coinbase (address): current block miner’s address
  • block.difficulty (uint): current block difficulty
  • block.gaslimit (uint): current block gaslimit
  • block.number (uint): current block number
  • block.timestamp (uint): current block timestamp as seconds since unix epoch
  • gasleft() returns (uint256): remaining gas
  • (bytes): complete calldata
  • msg.gas (uint): remaining gas - deprecated in version 0.4.21 and to be replaced by gasleft()
  • msg.sender (address): sender of the message (current call)
  • msg.sig (bytes4): first four bytes of the calldata (i.e. function identifier)
  • msg.value (uint): number of wei sent with the message
  • now (uint): current block timestamp (alias for block.timestamp)
  • tx.gasprice (uint): gas price of the transaction
  • tx.origin (address): sender of the transaction (full call chain)

Error Handling

  • assert(bool condition): throws if the condition is not met - to be used for internal errors.
  • require(bool condition): throws if the condition is not met - to be used for errors in inputs or external components.
  • revert(): abort execution and revert state changes

Mathematical and Cryptographic Functions

  • addmod(uint x, uint y, uint k) returns (uint):
  • mulmod(uint x, uint y, uint k) returns (uint):
  • keccak256(…) returns (bytes32):
  • sha256(…) returns (bytes32):
  • sha3(…) returns (bytes32):
  • ripemd160(…) returns (bytes20):
  • ecrecover(bytes32 hash, uint8 v, bytes32 r, bytes32 s) returns (address):
  • <address>.balance (uint256):
  • <address>.transfer(uint256 amount):
  • <address>.send(uint256 amount) returns (bool):
  • <address>.call(...) returns (bool):
  • <address>.callcode(...) returns (bool):
  • <address>.delegatecall(...) returns (bool):
  • this
  • selfdestruct(address recipient)
  • suicide(address recipient)

Doxity: Documentation Generator for Solidity.

All identifiers (contract names, function names and variable names) are restricted to the ASCII character set. It is possible to store UTF-8 encoded data in string variables.

  • public: The keyword public automatically generates a function that allows you to access the current value of the state variable from outside of the contract. Without this keyword, other contracts have no way to access the variable.
  • address: 160 bit value
  • mapping: Mappings can be seen as hash tables which are virtually initialized such that every possible key exists and is mapped to a value whose byte-representation is all zeros.
  • progma once

How to develop and debug?


  • 256bit register stack
  • e-WASM
  • send()
  • transfer()
  • call.value()


pragma solidity ^0.4.23;

interface ERC20 {
    function totalSupply() external view returns (uint supply);
    function balanceOf(address _owner) external view returns (uint balance);
    function transfer(address _to, uint _value) external returns (bool success);
    function transferFrom(address _from, address _to, uint _value) external returns (bool success);
    function approve(address _spender, uint _value) external returns (bool success);
    function allowance(address _owner, address _spender) external view returns (uint remaining);
    function decimals() external view returns(uint digits);
    event Approval(address indexed _owner, address indexed _spender, uint _value);
import java.util.HashMap;
import java.util.Map;

public class ERC20Coin {

    static class Address {
        private String val;
        private static final String PATTERN_ERC20_HASH = "0x[0-9A-Fa-f]{64}";

        public Address(String val) {
            assert (val.matches(PATTERN_ERC20_HASH));
            this.val = val;

        public String toString() {
            return this.val;

        public boolean equals(Object obj) {
            return (obj instanceof Address)
                    && ((Address) obj).val.equals(this.val);

    static class TransferEvent{
        public TransferEvent(Address _from, Address _to, Integer _value){


    Address sender;

    String name;
    String symbol;
    long totalSupply;
    Integer demicals;

    Map<Address, Integer> balances;
    Map<Address, Map<Address, Integer>> allowances;

    public ERC20Coin(String name, String symbol, long totalSupply, Integer demicals){
        balances = new HashMap<>();
        allowances = new HashMap<>(); = name;
        this.symbol = symbol;
        this.totalSupply = totalSupply;
        this.demicals = demicals;

    public Integer balanceOf(Address address) {
        return balances.get(address);

    public void transfer(Address from, Address to, Integer val) {
        Integer balanceFrom = balances.getOrDefault(from, 0);
        Integer balanceTo = balances.getOrDefault(to, 0);

        Map<Address, Integer> allowance;
            allowance = new HashMap<Address, Integer>();
            allowance = allowances.get(from);
        Integer allowed = allowance.get(sender);
        assert  (allowance.get(sender) >= val) &&
                (balanceFrom >= val);

        if(allowed < Integer.MAX_VALUE){
            allowance.put(sender, allowed - val);
            allowances.put(from, allowance);

        balances.put(from, balanceFrom - val);
        balances.put(to, balanceTo + val);

    public void approve(Address from, Address to, Integer val){
        Map<Address, Integer> allowance;
            allowance = new HashMap<Address, Integer>();
            allowance = allowances.get(from);

        allowance.put(to, val);
        allowances.put(from, allowance);


Airdrop 空投



空投之前需要找到你需要空投地址列表。 如何找到这些地址列表呢?

  1. 在 Slack 或者 telegram 等群里寻找鱼儿。
  2. 扫描 Ethereum 上 balance >0 的那些有效账号地址
parity export state --chain=ropsten -no-storage -no-code -min-balance=1000 -max-balance=500000 -at=2000000 balances.json

节省 gas 费用

如何一次向多个账号send token?

Spring Security

Servelet Interceptor

authenticationManager, accessDecisionManager, securityMetadataSource


1)首先我们自定义一个过滤器(调度器,这里我们命名为mySecurityInterceptor),这个过滤器继承AbstractSecurityInterceptor类(这里先说明,本文但凡不是自定义的类或接口都是Spring Security提供的,无须深究)。 它至少包含 authenticationManager,accessDecisionManager,securityMetadataSource三个属性,我们的所有控制将在这三个类中实现。

(2)登录验证:自定义类MyUserDetailService实现UserDetailsService接口和其loadUserByUsername方法,这个方法根据用户输入的用户名,从数据库里面获取该用户的所有权限细信息(统称用户信息)。Spring Security的AuthenticationProcessingFilter拦截器调用authenticationManager,类MyUserDetailService拿到用户信息后,authenticationManager对比用户的密码(即验证用户),如果通过了,那么相当于通过了AuthenticationProcessingFilter拦截器,也就是登录验证通过。




Spring Security中进行身份验证的是AuthenticationManager接口,ProviderManager是它的一个默认实现,但它并不用来处理身份认证,而是委托给配置好的AuthenticationProvider,每个AuthenticationProvider会轮流检查身份认证。检查后或者返回Authentication对象或者抛出异常。




Python multitasking



  • Methods:
    • start()
    • run()
    • is_alive()
    • join()

Threads are executed in their own system-level thread (e.g., a POSIX thread or Windows threads) that is fully managed by the host operating system. Once started, threads run independently until the target function returns.

from threading import Thread
t = Thread(target=countdown, args=(10,))

Due to a global interpreter lock (GIL), Python threads are restricted to an execution model that only allows one thread to execute in the interpreter at any given time. For this reason, Python threads should generally not be used for computationally intensive tasks where you are trying to achieve parallelism on multiple CPUs. They are much better suited for I/O handling and handling concurrent execution in code that performs blocking operations (e.g., waiting for I/O, waiting for results from a database, etc.).

  • Event
    • wait()
    • set()
  • Condition
    • wait()
    • notify_all()
  • Semaphore
sema = threading.Semaphore(0)
  • Lock
self._value_lock = threading.Lock()
with self._value_lock:
        self._value += delta
  • RLock

An RLock or re-entrant lock object is a lock that can be acquired multiple times by the same thread. It is primarily used to implement code based locking or synchronization based on a construct known as a “monitor.” With this kind of locking, only one thread is allowed to use an entire function or the methods of a class while the lock is held.

  • threading.local()


Perhaps the safest way to send data from one thread to another is to use a Queue from the queue library. To do this, you create a Queue instance that is shared by the threads. Threads then use put() or get() operations to add or remove items from the queue.

  • q.put(data)
  • data = in_q.get()
  • in_q.task_done()
  • q.qsize()
  • q.full()
  • q.empty()


In a nutshell, an actor is a concurrently executing task that simply acts upon messages sent to it. In response to these messages, it may decide to send further messages to other actors. Communication with actors is one way and asynchronous. Thus, the sender of a message does not know when a message actually gets delivered, nor does it receive a response or acknowledgment that the message has been processed.



  • Process
  • Exchanging objects between processes


class Pool(object):
    Process = Process

    def __init__(self, processes=None, initializer=None, initargs=(),
        self._taskqueue = Queue.Queue()
        self._cache = {}
        ... # stuff we don't care about
        self._worker_handler = threading.Thread(
            args=(self, )
        self._worker_handler.daemon = True
        self._worker_handler._state = RUN 

        self._task_handler = threading.Thread(
            args=(self._taskqueue, self._quick_put, self._outqueue,
                  self._pool, self._cache)
        self._task_handler.daemon = True
        self._task_handler._state = RUN 

        self._result_handler = threading.Thread(
            args=(self._outqueue, self._quick_get, self._cache)
        self._result_handler.daemon = True
        self._result_handler._state = RUN
  • queues:
    • self._taskqueue = Queue.Queue(): cache all submited tasks
    • self._inqueue = SimpleQueue(): _task_handler fetch task from _taskqueue and add to _inqueue for process to work on, this is shared among main process and worker process.
    • self._outqueue = SimpleQueue(): store the result.
  • threads:
    • _worker_handler = _handle_workers(pool): maintain the internal state
    • _task_handler = _handle_tasks(taskqueue, put, outqueue, pool, cache): fetch tasks and send to worker process
    • _result_handler = _handle_results(outqueue, get, cache): process result, callback etc.

Note: callback is called in _result_handler thread in caller process.



  • Executor.submit(fn, *args, **kwargs)
  •, *iterables, timeout=None)
  • Executor.shutdown(wait=True)


  • Use the ProcessPoolExecutor for CPU intensive tasks.
  • The ThreadPoolExecutor is better suited for network operations or I/O.


executor = ThreadPoolExecutor(max_workers=2)


executor = concurrent.futures.ProcessPoolExecutor(max_workers=None)

"""Implements ProcessPoolExecutor.

The follow diagram and text describe the data-flow through the system:

|======================= In-process =====================|== Out-of-process ==|

+----------+     +----------+       +--------+     +-----------+    +---------+
|          |  => | Work Ids |    => |        |  => | Call Q    | => |         |
|          |     +----------+       |        |     +-----------+    |         |
|          |     | ...      |       |        |     | ...       |    |         |
|          |     | 6        |       |        |     | 5, call() |    |         |
|          |     | 7        |       |        |     | ...       |    |         |
| Process  |     | ...      |       | Local  |     +-----------+    | Process |
|  Pool    |     +----------+       | Worker |                      |  #1..n  |
| Executor |                        | Thread |                      |         |
|          |     +----------- +     |        |     +-----------+    |         |
|          | <=> | Work Items | <=> |        | <=  | Result Q  | <= |         |
|          |     +------------+     |        |     +-----------+    |         |
|          |     | 6: call()  |     |        |     | ...       |    |         |
|          |     |    future  |     |        |     | 4, result |    |         |
|          |     | ...        |     |        |     | 3, except |    |         |
+----------+     +------------+     +--------+     +-----------+    +---------+

Executor.submit() called:
- creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict
- adds the id of the _WorkItem to the "Work Ids" queue

Local worker thread:
- reads work ids from the "Work Ids" queue and looks up the corresponding
  WorkItem from the "Work Items" dict: if the work item has been cancelled then
  it is simply removed from the dict, otherwise it is repackaged as a
  _CallItem and put in the "Call Q". New _CallItems are put in the "Call Q"
  until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because
  calls placed in the "Call Q" can no longer be cancelled with Future.cancel().
- reads _ResultItems from "Result Q", updates the future stored in the
  "Work Items" dict and deletes the dict entry

Process #1..n:
- reads _CallItems from "Call Q", executes the calls, and puts the resulting
  _ResultItems in "Request Q"


  • Future.cancel()
  • Future.cancelled()
  • Future.running()
  • Future.result(timeout=None)
  • Future.exception(timeout=None)
  • Future.add_done_callback(fn)

Internal methods, meant for use in unit tests and Executor implementations.

  • Future.set_running_or_notify_cancel()
  • Future.set_result(result)
  • Future.set_exception(exception)

module functions

  • concurrent.futures.wait(fs, timeout=None, return_when=ALL_COMPLETED)
  • concurrent.futures.as_completed(fs, timeout=None)

The main difference between the aforementioned map method with as_completed is that map returns the results in the order in which we pass the iterables. That is the first result from the map method is the result for the first item. On the other hand, the first result from the as_completed function is from whichever future completed first.

#Launching a Daemon Process on Unix

Debugging Python Multi-processes Program

Take aways

  • Another subtle aspect of pools is that mixing threads and process pools together can be a good way to make your head explode. If you are going to use both of these features together, it is often best to create the process pool as a singleton at program startup, prior to the creation of any threads. Threads will then use the same process pool for all of their computationally intensive work.

  • Many programmers, when faced with thread performance problems, are quick to blame the GIL for all of their ills. However, doing so is shortsighted and naive. Just as a real-world example, mysterious “stalls” in a multithreaded network program might be caused by something entirely different (e.g., a stalled DNS lookup) rather than anything related to the GIL. The bottom line is that you really need to study your code to know if the GIL is an issue or not. Again, realize that the GIL is mostly concerned with CPU-bound processing, not I/O.

  • Great care should be made when combining process pools and programs that use threads. In particular, you should probably create and launch process pools prior to the creation of any threads (e.g., create the pool in the main thread at program startup).

## References



TypeScript is a superset for javascript, enabling us to write more elegant javascript code.

TypeScript in 5 minutes


TypeScript –> Javascript –> Nodejs

  1. use tsc to transpile TypeScript into JavaScript,
  2. then run js by nodejs or browser.


npm install -g typescript
tsc --version
Version 2.7.2

tsc --help

Project management


See typescript101

➜  typescript101 git:(master) ✗ tree
├── HelloWorld.js
├── HelloWorld.ts
├── greeter.html
├── greeter.js
├── greeter.ts
└── tsconfig.json

With Visual Studio Code

Warning: Visual Studio for Mac Community Edition does not support TypeScript natively. So if on Mac, use Visual Studio Code instead. or Use Visual Studio on Windows for better IDE support.


  1. Editing TypeScript in VSCode
  2. Simplest TypeScript project using Visual Studio Code
    • File Scope
    • Explicit Project


This is the project definition file.

Spring boot

Sping Boot

Dependency Management

The curated list contains all the spring modules that you can use with Spring Boot as well as a refined list of third party libraries. The list is available as a standard Bills of Materials (spring-boot-dependencies) that can be used with both Maven and Gradle.

➜  NutNetwork git:(master) ✗ ./gradlew dependencies 
    'build.plugins.plugin[].dependencies.dependency.scope' for junit:junit:jar must be one of [compile, runtime, system] but is 'test'. in log4j:log4j:1.2.15
+--- org.springframework.boot:spring-boot-starter-data-rest -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot:2.0.0.RELEASE
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE
|    |    |    |    \--- org.springframework:spring-jcl:5.0.4.RELEASE
|    |    |    \--- org.springframework:spring-context:5.0.4.RELEASE
|    |    |         +--- org.springframework:spring-aop:5.0.4.RELEASE
|    |    |         |    +--- org.springframework:spring-beans:5.0.4.RELEASE
|    |    |         |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         \--- org.springframework:spring-expression:5.0.4.RELEASE
|    |    |              \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.0.0.RELEASE
|    |    |    \--- org.springframework.boot:spring-boot:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-logging:2.0.0.RELEASE
|    |    |    +--- ch.qos.logback:logback-classic:1.2.3
|    |    |    |    +--- ch.qos.logback:logback-core:1.2.3
|    |    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    |    +--- org.apache.logging.log4j:log4j-to-slf4j:2.10.0
|    |    |    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    |    |    \--- org.apache.logging.log4j:log4j-api:2.10.0
|    |    |    \--- org.slf4j:jul-to-slf4j:1.7.25
|    |    |         \--- org.slf4j:slf4j-api:1.7.25
|    |    +--- javax.annotation:javax.annotation-api:1.3.2
|    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.yaml:snakeyaml:1.19
|    +--- org.springframework.boot:spring-boot-starter-json:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework:spring-web:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.6.3
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0 -> 2.6.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    |    \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.4
|    |         +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |         \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    +--- org.springframework.boot:spring-boot-starter-web:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-json:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-tomcat:2.0.0.RELEASE
|    |    |    +--- javax.annotation:javax.annotation-api:1.3.2
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-core:8.5.28
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-el:8.5.28
|    |    |    \--- org.apache.tomcat.embed:tomcat-embed-websocket:8.5.28
|    |    |         \--- org.apache.tomcat.embed:tomcat-embed-core:8.5.28
|    |    +--- org.hibernate.validator:hibernate-validator:6.0.7.Final
|    |    |    +--- javax.validation:validation-api:2.0.1.Final
|    |    |    +--- org.jboss.logging:jboss-logging:3.3.0.Final -> 3.3.2.Final
|    |    |    \--- com.fasterxml:classmate:1.3.1 -> 1.3.4
|    |    +--- org.springframework:spring-web:5.0.4.RELEASE (*)
|    |    \--- org.springframework:spring-webmvc:5.0.4.RELEASE
|    |         +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|    |         \--- org.springframework:spring-web:5.0.4.RELEASE (*)
|    \---
|         +---
|         |    +--- org.springframework:spring-tx:5.0.4.RELEASE
|         |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         |    +--- org.springframework.hateoas:spring-hateoas:0.23.0.RELEASE -> 0.24.0.RELEASE
|         |    |    +--- org.springframework:spring-aop:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-beans:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-context:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-core:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-web:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-webmvc:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.25
|         |    +---
|         |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.25
|         |    +--- org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE
|         |    |    +--- org.springframework:spring-beans:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-context:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-aop:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.10 -> 1.7.25
|         |    +--- org.atteo:evo-inflector:1.2.2
|         |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.4 -> 2.6.3
|         |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.4 (*)
|         |    \--- org.slf4j:slf4j-api:1.7.25
|         +--- org.springframework:spring-webmvc:5.0.4.RELEASE (*)
|         +--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|         +--- com.fasterxml.jackson.core:jackson-annotations:2.9.4 -> 2.6.3
|         \--- org.slf4j:slf4j-api:1.7.25
+--- org.springframework.boot:spring-boot-starter-data-jpa -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-aop:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    \--- org.aspectj:aspectjweaver:1.8.13
|    +--- org.springframework.boot:spring-boot-starter-jdbc:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- com.zaxxer:HikariCP:2.7.8
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    \--- org.springframework:spring-jdbc:5.0.4.RELEASE
|    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |         \--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    +--- org.hibernate:hibernate-core:5.2.14.Final
|    |    +--- org.jboss.logging:jboss-logging:3.3.1.Final -> 3.3.2.Final
|    |    +--- org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final
|    |    +--- org.javassist:javassist:3.22.0-GA
|    |    +--- antlr:antlr:2.7.7
|    |    +--- org.jboss:jandex:2.0.3.Final
|    |    +--- com.fasterxml:classmate:1.3.0 -> 1.3.4
|    |    +--- dom4j:dom4j:1.6.1
|    |    \--- org.hibernate.common:hibernate-commons-annotations:5.0.1.Final
|    |         \--- org.jboss.logging:jboss-logging:3.3.0.Final -> 3.3.2.Final
|    +--- javax.transaction:javax.transaction-api:1.2
|    +---
|    |    +--- (*)
|    |    +--- org.springframework:spring-orm:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-jdbc:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- org.springframework:spring-aspects:5.0.4.RELEASE
|         \--- org.aspectj:aspectjweaver:1.8.13
+--- org.springframework.boot:spring-boot-starter-thymeleaf -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.thymeleaf:thymeleaf-spring5:3.0.9.RELEASE
|    |    +--- org.thymeleaf:thymeleaf:3.0.9.RELEASE
|    |    |    +--- org.attoparser:attoparser:2.0.4.RELEASE
|    |    |    +--- org.unbescape:unbescape:1.1.5.RELEASE
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- org.thymeleaf.extras:thymeleaf-extras-java8time:3.0.1.RELEASE
|         +--- org.thymeleaf:thymeleaf:3.0.0.RELEASE -> 3.0.9.RELEASE (*)
|         \--- org.slf4j:slf4j-api:1.6.6 -> 1.7.25
+--- org.springframework.boot:spring-boot-starter-security -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    +---
|    |    +---
|    |    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    \---
|         +--- (*)
|         +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|         \--- org.springframework:spring-web:5.0.4.RELEASE (*)
+--- org.springframework.boot:spring-boot-starter-data-redis -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +---
|    |    +---
|    |    |    +--- (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-oxm:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context-support:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- io.lettuce:lettuce-core:5.0.2.RELEASE
|         +--- io.projectreactor:reactor-core:3.1.4.RELEASE -> 3.1.5.RELEASE
|         |    \--- org.reactivestreams:reactive-streams:1.0.2
|         +--- io.netty:netty-common:4.1.21.Final -> 4.1.22.Final
|         +--- io.netty:netty-transport:4.1.21.Final -> 4.1.22.Final
|         |    +--- io.netty:netty-buffer:4.1.22.Final
|         |    |    \--- io.netty:netty-common:4.1.22.Final
|         |    \--- io.netty:netty-resolver:4.1.22.Final
|         |         \--- io.netty:netty-common:4.1.22.Final
|         \--- io.netty:netty-handler:4.1.21.Final -> 4.1.22.Final
|              +--- io.netty:netty-buffer:4.1.22.Final (*)
|              +--- io.netty:netty-transport:4.1.22.Final (*)
|              \--- io.netty:netty-codec:4.1.22.Final
|                   \--- io.netty:netty-transport:4.1.22.Final (*)
+--- org.springframework.boot:spring-boot-starter-web -> 2.0.0.RELEASE (*)
+--- com.h2database:h2 -> 1.4.196
+--- org.mybatis:mybatis:3.3.0
+--- org.mybatis:mybatis-spring:1.2.3
+--- com.mchange:c3p0:
|    \--- com.mchange:mchange-commons-java:0.2.10
+--- mysql:mysql-connector-java:5.1.31

Code Structure

 +- example
     +- myapplication
         +- customer
         |   +-
         |   +-
         |   +-
         |   +-
         +- order


Divide and Conqure


  1. divide n -> P1, P2, … Pk
  2. 最小子问题直接求解
  3. 递归/ 迭代求解子问题
  4. 合并子问题的解


Binary search a sorted array a[i]

  1. divide: n -> (0 – n/2) (n/2 – n)
  2. 最小子问题直接求解: n = 0, 1
  3. 递归/ 迭代求解子问题: each step, search either half.
  4. 合并子问题的解: 最后一个子问题的解就是整个问题的解







f(n) = f(n-1) , f(n-2)

e.g. Fibonacci 数列问题

此类问题依赖前面问题的求解, 如果是迭代求解可以缓存前面几个问题的解结果。 如果递归则不用


public int fibonacci(int n){
    if(n == 0 || n ==1) return n;
    return fibonacci(n-1)+ fibonacci(n-2); 

如果是迭代: 缓存中间结果。

public int fibonacci(int n){
    if(n == 0 || n ==1) return n;
    int f1 = 0;
    int f2 = 1;
    for(int i = 2; i<=n; i++){
        int t = f2;
        f2 = f1+t;
        f1 = t;
    return f2;

还有一种更加绝妙的解法, 注意

f(n+1) = f(n) + f(n-1)


[f(n+1) f(n)  ]       = [f(n)      f(n-1)] * [1  1]    = [1  1]^n
[f(n)   f(n-1)]         [ f(n-1)   f(n-2)]   [1  0]      [1  0]

通过矩阵的 n 次幂, 可以解出 F(n)

这里可以使用 Divide & Conqure 技巧的就是求解幂。

public double pow(x, n){
    if(n==0) return 1;
    int half = pow(x, n/2);
    if(n%2 ==1)
        return half * half * x;
        return half * half;

将上面的 乘法 * 换成矩阵乘法就可以求解 f(n) 了。 复杂度降为 O(logn).

f(n) = f(n/2) or more generally f(n/k)

此类问题经常对应搜索空间减半,从而实现 O(logn) 的时间复杂度。 binary search 是典型此类问题。

**二分查找的形式: **

while(l < r){
    if(==) return found result;
    else if (< ) l++
    else r--

Binary Search 的三个模板:

3 Parts of a Binary Search

  1. Pre-processing: Sort if collection is unsorted
  2. Binary Search: Using a loop or recursion to divide search space in half after each comparison
  3. Post-processing: Determine viable candidates in the remaining space.

3 Templates for Binary Search

  1. Template #1 is used to search for an element or condition which can be determined by accessing a single index in the array.
    • Search Condition can be determined without comparing to the element’s neighbors (or use specific elements around it)
    • No post-processing required because at each step, you are checking to see if the element has been found. If you reach the end, then you know the element is not found
    • Initial Condition: left = 0, right = length-1
    • Termination: left > right
int binarySearch(int[] nums, int target){
  if(nums == null || nums.length == 0)
    return -1;

  int left = 0, right = nums.length - 1;
  while(left <= right){
    // Prevent (left + right) overflow
    int mid = left + (right - left) / 2;
    if(nums[mid] == target){ return mid; }
    else if(nums[mid] < target) { left = mid + 1; }
    else { right = mid - 1; }

  // End Condition: left > right
  return -1;
  1. It is used to search for an element or condition which requires accessing the current index and its immediate right neighbor’s index in the array.
    • An advanced way to implement Binary Search.
    • Search Condition needs to access element’s immediate right neighbor
    • Use element’s right neighbor to determine if condition is met and decide whether to go left or right
    • Gurantees Search Space is at least 2 in size at each step
    • Post-processing required. Loop/Recursion ends when you have 1 element left. Need to assess if the remaining element meets the condition.
    • Initial Condition: left = 0, right = length
    • Termination: left == right
int binarySearch(int[] nums, int target){
  if(nums == null || nums.length == 0)
    return -1;

  int left = 0, right = nums.length;
  while(left < right){
    // Prevent (left + right) overflow
    int mid = left + (right - left) / 2;
    if(nums[mid] == target){ return mid; }
    else if(nums[mid] < target) { left = mid + 1; }
    else { right = mid; }

  // Post-processing:
  // End Condition: left == right
  if(left != nums.length && nums[left] == target) return left;
  return -1;
  1. Template #3 is another unique form of Binary Search. It is used to search for an element or condition which requires accessing the current index and its immediate left and right neighbor’s index in the array.
    • An alternative way to implement Binary Search
    • Search Condition needs to access element’s immediate left and right neighbors
    • Use element’s neighbors to determine if condition is met and decide whether to go left or right
    • Gurantees Search Space is at least 3 in size at each step
    • Post-processing required. Loop/Recursion ends when you have 2 elements left. Need to assess if the remaining elements meet the condition.
    • Initial Condition: left = 0, right = length-1
    • Termination: left + 1 == right
int binarySearch(int[] nums, int target) {
    if (nums == null || nums.length == 0)
        return -1;

    int left = 0, right = nums.length - 1;
    while (left + 1 < right){
        // Prevent (left + right) overflow
        int mid = left + (right - left) / 2;
        if (nums[mid] == target) {
            return mid;
        } else if (nums[mid] < target) {
            left = mid;
        } else {
            right = mid;

    // Post-processing:
    // End Condition: left + 1 == right
    if(nums[left] == target) return left;
    if(nums[right] == target) return right;
    return -1;

e.g. median of two sorted array a and b

 * Find all pairs (i, j) in sorted array `arr` such that `arr[i] + arr[j] > s`
public int countPairs(int[] arr, int s)
    int l = 0; r = arr.length-1;
    int count = 0;
    while(l < r){
        int t = arr[l] + arr[r];
        if(t >s) {
            count += r-l;
        else l++;
    return count;
 * find two numbers with sum equal to s in a sorted array.
public int[] countPairs(int[] arr, int s)
    int l = 0; r = arr.length-1;
    int count = 0;
    int[] result = {-1,-1};
    while(l < r){
        int t = arr[l] + arr[r];
        if(t == s) {
            result[0] = l;
            result[1] = r;
            return result;
        }else if(t<s){
        }else if(t>s){
    return result;

Dynamic Programming


  1. 建模
  2. 定义子问题
  3. 迭代、递归公式
  4. 边界条件

– Two eggs problem: You have two identical eggs and you are given access to a 100 story building. You would like to know the highest floor for the egg not to break when dropped. The problem is the egg might break on the 1st floor, or even the 100th floor, you just don’t know. Find the maximum number of trials you need to conduct in order to find the answer.

– You have 5 identical jars. 4 of the jars contain balls with identical size weighing 10g each, while 1 of the jar contains balls weighing 9g each. You are given a digital scale, find out the 1 jar that has the 9g balls with only 1 weighing.

Two Pointers


To iterate is human, to recurse divine (Anonymous)

solve( problem p )
    if ( p is simple )
        Solve the problem p directly
        Divide p into new sub-problems p1, p2, ...
        r1 = solve(p1)
        r2 = solve(p2)
        Reassemble r1, r2, ... to solve p


一次循环就是一轮状态转移(state transition)

初始状态 -> 按某种规则转移,有些状态属于终止状态,然后状态的某个变量就是所求

当前状态下,各个变量间互动,迁移,达到另一个状态,状态机的一般形式是 while (true) { //转移规则}, for (int i = 0; i < n; ++i ) {}这种是说,i这个状态分量每一轮的转移是固定的(++i),并且i == n 是触发状态机stop的方式之一。 状态机停止的一般形式是在转移的规则里的break或者 return

Recursively-defined functions

  • Factorial: n!
  • Fibonacci numbers
  • Ackermann Function
  • Tower of Hanoi
  • Fractals
  • Tree and data searches

tail recursion



public void print(Node root){
    print(root, "");

private void print(Node node, String ident){
    System.out.println(ident+ node.val);
    for(Node child : node.children){
        print(child, ident+" ");


圆形问题 round / circle/ rotation

  • Distributed Hashing Table
  • 数组循环
  • 队列 实现
  • Link list

Tail Recursion

The following section is exerpt from Stackoverflow.

A tail call [tail recursion] is a kind of goto dressed as a call. A tail call happens when a function calls another as its last action, so it has nothing else to do. For instance, in the following code, the call to g is a tail call:

function f (x)
  return g(x)

After f calls g, it has nothing else to do. In such situations, the program does not need to return to the calling function when the called function ends. Therefore, after the tail call, the program does not need to keep any information about the calling function in the stack. …

Because a proper tail call uses no stack space, there is no limit on the number of “nested” tail calls that a program can make. For instance, we can call the following function with any number as argument; it will never overflow the stack:

function foo (n)
  if n > 0 then return foo(n - 1) end

As I said earlier, a tail call is a kind of goto. As such, a quite useful application of proper tail calls in Lua is for programming state machines. Such applications can represent each state by a function; to change state is to go to (or to call) a specific function.

  • In traditional recursion, the typical model is that you perform your recursive calls first, and then you take the return value of the recursive call and calculate the result. In this manner, you don’t get the result of your calculation until you have returned from every recursive call.

      public int f(int n){
              return 0;
              int pre = f(n-1);
              return g(pre);
  • In tail recursion, you perform your calculations first, and then you execute the recursive call, passing the results of your current step to the next recursive step. This results in the last statement being in the form of “(return (recursive-function params))” (I think that’s the syntax for Lisp). Basically, the return value of any given recursive step is the same as the return value of the next recursive call.

      public int f(int n, int result){
              return result;
              result =  g(result, n);
              return f(n-1,result);

The consequence of this is that once you are ready to perform your next recursive step, you don’t need the current stack frame any more. This allows for some optimization. In fact, with an appropriately written compiler, you should never have a stack overflow snicker with a tail recursive call. Simply reuse the current stack frame for the next recursive step. I’m pretty sure Lisp does this.

Tail Recursion vs. Non-Tail Recursion

  • In general, (non-tail) recursive function calls put parameters on the stack
    • Every call grows the stack
    • On return, the parameters are needed to compute the result (together with the partial result returned)
  • In tail recursive functions, the parameters from the call before are not needed anymore
    • Instead, the result is directly handed to the parent
    • Hence, no parameters need to be put on the stack
  • Languages that use tail recursion optimization realize this and don’t grow the stack
  • The languages we cover next are optimized in this way So they are much more efficient when using recursion


Divide and Conquer Strategy for Problem Solving - Recursive Functions

  1. If the problem is small enough to be solved directly, do it
  2. If not, find a smaller problem and use its solution to create the solution to the larger problem


JuiceFS 服务将 云上的 OSS 存储空间 mount 到本地的目录。 当操作本地目录时, juicefs daemon 进程会监测本地目录文件变化, 实时地将本地目录上传到 OSS 空间上。 这比Dropbox 等要厉害很多。 也可以做协同开发等等。

curl -L -o juicefs && chmod +x juicefs
sudo ./juicefs mount buyuebuyue /jfs


在服务器可本地开发机器上分别: 安装 juicefs ,并将同一个 oss 上的 bucket mount 到服务器和本地。

这样服务器上的目录和本地的目录就能够保持实时同步。 同时 server 跑在远端。


Dynamic programming

Model Definition


  • 界定问题
  • 子问题
  • 递推公式
  • 边界条件
  • 复杂度


n 阶台阶,每步 1或者2阶, 有多少种走法

这是一个 Fibonacci 问题

迭代公式: f(n) = f(n-1) + f(n-2)

边界条件: f(0) = 1 f(1) =1


二维 2D 动态规划


sbt version

➜  /tmp cat /usr/local/bin/sbt
if [ -f "$HOME/.sbtconfig" ]; then
  echo "Use of ~/.sbtconfig is deprecated, please migrate global settings to /usr/local/etc/sbtopts" >&2
  . "$HOME/.sbtconfig"
exec "/usr/local/Cellar/sbt/0.13.15/libexec/bin/sbt" "$@"

While /usr/local/Cellar/sbt/0.13.15/libexec/bin/sbt is also a shell script wrapper. The real sbt exe and options are configured in file /usr/local/etc/sbtopts.

➜  /tmp cat /usr/local/etc/sbtopts
# ------------------------------------------------ #
#  The SBT Configuration file.                     #
# ------------------------------------------------ #

# Disable ANSI color codes

# Starts sbt even if the current directory contains no sbt project.

# Path to global settings/plugins directory (default: ~/.sbt)
#-sbt-dir  /etc/sbt

# Path to shared boot directory (default: ~/.sbt/boot in 0.11 series)
#-sbt-boot ~/.sbt/boot  

# Path to local Ivy repository (default: ~/.ivy2)
#-ivy ~/.ivy2

# set memory options
#-mem   <integer>  

# Use local caches for projects, no sharing.

# Put SBT in offline mode.

# Sets the SBT version to use.
#-sbt-version  0.11.3

# Scala version (default: latest release)
#-scala-home <path>        
#-scala-version <version>

# java version (default: java from PATH, currently $(java -version |& grep version))
#-java-home <path>


The real sbt jar/ bin are under path: /usr/local/Cellar/sbt

➜  sbt  tree -L 2 /usr/local/Cellar/sbt
├── 0.13.15
│   ├── INSTALL_RECEIPT.json
│   ├── bin
│   └── libexec
└── 0.13.9
    ├── INSTALL_RECEIPT.json
    ├── bin
    └── libexec

6 directories, 2 files

➜  0.13.9 tree  /usr/local/Cellar/sbt/0.13.9/ 
├── bin
│   └── sbt
└── libexec
    ├── sbt
    ├── sbt-launch-lib.bash
    └── sbt-launch.jar

2 directories, 5 files
➜  0.13.9 

If you want to use a specific sbt version for the project, configure it in project/ file.

➜  opencity-server git:(master)cat project/ 
➜  opencity-server git:(master)cat project/plugins.sbt     
addSbtPlugin("" % "sbt-plugin" % "2.6.11")
➜  opencity-server git:(master) ✗ tree -L 2 ~/.sbt/
├── 0.13
│   ├── staging
│   └── templates
├── 1.0
│   ├── server
│   ├── staging
│   └── zinc
├── boot
│   ├── sbt.boot.lock
│   ├── scala-2.10.2
│   ├── scala-2.10.4
│   ├── scala-2.10.5
│   ├── scala-2.10.6
│   ├── scala-2.11.5
│   ├── scala-2.11.7
│   ├── scala-2.11.8
│   ├── scala-2.12.4
│   └── update.log
├── launchers
│   └── sbt-launch-57d0f04f4b48b11ef7e764f4cea58dee4e806ffd.jar
├── preloaded
│   ├── com.jcraft
│   ├── com.thoughtworks.paranamer
│   ├── jline
│   ├── org.fusesource.jansi
│   ├── org.json4s
│   ├── org.scala-lang
│   ├── org.scala-lang.modules
│   ├── org.scala-sbt
│   ├── org.scala-sbt.ivy
│   ├── org.scala-tools.sbinary
│   ├── org.scalamacros
│   └── org.spire-math
└── repositories

version and dependency

There are many components of a play-framework project, you should take care of when configuring the versions.

  • play-framework
  • scala
  • sbt
  • sbt-plugin for play-framework


Firstly, you choose a play-framework verison, usally, the latest one. (the current latest is 2.6.11). Since play-framework is written in Scala, and you also need to write your application code in Scala, you have to choose a Scala version. The latest is 2.12.3. Configure it in build.sbt file.

➜  opencity-server git:(master) ✗ cat build.sbt 
name := """opencity-server"""
organization := "cn.opencity"

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.12.3"

libraryDependencies += guice
libraryDependencies += "" %% "scalatestplus-play" % "3.1.2" % Test

// Adds additional packages into Twirl
//TwirlKeys.templateImports += "cn.opencity.controllers._"

// Adds additional packages into conf/routes
// play.sbt.routes.RoutesKeys.routesImport += "cn.opencity.binders._"

build-tool versions

sbt and sbt-plugin for play-framework are used to build the project. There versions are specified under ./project/* dir configuring files.

  • issues to note:

Mill play-scala micro-services framework

Getting Started with Play 2.x with IntelliJ

Mill is a new build tool for Scala: it compiles your Scala code, packages it, runs it, and caches things to avoid doing unnecessary work. Mill aims to be better than Scala’s venerable old SBT build tool, learning from it’s mistakes and building upon ideas from functional programming to come up with a build tool that is fast, flexible, and easy to understand and use. This post will explore what makes Mill interesting to a Scala developer who is likely already using SBT

Scala and Play with Gradle

Create your own Ethereum token

Use local geth client

RPC client mode

  • Infura
  • local
    • RPC
    • IPC

Java Smart Contract Wrapper

Build and compile



Ethereum Name Service

Hello, world!

  1. Install ethereum command line tools


  • Geth: Go implementation Ethereum implementation
  • Parity:
  • Eth: The C++ implementation is simply called Eth. If you want added security by running two different implementations in parallel or are serious about GPU mining, then the C++ “Eth” client is for you.
  • Pyethapp: The Python implementation is called Pyethapp. If you are interested in understanding how Ethereum works and how to extend it, the code base of this client is probably the most readable and has a great contract tester library with fast development cycles. It is not meant for high-end usage as performance in this client is not as high priority as clarity and reliability. If you are a Python developer that wants to build decentralized apps or are interested in Ethereum for research or an academic purpose, this is a great client: we invite you to take a look and contribute to it.
  • Parity a Rust implementation by Parity Technologies
  • A Haskell implementation developed by Blockapps
  • If you are interested in developing a light application that will run entirely in a web browser, then we recommend using EthereumJS as a basis.
  • If you want to create a small hardware project, look into the implementation for the Raspberry Pi
  • If you want to install geth for non-ubuntu linux then we recommend you look into building from source
  • If you want more flexibility on the Mac, try Homebrew


Solidity is the major language to develop smart contracts on Ethereum blockchain. The documentation is at ##

Editors & IDEs

  • Remix: Browser-based IDE with integrated compiler and Solidity runtime environment without server-side components.
  • IntelliJ IDEA plugin: Solidity plugin for IntelliJ IDEA (and all other JetBrains IDEs)


decentralized storage service, like ipfs.

The swarm of Swarm is the collection of nodes of the devp2p network each of which run the bzz protocol on the same network id.


solc compiler



Whisper: decentralized messaging protocol

geth: command line tools

geth is listening on UDP/TCP port 30303

ps aux |grep geth
lucas            19806  35.5  8.0 558645868 1348204 s001  S+    4:24PM   1:45.52 geth console                                 
lucas            20355   0.0  0.0  4267768    892 s004  S+    4:30PM   0:00.00 grep geth                                      

lsof -i -a -p 19806
COMMAND   PID  USER   FD   TYPE            DEVICE SIZE/OFF NODE NAME                                                          
geth    19806 lucas   13u  IPv6 0x9295f2087a46ab9      0t0  UDP *:30303                                                       
geth    19806 lucas   16u  IPv4 0x9295f208ea01d21      0t0  TCP> (ESTABLISHED)        
geth    19806 lucas   21u  IPv6 0x9295f2082d4f599      0t0  TCP *:30303 (LISTEN)                                              
geth    19806 lucas   28u  IPv4 0x9295f208f6f6101      0t0  TCP> (ESTABLIS
geth    19806 lucas   39u  IPv4 0x9295f208ec0e101      0t0  TCP>
geth    19806 lucas   66u  IPv4 0x9295f2082332101      0t0  TCP> (SYN_SENT)             
geth    19806 lucas   75u  IPv4 0x9295f20823fed21      0t0  TCP> (ESTABLISHED)        
geth    19806 lucas   88u  IPv4 0x9295f20a183d681      0t0  TCP> (SYN_SENT)          

Ethereum Network Visualization



Operations on orders:

  • generating,
  • signing,
  • filling and
  • cancelling ,
  • verifying an orders signature,
  • setting or checking a users ERC20 token balance/allowance
  • and much more.


➜  0x.js-starter-project git:(master) ✗ yarn testrpc

yarn run v1.5.1
$ testrpc -p 8545 --networkId 50 --db ./0x_testrpc_snapshot -m "${npm_package_config_mnemonic}"

Upgrade to Android Studio 3.0

After upgrading to Android Studio 3.0, many configs changed. Please check for details.


Android Studio as an IDE often breaks because of the incompatiblity of these three components:

  • Android Studio
  • Gradle
  • Android Plugin for Gradle

Make sure they are compatible.

In latest version as 2018/02/23,

  • Android Studio v3.0.1
  • Gradle v4.1
  • Android Plugin for Gradle v3.0.1

The version for Android Studio and ` Android Plugin for Gradle` should be always the same.


implementation, api, compileOnly, and runtimeOnly.

flavor dimensions

Break Bitcoin Pow

Miners are required to do a Proof of Work (PoW) computation to find a x such that SHA256(digest+x) == 000102340... with specified number of leading zeros.

This is a less strict requirement than birthday attack which requires to find x and y such that SHA256(x) == SHA256(Y).

I guess by inspecting the internal state of SHA256 algorithm when doing a hash, one can dramatically reduce the work to find a x to satisfy PoW, which means he can mine the block much faster than others, which could be big economic incentive to optimize mining process, or, in other words, break the current bitcoin PoW mechanism.

By Using a neural network model to quick find x, can be a promising approach.



Lightening Network

## BlockStream
## Lightening Labs # Ethereum # Smart Contract # HOT/ DDEX # 0xproject # 21 wallet



Lightening Network

## BlockStream
## Lightening Labs # Ethereum # Smart Contract # HOT/ DDEX # 0xproject # 21 wallet

Python3 install

[root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 Python-3.6.4]# ./configure --enable-optimizations --prefix=/root/workspace/python3
make test
make install

[root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 python3]# export PATH=$PATH:/root/workspace/python3/Python-3.6.4:/root/workspace/python3/bin [root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 python3]# pip3 install virtualenv


basic operations

insert lookup delete

One-way hash

information theory

Hash chain

Cryptographic Hash

Universal Hash

perfect Hash



bit matrix

prime number

implementation in Java/ Python



MD5 public key

one time password

digital cash: bitcoin

Distributed hash table


IPFS is the Distributed Web

A peer-to-peer hypermedia protocol to make the web faster, safer, and more open.



Consistent Hashing


Python Pandas



index Series


Extract Transform Load pc4t

Tabular data Time series data


Python virtual enviroment 提供一个独立的 python 运行环境, 用来解决类似 Windows 上的 DLL HELL 问题。 将应用的依赖和版本等隔离开来。每个应用有自己的隔离环境。

Python 在用的版本有很多, 2.6, 2.7, 3.5, 3.6 等等。 每一个依赖也可能很多版本,中间很多兼容性问题。 如果将他们全都放到一起, 就会产生 DLL HELL 的问题。

➜  module2 l /usr/local/lib/python2.7/site-packages 
total 4392
drwxr-xr-x  95 lucas  admin   3.0K Nov 24 18:00 .
drwxr-xr-x   4 lucas  admin   128B Nov  1 20:26 ..
drwxr-xr-x   9 lucas  admin   288B Nov  3 23:25 astroid-1.5.3.dist-info
drwxr-xr-x   7 lucas  admin   224B Nov  3 23:25 backports
drwxr-xr-x   9 lucas  admin   288B Nov  3 23:25 backports.functools_lru_cache-1.4.dist-info


$ [sudo] pip install virtualenv

也可以安装源代码到本地,这样不需要 sudo 权限

$ python myVE
$ ls -lah
total 256K
drwxr-xr-x 10 lucas lucas 4.0K 2017-11-27 16:01 .
drwxr-xr-x  7 lucas lucas 4.0K 2017-11-27 16:01 ..
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 AUTHORS.txt
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 bin
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 docs
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 LICENSE.txt
-rw-r--r--  1 lucas lucas  345 2016-11-16 11:39
drwxr-xr-x  5 lucas lucas 4.0K 2017-11-27 16:01 myVE
-rw-r--r--  1 lucas lucas 3.4K 2016-11-16 11:39 PKG-INFO
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 README.rst
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 scripts
-rw-r--r--  1 lucas lucas   88 2016-11-16 11:39 setup.cfg
-rw-r--r--  1 lucas lucas 4.0K 2016-11-16 11:39
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 tests
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv.egg-info
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv_embedded
-rwxr-xr-x  1 lucas lucas  97K 2016-11-16 11:39
-rw-r--r--  1 lucas lucas  86K 2017-11-27 16:01 virtualenv.pyc
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv_support

核心文件是 当然要求 相关的类库存在。


➜  /tmp virtualenv ENV
New python executable in /private/tmp/ENV/bin/python2.7
Also creating executable in /private/tmp/ENV/bin/python
Installing setuptools, pip, wheel...done.

➜  /tmp tree -L 2 ENV
├── bin
│   ├── activate
│   ├── activate.csh
│   ├──
│   ├──
│   ├── easy_install
│   ├── easy_install-2.7
│   ├── pip
│   ├── pip2
│   ├── pip2.7
│   ├── python -> python2.7
│   ├── python-config
│   ├── python2 -> python2.7
│   ├── python2.7
│   └── wheel
├── include
│   └── python2.7 -> /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7
├── lib
│   └── python2.7
└── pip-selfcheck.json

5 directories, 15 files

  • ENV/lib/ and ENV/include/ are created, containing supporting library files for a new virtualenv python. Packages installed in this environment will live under ENV/lib/pythonX.X/site-packages/.
  • ENV/bin is created, where executables live - noticeably a new python. Thus running a script with #! /path/to/ENV/bin/python would run that script under this virtualenv’s python.
  • The crucial packages pip and setuptools are installed, which allow other packages to be easily installed to the environment. This associated pip can be run from ENV/bin/pip.


$ source bin/activate

``` $ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

$ source bin/activate

$ echo $PATH /home/xx/virtualenv-15.1.0/myVE/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games


Jupyter Notebook, 一款基于 Ipython Kernel 的 web 版 python 运行工具。 方便写 python 相关的报告。 混合 Python 运行环境和markdown 文档。




pip install jupyter


jupyter notebook




Running headless Chrome with Selenium in Python

Python Text Mining

Coursera Course: Applied Data Science with Python Specialization. Applied Text Mining in Python

Module 1: Working with Text in Python

  • Library name Minimum version
  • scikit-learn 0.18.1
  • scipy 0.19.0
  • numpy 1.12.1
  • pandas 0.20.1
  • matplotlib 2.0.1
  • seaborn 0.7.1
  • graphviz 0.7

  • String Operations
    # find unique words
    split(' ')
    t in s
  • Cleaning Text
  • Changing Text


DataFrame ###


Natural Language Processing Tookkit

Python Packaging

Python Process

Current Architecture

Disutils Setuptools Distribute Pip


  • rpm
  • dpkg
  • yum
  • apt
  • easy_install
  • pip

Python Process

Python Process

  • package multiprocessing
  • Pool map

  • Process

Exchanging objects between processes¶

  • Queue
  • Pipe

Synchronization between processes


  • Pool
  • Process
  • Exchanging Objects
    • Queue
    • Pipe
  • sync
    • Lock
  • Sharing state
    • Value
    • Array
    • Manager


Context-free Grammar

Word categories

  • noun
  • pronoun
  • verb
  • adjective
  • adverb
  • conjunction
  • preposition
  • interjection *


  • word
  • consituent phrases

CFG = Context-Free Grammar = Phrase Structure Grammar = BNF = Backus-Naur Form

Context-free grammar

G = <T,N, S,R> • T is set of terminals (lexicon) • N is set of non-terminals For NLP, we usually distinguish out a set P  N of preterminals which always rewrite as terminals. • S is start symbol (one of the nonterminals) • R is rules/productions of the form X ! , where X is a nonterminal and is a sequence of terminals and nonterminals (may be empty). • A grammar G generates a language L.

Parser combinator


  1. Design an algebra
  2. laws
  3. data types


algebra =

  • a collection of functions
  • data types
  • a set of laws over functions


  • YACC
  • algebraic design
  • how is regular expression implemented in various languages?


余华 『间奏』

余华的文字有种炫技的感觉, 但是看着又很舒服。技巧纯熟,所以不会有太做作的感觉。

记得以前在哪里看过韩寒表示读不下去余华的文字, 说是没有美感,可能他读的是「兄弟」吧。 那本小说和之前的一些小说文字确实不怎么美丽。

看了余华的一些散文, 其实文字功底是相当地强悍。 之前『活着』之类的作品文字上的直白,怕是有意为之。

回到这部作品, 写的是余华对音乐的一些感悟。 而感悟的点呢,当然有作家敏感的人生和生活等,但最主要的比较还是对文学。

比如对苏联作曲家肖斯塔科维奇的『第七交响曲』(列宁格勒交响曲),余华就把它同美国小说家霍桑小说『红字』做了细致的对比。本来是两个领域的两个东西, 没有可比性。 但是在情感上引起的共鸣又是那么的相似。 让我想起大一时老师给看的『Fantasy 2000』系列动画音乐作品。 本是一两百年前的大师古典乐,迪士尼为其配了动画故事。 结果看起来却像是200年前的大师为迪士尼动画配的乐。 这说明很多艺术所触发的情感本是相同。

余华听音乐时,第一层是听音,声音本身的旋律很好听。 第二层听故事, 旋律的跳动,影响这思考的节奏, 古典乐总是在叙说着一段故事。 第三层听人生,对比自己在文学和其他方面的感受, 有时是如此相似。


  • 叙述: 勃拉姆斯
  • 高潮: 肖斯塔科维奇和霍桑
  • 灵感: 莫扎特
  • 否定: 柏辽兹
  • 色彩: 拉赫玛尼诺夫, 绘画
  • 字与音: 门德尔松





Python Thread

Python Thread

  • __init__()
  • run()
  • start()


Python Optimize


最近使用 Python做了一个日志转换的小项目。原生 Python,没有使用任何的Framework. 写了约两周。 越写越顺手。 发现 Python 的整个设计比较人性化,没有太多的冷知识和需要记忆的。

项目逻辑比较简单。 一句话解释就是对日志格式进行转换。

原始日志(gzip/8.5M/300K lines), 22个/min
每行日志进行目标格式的转换,生成新的日志 (需要生成2种格式的日志,即两份)
每5分钟或者700M(压缩前大小, 压缩后大概100M) 滚动
新的日志 (gzip/100M/5M lines), 

对于单个原始文件, 最终优化到10s/个的处理速度。

最开始是需要12分钟的速度, 经过多次优化,达到现在的速度。

优化1: Python 字符串处理

字符串处理。 Python 本身的字符串处理速度是比较慢的。 每行进行格式转换的算法。

v1: string.split()

最开始是使用字符串空格切分,得到每个字段。由于每一行的字段数目不是完全一致,切分是分多次进行。 非常慢。

v2: regex

花了一个晚上的时间, 使用 regex 一次性获取所有字段,regex 只需要编译一次。 速度提升非常快。

优化2: bitarray

生成新格式其中一个需求是针对的某些符合需求的日志做替换。 如果日志中的 IP 不在给定的 IP 网段集合中, 则需要使用备用库中的 IP 替换原始 IP。 这涉及到一个判定 IP 在不在给定 IP 网段集合的算法。 IP 网段的形式是 (约有3000个网段。

v1: 直接循环判定

使用 python ipaddress library:

    for line in lines:
        subnet = ipaddress.ip_network(line)

    def ip_in_localnet_raw(self, ip):
        ''' deprecated raw method, which is slow. use  ip_in_localnet
        address = ipaddress.ip_address(unicode(ip))
        for subnet in self.localnet:
            if address in subnet:
                return True
        return False

这个算法当中, 每次 IP 都需要循环3000/2次判定在不在给定 IP 网段集合中。时间非常慢。

v2: bloomfilter

对于快速判定某元素在不在指定集合中, bloomfilter 是非常好的结构。 直接使用 bloomfilter, 将 IP 网段集合转换成 IP 集合。 但是3000个网段,对应的 IP 数量非常庞大,bloom filter 也非常大。

v3: bitarray

IP 数据有很好的特征。 IPV4 实际是32bit unsigned integer. 如果直接将一个 IP 在不在用一个 bit 表示,则需要2^32bit =512MB 的内存。 对于服务器来说, 可以接受。 当然可以直接使用一个 int array 。 但 Python 有现成的 library 来使用: bitarray

        self.ips = bitarray(2**32)

    def __load_localnet(self):
        self.localnet = []
        starttime =  
        count = 0
        with open(self.localnet_iplist_file, 'r') as f:
            for line in f:
                line = unicode(line.strip())
                count +=1
                    subnet = ipaddress.ip_network(line)
                    start = int(subnet.network_address)
                    end = start + subnet.num_addresses
                    for x in range(start, end):
                        self.ips[x] = 1
                except Exception as e:
        endtime = - starttime"GroupLog loading %s localnets done, takes %s" % (count, endtime))          

上面这个函数,将所有的 IP 网段转换成 IP 集合, 存到 ips[] bitarray 当中。 加载需要24s 左右。

v4: 加载速度优化

                    # for x in range(start, end):
                    #     self.ips[x] = 1
                    # This is far more faster than add one by one.
                    self.ips[start:end] = 1
GroupLog loading 2128 localnets done, takes 0:00:24.227503 (set one by one)
GroupLog loading 2128 localnets done, takes 0:00:00.214876 (set by [start:end])
benchmark: call 300000 times in 0:00:02.242983

经过多次优化, 加载和判定速度得到极大优化。

Python 多线程并不能利用多核

Threads are useful when the code is blocked by non bytecode execution, such as I/O or external process execution (C code, system calls, etc). If byte code execution is holding things up, the ProcessPool starts multiple interpreters that can execute in parallel. However, there is more overhead in spinning up these interpreters and in them communicating with the main thread through serialized representations (basically pickle or json over a socket if I understand correctly).


  • threading
  • thread


  • Pool
  • Process
  • Exchanging Objects
    • Queue
    • Pipe
  • sync
    • Lock
  • Sharing state
    • Value
    • Array
    • Manager


Global Interpreter Lock sub-process v.s. thread


  • ThreadPoolExecutor
  • ProcessdPoolExecutor

由于 GIL 的限制, python 的 multi-thread 并不能在多核上运行。 Python 的 multi-thread 虽然将运行线程化了,但是只是在单核上运行。 这样, 对 cpu-intensive 的计算没有意义,对于 I/O 类型的运算才有意义。


  • **kwargs
  • How does OS pip work?
  • multiprocessing.Pool v.s. concurrent.futures.ProcessPoolExecutor



interpreter instructions

import dis

str is text representation in bytes, unicode is text representation in characters.

Python Modules, Classes, Packages

Python Modules, Classes, Packages


  • A module is a file containing Python definitions and statements. The file name is the module name with the suffix ‘.py’ appended. Within a module, the module’s name (as a string) is available as the value of the global variable __name__.
import fibo
from fibo import fib, fib2
from fibo import *
  • symbol table

  • A module can contain executable statements as well as function definitions. These statements are intended to initialize the module. They are executed only the first time the module is imported somewhere.

  • When a module named ‘spam’ is imported, the interpreter searches for a file named ‘’ in the current directory, and then in the list of directories specified by the environment variable ‘PYTHONPATH’. This has the same syntax as the shell variable ‘PATH’, that is, a list of directory names. When ‘PYTHONPATH’ is not set, or when the file is not found there, the search continues in an installation-dependent default path; on UNIX, this is usually ‘.:/usr/local/lib/python’.
  • sys.path = the directory containing the input script (or the current directory), + ‘PYTHONPATH’ + the installation-dependent default.

  • dir() find out which names a module defines.
  • standard module __builtins__
# -*- coding: utf8 -*-
import os, sys

# demo module usage

def f():
    print "function f in module ''"
print "__name__ = ", __name__
print "os.environ['PYTHONPATH'] = ", os.environ.get("PYTHONPATH", '')
print "sys.path = ", sys.path
print "dir() = ", dir()
print "__builtins__ = ", __builtins__
print "dir(__builtins__) = ", dir(__builtins__)


➜  python101 python /Users/lucas/dev/workspace/python/python101/
__name__ =  __main__
os.environ['PYTHONPATH'] =
sys.path =  ['/Users/lucas/dev/workspace/python/python101', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages', '/usr/local/Cellar/protobuf/2.6.1/libexec/lib/python2.7/site-packages']
dir() =  ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'f', 'os', 'sys']
__builtins__ =  <module '__builtin__' (built-in)>
dir(__builtins__) =  ['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BufferError', 'BytesWarning', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError', 'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs', 'all', 'any', 'apply', 'basestring', 'bin', 'bool', 'buffer', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']


  • Packages are a way of structuring Python’s module namespace by using “dotted module names”.
  • The files are required to make Python treat the directories as containing packages;
  • __all__ = ["echo", "surround", "reverse"]
  • Intra-package References
  • __path__. This is initialized to be a list containing the name of the directory holding the package’s before the code in that file is executed.


# -*- coding: utf8 -*-

class Foo:
    def __init__(self, val):
        self.val = val
    def printVal(self):

foo = Foo(5)
  • __init__
      def __init__(self, val): # constructor
          self.val = val
  • self this me, etc.
  • methods
  • variables: Python creates variables on the fly, when they are first assigned, and class member variables are no different.
          self.val = val
  • instance we can change the definition of a class on the fly, or even create a completely new class at run-time!
      Foo.printVal = printVal
      del Foo.printVal
      MyNewClass = type('MyNewClass', (object,), dict())
  • standard methods:
    • __del__: Called when an instance is about to be destroyed, which lets you do any clean-up e.g. closing file handles or database connections
    • __repr__ and __str__: Both return a string representation of the object, but __repr__ should return a Python expression that can be used to re-create the object. The more commonly used one is __str__, which can return anything.
    • __cmp__: Called to compare the object with another object. Note that this is only used with Python 2.x. In Python 3.x, only rich comparison methods are used. Such as __lt__.
    • __lt__, __le__, __eq__, __ne__, __gt__ and __ge__: Called to compare the object with another object. These will be called if defined, otherwise Python will fall-back to using __cmp__.
    • __hash__: Called to calculate a hash for the object, which is used for placing objects in data structures such as sets and dictionaries.
    • __call__: Lets an object be “called” e.g. so that you can write things like this: obj(arg1,arg2,…).
    • Python also lets you define methods that let an object act like an array (so you can write things like this: obj[2] = "foo"), or like a numeric type (so you write things like this: print(obj1 + 3*obj2).
    • __dict__ − Dictionary containing the class’s namespace.
    • __doc__ − Class documentation string or none, if undefined.
    • __name__ − Class name.
    • __module__ − Module name in which the class is defined. This attribute is “__main__” in interactive mode.
    • __bases__ − A possibly empty tuple containing the base classes, in the order of their occurrence in the base class list.
  • In Python, methods and member variables are always public i.e. anyone can access them.
  • private methods and variables: Class methods and variable names that start with two underscores are considered to be private to that class, However, this is only a convention

  • Class Inheritance
      class Foo:
          def __init__(self, val):
              self.val = val
          def printVal(self):
      class DerivedFoo(Foo):
          def negateVal(self):
              self.val = -self.val
  • isinstance
  • issubclass
  • Class Iterators and Generators
      class Backwards:
          def __init__(self, val):
              self.val = val
              self.pos = len(val)
          def __iter__(self):
              return self
          def next(self):
              # We're done
              if self.pos <= 0:
                  raise StopIteration
              self.pos = self.pos - 1
              return self.val[self.pos]
      for x in Backwards([1,2,3]):
  • generator yield

  • class variable
  • instance variable
  • function overloading
  • inheritance

  • data memebers
  • documentation
      class Employee:
          'Common base class for all employees'
          empCount = 0
          def __init__(self, name, salary):
     = name
              self.salary = salary
              Employee.empCount += 1
          def displayCount(self):
              print "Total Employee %d" % Employee.empCount
          def displayEmployee(self):
              print "Name : ",,  ", Salary: ", self.salary

    The class has a documentation string, which can be accessed via ClassName.doc.

  • The getattr(obj, name[, default]) − to access the attribute of object.

  • The hasattr(obj,name) − to check if an attribute exists or not.

  • The setattr(obj,name,value) − to set an attribute. If attribute does not exist, then it would be created.

  • The delattr(obj, name) − to delete an attribute.

  • Destroying Objects (Garbage Collection)
         def __del__(self):


how to import module in parent directory?

check How does python find packages?

Python imports work by searching the directories listed in sys.path.

> import sys
> print '\n'.join(sys.path)


sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

Find module

> import imp
> imp.find_module('numpy')
(None, '/usr/local/lib/python2.7/dist-packages/numpy', ('', '', 5))


Python logging

Pythong logging module


  • Loggers expose the interface that application code directly uses.
  • Handlers send the log records (created by loggers) to the appropriate destination.
  • Filters provide a finer grained facility for determining which log records to output.
  • Formatters specify the layout of log records in the final output.
  • LogRecord



Python Logging

import logging

logger = logging.getLogger(__name__)

# create a file handler
handler = logging.FileHandler('hello.log')

# create a logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# add the handlers to the logger
logger.addHandler(handler)'Hello baby')


logger names track the package/module hierarchy, and it’s intuitively obvious where events are logged just from the logger name.

  • Logger.setLevel() specifies the lowest-severity log message a logger will handle, where debug is the lowest built-in severity level and critical is the highest built-in severity. For example, if the severity level is INFO, the logger will handle only INFO, WARNING, ERROR, and CRITICAL messages and will ignore DEBUG messages.
  • Logger.addHandler() and Logger.removeHandler() add and remove handler objects from the logger object. Handlers are covered in more detail in Handlers.
  • Logger.addFilter() and Logger.removeFilter() add and remove filter objects from the logger object. Filters are covered in more detail in Filter Objects.



three ways:

  • Creating loggers, handlers, and formatters explicitly using Python code that calls the configuration methods listed above.

  • Creating a logging config file and reading it using the fileConfig() function.
  • Creating a dictionary of configuration information and passing it to the dictConfig() function.

Use rotating file handler

Level/ Severity

debug(), info(), warning(), error() and critical().


  • class Handler()
  • class logging.StreamHandler(stream=None)
  • class logging.NullHandler
  • class logging.FileHandler(filename, mode='a', encoding=None, delay=False)
  • class logging.handlers.WatchedFileHandler(filename[, mode[, encoding[, delay]]])
  • class logging.handlers.RotatingFileHandler(filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0)
    • doRollover()
  • class logging.handlers.TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False)
  • class logging.handlers.SocketHandler(host, port)
  • class logging.handlers.DatagramHandler(host, port)
  • class logging.handlers.SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER, socktype=socket.SOCK_DGRAM)
  • class logging.handlers.NTEventLogHandler(appname, dllname=None, logtype='Application')
  • class logging.handlers.SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None)
  • class logging.handlers.BufferingHandler(capacity)
  • class logging.handlers.MemoryHandler(capacity, flushLevel=ERROR, target=None)
  • class logging.handlers.HTTPHandler(host, url, method='GET')


  • Logging variable data:
      logging.warning('%s before you %s', 'Look', 'leap!')
      from string import Template
      s = Template('$who likes $what')
      s.substitute(who='tim', what='kung pao')


  1. what is python module and __name__ variable? if the python interpreter is running that module (the source file) as the main program, it sets the special __name__ variable to have a value “__main__”. If this file is being imported from another module, __name__ will be set to the module’s name. A module’s __name__ is set equal to ‘__main__’ when read from standard input, a script, or from an interactive prompt. A module can discover whether or not it is running in the main scope by checking its own __name__, which allows a common idiom for conditionally executing code in a module when it is run as a script or with python -m but not when it is imported:

For a package, the same effect can be achieved by including a module, the contents of which will be executed when the module is run with -m.


Scrapy Learned



  • engine
  • spider
  • scheduler
  • downloader
  • item pipelines


  • crawl: walk the links
  • parser: parse the webpage html into structured data
  • spider: define how crawler and parser work
  • scraper: extract data from within a webpage.

A crawler gets web pages – i.e., given a starting address (or set of starting addresses) and some conditions (e.g., how many links deep to go, types of files to ignore) it downloads whatever is linked to from the starting point(s).

A scraper takes pages that have been downloaded [Edit: or, in a more general sense, data that’s formatted for display], and (attempts to) extract data from those pages, so that it can (for example) be stored in a database and manipulated as desired.

parse(self, response)


  • extract dict {}
  • Item
  • Request with parse() defined.


grammar: response.css(selector) or response.xpath(selector)

  • property: a ::text
  • child level: div.pre_post > a
  • class: div.pre_post
  • id: div#id


  • extract_first() css selection results are array, return the first element of the results, even though there are only one element in the results.

Scrapy Commands

  • scrapyd-deploy
  • scrapy.cfg

Global commands

  • startproject
  • genspider
  • settings
  • runspider
  • shell
  • fetch
  • view
  • version

Project commands

  • crawl
  • check
  • list
  • edit
  • parse
  • bench

project structure


Scrapy Shell


  • response.follow
  • yield result or page



The selection methods (.xpath() or .css()) return a list of selectors of the same type, so you can call the selection methods for those selectors too.

  • .xpath('//title/text()')
  • .css('title::text')
  • .re('a*')

In [20]: type(response.selector)
Out[20]: scrapy.selector.unified.Selector

In [21]: type(response.selector.xpath('//title/text()'))
Out[21]: scrapy.selector.unified.SelectorList

In [22]: type(response.selector.xpath('//title/text()')[0])
Out[22]: scrapy.selector.unified.Selector

In [23]: 
>>> response.xpath('//title/text()')
[<Selector (text) xpath=//title/text()>]
>>> response.css('title::text')
[<Selector (text) xpath=//title/text()>]
  • extract()
  • extract_first()

Relative Xpath

  • .// relative
  • // absolute



Scrapy Shell

Item pipeline

Feed Exports

Serialization formats Item exporters

  • JSON
  • JSON lines
  • CSV
  • XML

Requests and Responses

Downloader Middleware


    'myproject.middlewares.CustomDownloaderMiddleware': 543,


Engine -> mw1 -> mw2 -> mw3 -> ->   -> downloader

process_request(request, spider)

process_response(request, response, spider)

## Sitemap

  python curl
<?xml version="1.0" encoding="UTF-8"?>
	<?xml-stylesheet type="text/xsl" href="gss.xsl"?>
	<sitemapindex xmlns:xsi="" xsi:schemaLocation="" xmlns="">
curl |gzip -d  |less
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="" xmlns:mobile="">
 for index, link in enumerate(links):


Hello world

  1. import: <script src=""></script>
  2. HTML ```html
3. javascript

var app = new Vue({
  el: '#app',
  data: {
    message: 'hello, world'


HTML 中用一个 mount point <div id="app">, 在 javascript 创建一个 Vue 示例。


  1. 文本插值:
  2. v-bind::title e.g. ` `
  3. v-on: @ 事件处理 ` `
  4. 条件: v-if
  5. 循环: v-for
  6. 用户输入: v-model ` `


// 定义名为 todo-item 的新组件
Vue.component('todo-item', {
  template: '<li>这是个待办项</li>'


Vue class / instance

var vm = new Vue({



  1. el: 锚点
  2. data: 要绑定的数据
  3. computed: 复杂属性
  4. props: 对外的属性接口,类似 C# /Android 中 view 等属性。
  5. methods: 方法


global config

  • silent
  • optionMergeStrategies
  • devtools
  • errorHandler
  • warnHandler
  • ignoredElements
  • keyCodes
  • performance
  • productionTip

Global API

  • Vue.extend(options)
  • Vue.nextTick([callback, context])
  • Vue.set(target, key, value)
  • Vue.delete(target, key)
  • Vue.directive(id, [definition])
  • Vue.filter(id, [definition])
  • Vue.component(id, [definition])
  • Vue.use(plugin)
  • Vue.mixin(mixin)
  • Vue.compile(template)
  • Vue.version


  • data
  • props: A list/hash of attributes that are exposed to accept data from the parent component.
  • propsData: initial / default data for props
  • computed:


  1. what is javascript prototype?


  2. what is javascript context this?

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

javascript context/ scope/ this

  • context/ this: function

context means this keyword

Context is most often determined by how a function is invoked. When a function is called as a method of an object, this is set to the object the method is called on:

So for a global function, this refers to ‘window’ in browser

  • scope: variable
    • local
    • Global
    • block: let/ const

    Understanding Scope and Context in JavaScript

  • Execution Context

  • Execution stack

    The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

The event model for GUI application

Compare javascript event model and Android event model.

Tree structure of the GUI elements architecture.

The root window









Event propagation.

Animation framework


  1. $.each() v.s. $(“.cls”).each()
  2. document, window, $, jQuery()
  3. $() $(document).ready(function(){})
  4. $.fn.method, $.method
  5. $( document ).ready(), window.onload()
  6. $().call(“param”, callback), $().call(“param”, function(){ callback(param1, param2)}),
  7. Avoiding conflicting with other libs
     <!-- Putting jQuery into no-conflict mode. -->
     <script src="prototype.js"></script>
     <script src="jquery.js"></script>
     var $j = jQuery.noConflict();
     // $j is now an alias to the jQuery function; creating the new alias is optional.
     $j(document).ready(function() {
         $j( "div" ).hide();
     // The $ variable now has the prototype meaning, which is a shortcut for
     // document.getElementById(). mainDiv below is a DOM element, not a jQuery object.
     window.onload = function() {
         var mainDiv = $( "main" );
  8. Attributes
    • setter: .attr(key, value)
    • getter: .attr(key)
  9. Selecting Elements
    • Selecting Elements by ID $( "#myId" ); // Note IDs must be unique per page. * Selecting Elements by Class Name $( “.myClass” ); link Selecting Elements by Attribute 1 $( “input[name=’first_name’]” ); link Selecting Elements by Compound CSS Selector 1 $( “#contents ul.people li” ); link Selecting Elements with a Comma-separated List of Selectors 1 $( “div.myClass, ul.people” );

The jQuery object and the native DOM element. jQuery element is like a wrapper on a native DOM element, so you can use a lot of convinience methods and properties provided by jQuery. This is a bit like design pattern facet.

 var jQElem = $(domElem)
 var element = $(this)

Rapoo Mechanical Keyboard

Rapoo V500s I really cannot bear the awkward of the Mac Pro Keyboard for long time. Yesterday I bought this the Rapoo V500s Mechanical Keyboard. It is pretty shinning and easy to use. The most importantly, it is cost-effectve and simple to perform. The sound of percussion the keyboard is crisp and rhythmic.

Keyboard shortcuts and combo keys

FN+ key Usage
  W exchange the function of WSAD and
  SCROLL Backlight Switch mode
  F1 Homepage
  F2 Email
  F3 Calculator
  F4 Search
  F5 Player
  F6 Previous Music
  F7 Next Music
  F8 Play/Pause Music
  F9 Halt Key
  F10 Mute Key
  F11 Volume-
  F12 Volume+
  Backlights Brightness-
  Backlights Brightness+

Tech Positions in IT companies


Principle Dev Software


Fix Vim

One day I updated my MacOS to Sierra, it broke a lot of toolchain on my system.

And the most importantly, VIM is broken.

➜  _posts git:(master) ✗ vim
dyld: Library not loaded: /usr/local/opt/ruby/lib/libruby.2.3.0.dylib
  Referenced from: /usr/local/bin/vim
  Reason: image not found
[1]    6993 abort      vim

Run brew doctor to pinpoint the issue.

➜  _posts git:(master) ✗ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!

Warning: Some installed formula are missing dependencies.
You should `brew install` the missing dependencies:
  brew install libev libsodium udns

Run `brew missing` for more details.
➜  _posts git:(master) ✗ brew install libev libsodium udns
Updating Homebrew...

➜  _posts git:(master) ✗ brew upgrade vim
==> Upgrading 1 outdated package, with result:
vim 8.0.0495

➜  _posts git:(master) ✗ which vim
➜  _posts git:(master) ✗ vim
➜  _posts git:(master) ✗

Vim got fixed!

SIP System Integration Protection

sudo gem install bundler
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin15/rbconfig.rb:213: warning: Insecure world writable dir /Users/lucas/dev/android/dex2jar/dex2jar- in PATH, mode 040777
Fetching: bundler-1.13.1.gem (100%)
ERROR:  While executing gem ... (Errno::EPERM)
   Operation not permitted - /usr/bin/bundle

Google it find

After adding option -n /usr/local/bin, it works.

sudo gem install bundler -n /usr/local/bin
Successfully installed bundler-1.13.1
Parsing documentation for bundler-1.13.1
Installing ri documentation for bundler-1.13.1
1 gem installed

  • You can’t modify anything in /System, /bin, /sbin, or /usr (except /usr/local); or any of the built-in apps and utilities. Only Installer and software update can modify these areas, and even they only do it when installing Apple-signed packages. But since normal OS X-style customizations go in /Library (or ~/Library, or /Applications), and unix-style customizations (e.g. Homebrew) go in /usr/local (or sometimes /etc or /opt), this shouldn’t be a big deal. It also prevents block-level writes to the startup disk, so you can’t bypass it that way.

The full list of restricted directories (and exceptions like /usr/local and a few others) is in /System/Library/Sandbox/rootless.conf. Of course, this file is itself in a restricted area.

When you upgrade to El Capitan, it moves any “unauthorized” files from restricted areas to /Library/SystemMigration/History/Migration-(some UUID)/QuarantineRoot/.

  • You can’t attach to system processes (e.g. those running from those system locations) for things like debugging (or changing what dynamic libraries they load, or some other things). Again, not too much of a big deal; developers can still debug their own programs.

This does block some significant things like injecting code into the built-in Apple apps (notably the Finder). It also means that dtrace-based tools for system monitoring (e.g. opensnoop) will not be able to monitor & report on many system processes.

  • You can’t load kernel extensions (kexts) unless they’re properly signed (i.e. by Apple or an Apple-approved developer). Note that this replaces the old system for enforcing kext signing (and the old ways of bypassing it). But since v10.10.4 Apple has had a way to enable trim support for third-party SSDs, the #1 reason to use unsigned kexts has gone away.


Suffix Tree 后缀树


字符串模式匹配问题定义: -输入: 一个字符串模式 Pattern, 一个字符串文本 Text. –输出: 模式Pattern 作为子串出现在文本 Text 中的所有位置.


近似字符串模式匹配问题定义: -输入: 一个字符串模式 Pattern, 一个字符串文本 Text,正整数 d. –输出: 与模式 Pattern 距离最多为 d 的所有子串出现在文本 Text 中的所有位置.

Readings for Year 2016

Previous reading for Year 2013 & 2014:


  1. 清明上河图密码 : 隐藏在千古名画中的阴谋与杀局 by 冶文彪
  2. 清明上河图密码2 : 隐藏在千古名画中的阴谋与杀局 by 冶文彪
  3. 清明上河图密码3 : 隐藏在千古名画中的阴谋与杀局 by 冶文彪
  4. 禅茶濯剑录 by 诸相非
  5. 北京折叠 by 郝景芳
  6. 边城 by 沈从文
  7. 罗生门 by 〔日〕芥川龙之介 1.巴黎圣母院 by 〔法〕维克多·雨果 1. by 1. by 1. by

Loading resources from other Android applications.

This post tells you how to “steal” data and code from other Android applications at Runtime.

At the core of the trick is the method called “Context.createPackageContext”. Use this method to create a context for the applications you want to retrieve data from.

After building up a Context for the other application, get the “Resources” object by “context.getResources”, then with the new Resources object, you can do whatever you want. Check the following example:

This example shows how to retrieve all string values from the famous app “WeChat”.

private void testCreatePackageContext() {
    try {
        String packageName = "";
        Context c = createPackageContext(packageName,
        Resources otherR = c.getResources();

        Log.i(TAG, "classloader=" + this.getClassLoader());

        Log.i(TAG, "classloader=" + c.getClassLoader());

        Class RClass = Class.forName(packageName + ".a", true, c.getClassLoader());
        Class[] RTypes = RClass.getDeclaredClasses();
        for (Class RType : RTypes) {
            Field[] fields = RType.getDeclaredFields();
            for (Field f : fields) {
                try {
                    int id = f.getInt(null);
                    String formatStr = String.format("%s, %s=0x%x", otherR.getResourceName(id), f, id);
                    if ("string".equalsIgnoreCase(otherR.getResourceTypeName(id))) {
                        formatStr = String.format("%s, %s=0x%x, value=%s", otherR.getResourceName(id), f, id, otherR.getString(id));
                        Log.i(TAG, formatStr);
                } catch (Exception e) {
    } catch (PackageManager.NameNotFoundException e) {
    } catch (Exception e) {



18906   android.lucas.swip..  I  classloader=dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/android.lucas.swipeback.demo-2/base.apk"],nativeLibraryDirectories=[/vendor/lib, /system/lib]]]
18906   android.lucas.swip..  I  classloader=dalvik.system.PathClassLoader[DexPathList[[zip file "/system/framework/", zip file "/data/app/com.facebook.katana-1/base.apk"],nativeLibraryDirectories=[/data/app/com.facebook.katana-1/lib/arm, /vendor/lib, /system/lib]]]

As the above log shows, 2 classloaders were built, one for the current application, and the other for “WeChat” application.

String Resources

The following log prints out all string resources contained in “WeChat” package. Similarly, you can list all other major types of resources, such as layouts, drawables, raw assets, etc.

28058   android.lucas.swip..  I, public static final int$m.dtP=0x7f091187, value=Reminder: You can only link your own bank cards.
28058   android.lucas.swip..  I, public static final int$m.dtQ=0x7f091153, value=Last 4 digits
28058   android.lucas.swip..  I, public static final int$m.dtR=0x7f091186, value=Your information will be encrypted
28058   android.lucas.swip..  I, public static final int$m.dtS=0x7f091177, value=Add Now
28058   android.lucas.swip..  I, public static final int$m.dtT=0x7f091196, value=You've already linked this card on WeChat Payment
28058   android.lucas.swip..  I, public static final int$m.dtU=0x7f091176, value=Open Now
28058   android.lucas.swip..  I, public static final int$m.dtV=0x7f091156, value=Registered with bank
28058   android.lucas.swip..  I, public static final int$m.dtW=0x7f091189, value=Cardholder Description
28058   android.lucas.swip..  I, public static final int$m.dtX=0x7f09114c, value=Enter bank card number
28058   android.lucas.swip..  I, public static final int$m.dtY=0x7f09118d, value=Phone Number
28058   android.lucas.swip..  I, public static final int$m.dtZ=0x7f091191, value=Pay via both payment password and SMS verification
28058   android.lucas.swip..  I, public static final int$m.dta=0x7f0912b7, value=Details
28058   android.lucas.swip..  I, public static final int$m.dtb=0x7f0912b8, value=Top-up successful
28058   android.lucas.swip..  I, public static final int$m.dtc=0x7f0912b6, value=Details
28058   android.lucas.swip..  I, public static final int$m.dtd=0x7f0912aa, value=Amount (%s)
28058   android.lucas.swip..  I, public static final int$m.dte=0x7f0912ac, value=Input a correct amount
28058   android.lucas.swip..  I, public static final int$m.dtf=0x7f0912a9, value=Balance Top-up
28058   android.lucas.swip..  I, public static final int$m.dtg=0x7f0912bf, value=Select a bank card
28058   android.lucas.swip..  I, public static final int$m.dth=0x7f0911fe, value=The bank's system is currently undergoing maintena

Publish Android Library

Like you, as an Android developer that has benefitted from Github open source libraries a lot for long time, I start thinking how to repay this community. The best way, of course, is to share my contribution by publishing android library code. This post tells what to follow when you want to publish your Android library project on Github.

Github & Git

The first of the first, you should take some time to get familiar with Git and Github.

Gradle & Android Stuido

Familiarize yourself with Gradle and Android Studio.

Choose a License

Choose a license for your project. If you do not know how to, check this page:

“Apache License 2.0” is common to Android open source libraries on Github.

Add a readme

Add “” file to tell people what the project is about and how to use it.


When you are done with all the library code and Android Studio / gradle project code stuff, it is time to share it.

Android now jcenter uses by default. It is a good idea to share your library by jcenter too so that other Android developers and use your library by just adding gradle dependency, such as:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile ''
    compile 'android.lucas:swipeback:0.0.1'

There is no direct support in Android Studio or gradle now and it is a bit tricky to set up.

The post [TUT] Locally release an Android Library for JCenter or Maven Central inclusion is a good start to accomplish the task. To simplify the life, you can customize your library project build.gradle file based on the following template:

Then publish with command:

./gradew bintrayUpload

Github page

Give it a dedicated website or webpage makes your open source library project look professional. Github provides “Github Pages” feature so that you can create a page for your project very easily.

Project->Setting -> Options -> Github Pages -> Automatic Page Generator

Then follow the instructions to create a webpage for your project.

Share it on Twitter, G+, Facebook, etc

Finally, after all these done, share it to all social platforms to let people find it.

Git 多账号

git 账号

git 用户经常混淆用户和ssh key 等概念。git码农们经常泡在github,bitbucket 等网站,一方面需要给公司写代码,另外一方面不干寂寞的写开源代码,需要两个Email账户。一个是个人的(personal use),用户在github等网站使用, 比如, 另一个是工作用公司邮箱, 比如叫 在commit 时候如何区分开来呢?

  • 对于个人项目, 可以在项目下,使用 git config 添加email
  • 对于公司项目, 可以使用 git config

但是如果每一个项目都这样配置, 又太麻烦了。 如果公司项目比较多, 可以将 加到git global 配置中

git config —global

相反, 如果个人项目比较多, 可以

git config —global

使用git log 可以看到使用哪个账户commit的:

commit 977277ecc9ebccb2ddb93b99e69d71efbce9e7e3
Author: lucas <>
Date:   Fri Feb 27 13:47:57 2015 +0800

    inital project setup

commit 1a82f9456f86dca4d284a56024834ebfb6f26c4a
Author: lucas <>
Date:   Fri Feb 27 13:37:57 2015 +0800

    Initial commit

同样,email前面的名字也同理可以配置。 对于—global的配置, git 存储在~/.git/config 当中 对于项目的配置, git 存储在 ./.git/config 项目multi下

➜  SwipeBack git:(master) less .git/config

        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
        ignorecase = true
        precomposeunicode = true
[remote "origin"]
        url =
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
        email =
.git/config (END)

这样配置项目使用的user.email和 就能为公司项目和个人项目区分开用户账户了。

关于git 和 ssh key

ssh key 和git 本地使用没有关系。 如果需要同服务器同步代码, 才会用到。 git通信协议底层使用ssh, 故需要ssh key。 公司用的ssh key 多是用公司邮箱生成的, 如 , 同公司代码库通信或者登陆公司服务器,都使用此key。 ssh key 默认存储在~/.ssh/目录下,如果将公司和个人区分开来呢?

简单回答是不需要, 只用将pub key 加到github 上就可以了, 这个pub key 是可以公开的, 故将公司用的ssh key pubkey 放到 也无妨。 SSH key 和git 用户是两个独立的概念, 不应该混淆在一起 但是如果说一定要区分呢? 可以参考

在~/.ssh/config 中为每一个ssh server 使用IdentityFile 参数配置好private key.

Step 2: ssh config Set up multiple ssh profiles by creating/modifying ~/.ssh/config. Note the slightly differing ‘Host’ values:

# Default GitHub
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa

# Work GitHub
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa_work

Android SwipeBack pattern


It has been a widely used pattern by swiping from the left edge of the screen to return to the previous page or screen. iOS Safari presents such a feature. This navigation pattern is natural and convient, and it would be great to Android developers and users if it can be ported to Android platform. Take a look for the illustration.

<img src=”” width=200 />

The target of this project is to achieve the effect shown by iOS Safari, on Android Activity transition, like native Activity transition animation shown. Fragment transition animation is easy to implement by using ViewPager etc. and thus not discussed here.

Android implementation

how to: If the user swipes into the screen from the outside of the left edge, the previous page slides in from the left edge while the current page slides out to the right edge of the screen.

It looks like this: <img src=”” width=200px />


To make it more illusive, we can slide the previous page from 75% X-axis, that is, the previous page has already slided in 75% percent when the finder touches the left edge, while the content covered by the current page.

The pace of the previous page is 25% of the current page. So when the current page slides from the left to the right by 100%, the underlying previous page just 25%, which makes it 100% filling the screen perfectly.

This illusiveness is what Safari on iOS does.

To implement the effect on Android, we have to solve the following obstacles:

  1. Gesture detection
  2. To show part of the Activity / Page on the screen.

To show the animation, we can use the native animation support for the activity but controls it manually.

The most difficult part is to show 2 Activities simultaneously on the screen.

The solutions I found online are all workarounds by using Fragment.

The open source github projects sockeqwe/SwipeBack and ikew0ng/SwipeBackLayout replaces the decor view and show the swipe gesture, but cannot show the previous Activity correctly.


  1. Use GestureDetector to detect the touch event on Activity and set the root decorView of the Activity to the moved position.
  2. override Activity’s onTouchEvent method.
  3. Set the Activity theme to Translucent. ‘’

API Design

call android.lucas.swipeback.SwipeBack.enable()

Creating a new Activity is a bad idea because users may use different types of Activities and this also obeys the rule of “Prefer composition to inheritence”.

<img src=”” width=400 />

setBackgroundDrawable method takes effect only the parameter is TRANSPARENT. Setting the drawable to any other drawable does not work.

Add a translucent theme to the activity

<!-- Base application theme. -->
<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
  <item name="android:windowIsTranslucent">true</item>
  <item name="android:windowBackground">@android:color/transparent</item>
  <!-- Customize your theme here. -->

Another Idea: create a layout and insert it into decorview as a child and parent of LinearLayout. Intercept touch event on this layout.


  1. Dragging With ViewDragHelper
  2. Each Navigation Drawer Hides a ViewDragHelper
  3. sockeqwe/SwipeBack
  4. ikew0ng/SwipeBackLayout
  5. How to Implement a Floating Activity in an Android App

Readings for Year 2015

Previous reading for Year 2013 & 2014:


  1. 讲谈社•中国的历史(套装共10册) by (日) 宫本一夫 等
  2. Resource Revolution: How to Capture the Biggest Business Opportunity in a Century by Stefan Heck / Matt Rogers
  3. 算法帝国 by 克里斯托弗•斯坦纳


  1. 论人类不平等的起源 by 卢梭
  2. 人类群星闪耀时 by 〔奥〕茨威格
  3. The Glory and the Dream: A Narrative History of America, 1932-1972 (Goodreads) by William Manchester
  4. Design Patterns - Elements of Reusable Object-Oriented Software by Erich Gamma / Richard Helm / Ralph Johnson / John Vlissides
  5. Notes on the Synthesis of Form by Christopher Alexander
  6. Refactoring - Improving the Design of Existing Code by Martin Fowler / Kent Beck / John Brant / William Opdyke / Don Roberts


  1. 僧侣与哲学家 by 珍·法蘭可斯雷蒙 / 馬錫·理察
  2. 宝瓶同谋:大数据时代的思想聚变 by [美] 玛丽琳. 弗格森


  1. 光荣与梦想 by 威廉·曼彻斯特
  2. 从0到1 by 〔美〕蒂尔〔美〕马斯特斯

2014 Leftover

  1. Predictably Irrational, The Hidden Forces That Shape Our Decisions , Revised and Expanded Edition by Dan Ariely
  2. Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma / Richard Helm / Ralph Johnson / John Vlissides

  3. 动物庄园 by 乔治·奥威尔
  4. 1984 by George Orwell
  5. 数学与猜想(第1卷) by G. 波利亚
  6. 数学与猜想(第2卷) by G. 波利亚
  7. 怎样解题:数学思维的新方法 by G. 波利亚
  8. 你就是极客! - 软件开发人员生存指南 by Michael Lopp
  9. 快速阅读 by Tony Buzan
  10. 你一定爱读的极简欧洲史 by John Hirst

  11. 你一定爱读的极简欧洲史 by John Hirst
  12. Android软件安全与逆向分析 by 丰生强
  13. 咖啡大全 by 田口護
  14. 蒋公的面子 by 温方伊
  15. 中午吃什麼? 一個經濟學家的無星級開胃指南 by Tyler Cowen
  16. 长尾理论2.0 by 安德森

  17. 失控 by 凯文·凯利
  18. 冰与火之歌(卷一) by [美] 乔治·R. R. 马丁

  19. 三体 by 刘慈欣
  20. 信息简史 by 詹姆斯·格雷克
  21. The Singularity Is Near (奇点临近) by Ray Kurzweil
  22. 信息简史 by 詹姆斯·格雷克
  23. 大数据 : 正在到来的数据革命,以及它如何改变政府、商业与我们的生活 by 涂子沛

  24. 失控 by 凯文·凯利
  25. 奇点临近 by Ray Kurzweil
  26. 西藏生死书 by 索甲仁波切
  27. 讲谈社•中国的历史(套装共10册) by (日) 宫本一夫 等
  28. 宝瓶同谋:大数据时代的思想聚变 by [美] 玛丽琳. 弗格森
  29. 定价未来:撼动华尔街的量化金融史[美]乔治G.斯皮罗(George G. Szpiro)
  30. 僧侣与哲学家 by 珍·法蘭可斯雷蒙 / 馬錫·理察

  31. 自控力 by 凯利·麦格尼格尔
  32. 怪诞心理学2 by 理查德·怀斯曼
  33. Android 内核剖析 by 柯元旦
  34. 走出思维的误区
  35. 认知盈余
  36. 游戏改变世界
  37. Java并发编程实战 by Brian Goetz / Tim Peierls / Joshua Bloch / Joseph Bowbeer / David Holmes / Doug Le1.
  38. [走出思维的误区:批判性思维指南(修订第9版)]
  39. 对”伪心理学”说不(第8版)(How to think straight about psychology) by Keith E. Stanovich
  40. [找寻逝去的自我:大脑、心灵和往事的记忆]
  41. HIERARCHICAL TEMPORAL MEMORY including HTM Cortical Learning Algorithms
  42. On Intelligence by Jeff Hawkins.
  43. 竞争战略 by 迈克尔·波特
  44. 编程珠玑 by Jon Bentley
  45. 文明之光 (一、二)by 吴军


  1. 丰子恺
  2. 木心 《圆光》
  3. 弘一法师 (李叔同)
  4. 汪增祺
  5. 陈之藩《剑河倒影》
  6. 简媜,杨绛,汪曾祺,张炜,北岛,野夫,张晓风,周作人,史铁生,朱自清
  7. 陈丹青的《荒废集》《退步集》
  8. 胡兰成的《今生今世》
  9. 章诒和的《伶人往事》
  10. 林清玄《心的菩提》
  11. 迟子建散文
  12. 梁遇春《春醪集》
  13. 木心先生《哥伦比亚的倒影》
  14. 龙应台《目送》刘瑜《送你一颗子弹》
  15. 陈丹青《多余的素材》
  16. 野夫的《乡关何处》
  17. 阿城–汪曾祺–沈从文–周作人–张岱
  18. 周作人的散文合集,《时光阡陌,你一直未曾走远》
  19. 郁达夫《故都的秋》
  20. 沈从文的《湘行散记》
  21. 林清玄《煮雪》


  1. 陆谷孙《20篇:英美现当代散文》
  2. 《The Art of Travel》
  3. 《Beg, Borrow, Steal——A Writer’s Life》
  4. 黄源深编:《英语散文选读》,上海外语教育出版社,2007.
  5. 南建翀编:《经典英语美文》,世界图书出版社,2008.
  6. 杨自伍编:《英国散文名篇欣赏》,上海外语教育出版社,2010.
  7. 推荐看一下网易公开课上苏州大学王腊宝教授的公开课视频: “英语经典美文:从培根到伍尔芙


2014年自媒体急速发展, 通过互联网的方式做出了很多高水平的节目, 拜托了电视节目的限制。


<img src=”” width=100>

晓说 / 晓松奇谈

<img src=”,651508043&fm=58” width=100>


<img src=”” width=100>


<img src=”,2989452900&fm=20” width=100>

The BIG talk

<img src=”” width=100 />


<img src=”” width=100 />


<img src=”” width=100>


<img src=”” width=100 />


<img src=”” width=100 />


<img src=”” width=100 />


<img src=”” width=100 />

Try out Android-L Samples with Android studio

1. 导入



2. 更新文件


此文件为空, 添加内容

// Top-level build file where you can add configuration options common to all sub-projects/modules.

buildscript {
    repositories {
    dependencies {
        classpath ''

        // NOTE: Do not place your application dependencies here; they belong
        // in the individual module build.gradle files

allprojects {
    repositories {

3. 更新文件内容



buildscript {
    repositories {

    dependencies {
        classpath ''

apply plugin: 'android'

dependencies {

    compile ""
    compile ""


// The sample build uses multiple directories to
// keep boilerplate and common code separate from
// the main sample code.
List<String> dirs = [
    'main',     // main sample code; look here for the interesting stuff.
    'common',   // components that are reused by multiple samples
    'template'] // boilerplate code that is generated by the sample template process

android {
    compileSdkVersion "android-L"

    buildToolsVersion "20.0.0"

    sourceSets {
        main {
            dirs.each { dir ->
                java.srcDirs "src/${dir}/java"
                res.srcDirs "src/${dir}/res"
        androidTest.setRoot('tests') = ['tests/src']




apply plugin: ''

dependencies {

    compile ""
    compile ""


android {
    compileSdkVersion "android-L"

    buildToolsVersion "20.0.0"

    lintOptions {
        abortOnError false


错误Unable to locate a Java Runtime to invoke.

可能是gradlew daemon 进程死掉了。

解决方案: 手动杀掉gradle daemon 进程

➜  ClippingBasic  ps aux|grep gradle
lucas           14659   0.0  0.0  2432784    612 s006  S+    4:13PM   0:00.00 grep gradle
lucas           14529   0.0  4.3  3966132 359556   ??  S     4:07PM   0:22.98 /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/bin/java -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xmx1024m -Dfile.encoding=UTF-8 -cp /Users/lucas/.gradle/wrapper/dists/gradle-1.12-bin/64p5p2nte80b6rt6bb068pabi6/gradle-1.12/lib/gradle-launcher-1.12.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 1.12 /Users/lucas/.gradle/daemon 10800000 04d22f9b-874b-469b-8011-f7f2819ec409 -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xmx1024m -Dfile.encoding=UTF-8
➜  ClippingBasic  kill 14529
➜  ClippingBasic  

Could not create plugin of type 'AppPlugin'.

Gradle 版本和gradle android plugin 版本不对 修改gradle 版本

./gradlew --version

 vim ./gradle/wrapper/
#Wed Apr 10 15:27:10 PDT 2013

修改 distributionUrl 的值 gradle-1.*

关于Gradle 版本和gradle android plugin 版本的对应关系,见 gradle plugin version compatibility 页面映射表:



  1. 初步了解gradle的DSL 语言,能够看懂, 并简单改写。
  2. 了解Android Gradle build 系统
  3. 为Feedback Android SDK 书写gradle 打包系统,发布。
  4. 生成Eclipse 导入环境。 使得在Eclipse 中可用。


Gradle 是一个通用的编译构建工具。 Gradle 编译脚本使用Groovy的DSL 语言编写。 其强调配置 而不是编程

新的Android 工程使用Gradle 来编译, Android 团队提供了一个官方的Gradle plugin 插件来支持Android 工程使用Gradle 来编译。

Android 新的构建系统使得:

  • 重用代码和资源变得更容易(maven, ivy, aar 依赖管理)
  • 创建应用的变种更容易(收费,免费,等)
  • 配置、扩展、自定义构建过程更容易 (Gradle, 插件结构)
  • 和Android Studio IDE 合二为一(新的Android Studio 直接使用Gradle 编译, 使得脚本和IDE 完全使用一套编译系统)

Gradle 特征:

  • DSL
  • Groovy 构建文件
  • Maven / Ivy 的依赖管理
  • 灵活。 允许最佳实践,但是又不强制
  • plugins
  • IDE


一个Gradle 项目使用根目录下的build.gradle 来描述其构建配置。

  • java-only

    apply plugin: 'java'
  • Android

    buildscript {
        repositories {
        dependencies {
            classpath ''
    apply plugin: 'android'
    android {
        compileSdkVersion 19
        buildToolsVersion "19.0.0"
    • buildscript { … } 配置了构建过程需要的代码。 需要注意的是此中的dependencies 只是指定了构建过程需要的依赖, 而不是项目的依赖。
    • android 表明使用android 插件来构建此项目。
    • android{ … } 配置了Android 构建的所有参数。 是Android DSL 的入口。


Gradle 利用了习惯而不是配置的原则,尽可能的提供一些合理的默认值。

可以使用”sourceSets” 来配置或者改变默认值

  sourceSets {
      main {
          java {
              srcDir 'src/java'
          resources {
              srcDir 'src/resources'


  android {
      sourceSets {
          main {
              manifest.srcFile 'AndroidManifest.xml'
              java.srcDirs = ['src']
              resources.srcDirs = ['src']
              aidl.srcDirs = ['src']
              renderscript.srcDirs = ['src']
              res.srcDirs = ['res']
              assets.srcDirs = ['assets']



  • assemble
  • check
  • build
  • clean

Android 任务

  • assemble
    • assembleDebug
    • assembleRelease
  • check
    • lint
  • connectedCheck
    • connectedAndroidTest
    • connectedUiAutomatorTest
  • deviceCheck
  • build
  • clean


  • Manifest 参数
    android {
        compileSdkVersion 19
        buildToolsVersion "19.0.0"
        defaultConfig {
            versionCode 12
            versionName "2.0"
            minSdkVersion 16
            targetSdkVersion 16


debug, release instances of BuildType.

  android {
      buildTypes {
          debug {
              applicationIdSuffix ".debug"

          jnidebug {
              packageNameSuffix ".jnidebug"
              jnidebugBuild true


依赖,Android 类库项目, 多项目配置

  • dependencies
    • local
      • compile files(‘libs/foo.jar’)
    • remote
      • compile ‘’
    • multi-project
      • compile project(‘:libraries:lib1’)
         + app/
         + libraries/
            + lib1/
            + lib2/
  • java lib project and android lib project
    • apply plugin: ‘android-library’
    • generate a .aar Android archive output
  • Build flaors and types

Library publication


Lint support

Build variants

  • 同一个应用的不同版本, 专业版(pro) vs. 免费版
  • 多APK: 不同CPU, 不同系统版本, 不同屏幕。 multi-apk

Product flavors

Build Type + Product Flavor = Build Variant


  • java
  • aapt
  • dex

AAR format

The ‘aar’ bundle is the binary distribution of an Android Library Project.

The file extension is .aar, and the maven artifact type should be aar as well, but the file itself a simple zip file with the following entries: /AndroidManifest.xml (mandatory) /classes.jar (mandatory) /res/ (mandatory) /R.txt (mandatory) /assets/ (optional) /libs/*.jar (optional) /jni//*.so (optional) /proguard.txt (optional) /lint.jar (optional) These entries are directly at the root of the zip file.

The R.txt file is the output of aapt with –output-text-symbols.

Java Plugin

  apply plugin: 'java'

Source sets

  • main
  • test


    Project layout

    sourceSets {
      main {
          java {
              srcDir 'src/java'
          resources {
              srcDir 'src/resources'

Dependency management

Eclipse Plugin

  • None
  • Java
  • Groovy
  • Scala
  • War
  • Ear


  1. Gradle Java Plugin
  2. Gradle Plugin User Guide
  3. Gradle Build Language Reference

Quickly craft Android apps using open source tools

  • otto
  • Dagger
  • StaggeredGridView
  • Bootstrap

appcompat-v7 Theme demystified


增加一些工具类, 可以使用Fragment, NotificationCompat, ViewPager等。 作为一个jar包提供, 更多是一个工具类, 通过jar包方式提供高版本系统的一些API。



  • appcompat library
  • gridlayout library
  • mediarouter library

其中最重要的就是appcompat, 为API 7-10 系统提供在11上新加的ActionBar的设计模式。 所以如果你的应用只支持API 11+, 就可以不需要这个library。

appcompat 的核心支持是ActionBar。所以如果你决定你的应用要对API 7-10 支持ActionBar, 需要将Activity 继承 support/v7/app/ActionBarActivity, 并且将activity 的theme 设为Theme.AppCompat。


v0 表示定义在values/values.xml 中 v11 表示定义在values-11/values.xml 中 v14 表示定义在values-14/values.xml 中 app 表示应用定义的style

在v11 之后默认holo 风格

还有对应的Light 风格, 在每个Theme 的后面加上.Light


REST 缩写: REpresentation State Transfer 最初想法有Roy Fielding 在他的博士论文 Architectural Styles and the Design of Network-based Software Architectures 中提出来。

REST 有6大必要条件

  • Uniform Interface 统一接口
    统一接口定义了客户端和服务器之间的接口。简化并解耦了架构, 使得双方都可以独立进化。
    • Resource-based 基于资源 URI as resource identifiers
    • Manipulation of Resources Through Representations
    • Self Descriptive Messages
    • Hypermedia as the engine of Application State
  • Stateless 无状态
    服务器端处理请求需要的状态都包含在客户端发送的请求当中。状态可能是URI的一部分,查询字符串参数, body,也可能在Header 中。URI 唯一标识资源,body 包含了资源的状态。 无状态使得架构更具扩展性
  • Cacheable 可缓存
  • Client-Server 客户端-服务器模型
  • Layered System 分层设计的系统
  • Code on Demand (optional) 按需编码
    Java Applets or javascript

REST enables: performance, scalability, simplicity, modifiability, visibility, portability and reliability.

HTTP 状态码


  • 使用HTTP 动词
    • GET: 读取某个特定资源或者资源集合
    • PUT: 更新某个资源
    • POST: 创建新的资源。如果找不到合适的动词就用这个动词
    • DELETE: 删除某个资源

Provide Sensible Resource Names

设计一个好的API 80% 的艺术 + 20% 的科学

  • URL中使用标识, 而不是查询字符串, 如 /users/12345
  • 巧妙使用URL 的等级结构特性 来标识资源的结构
  • 为客户端设计, 而不是为数据设计
  • 资源名称使用名词, 使用HTTP 的方法来表示请求的动作部分
  • URL 中使用复数名词
  • 避免使用集合词
  • URL 使用小写, 使用’_‘或者’-‘来分割单词
  • 尽量保持URL 短小

Use HTTP Response Codes to Indicate Status

Offer Both JSON and XML

Create Fine-Grained Resources

Consider Connectedness




  • API 中的page设计

  • API Design



  • [django-rest-framework](

Python 学习笔记

May 14, 2014

由于项目的需求, 需要使用Django 技术栈。 Django 是基于Python 语言开发 的流行的Web 框架。 没办法,得学习Python。 由于时间关系, 不能花太多的时间去看大部头的书和文档, 需要快速上手。 在Slideshare 上找到一个不错的入门PPT。

Learn 90% of Python in 90 Minutes by Matt Harrison

Learn 90% of Python in 90 Minutes from Matt Harrison

标题看着很吓人, 也很励志, 90分钟搞定, 很适合我的需求。 start quickly and move fast.

确实大概花了100分钟的时间就看完了。 这个Slides 主要讲Python语言的语法。 多数语言的语法都相似,所有只要你很好的掌握了一门编程语言,学习其他的语言是相当简单容易的。 我之前有几年的C和Java 编程经验, 看起Python的语法,简单很多。

几乎所有基于C的编程语言语法大同小异, 介绍的ToC都会差不多。


  • 命名
  • 数据类型
  • 方法定义
  • 类定义
  • 类继承
  • 集合: List, Set, Dictionary, Map 等
  • 迭代
  • 操作符
  • 文件操作
  • debug

下面的MindMap 就总结了Python语言的这些方面:


  • Exception处理
  • 类库
  • 生态

Python 类库

Python for the web. python 是为web 而设计, 使用python 很大程度上为了于web 交互。 从而不可避免的要有一些web 相关的 库。


json 由于AJAX的兴起,大受欢迎, 以至于今天多数的web API 都以JSON 格式提供。 使用python 操作json 也成了日常工作中必不可少的技能。 Django 1.7 内置了新的json 类库, 但是1.7之前, 还是需要使用python 的json 类库.

如果熟悉json 格式的话,学习这个类库会很简单。 Python内部使用dictionary来存储json 数据, 没有class JSONObject 这样的类。

In [10]: type(x)
Out[10]: dict

In [11]: x={'__complex__':True, 'real':1, 'imag':2}

In [12]: type(x)
Out[12]: dict

In [13]: json.dumps(x)
Out[13]: '{"real": 1, "imag": 2, "__complex__": true}'

In [14]: str = json.dumps(x)

In [15]: type(str)
Out[15]: str

In [16]: j = json.loads(str)

In [17]: type(j)
Out[17]: dict

In [18]: j
Out[18]: {u'__complex__': True, u'imag': 2, u'real': 1}

json 类库主要有四个函数:

  • json.dumps() 将python 的dictionary dump 成字符串输出
  • json.loads() 将字符串转化成 dictionary
  • json.JSONEncoder.encode()
  • json.JSONDecoder.decode()

Pretty print json

curl |python -mjson.tool


Django 使用django.utils.simplejson 类库


Django 学习笔记

April 15, 2014

Writing your first Django app, part 1, Template

  1. Django follows MVC architecture pattern
  2. Install
  3. Hello, World
    • create project startproject mysite
    • project structure

      ➜  python  tree mysite
      └── mysite
    • Run

    • DB
    • debug
    • Applications
  5. Project v.s. Apps
  6. Create App
     $ python startapp polls

April 16, 2014

Writing your first Django app, part 2, Model

  1. start server python runserver
  2. admin site
  3. Make the poll app modifiable in the admin
  4. Customize the admin form
    • reorder fields
    • add field title
    • add html classes
    • add related objects, StackedInline, TabularInline
  5. Customize the admin change list
    • list_display
    • admin_order_field
    • short_description
    • list_filter
    • search_fields
    • change list pagination
  6. Customize the admin look and feel
    • Customize your project’s templates
    • django template file
    • customize the admin index page

April 16, 2014

Writing your first Django app, part 3, View

  1. Django views are web pages. URL pattern, urlconfs: map url patterns to views.
  2. Write your first view url() arguments, regex, view, kwargs, name
  3. write views that actually do something
    • return HttpResponse
    • Raise exception such as Http404
    • Django db API
    • Template System
    • loader
    • render
    • removing hardcoded urls in template by using {% %}
  4. Namespacing URL names

April 16, 2014

Writing your first Django app, part 4, Form

  1. Write a simple form
  2. Use generic views: Less code is better

April 16, 2014

Writing your first Django app, part 5, Testing

  1. automated testing
  2. basic testing strategy
  3. first test
  4. test a view

April 16, 2014

Writing your first Django app, part 6, static files

  1. Customize your app’s look and feel
  2. Adding a background-image

April 22, 2014

Advanced tutorial: How to write reusable apps

  1. package django apps to a tar file
  2. virtualenv
  3. python package index PyPI

May 14, 2014

i18n, l10n


May 15, 2014

Python 包管理

大部分现代语言都需要有包管理的机制, 使得开发的组件可以重用。

Java使用jar 打包组件, 使用maven来管理组件和组件之间的依赖。

Python 使用pip来安装和管理本地的组件。 其他工具有easy_install, distribute

pipPyPI (Python Package Index) 线上网站查找需要的包,如同maven 从mvncentral 查找jar包。

Python 使用Eggs, 如同Java 使用jar


Web Server Gateway interface


Werkzeug is a WSGI utility for Python


  1. Run server by python runserver. How to start it as background service?



某天在豆瓣上闲逛, 看到一个标题, “我是咖啡豆, 我为自己带盐”。 这本是借鉴陈欧为聚美优品做的广告,“我为自己代言”。 只是网络上的人越来故意的写错字用以卖萌。 听过各种的咖啡,包括诡异的猫屎咖啡, 但好像是从来没见过咖啡加盐。 想必会有一番风味。不过细想一下,这种简单搭配不存在, 说明味道不会怎么样。在网上搜索了一下, 还真有“盐咖啡”的存在。 讲的是一个美丽感人的爱情故事。

用Google英文搜索了一下, 发现真的存在咖啡加盐的喝法。 Life Hacker 上就有一篇讲到, 当咖啡煮坏了, 太苦的时候, 加一点盐有助于减轻苦味。

其实生活本无所定式和模式,尝试不同的组合, 即使没有发现很好的结果,或许尝试的过程本身就是一种享受。 有那么多的配料存在, 糖, 盐, 孜然, 胡椒, 蜂蜜,奶粉,牛奶,姜,各种各样, 和咖啡,茶混起来,或许能够找到适合的口味。 姜和茶就是老家老辈人很享受的组合。蒙古人会把牛奶和茶煮到一起。

附 盐咖啡的故事
译者: 梁軍

突然,他对服务员说,“请给我一点盐好吗? 我想把它加在我的咖啡里。”咖啡馆里每个人都惊奇地看着他——真是个怪人。他的脸变红了,但他仍把盐放入他的咖啡中,并喝下了那杯咖啡。她好奇地问他,“你为什么有这样的嗜好?”他回答,“小时候我住得离大海很近,我喜欢在海里玩,我能够感觉大海的味道,就象这杯盐咖啡的味道。现在,每当我喝盐咖啡,我总是想起我的童年、我的故乡,我非常想念我的家乡和依然生活在那里的父母。” 他说到这些,眼泪涌了出来。她深深地被感动了,
那真是一次非常不错的交谈,也是他们爱情故事的一个美丽开端。从那以后,他们继续约会。她发现事实上他是一个能满足她要求的男人:他有一颗善良的心,能容忍,多情、细心。他真是一个好人,而她差点错过了他! 真要谢谢他的盐咖啡!
四十年后,他去世了,留给她一封信,信中写道,“我最亲爱的,请原谅我。原谅我说了一个持续一生的谎言。这是我对你唯一的一次谎言——盐咖啡。记得我们第一次约会吗?当时我太紧张了,其实我想要一些糖,但却说成了盐。当时我觉得难以改口,只好随其发展了,却怎么也没想到那会是我们交往的开始! 在后来有许多次,我都想告诉你事实真相,但是我太担心,因为答应过你绝不说谎。现在我快要离开人世了,我不担心这么做了。所以,我告诉你,我不喜欢盐咖啡,那味道真是又怪、又难喝……但是我却喝它喝了一辈子。自从我认识你之后,为你做的事,我从不后悔。在我漫长的一生中,拥有你是我最大的幸福。如果我再活一次,我仍想与你相知,再次与你结合,过一辈子,即使我必须再喝盐咖啡我也愿意。”


Genymotion is based on VirtualBox, most issues come from VirtualBox. If you cannot start a Genymotion emulator, try reinstall VirtualBox, or repair VirtualBox.

Resolving Genymotion problems on Mac

  • genymotion error

sudo /Library/StartupItems/VirtualBox/VirtualBox restart


open Disk Utility (search Spotlight), try Repair Disk Permission

Restart Genymotion

Building CyanogenMod for Google Nexus 4

Android Derivative ROMs

  • AOSP
  • CyanogenMod
  • Kandroid
  • MIUI
  • Aliyun OS
  • AOKP
  • Baidu Yi
  • LeWa OS
  • OPhone
  • Replicant
  • XobotOS
  • Stock
    • HTC Sense
    • Samsung TouchWiz
    • LG Optimus UI
    • Sony UI

For more, check Wikipedia.

An excellent video tutorial on Youtube


Hardware driver (proprietary)

Using ./extract-files to extract vendor drivers from real device.

/media/android/cm/system/device/lge/mako$ ll 
-rwxrwxr-x 1 lucas lucas 354 Nov 18 23:19*

This pulls data from device’s /system directory to /vendor/lge/mako/proprietary directory.’

lucas@lucas-ubuntu:/media/android/cm/system/vendor/lge/mako/proprietary$ ll
total 24
drwxr-xr-x 6 lucas lucas 4096 Nov 19 11:29 ./
drwxr-xr-x 3 lucas lucas 4096 Nov 19 10:17 ../
drwxr-xr-x 2 lucas lucas 4096 Nov 19 11:29 bin/
drwxr-xr-x 3 lucas lucas 4096 Nov 19 11:29 etc/
drwxr-xr-x 5 lucas lucas 4096 Nov 19 11:29 lib/
drwxr-xr-x 4 lucas lucas 4096 Nov 19 11:29 vendor/

These includes drivers for CPU, NFC fromBroadcom, Camera, Sensors, Audio, DRM, Cryptography from LG, and imgs, GSM, Camera, GPS, Wi-Fi, Bluetooth, Sensors, Media, DRM, DSP, USB from Qualcomm. All these drivers are proprietary.


The next thing is to build the kernel source use breakfast mako (code name for Nexus 4) to download kernel source code from github



Enviroment setup

  • Ubuntu 64 bit
  • Android SDK
  • JDK
  • Python

Download source

  • repo
  • mkdir, /media/android/cm/system/

CM source file structure


  • bionic
  • build
  • bootable: cwm recovery
  • dalvik
  • art, replacement of dalvik, android runtime
  • device, board support packages and configurations for a particular device. device//, /device/] asus/grouper, device/lge/mako, where to put customized stuff for each device. `breakfast grouper && repo sync`
  • docs: android open source website
  • external, non-android utilities, with to work with android build system
  • frameworks, Android core
  • hardware, platform/hardware specific libs
  • kernel, /kernel//
  • ndk
  • out, make clobber
  • packages, Android standalone apps, like Settings, Browser, Launcher, etc
  • prebuilt(s)
  • system: linux shell, etc.
  • vendor: cm, lge, htc, etc

For Nexus 4:

  • vendor:cm
  • hardware: ti
  • kernel: google/msm
  • device: lge/mako

Repo manifest format

top level ./.repo/default.xml XML file.

  • manifest: the root element of the file.
  • remote: git url shared by one or more projects and the gerrit review server those projects upload changes through.
  • name: short git remote name
  • alias: override name if specified in each project’’s .git/config
  • fetch: git url prefix for all projects which use this remote
  • review: gerrit review server used by repo update
  • default: used when a project does not specify remtoe and revision.
  • manifest-server
  • project: single git repo.
  • annotation
  • remove-project
  • include

Local Manifests


adb pull /system/* to /vendor/lge/mako/proprietary

The problems encountered during downloading the cm source

Since the source repo is huge, the downloading process can be intermittant

3. build


build files


each module/ subproject has a file, telling the build system how to build the module, and where to put the output in the Android directory. 

The built files are put in /out/target/project/CODENAME, with zipped flashable recovery*.zip and fastboot*.img files. 

## $OUT directory
$OUT= /out/target/project/CODENAME

* kernel 
* /system, the /system folder on Android
* /root, contains the files that will be turned into ramdisk loaded and run by the kernel, including init, init.rc, init.CODENAME.rc
* /recovery/root, ramdisk contains the recovery mode 

mm: make modules, mm -B 
adb sync system

adb remount= adb shell mount -o rw,remount /system

Build Success

➜  mako  ls -la
total 803408
drwxrwxr-x 10 lucas lucas      4096 Nov 19 21:15 .
drwxrwxr-x  3 lucas lucas      4096 Nov 19 18:33 ..
-rw-rw-r--  1 lucas lucas        19 Nov 19 18:34 android-info.txt
-rw-r--r--  1 lucas lucas   6436864 Nov 19 21:02 boot.img
-rw-rw-r--  1 lucas lucas     34552 Nov 19 18:33
-rw-rw-r--  2 lucas lucas 189761342 Nov 19 21:15
-rw-rw-r--  1 lucas lucas       120 Nov 19 21:15
-rw-rw-r--  2 lucas lucas 189761342 Nov 19 21:15
drwxrwxr-x  2 lucas lucas      4096 Nov 19 20:46 data
drwxrwxr-x  4 lucas lucas      4096 Nov 19 18:39 external
drwxrwxr-x  2 lucas lucas      4096 Nov 19 21:10 fake_packages
-rw-rw-r--  1 lucas lucas     71456 Nov 19 21:13 installed-files.txt
-rwxrwxr-x  1 lucas lucas   5997088 Nov 19 20:36 kernel
drwxrwxr-x 18 lucas lucas      4096 Nov 19 21:13 obj
-rw-rw-r--  1 lucas lucas        49 Nov 19 21:14 ota_script_path
-rw-rw-r--  1 lucas lucas       669 Nov 19 18:33
-rw-rw-r--  1 lucas lucas    434701 Nov 19 21:01 ramdisk.img
-rw-rw-r--  1 lucas lucas   3845120 Nov 19 21:03 ramdisk-recovery.cpio
-rw-rw-r--  1 lucas lucas   2510001 Nov 19 21:04 ramdisk-recovery.img
drwxrwxr-x  3 lucas lucas      4096 Nov 19 21:02 recovery
-rw-r--r--  1 lucas lucas   8511488 Nov 19 21:04 recovery.img
drwxrwxr-x  9 lucas lucas      4096 Nov 19 21:01 root
drwxrwxr-x  5 lucas lucas      4096 Nov 19 20:59 symbols
drwxrwxr-x 15 lucas lucas      4096 Nov 19 21:01 system
-rw-r--r--  1 lucas lucas 314466532 Nov 19 21:14 system.img
-rw-r--r--  1 lucas lucas 100775392 Nov 19 20:46 userdata.img
➜  mako  


  1. Flash ClockworkMod recovery img. fastboot flash recovery recovery.img
  2. Flash
fastboot -w flashall

Building Kernel

  1. Building kernel from source

  2. Or, using prebuilt kernel image


Proprietary Drivers

Binaries for Nexus Devices

Restoring to factory state

upgrade to Android 4.4 on Nexus 4

Note: codename for factory image of Nexus 4 is occam

Android product naming rules



  2. Youtube //

Readings for Year 2013

说明: 此前由读《暗时间》 催生了读书的计划。 这篇博客用来记录2013 年看过的和想看但没看完或者还没开始看的书。 由于记不太清楚, 个别书可能是2012 年或者更早读过。



  1. 暗时间 by 刘未鹏
  2. 如丧 : 我们终于老得可以谈谈未来 by 高晓松
  3. 一个村庄里的中国 by 熊培云
  4. 狂热分子 : 群众运动圣经 by 埃里克·霍弗
  5. 人间失格 by 太宰治
  6. by 莫言
  7. 黑客与画家 by Paul Graham
  8. 天才在左 疯子在右 by 高铭
  9. 奇思妙想: 15位计算机天才及其重大发现 by Dennis E. Shasha / Cathy A. Lazere
  10. Java解惑:Traps, Pitfalls, and Corner Cases by (美)布洛克//加夫特
  11. 编写高质量代码:改善Java程序的151个建议 by 秦小波
  12. 乌合之众:大众心理学研究 by (法)古斯塔夫.勒庞
  13. 演讲之禅:一位技术演讲家的自白 by Scott Berkun


  1. 美妙的新世界 by 阿道斯·伦纳德·赫胥黎
  2. Predictably Irrational, The Hidden Forces That Shape Our Decisions , Revised and Expanded Edition by Dan Ariely
  3. Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma / Richard Helm / Ralph Johnson / John Vlissides


  1. 动物庄园 by 乔治·奥威尔
  2. 1984 by George Orwell
  3. 第七天 by 余华
  4. 可能与不可能的边界 by Lance Fortnow
  5. 人生中最美妙的事都是免费的 by 尼尔•帕斯理查
  6. 本能: 为什么我们管不住自己? Mean Genes by 特里·伯纳姆
  7. 数学与猜想(第1卷) by G. 波利亚
  8. 数学与猜想(第2卷) by G. 波利亚
  9. 怎样解题:数学思维的新方法 by G. 波利亚
  10. 你就是极客! - 软件开发人员生存指南 by Michael Lopp
  11. 快速阅读 by Tony Buzan


  1. 看见 by 柴静
  2. 猫力乱步 by 猫力 读完部分
  3. 你一定爱读的极简欧洲史 by John Hirst


  1. 猫力乱步 by 猫力
  2. The Performance of Open Source Applications II Secrets of Mobile Network Performance
  3. 你一定爱读的极简欧洲史 by John Hirst
  4. Android软件安全与逆向分析 by 丰生强
  5. 咖啡大全 by 田口護
  6. 蒋公的面子 by 温方伊
  7. 中午吃什麼? 一個經濟學家的無星級開胃指南 by Tyler Cowen
  8. 长尾理论2.0 by 安德森
  9. 快速阅读 by Tony Buzan


  1. 大数据变革: 让客户数据驱动利润奔跑 by 比约•布劳卿 / 拉斯•拉克 / 托马斯•拉姆什

    一本大烂书,完全没有任何引起我兴趣的见解,连新知识点都没有。 建议一小时内看完, 看不完也不要看了。

  2. 至关重要的关系(THE START-UP OF YOU) by Reid Hoffman / Ben Casnocha

    又是一本大烂书,适合在机场等飞机半小时看完的那种。 还不如心灵鸡汤类的成功学, 至少会有一些鸡血故事刺激一下。 自从学了“快速阅读” 的方法后, 看这种书真是可以一目十行。这本书是个奇葩, 看完目录,基本就相当于看完了, 剩下的内容, 找些段子,故事,填充下逻辑就好了。 不知道为什么这本书在豆瓣上评分8.3那么高。 这种商人写书, 真是渣渣。


  1. 十六周的人生蜕变: 找到意想不到的自己 by 胡蓉蓉 Vvivi Hu

    小资情调, 认真生活。 文字读起来很舒适。 4星

  2. 失控 by 凯文·凯利
  3. 冰与火之歌(卷一) by [美] 乔治·R. R. 马丁


  1. 车库咖啡 by 苏菂(口述) / 王海珍(撰稿)
  2. 互联网思维独孤九剑 by 赵大伟
  3. 三体 by 刘慈欣
  4. 罗辑思维 (Youku 视频节目)
  5. 信息简史 by 詹姆斯·格雷克
  6. The Singularity Is Near by Ray Kurzweil
  7. 互联网思维独孤九剑 by 赵大伟
  8. 互联网新思维:未来十年的企业变形计 by David Kerpen


  1. 信息简史 by 詹姆斯·格雷克
  2. 数据之巅:大数据革命,历史、现实与未来 by 涂子沛
  3. 大数据 : 正在到来的数据革命,以及它如何改变政府、商业与我们的生活 by 涂子沛
  4. 奇点临近 by Ray Kurzweil
  5. 短暂飞行 by 张向东


  1. 国际平面设计基础教程 by 加文·安布罗斯 [等]








  1. 失控 by 凯文·凯利
  2. 奇点临近 by Ray Kurzweil
  3. 西藏生死书 by 索甲仁波切
  4. 讲谈社•中国的历史(套装共10册) by (日) 宫本一夫 等
  5. 每天懂一点色彩心理学•实用篇(附色彩杂学练习册) by (日) 原田玲仁
  6. 配色设计原理 by 奥博斯科编辑部
  7. 宝瓶同谋:大数据时代的思想聚变 by [美] 玛丽琳. 弗格森
  8. 定价未来:撼动华尔街的量化金融史[美]乔治G.斯皮罗(George G. Szpiro)
  9. 僧侣与哲学家 by 珍·法蘭可斯雷蒙 / 馬錫·理察

Waiting List

  1. 自控力 by 凯利·麦格尼格尔
  2. 怪诞心理学2 by 理查德·怀斯曼
  3. Android 内核剖析 by 柯元旦
  4. 走出思维的误区
  5. 认知盈余
  6. 游戏改变世界
  7. Java并发编程实战 by Brian Goetz / Tim Peierls / Joshua Bloch / Joseph Bowbeer / David Holmes / Doug Le1.
  8. [走出思维的误区:批判性思维指南(修订第9版)]
  9. 对”伪心理学”说不(第8版)(How to think straight about psychology) by Keith E. Stanovich
  10. [找寻逝去的自我:大脑、心灵和往事的记忆]
  11. HIERARCHICAL TEMPORAL MEMORY including HTM Cortical Learning Algorithms
  12. On Intelligence by Jeff Hawkins.
  13. 竞争战略 by 迈克尔·波特
  14. 编程珠玑 by Jon Bentley
  15. 文明之光 (一、二)by 吴军

Programmer’s must read

  1. 程序员的自我修养 by 俞甲子 / 石凡 / 潘爱民
  2. Computer Systems:A Programmer’s Perspective by Randal E. Bryant / David R. O’Hallaron
  3. <img src=”” width=30px/>Structure and Interpretation of Computer Programs by Harold Abelson and Gerald Jay Sussman
  4. The Art of Unix Programming by Eric S. Raymond.
  5. The Performance of Open Source Applications part I
  6. The Performance of Open Source Applications part II
  7. The Performance of Open Source Applications by AOSA.

Deep Dive to Android Internals (resources)

  1. XDA Developers’ Android Hacker’s Toolkit: The Complete Guide to Rooting, ROMs and Theming by Jason Tyler (Author), Will Verduzco
  2. NewCirle’s Android Training
  3. Android Architecture
  4. Deep Dive into Android IPC/Binder Framework


  1. 丰子恺
  2. 木心 《圆光》
  3. 弘一法师 (李叔同)
  4. 汪增祺
  5. 陈之藩《剑河倒影》
  6. 简媜,杨绛,汪曾祺,张炜,北岛,野夫,张晓风,周作人,史铁生,朱自清
  7. 陈丹青的《荒废集》《退步集》
  8. 胡兰成的《今生今世》
  9. 章诒和的《伶人往事》
  10. 林清玄《心的菩提》
  11. 迟子建散文
  12. 梁遇春《春醪集》
  13. 木心先生《哥伦比亚的倒影》
  14. 龙应台《目送》刘瑜《送你一颗子弹》
  15. 陈丹青《多余的素材》
  16. 野夫的《乡关何处》
  17. 阿城–汪曾祺–沈从文–周作人–张岱
  18. 周作人的散文合集,《时光阡陌,你一直未曾走远》
  19. 郁达夫《故都的秋》
  20. 沈从文的《湘行散记》
  21. 林清玄《煮雪》


  1. 陆谷孙《20篇:英美现当代散文》
  2. 《The Art of Travel》
  3. 《Beg, Borrow, Steal——A Writer’s Life》
  4. 黄源深编:《英语散文选读》,上海外语教育出版社,2007.
  5. 南建翀编:《经典英语美文》,世界图书出版社,2008.
  6. 杨自伍编:《英国散文名篇欣赏》,上海外语教育出版社,2010.
  7. 推荐看一下网易公开课上苏州大学王腊宝教授的公开课视频:“英语经典美文:从培根到伍尔芙

Re-implement the works

前两天读到某某教授的一篇文章,作者提出一个“3遍阅读法(3-pass reading)”,最后一遍的要求叫"re-implement"。 这个词特别好。 我们读文章时, 或者因为使用某些知其然, 不知其所以然的类库, 知识时往往开始觉得很困惑, 经常用错。 源于我们对内在的原理不了解。 而“re-implement”要求我们从原作者的思维触发, 重铸作品, 达到知其所以然的目的。 当自己从头开始完全重铸意见作品时, 就能明白其中很多的条件和假设, 以及各种限制。 这些往往在我们使用是会忽略的因素。 大多数情况下, 一个工具,一套理论,一个开发类库, 都由其适用的条件和环境。 或许作者没有说清楚, 或者使用者没有仔细弄明白说明文档, 往往很容易用错。 这点对于程序员来说更甚。 往往拿到一个类库, 不会仔细阅读说明, 而是简单按照函数名称做一个猜测,忽略它的限制条件。 很多bug 就是由此引入的。 这些类库正常情况下使用良好, 然后在某些极端情况下会出问题。 带来各种bug。

由”re-implement” 一次我想到说其实类的设计充满这妥协和限制。 一个好的程序员应该站在类库作者的角度来考虑这些问题。简单说,即使不要求完全重写类库, 也应该在心里过一遍如果是自己来设计, 会如何做。 有条件的情况下, 多读读类库的源码。 一定会有很多收获。

Android 中的AsyncTask 就是这么一个很容易出错的类。 这个类android.os.AsyncTask 位于android.os 包下,虽然其实更应该是一个util类。这个类本是为了方便程序员写异步操作而设计, 然后它的API本省非常长。 而且在说明里告诉说这个类不适合长时间运行的任务,结果给出的例子就叫做DownloadingTask. 这是一个设计很糟糕的类。

对于它的设计, 如果没读过源码,可能会觉得有什么特别的, 因为位于android.os中。 然而它就是Handler和ThreadPool的简单封装而已。

  • <1.6 单线程
  • 1.6-3.0 多线程
  • =3.0 单线程

代码中使用了SerialExecutorThreadPoolExecutor, 有一个sDefaultExecutor静态成员变量来控制, 可以通过setDefaultExecutor函数来控制。

所有的回调函数都是通过Handler 和sendMessage 来实现。 非常简单,是不是?

另外, ThreadPool 中初始化时定义最大128 个, 同时5个thread 执行。


Looper 是线程见消息同步的机制。 一个线程发生了某件事情, 如何实时通知另外一个线程? 这是一个通用问题。 常见做法是有一个线程一直死循环, 检查一个变量, 另外的线程发生什么事情就写这个变量。 一直死循环在检查变量的线程就是Looper 线程。 Android 中UI线程就有一个Looper, 当把本身的事情都做完之后, 就会开始死循环, 一旦有消息, 在死循环里完成那个消息触发的事件, 继续死循环。

Android 线程相关

  • Handler, Message, Messenger, Process, AIDL, binder AsyncTask Thread Runnable ThreadPool


这里列出一些移动互联网从业者应该关注的站点, 大部分为新闻和评论站点。

  • kickstarter Kickstarter is the world’s largest funding platform for creative projects.

读书笔记: 黑客与画家

第九章 设计者的品味

Play Framework

Play around “Play”

  1. Install
  2. Create “Hello, world.”
    • play new todolist
  3. Anatomy of Play application / project structure
  4. Using the play console
    • launch: play
    • run
    • help play
    • compile
    • console: interactive
    • play debug
    • sbt features
    • play run
    • Ctl + D to exit
    • play clean-all
  5. IDE Support
  6. Samples
    • hello world
    • Computer database
    • CometClock
    • Bootstrap
    • WebSocket
  7. databases: CRUD, Ebean
  8. Enumerators, Enumeratees
  9. Akka actors
  10. The Apache Velocity Project
  11. Deploying to Heroku


  1. HTTP programming
  2. Async HTTP
  3. the template engine
  4. HTTP form submission and validation
  5. with Json
  6. With XML
  7. File Upload
  8. SQL DB
  9. Cache
  10. Calling WebServices
  11. Gloabal
  12. Testing

hand by hand introduction: Your first Play application

IO in Java

流,英文Stream. Something that is flowing. 信息流, 照片流,文字流, 数据流, 字节流

Android 自动化

Android 自动化开发流程 在android 社区中有一些列的开源工程帮助andorid 自动化。


Android maven plugin




Android Interview

Interview Questions Collection

  1. careercup,There is an accompany book: “Cracking the Coding Interview: 150 Programming Questions and Solutions” by Gayle Laakmann McDowell
  2. Cracking the coding interview–问题与解答
  3. Quora:Java Interview Questions. A collection of java interview questions on quora.

    Java Resources

  4. Qura: What questions are Java Software Engineers seeing the most of on technical interviews?
  5. Qura: [What are good interview questions to ask JAVA developers?] (
  6. Java puzzlers by Joshua Bloch. RECOMMENDED FOR SENIOR JAVA DEVELOPERS.
  8. Java Concurrency in Practice by Tim Peierls, Brian Goetz, Joshua Bloch, Joseph Bowbeer, Doug Lea, David Holmes. RECOMMENDED FOR SERVER DEVELOPERS.
  9. Inside the Java Virtual Machine by Bill Venners. Online book available.
  10. Books Related to the JVM, a list of jvm books.
  11. Thinking in Java by Bruce Eckel. JAVA 101
  12. The Java Job Interview Companion
  13. Quora: What are good books to test one’s programming skills?
  14. Wiki:List of data structures
  15. Wiki:List of algorithms
  16. Programming Interviews Exposed: Secrets to Landing Your Next Job
  17. Coding Interviews By Harre He . A Press
  18. Cracking the coding interview by Gayle as mentioned in other posts
  19. Algorithms For Interviews (9781453792995): Adnan Aziz, Amit Prakash: Books
  20. Programming Challenges by Steven Skiena
  21. Elements of Programming Interviews.
  22. Cracking the Coding Interview: 150 Programming Questions and Solutions
  23. Cracking The IT Interview 2nd Edition -
  24. Data Structures and Algorithms Made Easy 2nd Edition - I strongly recommend this book
  25. Cracking the C, C++ and Java Interview 1st Edition -
  26. Test your C Skills - Yashwant Kanetkar
  27. Test your C ++ Skills - Yashwant Kanetkar
  28. Exploring C - Yashwant Kanetkar
  29. Test Your Unix skills - Yashwant Kanetkar
  30. Top 15 Java threading interview questions asked in Investment banks
  31. 有道难题

I mostly agree with John Kurlak’s answer. Personally I feel Cracking the Coding Interview: 150 Programming Questions and Solutions is a good choice if you only know Java. Programming Interviews Exposed is a little elementary and the materials actually need to be updated. Elements of Programming Interviews: 300 Questions and Solutions seems for people who actually want to pass all interviews including the hardest one after concluding all reviews on Amazon, and it is mainly written in C++ which makes this as a great choice for people who can understand on C++.

Sample Questions

  1. Card Shuffle

The following shuffle method purports to shuffle its input array fairly. In other words, it purports to generate all permutations with equal likelihood, assuming that the underlying pseudorandom number generator is fair. Does it make good on its promise? If not, how do you fix it?

import java.util.Random;

public class Shuffle {
    private static Random rnd = new Random();

    public static void shuffle(Object[] a) {
        for (int i = 0; i < a.length; i++)
            swap(a, i, rnd.nextInt(a.length));

    private static void swap(Object[] a, int i, int j) {
        Object tmp = a[i];
        a[i] = a[j];
        a[j] = tmp;
  1. Java™ Puzzlers: Traps, Pitfalls, and Corner Cases Puzzle 45: Exhausting Workout
This puzzle tests your knowledge of recursion. What does this program do?

public class Workout {
    public static void main(String[] args) {
        System.out.println("It's nap time.");

    private static void workHard() {
        try {
        } finally {
  1. Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

  2. Clock puzzles - find the angle between hr,min,sec handles, etc

  3. From Twitter

"Consider the following picture:"

alt text

"In this picture we have walls of different heights. This picture is represented by an array of integers, where the value at each index is the height of the wall. The picture above is represented with an array as [2,5,1,2,3,4,7,7,6]."

"Now imagine it rains. How much water is going to be accumulated in puddles between walls?"

alt text

"We count volume in square blocks of 1X1. So in the picture above, everything to the left of index 1 spills out. Water to the right of index 7 also spills out. We are left with a puddle between 1 and 6 and the volume is 10."

Online Tests

  1. Test For Geeks, including java and android tests. Extremely challenging.
  2. Fizz Buzz Test
  3. Brainbench test


  1. Java Collection
  2. Multi Threading
  3. GC
  4. Design Patterns
  5. Algorithms
  6. Data Structure



  1. @知乎 如何面试android 工程师? 从哪些方面考察?
  2. 面试官如何写面试反馈?
  3. 整理一个PPT 出来, 分topic 出面试题, 做分享
  4. 整理知乎上的相关java / android 面试题

Android 面试题

  1. Android 中线程与线程,进程与进程之间如何通信?
  2. 解释 Binder 原理, 进程间通信
  3. activity,intent 和 service 是什么关系?
  4. handler 机制的原理是什么?
  5. 横竖屏切换时候 activity 的生命周期?
  6. 解释View 绘制过程
  7. 瀑布流如何回收资源
  8. 常用的Android 开源库有哪些, 如何选择?挑一个谈谈优缺点,如何改进?
  9. 什么情况下会出现ANR, 怎么找到原因, 如何解决?
  10. 起了很多线程, UI线程阻塞
  11. 后台线程, Service
  12. 异步线程
  13. listview 优化


两本和Big Data 相关的书, 一本是媒体书, 一本是技术书

Big Data 里讲到大数据的核心是预测。  Jeff Hawkins做Neural Networks 也说AI的核心是预测(On Intelligence)
可以参考之前的blog: [由《暗时间》所引起的读书计划](
大数据时代, 逐步放弃对精确性的追求。 
可以在google scholar 上查一查big data 对hedge fund, social network, twitter, finance, privacy, 大选, 民调 等领域的影响。大数据时代的隐私所面对的挑战不是单一数据源上的问题, 而是多数据源合并,通过reverse-anonymization 技术带来的挑战。 

大数据和物联网, 物联网时代每一个物体都在产生数据,核心将从功能转化成数据。 比如Fuel Band  就是一个数据收集器。 

大数据不是指数据绝对意义上的多, 而是采用全量不是采样的分析方法。 

预测只需要知道关联关系, 不需要知道因果关系。 
相关关系只是缺少一个人工的解释。 存在即合理。 

Tera data 数据分析公司

检查利用手机Sensor 累计数据的应用, 比如步速


在物理测量中, 极大极小意义上的测量和普通意义上的测量完全不同。 极小对应量子力学。 不确定原理。 
极大对应相对论,超光速, 黑洞等。 

数据处理也一样, 同样的计数, 在小数据的情况下, 普通的计数器。 
在大数据的情况下, 同步, 一致性, 等诸多问题。 

所以大数据时代产出的理论如同广义相对论, 囊括了各种不同情况下的描述。 而万有引力只是一种近似。 

找一本大数据的书, 讲述亚马逊的推荐系统。 

大数据时代的隐私保护。 privacy preserving in big data era. 

家具制造, 物流, 企业配料, 赌球赔率

Use probability to approach Pi

Once I found this interview question: calculate Pi. Where, all of us have learned calculating Pi in our primary math class by using regular polygon to approach a circle.

Today I am gonna show you a probability way.


Check the following graph. I hope it is self-explainatory.


class Pi{ 
        public static void main(String[] args){
                if(args.length < 1)
                long iter = Long.parseLong(args[0]);
                double pi = calculate(iter);
                System.out.println(String.format("after %d iterations, pi is %f", iter, pi ));  
        public static double calculate(long simNum){
                long i =0L;
                long count =0L;
                        double x = Math.random();
                        double y = Math.random();
                        double result = x*x+y*y;
                        if(result < 1.0D){
                return 4.0D* count/simNum;


lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 1000
after 1000 iterations, pi is 3.032000
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 10000
after 10000 iterations, pi is 3.137200
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 100000
after 100000 iterations, pi is 3.140240
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 1000000
after 1000000 iterations, pi is 3.144108
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 10000000
after 10000000 iterations, pi is 3.141199
lucas@lucas-ubuntu:~/dev/workspace/pi$ java Pi 100000000
after 100000000 iterations, pi is 3.141486


When iteration is set to 100,000,000, it takes about half a minute on my i7 Core Mac Pro with precision only to 3 digits.

  • This algorithm is unacceptable.
  • The precision cannot be guranteed as it is using a probabilistic way.

READING: Pragmatic Programmer

Pragmatic Programming: from journeyman to master.

Android Theme and Style

Android ListView Advanced


ListView is one of the most fundamental and important Android UI components. Due to the long rectangle shape of mobile device, ListView is widely used as the first level navigation to the more detailed content. Popular using scenarios include, but not limited to, news reader, reading list, e-commere product list, things to do list, ranking list, etc. If your applicaiton intends to providing updatable contents to users, you cannot avoid using this super powerful widgets.

ListView can be very simple and also extremely complex. There are many traps and tricks for novice developers. In this article, the author will summerize some typical traps developers may occur during their use of ListView and try to provide solutions or suggestions. In more general, the author will summerize typical advanced usages of ListView, such as pull to refresh ListView, HTTP cache, data binding, loader, etc.

Simple Example

Android ApiDemos Sample provide a bundle of examples for using ListView (under android-sdk-macosx/samples/android-17/ApiDemos/src/com/example/android/apis/view).

The following is a simple example.

  • /Users/lucas/dev/eclipse-juno-workspace/AdvancedListView/src/com/example/android/apis/view/advancedlistview/

import android.os.Bundle;
import android.widget.ArrayAdapter;
import android.widget.ListAdapter;
import android.widget.ListView;

public class SimpleListViewActivity extends Activity {

	protected void onCreate(Bundle savedInstanceState) {
		ListView lv = (ListView) this.findViewById(;

		ListAdapter adapter = new ArrayAdapter<String>(this,
				android.R.layout.simple_list_item_1, COUNTRIES);


	private static String[] COUNTRIES = new String[] { "CHINA", "US", "JAPAN" };
  • /Users/lucas/dev/eclipse-juno-workspace/AdvancedListView/res/layout/activity_simple_listview.xml
<RelativeLayout xmlns:android=""
    tools:context=".MainActivity" >

        android:layout_height="wrap_content" />


This is a very simple example of ListView.

Adapter Architecture

From the above simple example, we find 3 components for the ListView: the standard MVC pattern. Model, View, and Controller.


ListView is embeded in Activity in layout XML.


To provide data for each item in the ListView, we need a Adapter. Here we use ListAdapter, constructed from a static String array.


If you want users to interact with your app, you can add Listeners for user actions, for example, click event, touch event. This can be done by setting setOnItemClickListener.

Framework API

Official Tutorials

Non-Official Tutorials

API Explored





Header & Footer

Load Data Async

Data Binding



GroupList/ SectionList

Resources on Github


  1. Occasionally, you get the IllegalStateException error, stating

    “The content of the adapter has changed but “ + “ListView did not receive a notification. Make sure the content of “ + “your adapter is not modified from a background thread, but only from “ + “the UI thread. Make sure your adapter calls notifyDataSetChanged() “ + “when its content changes. “

By checking the source code, we find that when the ListView’s ‘mItemCount’ is different from the bound adapter’s count, framework will throw the exception.

Here ‘mItemCount’ is written when the adapter is first time set or when ‘onMeasure’ is called (refresh).

<img src=”imgs/listview_itemcount_write.png”.

            // Handle the empty set by removing all views that are visible
            // and calling it a day
            if (mItemCount == 0) {
            } else if (mItemCount != mAdapter.getCount()) {
                throw new IllegalStateException("The content of the adapter has changed but "
                        + "ListView did not receive a notification. Make sure the content of "
                        + "your adapter is not modified from a background thread, but only from "
                        + "the UI thread. Make sure your adapter calls notifyDataSetChanged() "
                        + "when its content changes. [in ListView(" + getId() + ", " + getClass()
                        + ") with Adapter(" + mAdapter.getClass() + ")]");

  1. Performance Tips for Android’s ListView
  2. [Google I/O 2010 - The world of ListView]<iframe width="560" height="315" src="//" frameborder="0" allowfullscreen></iframe>


Android Fragment

What is fragment stack?

There can be multiple fragments in a single Activity/ Screen.

What is transacation in fragment operation and why should there be the concept of transacion?

How to go back to the previous Fragment?

Compare fragment stack and activity stack.

  • change waiting progress bar if has already loaded .

W/Trace ( 7306): Unexpected value from nativeGetEnabledTags: 0


Wiki entry

Read all articles on CyanogenMod:qq

vim tutor

Vim provides a great tutorial tool called vim tutor. You can open it by using vim tutor or vimtutor.

By following the tutorial, you wil get a basic idea how to use vim more efficiently in 20-30 minutes.

Baiscally, vim command has the grammar:

operator times motion

Here, operator means the operation you want to do: like delete, and times is a integer that represents how many times your want to repeat the operation. motion is the sementics you want the operation does, like w, which represent a word, and e, end of a word.

so d5w mean delete the following 5 words. If operator is missing, it means moving by default. So 5w means moving the cursor by 5 words. There are 2 special motions: $, representing the end of the line and 0, representing the beginning of the line. Typing $ will take your cursor to the end of the line and d$ will delete from the cursor to the end of the line.

Due to the high frequency of deleting the whole line when editing in vim, deleting the whole line is simply dd. To delete multiple lines, use 5dd. You can use d5d, but that takes you longer time to type.


  • delete: d dw de
  • change: c ce
  • replace: r rx

Go to to line: 78 G, uppercase G

  • Search: /abc n for next N for previous
  • find matching parenthesis: %

visualize the block: v v, then move the cursor to hightlight the selected text.

Next, execute: operator number motion.

vimrc config: with VI improved features.

vim ~/.vimrc :r $VIMRUNTIME/vimrc_example.vim

Google Oauth 2.0





Mobile App Authentication




Maven with Eclipse


  1. 安装Maven
  2. 安装Eclipse
  3. 安装m2e
mvn archetype:generate -DgroupId={project-packaging} -DartifactId={project-name} -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false

To convert a maven project to a eclipse project

mvn eclipse:eclipse -Dwtpversion=2.0

Domestic Ads Players

# 调查国内的广告商 写一篇survey

Android Booting Process

Linux Booting

  • boot loader
  • boot
  • system
  • init.rc

Android Booting

  • Linux
  • init.rc
  • Dalvik
  • System server


  1. Android boot process
  2. General Android boot process
  3. Boot of Linux on Embedded Devices
    • Linux U-boot:
    • OMAP bring-up:
    • eLinux boot time:
  4. The Android boot image: boot.img
    • Unpack, re-pack boot image:

Configure ant for Android projects

There are 3 config files generated by android update project or android updatel ib-project.

lucas-mac:apf-framework lucas$ cat 
# This file is automatically generated by Android Tools.
# Do not modify this file -- YOUR CHANGES WILL BE ERASED!
# This file must *NOT* be checked into Version Control Systems,
# as it contains information specific to your local configuration.

# location of the SDK. This is only used by Ant
# For customization when using a Version Control System, please read the
# header note.

	The file can be created by you. It is only edited by the
     'android' tool to add properties to it.
     This is the place to change some Ant specific build properties.
     Here are some properties you may want to change/update:

         The name of the source directory. Default is 'src'.
         The name of the output directory. Default is 'bin'.

     For other overridable properties, look at the beginning of the rules
     files in the SDK, at tools/ant/build.xml

     Properties related to the SDK location or the project target should
     be updated using the 'android' tool with the 'update' action.

     This file is an integral part of the build system for your
     application and should be checked into Version Control Systems.

lucas-mac:apf-framework lucas$ cat 
# This file is automatically generated by Android Tools.
# Do not modify this file -- YOUR CHANGES WILL BE ERASED!
# This file must be checked in Version Control Systems.
# To customize properties used by the Ant build system edit
# "", and override values to adapt the script to your
# project structure.
# To enable ProGuard to shrink and obfuscate your code, uncomment this (available properties: sdk.dir, user.home):

# Project target.

Default build.xml generated by android tool

lucas-ma:apf-framework lucas$ cat build.xml 
<?xml version="1.0" encoding="UTF-8"?>
<project name="apf-framework" default="help">

    <!-- The file is created and updated by the 'android' tool.
         It contains the path to the SDK. It should *NOT* be checked into
         Version Control Systems. -->
    <property file="" />

    <!-- The file can be created by you. It is only edited by the
         'android' tool to add properties to it.
         This is the place to change some Ant specific build properties.
         Here are some properties you may want to change/update:

             The name of the source directory. Default is 'src'.
             The name of the output directory. Default is 'bin'.

         For other overridable properties, look at the beginning of the rules
         files in the SDK, at tools/ant/build.xml

         Properties related to the SDK location or the project target should
         be updated using the 'android' tool with the 'update' action.

         This file is an integral part of the build system for your
         application and should be checked into Version Control Systems.

    <property file="" />

    <!-- if sdk.dir was not set from one of the property file, then
         get it from the ANDROID_HOME env var.
         This must be done before we load since
         the proguard config can use sdk.dir -->
    <property environment="env" />
    <condition property="sdk.dir" value="${env.ANDROID_HOME}">
        <isset property="env.ANDROID_HOME" />

    <!-- The file is created and updated by the 'android'
         tool, as well as ADT.

         This contains project specific properties such as project target, and library
         dependencies. Lower level build properties are stored in
         (or in .classpath for Eclipse projects).

         This file is an integral part of the build system for your
         application and should be checked into Version Control Systems. -->
    <loadproperties srcFile="" />

    <!-- quick check on sdk.dir -->
            message="sdk.dir is missing. Make sure to generate using 'android update project' or to inject it through the ANDROID_HOME environment variable."

        Import per project custom build rules if present at the root of the project.
        This is the place to put custom intermediary targets such as:
            -post-compile (This is typically used for code obfuscation.
                           Compiled code location: ${out.classes.absolute.dir}
                           If this is not done in place, override ${out.dex.input.absolute.dir})
    <import file="custom_rules.xml" optional="true" />

    <!-- Import the actual build file.

         To customize existing targets, there are two options:
         - Customize only one target:
             - copy/paste the target into this file, *before* the
               <import> task.
             - customize it to your needs.
         - Customize the whole content of build.xml
             - copy/paste the content of the rules files (minus the top node)
               into this file, replacing the <import> task.
             - customize to your needs.

         ****** IMPORTANT ******
         In all cases you must update the value of version-tag below to read 'custom' instead of an integer,
         in order to avoid having your file be overridden by tools such as "android update project"
    <!-- version-tag: 1 -->
    <import file="${sdk.dir}/tools/ant/build.xml" />


Mobile Advertising

# Read a series of articles on Wikipedia. Get a introductroy sensation about mobile advertising and its fundamental concepts. Mobile advertising

Build Android Source

On Mac OSX


To build the latest source in a Mac OS environment, you will need an Intel/x86 machine running MacOS 10.6 (Snow Leopard) or MacOS 10.7 (Lion), along with Xcode 4.2 (Apple’s Developer Tools). Although Lion does not come with a JDK, it should install automatically when you attempt to build the source. ‘


  1. Creating a case-sensitive disk image
    # hdiutil create -type SPARSE -fs 'Case-sensitive Journaled HFS+' -size 40g ~/android.dmg
  2. add the following to your ~/.bash_profile to mount the image when you execute “mountAndroid”:
    # mount the android file image
    function mountAndroid { hdiutil attach ~/android.dmg -mountpoint /Volumes/android; } 
  3. Setting a file descriptor limit On MacOS the default limit on the number of simultaneous file descriptors open is too low and a highly parallel build process may exceed this limit. To increase the cap, add the following lines to your ~/.bash_profile:
    # set the number of open files to be 1024
    ulimit -S -n 1024
  4. Download the source

Android Build System

Android build system is make based.

In this document Understanding the makefile Layers Building the Android Platform Building the Android Kernel Build Variants


Understanding the makefile

A makefile defines how to build a particular application. Makefiles typically include all of the following elements:

  1. Name: Give your build a name (LOCAL_MODULE := ).
  2. Local Variables: Clear local variables with CLEAR_VARS (include $(CLEAR_VARS)).
  3. Files: Determine which files your application depends upon (LOCAL_SRC_FILES := main.c).
  4. Tags: Define tags, as necessary (LOCAL_MODULE_TAGS := eng development).
  5. Libraries: Define whether your application links with other libraries (LOCAL_SHARED_LIBRARIES := cutils).
  6. Template file: Include a template file to define underlining make tools for a particular target (include $(BUILD_EXECUTABLE)).

《Android 内核剖析》 第18章 Android 编译系统

  1. Make based 目标:条件 .PHONY 声明目标

  2. Android 源码文件结构

    drwxr-xr-x    6 lucas  staff   306 Mar  4 12:37 .repo
    -r--r--r--    1 lucas  staff    87 Oct  1 00:56 Makefile
    drwxr-xr-x    3 lucas  staff   136 Oct 15 08:17 abi
    drwxr-xr-x   10 lucas  staff   476 Oct 15 08:17 bionic
    drwxr-xr-x    5 lucas  staff   170 Mar  1 13:23 bootable
    drwxr-xr-x    7 lucas  staff   442 Mar  4 12:00 build
    drwxr-xr-x   11 lucas  staff   612 Mar  4 12:00 cts
    drwxr-xr-x   17 lucas  staff   782 Mar  4 12:00 dalvik
    drwxr-xr-x   18 lucas  staff   714 Nov 25 17:30 development
    drwxr-xr-x   11 lucas  staff   374 Mar  4 11:29 device
    drwxr-xr-x    3 lucas  staff   136 Oct 15 08:17 docs
    drwxr-xr-x  164 lucas  staff  5610 Mar  4 12:01 external
    drwxr-xr-x   15 lucas  staff   510 Jan 25 14:49 frameworks
    drwxr-xr-x   10 lucas  staff   442 Oct  1 01:01 gdk
    drwxr-xr-x   12 lucas  staff   408 Nov 25 17:32 hardware
    drwxr-xr-x   12 lucas  staff   680 Mar  4 12:01 libcore
    drwxr-xr-x    4 lucas  staff   306 Jan 28 02:08 libnativehelper
    drwxr-xr-x    8 lucas  staff   714 Jan 28 02:08 ndk
    drwxr-xr-x    4 lucas  staff   238 Mar  7 21:57 out
    drwxr-xr-x    8 lucas  staff   272 Nov 25 17:33 packages
    drwxr-xr-x    5 lucas  staff   238 Nov 25 17:33 pdk
    drwxr-xr-x   11 lucas  staff   374 Mar  4 12:00 prebuilts
    drwxr-xr-x   53 lucas  staff  2006 Jan 28 02:09 sdk
    drwxr-xr-x    9 lucas  staff   306 Oct  1 01:04 system
    drwxr-xr-x    3 lucas  staff   102 Mar  4 12:00 tools

Make 系统



  1. 编译整个Android系统 ```bash source ./build/ make PRODUCT-ful-crespo-eng
1. 编译子工程
make libbz
or mmm external bzip2

第19章 Android 系统编译

Android NDK

Android NDK Development

NDK is short for Native Development Kit. It does not benefit for all tasks. Typical good candidates for the NDK are self-contained, CPU-intensive operations that don’t allocate much memory, such as signal processing, physics simulation, and so on.

Add /Users/lucas/dev/android/android-ndk-r8d to ~/.bash_profile

export PATH=${PATH}:/Users/lucas/dev/android/android-ndk-r8d/

Here’s the general outline of how you work with the NDK tools:

  1. Place your native sources under /jni/...
  2. Create /jni/ to describe your native sources to the NDK build system
  3. Optional: Create /jni/
  4. Build your native code by running the ‘ndk-build’ script from your project’s directory. It is located in the top-level NDK directory: cd /ndk-build
  5. The build tools copy the stripped, shared libraries needed by your application to the proper location in the application’s project directory. Finally, compile your application using the SDK tools in the usual way. The SDK build tools will package the shared libraries in the application’s deployable .apk file.

On ARM emualtor

root@android:/data/data/com.example.hellojni/lib # ls
root@android:/data/data/com.example.hellojni/lib # 

Running JNI app on Intel Emulator.

# pwd
# ls

I/ActivityManager( 1013): START {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=com.example.hellojni/.HelloJni u=0} from pid 1151
D/dalvikvm( 1013): GC_FOR_ALLOC freed 376K, 9% free 12167K/13255K, paused 6ms, total 8ms
D/dalvikvm( 1800): Not late-enabling CheckJNI (already on)
I/dalvikvm( 1800): Turning on JNI app bug workarounds for target SDK version 3...
I/ActivityManager( 1013): Start proc com.example.hellojni for activity com.example.hellojni/.HelloJni: pid=1800 uid=10071 gids={1015, 1028}
E/Trace   ( 1800): error opening trace file: No such file or directory (2)
W/dalvikvm( 1800): Exception Ljava/lang/UnsatisfiedLinkError; thrown while initializing Lcom/example/hellojni/HelloJni;
W/dalvikvm( 1800): Class init failed in newInstance call (Lcom/example/hellojni/HelloJni;)
D/AndroidRuntime( 1800): Shutting down VM
W/dalvikvm( 1800): threadid=1: thread exiting with uncaught exception (group=0xb3fb7288)
E/AndroidRuntime( 1800): FATAL EXCEPTION: main
E/AndroidRuntime( 1800): java.lang.ExceptionInInitializerError
E/AndroidRuntime( 1800):  at java.lang.Class.newInstanceImpl(Native Method)
E/AndroidRuntime( 1800):  at java.lang.Class.newInstance(
E/AndroidRuntime( 1800):  at
E/AndroidRuntime( 1800):  at
E/AndroidRuntime( 1800):  at
E/AndroidRuntime( 1800):  at$600(
E/AndroidRuntime( 1800):  at$H.handleMessage(
E/AndroidRuntime( 1800):  at android.os.Handler.dispatchMessage(
E/AndroidRuntime( 1800):  at android.os.Looper.loop(
E/AndroidRuntime( 1800):  at
E/AndroidRuntime( 1800):  at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime( 1800):  at java.lang.reflect.Method.invoke(
E/AndroidRuntime( 1800):  at$
E/AndroidRuntime( 1800):  at
E/AndroidRuntime( 1800):  at dalvik.system.NativeStart.main(Native Method)
E/AndroidRuntime( 1800): Caused by: java.lang.UnsatisfiedLinkError: Couldn't load hello-jni: findLibrary returned null
E/AndroidRuntime( 1800):  at java.lang.Runtime.loadLibrary(
E/AndroidRuntime( 1800):  at java.lang.System.loadLibrary(
E/AndroidRuntime( 1800):  at com.example.hellojni.HelloJni.<clinit>(
E/AndroidRuntime( 1800):  ... 15 more
W/ActivityManager( 1013):   Force finishing activity com.example.hellojni/.HelloJni
D/dalvikvm( 1013): GC_FOR_ALLOC freed 43K, 7% free 12376K/13255K, paused 7ms, total 7ms
W/ActivityManager( 1013): Activity pause timeout for ActivityRecord{b4919ef8 com.example.hellojni/.HelloJni}
I/ActivityManager( 1013): No longer want (pid 1375): hidden #16
I/Process ( 1800): Sending signal. PID: 1800 SIG: 9
I/ActivityManager( 1013): Process com.example.hellojni (pid 1800) has died.
W/InputMethodManagerService( 1013): Window already focused, ignoring focus gain of:$Stub$Proxy@b49f4280 attribute=android.view.inputmethod.EditorInfo@b4b3d0f0
W/ActivityManager( 1013): Activity destroy timeout for ActivityRecord{b4919ef8 com.example.hellojni/.HelloJni}

Sample “NativeActivity”

On ARM emulator

W/dalvikvm(  956): threadid=1: thread exiting with uncaught exception (group=0x40a70930)
E/AndroidRuntime(  956): FATAL EXCEPTION: main
E/AndroidRuntime(  956): java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.native_activity/}: java.lang.IllegalArgumentException: Unable to find native library: native-activity
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at$600(
E/AndroidRuntime(  956):  at$H.handleMessage(
E/AndroidRuntime(  956):  at android.os.Handler.dispatchMessage(
E/AndroidRuntime(  956):  at android.os.Looper.loop(
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime(  956):  at java.lang.reflect.Method.invoke(
E/AndroidRuntime(  956):  at$
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at dalvik.system.NativeStart.main(Native Method)
E/AndroidRuntime(  956): Caused by: java.lang.IllegalArgumentException: Unable to find native library: native-activity
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  at
E/AndroidRuntime(  956):  ... 11 more
W/ActivityManager(  285):   Force finishing activity com.example.native_activity/
W/Trace   (  402): Unexpected value from nativeGetEnabledTags: 0
D/dalvikvm(  285): GC_FOR_ALLOC freed 467K, 21% free 11175K/14060K, paused 89ms, total 90ms