Algorithm Review

Algorithm Review

Stack

Queue

Heap

SORT

Math for algorithms

排序算法的商减 (entropy decrease)

Math for algorithms

Math basics for algorithm

A.1 求和公式及其性质

• 等差 $\sum_{k=1}^n k = \frac { n(n+1)}{2}$

$\sum_{k=1}^n k^2 = \frac {n(n+1)(2n+1)}{6}$  $\sum_{k=1}^n k^3 = \frac { n^2(n+1)^2}{4}$

• 线性性质

$\sum_{k=1}^n (ca_k+b_k) = c\sum_{k=1}^n a_k + \sum_{k=1}^n b_k$ $\sum_{k=1}^n \Theta\left(f(k)\right) = \Theta \left( \sum_{k=1}^n f(k) \right)$

• 几何级数

• 调和级数

• 级数积分与微分

• 列项级数 telescopes

$\sum_{k=0}^{n-1} (a_{k-1} - a_k) = a_0 - a_n$ $\sum_{k=1}^{n-1} \frac{1}{k(k+1)} = \sum_{k=1}^{n-1} (\frac{1}{k} - \frac{1}{k+1}) = 1- \frac{1}{n}$

• 乘积 $\lg\left(\prod_{k=1}^{n} a_k \right) = \sum_{k=1}^{n} \lg{a_k}$

A.2 确定求和时间的界

• 数学归纳法
• 确定级数中各项的界
• $$\sum_{k=1}^{\infty} \frac{1}{k} = \lim _{n \to \infty} {\sum_{k=1}^{n} \frac{1}{k} } = \lim _{n \to \infty} {\Theta(\lg n)} = \infty$$

• 通过积分求和的近似

  • if $f(x)$ 单调递增: $\int_{m-1}^{n} f(x)\,dx \leq \sum_{k=m}^n f(k) \leq \int_{m}^{n+1} f(x)\,dx$

  • if $f(x)$ 单调递减: $\int_{m}^{n+1} f(x)\,dx \leq \sum_{k=m}^n f(k) \leq \int_{m-1}^{n} f(x)\,dx$

  $H_n = \sum_{k=1}^n \frac{1}{n} \geq \int_{1}^{n+1} \frac{1}{x}\,dx = \ln{(n+1)}$ $\sum_{k=2}^n \frac{1}{n} \leq \int_{1}^{n} \frac{1}{x}\,dx = \ln{(n)}$

  $$H_n = \sum_{k=1}^n \frac{1}{n} \leq \ln{n}+1$$

重点

• Illustrate function call stack
• Describe how to convert recursion to iteration
  • one function call
  • 2 or more function calls
  • tail recursion
• PERMUTATION

PERM(A, k)
  if(k == n)
    PRINT(A)
  for i = 1 to n 
    SWAP(A, i, k)
    PERM(A, k+1)
    SWAP(A, i, k)


PERM(A, 1)

• DFS and stack
• BFS and queue

vue-router

• History

API

核心是使用 HTML window.history API

• vm
  • $options
    • _parentVnode
    • $router
    • $route

Vue.util.defineReactive(

• window
  • location
    • ancestorOrigins: DOMStringList {length: 0}
    • assign: ƒ ()
    • hash: “#html”
    • host: “router.vuejs.org”
    • hostname: “router.vuejs.org”
    • href: “https://router.vuejs.org/guide/#html”
    • origin: “https://router.vuejs.org”
    • pathname: “/guide/”
    • port: “”
    • protocol: “https:”
    • reload: ƒ reload()
    • replace: ƒ ()
    • search: “”
    • toString: ƒ toString()
  • history
  • performance
    • Performance Timeline API,
    • the High Resolution Time API,
    • the Navigation Timing API,
    • the User Timing API,
    • the Resource Timing API.

因为安全的问题， 不能获取 history stack 列表， 只能 push pop 操作。

vue-cli

vue-cli

Guide

type

• app
• library

• wc

• @vue/cli
• @vue/cli-service
• @vue/cli-plugin-
• vue-cli-plugin-
• Development
  • Browser Compatibility
    • browserslist
    • Polyfills
  • HTML 和静态资源
    • index file public/index.html
    • interpolation with loadash
      • e.g <link rel="icon" href="<%= BASE_URL %>favicon.ico">
      • <%= VALUE %>
      • <%- VALUE %>
      • <% expression %>
    • preload <link rel="preload">
    • prefetch <link rel="prefetch">
    • Disable Index Generation
    • Building a Multi-Page App
    • Static Assets Handling
    • Relative Path Imports
    • URL Transform Rules
      • absolute: /images/foo.png
      • relative: ./
      • module: <img src="~some-npm-package/foo.png">
      • module: <img src="@some-npm-package/foo.png">, interpret to <projectRoot>/src
    • The public Folder

        <link rel="icon" href="<%= BASE_URL %>favicon.ico">
      
        data () {
            return {
                baseUrl: process.env.BASE_URL
            }
        }       
                  
        <img :src="`${baseUrl}my-image.png`">
      
      

  • CSS 相关
  • webpack 相关
  • Enviroment Variables and Modes
  • Build Targets
  • Deployments

    Config Reference

• Global Config
  • ~/.vuerc
  • vue config
• Target Browsers
• vue.config.js
  • loaded by @vue/cli-service
  • or by vue section in package.json

      module.exports = {
          baseUrl: '/my-app/' or './',
          outputDir: 'dist',
          assetsDir: '',
          indexPath: 'index.html',
          filenameHashing: false,
          pages: undefined,
          lintOnSave: 'true',
          lintOnSave: process.env.NODE_ENV !== 'production',
          runtimeCompiler: 'false',
          transpileDependencies:[], // 转译 node_modules/ 中的依赖
          productionSourceMap: true,
          crossorigin: undefined,
          integrity: false,
          configureWebpack: {
              plugins: [
              new MyAwesomeWebpackPlugin()
              ]
          },
          chainWebpack: '',
          css.modules: false,
          css.extract: false,
          css.sourceMap: false,
          css: {
              loaderOptions: {
                  css: {
                      // 这里的选项会传递给 css-loader
                  },
                  postcss: {
                      // 这里的选项会传递给 postcss-loader
                  }
              }
          },
          devServer: {
              proxy: 'http://localhost:4000'
          },
          parallel: true,
          pwa: {},
          pluginOptions: {
              foo: {
              // 插件可以作为 `options.pluginOptions.foo` 访问这些选项。
              }
          },
          chainWebpack: config => {
              config.plugins.delete('html')
              config.plugins.delete('preload')
              config.plugins.delete('prefetch')
          }
    
    
                
    
    
    
    
    
    
    
                
      }
    

• Babel
  • @vue/babel-preset-app
  • babel.config.js
  • Polyfills

• ESLint @vue/cli-plugin-eslint

  • .eslintrc
  • eslintConfig in package.json
• TypeScript @vue/cli-plugin-typescript
  • tsconfig.json
• Unit Testing
  • Jest
  • Mocha
• E2E Testing
  • Cypress
  • Nightwatch

Plugin Dev Guide

Plugins

Q & A

1. what is vue-loader, babel-loader, lint-loader etc. ?
2. Javascript moduel, moduleA.js? moduleB/index.js?

   See Learn the basics of the JavaScript module system and build your own library

   Telling stories is as basic to human beings as eating. More so, in fact, for while food makes us live, stories are what make our lives worth living - Richard Kearney

   • tools
     • CommonJS
     • AMD
     • UMD
     • ES6 (Harmony)
   • Best Practices
     • Tree shaking: import , export
     • Webpack vs Rollup vs Babel?
     • webpack:
       • code splitting,
       • async loading of bundles,
       • tree shaking,
     • UI Libraries have a dist folder that has the bundled and minified version for ES and UMD/CJS module systems as a target. There is a lib folder that has the transpiled version of the library.
     • Core Packages have just one folder which has the bundled and minified version for CJS or UMD module system as a target.

Cascading Style Sheets

• ECMAScript 6 Primer
  • 第1章 ECMAScript 6 简介
  • 第2章 let 和 const 命令
    • 2.1 let
    • 2.2 const
    • 顶层对象
  • 第 3 章 变量的解构赋值 Destructuring assignment
    • 3.1 数组的 Destructuring assignment
    • 3.2 对象的解构赋值
    • Map
  • TODO:
  • Q & A

CSS

• Syntax
  • 
• selectors
  • select by id #id
  • select by element type p {text-align: center; color: red;}
  • select by class: .center
  • combined: p.center
  • group:

      h1 {
          text-align: center;
          color: red;
      }
    
      h2 {
          text-align: center;
          color: red;
      }
    
      p {
          text-align: center;
          color: red;
      }
    
      h1, h2, p {
          text-align: center;
          color: red;
      }
    

  • comments:

      p {
          color: red;
          /* This is a single-line comment */
          text-align: center;
      }
    
      /* This is
      a multi-line
      comment */
    

• refer to style sheet:
  • External style sheet

      <head>
      <link rel="stylesheet" type="text/css" href="mystyle.css">
      </head>
    

  • Internal style sheet

      <head>
      <style>
      body {
          background-color: linen;
      }
    
      h1 {
          color: maroon;
          margin-left: 40px;
      } 
      </style>
      </head>
    

  • Inline style

      <h1 style="color:blue;margin-left:30px;">This is a heading</h1>
    

  • Cascading Order: all the styles will “cascade” into a new “virtual” style sheet by the following rules:
    • Inline style (inside an HTML element)
    • External and internal style sheets (in the head section)
    • Browser default
• Colors
  • Background Color: background-color:Tomato;
  • text color: color:MediumSeaGreen;
  • border: border:2px solid Violet;
  • Color values:
    • rgba(255, 99, 71, 0.5), or hex: #ee82ee
    • hsla(9, 100%, 64%, 0.5)
      • Hue is a degree on the color wheel from 0 to 360. 0 is red, 120 is green, and 240 is blue.
      • Saturation is a percentage value, 0% means a shade of gray, and 100% is the full color.
      • Lightness is also a percentage, 0% is black, 50% is neither light or dark, 100% is white
    • Shades of gray are often defined using equal values for all the 3 light sources: rgb(180, 180, 180) hsl(0, 0%, 0%)

• backgrounds

    body {
        background-color: lightblue;
        background-image: url("paper.gif");
        background-repeat: repeat-x|no-repeat;
        background-attachment: fixed;
        background-position: right top;
        background-size: 300px 100px;
        background-size: auto|length|cover|contain|initial|inherit;
  
    }
  

  short:

    background: #ffffff url("img_tree.png") no-repeat right top;
    background: bg-color bg-image position/bg-size bg-repeat bg-origin bg-clip bg-attachment initial|inherit;
  
  

  When using the shorthand property the order of the property values is:

  • background-color
  • background-image
  • background-repeat
  • background-attachment
  • background-position
• Borders

    p.one {
        border-style: solid;
        border-top-style: dotted;
        border-right-style: solid;
        border-bottom-style: dotted;
        border-left-style: solid;
        border-width: 5px;
        border-color: red;
        border: 5px solid red;
        border-radius: 5px;
        border-collapse: separate|collapse|initial|inherit;
  
    }
  
  
  

• Margins
• Paddings
• Height/Width
• Box Model
• Outline
• Text

    h1 {
        color: blue;
        text-align: left|right|center|justify|initial|inherit;
        text-decoration: text-decoration-line text-decoration-color text-decoration-style|initial|inherit;
        text-decoration-line: none|underline|overline|line-through|initial|inherit;
        text-transform: uppercase|lowercase|capitalize;
        text-indent: 50px;
        letter-spacing: 3px;
        line-height: 0.8;
        direction: rtl;
        word-spacing: 10px;
        text-shadow: 3px 2px red;
  
  
  
    }
  

• Fonts
• Icons
• Links
• Lists
• Tables
• Display
  • Block
  • inline
  • display: none|inline|block;
• max-width
• postion

    position: static|relative|fixed|absolute|sticky
  

Deep Learning

Deep Learning

$z_j^{l+1} = \sum_{k=1} (w_{jk}^la_k^l) + b_j^{l+1}$ $z= w \cdot a + b$

Cost for sample $x$ $C_x= \frac 1 2 \sum_{j=1}^m (y_j-a_j)^2$

total Cost:

the objective function: $min(C), w.t. w_{jk}, b_j$

define: $\delta^l_j \equiv \frac {\partial C} {\partial z_j^l} \\$

$z_j^{l+1} = \sum_{k=1} (w_{jk}^l\sigma(z_k^l) + b_j^{l+1} \\ \frac {\partial z_j^{l+1}} {\partial z_k^{l}} = w_{jk} \cdot \sigma^{\prime} = w_{jk}^{l+1} \cdot \sigma(z_k^{l})(1-\sigma(z_k^{l}))$ to z $$ \frac {\partial C_x} {\partial z_k^l} = \sum_j { \frac {\partial C_x} {\partial z_j^{l+1} } \cdot \frac {\partial z_j^{l+1}} {\partial z_k^{l}} }\

\delta^l_k = \sum_j \delta^{l+1}_j \cdot {\frac {\partial z_j^{l+1}} {\partial z_k^{l} }}

$Output layer:$ \delta^L_j \equiv \frac {\partial C} {\partial z_j^L}
=\frac {\partial C} {\partial a_j^L} \cdot \sigma^{\prime}(z_j^L)
= ( a_j^L - y_j) \cdot \sigma(z_j^L) \cdot (1 - \sigma(z_j^L)) $$

matrix based: $\delta^L \equiv \frac {\partial C} {\partial z^L} \\ = \nabla_aC \odot {\sigma^{\prime}(z^L)} \\ = ( a^L - y) \odot {\sigma^{\prime}(z^L)}$ back propagation:

$$

\delta^l_k = \sum_j {\delta^{l+1}_k}

\delta^L = \nabla_aC \odot {\sigma^{\prime}(z^L)} \

\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot {\sigma^{\prime}(z^l)} \

$$

with $w_{jk}$ $$ \nabla_{b^l}C = \delta^l
\nabla_{w^l}C = \delta^l \sigma^{l-1} \

$$

ES6标准入门

• ECMAScript 6 Primer
  • 第1章 ECMAScript 6 简介
  • 第2章 let 和 const 命令
    • 2.1 let
    • 2.2 const
    • 顶层对象
  • 第 3 章 变量的解构赋值 Destructuring assignment
    • 3.1 数组的 Destructuring assignment
    • 3.2 对象的解构赋值
    • Map
  • TODO:
  • Q & A

ECMAScript 6 Primer

Date: 20180904

第1章 ECMAScript 6 简介

第2章 let 和 const 命令

2.1 let

• 类似 var，但是又 locality
• 代码块内有效

    {
        let a = 10; // 超出此代码块无效, 局部变量
        var b = 5;  // 代码块外有效, 作用域: 函数 + 全局
    }
  

• 变量提升
• 暂时性死去 temporal dead zone (TDZ)
• 不允许重复声明
• block level domain
• do

    let x = do {
        let t = f();
        t * t + 1;
    }
  

  2.2 const

  定义常量

  const PI = 3.1415
  const a = []
  a.push('Hello')
  

• 声明变量的6中方法
  • var
  • function
  • let
  • const
  • import
  • class

顶层对象

• 浏览器: window
• Node / web workder: global
• this

第 3 章 变量的解构赋值 Destructuring assignment

模式匹配

3.1 数组的 Destructuring assignment

let [firstName, surname] = arr;

function fibs() {
    let a = 0;
    let b = 1;
    whiile (true) {
        yield a; 
        [a, b] = [b, a+b];
    }
}
let [first, second, third, fourth, fifth, sixth] = fibs();
sixth

3.2 对象的解构赋值

let jsonData ={
    id: 42,
    status: 'OK',
    data: [867, 5309]
};
let {id, status, data: number } = jsonData;

Map

var map = new Map();
map.set('first', 'hello');
map.set('second', 'world');
for(let [key, value] of map){
    console.log(key + ' is ' value);
}

TODO:

• 第 4 章: 字符串的扩展

Q & A

• javascript 中 ; 使用
  • Stackoverflow: Function expression ending with ; vs. not

    At it’s root, what you have there is an assignment statement, and according to the Google Javascript Style Guide, all statements should end in a ;. Especially assignment statements. JSLint requires ; or else it won’t pass.

• javascript prototype
  • 动态向 javascript object 添加属性, 每个函数都有一个prototype属性，这个属性是指向一个对象的引用，这个对象称为原型对象，原型对象包含函数实例共享的方法和属性，也就是说将函数用作构造函数调用（使用new操作符调用）的时候，新创建的对象会从原型对象上继承属性和方法。

    function employee(name,job,born)
    {
        this.name=name;
        this.job=job;
        this.born=born;
    }
  
    var bill=new employee("Bill Gates","Engineer",1985);
  
    employee.prototype.salary=null;
    bill.salary=20000;
  
  

How Music Works

How Music Works

Note

Physics

Frequency Hertz Amplitude Wavelength Speed (1.4) Musical Instrument Tone

There is a huge variety of musical instruments and sounds, as you would already know from your experience with music. Even two instruments playing the same note can sound very different.

This is because a musical instrument produces a sound wave which is a combination of different but related frequencies (known as harmonics) which all mix together to create the distinctive tone or voice of the instrument.

The lowest frequency is usually dominant, and you perceive this one as the pitch. The combination of the other harmonics provides the distinctive shape of the waveform, and thereby the distinctive tone of the instrument.

String instrument frequency determined by the length of the string. Guitar http://hyperphysics.phy-astr.gsu.edu/hbase/Waves/string.html

Mathematics of Music

https://www.lightnote.co/ https://ianring.com/musictheory/scales/#citation2 https://www.ams.jhu.edu/dan-mathofmusic/ https://news.ycombinator.com/item?id=12792063 https://quariety.com/2018/04/28/a-laymans-intro-to-western-classical-music/ https://www.soundslice.com/scores/auld-lang-syne/

MIT Music21 project

Music21 is a set of tools for helping scholars and other active listeners answer questions about music quickly and simply. If you’ve ever asked yourself a question like, “I wonder how often Bach does that” or “I wish I knew which band was the first to use these chords in this order,” or “I’ll bet we’d know more about Renaissance counterpoint (or Indian ragas or post-tonal pitch structures or the form of minuets) if I could write a program to automatically write more of them,” then music21 can help you with your work.

• Installation:
  • libs

      brew install portaudio 
      pip3 install matplotlib numpy scipy pyaudio pygame pillow
    

  • MuseScore
  • Finale or Finale NotePad
  • Lilypond

Music Theory

Piano key frequencies

• scientific notation

    Eb4, C#,,, C4, d#'', Ab2
  

• Helmholtz

  libraries

  tone.js music synthesis,

  Teoria.js for theory

• Note
• Chord
• Scale
• Interval

Web Audio API

Event Driven Design

Introduction

Observer Pattern

Centralized Event Management

Decouple event from event handling

Implementation

Queue Single Thread, queue, watch

Operating System, Kernel mode, user mode

select / epoll model

GUI Design

Updating UI in dedicated thread. Single Thread UI / main thread

epoll

Windows

Android

AsyncTask

Vue

vm._events[event] stores an array of listener functions: (vm._events[event] || (vm._events[event] = [])).push(fn)

• vm.$emit
• vm.$on
• vm.$once( event, callback )
• vm.$off( [event, callback] )

// node_modules/vue/src/core/instance/events.js
export function eventsMixin (Vue: Class<Component>) {
  const hookRE = /^hook:/
  Vue.prototype.$on = function (event: string | Array<string>, fn: Function): Component {
    const vm: Component = this
    if (Array.isArray(event)) {
      for (let i = 0, l = event.length; i < l; i++) {
        this.$on(event[i], fn)
      }
    } else {
      (vm._events[event] || (vm._events[event] = [])).push(fn)
      // optimize hook:event cost by using a boolean flag marked at registration
      // instead of a hash lookup
      if (hookRE.test(event)) {
        vm._hasHookEvent = true
      }
    }
    return vm
  }

Javascript Event

https://developer.mozilla.org/en-US/docs/Web/API/Event

Java Event

In Java, we implement event using interface, See https://docs.oracle.com/javase/tutorial/uiswing/events/intro.html

Event Hierarchy

Centralized management

sub event

distributed management

register

pub/sub

Programming Models and Phylosophy

A Core Pattern for Events Adila A. Krisnadhi1;2 and Pascal Hitzler1 1 Wright State University, OH, USA 2 Universitas Indonesia, Depok, Indonesia

Refernces:

• https://github.com/mspnp/architecture-center/blob/master/docs/patterns/event-sourcing.md

AMAP高德 地图

适用平台： Web Android iOS Web Service

位置产品

1. 地图
2. 定位
3. 导航
4. 路线规划
5. 搜索
6. 室内地图
7. 室内定位

自定义地图 位置数据可视化 海外LBS服务

Java SPI

SPI Service Provider Interface

checkout github repo: https://github.com/xianminx/hellospi

Spring mongo data naming

Mongo Naming

JSON Naming

Google JSON Style Guide

https://publicobject.com/2016/01/20/strict-naming-conventions-are-a-liability/ Gson has built-in magic to convert snakes into camels with its LOWER_CASE_WITH_UNDERSCORES FieldNamingPolicy. We make a global configuration change to our Gson instance and the problem is solved.

Java Naming

Spring data mongo

https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#repositories.query-methods.query-property-expressions

8.4.3. Property Expressions
Property expressions can refer only to a direct property of the managed entity, as shown in the preceding example. At query creation time, you already make sure that the parsed property is a property of the managed domain class. However, you can also define constraints by traversing nested properties. Consider the following method signature:

List<Person> findByAddressZipCode(ZipCode zipCode);
Assume a Person has an Address with a ZipCode. In that case, the method creates the property traversal x.address.zipCode. The resolution algorithm starts by interpreting the entire part (AddressZipCode) as the property and checks the domain class for a property with that name (uncapitalized). If the algorithm succeeds, it uses that property. If not, the algorithm splits up the source at the camel case parts from the right side into a head and a tail and tries to find the corresponding property — in our example, AddressZip and Code. If the algorithm finds a property with that head, it takes the tail and continues building the tree down from there, splitting the tail up in the way just described. If the first split does not match, the algorithm moves the split point to the left (Address, ZipCode) and continues.

Although this should work for most cases, it is possible for the algorithm to select the wrong property. Suppose the Person class has an addressZip property as well. The algorithm would match in the first split round already, choose the wrong property, and fail (as the type of addressZip probably has no code property).

To resolve this ambiguity you can use \_ inside your method name to manually define traversal points. So our method name would be as follows:

List<Person> findByAddress_ZipCode(ZipCode zipCode);
Because we treat the underscore character as a reserved character, we strongly advise following standard Java naming conventions (that is, not using underscores in property names but using camel case instead).

Mongo References:

• manual references
• DBRef

Manual references are named after the referenced collection. Use the document type (collection name in singular) followed by _id ( _id ). This is the only case where you can use underscore in the middle. For example, in a dogs collection, a document would have manual references to external documents named like this: { name: 'fido', owner_id: '5358e4249611f4a65e3068ab', race_id: '5358ee549611f4a65e3068ac', colour: 'yellow' ... }

https://docs.mongodb.com/manual/reference/database-references/#document-references

Mongo follows javascript naming convention using camelCase

Mongo has no preference on camelCase v.s. underscore_style just stay consistent with the language. But also be consistent with on-wire json/xml format, also consider for Android/ iOS / Javascript web consumer sides.

http://christophermaier.name/2011/05/22/MongoDB-key-names/

different to relational database, in mongoldb, key names are stored in every document, so avoid using long names.

db.collection.stats()

using shorter keys can potentially save a lot of space and even increase performance (after all, more of a smaller database can fit into memory). WiredTiger engine in Mongo 3.0 supports compression, the point is rather moot.

http://arkusnexus.com/2016/09/12/coding-guidelines-mongodb/

GENERAL GUIDELINES

• Use tabs to indent. This applies to all MongoDB-specific code (chained functions) and objects used by MongoDB (queries, projections, documents).
• Always have a space after a : colon.
• Comma-last. If you divide the components of an object/array into various lines, use one line for each component. The } closing curly brace should follow the last component (except for aggregation). GENERAL GUIDELINES
• No _ underscores in the middle of names (database, collection, fields) except for manual references.
• Collections database, variables, properties and function names should uselowerCamelCase. They should also be descriptive. Single character variables and uncommon abbreviations should generally be avoided.
• Place spaces between nested parentheticals and elements in JavaScript examples. For example, prefer { [ a, a, a ] } over {[a,a,a]}.

COLLECTION NAMES

• The name should be a plural of the types of documents to be saved.
• Use camelCase. Normally you shouldn’t have to because the collection name will be one word (plural of the types of documents saved).
• A collection with “” empty string is not a valid collection name.
• A collection name should not contain the null character because this defines the end of collection name.
• Collection name should not start with the prefix “system.” as this is reserved for internal collections.

• It would be good to not contain the character “$” in the collection name as various drivers available for database do not support “$” in collection name. DATABASE NAMES

• Try to have the database named after the project and one database per project.
• Use camelCase.
• A database with “” empty string is not a valid database name.
• Database name cannot be more than 64 bytes.
• Database name are case-sensitive, even on non-case-sensitive file systems. Thus it is good to keep name in lower case.

• A database name cannot contain any of these characters “/, \, ., “, *, <, >, :, |, ?, $,”. It also cannot contain a single space or null character. FIELD NAMES

• Use camelCase.
• Don’t use _ underscore as the starting character of the field name. The only field with _ undescore should be _id.

• Field names cannot contain . dots or null characters and must not start with a $ dollar sign. Manual references are named after the referenced collection. Use the document type (collection name in singular) followed by _id ( _id ). This is the only case where you can use underscore in the middle. For example, in a dogs collection, a document would have manual references to external documents named like this: { name: 'fido', owner_id: '5358e4249611f4a65e3068ab', race_id: '5358ee549611f4a65e3068ac', colour: 'yellow' ... } FUNCTIONS

• One method per line should be used when chaining methods.
• You should also indent these methods with a tab so it’s easier to tell they are part of the same chain. If you need to use a long query, projection or options object for a function, assign it to a var and use the var in the function call to improve readability.

vue-element-admin

• axios
• mock.js
  • random
• ES2015+
• vuex
• vue-router

axios

Promise based HTTP client for the browser and node.js

mock.js

create project and download deps.

vue create hello_mockjs
cd hello_mockjs 
npm install mockjs

npm run serve

• 数据模板定义规范 DTD ` ‘name|rule’: value `

random

Random is a useful nodejs commandline / library for generating random objects, like url, email, etc.

npm install random -g 

➜  hello_mockjs git:(master) ✗ random

  Usage: random [options] [command]

  Options:

    -V, --version  output the version number
    -h, --help     output usage information

  Commands:

    boolean        Random.boolean( min,  max,  cur )
    bool           Random.bool( min,  max,  cur )
    natural        Random.natural( min,  max )
    integer        Random.integer( min,  max )
    int            Random.int( min,  max )
    float          Random.float( min,  max,  dmin,  dmax )
    character      Random.character( pool )
    char           Random.char( pool )
    string         Random.string( pool,  min,  max )
    str            Random.str()
    range          Random.range( start,  stop,  step )
    date           Random.date( format )
    time           Random.time( format )
    datetime       Random.datetime( format )
    now            Random.now( unit,  format )
    image          Random.image( size,  background,  foreground,  format,  text )
    img            Random.img()
    color          Random.color( name )
    hex            Random.hex()
    rgb            Random.rgb()
    rgba           Random.rgba()
    hsl            Random.hsl()
    paragraph      Random.paragraph( min,  max )
    cparagraph     Random.cparagraph( min,  max )
    sentence       Random.sentence( min,  max )
    csentence      Random.csentence( min,  max )
    word           Random.word( min,  max )
    cword          Random.cword( pool,  min,  max )
    title          Random.title( min,  max )
    ctitle         Random.ctitle( min,  max )
    first          Random.first()
    last           Random.last()
    name           Random.name( middle )
    cfirst         Random.cfirst()
    clast          Random.clast()
    cname          Random.cname()
    url            Random.url( protocol,  host )
    protocol       Random.protocol()
    domain         Random.domain( tld )
    tld            Random.tld()
    email          Random.email( domain )
    ip             Random.ip()
    region         Random.region()
    province       Random.province()
    city           Random.city( prefix )
    county         Random.county( prefix )
    zip            Random.zip( len )
    d4             Random.d4()
    d6             Random.d6()
    d8             Random.d8()
    d12            Random.d12()
    d20            Random.d20()
    d100           Random.d100()
    guid           Random.guid()
    uuid           Random.uuid()
    id             Random.id()
  Examples:

    $ random date yyyy-MM-dd
    $ random time HH:mm:ss

ES2015+

vuex

每一个 Vuex 应用的核心就是 store（仓库）。“store”基本上就是一个容器，它包含着你的应用中大部分的状态 (state)。

• reactive in store state vars.
• commit mutation

vue-router

Questions

• import request from '@/utils/request' what is @?
• requests interceptor 中 set login token
• response interceptor 中验证 response 错误

• arrow functions

  sidebar: state => state.app.sidebar,
  

• store
  • getters
  • store modules

    const store = new Vuex.Store({
      modules: {
        app,
        user
      },
      getters
    })
    
    export default store
    

• export function
• router

  {
    path: '/',
    component: Layout,
    redirect: '/dashboard',
    name: 'Dashboard',
    hidden: true,
    children: [{
      path: 'dashboard',
      component: () => import('@/views/dashboard/index')
    }]
  },
  

• vuex
  • getters
  • mapGetters
  • mutations

CSS

Box model

</img>

border-width:
border-top-width:
border-bottom-width:
border-left-width: 
border-right-width:

box-sizing: content-box/ border-box

line-width: normal/1.6/30px/0.8em

margin
margin-top
margin-bottom: 2px/3em/10%/auto
margin-left
margin-right

height:
max-height: none/2000px
max-width:
min-height: 0/200px
min-width:

padding
padding-top:
padding-bottom:
padding-left:
padding-right:

width: auto/240px/50%



overflow: auto/hidden/visible
background-cli: border-box/padding-box/content-box
display: block/inline/inline-block/positioned/table/flex

• You can use unitless values: the line height will be relative to the font size. line-height: 1.6;
• You can use pixel values. line-height: 30px;
• You can use em values: like with unitless values, the line height will be relative to the font size. line-height: 0.8em;

Flexbox

Before the Flexbox Layout module, there were four layout modes:

• Block, for sections in a webpage
• Inline, for text
• Table, for two-dimensional table data
• Positioned, for explicit position of an element

flexbox
align-content: stretch/flex-start/flex-end/center/space-between/space-around/
align-items: flex-start/flex-end/center/baseline/stretch
align-self: auto/flex-start/flex-end/center/baseline/stretch
flex-basis: auto/80px/
flex-direction: row/row-reverse/column/column-reverse
flex-flow
flex-grow: 0/1
flex-shrink: 0/1
flex-wrap: nowrap/wrap/wrap-reverse
justify-conent: flex-start/flex-end/center/space-between/space-around
order: 0/1/-1/9

Grid

Node.js development stack

• Node.js development stack
  • Node.js
    • Modules
    • Node.js and MySQL
    • Node.js and MongoDB
    • package.json
  • Learn ES2015
• Vue.js
  • Hello world
  • HTML 渲染
  • Component
    • Organizing components
    • SFC
  • Vue class / instance
    • Options/*
  • API
    • global config
    • Global API
    • Options/data
    • events
  • Questions
  • javascript context/ scope/ this
  • Vue-cli
    • UI
    • vue-cli-service
  • MISCS
  • vue-loader
  • AMD v.s. CommonJS v.s. UMD
  • MISC
  • - vuex
  • MISC

Node.js development stack

• ES2015+
• vue
• vuex
• vue-router
• axios
• element-ui
• Mock.js

Node.js

https://www.w3schools.com/nodejs/default.asp

• Node.js uses asynchronous programming!
• Node.js can generate dynamic page content
• Node.js can create, open, read, write, delete, and close files on the server
• Node.js can collect form data
• Node.js can add, delete, modify data in your database

Modules

• What is a Module in Node.js?

  Consider modules to be the same as JavaScript libraries.A set of functions you want to include in your application.

• built-in modules

• Include Modules

  To include a module, use the require() function with the name of the module:

    var http = require('http');
  

• Create Your Own Modules

    // save as mymodule.j* 
    exports.myDateTime = function () {
        return Date();
    };
  

    // in another main.js
    var mymodule = require('./mymodule')
    var datetime = mymodule.myDateTime()
  

• HTTP Module

    var http = require('http');
  
    //create a server object:
    http.createServer(function (req, res) {
    res.write('Hello World!'); //write a response to the client
    res.end(); //end the response
    }).listen(8080);
  

  • Add an HTTP Header

    res.writeHead(200, {'Content-Type': 'text/html'});

  • Read the Query String

    res.write(req.url);

  • Split the Query String

      var q = url.parse(req.url, true).query;
      var txt = q.year + " " + q.month;
    
• File System Module
  • Node.js as a File Server

      var fs = require('fs');
    

  • Read Files

      fs.readFile('demofile1.html', function(err, data) {
    

  • Create Files

      var fs = require('fs');
    
      fs.appendFile('mynewfile1.txt', 'Hello content!', function (err) {
          if (err) throw err;
          console.log('Saved!');
      });
    

  • Delete Files

      var fs = require('fs');
    
      fs.unlink('mynewfile2.txt', function (err) {
      if (err) throw err;
      console.log('File deleted!');
      });
    
    

  • rename files

      fs.rename('mynewfile1.txt', 'myrenamedfile.txt', function (err) {
    

  • Upload Files

      var formidable = require('formidable');
    

• Send an Email
  • nodemailer module var nodemailer = require('nodemailer');
  • multiple receivers
  • html
• Events module
  • addListener()
  • defaultMaxListeners
  • emit()
  • eventNames()
  • getMaxListeners()
  • listenerCount()
  • listeners()
  • on()
  • once()
  • prependListener()
  • prependOnceListener()
  • removeAllListeners()
  • removeListener()
  • setMaxListeners()

    var events = require('events');
    var eventEmitter = new events.EventEmitter();
  
    eventEmitter.on('scream', function() {
    console.log('A scream is detected!');
    });
    eventEmitter.emit('scream');
  

• NPM
  • download: npm install upper-case --save
  • use: var uc = require('upper-case')
• URL

    var url = require('url');
    var adr = 'http://localhost:8080/default.htm?year=2017&month=february';
    var q = url.parse(adr, true);
  
    console.log(q.host); //returns 'localhost:8080'
    console.log(q.pathname); //returns '/default.htm'
    console.log(q.search); //returns '?year=2017&month=february'
  
    var qdata = q.query; //returns an object: { year: 2017, month: 'february' }
    console.log(qdata.month); //returns 'february'
  

Node.js and MySQL

Node.js and MongoDB

• install driver: npm install mongodb --save
• var mongo = require('mongodb');
• create collections

    var MongoClient = require('mongodb').MongoClient;
    var url = "mongodb://localhost:27017/";
  
    MongoClient.connect(url, function(err, db) {
        if (err) throw err;
        var dbo = db.db("mydb");
        dbo.createCollection("customers", function(err, res) {
            if (err) throw err;
            console.log("Collection created!");
            db.close();
        });
  
        var myobj = { name: "Company Inc", address: "Highway 37" };
        dbo.collection("customers").insertOne(myobj, function(err, res) {
            if (err) throw err;
            console.log("1 document inserted");
            db.close();
        });
          
    });
  

package.json

• package.json

  The package.json file is core to the Node.js ecosystem and is a basic part of understanding and working with Node.js, npm, and even modern JavaScript. The package.json is used as what equates to a manifest about applications, modules, packages, and more - it’s a tool to that’s used to make modern development streamlined, modular, and efficient.

  • name
  • version
  • description
  • license
  • keywords
  • dependencies
  • devDependencies
  • “main”: “app.js”,
  • repository
  • scripts

Learn ES2015

https://old.babeljs.io/learn-es2015/

• Arrows and Lexical This

Vue.js

Hello world

1. import: <script src="https://unpkg.com/vue"></script>
2. HTML ```html

3. javascript

```js
var app = new Vue({
  el: '#app',
  data: {
    message: 'hello, world'
  }

})

HTML 中用一个 mount point <div id="app">, 在 javascript 创建一个 Vue 示例。

HTML 渲染

1. 文本插值: ``
2. v-bind：:title e.g. ` `
3. v-on: @click 事件处理 ` 逆转消息`
4. 条件: v-if
5. 循环： v-for
6. 用户输入： v-model: ` `

Component

// 定义名为 todo-item 的新组件
Vue.component('todo-item', {
  template: '<li>这是个待办项</li>'
})

Instead, a component’s data option must be a function, so that each instance can maintain an independent copy of the returned data object:

data: function () {
  return {
    count: 0
  }
}

Organizing components

• registration:
  • global
    • Vue.component('name', opts)
    • Globally registered components can be used in the template of any root Vue instance (new Vue) created afterwards – and even inside all subcomponents of that Vue instance’s component tree.
  • local
• Props
  • props is like interface to class, and data is used to maitain interal state, as private fields to class.

• SFC

Vue.

• Passing Data to Child Components with Props
• Sending Messages to Parents with Events
• v-model
• Content Distribution with Slots
• Dynamic Components <component v-bind:is="currentTabComponent"></component>

Vue class / instance

var vm = new Vue({

})

Options/*

1. el: 锚点
2. data: 要绑定的数据
3. computed: 复杂属性
4. props: 对外的属性接口，类似 C# /Android 中 view 等属性。
5. methods: 方法

API

global config

• silent
• optionMergeStrategies
• devtools
• errorHandler
• warnHandler
• ignoredElements
• keyCodes
• performance
• productionTip

Global API

• Vue.extend(options)
• Vue.nextTick([callback, context])
• Vue.set(target, key, value)
• Vue.delete(target, key)
• Vue.directive(id, [definition])
• Vue.filter(id, [definition])
• Vue.component(id, [definition])
• Vue.use(plugin)
• Vue.mixin(mixin)
• Vue.compile(template)
• Vue.version

Options/data

• data
• props: A list/hash of attributes that are exposed to accept data from the parent component.
• propsData: initial / default data for props
• computed:

events

• $emit

synthetic events Creating and triggering events

vm.$emit( eventName, […args] )

Questions

1. what is javascript prototype?

   fdf

2. what is javascript context this?

   The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

javascript context/ scope/ this

• context/ this: function

context means this keyword

Context is most often determined by how a function is invoked. When a function is called as a method of an object, this is set to the object the method is called on:

So for a global function, this refers to ‘window’ in browser

• scope: variable
  • local
  • Global
  • block: let/ const

  Understanding Scope and Context in JavaScript

• Execution Context

• Execution stack

  The answer is short and simple: Scope pertains to the visibility of variables, and context refers to the object within which a function is executed.

Vue-cli

https://cli.vuejs.org/

new 3.x is @vue/cli, the old is vue-cli

• instant prototyping

    vue serve
  

• Create a project

    vue create
  

• Plugins
• Presets
• CLI service

UI

• UI template

    vue ui
  

vue-cli-service

➜  hello-vue git:(master) ✗ ./node_modules/.bin/vue-cli-service 

  Usage: vue-cli-service <command> [options]

  Commands:

    serve     start development server
    build     build for production
    inspect   inspect internal webpack config
    lint      lint and fix source files

  run vue-cli-service help [command] for usage of a specific command.

• Using the Binary
• vue-cli-service serve
• vue-cli-service build
• vue-cli-service inspect
• Checking All Available Commands
• Caching and Parallelization
• Git Hooks
• Configuration without Ejecting
• webpack-dev-server

MISCS

• UMD
• AMD
• CommonJS is a project with the goal of specifying an ecosystem for JavaScript outside the browser (for example, on the server or for native desktop applications).

vue-loader

是一个 webpack 的 loader，可以将用下面这个格式编写的 Vue 组件转换为 JavaScript 模块：

AMD v.s. CommonJS v.s. UMD

https://www.davidbcalhoun.com/2014/what-is-amd-commonjs-and-umd/

AMD v.s. CommonJS

Sandboxed

Asynchronous Module Definition

• AMD <- RequireJS

    //    filename: foo.js
    define(['jquery', 'underscore'], function ($, _) {
        //    methods
        function a(){};    //    private because it's not returned (see below)
        function b(){};    //    public because it's returned
        function c(){};    //    public because it's returned
  
        //    exposed public methods
        return {
            b: b,
            c: c
        }
    });
  

• CommonJS <- Browserify

    //    filename: foo.js
    var $ = require('jquery');
    var _ = require('underscore');
  
    //    methods
    function a(){};    //    private because it's omitted from module.exports (see below)
    function b(){};    //    public because it's defined in module.exports
    function c(){};    //    public because it's defined in module.exports
  
    //    exposed public methods
    module.exports = {
        b: b,
        c: c
    };
  

• UMD: The pattern is admittedly ugly, but is both AMD and CommonJS compatible, as well as supporting the old-style “global” variable definition:

    (function (root, factory) {
        if (typeof define === 'function' && define.amd) {
            // AMD
            define(['jquery'], factory);
        } else if (typeof exports === 'object') {
            // Node, CommonJS-like
            module.exports = factory(require('jquery'));
        } else {
            // Browser globals (root is window)
            root.returnExports = factory(root.jQuery);
        }
    }(this, function ($) {
        //    methods
        function myFunc(){};
  
        //    exposed public method
        return myFunc;
    }));
  

MISC

• js === v.s. ==

##

vuex

vuex 集中管理应用中的状态。 类似 Java Spring 中的配置中心。 一种管理的哲学思想 commit 的方式相当于 会隐藏了 setOnListener 的调用, 类似 set 方法， 而不是直接访问状态字段

• state
• mutations
• 触发变化 store.commit('increment')
• 调用状态: 计算属性

   computed: {
    count () {
      return store.state.count
    }
  }
  

• 单一状态树， 唯一数据源 SSOT, 状态快照

So actions/mutations allow you to:

• centrally define allowed changes to the state, and by that,
• how those changes are connected and logically depend on one another (i.e. by reading an action that commits a series of mutations)
• track these changes in the devtools very easily, as you see all the Mutation names you just read in the list there.

• writing expressive code that, to an extend, documents your business logic. You can understand a lot about an app’s business logic by just reading its store actions & mutations, without digging through hundreds of components looking where some piece of state is altered under which circumstances.

• mapGetter 类同 Java Bean 中的自动生成 get 函数
• mapMutations 类同 Java Bean 中自动生成 set 函数
• 可以手写 get / set 函数， 以定制化
  • get: getters:
  • set: mutations:
  • mutaions = event + event handler (type + handler)
  • computed: mapState
• Actions
  • …mapActions

MISC

• setInterval
• clearInterval
• setTimeout clearTimeout

Spring Jackson

Jackson Exceptions – Problems and Solutions

JsonMappingException UnrecognizedPropertyException

Spring Mongo

• root Spring Data’s MongoDB abstraction.
• .config Spring XML namespace configuration for MongoDB specific repositories.
• .core MongoDB core support.
• .core.aggregation Support for the MongoDB aggregation framework.
• .core.convert Spring Data MongoDB specific converter infrastructure.
• .core.geo Support for MongoDB geo-spatial queries.
• .core.index Support for MongoDB document indexing.
• .core.mapping Infrastructure for the MongoDB document-to-object mapping subsystem.
• .core.mapping.event Mapping event callback infrastructure for the MongoDB document-to-object mapping subsystem.
• .core.mapreduce Support for MongoDB map-reduce operations.
• .core.query MongoDB specific query and update support.
• .core.script Abstraction classes javascript function execution within MongoDB Server.
• .core.spel Support classes to transform SpEL expressions into MongoDB expressions.
• .crossstore Infrastructure for Spring Data’s MongoDB cross store support.
• .gridfs Support for MongoDB GridFS feature.
• .monitor MongoDB specific JMX monitoring support.
• .repository MongoDB specific repository implementation.
• .repository.cdi CDI support for MongoDB specific repository implementation.
• .repository.config Support infrastructure for the configuration of MongoDB specific repositories.
• .repository.query Query derivation mechanism for MongoDB specific repositories.
• .repository.support Support infrastructure for query derivation of MongoDB specific repositories.
• .util MongoDB driver-specific utility classes for Bson and DBObject interaction.

修改错误

{
    "timestamp": "2018-06-08T06:47:54.222+0000",
    "status": 500,
    "error": "Internal Server Error",
    "message": "Could not extract response: no suitable HttpMessageConverter found for response type [class com.google.gson.JsonObject] and content type [text/plain]",
    "path": "/auth/login/wechat"
}

LetsEncrypt

https://certbot.eff.org/lets-encrypt/ubuntutrusty-nginx

Install LetsEncrypt

root@open2:/etc/nginx/sites-enabled# ls
city.opentown.cn  opentown.cn
root@open2:/etc/nginx/sites-enabled# git diff city.opentown.cn
root@open2:/etc/nginx/sites-enabled# vim city.opentown.cn 
root@open2:/etc/nginx/sites-enabled# certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: opentown.cn
2: adunk.opentown.cn
3: api.opentown.cn
4: cityapi.opentown.cn
5: m.opentown.cn
6: www.opentown.cn
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 4
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for cityapi.opentown.cn
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/city.opentown.cn

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://cityapi.opentown.cn

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=cityapi.opentown.cn
-------------------------------------------------------------------------------

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/cityapi.opentown.cn/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/cityapi.opentown.cn/privkey.pem
   Your cert will expire on 2018-09-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Ethereum and Solidity

Tooling

• Remix
• npm / Node.js

web3j web3.js

Introduction to Smart Contracts

Solidity 是 Ethereum 上智能合约实现的主要语言之一。 运行在 EVM 上。

• static typing
• variadic return types
• Complex member variables for contracts including arbitrarily hierarchical mappings and structs
• Contracts support inheritance, including multiple inheritance with C3 linearization.
• An application binary interface (ABI) facilitating multiple type-safe functions within a single contract was also introduced (and later supported by Serpent).
• A documentation system for specifying a user-centric description of the ramifications of a method-call was also included in the proposal, known as “Natural Language Specification”.
• on EVM
• compile to bytecode
• designed around ECMAScript

Blockchain Basics

• Transactions
• Blocks
• EVM
  • Accounts: external accounts controlled by public private key pairs
  • contract accounts controlled by the code stored together with the account.
  • Every account has a persistent key-value store mapping 256-bit words to 256-bit words called storage.
• Storage:a key-value store that maps 256-bit words to 256-bit words. It is not possible to enumerate storage from within a contract and it is comparatively costly to read and even more so, to modify storage. A contract can neither read nor write to any storage apart from its own.
• Memory: a contract obtains a freshly cleared instance for each message call. Memory is linear and can be addressed at byte level, but reads are limited to a width of 256 bits, while writes can be either 8 bits or 256 bits wide. Memory is expanded by a word (256-bit), when accessing (either reading or writing) a previously untouched memory word (ie. any offset within a word). At the time of expansion, the cost in gas must be paid. Memory is more costly the larger it grows (it scales quadratically).
• Stack
• Instruction Set
• Message Calls
• Delegatecall / Callcode and Libraries
• Logs
• Create
• self-destruct

Installing the Solidity Compiler

Solidity by Example

Solidity in Depth

Security Considerations

Using the compiler

Contract Metadata

Application Binary Interface Specification

Joyfully Universal Language for (Inline) Assembly

Style Guide

Common Patterns

List of Known Bugs

Contributing

Frequently Asked Questions

##

Gloabal variable

Ether Units

• wei,
• finney,
• szabo or
• ether

Time Units

• 1 == 1 seconds
• 1 minutes == 60 seconds
• 1 hours == 60 minutes
• 1 days == 24 hours
• 1 weeks == 7 days
• 1 years == 365 days

Globals

• block.blockhash(uint blockNumber) returns (bytes32): hash of the given block - only works for 256 most recent blocks excluding current
• block.coinbase (address): current block miner’s address
• block.difficulty (uint): current block difficulty
• block.gaslimit (uint): current block gaslimit
• block.number (uint): current block number
• block.timestamp (uint): current block timestamp as seconds since unix epoch
• gasleft() returns (uint256): remaining gas
• msg.data (bytes): complete calldata
• msg.gas (uint): remaining gas - deprecated in version 0.4.21 and to be replaced by gasleft()
• msg.sender (address): sender of the message (current call)
• msg.sig (bytes4): first four bytes of the calldata (i.e. function identifier)
• msg.value (uint): number of wei sent with the message
• now (uint): current block timestamp (alias for block.timestamp)
• tx.gasprice (uint): gas price of the transaction
• tx.origin (address): sender of the transaction (full call chain)

Error Handling

• assert(bool condition): throws if the condition is not met - to be used for internal errors.
• require(bool condition): throws if the condition is not met - to be used for errors in inputs or external components.
• revert(): abort execution and revert state changes

Mathematical and Cryptographic Functions

• addmod(uint x, uint y, uint k) returns (uint):
• mulmod(uint x, uint y, uint k) returns (uint):
• keccak256(…) returns (bytes32):
• sha256(…) returns (bytes32):
• sha3(…) returns (bytes32):
• ripemd160(…) returns (bytes20):
• ecrecover(bytes32 hash, uint8 v, bytes32 r, bytes32 s) returns (address):

Adddress Related

• <address>.balance (uint256):
• <address>.transfer(uint256 amount):
• <address>.send(uint256 amount) returns (bool):
• <address>.call(...) returns (bool):
• <address>.callcode(...) returns (bool):
• <address>.delegatecall(...) returns (bool):

Contract Related

• this
• selfdestruct(address recipient)
• suicide(address recipient)

Doxity: Documentation Generator for Solidity.

All identifiers (contract names, function names and variable names) are restricted to the ASCII character set. It is possible to store UTF-8 encoded data in string variables.

• public: The keyword public automatically generates a function that allows you to access the current value of the state variable from outside of the contract. Without this keyword, other contracts have no way to access the variable.
• address: 160 bit value
• mapping: Mappings can be seen as hash tables which are virtually initialized such that every possible key exists and is mapped to a value whose byte-representation is all zeros.
• progma once
• Coin.balances.call(arg)

How to develop and debug?

EVM

• 256bit register stack
• e-WASM
• send()
• transfer()
• call.value()

ERC20

pragma solidity ^0.4.23;


// https://github.com/ethereum/EIPs/issues/20
interface ERC20 {
    function totalSupply() external view returns (uint supply);
    function balanceOf(address _owner) external view returns (uint balance);
    function transfer(address _to, uint _value) external returns (bool success);
    function transferFrom(address _from, address _to, uint _value) external returns (bool success);
    function approve(address _spender, uint _value) external returns (bool success);
    function allowance(address _owner, address _spender) external view returns (uint remaining);
    function decimals() external view returns(uint digits);
    event Approval(address indexed _owner, address indexed _spender, uint _value);
}

import java.util.HashMap;
import java.util.Map;

public class ERC20Coin {

    static class Address {
        private String val;
        private static final String PATTERN_ERC20_HASH = "0x[0-9A-Fa-f]{64}";

        public Address(String val) {
            assert (val.matches(PATTERN_ERC20_HASH));
            this.val = val;
        }

        public String toString() {
            return this.val;
        }

        @Override
        public boolean equals(Object obj) {
            return (obj instanceof Address)
                    && ((Address) obj).val.equals(this.val);
        }
    }

    static class TransferEvent{
        public TransferEvent(Address _from, Address _to, Integer _value){

        }
    }

    Address sender;


    String name;
    String symbol;
    long totalSupply;
    Integer demicals;

    Map<Address, Integer> balances;
    Map<Address, Map<Address, Integer>> allowances;


    public ERC20Coin(String name, String symbol, long totalSupply, Integer demicals){
        balances = new HashMap<>();
        allowances = new HashMap<>();
        this.name = name;
        this.symbol = symbol;
        this.totalSupply = totalSupply;
        this.demicals = demicals;
    }

    public Integer balanceOf(Address address) {
        return balances.get(address);
    }

    public void transfer(Address from, Address to, Integer val) {
        Integer balanceFrom = balances.getOrDefault(from, 0);
        Integer balanceTo = balances.getOrDefault(to, 0);

        Map<Address, Integer> allowance;
        if(!allowances.containsKey(from)){
            allowance = new HashMap<Address, Integer>();
        }else{
            allowance = allowances.get(from);
        }
        Integer allowed = allowance.get(sender);
        assert  (allowance.get(sender) >= val) &&
                (balanceFrom >= val);

        if(allowed < Integer.MAX_VALUE){
            allowance.put(sender, allowed - val);
            allowances.put(from, allowance);
        }

        balances.put(from, balanceFrom - val);
        balances.put(to, balanceTo + val);
    }

    public void approve(Address from, Address to, Integer val){
        Map<Address, Integer> allowance;
        if(!allowances.containsKey(from)){
            allowance = new HashMap<Address, Integer>();
        }else{
            allowance = allowances.get(from);
        }

        allowance.put(to, val);
        allowances.put(from, allowance);
    }



}

Airdrop 空投

参考: https://medium.com/bitfwd/how-to-airdrop-your-first-token-3ba187ff7cbf

空投地址列表

空投之前需要找到你需要空投地址列表。 如何找到这些地址列表呢？

1. 在 Slack 或者 telegram 等群里寻找鱼儿。
2. 扫描 Ethereum 上 balance >0 的那些有效账号地址

parity export state --chain=ropsten -no-storage -no-code -min-balance=1000 -max-balance=500000 -at=2000000 balances.json

节省 gas 费用

如何一次向多个账号send token?

Spring Security

Servelet Interceptor

authenticationManager, accessDecisionManager, securityMetadataSource

UserDetailsService接口和其loadUserByUsername方法

1）首先我们自定义一个过滤器（调度器，这里我们命名为mySecurityInterceptor），这个过滤器继承AbstractSecurityInterceptor类（这里先说明，本文但凡不是自定义的类或接口都是Spring Security提供的，无须深究）。 它至少包含 authenticationManager,accessDecisionManager,securityMetadataSource三个属性，我们的所有控制将在这三个类中实现。

（2）登录验证：自定义类MyUserDetailService实现UserDetailsService接口和其loadUserByUsername方法，这个方法根据用户输入的用户名，从数据库里面获取该用户的所有权限细信息（统称用户信息）。Spring Security的AuthenticationProcessingFilter拦截器调用authenticationManager，类MyUserDetailService拿到用户信息后，authenticationManager对比用户的密码（即验证用户），如果通过了，那么相当于通过了AuthenticationProcessingFilter拦截器，也就是登录验证通过。

（3）资源访问控制：MySecurityInterceptor继承AbstractSecurityInterceptor、实现Filter是必须的。登陆后，每次访问资源都会被MySecurityInterceptor这个拦截器拦截，它首先会调用MyFilterInvocationSecurityMetadataSource类的getAttributes方法获取被拦截url所需的权限，在调用MyAccessDecisionManager类decide方法判断用户是否够权限。

可能文字描述还是比较抽象，通过实例应该能让大家更加清楚其原理。

UserDetailsService在身份认证中的作用：

Spring Security中进行身份验证的是AuthenticationManager接口，ProviderManager是它的一个默认实现，但它并不用来处理身份认证，而是委托给配置好的AuthenticationProvider，每个AuthenticationProvider会轮流检查身份认证。检查后或者返回Authentication对象或者抛出异常。

验证身份就是加载响应的UserDetails，看看是否和用户输入的账号、密码、权限等信息匹配。此步骤由实现AuthenticationProvider的DaoAuthenticationProvider（它利用UserDetailsService验证用户名、密码和授权）处理。

因此，登录认证其实可以不实现UserDetailsService，而是实现AuthenticationProvider，然后在AuthenticationProvider里面获取用户输入的用户名和密码进行校验也是可以的。或者两者一起使用。

UsernamePasswordAuthenticationToken

http://up.2cto.com/kf/201103/20110314161540730.png

Python multitasking

• threading
  • Thread
  • Queue
  • Actor
• multiprocessing
  • Pool
• concurrent.futures
  • Executor
  • Difference
  • ThreadPoolExecutor
  • ProcessPoolExecutor
  • Future
  • module functions
• Debugging Python Multi-processes Program
• Take aways
• References

threading

Thread

• Methods:
  • start()
  • run()
  • is_alive()
  • join()

Threads are executed in their own system-level thread (e.g., a POSIX thread or Windows threads) that is fully managed by the host operating system. Once started, threads run independently until the target function returns.

from threading import Thread
t = Thread(target=countdown, args=(10,))
t.start()

Due to a global interpreter lock (GIL), Python threads are restricted to an execution model that only allows one thread to execute in the interpreter at any given time. For this reason, Python threads should generally not be used for computationally intensive tasks where you are trying to achieve parallelism on multiple CPUs. They are much better suited for I/O handling and handling concurrent execution in code that performs blocking operations (e.g., waiting for I/O, waiting for results from a database, etc.).

• Event
  • wait()
  • set()
• Condition
  • wait()
  • notify_all()
• Semaphore

sema = threading.Semaphore(0)
sema.release()

• Lock

self._value_lock = threading.Lock()
with self._value_lock:
        self._value += delta

• RLock

An RLock or re-entrant lock object is a lock that can be acquired multiple times by the same thread. It is primarily used to implement code based locking or synchronization based on a construct known as a “monitor.” With this kind of locking, only one thread is allowed to use an entire function or the methods of a class while the lock is held.

• threading.local()

Queue

Perhaps the safest way to send data from one thread to another is to use a Queue from the queue library. To do this, you create a Queue instance that is shared by the threads. Threads then use put() or get() operations to add or remove items from the queue.

• q.put(data)
• data = in_q.get()
• in_q.task_done()
• q.qsize()
• q.full()
• q.empty()

Actor

https://www.oreilly.com/learning/python-cookbook-concurrency#actors

In a nutshell, an actor is a concurrently executing task that simply acts upon messages sent to it. In response to these messages, it may decide to send further messages to other actors. Communication with actors is one way and asynchronous. Thus, the sender of a message does not know when a message actually gets delivered, nor does it receive a response or acknowledgment that the message has been processed.

multiprocessing

multiprocessing

• Process
• Exchanging objects between processes

Pool

class Pool(object):
    Process = Process

    def __init__(self, processes=None, initializer=None, initargs=(),
                 maxtasksperchild=None):
        self._setup_queues()
        self._taskqueue = Queue.Queue()
        self._cache = {}
        ... # stuff we don't care about
        self._worker_handler = threading.Thread(
            target=Pool._handle_workers,
            args=(self, )
            )
        self._worker_handler.daemon = True
        self._worker_handler._state = RUN 
        self._worker_handler.start()

        self._task_handler = threading.Thread(
            target=Pool._handle_tasks,
            args=(self._taskqueue, self._quick_put, self._outqueue,
                  self._pool, self._cache)
            )
        self._task_handler.daemon = True
        self._task_handler._state = RUN 
        self._task_handler.start()

        self._result_handler = threading.Thread(
            target=Pool._handle_results,
            args=(self._outqueue, self._quick_get, self._cache)
            )
        self._result_handler.daemon = True
        self._result_handler._state = RUN
        self._result_handler.start()

• queues:
  • self._taskqueue = Queue.Queue(): cache all submited tasks
  • self._inqueue = SimpleQueue(): _task_handler fetch task from _taskqueue and add to _inqueue for process to work on, this is shared among main process and worker process.
  • self._outqueue = SimpleQueue(): store the result.
• threads:
  • _worker_handler = _handle_workers(pool): maintain the internal state
  • _task_handler = _handle_tasks(taskqueue, put, outqueue, pool, cache): fetch tasks and send to worker process
  • _result_handler = _handle_results(outqueue, get, cache): process result, callback etc.

Note: callback is called in _result_handler thread in caller process.

concurrent.futures

Executor

• Executor.submit(fn, *args, **kwargs)
• Executor.map(func, *iterables, timeout=None)
• Executor.shutdown(wait=True)

Difference

• Use the ProcessPoolExecutor for CPU intensive tasks.
• The ThreadPoolExecutor is better suited for network operations or I/O.

ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=2)

ProcessPoolExecutor

executor = concurrent.futures.ProcessPoolExecutor(max_workers=None)

"""Implements ProcessPoolExecutor.

The follow diagram and text describe the data-flow through the system:

|======================= In-process =====================|== Out-of-process ==|

+----------+     +----------+       +--------+     +-----------+    +---------+
|          |  => | Work Ids |    => |        |  => | Call Q    | => |         |
|          |     +----------+       |        |     +-----------+    |         |
|          |     | ...      |       |        |     | ...       |    |         |
|          |     | 6        |       |        |     | 5, call() |    |         |
|          |     | 7        |       |        |     | ...       |    |         |
| Process  |     | ...      |       | Local  |     +-----------+    | Process |
|  Pool    |     +----------+       | Worker |                      |  #1..n  |
| Executor |                        | Thread |                      |         |
|          |     +----------- +     |        |     +-----------+    |         |
|          | <=> | Work Items | <=> |        | <=  | Result Q  | <= |         |
|          |     +------------+     |        |     +-----------+    |         |
|          |     | 6: call()  |     |        |     | ...       |    |         |
|          |     |    future  |     |        |     | 4, result |    |         |
|          |     | ...        |     |        |     | 3, except |    |         |
+----------+     +------------+     +--------+     +-----------+    +---------+

Executor.submit() called:
- creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict
- adds the id of the _WorkItem to the "Work Ids" queue

Local worker thread:
- reads work ids from the "Work Ids" queue and looks up the corresponding
  WorkItem from the "Work Items" dict: if the work item has been cancelled then
  it is simply removed from the dict, otherwise it is repackaged as a
  _CallItem and put in the "Call Q". New _CallItems are put in the "Call Q"
  until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because
  calls placed in the "Call Q" can no longer be cancelled with Future.cancel().
- reads _ResultItems from "Result Q", updates the future stored in the
  "Work Items" dict and deletes the dict entry

Process #1..n:
- reads _CallItems from "Call Q", executes the calls, and puts the resulting
  _ResultItems in "Request Q"
"""

Future

• Future.cancel()
• Future.cancelled()
• Future.running()
• Future.result(timeout=None)
• Future.exception(timeout=None)
• Future.add_done_callback(fn)

Internal methods, meant for use in unit tests and Executor implementations.

• Future.set_running_or_notify_cancel()
• Future.set_result(result)
• Future.set_exception(exception)

module functions

• concurrent.futures.wait(fs, timeout=None, return_when=ALL_COMPLETED)
• concurrent.futures.as_completed(fs, timeout=None)

The main difference between the aforementioned map method with as_completed is that map returns the results in the order in which we pass the iterables. That is the first result from the map method is the result for the first item. On the other hand, the first result from the as_completed function is from whichever future completed first.

#Launching a Daemon Process on Unix

Debugging Python Multi-processes Program

Take aways

• Another subtle aspect of pools is that mixing threads and process pools together can be a good way to make your head explode. If you are going to use both of these features together, it is often best to create the process pool as a singleton at program startup, prior to the creation of any threads. Threads will then use the same process pool for all of their computationally intensive work.

• Many programmers, when faced with thread performance problems, are quick to blame the GIL for all of their ills. However, doing so is shortsighted and naive. Just as a real-world example, mysterious “stalls” in a multithreaded network program might be caused by something entirely different (e.g., a stalled DNS lookup) rather than anything related to the GIL. The bottom line is that you really need to study your code to know if the GIL is an issue or not. Again, realize that the GIL is mostly concerned with CPU-bound processing, not I/O.

• Great care should be made when combining process pools and programs that use threads. In particular, you should probably create and launch process pools prior to the creation of any threads (e.g., create the pool in the main thread at program startup).

## References

• Python Cookbook: Concurrency
• concurrent.futures
• Python Concurrency Cheatsheet

TypeScript

TypeScript

TypeScript is a superset for javascript, enabling us to write more elegant javascript code.

TypeScript in 5 minutes

Process

TypeScript –> Javascript –> Nodejs

1. use tsc to transpile TypeScript into JavaScript,
2. then run js by nodejs or browser.

Install

npm install -g typescript
tsc --version
Version 2.7.2

tsc --help

Project management

structure

See typescript101

➜  typescript101 git:(master) ✗ tree
.
├── HelloWorld.js
├── HelloWorld.js.map
├── HelloWorld.ts
├── README.md
├── greeter.html
├── greeter.js
├── greeter.js.map
├── greeter.ts
└── tsconfig.json

With Visual Studio Code

Warning: Visual Studio for Mac Community Edition does not support TypeScript natively. So if on Mac, use Visual Studio Code instead. or Use Visual Studio on Windows for better IDE support.

References:

1. Editing TypeScript in VSCode
2. Simplest TypeScript project using Visual Studio Code
   • File Scope
   • Explicit Project

tsconfig.json

This is the project definition file.

Spring boot

Sping Boot

Dependency Management

The curated list contains all the spring modules that you can use with Spring Boot as well as a refined list of third party libraries. The list is available as a standard Bills of Materials (spring-boot-dependencies) that can be used with both Maven and Gradle.

BUILD FAILED in 4s
➜  NutNetwork git:(master) ✗ ./gradlew dependencies 
/Users/lucas/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.15/1483490805e8f86c21240958629905e733d079eb/log4j-1.2.15.pom:
    'build.plugins.plugin[io.spring.gradle.dependencymanagement.org.apache.maven.plugins:maven-antrun-plugin].dependencies.dependency.scope' for junit:junit:jar must be one of [compile, runtime, system] but is 'test'. in log4j:log4j:1.2.15
+--- org.springframework.boot:spring-boot-starter-data-rest -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot:2.0.0.RELEASE
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE
|    |    |    |    \--- org.springframework:spring-jcl:5.0.4.RELEASE
|    |    |    \--- org.springframework:spring-context:5.0.4.RELEASE
|    |    |         +--- org.springframework:spring-aop:5.0.4.RELEASE
|    |    |         |    +--- org.springframework:spring-beans:5.0.4.RELEASE
|    |    |         |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |         \--- org.springframework:spring-expression:5.0.4.RELEASE
|    |    |              \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.0.0.RELEASE
|    |    |    \--- org.springframework.boot:spring-boot:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-logging:2.0.0.RELEASE
|    |    |    +--- ch.qos.logback:logback-classic:1.2.3
|    |    |    |    +--- ch.qos.logback:logback-core:1.2.3
|    |    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    |    +--- org.apache.logging.log4j:log4j-to-slf4j:2.10.0
|    |    |    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    |    |    \--- org.apache.logging.log4j:log4j-api:2.10.0
|    |    |    \--- org.slf4j:jul-to-slf4j:1.7.25
|    |    |         \--- org.slf4j:slf4j-api:1.7.25
|    |    +--- javax.annotation:javax.annotation-api:1.3.2
|    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.yaml:snakeyaml:1.19
|    +--- org.springframework.boot:spring-boot-starter-json:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework:spring-web:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.6.3
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0 -> 2.6.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    |    \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.4
|    |         +--- com.fasterxml.jackson.core:jackson-core:2.9.4 -> 2.6.3
|    |         \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|    +--- org.springframework.boot:spring-boot-starter-web:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-json:2.0.0.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-tomcat:2.0.0.RELEASE
|    |    |    +--- javax.annotation:javax.annotation-api:1.3.2
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-core:8.5.28
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-el:8.5.28
|    |    |    \--- org.apache.tomcat.embed:tomcat-embed-websocket:8.5.28
|    |    |         \--- org.apache.tomcat.embed:tomcat-embed-core:8.5.28
|    |    +--- org.hibernate.validator:hibernate-validator:6.0.7.Final
|    |    |    +--- javax.validation:validation-api:2.0.1.Final
|    |    |    +--- org.jboss.logging:jboss-logging:3.3.0.Final -> 3.3.2.Final
|    |    |    \--- com.fasterxml:classmate:1.3.1 -> 1.3.4
|    |    +--- org.springframework:spring-web:5.0.4.RELEASE (*)
|    |    \--- org.springframework:spring-webmvc:5.0.4.RELEASE
|    |         +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|    |         \--- org.springframework:spring-web:5.0.4.RELEASE (*)
|    \--- org.springframework.data:spring-data-rest-webmvc:3.0.5.RELEASE
|         +--- org.springframework.data:spring-data-rest-core:3.0.5.RELEASE
|         |    +--- org.springframework:spring-tx:5.0.4.RELEASE
|         |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         |    +--- org.springframework.hateoas:spring-hateoas:0.23.0.RELEASE -> 0.24.0.RELEASE
|         |    |    +--- org.springframework:spring-aop:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-beans:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-context:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-core:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-web:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-webmvc:4.3.12.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.25
|         |    +--- org.springframework.data:spring-data-commons:2.0.5.RELEASE
|         |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.25
|         |    +--- org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE
|         |    |    +--- org.springframework:spring-beans:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-context:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    +--- org.springframework:spring-aop:4.0.9.RELEASE -> 5.0.4.RELEASE (*)
|         |    |    \--- org.slf4j:slf4j-api:1.7.10 -> 1.7.25
|         |    +--- org.atteo:evo-inflector:1.2.2
|         |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.4 -> 2.6.3
|         |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.4 (*)
|         |    \--- org.slf4j:slf4j-api:1.7.25
|         +--- org.springframework:spring-webmvc:5.0.4.RELEASE (*)
|         +--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.6.3 (*)
|         +--- com.fasterxml.jackson.core:jackson-annotations:2.9.4 -> 2.6.3
|         \--- org.slf4j:slf4j-api:1.7.25
+--- org.springframework.boot:spring-boot-starter-data-jpa -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-aop:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    \--- org.aspectj:aspectjweaver:1.8.13
|    +--- org.springframework.boot:spring-boot-starter-jdbc:2.0.0.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    |    +--- com.zaxxer:HikariCP:2.7.8
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    \--- org.springframework:spring-jdbc:5.0.4.RELEASE
|    |         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |         \--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    +--- org.hibernate:hibernate-core:5.2.14.Final
|    |    +--- org.jboss.logging:jboss-logging:3.3.1.Final -> 3.3.2.Final
|    |    +--- org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final
|    |    +--- org.javassist:javassist:3.22.0-GA
|    |    +--- antlr:antlr:2.7.7
|    |    +--- org.jboss:jandex:2.0.3.Final
|    |    +--- com.fasterxml:classmate:1.3.0 -> 1.3.4
|    |    +--- dom4j:dom4j:1.6.1
|    |    \--- org.hibernate.common:hibernate-commons-annotations:5.0.1.Final
|    |         \--- org.jboss.logging:jboss-logging:3.3.0.Final -> 3.3.2.Final
|    +--- javax.transaction:javax.transaction-api:1.2
|    +--- org.springframework.data:spring-data-jpa:2.0.5.RELEASE
|    |    +--- org.springframework.data:spring-data-commons:2.0.5.RELEASE (*)
|    |    +--- org.springframework:spring-orm:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-jdbc:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- org.springframework:spring-aspects:5.0.4.RELEASE
|         \--- org.aspectj:aspectjweaver:1.8.13
+--- org.springframework.boot:spring-boot-starter-thymeleaf -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.thymeleaf:thymeleaf-spring5:3.0.9.RELEASE
|    |    +--- org.thymeleaf:thymeleaf:3.0.9.RELEASE
|    |    |    +--- org.attoparser:attoparser:2.0.4.RELEASE
|    |    |    +--- org.unbescape:unbescape:1.1.5.RELEASE
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- org.thymeleaf.extras:thymeleaf-extras-java8time:3.0.1.RELEASE
|         +--- org.thymeleaf:thymeleaf:3.0.0.RELEASE -> 3.0.9.RELEASE (*)
|         \--- org.slf4j:slf4j-api:1.6.6 -> 1.7.25
+--- org.springframework.boot:spring-boot-starter-security -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    +--- org.springframework.security:spring-security-config:5.0.3.RELEASE
|    |    +--- org.springframework.security:spring-security-core:5.0.3.RELEASE
|    |    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    \--- org.springframework.security:spring-security-web:5.0.3.RELEASE
|         +--- org.springframework.security:spring-security-core:5.0.3.RELEASE (*)
|         +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-core:5.0.4.RELEASE (*)
|         +--- org.springframework:spring-expression:5.0.4.RELEASE (*)
|         \--- org.springframework:spring-web:5.0.4.RELEASE (*)
+--- org.springframework.boot:spring-boot-starter-data-redis -> 2.0.0.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.0.0.RELEASE (*)
|    +--- org.springframework.data:spring-data-redis:2.0.5.RELEASE
|    |    +--- org.springframework.data:spring-data-keyvalue:2.0.5.RELEASE
|    |    |    +--- org.springframework.data:spring-data-commons:2.0.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    |    \--- org.slf4j:slf4j-api:1.7.25
|    |    +--- org.springframework:spring-tx:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-oxm:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.0.4.RELEASE (*)
|    |    +--- org.springframework:spring-context-support:5.0.4.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.0.4.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.0.4.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.0.4.RELEASE (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25
|    \--- io.lettuce:lettuce-core:5.0.2.RELEASE
|         +--- io.projectreactor:reactor-core:3.1.4.RELEASE -> 3.1.5.RELEASE
|         |    \--- org.reactivestreams:reactive-streams:1.0.2
|         +--- io.netty:netty-common:4.1.21.Final -> 4.1.22.Final
|         +--- io.netty:netty-transport:4.1.21.Final -> 4.1.22.Final
|         |    +--- io.netty:netty-buffer:4.1.22.Final
|         |    |    \--- io.netty:netty-common:4.1.22.Final
|         |    \--- io.netty:netty-resolver:4.1.22.Final
|         |         \--- io.netty:netty-common:4.1.22.Final
|         \--- io.netty:netty-handler:4.1.21.Final -> 4.1.22.Final
|              +--- io.netty:netty-buffer:4.1.22.Final (*)
|              +--- io.netty:netty-transport:4.1.22.Final (*)
|              \--- io.netty:netty-codec:4.1.22.Final
|                   \--- io.netty:netty-transport:4.1.22.Final (*)
+--- org.springframework.boot:spring-boot-starter-web -> 2.0.0.RELEASE (*)
+--- com.h2database:h2 -> 1.4.196
+--- org.mybatis:mybatis:3.3.0
+--- org.mybatis:mybatis-spring:1.2.3
+--- com.mchange:c3p0:0.9.5.1
|    \--- com.mchange:mchange-commons-java:0.2.10
+--- mysql:mysql-connector-java:5.1.31

Code Structure

com
 +- example
     +- myapplication
         +- Application.java
         |
         +- customer
         |   +- Customer.java
         |   +- CustomerController.java
         |   +- CustomerService.java
         |   +- CustomerRepository.java
         |
         +- order
             +- Order.java
             +- OrderController.java
             +- OrderService.java
             +- OrderRepository.java

Algorithm

• Divide and Conqure
  • Steps
    • Examples
  • 求解法
    • 递归
    • 迭代
  • 子问题类型
    • f(n) = f(n-1) , f(n-2)
      • e.g. Fibonacci 数列问题
    • f(n) = f(n/2) or more generally f(n/k)
      • Binary Search 的三个模板:
• Dynamic Programming
  • Steps
• Two Pointers
• Recursion
  • Recursively-defined functions
  • tail recursion
• DFS
• 圆形问题 round / circle/ rotation
  • Tail Recursion
    • Tail Recursion vs. Non-Tail Recursion
• References:
  • Divide and Conquer Strategy for Problem Solving - Recursive Functions

Divide and Conqure

Steps

1. divide n -> P1, P2, … Pk
2. 最小子问题直接求解
3. 递归/ 迭代求解子问题
4. 合并子问题的解

Examples

Binary search a sorted array a[i]

1. divide: n -> (0 – n/2) (n/2 – n)
2. 最小子问题直接求解: n = 0, 1
3. 递归/ 迭代求解子问题: each step, search either half.
4. 合并子问题的解: 最后一个子问题的解就是整个问题的解

求解法

递归

递归是最简单的形式

迭代

部分递归可以转换为迭代

子问题类型

f(n) = f(n-1) , f(n-2)

e.g. Fibonacci 数列问题

此类问题依赖前面问题的求解， 如果是迭代求解可以缓存前面几个问题的解结果。 如果递归则不用

比如递归:

public int fibonacci(int n){
    if(n == 0 || n ==1) return n;
    return fibonacci(n-1)+ fibonacci(n-2); 
}

如果是迭代: 缓存中间结果。

public int fibonacci(int n){
    if(n == 0 || n ==1) return n;
    int f1 = 0;
    int f2 = 1;
    for(int i = 2; i<=n; i++){
        int t = f2;
        f2 = f1+t;
        f1 = t;
    }
    return f2;
}

还有一种更加绝妙的解法, 注意

f(n+1) = f(n) + f(n-1)

所以有:

[f(n+1) f(n)  ]       = [f(n)      f(n-1)] * [1  1]    = [1  1]^n
[f(n)   f(n-1)]         [ f(n-1)   f(n-2)]   [1  0]      [1  0]

通过矩阵的 n 次幂， 可以解出 F(n)

这里可以使用 Divide & Conqure 技巧的就是求解幂。

public double pow(x, n){
    if(n==0) return 1;
    int half = pow(x, n/2);
    if(n%2 ==1)
        return half * half * x;
    else 
        return half * half;
}

将上面的 乘法 * 换成矩阵乘法就可以求解 f(n) 了。 复杂度降为 O(logn).

f(n) = f(n/2) or more generally f(n/k)

此类问题经常对应搜索空间减半，从而实现 O(logn) 的时间复杂度。 binary search 是典型此类问题。

**二分查找的形式: **

while(l < r){
    if(==) return found result;
    else if (< ) l++
    else r--
}

Binary Search 的三个模板:

3 Parts of a Binary Search

1. Pre-processing: Sort if collection is unsorted
2. Binary Search: Using a loop or recursion to divide search space in half after each comparison
3. Post-processing: Determine viable candidates in the remaining space.

3 Templates for Binary Search

1. Template #1 is used to search for an element or condition which can be determined by accessing a single index in the array.
   • Search Condition can be determined without comparing to the element’s neighbors (or use specific elements around it)
   • No post-processing required because at each step, you are checking to see if the element has been found. If you reach the end, then you know the element is not found
   • Initial Condition: left = 0, right = length-1
   • Termination: left > right

int binarySearch(int[] nums, int target){
  if(nums == null || nums.length == 0)
    return -1;

  int left = 0, right = nums.length - 1;
  while(left <= right){
    // Prevent (left + right) overflow
    int mid = left + (right - left) / 2;
    if(nums[mid] == target){ return mid; }
    else if(nums[mid] < target) { left = mid + 1; }
    else { right = mid - 1; }
  }

  // End Condition: left > right
  return -1;
}

1. It is used to search for an element or condition which requires accessing the current index and its immediate right neighbor’s index in the array.
   • An advanced way to implement Binary Search.
   • Search Condition needs to access element’s immediate right neighbor
   • Use element’s right neighbor to determine if condition is met and decide whether to go left or right
   • Gurantees Search Space is at least 2 in size at each step
   • Post-processing required. Loop/Recursion ends when you have 1 element left. Need to assess if the remaining element meets the condition.
   • Initial Condition: left = 0, right = length
   • Termination: left == right

int binarySearch(int[] nums, int target){
  if(nums == null || nums.length == 0)
    return -1;

  int left = 0, right = nums.length;
  while(left < right){
    // Prevent (left + right) overflow
    int mid = left + (right - left) / 2;
    if(nums[mid] == target){ return mid; }
    else if(nums[mid] < target) { left = mid + 1; }
    else { right = mid; }
  }

  // Post-processing:
  // End Condition: left == right
  if(left != nums.length && nums[left] == target) return left;
  return -1;
}

1. Template #3 is another unique form of Binary Search. It is used to search for an element or condition which requires accessing the current index and its immediate left and right neighbor’s index in the array.
   • An alternative way to implement Binary Search
   • Search Condition needs to access element’s immediate left and right neighbors
   • Use element’s neighbors to determine if condition is met and decide whether to go left or right
   • Gurantees Search Space is at least 3 in size at each step
   • Post-processing required. Loop/Recursion ends when you have 2 elements left. Need to assess if the remaining elements meet the condition.
   • Initial Condition: left = 0, right = length-1
   • Termination: left + 1 == right

int binarySearch(int[] nums, int target) {
    if (nums == null || nums.length == 0)
        return -1;

    int left = 0, right = nums.length - 1;
    while (left + 1 < right){
        // Prevent (left + right) overflow
        int mid = left + (right - left) / 2;
        if (nums[mid] == target) {
            return mid;
        } else if (nums[mid] < target) {
            left = mid;
        } else {
            right = mid;
        }
    }

    // Post-processing:
    // End Condition: left + 1 == right
    if(nums[left] == target) return left;
    if(nums[right] == target) return right;
    return -1;
}

e.g. median of two sorted array a and b

/** 
 * Find all pairs (i, j) in sorted array `arr` such that `arr[i] + arr[j] > s`
 */
public int countPairs(int[] arr, int s)
    int l = 0; r = arr.length-1;
    int count = 0;
    while(l < r){
        int t = arr[l] + arr[r];
        if(t >s) {
            count += r-l;
            r--;
        }
        else l++;
    }
    return count;

/**
 * find two numbers with sum equal to s in a sorted array.
 */
public int[] countPairs(int[] arr, int s)
    int l = 0; r = arr.length-1;
    int count = 0;
    int[] result = {-1,-1};
    while(l < r){
        int t = arr[l] + arr[r];
        if(t == s) {
            result[0] = l;
            result[1] = r;
            return result;
        }else if(t<s){
            l++;
        }else if(t>s){
            r--;
        }
    }
    return result;

Dynamic Programming

Steps

1. 建模
2. 定义子问题
3. 迭代、递归公式
4. 边界条件

– Two eggs problem: You have two identical eggs and you are given access to a 100 story building. You would like to know the highest floor for the egg not to break when dropped. The problem is the egg might break on the 1st floor, or even the 100th floor, you just don’t know. Find the maximum number of trials you need to conduct in order to find the answer.

– You have 5 identical jars. 4 of the jars contain balls with identical size weighing 10g each, while 1 of the jar contains balls weighing 9g each. You are given a digital scale, find out the 1 jar that has the 9g balls with only 1 weighing.

Two Pointers

Recursion

To iterate is human, to recurse divine (Anonymous)

solve( problem p )
    if ( p is simple )
        Solve the problem p directly
    else
        Divide p into new sub-problems p1, p2, ...
        r1 = solve(p1)
        r2 = solve(p2)
        Reassemble r1, r2, ... to solve p
    end
end

状态是相关变量的组合状态，一个变量相当于状态的一个分量

一次循环就是一轮状态转移（state transition）

初始状态 -> 按某种规则转移，有些状态属于终止状态，然后状态的某个变量就是所求

当前状态下，各个变量间互动，迁移，达到另一个状态，状态机的一般形式是 while (true) { //转移规则}， for (int i = 0; i < n; ++i ) {}这种是说，i这个状态分量每一轮的转移是固定的（++i），并且i == n 是触发状态机stop的方式之一。 状态机停止的一般形式是在转移的规则里的break或者 return

Recursively-defined functions

• Factorial: n!
• Fibonacci numbers
• Ackermann Function
• Tower of Hanoi
• Fractals
• Tree and data searches

tail recursion

递归带参数

打印目录树结构带缩进

public void print(Node root){
    print(root, "");
}

private void print(Node node, String ident){
    System.out.println(ident+ node.val);
    for(Node child : node.children){
        print(child, ident+" ");
    }
}

DFS

圆形问题 round / circle/ rotation

• Distributed Hashing Table
• 数组循环
• 队列 实现
• Link list

Tail Recursion

The following section is exerpt from Stackoverflow.

A tail call [tail recursion] is a kind of goto dressed as a call. A tail call happens when a function calls another as its last action, so it has nothing else to do. For instance, in the following code, the call to g is a tail call:

function f (x)
  return g(x)
end

After f calls g, it has nothing else to do. In such situations, the program does not need to return to the calling function when the called function ends. Therefore, after the tail call, the program does not need to keep any information about the calling function in the stack. …

Because a proper tail call uses no stack space, there is no limit on the number of “nested” tail calls that a program can make. For instance, we can call the following function with any number as argument; it will never overflow the stack:

function foo (n)
  if n > 0 then return foo(n - 1) end
end

As I said earlier, a tail call is a kind of goto. As such, a quite useful application of proper tail calls in Lua is for programming state machines. Such applications can represent each state by a function; to change state is to go to (or to call) a specific function.

• In traditional recursion, the typical model is that you perform your recursive calls first, and then you take the return value of the recursive call and calculate the result. In this manner, you don’t get the result of your calculation until you have returned from every recursive call.

    public int f(int n){
        if(n==0)
            return 0;
        else{
            int pre = f(n-1);
            return g(pre);
        }
    }
  

• In tail recursion, you perform your calculations first, and then you execute the recursive call, passing the results of your current step to the next recursive step. This results in the last statement being in the form of “(return (recursive-function params))” (I think that’s the syntax for Lisp). Basically, the return value of any given recursive step is the same as the return value of the next recursive call.

    public int f(int n, int result){
        if(n==0)
            return result;
        else{
            result =  g(result, n);
            return f(n-1,result);
        }
    }
  

The consequence of this is that once you are ready to perform your next recursive step, you don’t need the current stack frame any more. This allows for some optimization. In fact, with an appropriately written compiler, you should never have a stack overflow snicker with a tail recursive call. Simply reuse the current stack frame for the next recursive step. I’m pretty sure Lisp does this.

Tail Recursion vs. Non-Tail Recursion

• In general, (non-tail) recursive function calls put parameters on the stack
  • Every call grows the stack
  • On return, the parameters are needed to compute the result (together with the partial result returned)
• In tail recursive functions, the parameters from the call before are not needed anymore
  • Instead, the result is directly handed to the parent
  • Hence, no parameters need to be put on the stack
• Languages that use tail recursion optimization realize this and don’t grow the stack
• The languages we cover next are optimized in this way So they are much more efficient when using recursion

References:

Divide and Conquer Strategy for Problem Solving - Recursive Functions

1. If the problem is small enough to be solved directly, do it
2. If not, find a smaller problem and use its solution to create the solution to the larger problem

juicefs

JuiceFS 服务将 云上的 OSS 存储空间 mount 到本地的目录。 当操作本地目录时， juicefs daemon 进程会监测本地目录文件变化， 实时地将本地目录上传到 OSS 空间上。 这比Dropbox 等要厉害很多。 也可以做协同开发等等。

curl -L juicefs.io/static/juicefs -o juicefs && chmod +x juicefs
sudo ./juicefs mount buyuebuyue /jfs

一个创新的做法是 https://www.douban.com/note/656209469/

在服务器可本地开发机器上分别: 安装 juicefs ，并将同一个 oss 上的 bucket mount 到服务器和本地。

这样服务器上的目录和本地的目录就能够保持实时同步。 同时 server 跑在远端。

做多人协作也很好。

Dynamic programming

Model Definition

components

• 界定问题
• 子问题
• 递推公式
• 边界条件
• 复杂度

示例问题分析

n 阶台阶，每步 1或者2阶， 有多少种走法

这是一个 Fibonacci 问题

迭代公式: f(n) = f(n-1) + f(n-2)

边界条件: f(0) = 1 f(1) =1

字符串相似度(编辑距离)

二维 2D 动态规划

sbt

sbt version

➜  /tmp cat /usr/local/bin/sbt
#!/bin/sh
if [ -f "$HOME/.sbtconfig" ]; then
  echo "Use of ~/.sbtconfig is deprecated, please migrate global settings to /usr/local/etc/sbtopts" >&2
  . "$HOME/.sbtconfig"
fi
exec "/usr/local/Cellar/sbt/0.13.15/libexec/bin/sbt" "$@"

While /usr/local/Cellar/sbt/0.13.15/libexec/bin/sbt is also a shell script wrapper. The real sbt exe and options are configured in file /usr/local/etc/sbtopts.

➜  /tmp cat /usr/local/etc/sbtopts
# ------------------------------------------------ #
#  The SBT Configuration file.                     #
# ------------------------------------------------ #


# Disable ANSI color codes
#
#-no-colors

# Starts sbt even if the current directory contains no sbt project.
#
-sbt-create

# Path to global settings/plugins directory (default: ~/.sbt)
#
#-sbt-dir  /etc/sbt

# Path to shared boot directory (default: ~/.sbt/boot in 0.11 series)
#
#-sbt-boot ~/.sbt/boot  

# Path to local Ivy repository (default: ~/.ivy2)
#
#-ivy ~/.ivy2

# set memory options
#
#-mem   <integer>  

# Use local caches for projects, no sharing.
#
#-no-share

# Put SBT in offline mode.
#
#-offline

# Sets the SBT version to use.
#-sbt-version  0.11.3

# Scala version (default: latest release)
#
#-scala-home <path>        
#-scala-version <version>

# java version (default: java from PATH, currently $(java -version |& grep version))
#
#-java-home <path>

-J-Xmx2G
-J-XX:+CMSClassUnloadingEnabled

The real sbt jar/ bin are under path: /usr/local/Cellar/sbt

➜  sbt  tree -L 2 /usr/local/Cellar/sbt
/usr/local/Cellar/sbt
├── 0.13.15
│   ├── INSTALL_RECEIPT.json
│   ├── bin
│   └── libexec
└── 0.13.9
    ├── INSTALL_RECEIPT.json
    ├── bin
    └── libexec

6 directories, 2 files

➜  0.13.9 tree  /usr/local/Cellar/sbt/0.13.9/ 
/usr/local/Cellar/sbt/0.13.9/
├── INSTALL_RECEIPT.json
├── bin
│   └── sbt
└── libexec
    ├── sbt
    ├── sbt-launch-lib.bash
    └── sbt-launch.jar

2 directories, 5 files
➜  0.13.9 

If you want to use a specific sbt version for the project, configure it in project/build.properties file.

➜  opencity-server git:(master) ✗ cat project/build.properties 
sbt.version=1.0.4

➜  opencity-server git:(master) ✗ cat project/plugins.sbt     
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.6.11")

➜  opencity-server git:(master) ✗ tree -L 2 ~/.sbt/
/Users/lucas/.sbt/
├── 0.13
│   ├── staging
│   └── templates
├── 1.0
│   ├── server
│   ├── staging
│   └── zinc
├── boot
│   ├── sbt.boot.lock
│   ├── scala-2.10.2
│   ├── scala-2.10.4
│   ├── scala-2.10.5
│   ├── scala-2.10.6
│   ├── scala-2.11.5
│   ├── scala-2.11.7
│   ├── scala-2.11.8
│   ├── scala-2.12.4
│   └── update.log
├── launchers
│   └── sbt-launch-57d0f04f4b48b11ef7e764f4cea58dee4e806ffd.jar
├── preloaded
│   ├── com.jcraft
│   ├── com.thoughtworks.paranamer
│   ├── jline
│   ├── org.fusesource.jansi
│   ├── org.json4s
│   ├── org.scala-lang
│   ├── org.scala-lang.modules
│   ├── org.scala-sbt
│   ├── org.scala-sbt.ivy
│   ├── org.scala-tools.sbinary
│   ├── org.scalamacros
│   └── org.spire-math
└── repositories

version and dependency

There are many components of a play-framework project, you should take care of when configuring the versions.

• play-framework
• scala
• sbt
• sbt-plugin for play-framework

application-releated

Firstly, you choose a play-framework verison, usally, the latest one. (the current latest is 2.6.11). Since play-framework is written in Scala, and you also need to write your application code in Scala, you have to choose a Scala version. The latest is 2.12.3. Configure it in build.sbt file.

➜  opencity-server git:(master) ✗ cat build.sbt 
name := """opencity-server"""
organization := "cn.opencity"

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.12.3"

libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % Test

// Adds additional packages into Twirl
//TwirlKeys.templateImports += "cn.opencity.controllers._"

// Adds additional packages into conf/routes
// play.sbt.routes.RoutesKeys.routesImport += "cn.opencity.binders._"

build-tool versions

sbt and sbt-plugin for play-framework are used to build the project. There versions are specified under ./project/* dir configuring files.

https://github.com/coursier/coursier#sbt-plugin

• issues to note:
  • https://github.com/coursier/coursier/issues/450

Mill play-scala micro-services framework https://www.lagomframework.com/

Getting Started with Play 2.x with IntelliJ

• Mill: Better Scala Builds

Mill is a new build tool for Scala: it compiles your Scala code, packages it, runs it, and caches things to avoid doing unnecessary work. Mill aims to be better than Scala’s venerable old SBT build tool, learning from it’s mistakes and building upon ideas from functional programming to come up with a build tool that is fast, flexible, and easy to understand and use. This post will explore what makes Mill interesting to a Scala developer who is likely already using SBT

Scala and Play with Gradle

Create your own Ethereum token

web3j.io

Use local geth client

RPC client mode

• Infura
• local
  • RPC
  • IPC

Java Smart Contract Wrapper

Build and compile

solc

ENS

Ethereum Name Service

Hello, world!

https://www.ethereum.org/greeter

1. Install ethereum command line tools

Clients

• Geth: Go implementation Ethereum implementation
• Parity:
• Eth: The C++ implementation is simply called Eth. If you want added security by running two different implementations in parallel or are serious about GPU mining, then the C++ “Eth” client is for you.
• Pyethapp: The Python implementation is called Pyethapp. If you are interested in understanding how Ethereum works and how to extend it, the code base of this client is probably the most readable and has a great contract tester library with fast development cycles. It is not meant for high-end usage as performance in this client is not as high priority as clarity and reliability. If you are a Python developer that wants to build decentralized apps or are interested in Ethereum for research or an academic purpose, this is a great client: we invite you to take a look and contribute to it.
• Parity a Rust implementation by Parity Technologies
• A Haskell implementation developed by Blockapps
• If you are interested in developing a light application that will run entirely in a web browser, then we recommend using EthereumJS as a basis.
• If you want to create a small hardware project, look into the implementation for the Raspberry Pi
• If you want to install geth for non-ubuntu linux then we recommend you look into building from source
• If you want more flexibility on the Mac, try Homebrew

Solidity

Solidity is the major language to develop smart contracts on Ethereum blockchain. The documentation is at https://solidity.readthedocs.io/en/develop/ ##

Editors & IDEs

• Remix: Browser-based IDE with integrated compiler and Solidity runtime environment without server-side components.
• IntelliJ IDEA plugin: Solidity plugin for IntelliJ IDEA (and all other JetBrains IDEs)

SWARM

decentralized storage service, like ipfs.

The swarm of Swarm is the collection of nodes of the devp2p network each of which run the bzz protocol on the same network id.

ABI

solc compiler

Events

misc

Whisper: decentralized messaging protocol

geth: command line tools

geth is listening on UDP/TCP port 30303

ps aux |grep geth
lucas            19806  35.5  8.0 558645868 1348204 s001  S+    4:24PM   1:45.52 geth console                                 
lucas            20355   0.0  0.0  4267768    892 s004  S+    4:30PM   0:00.00 grep geth                                      
                                                                           
                

lsof -i -a -p 19806
COMMAND   PID  USER   FD   TYPE            DEVICE SIZE/OFF NODE NAME                                                          
geth    19806 lucas   13u  IPv6 0x9295f2087a46ab9      0t0  UDP *:30303                                                       
geth    19806 lucas   16u  IPv4 0x9295f208ea01d21      0t0  TCP 192.168.1.116:64563->39.107.26.140:61910 (ESTABLISHED)        
geth    19806 lucas   21u  IPv6 0x9295f2082d4f599      0t0  TCP *:30303 (LISTEN)                                              
geth    19806 lucas   28u  IPv4 0x9295f208f6f6101      0t0  TCP 192.168.1.116:63606->ns3066492.ip-79-137-70.eu:30303 (ESTABLIS
HED)                                                                                                                          
geth    19806 lucas   39u  IPv4 0x9295f208ec0e101      0t0  TCP 192.168.1.116:64180->hst-46-166-161-114.balticservers.eu:30303
 (ESTABLISHED)                                                                                                                
geth    19806 lucas   66u  IPv4 0x9295f2082332101      0t0  TCP 192.168.1.116:64552->47.74.5.209:30821 (SYN_SENT)             
geth    19806 lucas   75u  IPv4 0x9295f20823fed21      0t0  TCP 192.168.1.116:64070->47.104.15.188:30303 (ESTABLISHED)        
geth    19806 lucas   88u  IPv4 0x9295f20a183d681      0t0  TCP 192.168.1.116:64559->101.207.224.48:40145 (SYN_SENT)          
                                                                                                                              

Ethereum Network Visualization

http://ethviewer.live/

https://ethstats.net/

0x

mempools

Operations on orders:

• generating,
• signing,
• filling and
• cancelling ,
• verifying an orders signature,
• setting or checking a users ERC20 token balance/allowance
• and much more.

TestRPC

➜  0x.js-starter-project git:(master) ✗ yarn testrpc

yarn run v1.5.1
$ testrpc -p 8545 --networkId 50 --db ./0x_testrpc_snapshot -m "${npm_package_config_mnemonic}"

Upgrade to Android Studio 3.0

After upgrading to Android Studio 3.0, many configs changed. Please check https://developer.android.com/studio/build/gradle-plugin-3-0-0-migration.html for details.

Versions

Android Studio as an IDE often breaks because of the incompatiblity of these three components:

• Android Studio
• Gradle
• Android Plugin for Gradle

Make sure they are compatible.

In latest version as 2018/02/23,

• Android Studio v3.0.1
• Gradle v4.1
• Android Plugin for Gradle v3.0.1

The version for Android Studio and ` Android Plugin for Gradle` should be always the same.

Naming

com.android.feature

implementation, api, compileOnly, and runtimeOnly.

flavor dimensions

Break Bitcoin Pow

Miners are required to do a Proof of Work (PoW) computation to find a x such that SHA256(digest+x) == 000102340... with specified number of leading zeros.

This is a less strict requirement than birthday attack which requires to find x and y such that SHA256(x) == SHA256(Y).

I guess by inspecting the internal state of SHA256 algorithm when doing a hash, one can dramatically reduce the work to find a x to satisfy PoW, which means he can mine the block much faster than others, which could be big economic incentive to optimize mining process, or, in other words, break the current bitcoin PoW mechanism.

By Using a neural network model to quick find x, can be a promising approach.

0xproject

Bitcoin

Lightening Network

## BlockStream
c-lightening
## Lightening Labs # Ethereum # Smart Contract # HOT/ DDEX # 0xproject # 21 wallet

BlockChain

Bitcoin

Lightening Network

## BlockStream
c-lightening
## Lightening Labs # Ethereum # Smart Contract # HOT/ DDEX # 0xproject # 21 wallet

Python3 install

https://passingcuriosity.com/2015/installing-python-from-source/

[root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 Python-3.6.4]# ./configure --enable-optimizations --prefix=/root/workspace/python3
make
make test
make install

[root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 python3]# export PATH=$PATH:/root/workspace/python3/Python-3.6.4:/root/workspace/python3/bin [root@ZJHZ-PS-DNS4-Log01-SQ3#7FJ09 python3]# pip3 install virtualenv

Hashing

basic operations

insert lookup delete

One-way hash

information theory

Hash chain

Cryptographic Hash

Universal Hash

perfect Hash

Dynamic

Construction

bit matrix

prime number

implementation in Java/ Python

Applications

cryptography

MD5 public key

one time password

digital cash: bitcoin

Distributed hash table

IPFS

IPFS is the Distributed Web

A peer-to-peer hypermedia protocol to make the web faster, safer, and more open.

###CDN

Routing

Consistent Hashing

MERKEL TREE

Python Pandas

model

DataFrame

index Series

Join

Extract Transform Load pc4t

Tabular data Time series data

Python VIRTUAL ENV

Python virtual enviroment 提供一个独立的 python 运行环境， 用来解决类似 Windows 上的 DLL HELL 问题。 将应用的依赖和版本等隔离开来。每个应用有自己的隔离环境。

Python 在用的版本有很多， 2.6， 2.7， 3.5， 3.6 等等。 每一个依赖也可能很多版本，中间很多兼容性问题。 如果将他们全都放到一起， 就会产生 DLL HELL 的问题。

➜  module2 l /usr/local/lib/python2.7/site-packages 
total 4392
drwxr-xr-x  95 lucas  admin   3.0K Nov 24 18:00 .
drwxr-xr-x   4 lucas  admin   128B Nov  1 20:26 ..
drwxr-xr-x   9 lucas  admin   288B Nov  3 23:25 astroid-1.5.3.dist-info
drwxr-xr-x   7 lucas  admin   224B Nov  3 23:25 backports
drwxr-xr-x   9 lucas  admin   288B Nov  3 23:25 backports.functools_lru_cache-1.4.dist-info

Install

$ [sudo] pip install virtualenv

也可以安装源代码到本地，这样不需要 sudo 权限

$ python virtualenv.py myVE
$ ls -lah
total 256K
drwxr-xr-x 10 lucas lucas 4.0K 2017-11-27 16:01 .
drwxr-xr-x  7 lucas lucas 4.0K 2017-11-27 16:01 ..
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 AUTHORS.txt
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 bin
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 docs
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 LICENSE.txt
-rw-r--r--  1 lucas lucas  345 2016-11-16 11:39 MANIFEST.in
drwxr-xr-x  5 lucas lucas 4.0K 2017-11-27 16:01 myVE
-rw-r--r--  1 lucas lucas 3.4K 2016-11-16 11:39 PKG-INFO
-rw-r--r--  1 lucas lucas 1.2K 2016-11-16 11:39 README.rst
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 scripts
-rw-r--r--  1 lucas lucas   88 2016-11-16 11:39 setup.cfg
-rw-r--r--  1 lucas lucas 4.0K 2016-11-16 11:39 setup.py
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 tests
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv.egg-info
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv_embedded
-rwxr-xr-x  1 lucas lucas  97K 2016-11-16 11:39 virtualenv.py
-rw-r--r--  1 lucas lucas  86K 2017-11-27 16:01 virtualenv.pyc
drwxr-xr-x  2 lucas lucas 4.0K 2016-11-16 11:39 virtualenv_support

核心文件是virtualenv.py 当然要求 相关的类库存在。

使用

➜  /tmp virtualenv ENV
New python executable in /private/tmp/ENV/bin/python2.7
Also creating executable in /private/tmp/ENV/bin/python
Installing setuptools, pip, wheel...done.

➜  /tmp tree -L 2 ENV
ENV
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── easy_install
│   ├── easy_install-2.7
│   ├── pip
│   ├── pip2
│   ├── pip2.7
│   ├── python -> python2.7
│   ├── python-config
│   ├── python2 -> python2.7
│   ├── python2.7
│   └── wheel
├── include
│   └── python2.7 -> /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7
├── lib
│   └── python2.7
└── pip-selfcheck.json

5 directories, 15 files

• ENV/lib/ and ENV/include/ are created, containing supporting library files for a new virtualenv python. Packages installed in this environment will live under ENV/lib/pythonX.X/site-packages/.
• ENV/bin is created, where executables live - noticeably a new python. Thus running a script with #! /path/to/ENV/bin/python would run that script under this virtualenv’s python.
• The crucial packages pip and setuptools are installed, which allow other packages to be easily installed to the environment. This associated pip can be run from ENV/bin/pip.

激活

$ source bin/activate

``` $ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

$ source bin/activate

$ echo $PATH /home/xx/virtualenv-15.1.0/myVE/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Jupyter

Jupyter Notebook， 一款基于 Ipython Kernel 的 web 版 python 运行工具。 方便写 python 相关的报告。 混合 Python 运行环境和markdown 文档。

架构

安装

pip install jupyter

运行

jupyter notebook

其他

Headless-chrome

Running headless Chrome with Selenium in Python

Python Text Mining

Coursera Course: Applied Data Science with Python Specialization. Applied Text Mining in Python

Module 1: Working with Text in Python

• Library name Minimum version
• scikit-learn 0.18.1
• scipy 0.19.0
• numpy 1.12.1
• pandas 0.20.1
• matplotlib 2.0.1
• seaborn 0.7.1

• graphviz 0.7

• String Operations

  # find unique words
  split(' ')
  len()
  set()
  captalize()
  startswith(s)
  endswith(s)
  t in s
  isupper()
  islower()
  isTitle()
  isalpha()
  isdigit()
  isalnum()
  

• Cleaning Text

  stip()
  

• Changing Text

pandas

DataFrame ###

NLKT

Natural Language Processing Tookkit

Python Packaging

Python Process

Current Architecture

Disutils Setuptools Distribute Pip

Miscs

• rpm
• dpkg
• yum
• apt
• easy_install
• pip

Python Process

Python Process

• package multiprocessing

• Pool map

• Process

Exchanging objects between processes¶

• Queue
• Pipe

Synchronization between processes

multiprocessing

• Pool
• Process
• Exchanging Objects
  • Queue
  • Pipe
• sync
  • Lock
• Sharing state
  • Value
  • Array
  • Manager

Parsing

Context-free Grammar

Word categories

• noun
• pronoun
• verb
• adjective
• adverb
• conjunction
• preposition
• interjection *

##

• word
• consituent phrases

CFG = Context-Free Grammar = Phrase Structure Grammar = BNF = Backus-Naur Form

Context-free grammar

G = <T,N, S,R> • T is set of terminals (lexicon) • N is set of non-terminals For NLP, we usually distinguish out a set P  N of preterminals which always rewrite as terminals. • S is start symbol (one of the nonterminals) • R is rules/productions of the form X ! , where X is a nonterminal and is a sequence of terminals and nonterminals (may be empty). • A grammar G generates a language L.

Parser combinator

Parser

1. Design an algebra
2. laws
3. data types

Algebra

algebra =

• a collection of functions
• data types
• a set of laws over functions

miscs

• YACC
• ANTLR
• algebraic design
• how is regular expression implemented in various languages?

Interlude

余华 『间奏』

余华的文字有种炫技的感觉， 但是看着又很舒服。技巧纯熟，所以不会有太做作的感觉。

记得以前在哪里看过韩寒表示读不下去余华的文字， 说是没有美感，可能他读的是「兄弟」吧。 那本小说和之前的一些小说文字确实不怎么美丽。

看了余华的一些散文， 其实文字功底是相当地强悍。 之前『活着』之类的作品文字上的直白，怕是有意为之。

回到这部作品， 写的是余华对音乐的一些感悟。 而感悟的点呢，当然有作家敏感的人生和生活等，但最主要的比较还是对文学。

比如对苏联作曲家肖斯塔科维奇的『第七交响曲』（列宁格勒交响曲），余华就把它同美国小说家霍桑小说『红字』做了细致的对比。本来是两个领域的两个东西， 没有可比性。 但是在情感上引起的共鸣又是那么的相似。 让我想起大一时老师给看的『Fantasy 2000』系列动画音乐作品。 本是一两百年前的大师古典乐，迪士尼为其配了动画故事。 结果看起来却像是200年前的大师为迪士尼动画配的乐。 这说明很多艺术所触发的情感本是相同。

余华听音乐时，第一层是听音，声音本身的旋律很好听。 第二层听故事， 旋律的跳动，影响这思考的节奏， 古典乐总是在叙说着一段故事。 第三层听人生，对比自己在文学和其他方面的感受， 有时是如此相似。

章节

• 叙述: 勃拉姆斯
• 高潮: 肖斯塔科维奇和霍桑
• 灵感: 莫扎特
• 否定: 柏辽兹
• 色彩: 拉赫玛尼诺夫， 绘画
• 字与音: 门德尔松

文字

这本书里的很多文字都非常有艺术想象力。读起来很美。比如：

然后他（肖斯塔科维奇）发现一个时代找上门来了。一九四二年三月五日，『第七交响曲』在后方城市古比雪夫首演后，立刻成为了这个正在遭受耻辱的民族的抗击之声，另外一个标题『列宁格勒交响曲』也立刻覆盖了原有的标题『第七交响曲』。

这几乎是一切叙述作品的命运，它们需要获得一个时代的青睐，才能使自己得到成功的位置，然后一劳永逸地坐下去。

Python Thread

Python Thread

• __init__()
• run()
• start()

子进程出错了，如何让整个程序退出？

Python Optimize

项目优化

最近使用 Python做了一个日志转换的小项目。原生 Python，没有使用任何的Framework. 写了约两周。 越写越顺手。 发现 Python 的整个设计比较人性化，没有太多的冷知识和需要记忆的。

项目逻辑比较简单。 一句话解释就是对日志格式进行转换。

原始日志(gzip/8.5M/300K lines)， 22个/min
                ↓
每行日志进行目标格式的转换，生成新的日志 （需要生成2种格式的日志，即两份）
                ↓
每5分钟或者700M(压缩前大小， 压缩后大概100M) 滚动
                ↓
新的日志 (gzip/100M/5M lines), 

对于单个原始文件， 最终优化到10s/个的处理速度。

最开始是需要12分钟的速度， 经过多次优化，达到现在的速度。

优化1: Python 字符串处理

字符串处理。 Python 本身的字符串处理速度是比较慢的。 每行进行格式转换的算法。

v1: string.split()

最开始是使用字符串空格切分，得到每个字段。由于每一行的字段数目不是完全一致，切分是分多次进行。 非常慢。

v2: regex

花了一个晚上的时间， 使用 regex 一次性获取所有字段，regex 只需要编译一次。 速度提升非常快。

优化2: bitarray

生成新格式其中一个需求是针对的某些符合需求的日志做替换。 如果日志中的 IP 不在给定的 IP 网段集合中， 则需要使用备用库中的 IP 替换原始 IP。 这涉及到一个判定 IP 在不在给定 IP 网段集合的算法。 IP 网段的形式是 (1.1.1.0/24).约有3000个网段。

v1: 直接循环判定

使用 python ipaddress library:

    for line in lines:
        subnet = ipaddress.ip_network(line)
        self.localnet.append(subnet)

    def ip_in_localnet_raw(self, ip):
        ''' deprecated raw method, which is slow. use  ip_in_localnet
        '''
        address = ipaddress.ip_address(unicode(ip))
        for subnet in self.localnet:
            if address in subnet:
                return True
        return False
    

这个算法当中， 每次 IP 都需要循环3000/2次判定在不在给定 IP 网段集合中。时间非常慢。

v2: bloomfilter

对于快速判定某元素在不在指定集合中， bloomfilter 是非常好的结构。 直接使用 bloomfilter， 将 IP 网段集合转换成 IP 集合。 但是3000个网段，对应的 IP 数量非常庞大，bloom filter 也非常大。

v3: bitarray

IP 数据有很好的特征。 IPV4 实际是32bit unsigned integer. 如果直接将一个 IP 在不在用一个 bit 表示，则需要2^32bit =512MB 的内存。 对于服务器来说， 可以接受。 当然可以直接使用一个 int array 。 但 Python 有现成的 library 来使用: bitarray

        self.ips = bitarray(2**32)

    def __load_localnet(self):
        self.localnet = []
        starttime = datetime.datetime.now()  
        count = 0
        with open(self.localnet_iplist_file, 'r') as f:
            for line in f:
                line = unicode(line.strip())
                count +=1
                try:
                    subnet = ipaddress.ip_network(line)
                    self.localnet.append(subnet)
                    start = int(subnet.network_address)
                    end = start + subnet.num_addresses
                    for x in range(start, end):
                        self.ips[x] = 1
                except Exception as e:
                    self.logger.exception(e.message)
        endtime = datetime.datetime.now() - starttime 
        self.logger.info("GroupLog loading %s localnets done, takes %s" % (count, endtime))          

上面这个函数，将所有的 IP 网段转换成 IP 集合， 存到 ips[] bitarray 当中。 加载需要24s 左右。

v4: 加载速度优化

                    # for x in range(start, end):
                    #     self.ips[x] = 1
                    # This is far more faster than add one by one.
                    self.ips[start:end] = 1

GroupLog loading 2128 localnets done, takes 0:00:24.227503 (set one by one)
GroupLog loading 2128 localnets done, takes 0:00:00.214876 (set by [start:end])
benchmark: call 300000 times in 0:00:02.242983

经过多次优化， 加载和判定速度得到极大优化。

Python 多线程并不能利用多核

Threads are useful when the code is blocked by non bytecode execution, such as I/O or external process execution (C code, system calls, etc). If byte code execution is holding things up, the ProcessPool starts multiple interpreters that can execute in parallel. However, there is more overhead in spinning up these interpreters and in them communicating with the main thread through serialized representations (basically pickle or json over a socket if I understand correctly).

multithread

• threading
• thread

  multiprocessing

• Pool
• Process
• Exchanging Objects
  • Queue
  • Pipe
• sync
  • Lock
• Sharing state
  • Value
  • Array
  • Manager

GIL

Global Interpreter Lock sub-process v.s. thread

concurrent.futures

• ThreadPoolExecutor
• ProcessdPoolExecutor

由于 GIL 的限制， python 的 multi-thread 并不能在多核上运行。 Python 的 multi-thread 虽然将运行线程化了，但是只是在单核上运行。 这样， 对 cpu-intensive 的计算没有意义，对于 I/O 类型的运算才有意义。

msic

• **kwargs
• How does OS pip work?
• multiprocessing.Pool v.s. concurrent.futures.ProcessPoolExecutor

GIL

tick

interpreter instructions

import dis
dis.dis(countdown)

str is text representation in bytes, unicode is text representation in characters.

Python Modules, Classes, Packages

Python Modules, Classes, Packages

Module

• A module is a file containing Python definitions and statements. The file name is the module name with the suffix ‘.py’ appended. Within a module, the module’s name (as a string) is available as the value of the global variable __name__.

__name__
import fibo
from fibo import fib, fib2
from fibo import *

• symbol table

• A module can contain executable statements as well as function definitions. These statements are intended to initialize the module. They are executed only the first time the module is imported somewhere.

• When a module named ‘spam’ is imported, the interpreter searches for a file named ‘spam.py’ in the current directory, and then in the list of directories specified by the environment variable ‘PYTHONPATH’. This has the same syntax as the shell variable ‘PATH’, that is, a list of directory names. When ‘PYTHONPATH’ is not set, or when the file is not found there, the search continues in an installation-dependent default path; on UNIX, this is usually ‘.:/usr/local/lib/python’.

• sys.path = the directory containing the input script (or the current directory), + ‘PYTHONPATH’ + the installation-dependent default.

• dir() find out which names a module defines.
• standard module __builtins__

# -*- coding: utf8 -*-
import os, sys

# demo module usage

def f():
    print "function f in module 'module.py'"
print "__name__ = ", __name__
print "os.environ['PYTHONPATH'] = ", os.environ.get("PYTHONPATH", '')
print "sys.path = ", sys.path
print "dir() = ", dir()
print "__builtins__ = ", __builtins__
print "dir(__builtins__) = ", dir(__builtins__)

outputs：

➜  python101 python /Users/lucas/dev/workspace/python/python101/module.py
__name__ =  __main__
os.environ['PYTHONPATH'] =
sys.path =  ['/Users/lucas/dev/workspace/python/python101', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages', '/usr/local/Cellar/protobuf/2.6.1/libexec/lib/python2.7/site-packages']
dir() =  ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'f', 'os', 'sys']
__builtins__ =  <module '__builtin__' (built-in)>
dir(__builtins__) =  ['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BufferError', 'BytesWarning', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError', 'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs', 'all', 'any', 'apply', 'basestring', 'bin', 'bool', 'buffer', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']

package

• Packages are a way of structuring Python’s module namespace by using “dotted module names”.
• The __init__.py files are required to make Python treat the directories as containing packages;
• __all__ = ["echo", "surround", "reverse"]
• Intra-package References
• __path__. This is initialized to be a list containing the name of the directory holding the package’s __init__.py before the code in that file is executed.

Class

# -*- coding: utf8 -*-

class Foo:
    val=0
    def __init__(self, val):
        self.val = val
        pass
    
    def printVal(self):
        print(self.val)


foo = Foo(5)
foo.printVal()

• __init__

    def __init__(self, val): # constructor
        self.val = val
  

• self this me, etc.
• methods
• variables: Python creates variables on the fly, when they are first assigned, and class member variables are no different.

        self.val = val
  

• instance we can change the definition of a class on the fly, or even create a completely new class at run-time!

    Foo.printVal = printVal
    obj.printVal()
    del Foo.printVal
  
    MyNewClass = type('MyNewClass', (object,), dict())
  

• standard methods:
  • __del__: Called when an instance is about to be destroyed, which lets you do any clean-up e.g. closing file handles or database connections
  • __repr__ and __str__: Both return a string representation of the object, but __repr__ should return a Python expression that can be used to re-create the object. The more commonly used one is __str__, which can return anything.
  • __cmp__: Called to compare the object with another object. Note that this is only used with Python 2.x. In Python 3.x, only rich comparison methods are used. Such as __lt__.
  • __lt__, __le__, __eq__, __ne__, __gt__ and __ge__: Called to compare the object with another object. These will be called if defined, otherwise Python will fall-back to using __cmp__.
  • __hash__: Called to calculate a hash for the object, which is used for placing objects in data structures such as sets and dictionaries.
  • __call__: Lets an object be “called” e.g. so that you can write things like this: obj(arg1,arg2,…).
  • Python also lets you define methods that let an object act like an array (so you can write things like this: obj[2] = "foo"), or like a numeric type (so you write things like this: print(obj1 + 3*obj2).
  • __dict__ − Dictionary containing the class’s namespace.
  • __doc__ − Class documentation string or none, if undefined.
  • __name__ − Class name.
  • __module__ − Module name in which the class is defined. This attribute is “__main__” in interactive mode.
  • __bases__ − A possibly empty tuple containing the base classes, in the order of their occurrence in the base class list.
• In Python, methods and member variables are always public i.e. anyone can access them.

• private methods and variables: Class methods and variable names that start with two underscores are considered to be private to that class, However, this is only a convention

• Class Inheritance

    class Foo:
        def __init__(self, val):
            self.val = val
        def printVal(self):
            print(self.val)
      
    class DerivedFoo(Foo):
        def negateVal(self):
            self.val = -self.val
  

• isinstance
• issubclass
• Class Iterators and Generators

  
    class Backwards:
        def __init__(self, val):
            self.val = val
            self.pos = len(val)
      
        def __iter__(self):
            return self
      
        def next(self):
            # We're done
            if self.pos <= 0:
                raise StopIteration
      
            self.pos = self.pos - 1
            return self.val[self.pos]
  
    for x in Backwards([1,2,3]):
        print(x)
  

• generator yield

• class variable
• instance variable
• function overloading

• inheritance

• data memebers
• documentation

    class Employee:
        'Common base class for all employees'
        empCount = 0
  
        def __init__(self, name, salary):
            self.name = name
            self.salary = salary
            Employee.empCount += 1
          
        def displayCount(self):
            print "Total Employee %d" % Employee.empCount
  
        def displayEmployee(self):
            print "Name : ", self.name,  ", Salary: ", self.salary
            class_suite
  

  The class has a documentation string, which can be accessed via ClassName.doc.

• The getattr(obj, name[, default]) − to access the attribute of object.

• The hasattr(obj,name) − to check if an attribute exists or not.

• The setattr(obj,name,value) − to set an attribute. If attribute does not exist, then it would be created.

• The delattr(obj, name) − to delete an attribute.

• Destroying Objects (Garbage Collection)

       def __del__(self):
  

import

how to import module in parent directory?

check How does python find packages?

Python imports work by searching the directories listed in sys.path.

> import sys
> print '\n'.join(sys.path)

/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages