golang channel buffered vs unbuffered

buffered vs unbuffered channel 有些網路上文章以 同步與異步來解釋，雖然感覺上有同步與異步，但其並非同步與異步。非常單純就只是有沒有多buffer空間

先看一個案例

func main() {
    c := make(chan bool, 1)
    go func() {
        fmt.Println("go func()", <-c)
        fmt.Println("go func()", <-c)
    }()

    c <- true
    time.Sleep(1 * time.Second)

    c <- false
    c <- true

    fmt.Println("main()")

}

執行結果

go func() true
... (sleep 1 sec) ...
go func() false
main()

就寫入channel而言，只要還有buffer，就在寫入後立即往下執行，反之，會等待有多餘的空間。因此有同步的感覺，事實上，程式仍是在不同的協程中以非同步進行。

就讀出channel而言，有資料就會讀出後立即往下執行，反之，會等待有資料才讀出往下執行。

所以別再以同步或非同步(異步)來理解channel。而是以 阻塞(blocked)來理解才是正確的唷!

另外加以下二個思考題，請大家自行測試，

channel宣告改為以下buffer size = 2 ，執行結果如何?

    c := make(chan bool, 2)

channel宣告的buffer size = 0，大家覺得成立嗎，還是會有error呢?

    c := make(chan bool, 0)

希望以上的解說，能導正對channel的認知

程式效能之路(1) 變數宣告的時機影響記憶體分配的次數

情境: 每次loop都需要新的變數來進行運算，且為確保每次運算獨立，所以在每進入loop就重新宣告(配置)變數。

func () test() {
    for i := 0; i < 1000000000; i++ {
        var (
            v11, v12, v13, v21, v22, v23 float64
        )
        //...
        //...
        //...
    }
}

這樣看似平常，但在每次loop都會進行記憶體分配與釋放，都是很花費cpu time。


所以應調整成如下:

func () test() {
    var (
        v11, v12, v13, v21, v22, v23 float64

        initVar = func() {
            v11, v12, v13, v21, v22, v23 = 0, 0, 0, 0, 0, 0
        }
    )

    for i := 0; i < 1000000000; i++ {
        initVar()
        //...
        //...
        //...
    }
}

只有進loop時做宣告(配置)變數，而在每次loop開始將變數初始化

golang inner func() 讓code變優雅變精巧

原始 for loop裡在進入時會進行許多的資料判別是否可以往下運行。程式看起來非常的冗長

func (a *ArbTriangular) stg() { var ( askDetph map[string]db.Order10Depth lastTicker map[string]db.LastTicker ) // ... // ... // ... for _, st := range *a.sTriangular { if st.isTrading { continue } if _, b := askDetph[st.a]; !b { continue } if _, b := askDetph[st.b]; !b { continue } if _, b := askDetph[st.c]; !b { continue } if _, b := lastTicker[st.a]; !b { continue } if _, b := lastTicker[st.b]; !b { continue } if _, b := lastTicker[st.c]; !b { continue } // ... // ... // ... } }

進行一次的重構，for loop 看起來好多了，但是冗長的if判別看起來還是不夠優雅

func (a *ArbTriangular) stg() { var ( askDetph map[string]db.Order10Depth bidDetph map[string]db.Order10Depth ) checkData := func(st SymboleTriangular) (b bool) { if st.isTrading { return } if _, b = askDetph[st.a]; !b { return } if _, b = askDetph[st.b]; !b { return } if _, b = askDetph[st.c]; !b { return } if _, b = bidDetph[st.a]; !b { return } if _, b = bidDetph[st.b]; !b { return } if _, b = bidDetph[st.c]; !b { return } b = true return } //... //... //... for _, st := range *a.sTriangular { if !checkData(st) { continue } //... //... //... } }

進行第二次的重構，這樣的成果優雅又精簡巧

func (a *ArbTriangular) stg() { var ( askDetph map[string]db.Order10Depth bidDetph map[string]db.Order10Depth existAsk = func(s string) (b bool) { _, b = askDetph[s] return } existBid = func(s string) (b bool) { _, b = bidDetph[s] return } checkData = func(st SymboleTriangular) bool { if st.isTrading || !existAsk(st.a) || !existAsk(st.b) || !existAsk(st.c) || !existBid(st.a) || !existBid(st.b) || !existBid(st.c) { return false } return true } ) //... //... //... for _, st := range *a.sTriangular { if !checkData(st) { continue } //... //... //... } }

golang context 讓程式變髒了

為了解決goroutine間(特別是多層樹狀)取消( cancelation )
在 go 1.7 提供了 context standard library。
https://blog.golang.org/context

雖然提供了方便簡潔的操控goroutine的取消方法，並利用channel自動的通知取消給所有子執行緒。以及讓整顆樹的goroutine都能共享資料。

但官方希望我們不要二次封裝 context，而採用參數方式傳到每一個相關的function裡，連變數名稱都幫我們想好( ctx )，而且還建議放第一個參數。
這.... ，說真的，我不太能接受。

我們原本優雅的程式，讓這 ctx 污了整個code。

func doSomething(...interface{}) {
...
}

硬生生要變成這樣...

func doSomething(ctx context.Context, ...interface{}) {
...
   ctx.Done()
}

context 讓程式變髒了，就為了執行 ctx.Done()，ctx.Value(key)

@@

其實在讀完context的原始碼後，大概能理解原由。
雖然context source code寫得相當精要傳神，但個人認為其還是有許多的改善空間。

golang sqlbuilder

https://github.com/eehsiao/sqlbuilder

sqlbuilder

sqlbuilder is a simple sql query string builder

sqlbuilder its recursive struct call, that you can easy to build sql string

ex: dao.Select().From().Join().Where().Limit()

SqlBuilder functions

• build select :
  • Select(f ...string)
    • f is a fileds list of strings
    • Select("filed1", "filed2", "filed3")
    • Select(lib.Struce4QuerySlice(DaoStructType)...)
      • the library ref : https://github.com/eehsiao/go-models-lib
  • Distinct(b bool)
    • its default in builder is set false
  • Top(i int)
    • only support mssql
  • From(t ...string)
    • t is table name
  • Where(c string)
    • c is condition, ex Where("field1=1 and filed2='b'")
  • WhereAnd(c ...string)
  • WhereOr(c ...string)
  • Limit(i ...int)
    • support 2 parms
    • only support mysql
  • Join(t string, c string)
    • t is table name
    • c is condition
  • InnerJoin(t string, c string)
  • LeftJoin(t string, c string)
  • RightJoin(t string, c string)
  • FullJoin(t string, c string)
  • GroupBy(f ...string)
    • f is a fileds list of strings
  • OrderBy(f ...string)
    • f is a fileds list of strings
  • OrderByAsc(f ...string)
  • OrderByDesc(f ...string)
  • Having(s string)
    • s is having condition string
  • BuildSelectSQL()
    • check and build sql string.
    • you can get sql string via BuildedSQL()
• build update :
  • Set(s map[string]interface{})
  • FromOne(t string)
    • reset the table for only one
  • BuildUpdateSQL()
    • check and build sql string.
    • you can get sql string via BuildedSQL()
• build insert :
  • Into(t string)
    • set the insert table
  • Fields(f ...string)
    • f is a fileds list of strings
  • Values(v ...[]interface{})
    • v is a values list of interface{}
  • BuildInsertSQL()
    • check and build sql string.
    • you can get sql string via BuildedSQL()
• build delete :
  • BuildDeleteSQL()
    • check and build sql string.
    • you can get sql string via BuildedSQL()
• common :
  • ClearBuilder()
    • reset builder
  • BuildedSQL()
    • return the builded sql string, if build success.
  • SetDbName(s string)
  • SetTbName(s string)
  • SwitchPanicToErrorLog(b bool)
  • PanicOrErrorLog(s string)

Inter-Process Communication via websocket (golang)

github : https://github.com/eehsiao/websocket-ipc

websocket-ipc

Inter-Process Communication via websocket

how to be a deamon

install : install into service start : run as a deamon

sudo ./example install
sudo ./example start

how to stop deamon

when the deamon start

sudo ./example stop

how to remove deamon

when the deamon stop

sudo ./example remove

how to Communication

when the deamon start

sudo ./example {any string otherwise commands}

golang 連接mysql, redis 讀取資料的套件，簡單又好用

套件位置:
https://github.com/eehsiao/go-models-mysql
https://github.com/eehsiao/go-models-redis
https://github.com/eehsiao/go-models-lib
https://github.com/eehsiao/sqlbuilder
godoc:
https://godoc.org/github.com/eehsiao/go-models-mysql
https://godoc.org/github.com/eehsiao/go-models-redis
https://godoc.org/github.com/eehsiao/go-models-lib
https://godoc.org/github.com/eehsiao/sqlbuilder

go-models-mysql

go-models-mysql its lite and easy model.

Features

• Field scanning has become easier since the original driver was extended. Assumption: we have 5 fields to scan

type Tb struct {
 field0 sql.NullString,
 field1 sql.NullString,
 field2 sql.NullString,
 field3 sql.NullString,
 field4 sql.NullString,
}

In original driver, it can't dynamic. how many fields, that you must write fields many how. it you have 20 fileds, you must write 20 times.

var tb Tb
err = rows.Scan(&tb.field0, &tb.field1, &tb.field2, &tb.field3, &tb.field4)

In go-models-mysql , you just fill struct nil pointer.

if val, err = myDao.ScanRowType(row, (*Tb)(nil)); err == nil {
 u, _ := val.(*Tb)
 fmt.Println("Tb", u)
}

• DAO layer let you operate mysql more Intuitively.
  • Original driver (sql.DB) was extended, so you can operate original commands.
    • ex: Query, QueryRow, Exec ....
  • Import the sqlbuilder that help access sql db easily.

   myDao.Select("Host", "User", "Select_priv").From("user").Where("User='root'").Limit(1)

  • Set the default table in DAO, that you can design your dao layer friendly.

   // set a struct for dao as default model (option)
   // (*UserTb)(nil) : nil pointer of the UserTb struct
   // "user" : is real table name in the db
   myUserDao.SetDefaultModel((*UserTb)(nil), "user")
  
   // call model's Get() , get all rows in user table
   // return (rows *sql.Rows, err error)
   rows, err = myDao.Get()

Requirements

• Go 1.12 or higher.
• database/sql package
• go-sql-driver/mysql package

Go-Module

create go.mod file in your package folder, and fill below

module github.com/eehsiao/go-models-example

go 1.13

require (
 github.com/eehsiao/go-models-lib latest
 github.com/eehsiao/go-models-mysql latest
 github.com/eehsiao/go-models-redis latest
 github.com/eehsiao/sqlbuilder latest
 github.com/go-sql-driver/mysql v1.4.1
)

Docker

Easy to start the test evn. That you can run the example code.

$ docker-compose up -d

Usage

import (
    "database/sql"
 "fmt"

 mysql "github.com/eehsiao/go-models-mysql"
 redis "github.com/eehsiao/go-models-redis"
)

// UserTb : sql table struct that to store into mysql
type UserTb struct {
 Host       sql.NullString `TbField:"Host"`
 User       sql.NullString `TbField:"User"`
 SelectPriv sql.NullString `TbField:"Select_priv"`
}

//new mysql dao
myUserDao := &MyUserDao{
    Dao: mysql.NewDao().SetConfig("root", "mYaDmin", "127.0.0.1:3306", "mysql").OpenDB(),
}

// example 1 : directly use the sqlbuilder
myUserDao.Select("Host", "User", "Select_priv").From("user").Where("User='root'").Limit(1)
fmt.Println("sqlbuilder", myUserDao.BuildSelectSQL())
if row, err = myUserDao.GetRow(); err == nil {
    if val, err = myUserDao.ScanRowType(row, (*UserTb)(nil)); err == nil {
        u, _ := val.(*UserTb)
        fmt.Println("UserTb", u)
    }
}
    
// set a struct for dao as default model (option)
// (*UserTb)(nil) : nil pointer of the UserTb struct
// "user" : is real table name in the db
myUserDao.SetDefaultModel((*UserTb)(nil), "user")

// call model's Get() , get all rows in user table
// return (rows *sql.Rows, err error)
rows, err = myDao.Get()

// call model's GetRow() , get first row in user rows
// return (row *sql.Row, err error)
row, err = myDao.GetRow()

Example

1 build-in

example.go

The example will connect to local mysql and get user data. Then connect to local redis and set user data, and get back.

2 example

https://github.com/eehsiao/go-models-example/

How-to

How to design model data logical

MySQL

1.

create a table struct, and add the tag TbField:"real table filed"

TbField the tag is musted. read table filed also be same the table field.

type UserTb struct {
 Host       sql.NullString `TbField:"Host"`
 User       sql.NullString `TbField:"User"`
 SelectPriv sql.NullString `TbField:"Select_priv"`
}

2.

use Struce4QuerySlice to gen the sqlbuilder select fields

m := mysql.NewDao().SetConfig("root", "mYaDmin", "127.0.0.1:3306", "mysql").OpenDB()
m.Select(lib.Struce4QuerySlice(m.DaoStructType)...).From(m.TbName).Limit(3)

3.

scan the sql result to the struct of object

row, err = m.GetRow()
if val, err = m.ScanRowType(row, (*UserTb)(nil)); err == nil {
    u, _ := val.(*UserTb)
    fmt.Println("UserTb", u)
}

golang go mod reading https://sum.golang.org/lookup/github.com/... 410 Gone (slove)

研究 go module 且跨多的module引用時, 發生

verifying github.com/eehsiao/go-models-lib@v0.0.1/go.mod: github.com/eehsiao/go-models-lib@v0.0.1/go.mod: reading https://sum.golang.org/lookup/github.com/eehsiao/go-models-lib@v0.0.1: 410 Gone

 解決方法 (solve)

go mod tidy