hot fix issue in router registry

.gitee/ISSUE_TEMPLATE.MD

@@ -0,0 +1,33 @@
+<!-- 为更高效率地交流并解决问题，请按照以下模板提交issue，感谢！ -->
+### 1. 您当前使用的`Go`版本（将终端`go version`指令结果粘贴到下面）？
+
+<pre>
+$ go version
+
+</pre>
+
+### 2. 您当前使用的`GoFrame`版本（可以查看`go.mod`/`version.go`/`gf.VERSION`）？
+
+
+
+### 3. 更新到最新的框架版本是否能够解决问题？
+
+
+
+### 4. 问题描述？
+
+<!--
+请您尽可能地提供一份最短的，可复现问题的代码。
+代码尽可能地完整，最好是可以直接编译运行。
+-->
+
+
+
+### 5. 您期望得到的结果？
+
+
+
+### 6. 您实际得到的结果？
+
+
+

.github/ISSUE_TEMPLATE.MD

@@ -0,0 +1,33 @@
+<!-- Please answer these questions before submitting your issue. Thanks! -->
+### 1. What version of `Go` are you using (`go version`)?
+
+<pre>
+$ go version
+
+</pre>
+
+### 2. What version of `GoFrame` are you using?
+
+
+
+### 3. Does this issue reproduce with the latest release?
+
+
+
+### 4. What did you do?
+
+<!--
+If possible, provide a copy of shortest codes for reproducing the error.
+A complete runnable program is best.
+-->
+
+
+
+### 5. What did you expect to see?
+
+
+
+### 6. What did you see instead?
+
+
+

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2017 john@johng.cn http://johng.cn
+Copyright (c) 2017 john@goframe.org https://goframe.org
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.MD

@@ -14,13 +14,13 @@
 [![Code Helper](https://www.codetriage.com/gogf/gf/badges/users.svg)](https://www.codetriage.com/gogf/gf) 
 --> 
 
-`GF(GoFrame)` is a modular, lightweight, loosely coupled, high performance application development framework written in Go. Supporting graceful server, hot updates, multi-domain, multi-port, multi-service, HTTP/HTTPS, dynamic/hook routing and many more features. Providing a series of core components and dozens of practical modules.
+`GF(GoFrame)` is a modular, loose-coupled and production-ready application development framework written in Go. Providing a series of core components and dozens of practical modules, such as: cache, logging, array/queue/set/map, timer/timing tasks, file/memory lock, object pool, validator, database ORM, etc. Supporting web server with graceful server, hot updates, multi-domain, multi-port, multi-service, HTTP/HTTPS, dynamic/hook routing, rewrite rules and many more features. 
 
 # Installation
 ```
 go get -u github.com/gogf/gf
 ```
-or use `go.mod`
+or use `go.mod`:
 ```
 require github.com/gogf/gf latest
 ```
@@ -32,6 +32,7 @@ golang version >= 1.9.2
 
 # Documentation
 
+* [GoDoc](https://godoc.org/github.com/gogf/gf)
 * [中文文档](https://goframe.org)
 
 # Architecture
@@ -92,6 +93,8 @@ func main() {
 
 # Donators
 
+<a href="https://gitee.com/tiangenglan" target="_blank" title="zhuhuan12"><img src="https://images.gitee.com/uploads/99/1167099_tiangenglan.png" width="60" align="left"></a>
+
 <a href="https://gitee.com/zhuhuan12" target="_blank" title="zhuhuan12"><img src="https://gitee.com/uploads/39/751839_zhuhuan12.png" width="60" align="left"></a>
 
 <a href="https://gitee.com/zfan_codes" target="_blank" title="范钟"><img src="https://images.gitee.com/uploads/32/2044832_zfan_codes.png" width="60" align="left"></a>

README_ZH.MD

@@ -32,7 +32,7 @@
 go get -u github.com/gogf/gf
 ```
 或者
-`go.mod`
+`go.mod`:
 ```
 require github.com/gogf/gf latest
 ```
@@ -81,6 +81,14 @@ func main() {
 
 `GF` 使用非常友好的 [MIT](LICENSE) 开源协议进行发布，永久`100%`开源免费。
 
+# 捐赠
+
+捐赠支持`GF`框架的研发，
+请在捐赠时备注您的`github`/`gitee`账号名称。
+
+<a href="https://goframe.org/images/donate.png" target="_blank">
+<img src="https://goframe.org/images/donate.png" width="300"/>
+</a>
 
 # 贡献者(TOP 10)
 
@@ -100,6 +108,8 @@ func main() {
 
 # 捐赠者
 
+<a href="https://gitee.com/tiangenglan" target="_blank" title="zhuhuan12"><img src="https://images.gitee.com/uploads/99/1167099_tiangenglan.png" width="60" align="left"></a>
+
 <a href="https://gitee.com/zhuhuan12" target="_blank" title="zhuhuan12"><img src="https://gitee.com/uploads/39/751839_zhuhuan12.png" width="60" align="left"></a>
 
 <a href="https://gitee.com/zfan_codes" target="_blank" title="范钟"><img src="https://images.gitee.com/uploads/32/2044832_zfan_codes.png" width="60" align="left"></a>

RELEASE.MD

@@ -1,3 +1,38 @@
+# `v1.5.8` (2019-02-28)
+
+## 新特性
+1. 主库从`gitee`迁移到了`github`( https://github.com/gogf/gf )，`gitee`作为镜像站，用于国内的代码贡献及ISSUE提交，迁移说明详见：https://goframe.org/upgradeto150
+1. 对常用的`container`数组模块: `garray`做了大量改进/完善工作，新增大量常用方法，并完善单元测试用例及方法注释，详见API文档：https://godoc.org/github.com/gogf/gf/g/container/garray
+1. 对常用的`container`集合模块: `gset`做了大量改进/完善工作，新增大量常用方法，并完善单元测试用例及方法注释，详见API文档：https://godoc.org/github.com/gogf/gf/g/container/gset
+1. 对常用的`container`MAP模块: `gmap`做了大量改进/完善工作，新增大量常用方法，并完善单元测试用例及方法注释，详见API文档：https://godoc.org/github.com/gogf/gf/g/container/gmap
+1. 对常用的字符串模块: `gstr`做了大量改进/完善工作，新增大量常用方法，并完善单元测试用例及方法注释，详见API文档：https://godoc.org/github.com/gogf/gf/g/text/gstr
+1. 改进`gform`中对`struct`/`*struct`参数的支持，`*Insert/*Save/*Replace/*Update/Where/Data`方法的参数调整为`interface{}`类型，并支持任意类型的: `string/map/slice/struct/*struct`参数传递，具体请参考：https://goframe.org/database/orm/chaining
+1. 新增/完善若干模块的单元测试用例, 包括：`gvalid`/`gregex`/`garray`/`gset`/`gmap`/`gstr`/`gconv`/`ghttp`/`gdb`；
+1. 由于`gkafka`模块比较重，且不是框架核心模块，因此将该模块迁移到新的仓库中独立管理，并去掉相关依赖包：https://github.com/gogf/gkafka
+1. 新增`greuseport`模块，用以实现TCP的`REUSEPORT`特性：https://godoc.org/github.com/gogf/gf/g/net/greuseport
+
+## 新功能/改进
+1. 去掉模板引擎内置变量中自动初始化`session`对象带来的内存占用问题；
+1. `ghttp.Client`改进，增加若干方法，详见：https://goframe.org/net/ghttp/client 
+1. `ghttp`分组路由增加`COMMON`方法，用以注册常用的`HTTP METHOD`(`GET/PUT/POST/DELETE`)路由；
+1. 更新框架依赖的`golang.org/x/sys`模块；
+1. 改进`gform`的批量操作(`Batch*`操作)返回结果对象，可以通过该结果对象获得批量操作准确的受影响记录行数；
+1. 将`gstr`/`gregex`模块从`util`分类迁移到了`text`分类目录下；
+1. 将`gtest`模块从`util`分类迁移到了`test`分类目录下；
+1. 完善`glog`方法注释；
+
+## Bug Fix
+1. 修复带点的邮件格式,用`gvalid.Check`的"`email`"规则不能匹配成功;
+1. 修复`gvalid.Check`在`regex`规则下的检查失败问题；
+1. 修复`gcron`模块定时规则中天和周不允许`?`符号的问题；
+1. 修复`ghttp.Server`在部分异常情况下仍然返回`200`状态码的问题；
+1. 修复`gfpool`模块中由于原子操作问题造成的高并发"内存泄露"问题；
+1. 修复分组路由注册对象/控制时，方法`Index`的路由仅能通过`/xxx/index`访问的问题；
+1. 修复模板引擎使用中，当不存在`config.toml`(即使没使用)配置文件时的报错问题；
+1. 其他一些修复；
+
+
+
 # `v1.4.6` (2019-01-24)
 
 ## 新特性

TODO.MD

@@ -51,8 +51,8 @@
 1. 权限管理模块；
 1. 从ghttp中剥离SESSION功能构成单独的模块gsession；
 1. 改进gproc进程间通信处理逻辑，提高稳定性，以应对进程间大批量的数据发送/接收；
-
-
+1. gdb的Data方法支持struct参数传入；
+1. gfcache依旧使用gcache作为缓存控制对象，不要使用gmap；
 
 
 

g/database/gdb/gdb.go

@@ -37,8 +37,8 @@ type DB interface {
     doQuery(link dbLink, query string, args ...interface{}) (rows *sql.Rows, err error)
     doExec(link dbLink, query string, args ...interface{}) (result sql.Result, err error)
     doPrepare(link dbLink, query string) (*sql.Stmt, error)
-    doInsert(link dbLink, table string, data Map, option int) (result sql.Result, err error)
-    doBatchInsert(link dbLink, table string, list List, batch int, option int) (result sql.Result, err error)
+    doInsert(link dbLink, table string, data interface{}, option int, batch...int) (result sql.Result, err error)
+    doBatchInsert(link dbLink, table string, list interface{}, option int, batch...int) (result sql.Result, err error)
     doUpdate(link dbLink, table string, data interface{}, condition interface{}, args ...interface{}) (result sql.Result, err error)
     doDelete(link dbLink, table string, condition interface{}, args ...interface{}) (result sql.Result, err error)
 
@@ -61,14 +61,14 @@ type DB interface {
 	Begin() (*TX, error)
 
 	// 数据表插入/更新/保存操作
-	Insert(table string, data Map) (sql.Result, error)
-	Replace(table string, data Map) (sql.Result, error)
-	Save(table string, data Map) (sql.Result, error)
+	Insert(table string, data interface{}, batch...int) (sql.Result, error)
+	Replace(table string, data interface{}, batch...int) (sql.Result, error)
+	Save(table string, data interface{}, batch...int) (sql.Result, error)
 
 	// 数据表插入/更新/保存操作(批量)
-	BatchInsert(table string, list List, batch int) (sql.Result, error)
-	BatchReplace(table string, list List, batch int) (sql.Result, error)
-	BatchSave(table string, list List, batch int) (sql.Result, error)
+	BatchInsert(table string, list interface{}, batch...int) (sql.Result, error)
+	BatchReplace(table string, list interface{}, batch...int) (sql.Result, error)
+	BatchSave(table string, list interface{}, batch...int) (sql.Result, error)
 
 	// 数据修改/删除
 	Update(table string, data interface{}, condition interface{}, args ...interface{}) (sql.Result, error)
@@ -149,8 +149,11 @@ const (
     OPTION_REPLACE = 1
     OPTION_SAVE    = 2
     OPTION_IGNORE  = 3
+    // 默认批量操作的数量值(Batch*操作)
+    gDEFAULT_BATCH_NUM          = 10
     // 默认的连接池连接存活时间(秒)
     gDEFAULT_CONN_MAX_LIFE_TIME = 30
+
 )
 
 // 使用默认/指定分组配置进行连接，数据库集群配置项：default

g/database/gdb/gdb_base.go

@@ -14,8 +14,8 @@ import (
     "github.com/gogf/gf/g/container/gvar"
     "github.com/gogf/gf/g/os/gcache"
     "github.com/gogf/gf/g/os/gtime"
-    "github.com/gogf/gf/g/util/gconv"
     "github.com/gogf/gf/g/text/gregex"
+    "github.com/gogf/gf/g/util/gconv"
     "reflect"
     "strings"
 )
@@ -234,33 +234,60 @@ func (bs *dbBase) Begin() (*TX, error) {
     }
 }
 
-// CURD操作:单条数据写入, 仅仅执行写入操作，如果存在冲突的主键或者唯一索引，那么报错返回
-func (bs *dbBase) Insert(table string, data Map) (sql.Result, error) {
-    return bs.db.doInsert(nil, table, data, OPTION_INSERT)
+// CURD操作:单条数据写入, 仅仅执行写入操作，如果存在冲突的主键或者唯一索引，那么报错返回。
+// 参数data支持map/struct/*struct/slice类型，
+// 当为slice(例如[]map/[]struct/[]*struct)类型时，batch参数生效，并自动切换为批量操作。
+func (bs *dbBase) Insert(table string, data interface{}, batch...int) (sql.Result, error) {
+    return bs.db.doInsert(nil, table, data, OPTION_INSERT, batch...)
 }
 
-// CURD操作:单条数据写入, 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条
-func (bs *dbBase) Replace(table string, data Map) (sql.Result, error) {
-    return bs.db.doInsert(nil, table, data, OPTION_REPLACE)
+// CURD操作:单条数据写入, 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条。
+// 参数data支持map/struct/*struct/slice类型，
+// 当为slice(例如[]map/[]struct/[]*struct)类型时，batch参数生效，并自动切换为批量操作。
+func (bs *dbBase) Replace(table string, data interface{}, batch...int) (sql.Result, error) {
+    return bs.db.doInsert(nil, table, data, OPTION_REPLACE, batch...)
 }
 
-// CURD操作:单条数据写入, 如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据
-func (bs *dbBase) Save(table string, data Map) (sql.Result, error) {
-    return bs.db.doInsert(nil, table, data, OPTION_SAVE)
+// CURD操作:单条数据写入, 如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据。
+// 参数data支持map/struct/*struct/slice类型，
+// 当为slice(例如[]map/[]struct/[]*struct)类型时，batch参数生效，并自动切换为批量操作。
+func (bs *dbBase) Save(table string, data interface{}, batch...int) (sql.Result, error) {
+    return bs.db.doInsert(nil, table, data, OPTION_SAVE, batch...)
 }
 
-// insert、replace, save， ignore操作
-// 0: insert:  仅仅执行写入操作，如果存在冲突的主键或者唯一索引，那么报错返回
-// 1: replace: 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条
-// 2: save:    如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据
-// 3: ignore:  如果数据存在(主键或者唯一索引)，那么什么也不做
-func (bs *dbBase) doInsert(link dbLink, table string, data Map, option int) (result sql.Result, err error) {
-    var fields []string
-    var values []string
-    var params []interface{}
-    charl, charr := bs.db.getChars()
-    for k, v := range data {
-        fields = append(fields,   charl + k + charr)
+// 支持insert、replace, save， ignore操作。
+// 0: insert:  仅仅执行写入操作，如果存在冲突的主键或者唯一索引，那么报错返回;
+// 1: replace: 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条;
+// 2: save:    如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据;
+// 3: ignore:  如果数据存在(主键或者唯一索引)，那么什么也不做;
+//
+// 参数data支持map/struct/*struct/slice类型，
+// 当为slice(例如[]map/[]struct/[]*struct)类型时，batch参数生效，并自动切换为批量操作。
+func (bs *dbBase) doInsert(link dbLink, table string, data interface{}, option int, batch...int) (result sql.Result, err error) {
+    var fields  []string
+    var values  []string
+    var params  []interface{}
+    var dataMap Map
+    // 使用反射判断data数据类型，如果为slice类型，那么自动转为批量操作
+    rv   := reflect.ValueOf(data)
+    kind := rv.Kind()
+    if kind == reflect.Ptr {
+        rv   = rv.Elem()
+        kind = rv.Kind()
+    }
+    switch kind {
+        case reflect.Slice: fallthrough
+        case reflect.Array:
+            return bs.db.doBatchInsert(link, table, data, option, batch...)
+        case reflect.Map:   fallthrough
+        case reflect.Struct:
+            dataMap = Map(gconv.Map(data))
+        default:
+            return result, errors.New(fmt.Sprint("unsupported data type:", kind))
+    }
+    charL, charR := bs.db.getChars()
+    for k, v := range dataMap {
+        fields = append(fields, charL + k + charR)
         values = append(values, "?")
         params = append(params, v)
     }
@@ -268,11 +295,11 @@ func (bs *dbBase) doInsert(link dbLink, table string, data Map, option int) (res
     updateStr := ""
     if option == OPTION_SAVE {
         var updates []string
-        for k, _ := range data {
+        for k, _ := range dataMap {
             updates = append(updates,
                 fmt.Sprintf("%s%s%s=VALUES(%s%s%s)",
-                    charl, k, charr,
-                    charl, k, charr,
+                    charL, k, charR,
+                    charL, k, charR,
                 ),
             )
         }
@@ -290,28 +317,55 @@ func (bs *dbBase) doInsert(link dbLink, table string, data Map, option int) (res
 }
 
 // CURD操作:批量数据指定批次量写入
-func (bs *dbBase) BatchInsert(table string, list List, batch int) (sql.Result, error) {
-    return bs.db.doBatchInsert(nil, table, list, batch, OPTION_INSERT)
+func (bs *dbBase) BatchInsert(table string, list interface{}, batch...int) (sql.Result, error) {
+    return bs.db.doBatchInsert(nil, table, list, OPTION_INSERT, batch...)
 }
 
 // CURD操作:批量数据指定批次量写入, 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条
-func (bs *dbBase) BatchReplace(table string, list List, batch int) (sql.Result, error) {
-    return bs.db.doBatchInsert(nil, table, list, batch, OPTION_REPLACE)
+func (bs *dbBase) BatchReplace(table string, list interface{}, batch...int) (sql.Result, error) {
+    return bs.db.doBatchInsert(nil, table, list, OPTION_REPLACE, batch...)
 }
 
 // CURD操作:批量数据指定批次量写入, 如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据
-func (bs *dbBase) BatchSave(table string, list List, batch int) (sql.Result, error) {
-    return bs.db.doBatchInsert(nil, table, list, batch, OPTION_SAVE)
+func (bs *dbBase) BatchSave(table string, list interface{}, batch...int) (sql.Result, error) {
+    return bs.db.doBatchInsert(nil, table, list, OPTION_SAVE, batch...)
 }
 
-// 批量写入数据
-func (bs *dbBase) doBatchInsert(link dbLink, table string, list List, batch int, option int) (result sql.Result, err error) {
-    var keys    []string
-    var values  []string
-    var bvalues []string
-    var params  []interface{}
+// 批量写入数据, 参数list支持slice类型，例如: []map/[]struct/[]*struct。
+func (bs *dbBase) doBatchInsert(link dbLink, table string, list interface{}, option int, batch...int) (result sql.Result, err error) {
+    var keys   []string
+    var values []string
+    var params []interface{}
+    listMap := (List)(nil)
+    switch v := list.(type) {
+        case List:
+            listMap = v
+        case Map:
+            listMap = List{v}
+        default:
+            rv   := reflect.ValueOf(list)
+            kind := rv.Kind()
+            if kind == reflect.Ptr {
+                rv   = rv.Elem()
+                kind = rv.Kind()
+            }
+            switch kind {
+                // 如果是slice，那么转换为List类型
+                case reflect.Slice: fallthrough
+                case reflect.Array:
+                    listMap = make(List, rv.Len())
+                    for i := 0; i < rv.Len(); i++ {
+                        listMap[i] = gconv.Map(rv.Index(i).Interface())
+                    }
+                case reflect.Map:   fallthrough
+                case reflect.Struct:
+                    listMap = List{Map(gconv.Map(list))}
+                default:
+                    return result, errors.New(fmt.Sprint("unsupported list type:", kind))
+            }
+    }
     // 判断长度
-    if len(list) < 1 {
+    if len(listMap) < 1 {
         return result, errors.New("empty data list")
     }
     if link == nil {
@@ -320,14 +374,15 @@ func (bs *dbBase) doBatchInsert(link dbLink, table string, list List, batch int,
         }
     }
     // 首先获取字段名称及记录长度
-    for k, _ := range list[0] {
-        keys   = append(keys,   k)
-        values = append(values, "?")
+    holders := []string(nil)
+    for k, _ := range listMap[0] {
+        keys    = append(keys,   k)
+        holders = append(holders, "?")
     }
     batchResult    := new(batchSqlResult)
-    charl, charr   := bs.db.getChars()
-    keyStr         := charl + strings.Join(keys, charl + "," + charr) + charr
-    valueHolderStr := "(" + strings.Join(values, ",") + ")"
+    charL, charR   := bs.db.getChars()
+    keyStr         := charL + strings.Join(keys, charL + "," + charR) + charR
+    valueHolderStr := "(" + strings.Join(holders, ",") + ")"
     // 操作判断
     operation := getInsertOperationByOption(option)
     updateStr := ""
@@ -336,22 +391,26 @@ func (bs *dbBase) doBatchInsert(link dbLink, table string, list List, batch int,
         for _, k := range keys {
             updates = append(updates,
                 fmt.Sprintf("%s%s%s=VALUES(%s%s%s)",
-                    charl, k, charr,
-                    charl, k, charr,
+                    charL, k, charR,
+                    charL, k, charR,
                 ),
             )
         }
         updateStr = fmt.Sprintf(" ON DUPLICATE KEY UPDATE %s", strings.Join(updates, ","))
     }
     // 构造批量写入数据格式(注意map的遍历是无序的)
-    for i := 0; i < len(list); i++ {
+    batchNum := gDEFAULT_BATCH_NUM
+    if len(batch) > 0 {
+        batchNum = batch[0]
+    }
+    for i := 0; i < len(listMap); i++ {
         for _, k := range keys {
-            params = append(params, list[i][k])
+            params = append(params, listMap[i][k])
         }
-        bvalues = append(bvalues, valueHolderStr)
-        if len(bvalues) == batch {
+        values = append(values, valueHolderStr)
+        if len(values) == batchNum {
             r, err := bs.db.doExec(link, fmt.Sprintf("%s INTO %s(%s) VALUES%s %s",
-                operation, table, keyStr, strings.Join(bvalues, ","),
+                operation, table, keyStr, strings.Join(values, ","),
                 updateStr),
                 params...)
             if err != nil {
@@ -363,14 +422,14 @@ func (bs *dbBase) doBatchInsert(link dbLink, table string, list List, batch int,
                 batchResult.lastResult    = r
                 batchResult.rowsAffected += n
             }
-            params  = params[:0]
-            bvalues = bvalues[:0]
+            params = params[:0]
+            values = values[:0]
         }
     }
     // 处理最后不构成指定批量的数据
-    if len(bvalues) > 0 {
+    if len(values) > 0 {
         r, err := bs.db.doExec(link, fmt.Sprintf("%s INTO %s(%s) VALUES%s %s",
-            operation, table, keyStr, strings.Join(bvalues, ","),
+            operation, table, keyStr, strings.Join(values, ","),
             updateStr),
             params...)
         if err != nil {
@@ -386,8 +445,8 @@ func (bs *dbBase) doBatchInsert(link dbLink, table string, list List, batch int,
     return batchResult, nil
 }
 
-// CURD操作:数据更新，统一采用sql预处理
-// data参数支持字符串或者关联数组类型，内部会自行做判断处理
+// CURD操作:数据更新，统一采用sql预处理。
+// data参数支持string/map/struct/*struct类型。
 func (bs *dbBase) Update(table string, data interface{}, condition interface{}, args ...interface{}) (sql.Result, error) {
     link, err := bs.db.Master()
     if err != nil {
@@ -396,23 +455,30 @@ func (bs *dbBase) Update(table string, data interface{}, condition interface{},
     return bs.db.doUpdate(link, table, data, condition, args ...)
 }
 
-// CURD操作:数据更新，统一采用sql预处理
-// data参数支持字符串或者关联数组类型，内部会自行做判断处理
+// CURD操作:数据更新，统一采用sql预处理。
+// data参数支持string/map/struct/*struct类型类型。
 func (bs *dbBase) doUpdate(link dbLink, table string, data interface{}, condition interface{}, args ...interface{}) (result sql.Result, err error) {
     params       := ([]interface{})(nil)
     updates      := ""
-    charl, charr := bs.db.getChars()
-    refValue     := reflect.ValueOf(data)
-    if refValue.Kind() == reflect.Map {
-        var fields []string
-        keys := refValue.MapKeys()
-        for _, k := range keys {
-            fields = append(fields, fmt.Sprintf("%s%s%s=?", charl, k, charr))
-            params = append(params, gconv.String(refValue.MapIndex(k).Interface()))
-        }
-        updates = strings.Join(fields, ",")
-    } else {
-        updates = gconv.String(data)
+    charL, charR := bs.db.getChars()
+    // 使用反射进行类型判断
+    rv   := reflect.ValueOf(data)
+    kind := rv.Kind()
+    if kind == reflect.Ptr {
+        rv   = rv.Elem()
+        kind = rv.Kind()
+    }
+    switch kind {
+        case reflect.Map:   fallthrough
+        case reflect.Struct:
+            var fields []string
+            for k, v := range gconv.Map(data) {
+                fields = append(fields, fmt.Sprintf("%s%s%s=?", charL, k, charR))
+                params = append(params, gconv.String(v))
+            }
+            updates = strings.Join(fields, ",")
+        default:
+            updates = gconv.String(data)
     }
     for _, v := range args {
         params = append(params, gconv.String(v))

g/database/gdb/gdb_func.go

@@ -24,23 +24,30 @@ import (
 func formatCondition(where interface{}, args []interface{}) (string, []interface{}) {
     // 条件字符串处理
     buffer := bytes.NewBuffer(nil)
-    if reflect.ValueOf(where).Kind() == reflect.Map {
-        ks := reflect.ValueOf(where).MapKeys()
-        vs := reflect.ValueOf(where)
-        for _, k := range ks {
-            key   := gconv.String(k.Interface())
-            value := gconv.String(vs.MapIndex(k).Interface())
-            if buffer.Len() > 0 {
-                buffer.WriteString(" AND ")
+    // 使用反射进行类型判断
+    rv   := reflect.ValueOf(where)
+    kind := rv.Kind()
+    if kind == reflect.Ptr {
+        rv   = rv.Elem()
+        kind = rv.Kind()
+    }
+    switch kind {
+        case reflect.Map:   fallthrough
+        case reflect.Struct:
+            for k, v := range gconv.Map(where) {
+                key   := gconv.String(k)
+                value := gconv.String(v)
+                if buffer.Len() > 0 {
+                    buffer.WriteString(" AND ")
+                }
+                if gstr.IsNumeric(value) || value == "?" {
+                    buffer.WriteString(key + "=" + value)
+                } else {
+                    buffer.WriteString(key + "='" + value + "'")
+                }
             }
-            if gstr.IsNumeric(value) || value == "?" {
-                buffer.WriteString(key + "=" + value)
-            } else {
-                buffer.WriteString(key + "='" + value + "'")
-            }
-        }
-    } else {
-        buffer.Write(gconv.Bytes(where))
+        default:
+            buffer.WriteString(gconv.String(where))
     }
     if buffer.Len() == 0 {
         buffer.WriteString("1=1")
@@ -57,6 +64,7 @@ func formatCondition(where interface{}, args []interface{}) (string, []interface
                 kind = rv.Kind()
             }
             switch kind {
+                // Where条件参数支持slice类型
                 case reflect.Slice: fallthrough
                 case reflect.Array:
                     for i := 0; i < rv.Len(); i++ {
@@ -109,13 +117,13 @@ func formatError(err error, query string, args ...interface{}) error {
 
 // 根据insert选项获得操作名称
 func getInsertOperationByOption(option int) string {
-    oper := "INSERT"
+    operator := "INSERT"
     switch option {
         case OPTION_REPLACE:
-            oper = "REPLACE"
+            operator = "REPLACE"
         case OPTION_SAVE:
         case OPTION_IGNORE:
-            oper = "INSERT IGNORE"
+            operator = "INSERT IGNORE"
     }
-    return oper
+    return operator
 }

g/database/gdb/gdb_model.go

@@ -200,13 +200,14 @@ func (md *Model) Cache(time int, name ... string) *Model {
     return model
 }
 
-// 链式操作，操作数据记录项，可以是string/Map, 也可以是：key,value,key,value,...
-func (md *Model) Data(data ...interface{}) (*Model) {
+// 链式操作，操作数据项，参数data类型支持 string/map/slice/struct/*struct ,
+// 也可以是：key,value,key,value,...。
+func (md *Model) Data(data ...interface{}) *Model {
     model := md.Clone()
 	if len(data) > 1 {
 		m := make(map[string]interface{})
 		for i := 0; i < len(data); i += 2 {
-			m[gconv.String(data[i])] = data[i+1]
+			m[gconv.String(data[i])] = data[i + 1]
 		}
         model.data = m
 	} else {
@@ -223,6 +224,7 @@ func (md *Model) Data(data ...interface{}) (*Model) {
                     kind = rv.Kind()
                 }
                 switch kind {
+                	// 如果是slice，那么转换为List类型
                     case reflect.Slice: fallthrough
                     case reflect.Array:
                         list := make(List, rv.Len())
@@ -230,8 +232,9 @@ func (md *Model) Data(data ...interface{}) (*Model) {
                             list[i] = gconv.Map(rv.Index(i).Interface())
                         }
                         model.data = list
-                    case reflect.Map:
-                        model.data = gconv.Map(data[0])
+                    case reflect.Map:   fallthrough
+                    case reflect.Struct:
+                        model.data = Map(gconv.Map(data[0]))
                     default:
                         model.data = data[0]
                 }
@@ -240,7 +243,9 @@ func (md *Model) Data(data ...interface{}) (*Model) {
 	return model
 }
 
-// 链式操作， CURD - Insert/BatchInsert
+// 链式操作， CURD - Insert/BatchInsert。
+// 根据Data方法传递的参数类型决定该操作是单条操作还是批量操作，
+// 如果Data方法传递的是slice类型，那么为批量操作。
 func (md *Model) Insert() (result sql.Result, err error) {
 	defer func() {
 		if err == nil {
@@ -279,7 +284,9 @@ func (md *Model) Insert() (result sql.Result, err error) {
 	return nil, errors.New("inserting into table with invalid data type")
 }
 
-// 链式操作， CURD - Replace/BatchReplace
+// 链式操作， CURD - Replace/BatchReplace。
+// 根据Data方法传递的参数类型决定该操作是单条操作还是批量操作，
+// 如果Data方法传递的是slice类型，那么为批量操作。
 func (md *Model) Replace() (result sql.Result, err error) {
 	defer func() {
 		if err == nil {
@@ -318,7 +325,9 @@ func (md *Model) Replace() (result sql.Result, err error) {
 	return nil, errors.New("replacing into table with invalid data type")
 }
 
-// 链式操作， CURD - Save/BatchSave
+// 链式操作， CURD - Save/BatchSave。
+// 根据Data方法传递的参数类型决定该操作是单条操作还是批量操作，
+// 如果Data方法传递的是slice类型，那么为批量操作。
 func (md *Model) Save() (result sql.Result, err error) {
 	defer func() {
 		if err == nil {
@@ -330,7 +339,7 @@ func (md *Model) Save() (result sql.Result, err error) {
 	}
 	// 批量操作
 	if list, ok := md.data.(List); ok {
-		batch := 10
+		batch := gDEFAULT_BATCH_NUM
 		if md.batch > 0 {
 			batch = md.batch
 		}

g/database/gdb/gdb_structure.go

@@ -85,8 +85,8 @@ func (bs *dbBase) getTableFields(table string) (fields map[string]string, err er
     // 缓存不存在时会查询数据表结构，缓存后不过期，直至程序重启(重新部署)
     v := bs.cache.GetOrSetFunc("table_fields_" + table, func() interface{} {
         result       := (Result)(nil)
-        charl, charr := bs.db.getChars()
-        result, err   = bs.GetAll(fmt.Sprintf(`SHOW COLUMNS FROM %s%s%s`, charl, table, charr))
+        charL, charR := bs.db.getChars()
+        result, err   = bs.GetAll(fmt.Sprintf(`SHOW COLUMNS FROM %s%s%s`, charL, table, charR))
         if err != nil {
             return nil
         }

g/database/gdb/gdb_transaction.go

@@ -100,33 +100,33 @@ func (tx *TX) GetCount(query string, args ...interface{}) (int, error) {
 }
 
 // CURD操作:单条数据写入, 仅仅执行写入操作，如果存在冲突的主键或者唯一索引，那么报错返回
-func (tx *TX) Insert(table string, data Map) (sql.Result, error) {
-    return tx.db.doInsert(tx.tx, table, data, OPTION_INSERT)
+func (tx *TX) Insert(table string, data interface{}, batch...int) (sql.Result, error) {
+    return tx.db.doInsert(tx.tx, table, data, OPTION_INSERT, batch...)
 }
 
 // CURD操作:单条数据写入, 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条
-func (tx *TX) Replace(table string, data Map) (sql.Result, error) {
-    return tx.db.doInsert(tx.tx, table, data, OPTION_REPLACE)
+func (tx *TX) Replace(table string, data interface{}, batch...int) (sql.Result, error) {
+    return tx.db.doInsert(tx.tx, table, data, OPTION_REPLACE, batch...)
 }
 
 // CURD操作:单条数据写入, 如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据
-func (tx *TX) Save(table string, data Map) (sql.Result, error) {
-    return tx.db.doInsert(tx.tx, table, data, OPTION_SAVE)
+func (tx *TX) Save(table string, data interface{}, batch...int) (sql.Result, error) {
+    return tx.db.doInsert(tx.tx, table, data, OPTION_SAVE, batch...)
 }
 
 // CURD操作:批量数据指定批次量写入
-func (tx *TX) BatchInsert(table string, list List, batch int) (sql.Result, error) {
-    return tx.db.doBatchInsert(tx.tx, table, list, batch, OPTION_INSERT)
+func (tx *TX) BatchInsert(table string, list interface{}, batch...int) (sql.Result, error) {
+    return tx.db.doBatchInsert(tx.tx, table, list, OPTION_INSERT, batch...)
 }
 
 // CURD操作:批量数据指定批次量写入, 如果数据存在(主键或者唯一索引)，那么删除后重新写入一条
-func (tx *TX) BatchReplace(table string, list List, batch int) (sql.Result, error) {
-    return tx.db.doBatchInsert(tx.tx, table, list, batch, OPTION_REPLACE)
+func (tx *TX) BatchReplace(table string, list interface{}, batch...int) (sql.Result, error) {
+    return tx.db.doBatchInsert(tx.tx, table, list, OPTION_REPLACE, batch...)
 }
 
 // CURD操作:批量数据指定批次量写入, 如果数据存在(主键或者唯一索引)，那么更新，否则写入一条新数据
-func (tx *TX) BatchSave(table string, list List, batch int) (sql.Result, error) {
-    return tx.db.doBatchInsert(tx.tx, table, list, batch, OPTION_SAVE)
+func (tx *TX) BatchSave(table string, list interface{}, batch...int) (sql.Result, error) {
+    return tx.db.doBatchInsert(tx.tx, table, list, OPTION_SAVE, batch...)
 }
 
 // CURD操作:数据更新，统一采用sql预处理

g/database/gdb/gdb_unit_method_test.go

@@ -56,6 +56,91 @@ func TestDbBase_Insert(t *testing.T) {
     }); err != nil {
         gtest.Fatal(err)
     }
+    // normal map
+    result, err := db.Insert("user", map[interface{}]interface{} {
+        "id"          : "2",
+        "passport"    : "t2",
+        "password"    : "25d55ad283aa400af464c76d713c07ad",
+        "nickname"    : "T2",
+        "create_time" : gtime.Now().String(),
+    })
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ := result.RowsAffected()
+    gtest.Assert(n, 1)
+
+    // struct
+    type User struct {
+        Id         int    `gconv:"id"`
+        Passport   string `json:"passport"`
+        Password   string `gconv:"password"`
+        Nickname   string `gconv:"nickname"`
+        CreateTime string `json:"create_time"`
+    }
+    result, err = db.Insert("user", User{
+        Id          : 3,
+        Passport    : "t3",
+        Password    : "25d55ad283aa400af464c76d713c07ad",
+        Nickname    : "T3",
+        CreateTime  : gtime.Now().String(),
+    })
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 1)
+    value, err := db.GetValue("select `passport` from `user` where id=?", 3)
+    gtest.Assert(err, nil)
+    gtest.Assert(value.String(), "t3")
+
+    // *struct
+    result, err = db.Insert("user", &User{
+        Id          : 4,
+        Passport    : "t4",
+        Password    : "25d55ad283aa400af464c76d713c07ad",
+        Nickname    : "T4",
+        CreateTime  : gtime.Now().String(),
+    })
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 1)
+    value, err = db.GetValue("select `passport` from `user` where id=?", 4)
+    gtest.Assert(err, nil)
+    gtest.Assert(value.String(), "t4")
+
+    // batch with Insert
+    if r, err := db.Insert("user", []interface{} {
+        map[interface{}]interface{} {
+            "id"          : 200,
+            "passport"    : "t200",
+            "password"    : "25d55ad283aa400af464c76d713c07ad",
+            "nickname"    : "T200",
+            "create_time" : gtime.Now().String(),
+        },
+        map[interface{}]interface{} {
+            "id"          : 300,
+            "passport"    : "t300",
+            "password"    : "25d55ad283aa400af464c76d713c07ad",
+            "nickname"    : "T300",
+            "create_time" : gtime.Now().String(),
+        },
+    }); err != nil {
+        gtest.Fatal(err)
+    } else {
+        n, _ := r.RowsAffected()
+        gtest.Assert(n, 2)
+    }
+
+    // clear unnecessary data
+    result, err = db.Delete("user", "id>?", 1)
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 5)
 }
 
 func TestDbBase_BatchInsert(t *testing.T) {
@@ -80,6 +165,36 @@ func TestDbBase_BatchInsert(t *testing.T) {
         n, _ := r.RowsAffected()
         gtest.Assert(n, 2)
     }
+
+    result, err := db.Delete("user", "id>?", 1)
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ := result.RowsAffected()
+    gtest.Assert(n, 2)
+
+    // []interface{}
+    if r, err := db.BatchInsert("user", []interface{} {
+        map[interface{}]interface{} {
+            "id"          : 2,
+            "passport"    : "t2",
+            "password"    : "25d55ad283aa400af464c76d713c07ad",
+            "nickname"    : "T2",
+            "create_time" : gtime.Now().String(),
+        },
+        map[interface{}]interface{} {
+            "id"          : 3,
+            "passport"    : "t3",
+            "password"    : "25d55ad283aa400af464c76d713c07ad",
+            "nickname"    : "T3",
+            "create_time" : gtime.Now().String(),
+        },
+    }, 1); err != nil {
+        gtest.Fatal(err)
+    } else {
+        n, _ := r.RowsAffected()
+        gtest.Assert(n, 2)
+    }
 }
 
 func TestDbBase_Save(t *testing.T) {

g/database/gdb/gdb_unit_model_test.go

@@ -9,6 +9,7 @@ import (
     "testing"
 )
 
+// 基本测试
 func TestModel_Insert(t *testing.T) {
     result, err := db.Table("user").Filter().Data(g.Map{
         "id"          : 1,
@@ -23,6 +24,69 @@ func TestModel_Insert(t *testing.T) {
     }
     n, _ := result.LastInsertId()
     gtest.Assert(n, 1)
+
+    result, err = db.Table("user").Filter().Data(map[interface{}]interface{} {
+        "id"          : "2",
+        "uid"         : "2",
+        "passport"    : "t2",
+        "password"    : "25d55ad283aa400af464c76d713c07ad",
+        "nickname"    : "T2",
+        "create_time" : gtime.Now().String(),
+    }).Insert()
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 1)
+
+    type User struct {
+        Id         int    `gconv:"id"`
+        Uid        int    `gconv:"uid"`
+        Passport   string `json:"passport"`
+        Password   string `gconv:"password"`
+        Nickname   string `gconv:"nickname"`
+        CreateTime string `json:"create_time"`
+    }
+    result, err = db.Table("user").Filter().Data(User{
+        Id          : 3,
+        Uid         : 3,
+        Passport    : "t3",
+        Password    : "25d55ad283aa400af464c76d713c07ad",
+        Nickname    : "T3",
+        CreateTime  : gtime.Now().String(),
+    }).Insert()
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 1)
+    value, err := db.Table("user").Fields("passport").Where("id=3").Value()
+    gtest.Assert(err, nil)
+    gtest.Assert(value.String(), "t3")
+
+    result, err = db.Table("user").Filter().Data(&User{
+        Id          : 4,
+        Uid         : 4,
+        Passport    : "t4",
+        Password    : "25d55ad283aa400af464c76d713c07ad",
+        Nickname    : "T4",
+        CreateTime  : gtime.Now().String(),
+    }).Insert()
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 1)
+    value, err = db.Table("user").Fields("passport").Where("id=4").Value()
+    gtest.Assert(err, nil)
+    gtest.Assert(value.String(), "t4")
+
+    result, err = db.Table("user").Where("id>?", 1).Delete()
+    if err != nil {
+        gtest.Fatal(err)
+    }
+    n, _ = result.RowsAffected()
+    gtest.Assert(n, 3)
 }
 
 func TestModel_Batch(t *testing.T) {
@@ -191,6 +255,7 @@ func TestModel_GroupBy(t *testing.T) {
     gtest.Assert(result[0]["nickname"].String(), "T111")
 }
 
+// slice
 func TestModel_Where1(t *testing.T) {
     result, err := db.Table("user").Where("id IN(?)", g.Slice{1,3}).OrderBy("id ASC").All()
     if err != nil {
@@ -201,6 +266,7 @@ func TestModel_Where1(t *testing.T) {
     gtest.Assert(result[1]["id"].Int(), 3)
 }
 
+// slice
 func TestModel_Where2(t *testing.T) {
     result, err := db.Table("user").Where("nickname=? AND id IN(?)", "T3", g.Slice{1,3}).OrderBy("id ASC").All()
     if err != nil {

g/database/gkafka/gkafka.go

@@ -1,242 +0,0 @@
-// Copyright 2018 gf Author(https://github.com/gogf/gf). All Rights Reserved.
-//
-// This Source Code Form is subject to the terms of the MIT License.
-// If a copy of the MIT was not distributed with this file,
-// You can obtain one at https://github.com/gogf/gf.
-
-// Package gkafka provides producer and consumer client for kafka server.
-//
-// Kafka客户端.
-package gkafka
-
-import (
-    "github.com/gogf/gf/g/os/glog"
-    "github.com/gogf/gf/third/github.com/Shopify/sarama"
-    "github.com/gogf/gf/third/github.com/johng-cn/sarama-cluster"
-    "strings"
-    "time"
-)
-
-var (
-    // 当使用Topics方法获取所有topic后，进行过滤忽略的topic，多个以','号分隔
-    ignoreTopics = map[string]bool {
-        "__consumer_offsets" : true,
-    }
-)
-
-// kafka Client based on sarama.Config
-type Config struct {
-    GroupId        string // group id for consumer.
-    Servers        string // server list, multiple servers joined by ','.
-    Topics         string // topic list, multiple topics joined by ','.
-    AutoMarkOffset bool   // auto mark message read after consumer message from server
-    sarama.Config
-}
-
-// Kafka Client(Consumer/SyncProducer/AsyncProducer)
-type Client struct {
-    Config        *Config
-    consumer      *cluster.Consumer
-    rawConsumer   sarama.Consumer
-    syncProducer  sarama.SyncProducer
-    asyncProducer sarama.AsyncProducer
-}
-
-// Kafka Message.
-type Message struct {
-    Value          []byte
-    Key            []byte
-    Topic          string
-    Partition      int
-    Offset         int
-    client         *Client
-    consumerMsg    *sarama.ConsumerMessage
-}
-
-
-// New a kafka client.
-func NewClient(config *Config) *Client {
-    return &Client {
-        Config : config,
-    }
-}
-
-// New a default configuration object.
-func NewConfig() *Config {
-    config       := &Config{}
-    config.Config = *sarama.NewConfig()
-
-    // default config for consumer
-    config.Consumer.Return.Errors          = true
-    config.Consumer.Offsets.Initial        = sarama.OffsetOldest
-    config.Consumer.Offsets.CommitInterval = 1 * time.Second
-
-    // default config for producer
-    config.Producer.Return.Errors          = true
-    config.Producer.Return.Successes       = true
-    config.Producer.Timeout                = 5 * time.Second
-
-    config.AutoMarkOffset                  = true
-    return config
-}
-
-// Close client.
-func (client *Client) Close() {
-    if client.rawConsumer != nil {
-        client.rawConsumer.Close()
-    }
-    if client.consumer != nil {
-        client.consumer.Close()
-    }
-    if client.syncProducer != nil {
-        client.syncProducer.Close()
-    }
-    if client.asyncProducer != nil {
-        client.asyncProducer.Close()
-    }
-}
-
-// Get all topics from kafka server.
-// 这里创建独立的消费客户端获取topics，获取完之后销毁该客户端对象。
-func (client *Client) Topics() ([]string, error) {
-    if c, err := sarama.NewConsumer(strings.Split(client.Config.Servers, ","), &client.Config.Config); err != nil {
-        return nil, err
-    } else {
-        if topics, err := c.Topics(); err == nil {
-            for k, v := range topics {
-                if _, ok := ignoreTopics[v]; ok {
-                    topics = append(topics[ : k], topics[k + 1 : ]...)
-                }
-            }
-            c.Close()
-            return topics, nil
-        } else {
-            return nil, err
-        }
-    }
-}
-
-// 初始化内部消费客户端
-func (client *Client) initConsumer() error {
-    if client.consumer == nil {
-        config       := cluster.NewConfig()
-        config.Config = client.Config.Config
-        config.Group.Return.Notifications = false
-        c, err := cluster.NewConsumer(strings.Split(client.Config.Servers, ","), client.Config.GroupId, strings.Split(client.Config.Topics, ","), config)
-        if err != nil {
-            return err
-        } else {
-            client.consumer = c
-        }
-    }
-    return nil
-}
-
-// 标记指定topic 分区开始读取位置
-func (client *Client) MarkOffset(topic string, partition int, offset int, metadata...string) error {
-    if err := client.initConsumer(); err != nil {
-        return err
-    }
-    meta := ""
-    if len(metadata) > 0 {
-        meta = metadata[0]
-    }
-    client.consumer.MarkPartitionOffset(topic, int32(partition), int64(offset), meta)
-    return nil
-}
-
-
-// Receive message from kafka from specified topics in config, in BLOCKING way, gkafka will handle offset tracking automatically.
-func (client *Client) Receive() (*Message, error) {
-    if err := client.initConsumer(); err != nil {
-        return nil, err
-    }
-    errorsChan  := client.consumer.Errors()
-    notifyChan  := client.consumer.Notifications()
-    messageChan := client.consumer.Messages()
-    for {
-        select {
-            case msg := <- messageChan:
-                if client.Config.AutoMarkOffset {
-                    client.consumer.MarkOffset(msg, "")
-                }
-                return &Message {
-                    Value       : msg.Value,
-                    Key         : msg.Key,
-                    Topic       : msg.Topic,
-                    Partition   : int(msg.Partition),
-                    Offset      : int(msg.Offset),
-                    client      : client,
-                    consumerMsg : msg,
-                }, nil
-
-            case err := <-errorsChan:
-                if err != nil {
-                    return nil, err
-                }
-
-            case <-notifyChan:
-        }
-    }
-}
-
-// Send data to kafka in synchronized way.
-func (client *Client) SyncSend(message *Message) error {
-    if client.syncProducer == nil {
-        if p, err := sarama.NewSyncProducer(strings.Split(client.Config.Servers, ","), &client.Config.Config); err != nil {
-            return err
-        } else {
-            client.syncProducer = p
-        }
-    }
-    for _, topic := range strings.Split(client.Config.Topics, ",") {
-        msg := messageToProducerMessage(message)
-        msg.Topic = topic
-        if _, _, err := client.syncProducer.SendMessage(msg); err != nil {
-            return err
-        }
-    }
-    return nil
-}
-
-// Send data to kafka in asynchronized way(concurrent safe).
-func (client *Client) AsyncSend(message *Message) error {
-    if client.asyncProducer == nil {
-        if p, err := sarama.NewAsyncProducer(strings.Split(client.Config.Servers, ","), &client.Config.Config); err != nil {
-            return err
-        } else {
-            client.asyncProducer = p
-            go func(p sarama.AsyncProducer) {
-               errors  := p.Errors()
-               success := p.Successes()
-               for {
-                   select {
-                       case err := <-errors:
-                           if err != nil {
-                               glog.Error(err)
-                           }
-                       case <-success:
-                   }
-               }
-            }(client.asyncProducer)
-        }
-    }
-
-    for _, topic := range strings.Split(client.Config.Topics, ",") {
-        msg      := messageToProducerMessage(message)
-        msg.Topic = topic
-        client.asyncProducer.Input() <- msg
-    }
-    return nil
-}
-
-// Convert *gkafka.Message to *sarama.ProducerMessage
-func messageToProducerMessage(message *Message) *sarama.ProducerMessage {
-    return &sarama.ProducerMessage {
-        Topic     : message.Topic,
-        Key       : sarama.ByteEncoder(message.Key),
-        Value     : sarama.ByteEncoder(message.Value),
-        Partition : int32(message.Partition),
-        Offset    : int64(message.Offset),
-    }
-}

g/database/gkafka/gkafka_message.go

@@ -1,14 +0,0 @@
-// Copyright 2018 gf Author(https://github.com/gogf/gf). All Rights Reserved.
-//
-// This Source Code Form is subject to the terms of the MIT License.
-// If a copy of the MIT was not distributed with this file,
-// You can obtain one at https://github.com/gogf/gf.
-
-package gkafka
-
-// 自动标记已读取
-func (msg *Message) MarkOffset() {
-    if msg.consumerMsg != nil && msg.client != nil && msg.client.consumer != nil {
-        msg.client.consumer.MarkOffset(msg.consumerMsg, "")
-    }
-}

g/g_func.go

@@ -23,27 +23,36 @@ const (
     LOG_LEVEL_CRIT = glog.LEVEL_CRIT
 )
 
+// NewVar creates a *Var.
+//
 // 动态变量
 func NewVar(i interface{}, unsafe...bool) *Var {
     return gvar.New(i, unsafe...)
 }
 
+// Wait blocks until all the web servers shutdown.
+//
 // 阻塞等待HTTPServer执行完成(同一进程多HTTPServer情况下)
 func Wait() {
     ghttp.Wait()
 }
 
+// Dump dumps a variable to stdout with more manually readable.
+//
 // 打印变量
 func Dump(i...interface{}) {
     gutil.Dump(i...)
 }
 
+// Throw throws a exception, which can be caught by Catch function.
+// It always be used in TryCatch function.
+//
 // 抛出一个异常
 func Throw(exception interface{}) {
     gutil.Throw(exception)
 }
 
-// try...catch...
+// TryCatch does the try...catch... logic.
 func TryCatch(try func(), catch ... func(exception interface{})) {
     gutil.TryCatch(try, catch...)
 }

g/g_logger.go

@@ -10,16 +10,22 @@ import (
     "github.com/gogf/gf/g/os/glog"
 )
 
+// Disable/Enabled debug of logging globally.
+//
 // 是否显示调试信息
 func SetDebug(debug bool) {
     glog.SetDebug(debug)
 }
 
+// Set the logging level globally.
+//
 // 设置日志的显示等级
 func SetLogLevel(level int) {
     glog.SetLevel(level)
 }
 
+// Get the global logging level.
+//
 // 获取设置的日志显示等级
 func GetLogLevel() int {
     return glog.GetLevel()

g/g_object.go

@@ -17,42 +17,58 @@ import (
     "github.com/gogf/gf/g/os/gcfg"
 )
 
+// Get an instance of http server with specified name.
+//
 // HTTPServer单例对象
 func Server(name...interface{}) *ghttp.Server {
     return ghttp.GetServer(name...)
 }
 
+// Get an instance of tcp server with specified name.
+//
 // TCPServer单例对象
 func TCPServer(name...interface{}) *gtcp.Server {
     return gtcp.GetServer(name...)
 }
 
+// Get an instance of udp server with specified name.
+//
 // UDPServer单例对象
 func UDPServer(name...interface{}) *gudp.Server {
     return gudp.GetServer(name...)
 }
 
+// Get an instance of template engine object with specified name.
+//
 // 核心对象：View
 func View(name...string) *gview.View {
     return gins.View(name...)
 }
 
-// Config配置管理对象
+// Get an instance of config object with specified default config file name.
+//
+// Config配置管理对象,
 // 配置文件目录查找依次为：启动参数cfgpath、当前程序运行目录
 func Config(file...string) *gcfg.Config {
     return gins.Config(file...)
 }
 
+// Get an instance of database ORM object with specified configuration group name.
+//
 // 数据库操作对象，使用了连接池
 func Database(name...string) gdb.DB {
     return gins.Database(name...)
 }
 
+// Alias of Database.
+//
 // (别名)Database
 func DB(name...string) gdb.DB {
     return gins.Database(name...)
 }
 
+// Get an instance of redis client with specified configuration group name.
+//
 // Redis操作对象，使用了连接池
 func Redis(name...string) *gredis.Redis {
     return gins.Redis(name...)

g/g_setting.go

@@ -0,0 +1,16 @@
+// Copyright 2019 gf Author(https://github.com/gogf/gf). All Rights Reserved.
+//
+// This Source Code Form is subject to the terms of the MIT License.
+// If a copy of the MIT was not distributed with this file,
+// You can obtain one at https://github.com/gogf/gf.
+
+package g
+
+import "github.com/gogf/gf/g/net/ghttp"
+
+// SetServerGraceful enables/disables graceful reload feature of ghttp Web Server.
+//
+// 是否开启WebServer的平滑重启特性。
+func SetServerGraceful(enabled bool) {
+    ghttp.SetGraceful(enabled)
+}

g/net/ghttp/ghttp_client_request_api.go

@@ -9,44 +9,44 @@
 package ghttp
 
 func Get(url string) (*ClientResponse, error) {
-    return DoRequest("GET", url, []byte(""))
+    return DoRequest("GET", url)
 }
 
-func Put(url, data string) (*ClientResponse, error) {
-    return DoRequest("PUT", url, []byte(data))
+func Put(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("PUT", url, data...)
 }
 
-func Post(url, data string) (*ClientResponse, error) {
-    return DoRequest("POST", url, []byte(data))
+func Post(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("POST", url, data...)
 }
 
-func Delete(url, data string) (*ClientResponse, error) {
-    return DoRequest("DELETE", url, []byte(data))
+func Delete(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("DELETE", url, data...)
 }
 
-func Head(url, data string) (*ClientResponse, error) {
-    return DoRequest("HEAD", url, []byte(data))
+func Head(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("HEAD", url, data...)
 }
 
-func Patch(url, data string) (*ClientResponse, error) {
-    return DoRequest("PATCH", url, []byte(data))
+func Patch(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("PATCH", url, data...)
 }
 
-func Connect(url, data string) (*ClientResponse, error) {
-    return DoRequest("CONNECT", url, []byte(data))
+func Connect(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("CONNECT", url, data...)
 }
 
-func Options(url, data string) (*ClientResponse, error) {
-    return DoRequest("OPTIONS", url, []byte(data))
+func Options(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("OPTIONS", url, data...)
 }
 
-func Trace(url, data string) (*ClientResponse, error) {
-    return DoRequest("TRACE", url, []byte(data))
+func Trace(url string, data...string) (*ClientResponse, error) {
+    return DoRequest("TRACE", url, data...)
 }
 
 // 该方法支持二进制提交数据
-func DoRequest(method, url string, data []byte) (*ClientResponse, error) {
-    return NewClient().DoRequest(method, url, data)
+func DoRequest(method, url string, data...string) (*ClientResponse, error) {
+    return NewClient().DoRequest(method, url, data...)
 }
 
 // GET请求并返回服务端结果(内部会自动读取服务端返回结果并关闭缓冲区指针)

g/net/ghttp/ghttp_client_request_client.go

@@ -24,11 +24,13 @@ import (
 
 // http客户端
 type Client struct {
-    http.Client                // 底层http client对象
-    header   map[string]string // HEADER信息Map
-    prefix   string            // 设置请求的URL前缀
-    authUser string            // HTTP基本权限设置：名称
-    authPass string            // HTTP基本权限设置：密码
+    http.Client                   // 底层http client对象
+    header      map[string]string // HEADER信息Map
+    cookies     map[string]string // 自定义COOKIE
+    prefix      string            // 设置请求的URL前缀
+    authUser    string            // HTTP基本权限设置：名称
+    authPass    string            // HTTP基本权限设置：密码
+    browserMode bool              // 是否模拟浏览器模式(自动保存提交COOKIE)
 }
 
 // http客户端对象指针
@@ -39,10 +41,16 @@ func NewClient() (*Client) {
                 DisableKeepAlives: true,
             },
         },
-        header : make(map[string]string),
+        header  : make(map[string]string),
+        cookies : make(map[string]string),
     }
 }
 
+// 是否模拟浏览器模式(自动保存提交COOKIE)
+func (c *Client) SetBrowserMode(enabled bool) {
+    c.browserMode = enabled
+}
+
 // 设置HTTP Header
 func (c *Client) SetHeader(key, value string) {
     c.header[key] = value
@@ -58,6 +66,18 @@ func (c *Client) SetHeaderRaw(header string) {
     }
 }
 
+// 设置COOKIE
+func (c *Client) SetCookie(key, value string) {
+    c.cookies[key] = value
+}
+
+// 使用Map设置COOKIE
+func (c *Client) SetCookieMap(cookieMap map[string]string) {
+    for k, v := range cookieMap {
+        c.cookies[k] = v
+    }
+}
+
 // 设置请求的URL前缀
 func (c *Client) SetPrefix(prefix string) {
     c.prefix = prefix
@@ -76,26 +96,30 @@ func (c *Client) SetBasicAuth(user, pass string) {
 
 // GET请求
 func (c *Client) Get(url string) (*ClientResponse, error) {
-    return c.DoRequest("GET", url, []byte(""))
+    return c.DoRequest("GET", url)
 }
 
 // PUT请求
-func (c *Client) Put(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("PUT", url, []byte(data))
+func (c *Client) Put(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("PUT", url, data...)
 }
 
 // POST请求提交数据，默认使用表单方式提交数据(绝大部分场景下也是如此)。
 // 如果服务端对Content-Type有要求，可使用Client对象进行请求，单独设置相关属性。
 // 支持文件上传，需要字段格式为：FieldName=@file:
-func (c *Client) Post(url, data string) (*ClientResponse, error) {
+func (c *Client) Post(url string, data...string) (*ClientResponse, error) {
     if len(c.prefix) > 0 {
         url = c.prefix + url
     }
+    param := ""
+    if len(data) > 0 {
+        param = data[0]
+    }
     req := (*http.Request)(nil)
-    if strings.Contains(data, "@file:") {
+    if strings.Contains(param, "@file:") {
         buffer := new(bytes.Buffer)
         writer := multipart.NewWriter(buffer)
-        for _, item := range strings.Split(data, "&") {
+        for _, item := range strings.Split(param, "&") {
             array := strings.Split(item, "=")
             if len(array[1]) > 6 && strings.Compare(array[1][0:6], "@file:") == 0 {
                 path := array[1][6:]
@@ -126,7 +150,7 @@ func (c *Client) Post(url, data string) (*ClientResponse, error) {
             req.Header.Set("Content-Type", writer.FormDataContentType())
         }
     } else {
-        if r, err := http.NewRequest("POST", url, bytes.NewReader([]byte(data))); err != nil {
+        if r, err := http.NewRequest("POST", url, bytes.NewReader([]byte(param))); err != nil {
             return nil, err
         } else {
             req = r
@@ -139,6 +163,19 @@ func (c *Client) Post(url, data string) (*ClientResponse, error) {
             req.Header.Set(k, v)
         }
     }
+    // COOKIE
+    if len(c.cookies) > 0 {
+        headerCookie := ""
+        for k, v := range c.cookies {
+            if len(headerCookie) > 0 {
+                headerCookie += ";"
+            }
+            headerCookie += k + "=" + v
+        }
+        if len(headerCookie) > 0 {
+            req.Header.Set("Cookie", headerCookie)
+        }
+    }
     // HTTP账号密码
     if len(c.authUser) > 0 {
         req.SetBasicAuth(c.authUser, c.authPass)
@@ -148,34 +185,36 @@ func (c *Client) Post(url, data string) (*ClientResponse, error) {
     if err != nil {
         return nil, err
     }
-    r := &ClientResponse{}
-    r.Response = *resp
+    r := &ClientResponse{
+        cookies : make(map[string]string),
+    }
+    r.Response = resp
     return r, nil
 }
 
 // DELETE请求
-func (c *Client) Delete(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("DELETE", url, []byte(data))
+func (c *Client) Delete(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("DELETE", url, data...)
 }
 
-func (c *Client) Head(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("HEAD", url, []byte(data))
+func (c *Client) Head(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("HEAD", url, data...)
 }
 
-func (c *Client) Patch(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("PATCH", url, []byte(data))
+func (c *Client) Patch(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("PATCH", url, data...)
 }
 
-func (c *Client) Connect(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("CONNECT", url, []byte(data))
+func (c *Client) Connect(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("CONNECT", url, data...)
 }
 
-func (c *Client) Options(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("OPTIONS", url, []byte(data))
+func (c *Client) Options(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("OPTIONS", url, data...)
 }
 
-func (c *Client) Trace(url, data string) (*ClientResponse, error) {
-    return c.DoRequest("TRACE", url, []byte(data))
+func (c *Client) Trace(url string, data...string) (*ClientResponse, error) {
+    return c.DoRequest("TRACE", url, data...)
 }
 
 // GET请求并返回服务端结果(内部会自动读取服务端返回结果并关闭缓冲区指针)
@@ -220,11 +259,7 @@ func (c *Client) TraceContent(url string, data...string) string {
 
 // 请求并返回服务端结果(内部会自动读取服务端返回结果并关闭缓冲区指针)
 func (c *Client) DoRequestContent(method string, url string, data...string) string {
-    content := ""
-    if len(data) > 0 {
-        content = data[0]
-    }
-    response, err := c.DoRequest(method, url, []byte(content))
+    response, err := c.DoRequest(method, url, data...)
     if err != nil {
         return ""
     }
@@ -233,14 +268,18 @@ func (c *Client) DoRequestContent(method string, url string, data...string) stri
 }
 
 // 请求并返回response对象，该方法支持二进制提交数据
-func (c *Client) DoRequest(method, url string, data []byte) (*ClientResponse, error) {
+func (c *Client) DoRequest(method, url string, data...string) (*ClientResponse, error) {
     if strings.EqualFold("POST", method) {
-        return c.Post(url, string(data))
+        return c.Post(url, data...)
     }
     if len(c.prefix) > 0 {
         url = c.prefix + url
     }
-    req, err := http.NewRequest(strings.ToUpper(method), url, bytes.NewReader(data))
+    param := ""
+    if len(data) > 0 {
+        param = data[0]
+    }
+    req, err := http.NewRequest(strings.ToUpper(method), url, bytes.NewReader([]byte(param)))
     if err != nil {
         return nil, err
     }
@@ -250,13 +289,40 @@ func (c *Client) DoRequest(method, url string, data []byte) (*ClientResponse, er
             req.Header.Set(k, v)
         }
     }
+    // COOKIE
+    if len(c.cookies) > 0 {
+        headerCookie := ""
+        for k, v := range c.cookies {
+            if len(headerCookie) > 0 {
+                headerCookie += ";"
+            }
+            headerCookie += k + "=" + v
+        }
+        if len(headerCookie) > 0 {
+            req.Header.Set("Cookie", headerCookie)
+        }
+    }
     // 执行请求
     resp, err := c.Do(req)
     if err != nil {
         return nil, err
     }
-    r := &ClientResponse{}
-    r.Response = *resp
+    r := &ClientResponse{
+        cookies : make(map[string]string),
+    }
+    r.Response = resp
+    // 浏览器模式
+    if c.browserMode {
+        now := time.Now()
+        for _, v := range r.Cookies() {
+            if v.Expires.UnixNano() < now.UnixNano() {
+                delete(c.cookies, v.Name)
+            } else {
+                c.cookies[v.Name] = v.Value
+            }
+        }
+    }
+    //fmt.Println(url, c.cookies)
     return r, nil
 }
 

g/net/ghttp/ghttp_client_response.go

@@ -10,14 +10,30 @@ package ghttp
 import (
     "io/ioutil"
     "net/http"
+    "time"
 )
 
 // 客户端请求结果对象
 type ClientResponse struct {
-    http.Response
+    *http.Response
+    cookies map[string]string
 }
 
-// 获取返回的数据
+// 获得返回的指定COOKIE值
+func (r *ClientResponse) GetCookie(key string) string {
+    if r.cookies == nil {
+        now := time.Now()
+        for _, v := range r.Cookies() {
+            if v.Expires.UnixNano() < now.UnixNano() {
+                continue
+            }
+            r.cookies[v.Name] = v.Value
+        }
+    }
+    return r.cookies[key]
+}
+
+// 获取返回的数据(二进制).
 func (r *ClientResponse) ReadAll() []byte {
     body, err := ioutil.ReadAll(r.Body)
     if err != nil {
@@ -26,6 +42,11 @@ func (r *ClientResponse) ReadAll() []byte {
     return body
 }
 
+// 获取返回的数据(字符串).
+func (r *ClientResponse) ReadAllString() string {
+    return string(r.ReadAll())
+}
+
 // 关闭返回的HTTP链接
 func (r *ClientResponse) Close()  {
     r.Response.Close = true

g/net/ghttp/ghttp_request.go

@@ -79,7 +79,7 @@ func (r *Request) GetVar(key string, def ... interface{}) gvar.VarRead {
     return r.GetRequestVar(key, def...)
 }
 
-// 获取原始请求输入字符串
+// 获取原始请求输入二进制。
 func (r *Request) GetRaw() []byte {
     if r.rawContent == nil {
         r.rawContent, _ = ioutil.ReadAll(r.Body)
@@ -87,6 +87,14 @@ func (r *Request) GetRaw() []byte {
     return r.rawContent
 }
 
+// 获取原始请求输入字符串。
+func (r *Request) GetRawString() string {
+    if r.rawContent == nil {
+        r.rawContent, _ = ioutil.ReadAll(r.Body)
+    }
+    return string(r.rawContent)
+}
+
 // 获取原始json请求输入字符串，并解析为json对象
 func (r *Request) GetJson() *gjson.Json {
     data := r.GetRaw()

g/net/ghttp/ghttp_request_query.go

@@ -16,9 +16,10 @@ func (r *Request) initGet() {
     if !r.parsedGet {
         r.queryVars = r.URL.Query()
         if strings.EqualFold(r.Method, "GET") {
-            if raw := r.GetRaw(); len(raw) > 0 {
-                for _, item := range strings.Split(string(raw), "&") {
-                    array                := strings.Split(item, "=")
+            if raw := r.GetRawString(); len(raw) > 0 {
+                var array []string
+                for _, item := range strings.Split(raw, "&") {
+                    array                 = strings.Split(item, "=")
                     r.queryVars[array[0]] = append(r.queryVars[array[0]], array[1])
                 }
             }

g/net/ghttp/ghttp_response.go

@@ -32,7 +32,6 @@ func newResponse(s *Server, w http.ResponseWriter) *Response {
         Server         : s,
         ResponseWriter : ResponseWriter {
             ResponseWriter : w,
-            Status         : http.StatusOK,
             buffer         : bytes.NewBuffer(nil),
         },
     }
@@ -137,9 +136,8 @@ func (r *Response) WriteStatus(status int, content...string) {
         if status != http.StatusOK {
             if f := r.request.Server.getStatusHandler(status, r.request); f != nil {
                 f(r.request)
-                // 如果是http.StatusOK那么表示回调函数内部没有设置header status，
-                // 那么这里就可以设置status，防止多次设置(http: multiple response.WriteHeader calls)
-                if r.Status == http.StatusOK {
+                // 防止多次设置(http: multiple response.WriteHeader calls)
+                if r.Status == 0 {
                     r.WriteHeader(status)
                 }
                 return

g/net/ghttp/ghttp_response_view.go

@@ -65,7 +65,6 @@ func (r *Response) buildInVars(params map[string]interface{}) map[string]interfa
     if params == nil {
         params = make(map[string]interface{})
     }
-
     c := gins.Config()
     if c.GetFilePath() != "" {
         params["Config"]  = c.GetMap("")

g/net/ghttp/ghttp_server.go

@@ -140,17 +140,24 @@ var (
     doneChan         = make(chan struct{}, 1000)
 
     // 用于服务进程初始化，只能初始化一次，采用“懒初始化”(在server运行时才初始化)
-    serverProcInited = gtype.NewBool()
+    serverProcessInited = gtype.NewBool()
+
+    // 是否开启WebServer平滑重启特性, 会开启额外的本地端口监听，用于进程管理通信
+    gracefulEnabled     = true
 )
 
+// 是否开启平滑重启特性
+func SetGraceful(enabled bool) {
+    gracefulEnabled = enabled
+}
 
 // Web Server进程初始化.
 // 注意该方法不能放置于包初始化方法init中，不使用ghttp.Server的功能便不能初始化对应的协程goroutine逻辑.
-func serverProcInit() {
-    if serverProcInited.Val() {
+func serverProcessInit() {
+    if serverProcessInited.Val() {
         return
     }
-    serverProcInited.Set(true)
+    serverProcessInited.Set(true)
     // 如果是完整重启，那么需要等待主进程销毁后，才开始执行监听，防止端口冲突
     if genv.Get(gADMIN_ACTION_RESTART_ENVKEY) != "" {
         if p, e := os.FindProcess(gproc.PPid()); e == nil {
@@ -164,7 +171,9 @@ func serverProcInit() {
     // 信号量管理操作监听
     go handleProcessSignal()
     // 异步监听进程间消息
-    go handleProcessMessage()
+    if gracefulEnabled {
+        go handleProcessMessage()
+    }
 }
 
 // 获取/创建一个默认配置的HTTP Server(默认监听端口是80)
@@ -207,7 +216,7 @@ func GetServer(name...interface{}) (*Server) {
 // 需要结合Wait方式一起使用
 func (s *Server) Start() error {
     // 服务进程初始化，只会初始化一次
-    serverProcInit()
+    serverProcessInit()
 
     // 当前Web Server状态判断
     if s.Status() == SERVER_STATUS_RUNNING {
@@ -253,7 +262,7 @@ func (s *Server) Start() error {
     if gproc.IsChild() {
         gtimer.SetTimeout(2*time.Second, func() {
             if err := gproc.Send(gproc.PPid(), []byte("exit"), gADMIN_GPROC_COMM_GROUP); err != nil {
-                panic(err)
+                glog.Error("ghttp server error in process communication:", err)
             }
         })
     }

g/net/ghttp/ghttp_server_cookie.go

@@ -42,6 +42,7 @@ func GetCookie(r *Request) *Cookie {
     }
     return &Cookie {
         request : r,
+        server  : r.Server,
     }
 }
 
@@ -52,7 +53,6 @@ func (c *Cookie) init() {
         c.path     = c.request.Server.GetCookiePath()
         c.domain   = c.request.Server.GetCookieDomain()
         c.maxage   = c.request.Server.GetCookieMaxAge()
-        c.server   = c.request.Server
         c.response = c.request.Response
         // 如果没有设置COOKIE有效域名，那么设置HOST为默认有效域名
         if c.domain == "" {
@@ -115,6 +115,11 @@ func (c *Cookie) SetCookie(key, value, domain, path string, maxAge int, httpOnly
     }
 }
 
+// 获得客户端提交的SessionId
+func (c *Cookie) GetSessionId() string {
+    return c.Get(c.server.GetSessionIdName())
+}
+
 // 设置SessionId
 func (c *Cookie) SetSessionId(id string) {
     c.Set(c.server.GetSessionIdName(), id)
@@ -133,9 +138,14 @@ func (c *Cookie) Get(key string) string {
     return ""
 }
 
+// 删除COOKIE，使用默认的domain&path
+func (c *Cookie) Remove(key string) {
+    c.SetCookie(key, "", c.domain, c.path, -86400)
+}
+
 // 标记该cookie在对应的域名和路径失效
 // 删除cookie的重点是需要通知浏览器客户端cookie已过期
-func (c *Cookie) Remove(key, domain, path string) {
+func (c *Cookie) RemoveCookie(key, domain, path string) {
     c.SetCookie(key, "", domain, path, -86400)
 }
 

g/net/ghttp/ghttp_server_graceful.go

@@ -11,7 +11,6 @@ import (
     "crypto/tls"
     "errors"
     "fmt"
-    "github.com/gogf/gf/g/net/greuseport"
     "github.com/gogf/gf/g/os/glog"
     "github.com/gogf/gf/g/os/gproc"
     "net"
@@ -146,7 +145,7 @@ func (s *gracefulServer) getNetListener(addr string) (net.Listener, error) {
     } else {
         // 如果监听失败，1秒后重试，最多重试3次
         for i := 0; i < 3; i++ {
-            ln, err = greuseport.Listen("tcp", addr)
+            ln, err = net.Listen("tcp", addr)
             if err != nil {
                 err = fmt.Errorf("%d: net.Listen error: %v", gproc.Pid(), err)
                 time.Sleep(time.Second)

g/net/ghttp/ghttp_server_handler.go

@@ -46,14 +46,28 @@ func (s *Server)handleRequest(w http.ResponseWriter, r *http.Request) {
     request := newRequest(s, r, w)
 
     defer func() {
-        if request.LeaveTime == 0 {
-            request.LeaveTime = gtime.Microsecond()
+        // 设置请求完成时间
+        request.LeaveTime = gtime.Microsecond()
+        // 事件 - BeforeOutput
+        if !request.IsExited() {
+            s.callHookHandler(HOOK_BEFORE_OUTPUT, request)
         }
+        // 输出Cookie
+        request.Cookie.Output()
+        // 输出缓冲区
+        request.Response.OutputBuffer()
+        // 事件 - AfterOutput
+        if !request.IsExited() {
+            s.callHookHandler(HOOK_AFTER_OUTPUT, request)
+        }
+
+        // 事件 - BeforeClose
         s.callHookHandler(HOOK_BEFORE_CLOSE, request)
         // access log
         s.handleAccessLog(request)
         // error log使用recover进行判断
         if e := recover(); e != nil {
+            request.Response.WriteStatus(http.StatusInternalServerError)
             s.handleErrorLog(e, request)
         }
         // 更新Session会话超时时间
@@ -125,22 +139,6 @@ func (s *Server)handleRequest(w http.ResponseWriter, r *http.Request) {
     if !request.IsExited() {
         s.callHookHandler(HOOK_AFTER_SERVE, request)
     }
-
-    // 设置请求完成时间
-    request.LeaveTime = gtime.Microsecond()
-
-    // 事件 - BeforeOutput
-    if !request.IsExited() {
-        s.callHookHandler(HOOK_BEFORE_OUTPUT, request)
-    }
-    // 输出Cookie
-    request.Cookie.Output()
-    // 输出缓冲区
-    request.Response.OutputBuffer()
-    // 事件 - AfterOutput
-    if !request.IsExited() {
-        s.callHookHandler(HOOK_AFTER_OUTPUT, request)
-    }
 }
 
 // 查找静态文件的绝对路径

g/net/ghttp/ghttp_server_log.go

@@ -10,7 +10,6 @@ package ghttp
 import (
     "fmt"
     "github.com/gogf/gf/g/os/gfile"
-    "net/http"
 )
 
 // 处理服务错误信息，主要是panic，http请求的status由access log进行管理
@@ -34,8 +33,6 @@ func (s *Server) handleAccessLog(r *Request) {
 
 // 处理服务错误信息，主要是panic，http请求的status由access log进行管理
 func (s *Server) handleErrorLog(error interface{}, r *Request) {
-    r.Response.WriteStatus(http.StatusInternalServerError)
-
     // 错误输出默认是开启的
     if !s.IsErrorLogEnabled() && gfile.MainPkgPath() == "" {
         return

g/net/ghttp/ghttp_server_service_controller.go

@@ -64,15 +64,15 @@ func (s *Server)BindController(pattern string, c Controller, methods...string) e
             faddr : nil,
         }
         // 如果方法中带有Index方法，那么额外自动增加一个路由规则匹配主URI
+        // 例如: pattern为/user, 那么会同时注册/user及/user/index,
+        // 这里处理新增/user路由绑定
         if strings.EqualFold(mname, "Index") {
-            p := key
-            if strings.EqualFold(p[len(p) - 6:], "/index") {
-                p = p[0 : len(p) - 6]
-                if len(p) == 0 {
-                    p = "/"
-                }
+            p := gstr.PosR(key, "/index")
+            k := key[0 : p] + key[p + 6 : ]
+            if len(k) == 0 {
+                k = "/"
             }
-            m[p] = &handlerItem {
+            m[k] = &handlerItem {
                 name  : fmt.Sprintf(`%s.%s.%s`, pkgPath, ctlName, mname),
                 rtype : gROUTE_REGISTER_CONTROLLER,
                 ctype : v.Elem().Type(),

g/net/ghttp/ghttp_server_service_object.go

@@ -74,14 +74,12 @@ func (s *Server)BindObject(pattern string, obj interface{}, methods...string) er
         }
         // 如果方法中带有Index方法，那么额外自动增加一个路由规则匹配主URI
         if strings.EqualFold(mname, "Index") {
-            p := key
-            if strings.EqualFold(p[len(p) - 6:], "/index") {
-                p = p[0 : len(p) - 6]
-                if len(p) == 0 {
-                    p = "/"
-                }
+            p := gstr.PosR(key, "/index")
+            k := key[0 : p] + key[p + 6 : ]
+            if len(k) == 0 {
+                k = "/"
             }
-            m[p] = &handlerItem {
+            m[k] = &handlerItem {
                 name  : fmt.Sprintf(`%s.%s.%s`, pkgPath, objName, mname),
                 rtype : gROUTE_REGISTER_OBJECT,
                 ctype : nil,

g/net/ghttp/ghttp_server_session.go

@@ -52,7 +52,7 @@ func (s *Session) init() {
     }
 }
 
-// 获取SessionId
+// 获取/创建SessionId
 func (s *Session) Id() string {
     s.init()
     return s.id
@@ -60,8 +60,11 @@ func (s *Session) Id() string {
 
 // 获取当前session所有数据
 func (s *Session) Data() map[string]interface{} {
-    s.init()
-    return s.data.Map()
+    if len(s.id) > 0 || s.request.Cookie.GetSessionId() != "" {
+        s.init()
+        return s.data.Map()
+    }
+    return nil
 }
 
 // 设置session
@@ -84,70 +87,158 @@ func (s *Session) BatchSet(m map[string]interface{}) {
 
 // 判断键名是否存在
 func (s *Session) Contains (key string) bool {
-    s.init()
-    return s.data.Contains(key)
+    if len(s.id) > 0 || s.request.Cookie.GetSessionId() != "" {
+        s.init()
+        return s.data.Contains(key)
+    }
+    return false
 }
 
 // 获取SESSION
 func (s *Session) Get (key string) interface{}  {
-    s.init()
-    return s.data.Get(key)
+    if len(s.id) > 0 || s.request.Cookie.GetSessionId() != "" {
+        s.init()
+        return s.data.Get(key)
+    }
+    return nil
 }
 
 // 获取SESSION，建议都用该方法获取参数
 func (s *Session) GetVar(key string) gvar.VarRead  {
-    s.init()
-    return gvar.NewRead(s.data.Get(key), true)
-}
-
-
-func (s *Session) GetString (key string)        string          { return gconv.String(s.Get(key)) }
-func (s *Session) GetBool(key string)           bool            { return gconv.Bool(s.Get(key))   }
-
-func (s *Session) GetInt(key string)            int             { return gconv.Int(s.Get(key))   }
-func (s *Session) GetInt8(key string)           int8            { return gconv.Int8(s.Get(key))  }
-func (s *Session) GetInt16(key string)          int16           { return gconv.Int16(s.Get(key)) }
-func (s *Session) GetInt32(key string)          int32           { return gconv.Int32(s.Get(key)) }
-func (s *Session) GetInt64(key string)          int64           { return gconv.Int64(s.Get(key)) }
-
-func (s *Session) GetUint(key string)           uint            { return gconv.Uint(s.Get(key))   }
-func (s *Session) GetUint8(key string)          uint8           { return gconv.Uint8(s.Get(key))  }
-func (s *Session) GetUint16(key string)         uint16          { return gconv.Uint16(s.Get(key)) }
-func (s *Session) GetUint32(key string)         uint32          { return gconv.Uint32(s.Get(key)) }
-func (s *Session) GetUint64(key string)         uint64          { return gconv.Uint64(s.Get(key)) }
-
-func (s *Session) GetFloat32 (key string)       float32         { return gconv.Float32(s.Get(key)) }
-func (s *Session) GetFloat64 (key string)       float64         { return gconv.Float64(s.Get(key)) }
-
-func (s *Session) GetBytes(key string)          []byte          { return gconv.Bytes(s.Get(key))      }
-func (s *Session) GetInts(key string)           []int           { return gconv.Ints(s.Get(key))       }
-func (s *Session) GetFloats(key string)         []float64       { return gconv.Floats(s.Get(key))     }
-func (s *Session) GetStrings(key string)        []string        { return gconv.Strings(s.Get(key))    }
-func (s *Session) GetInterfaces(key string)     []interface{}   { return gconv.Interfaces(s.Get(key)) }
-
-func (s *Session) GetTime(key string, format...string) time.Time       { return gconv.Time(s.Get(key), format...) }
-func (s *Session) GetTimeDuration(key string)          time.Duration   { return gconv.TimeDuration(s.Get(key)) }
-
-// 将变量转换为对象，注意 objPointer 参数必须为struct指针
-func (s *Session) GetStruct(key string, objPointer interface{}, attrMapping...map[string]string) error {
-    return gconv.Struct(s.Get(key), objPointer, attrMapping...)
+    return gvar.NewRead(s.Get(key), true)
 }
 
 // 删除session
 func (s *Session) Remove(key string) {
-    s.init()
-    s.data.Remove(key)
+    if len(s.id) > 0 || s.request.Cookie.GetSessionId() != "" {
+        s.init()
+        s.data.Remove(key)
+    }
 }
 
 // 清空session
 func (s *Session) Clear() {
-    s.init()
-    s.data.Clear()
+    if len(s.id) > 0 || s.request.Cookie.GetSessionId() != "" {
+        s.init()
+        s.data.Clear()
+    }
 }
 
 // 更新过期时间(如果用在守护进程中长期使用，需要手动调用进行更新，防止超时被清除)
 func (s *Session) UpdateExpire() {
-    if len(s.id) > 0 {
+    if len(s.id) > 0 && s.data.Size() > 0 {
         s.server.sessions.Set(s.id, s.data, s.server.GetSessionMaxAge()*1000)
     }
-}
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetString(key string) string {
+    return gconv.String(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetBool(key string) bool {
+    return gconv.Bool(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInt(key string) int {
+    return gconv.Int(s.Get(key)) }
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInt8(key string) int8 {
+    return gconv.Int8(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInt16(key string) int16 {
+    return gconv.Int16(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInt32(key string) int32 {
+    return gconv.Int32(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInt64(key string) int64 {
+    return gconv.Int64(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetUint(key string) uint {
+    return gconv.Uint(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetUint8(key string) uint8 {
+    return gconv.Uint8(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetUint16(key string) uint16 {
+    return gconv.Uint16(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetUint32(key string) uint32 {
+    return gconv.Uint32(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetUint64(key string) uint64 {
+    return gconv.Uint64(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetFloat32(key string) float32 {
+    return gconv.Float32(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetFloat64(key string) float64 {
+    return gconv.Float64(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetBytes(key string) []byte {
+    return gconv.Bytes(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInts(key string) []int {
+    return gconv.Ints(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetFloats(key string) []float64 {
+    return gconv.Floats(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetStrings(key string) []string {
+    return gconv.Strings(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetInterfaces(key string) []interface{} {
+    return gconv.Interfaces(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetTime(key string, format...string) time.Time {
+    return gconv.Time(s.Get(key), format...)
+}
+
+// Deprecated, use GetVar instead.
+func (s *Session) GetTimeDuration(key string) time.Duration {
+    return gconv.TimeDuration(s.Get(key))
+}
+
+// Deprecated, use GetVar instead.
+// (已废弃, 请使用GetVar) 将变量转换为对象，注意 objPointer 参数必须为struct指针
+func (s *Session) GetStruct(key string, objPointer interface{}, attrMapping...map[string]string) error {
+    return gconv.Struct(s.Get(key), objPointer, attrMapping...)
+}
+
+

g/net/ghttp/ghttp_unit_1_test.go

@@ -1,55 +0,0 @@
-// Copyright 2018 gf Author(https://github.com/gogf/gf). All Rights Reserved.
-//
-// This Source Code Form is subject to the terms of the MIT License.
-// If a copy of the MIT was not distributed with this file,
-// You can obtain one at https://github.com/gogf/gf.
-
-// 基本路由功能以及优先级测试
-package ghttp_test
-
-import (
-    "github.com/gogf/gf/g"
-    "github.com/gogf/gf/g/net/ghttp"
-    "github.com/gogf/gf/g/os/gtime"
-    "github.com/gogf/gf/g/test/gtest"
-    "testing"
-    "time"
-)
-
-
-func Test_Router_Basic(t *testing.T) {
-    s := g.Server(gtime.Nanosecond())
-    s.BindHandler("/:name", func(r *ghttp.Request){
-        r.Response.Write("/:name")
-    })
-    s.BindHandler("/:name/update", func(r *ghttp.Request){
-        r.Response.Write(r.Get("name"))
-    })
-    s.BindHandler("/:name/:action", func(r *ghttp.Request){
-        r.Response.Write(r.Get("action"))
-    })
-    s.BindHandler("/:name/*any", func(r *ghttp.Request){
-        r.Response.Write(r.Get("any"))
-    })
-    s.BindHandler("/user/list/{field}.html", func(r *ghttp.Request){
-        r.Response.Write(r.Get("field"))
-    })
-    s.SetPort(8100)
-    s.SetDumpRouteMap(false)
-    go s.Run()
-    defer func() {
-        s.Shutdown()
-        time.Sleep(time.Second)
-    }()
-    // 等待启动完成
-    time.Sleep(time.Second)
-    gtest.Case(t, func() {
-        client := ghttp.NewClient()
-        client.SetPrefix("http://127.0.0.1:8100")
-        
-        gtest.Assert(client.GetContent("/john"),               "")
-        gtest.Assert(client.GetContent("/john/update"),        "john")
-        gtest.Assert(client.GetContent("/john/edit"),          "edit")
-        gtest.Assert(client.GetContent("/user/list/100.html"), "100")
-    })
-}

g/net/ghttp/ghttp_unit_cookie_test.go

@@ -0,0 +1,62 @@
+// Copyright 2018 gf Author(https://github.com/gogf/gf). All Rights Reserved.
+//
+// This Source Code Form is subject to the terms of the MIT License.
+// If a copy of the MIT was not distributed with this file,
+// You can obtain one at https://github.com/gogf/gf.
+
+// COOKIE测试
+package ghttp_test
+
+import (
+    "github.com/gogf/gf/g"
+    "github.com/gogf/gf/g/net/ghttp"
+    "github.com/gogf/gf/g/os/gtime"
+    "github.com/gogf/gf/g/test/gtest"
+    "testing"
+    "time"
+)
+
+func Test_Cookie(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("/set", func(r *ghttp.Request){
+        r.Cookie.Set(r.Get("k"), r.Get("v"))
+    })
+    s.BindHandler("/get", func(r *ghttp.Request){
+        //fmt.Println(r.Cookie.Map())
+        r.Response.Write(r.Cookie.Get(r.Get("k")))
+    })
+    s.BindHandler("/remove", func(r *ghttp.Request){
+        r.Cookie.Remove(r.Get("k"))
+    })
+    s.SetPort(8500)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetBrowserMode(true)
+        client.SetPrefix("http://127.0.0.1:8500")
+        r1, e1 := client.Get("/set?k=key1&v=100")
+        if r1 != nil {
+            defer r1.Close()
+        }
+        gtest.Assert(e1, nil)
+        gtest.Assert(r1.ReadAllString(), "")
+
+        gtest.Assert(client.GetContent("/set?k=key2&v=200"), "")
+
+        gtest.Assert(client.GetContent("/get?k=key1"),    "100")
+        gtest.Assert(client.GetContent("/get?k=key2"),    "200")
+        gtest.Assert(client.GetContent("/get?k=key3"),    "")
+        gtest.Assert(client.GetContent("/remove?k=key1"), "")
+        gtest.Assert(client.GetContent("/remove?k=key3"), "")
+        gtest.Assert(client.GetContent("/remove?k=key4"), "")
+        gtest.Assert(client.GetContent("/get?k=key1"),    "")
+        gtest.Assert(client.GetContent("/get?k=key2"),    "200")
+    })
+}

g/net/ghttp/ghttp_unit_param_test.go

@@ -16,7 +16,7 @@ import (
     "time"
 )
 
-func Test_Params(t *testing.T) {
+func Test_Params_Basic(t *testing.T) {
     type User struct {
         Id    int
         Name  string

g/net/ghttp/ghttp_unit_router_basic_test.go

@@ -0,0 +1,208 @@
+// Copyright 2018 gf Author(https://github.com/gogf/gf). All Rights Reserved.
+//
+// This Source Code Form is subject to the terms of the MIT License.
+// If a copy of the MIT was not distributed with this file,
+// You can obtain one at https://github.com/gogf/gf.
+
+// 基本路由功能以及优先级测试
+package ghttp_test
+
+import (
+    "github.com/gogf/gf/g"
+    "github.com/gogf/gf/g/net/ghttp"
+    "github.com/gogf/gf/g/os/gtime"
+    "github.com/gogf/gf/g/test/gtest"
+    "testing"
+    "time"
+)
+
+
+// 基本路由功能测试
+func Test_Router_Basic(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("/:name", func(r *ghttp.Request){
+        r.Response.Write("/:name")
+    })
+    s.BindHandler("/:name/update", func(r *ghttp.Request){
+        r.Response.Write(r.Get("name"))
+    })
+    s.BindHandler("/:name/:action", func(r *ghttp.Request){
+        r.Response.Write(r.Get("action"))
+    })
+    s.BindHandler("/:name/*any", func(r *ghttp.Request){
+        r.Response.Write(r.Get("any"))
+    })
+    s.BindHandler("/user/list/{field}.html", func(r *ghttp.Request){
+        r.Response.Write(r.Get("field"))
+    })
+    s.SetPort(8100)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetPrefix("http://127.0.0.1:8100")
+        
+        gtest.Assert(client.GetContent("/john"),               "")
+        gtest.Assert(client.GetContent("/john/update"),        "john")
+        gtest.Assert(client.GetContent("/john/edit"),          "edit")
+        gtest.Assert(client.GetContent("/user/list/100.html"), "100")
+    })
+}
+
+// 测试HTTP Method注册.
+func Test_Router_Method(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("GET:/get", func(r *ghttp.Request){
+
+    })
+    s.BindHandler("POST:/post", func(r *ghttp.Request){
+
+    })
+    s.SetPort(8105)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetPrefix("http://127.0.0.1:8105")
+
+        resp1, err := client.Get("/get")
+        defer resp1.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp1.StatusCode, 200)
+
+        resp2, err := client.Post("/get")
+        defer resp2.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp2.StatusCode, 404)
+
+        resp3, err := client.Get("/post")
+        defer resp3.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp3.StatusCode, 404)
+
+        resp4, err := client.Post("/post")
+        defer resp4.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp4.StatusCode, 200)
+    })
+}
+
+// 测试状态返回.
+func Test_Router_Status(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("/200", func(r *ghttp.Request){
+        r.Response.WriteStatus(200)
+    })
+    s.BindHandler("/300", func(r *ghttp.Request){
+        r.Response.WriteStatus(300)
+    })
+    s.BindHandler("/400", func(r *ghttp.Request){
+        r.Response.WriteStatus(400)
+    })
+    s.BindHandler("/500", func(r *ghttp.Request){
+        r.Response.WriteStatus(500)
+    })
+    s.SetPort(8110)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetPrefix("http://127.0.0.1:8110")
+
+        resp1, err := client.Get("/200")
+        defer resp1.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp1.StatusCode, 200)
+
+        resp2, err := client.Get("/300")
+        defer resp2.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp2.StatusCode, 300)
+
+        resp3, err := client.Get("/400")
+        defer resp3.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp3.StatusCode, 400)
+
+        resp4, err := client.Get("/500")
+        defer resp4.Close()
+        gtest.Assert(err,              nil)
+        gtest.Assert(resp4.StatusCode, 500)
+    })
+}
+
+// 自定义状态码处理.
+func Test_Router_CustomStatusHandler(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("/", func(r *ghttp.Request){
+        r.Response.Write("hello")
+    })
+    s.BindStatusHandler(404, func(r *ghttp.Request){
+        r.Response.Write("404 page")
+    })
+    s.SetPort(8120)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetPrefix("http://127.0.0.1:8120")
+
+        gtest.Assert(client.GetContent("/"), "hello")
+        resp, err := client.Get("/ThisDoesNotExist")
+        defer resp.Close()
+        gtest.Assert(err,             nil)
+        gtest.Assert(resp.StatusCode, 404)
+        gtest.Assert(resp.ReadAllString(), "404 page")
+    })
+}
+
+// 测试不存在的路由.
+func Test_Router_404(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("/", func(r *ghttp.Request){
+        r.Response.Write("hello")
+    })
+    s.SetPort(8130)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetPrefix("http://127.0.0.1:8130")
+
+        gtest.Assert(client.GetContent("/"), "hello")
+        resp, err := client.Get("/ThisDoesNotExist")
+        defer resp.Close()
+        gtest.Assert(err,             nil)
+        gtest.Assert(resp.StatusCode, 404)
+    })
+}

g/net/ghttp/ghttp_unit_router_group_test.go

@@ -20,6 +20,10 @@ import (
 // 执行对象
 type Object struct {}
 
+func (o *Object) Index(r *ghttp.Request) {
+    r.Response.Write("Object Index")
+}
+
 func (o *Object) Show(r *ghttp.Request) {
     r.Response.Write("Object Show")
 }
@@ -33,6 +37,10 @@ type Controller struct {
     gmvc.Controller
 }
 
+func (c *Controller) Index() {
+    c.Response.Write("Controller Index")
+}
+
 func (c *Controller) Show() {
     c.Response.Write("Controller Show")
 }
@@ -72,16 +80,27 @@ func Test_Router_Group1(t *testing.T) {
 
         gtest.Assert(client.GetContent ("/api/handler"),     "Handler")
 
+        gtest.Assert(client.GetContent ("/api/ctl"),         "Controller Index")
+        gtest.Assert(client.GetContent ("/api/ctl/"),        "Controller Index")
+        gtest.Assert(client.GetContent ("/api/ctl/index"),   "Controller Index")
         gtest.Assert(client.GetContent ("/api/ctl/my-show"), "Controller Show")
         gtest.Assert(client.GetContent ("/api/ctl/post"),    "Controller REST Post")
         gtest.Assert(client.GetContent ("/api/ctl/show"),    "Controller Show")
         gtest.Assert(client.PostContent("/api/ctl/rest"),    "Controller REST Post")
 
+        gtest.Assert(client.GetContent ("/api/obj"),         "Object Index")
+        gtest.Assert(client.GetContent ("/api/obj/"),        "Object Index")
+        gtest.Assert(client.GetContent ("/api/obj/index"),   "Object Index")
         gtest.Assert(client.GetContent ("/api/obj/delete"),  "Object REST Delete")
         gtest.Assert(client.GetContent ("/api/obj/my-show"), "Object Show")
         gtest.Assert(client.GetContent ("/api/obj/show"),    "Object Show")
         gtest.Assert(client.DeleteContent("/api/obj/rest"),  "Object REST Delete")
 
+        // 测试404
+        resp, err := client.Get("/ThisDoesNotExist")
+        defer resp.Close()
+        gtest.Assert(err,             nil)
+        gtest.Assert(resp.StatusCode, 404)
     })
 }
 
@@ -122,5 +141,11 @@ func Test_Router_Group2(t *testing.T) {
         gtest.Assert(client.GetContent ("/api/obj/my-show"), "Object Show")
         gtest.Assert(client.GetContent ("/api/obj/show"),    "Object Show")
         gtest.Assert(client.DeleteContent("/api/obj/rest"),  "Object REST Delete")
+
+        // 测试404
+        resp, err := client.Get("/ThisDoesNotExist")
+        defer resp.Close()
+        gtest.Assert(err,             nil)
+        gtest.Assert(resp.StatusCode, 404)
     })
 }

g/net/ghttp/ghttp_unit_session_test.go

@@ -0,0 +1,66 @@
+// Copyright 2018 gf Author(https://github.com/gogf/gf). All Rights Reserved.
+//
+// This Source Code Form is subject to the terms of the MIT License.
+// If a copy of the MIT was not distributed with this file,
+// You can obtain one at https://github.com/gogf/gf.
+
+// SESSION测试
+package ghttp_test
+
+import (
+    "github.com/gogf/gf/g"
+    "github.com/gogf/gf/g/net/ghttp"
+    "github.com/gogf/gf/g/os/gtime"
+    "github.com/gogf/gf/g/test/gtest"
+    "testing"
+    "time"
+)
+
+func Test_Session(t *testing.T) {
+    s := g.Server(gtime.Nanosecond())
+    s.BindHandler("/set", func(r *ghttp.Request){
+        r.Session.Set(r.Get("k"), r.Get("v"))
+    })
+    s.BindHandler("/get", func(r *ghttp.Request){
+        r.Response.Write(r.Session.Get(r.Get("k")))
+    })
+    s.BindHandler("/remove", func(r *ghttp.Request){
+        r.Session.Remove(r.Get("k"))
+    })
+    s.BindHandler("/clear", func(r *ghttp.Request){
+        r.Session.Clear()
+    })
+    s.SetPort(8600)
+    s.SetDumpRouteMap(false)
+    go s.Run()
+    defer func() {
+        s.Shutdown()
+        time.Sleep(time.Second)
+    }()
+    // 等待启动完成
+    time.Sleep(time.Second)
+    gtest.Case(t, func() {
+        client := ghttp.NewClient()
+        client.SetBrowserMode(true)
+        client.SetPrefix("http://127.0.0.1:8600")
+        r1, e1 := client.Get("/set?k=key1&v=100")
+        if r1 != nil {
+            defer r1.Close()
+        }
+        gtest.Assert(e1, nil)
+        gtest.Assert(r1.ReadAllString(), "")
+
+        gtest.Assert(client.GetContent("/set?k=key2&v=200"), "")
+
+        gtest.Assert(client.GetContent("/get?k=key1"),    "100")
+        gtest.Assert(client.GetContent("/get?k=key2"),    "200")
+        gtest.Assert(client.GetContent("/get?k=key3"),    "")
+        gtest.Assert(client.GetContent("/remove?k=key1"), "")
+        gtest.Assert(client.GetContent("/remove?k=key3"), "")
+        gtest.Assert(client.GetContent("/remove?k=key4"), "")
+        gtest.Assert(client.GetContent("/get?k=key1"),    "")
+        gtest.Assert(client.GetContent("/get?k=key2"),    "200")
+        gtest.Assert(client.GetContent("/clear"),         "")
+        gtest.Assert(client.GetContent("/get?k=key2"),    "")
+    })
+}

g/os/gcfg/gcfg.go

@@ -63,13 +63,8 @@ func (c *Config) filePath(file...string) (path string) {
     }
     c.paths.RLockFunc(func(array []string) {
         for _, v := range array {
-            //fmt.Println("search:", v, name)
             if path, _ = gspath.Search(v, name); path != "" {
                 break
-            } else {
-                //if strings.EqualFold(v, "/Users/john/Temp/config") {
-                //    gutil.Dump(gspath.Get(v).AllPaths())
-                //}
             }
         }
     })
@@ -94,6 +89,10 @@ func (c *Config) SetPath(path string) error {
         glog.Error(fmt.Sprintf(`[gcfg] SetPath failed: %s`, err.Error()))
         return err
     }
+    // 重复判断
+    if c.paths.Search(realPath) != -1 {
+        return nil
+    }
     c.jsons.Clear()
     c.paths.Clear()
     c.paths.Append(realPath)
@@ -116,14 +115,29 @@ func (c *Config) AddPath(path string) error {
         glog.Error(fmt.Sprintf(`[gcfg] AddPath failed: %s`, err.Error()))
         return err
     }
+    // 重复判断
+    if c.paths.Search(realPath) != -1 {
+        return nil
+    }
     c.paths.Append(realPath)
     glog.Debug("[gcfg] AddPath:", realPath)
     return nil
 }
 
-// 获取指定文件的绝对路径，默认获取默认的配置文件路径
-func (c *Config) GetFilePath(file...string) string {
-    return c.filePath(file...)
+// 获取指定文件的绝对路径，默认获取默认的配置文件路径，当指定的配置文件不存在时，返回空字符串，并且不会报错。
+func (c *Config) GetFilePath(file...string) (path string) {
+    name := c.name.Val()
+    if len(file) > 0 {
+        name = file[0]
+    }
+    c.paths.RLockFunc(func(array []string) {
+        for _, v := range array {
+            if path, _ = gspath.Search(v, name); path != "" {
+                break
+            }
+        }
+    })
+    return
 }
 
 // 设置配置管理对象的默认文件名称

g/os/gcron/gcron_schedule.go

@@ -117,23 +117,23 @@ func newSchedule(pattern string) (*cronSchedule, error) {
             schedule.hour = m
         }
         // 天
-        if m, err := parseItem(match[4], 1, 31, false); err != nil {
+        if m, err := parseItem(match[4], 1, 31, true); err != nil {
             return nil, err
         } else {
             schedule.day = m
         }
-        // 周
-        if m, err := parseItem(match[5], 0, 6, false); err != nil {
-            return nil, err
-        } else {
-            schedule.week = m
-        }
         // 月
-        if m, err := parseItem(match[6], 1, 12, false); err != nil {
+        if m, err := parseItem(match[5], 1, 12, false); err != nil {
             return nil, err
         } else {
             schedule.month = m
         }
+        // 周
+        if m, err := parseItem(match[6], 0, 6, true); err != nil {
+            return nil, err
+        } else {
+            schedule.week = m
+        }
         return schedule, nil
     } else {
         return nil, errors.New(fmt.Sprintf(`invalid pattern: "%s"`, pattern))
@@ -200,14 +200,14 @@ func parseItemValue(value string, valueType byte) (int, error) {
     } else {
         // 英文字母
         switch valueType {
-            case 'w':
-                if i, ok := weekMap[strings.ToLower(value)]; ok {
-                    return int(i), nil
-                }
             case 'm':
                 if i, ok := monthMap[strings.ToLower(value)]; ok {
                     return int(i), nil
                 }
+            case 'w':
+                if i, ok := weekMap[strings.ToLower(value)]; ok {
+                    return int(i), nil
+                }
         }
     }
     return 0, errors.New(fmt.Sprintf(`invalid pattern value: "%s"`, value))
@@ -234,10 +234,10 @@ func (s *cronSchedule) meet(t time.Time) bool {
         if _, ok := s.day[t.Day()]; !ok {
             return false
         }
-        if _, ok := s.week[int(t.Weekday())]; !ok {
+        if _, ok := s.month[int(t.Month())]; !ok {
             return false
         }
-        if _, ok := s.month[int(t.Month())]; !ok {
+        if _, ok := s.week[int(t.Weekday())]; !ok {
             return false
         }
         return true

g/os/gfcache/gfcache_cache.go

@@ -47,31 +47,16 @@ func (c *Cache) GetBinContents(path string) []byte {
     return b
 }
 
-// 添加文件监控
+// 添加文件监控，一旦文件有变化立即清除缓存，下一次读取的时候再执行缓存。
 func (c *Cache) addMonitor(path string) {
     // 防止多goroutine同时调用
     if c.cache.Contains(path) {
         return
     }
     gfsnotify.Add(path, func(event *gfsnotify.Event) {
-        //glog.Debug("gfcache:", event)
-        length := 0
         if r := c.cache.Get(path); r != nil {
-            length = len(r.([]byte))
-        }
-        // 是否删除
-        if event.IsRemove() {
             c.cache.Remove(path)
-            c.size.Add(-length)
-            return
-        }
-        // 更新缓存内容
-        if c.cap.Val() == 0 || c.size.Val() < c.cap.Val() {
-            b := gfile.GetBinContents(path)
-            if len(b) > 0 {
-                c.size.Add(len(b) - length)
-                c.cache.Set(path, b)
-            }
+            c.size.Add(-len(r.([]byte)))
         }
     })
 }

g/os/gfile/gfile.go

@@ -28,8 +28,8 @@ import (
     "time"
 )
 
-// 文件分隔符
 const (
+    // 文件分隔符
     Separator     = string(filepath.Separator)
     // 默认的文件打开权限
     gDEFAULT_PERM = 0666
@@ -43,7 +43,9 @@ var (
     goRootOfBuild = gtype.NewString()
 )
 
-// 给定文件的绝对路径创建文件
+// Create directories recursively.
+//
+// 给定目录的绝对路径创建目录(递归创建)。
 func Mkdir(path string) error {
     err  := os.MkdirAll(path, os.ModePerm)
     if err != nil {
@@ -52,29 +54,33 @@ func Mkdir(path string) error {
     return nil
 }
 
-// 给定文件的绝对路径创建文件
-func Create(path string) error {
+// Create file with given path recursively.
+//
+// 给定文件的绝对路径创建文件。
+func Create(path string) (*os.File, error) {
     dir := Dir(path)
     if !Exists(dir) {
         Mkdir(dir)
     }
-    f, err  := os.Create(path)
-    if err != nil {
-        return err
-    }
-    f.Close()
-    return nil
+    return os.Create(path)
 }
 
-// 打开文件(os.O_RDWR|os.O_CREATE, 0666)
+// Open file/directory with readonly.
+//
+// 只读打开文件
 func Open(path string) (*os.File, error) {
-    f, err  := os.OpenFile(path, os.O_RDWR|os.O_CREATE, gDEFAULT_PERM)
-    if err != nil {
-        return nil, err
-    }
-    return f, nil
+    return os.Open(path)
 }
 
+// Open file/directory with given <flag> and <perm>.
+//
+// 打开文件(带flag&perm)
+func OpenFile(path string, flag int, perm os.FileMode) (*os.File, error) {
+    return os.OpenFile(path, flag, perm)
+}
+
+// Open file/directory with default perm and given <flag>.
+//
 // 打开文件(带flag)
 func OpenWithFlag(path string, flag int) (*os.File, error) {
     f, err  := os.OpenFile(path, flag, gDEFAULT_PERM)
@@ -84,6 +90,8 @@ func OpenWithFlag(path string, flag int) (*os.File, error) {
     return f, nil
 }
 
+// Open file/directory with given <flag> and <perm>.
+//
 // 打开文件(带flag&perm)
 func OpenWithFlagPerm(path string, flag int, perm int) (*os.File, error) {
     f, err  := os.OpenFile(path, flag, os.FileMode(perm))
@@ -93,6 +101,8 @@ func OpenWithFlagPerm(path string, flag int, perm int) (*os.File, error) {
     return f, nil
 }
 
+// Check whether given path exist.
+//
 // 判断所给路径文件/文件夹是否存在
 func Exists(path string) bool {
     if _, err := os.Stat(path); !os.IsNotExist(err) {
@@ -101,6 +111,8 @@ func Exists(path string) bool {
     return false
 }
 
+// Check whether given path a directory.
+//
 // 判断所给路径是否为文件夹
 func IsDir(path string) bool {
     s, err := os.Stat(path)
@@ -110,11 +122,15 @@ func IsDir(path string) bool {
     return s.IsDir()
 }
 
+// Get current working absolute directory path.
+//
 // 获取当前工作目录(SelfDir()方法的别名)
 func Pwd() string {
     return SelfDir()
 }
 
+// Check whether given path a file(not a directory).
+//
 // 判断所给路径是否为文件
 func IsFile(path string) bool {
     s, err := os.Stat(path)
@@ -124,36 +140,43 @@ func IsFile(path string) bool {
     return !s.IsDir()
 }
 
-// 获取文件或目录信息
-func Info(path string) *os.FileInfo {
-    info, err := os.Stat(path)
-    if err != nil {
-        return nil
-    }
-    return &info
+// Info returns a FileInfo describing the named file.
+// If there is an error, it will be of type *PathError.
+//
+// 获取文件或目录信息.
+func Info(path string) (os.FileInfo, error) {
+    return os.Stat(path)
 }
 
+// Move renames (moves) src to dst path.
+//
 // 文件移动/重命名
 func Move(src string, dst string) error {
     return os.Rename(src, dst)
 }
 
-
-// 文件移动/重命名
+// Rename renames (moves) src to dst path.
+//
+// 文件移动/重命名.
 func Rename(src string, dst string) error {
     return Move(src, dst)
 }
 
-// 文件复制
+// Copy file from src to dst.
+//
+// 文件复制.
+// @TODO 支持目录复制.
 func Copy(src string, dst string) error {
-    srcFile, err := os.Open(src)
+    srcFile, err := Open(src)
     if err != nil {
         return err
     }
-    dstFile, err := os.Create(dst)
+    defer srcFile.Close()
+    dstFile, err := Create(dst)
     if err != nil {
         return err
     }
+    defer dstFile.Close()
     _, err = io.Copy(dstFile, srcFile)
     if err != nil {
         return err
@@ -162,11 +185,11 @@ func Copy(src string, dst string) error {
     if err != nil {
         return err
     }
-    srcFile.Close()
-    dstFile.Close()
     return nil
 }
 
+// Get sub-file names of path.
+//
 // 返回目录下的文件名列表
 func DirNames(path string) ([]string, error) {
     f, err := os.Open(path)
@@ -181,6 +204,15 @@ func DirNames(path string) ([]string, error) {
     return list, nil
 }
 
+// Glob returns the names of all files matching pattern or nil
+// if there is no matching file. The syntax of patterns is the same
+// as in Match. The pattern may describe hierarchical names such as
+// /usr/*/bin/ed (assuming the Separator is '/').
+//
+// Glob ignores file system errors such as I/O errors reading directories.
+// The only possible returned error is ErrBadPattern, when pattern
+// is malformed.
+//
 // 文件名正则匹配查找，第二个可选参数指定返回的列表是否仅为文件名(非绝对路径)，默认返回绝对路径
 func Glob(pattern string, onlyNames...bool) ([]string, error) {
     if list, err := filepath.Glob(pattern); err == nil {
@@ -197,11 +229,15 @@ func Glob(pattern string, onlyNames...bool) ([]string, error) {
     }
 }
 
+// Remove file/directory with <path> parameter.
+//
 // 文件/目录删除
 func Remove(path string) error {
     return os.RemoveAll(path)
 }
 
+// Check whether given <path> is readable.
+//
 // 文件是否可读(支持文件/目录)
 func IsReadable(path string) bool {
     result    := true
@@ -213,16 +249,19 @@ func IsReadable(path string) bool {
     return result
 }
 
+// Check whether given <path> is writable.
+//
 // 文件是否可写(支持文件/目录)
+// @TODO 改进性能，利用 golang.org/x/sys 来实现跨平台的权限判断。
 func IsWritable(path string) bool {
     result := true
     if IsDir(path) {
         // 如果是目录，那么创建一个临时文件进行写入测试
         tmpFile := strings.TrimRight(path, Separator) + Separator + gconv.String(time.Now().UnixNano())
-        err     := Create(tmpFile)
-        if err != nil || !Exists(tmpFile){
+        if f, err := Create(tmpFile); err != nil || !Exists(tmpFile){
             result = false
         } else {
+            f.Close()
             Remove(tmpFile)
         }
     } else {
@@ -236,11 +275,16 @@ func IsWritable(path string) bool {
     return result
 }
 
+// See os.Chmod.
+//
 // 修改文件/目录权限
 func Chmod(path string, mode os.FileMode) error {
     return os.Chmod(path, mode)
 }
 
+// Get all sub-files(absolute) of given <path>,
+// can be recursively with given parameter <recursive> true.
+//
 // 打开目录，并返回其下一级文件列表(绝对路径)，按照文件名称大小写进行排序，支持目录递归遍历。
 func ScanDir(path string, pattern string, recursive ... bool) ([]string, error) {
     list, err := doScanDir(path, pattern, recursive...)
@@ -287,6 +331,8 @@ func doScanDir(path string, pattern string, recursive ... bool) ([]string, error
     return list, nil
 }
 
+// See filepath.Abs.
+//
 // 将所给定的路径转换为绝对路径
 // 并判断文件路径是否存在，如果文件不存在，那么返回空字符串
 func RealPath(path string) string {
@@ -300,33 +346,44 @@ func RealPath(path string) string {
     return p
 }
 
-
+// Get absolute file path of current running process(binary).
+//
 // 获取当前执行文件的绝对路径
 func SelfPath() string {
     p, _ := filepath.Abs(os.Args[0])
     return p
 }
 
+// Get absolute directory path of current running process(binary).
+//
 // 获取当前执行文件的目录绝对路径
 func SelfDir() string {
     return filepath.Dir(SelfPath())
 }
 
+// See filepath.Base.
+//
 // 获取指定文件路径的文件名称
 func Basename(path string) string {
     return filepath.Base(path)
 }
 
-// 获取指定文件路径的目录地址绝对路径
+// See filepath.Dir.
+//
+// 获取指定文件路径的目录地址绝对路径.
 func Dir(path string) string {
     return filepath.Dir(path)
 }
 
+// See filepath.Ext.
+//
 // 获取指定文件路径的文件扩展名(包含"."号)
 func Ext(path string) string {
     return filepath.Ext(path)
 }
 
+// Get absolute home directory path of current user.
+//
 // 获取用户主目录
 func Home() (string, error) {
     u, err := user.Current()
@@ -372,6 +429,9 @@ func homeWindows() (string, error) {
     return home, nil
 }
 
+// Get absolute file path of main file, which contains the entrance function main.
+// Available in develop environment.
+//
 // 获取入口函数文件所在目录(main包文件目录),
 // **仅对源码开发环境有效(即仅对生成该可执行文件的系统下有效)**
 func MainPkgPath() string {
@@ -418,6 +478,8 @@ func MainPkgPath() string {
     return ""
 }
 
+// See os.TempDir().
+//
 // 系统临时目录
 func TempDir() string {
     return os.TempDir()

g/os/gfpool/gfpool.go

@@ -130,7 +130,9 @@ func (p *Pool) File() (*File, error) {
                return nil, err
            }
         }
-        if !p.inited.Val() || p.inited.Set(true) == false {
+        // !p.inited.Val() 使用原子读取操作判断，保证该操作判断的效率；
+        // p.inited.Set(true) == false 使用原子写入操作，保证该操作的原子性；
+        if !p.inited.Val() && p.inited.Set(true) == false {
             gfsnotify.Add(f.path, func(event *gfsnotify.Event) {
                 // 如果文件被删除或者重命名，立即重建指针池
                 if event.IsRemove() || event.IsRename() {

g/os/glog/glog.go

@@ -7,8 +7,7 @@
 
 // Package glog implements powerful and easy-to-use levelled logging functionality.
 // 
-// 日志模块,
-// 直接文件/输出操作，没有异步逻辑，没有使用缓存或者通道
+// 日志模块, 直接文件/输出操作，没有异步逻辑，没有使用缓存或者通道
 package glog
 
 import (
@@ -28,10 +27,10 @@ const (
 )
 
 var (
-    // glog默认的日志等级，影响全局
+    // default level for log
     defaultLevel = gtype.NewInt(LEVEL_ALL)
 
-    // 默认的日志对象
+    // default logger object, for package method usage
     logger = New()
 )
 
@@ -39,101 +38,157 @@ func init() {
     SetDebug(cmdenv.Get("gf.glog.debug", true).Bool())
 }
 
-// 日志日志目录绝对路径
+// SetPath sets the directory path for file logging.
+//
+// 日志日志目录绝对路径.
 func SetPath(path string) {
     logger.SetPath(path)
 }
 
-// 日志文件名称
-func SetFile(file string) {
-    logger.SetFile(file)
+// SetFile sets the file name <pattern> for file logging.
+// Datetime pattern can be used in <pattern>, eg: access-{Ymd}.log.
+// The default file name pattern is: Y-m-d.log, eg: 2018-01-01.log
+//
+// 日志文件名称.
+func SetFile(pattern string) {
+    logger.SetFile(pattern)
 }
 
-// 设置全局的日志记录等级
+// SetLevel sets the default logging level.
+//
+// 设置全局的日志记录等级.
 func SetLevel(level int) {
     logger.SetLevel(level)
     defaultLevel.Set(level)
 }
 
-// 可自定义IO接口，IO可以是文件输出、标准输出、网络输出
+// SetWriter sets the customed logging <writer> for logging.
+// The <writer> object should implements the io.Writer interface.
+// Developer can use customed logging <writer> to redirect logging output to another service, 
+// eg: kafka, mysql, mongodb, etc.
+//
+// 可自定义IO接口，IO可以是文件输出、标准输出、网络输出.
 func SetWriter(writer io.Writer) {
     logger.SetWriter(writer)
 }
 
-// 返回自定义的IO，默认为nil
+// GetWriter returns the customed writer object, which implements the io.Writer interface.
+// It returns nil if no customed writer set.
+// 
+// 返回自定义的IO，默认为nil.
 func GetWriter() io.Writer {
     return logger.GetWriter()
 }
 
-// 获取全局的日志记录等级
+// GetLevel returns the default logging level value.
+// 
+// 获取全局的日志记录等级.
 func GetLevel() int {
     return defaultLevel.Val()
 }
 
-// 设置是否允许输出DEBUG信息
+// SetDebug enables/disables the debug level for default logger.
+// The debug level is enbaled in default.
+// 
+// 设置是否允许输出DEBUG信息.
 func SetDebug(debug bool) {
     logger.SetDebug(debug)
 }
 
+// SetStdPrint sets whether ouptput the logging contents to stdout, which is false indefault.
+// 
 // 设置写日志的同时开启or关闭控制台打印，默认是关闭的
 func SetStdPrint(open bool) {
     logger.SetStdPrint(open)
 }
 
+// GetPath returns the logging directory path for file logging.
+// It returns empty string if no directory path set.
+//
 // 获取日志目录绝对路径
 func GetPath() string {
     return logger.GetPath()
 }
 
+// PrintBacktrace prints the caller backtrace, 
+// the optional parameter <skip> specify the skipped backtraces offset from the end point.
+//
 // 打印文件调用回溯信息
 func PrintBacktrace(skip...int) {
     logger.PrintBacktrace(skip...)
 }
 
-// 获取文件调用回溯信息
+// GetBacktrace returns the caller backtrace content, 
+// the optional parameter <skip> specify the skipped backtraces offset from the end point.
+//
+// 获取文件调用回溯信息.
 func GetBacktrace(skip...int) string {
     return logger.GetBacktrace(skip...)
 }
 
+// SetBacktrace enables/disables the backtrace feature in failure logging outputs.
+//
 // 是否关闭全局的backtrace信息
 func SetBacktrace(enabled bool) {
     logger.SetBacktrace(enabled)
 }
 
+// To is a chaining function, 
+// which redirects current logging content output to the sepecified <writer>.
+// 
 // 链式操作，设置下一次写入日志内容的Writer
 func To(writer io.Writer) *Logger {
     return logger.To(writer)
 }
 
-// 设置下一次输出的分类，支持多级分类设置
+// Cat is a chaining function, 
+// which sets the category to <category> for current logging content output.
+// 
+// 设置下一次输出的分类，支持多级分类设置.
 func Cat(category string) *Logger {
     return logger.Cat(category)
 }
 
+// File is a chaining function, 
+// which sets file name <pattern> for the current logging content output.
+//
 // 设置日志输出文件名称格式
-func File(file string) *Logger {
-    return logger.File(file)
+func File(pattern string) *Logger {
+    return logger.File(pattern)
 }
 
-// 设置日志打印等级
+// Level is a chaining function, 
+// which sets logging level for the current logging content output.
+//
+// 设置日志打印等级.
 func Level(level int) *Logger {
     return logger.Level(level)
 }
 
-// 设置文件调用回溯信息
+// Backtrace is a chaining function, 
+// which sets backtrace options for the current logging content output .
+//
+// 设置文件调用回溯信息.
 func Backtrace(enabled bool, skip...int) *Logger {
     return logger.Backtrace(enabled, skip...)
 }
 
+// StdPrint is a chaining function, 
+// which enables/disables stdout for the current logging content output.
+//
 // 是否允许在设置输出文件时同时也输出到终端
 func StdPrint(enabled bool) *Logger {
     return logger.StdPrint(enabled)
 }
 
+// Header is a chaining function, 
+// which enables/disables log header for the current logging content output.
+//
 // 是否打印每行日志头信息(默认开启)
 func Header(enabled bool) *Logger {
     return logger.Header(enabled)
 }
+
 func Print(v ...interface{}) {
     logger.Print(v ...)
 }
@@ -150,14 +205,17 @@ func Printfln(format string, v ...interface{}) {
     logger.Printfln(format, v ...)
 }
 
+// Fatal prints the logging content with [FATA] header and newline, then exit the current process.
 func Fatal(v ...interface{}) {
     logger.Fatal(v ...)
 }
 
+// Fatalf prints the logging content with [FATA] header and custom format, then exit the current process.
 func Fatalf(format string, v ...interface{}) {
     logger.Fatalf(format, v ...)
 }
 
+// Fatalf prints the logging content with [FATA] header, custom format and newline, then exit the current process.
 func Fatalfln(format string, v ...interface{}) {
     logger.Fatalfln(format, v ...)
 }

g/os/glog/glog_logger.go

@@ -58,6 +58,8 @@ func init() {
     }
 }
 
+// New creates a custom logger.
+//
 // 新建自定义的日志操作对象
 func New() *Logger {
     return &Logger {
@@ -72,7 +74,9 @@ func New() *Logger {
     }
 }
 
-// Logger深拷贝
+// Clone returns a new logger, which is the clone the current logger.
+//
+// Logger拷贝.
 func (l *Logger) Clone() *Logger {
     return &Logger {
         pr           : l,
@@ -87,16 +91,23 @@ func (l *Logger) Clone() *Logger {
     }
 }
 
+// SetLevel sets the logging level.
+//
 // 设置日志记录等级
 func (l *Logger) SetLevel(level int) {
     l.level.Set(level)
 }
 
+// GetLevel returns the logging level value.
+//
 // 获取日志记录等级
 func (l *Logger) GetLevel() int {
     return l.level.Val()
 }
 
+// SetDebug enables/disables the debug level for logger.
+// The debug level is enbaled in default.
+// 
 // 快捷方法，打开或关闭DEBU日志信息
 func (l *Logger) SetDebug(debug bool) {
     if debug {
@@ -106,6 +117,7 @@ func (l *Logger) SetDebug(debug bool) {
     }
 }
 
+// SetBacktrace enables/disables the backtrace feature in failure logging outputs.
 func (l *Logger) SetBacktrace(enabled bool) {
     if enabled {
         l.btStatus.Set(1)
@@ -115,11 +127,16 @@ func (l *Logger) SetBacktrace(enabled bool) {
 
 }
 
-// 设置BacktraceSkip
+// SetBacktraceSkip sets the backtrace offset from the end point.
 func (l *Logger) SetBacktraceSkip(skip int) {
     l.btSkip.Set(skip)
 }
 
+// SetWriter sets the customed logging <writer> for logging.
+// The <writer> object should implements the io.Writer interface.
+// Developer can use customed logging <writer> to redirect logging output to another service, 
+// eg: kafka, mysql, mongodb, etc.
+//
 // 可自定义IO接口，IO可以是文件输出、标准输出、网络输出
 func (l *Logger) SetWriter(writer io.Writer) {
     l.mu.Lock()
@@ -127,6 +144,9 @@ func (l *Logger) SetWriter(writer io.Writer) {
     l.mu.Unlock()
 }
 
+// GetWriter returns the customed writer object, which implements the io.Writer interface.
+// It returns nil if no customed writer set.
+//
 // 返回自定义的IO，默认为nil
 func (l *Logger) GetWriter() io.Writer {
     l.mu.RLock()
@@ -135,7 +155,10 @@ func (l *Logger) GetWriter() io.Writer {
     return r
 }
 
-// 获取默认的文件IO
+// getFilePointer returns the file pinter for file logging.
+// It returns nil if file logging disabled, or file open fails.
+//
+// 获取默认的文件IO.
 func (l *Logger) getFilePointer() *gfpool.File {
     if path := l.path.Val(); path != "" {
         // 文件名称中使用"{}"包含的内容使用gtime格式化
@@ -159,7 +182,9 @@ func (l *Logger) getFilePointer() *gfpool.File {
     return nil
 }
 
-// 设置日志文件的存储目录路径
+// SetPath sets the directory path for file logging.
+//
+// 设置日志文件的存储目录路径.
 func (l *Logger) SetPath(path string) error {
     // path必须有值
     if path == "" {
@@ -176,16 +201,25 @@ func (l *Logger) SetPath(path string) error {
     return nil
 }
 
+// GetPath returns the logging directory path for file logging.
+// It returns empty string if no directory path set.
+//
 // 获取设置的日志目录路径
 func (l *Logger) GetPath() string {
     return l.path.Val()
 }
 
-// 日志文件名称
-func (l *Logger) SetFile(file string) {
-    l.file.Set(file)
+// SetFile sets the file name <pattern> for file logging.
+// Datetime pattern can be used in <pattern>, eg: access-{Ymd}.log.
+// The default file name pattern is: Y-m-d.log, eg: 2018-01-01.log
+//
+// 设置日志文件名称格式.
+func (l *Logger) SetFile(pattern string) {
+    l.file.Set(pattern)
 }
 
+// SetStdPrint sets whether ouptput the logging contents to stdout, which is false indefault.
+// 
 // 设置写日志时开启or关闭控制台打印，默认是关闭的
 func (l *Logger) SetStdPrint(enabled bool) {
     l.alsoStdPrint.Set(enabled)
@@ -263,11 +297,17 @@ func (l *Logger) appendBacktrace(s string, skip...int) string {
     return s
 }
 
+// PrintBacktrace prints the caller backtrace, 
+// the optional parameter <skip> specify the skipped backtraces offset from the end point.
+//
 // 直接打印回溯信息，参数skip表示调用端往上多少级开始回溯
 func (l *Logger) PrintBacktrace(skip...int) {
     l.Println(l.appendBacktrace("", skip...))
 }
 
+// GetBacktrace returns the caller backtrace content, 
+// the optional parameter <skip> specify the skipped backtraces offset from the end point.
+//
 // 获取文件调用回溯字符串，参数skip表示调用端往上多少级开始回溯
 func (l *Logger) GetBacktrace(skip...int) string {
     customSkip := 0
@@ -322,16 +362,19 @@ func (l *Logger) Printfln(format string, v ...interface{}) {
     l.stdPrint(fmt.Sprintf(format + ln, v...))
 }
 
+// Fatal prints the logging content with [FATA] header and newline, then exit the current process.
 func (l *Logger) Fatal(v ...interface{}) {
     l.errPrint("[FATA] " + fmt.Sprintln(v...))
     os.Exit(1)
 }
 
+// Fatalf prints the logging content with [FATA] header and custom format, then exit the current process.
 func (l *Logger) Fatalf(format string, v ...interface{}) {
     l.errPrint("[FATA] " + fmt.Sprintf(format, v...))
     os.Exit(1)
 }
 
+// Fatalf prints the logging content with [FATA] header, custom format and newline, then exit the current process.
 func (l *Logger) Fatalfln(format string, v ...interface{}) {
     l.errPrint("[FATA] " + fmt.Sprintf(format + ln, v...))
     os.Exit(1)
@@ -463,6 +506,8 @@ func (l *Logger) Criticalfln(format string, v ...interface{}) {
     }
 }
 
+// checkLevel checks whether the given <level> could be output.
+//
 // 判断给定level是否满足
 func (l *Logger) checkLevel(level int) bool {
     return l.level.Val() & level > 0

g/os/glog/glog_logger_chaining.go

@@ -11,6 +11,9 @@ import (
     "io"
 )
 
+// To is a chaining function, 
+// which redirects current logging content output to the sepecified <writer>.
+// 
 // 链式操作，设置下一次写入日志内容的Writer
 func (l *Logger) To(writer io.Writer) *Logger {
     logger := (*Logger)(nil)
@@ -23,6 +26,9 @@ func (l *Logger) To(writer io.Writer) *Logger {
     return logger
 }
 
+// Cat is a chaining function, 
+// which sets the category to <category> for current logging content output.
+// 
 // 链式操作，设置下一次输出的日志分类(可以按照文件目录层级设置)，在当前logpath或者当前工作目录下创建category目录，
 // 这是一个链式操作，可以设置多个分类，将会创建层级的日志分类目录。
 func (l *Logger) Cat(category string) *Logger {
@@ -39,6 +45,9 @@ func (l *Logger) Cat(category string) *Logger {
     return logger
 }
 
+// File is a chaining function, 
+// which sets file name <pattern> for the current logging content output.
+//
 // 日志文件格式
 func (l *Logger) File(file string) *Logger {
     logger := (*Logger)(nil)
@@ -51,6 +60,9 @@ func (l *Logger) File(file string) *Logger {
     return logger
 }
 
+// Level is a chaining function, 
+// which sets logging level for the current logging content output.
+//
 // 设置日志打印等级
 func (l *Logger) Level(level int) *Logger {
     logger := (*Logger)(nil)
@@ -63,6 +75,9 @@ func (l *Logger) Level(level int) *Logger {
     return logger
 }
 
+// Backtrace is a chaining function, 
+// which sets backtrace options for the current logging content output .
+//
 // 设置文件调用回溯信息
 func (l *Logger) Backtrace(enabled bool, skip...int) *Logger {
     logger := (*Logger)(nil)
@@ -78,6 +93,9 @@ func (l *Logger) Backtrace(enabled bool, skip...int) *Logger {
     return logger
 }
 
+// StdPrint is a chaining function, 
+// which enables/disables stdout for the current logging content output.
+//
 // 是否允许在设置输出文件时同时也输出到终端
 func (l *Logger) StdPrint(enabled bool) *Logger {
     logger := (*Logger)(nil)
@@ -90,6 +108,9 @@ func (l *Logger) StdPrint(enabled bool) *Logger {
     return logger
 }
 
+// Header is a chaining function, 
+// which enables/disables log header for the current logging content output.
+//
 // 是否打印每行日志头信息(默认开启)
 func (l *Logger) Header(enabled bool) *Logger {
     logger := (*Logger)(nil)

g/os/gproc/gproc.go

@@ -27,8 +27,10 @@ const (
     gPROC_TEMP_DIR_ENV_KEY = "GPROC_TEMP_DIR"
 )
 
-// 进程开始执行时间
-var processStartTime = time.Now()
+var (
+    // 进程开始执行时间
+    processStartTime = time.Now()
+)
 
 // 获取当前进程ID
 func Pid() int {

g/os/gproc/gproc_comm_receive.go

@@ -3,6 +3,7 @@
 // This Source Code Form is subject to the terms of the MIT License.
 // If a copy of the MIT was not distributed with this file,
 // You can obtain one at https://github.com/gogf/gf.
+
 // "不要通过共享内存来通信，而应该通过通信来共享内存"
 
 
@@ -26,16 +27,43 @@ const (
 )
 
 var (
-    // 是否已开启TCP端口监听服务(使用int而非bool，以便于使用原子操作判断是否开启)
-    tcpListeningCount = gtype.NewInt()
+    // 是否已开启TCP端口监听服务
+    tcpListened = gtype.NewBool()
 )
 
+// 获取其他进程传递到当前进程的消息包，阻塞执行。
+// 进程只有在执行该方法后才会打开请求端口，默认情况下不允许进程间通信。
+func Receive(group...string) *Msg {
+    // 一个进程只能开启一个监听goroutine
+    if tcpListened.Set(true) == false {
+        go startTcpListening()
+    }
+    queue     := (*gqueue.Queue)(nil)
+    groupName := gPROC_COMM_DEAFULT_GRUOP_NAME
+    if len(group) > 0 {
+        groupName = group[0]
+    }
+    if v := commReceiveQueues.Get(groupName); v == nil {
+        commReceiveQueues.LockFunc(func(m map[string]interface{}) {
+            if v, ok := m[groupName]; ok {
+                queue        = v.(*gqueue.Queue)
+            } else {
+                queue        = gqueue.New(gPROC_MSG_QUEUE_MAX_LENGTH)
+                m[groupName] = queue
+            }
+        })
+    } else {
+        queue = v.(*gqueue.Queue)
+    }
+
+    if v := queue.Pop(); v != nil {
+        return v.(*Msg)
+    }
+    return nil
+}
+
 // 创建本地进程TCP通信服务
 func startTcpListening() {
-    // 一个进程只能开启一个监听goroutine
-    if tcpListeningCount.Add(1) != 1 {
-        return
-    }
     var listen *net.TCPListener
     for i := gPROC_DEFAULT_TCP_PORT; ; i++ {
         addr, err := net.ResolveTCPAddr("tcp", fmt.Sprintf("127.0.0.1:%d", i))
@@ -48,7 +76,6 @@ func startTcpListening() {
         }
         // 将监听的端口保存到通信文件中(字符串类型存放)
         gfile.PutContents(getCommFilePath(Pid()), gconv.String(i))
-        //glog.Printfln("%d: gproc listening on [%s]", Pid(), addr)
         break
     }
     for  {
@@ -133,32 +160,3 @@ func bufferToMsgs(buffer []byte) []*Msg {
     return msgs
 }
 
-// 获取其他进程传递到当前进程的消息包，阻塞执行。
-func Receive(group...string) *Msg {
-    // 开启接收协程时才会开启端口监听
-    go startTcpListening()
-
-    var queue *gqueue.Queue
-    groupName := gPROC_COMM_DEAFULT_GRUOP_NAME
-    if len(group) > 0 {
-        groupName = group[0]
-    }
-
-    if v := commReceiveQueues.Get(groupName); v == nil {
-        commReceiveQueues.LockFunc(func(m map[string]interface{}) {
-            if v, ok := m[groupName]; ok {
-                queue        = v.(*gqueue.Queue)
-            } else {
-                queue        = gqueue.New(gPROC_MSG_QUEUE_MAX_LENGTH)
-                m[groupName] = queue
-            }
-        })
-    } else {
-        queue = v.(*gqueue.Queue)
-    }
-
-    if v := queue.Pop(); v != nil {
-        return v.(*Msg)
-    }
-    return nil
-}

g/os/gproc/gproc_comm_send.go

@@ -7,16 +7,16 @@
 package gproc
 
 import (
-    "github.com/gogf/gf/g/net/gtcp"
-    "github.com/gogf/gf/g/os/gfile"
-    "github.com/gogf/gf/g/util/gconv"
-    "github.com/gogf/gf/g/encoding/gbinary"
-    "fmt"
-    "errors"
-    "time"
     "bytes"
+    "errors"
+    "fmt"
+    "github.com/gogf/gf/g/encoding/gbinary"
+    "github.com/gogf/gf/g/net/gtcp"
+    "github.com/gogf/gf/g/os/gfcache"
     "github.com/gogf/gf/g/os/glog"
+    "github.com/gogf/gf/g/util/gconv"
     "io"
+    "time"
 )
 
 const (
@@ -26,7 +26,7 @@ const (
     gPROC_COMM_DEAFULT_GRUOP_NAME    = ""   // 默认分组名称
 )
 
-// 向指定gproc进程发送数据
+// 向指定gproc进程发送数据.
 // 数据格式：总长度(24bit)|发送进程PID(24bit)|接收进程PID(24bit)|分组长度(8bit)|分组名称(变长)|校验(32bit)|参数(变长)
 func Send(pid int, data []byte, group...string) error {
     groupName := gPROC_COMM_DEAFULT_GRUOP_NAME
@@ -85,6 +85,6 @@ func getConnByPid(pid int) (*gtcp.Conn, error) {
 // 获取指定进程监听的端口号
 func getPortByPid(pid int) int {
     path    := getCommFilePath(pid)
-    content := gfile.GetContents(path)
+    content := gfcache.GetContents(path)
     return gconv.Int(content)
 }

g/os/gview/gview.go

@@ -115,6 +115,10 @@ func (view *View) SetPath(path string) error {
         glog.Error(fmt.Sprintf(`[gview] SetPath failed: %s`, err.Error()))
         return err
     }
+    // 重复判断
+    if view.paths.Search(realPath) != -1 {
+        return nil
+    }
     view.paths.Clear()
     view.paths.Append(realPath)
     glog.Debug("[gview] SetPath:", realPath)
@@ -132,6 +136,10 @@ func (view *View) AddPath(path string) error {
         glog.Error(fmt.Sprintf(`[gview] AddPath failed: %s`, err.Error()))
         return err
     }
+    // 重复判断
+    if view.paths.Search(realPath) != -1 {
+        return nil
+    }
     view.paths.Append(realPath)
     glog.Debug("[gview] AddPath:", realPath)
     return nil

g/test/gtest/gtest.go

@@ -215,12 +215,12 @@ func AssertNI(value, expect interface{}) {
     }
 }
 
-// 提示错误不退出
+// 提示错误不退出进程执行
 func Error(message...interface{}) {
     fmt.Fprintf(os.Stderr, "[ERROR] %s\n%s", fmt.Sprint(message...), getBacktrace())
 }
 
-// 提示错误并退出
+// 提示错误并退出进程执行
 func Fatal(message...interface{}) {
     fmt.Fprintf(os.Stderr, "[FATAL] %s\n%s", fmt.Sprint(message...), getBacktrace())
     os.Exit(1)
@@ -270,7 +270,7 @@ func getBacktrace(skip...int) string {
     // 首先定位业务文件开始位置
     for i := 0; i < 10; i++ {
         if _, file, _, ok := runtime.Caller(i); ok {
-            if reg, _  := regexp.Compile("/g/util/gtest/.+$"); !reg.MatchString(file) {
+            if reg, _  := regexp.Compile(`gtest\.go$`); !reg.MatchString(file) {
                 from = i
                 break
             }
@@ -283,7 +283,7 @@ func getBacktrace(skip...int) string {
             if reg, _  := regexp.Compile(`<autogenerated>`); reg.MatchString(file) {
                 continue
             }
-            if reg, _  := regexp.Compile("/g/test/gtest/.+$"); reg.MatchString(file) {
+            if reg, _  := regexp.Compile(`gtest\.go$`); reg.MatchString(file) {
                 continue
             }
             if goRoot != "" {

g/text/gregex/gregex.go

@@ -13,17 +13,25 @@ import (
     "regexp"
 )
 
+// Quote quotes <s> by replacing special chars in <s>
+// to match the rules of regular expression pattern.
+// And returns the copy.
+//
 // 转移正则规则字符串，例如：Quote(`[foo]`) 返回 `\[foo\]`
 func Quote(s string) string {
     return regexp.QuoteMeta(s)
 }
 
+// Validate checks whether given regular expression pattern <pattern> valid.
+//
 // 校验所给定的正则表达式是否符合规范
 func Validate(pattern string) error {
     _, err := getRegexp(pattern)
     return err
 }
 
+// IsMatch checks whether given bytes <src> matches <pattern>.
+//
 // 正则表达式是否匹配
 func IsMatch(pattern string, src []byte) bool {
     if r, err := getRegexp(pattern); err == nil {
@@ -32,10 +40,15 @@ func IsMatch(pattern string, src []byte) bool {
     return false
 }
 
+// IsMatchString checks whether given string <src> matches <pattern>.
+//
+// 判断给定的字符串<src>是否满足正则表达式<pattern>.
 func IsMatchString(pattern string, src string) bool {
     return IsMatch(pattern, []byte(src))
 }
 
+// MatchString return bytes slice that matched <pattern>.
+//
 // 正则匹配，并返回匹配的列表(参数[]byte)
 func Match(pattern string, src []byte) ([][]byte, error) {
     if r, err := getRegexp(pattern); err == nil {
@@ -45,6 +58,8 @@ func Match(pattern string, src []byte) ([][]byte, error) {
     }
 }
 
+// MatchString return strings that matched <pattern>.
+//
 // 正则匹配，并返回匹配的列表(参数[]string)
 func MatchString(pattern string, src string) ([]string, error) {
     if r, err := getRegexp(pattern); err == nil {
@@ -54,6 +69,8 @@ func MatchString(pattern string, src string) ([]string, error) {
     }
 }
 
+// MatchAll return all bytes slices that matched <pattern>.
+//
 // 正则匹配，并返回所有匹配的列表(参数[]string)
 func MatchAll(pattern string, src []byte) ([][][]byte, error) {
     if r, err := getRegexp(pattern); err == nil {
@@ -63,7 +80,9 @@ func MatchAll(pattern string, src []byte) ([][][]byte, error) {
     }
 }
 
-// 正则匹配，并返回所有匹配的列表(参数[][]string)
+// MatchAllString return all strings that matched <pattern>.
+//
+// 正则匹配，并返回所有匹配的列表(参数[][]string).
 func MatchAllString(pattern string, src string) ([][]string, error) {
     if r, err := getRegexp(pattern); err == nil {
         return r.FindAllStringSubmatch(src, -1), nil
@@ -72,7 +91,9 @@ func MatchAllString(pattern string, src string) ([][]string, error) {
     }
 }
 
-// 正则替换(全部替换)
+// ReplaceString replace all matched <pattern> in bytes <src> with bytes <replace>.
+//
+// 正则替换(全部替换).
 func Replace(pattern string, replace, src []byte) ([]byte, error) {
     if r, err := getRegexp(pattern); err == nil {
         return r.ReplaceAll(src, replace), nil
@@ -81,12 +102,17 @@ func Replace(pattern string, replace, src []byte) ([]byte, error) {
     }
 }
 
+// ReplaceString replace all matched <pattern> in string <src> with string <replace>.
+//
 // 正则替换(全部替换)，字符串
 func ReplaceString(pattern, replace, src string) (string, error) {
     r, e := Replace(pattern, []byte(replace), []byte(src))
     return string(r), e
 }
 
+// ReplaceFunc replace all matched <pattern> in bytes <src>
+// with custom replacement function <replaceFunc>.
+//
 // 正则替换(全部替换)，给定自定义替换方法
 func ReplaceFunc(pattern string, src []byte, replaceFunc func(b []byte) []byte) ([]byte, error) {
     if r, err := getRegexp(pattern); err == nil {
@@ -96,6 +122,9 @@ func ReplaceFunc(pattern string, src []byte, replaceFunc func(b []byte) []byte)
     }
 }
 
+// ReplaceStringFunc replace all matched <pattern> in string <src>
+// with custom replacement function <replaceFunc>.
+//
 // 正则替换(全部替换)，给定自定义替换方法
 func ReplaceStringFunc(pattern string, src string, replaceFunc func(s string) string) (string, error) {
     bytes, err := ReplaceFunc(pattern, []byte(src), func(bytes []byte) []byte {

g/text/gregex/gregex_z_bench_test.go

@@ -6,9 +6,10 @@
 
 // go test *.go -bench=".*"
 
-package gregex
+package gregex_test
 
 import (
+    "github.com/gogf/gf/g/text/gregex"
     "testing"
 )
 
@@ -18,42 +19,42 @@ var replace = "johng.cn"
 
 func BenchmarkValidate(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        Validate(pattern)
+        gregex.Validate(pattern)
     }
 }
 
 func BenchmarkIsMatch(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        IsMatch(pattern, []byte(src))
+        gregex.IsMatch(pattern, []byte(src))
     }
 }
 
 func BenchmarkIsMatchString(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        IsMatchString(pattern, src)
+        gregex.IsMatchString(pattern, src)
     }
 }
 
 func BenchmarkMatchString(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        MatchString(pattern, src)
+        gregex.MatchString(pattern, src)
     }
 }
 
 func BenchmarkMatchAllString(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        MatchAllString(pattern, src)
+        gregex.MatchAllString(pattern, src)
     }
 }
 
 func BenchmarkReplace(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        Replace(pattern, []byte(replace), []byte(src))
+        gregex.Replace(pattern, []byte(replace), []byte(src))
     }
 }
 
 func BenchmarkReplaceString(b *testing.B) {
     for i := 0; i < b.N; i++ {
-        ReplaceString(pattern, replace, src)
+        gregex.ReplaceString(pattern, replace, src)
     }
 }

g/text/gstr/gstr.go

@@ -88,25 +88,6 @@ func UcWords(str string) string {
     return strings.Title(str)
 }
 
-// Traverse the array to find the string index position, if not exist, return-1.
-//
-// 遍历数组查找字符串索引位置，如果不存在则返回-1，使用完整遍历查找.
-func SearchArray (a []string, s string) int {
-    for i, v := range a {
-        if s == v {
-            return i
-        }
-    }
-    return -1
-}
-
-// InArray tests whether the given string s is in string array a.
-//
-// 判断字符串是否在数组中
-func InArray (a []string, s string) bool {
-    return SearchArray(a, s) != -1
-}
-
 // IsLetterLower tests whether the given byte b is in lower case.
 //
 // 判断给定字符是否小写
@@ -490,15 +471,7 @@ func Explode(delimiter, str string) []string {
 //
 // 用glue将字符串数组pieces连接为一个字符串。
 func Implode(glue string, pieces []string) string {
-    var buf bytes.Buffer
-    l := len(pieces)
-    for _, str := range pieces {
-        buf.WriteString(str)
-        if l--; l > 0 {
-            buf.WriteString(glue)
-        }
-    }
-    return buf.String()
+    return strings.Join(pieces, glue)
 }
 
 // Generate a single-byte string from a number.

g/text/gstr/gstr_pos.go

@@ -9,6 +9,7 @@ package gstr
 import "strings"
 
 // Find the position of the first occurrence of a substring in a string.
+// It returns -1, if none found.
 //
 // 返回 needle 在 haystack 中首次出现的数字位置，找不到返回-1。
 func Pos(haystack, needle string, startOffset...int) int {
@@ -32,6 +33,7 @@ func Pos(haystack, needle string, startOffset...int) int {
 }
 
 // Find the position of the first occurrence of a case-insensitive substring in a string.
+// It returns -1, if none found.
 //
 // 返回在字符串 haystack 中 needle 首次出现的数字位置（不区分大小写），找不到返回-1。
 func PosI(haystack, needle string, startOffset...int) int {
@@ -56,6 +58,7 @@ func PosI(haystack, needle string, startOffset...int) int {
 }
 
 // Find the position of the last occurrence of a substring in a string.
+// It returns -1, if none found.
 //
 // 查找指定字符串在目标字符串中最后一次出现的位置，找不到返回-1。
 func PosR(haystack, needle string, startOffset...int) int {
@@ -81,6 +84,7 @@ func PosR(haystack, needle string, startOffset...int) int {
 }
 
 // Find the position of the last occurrence of a case-insensitive substring in a string.
+// It returns -1, if none found.
 //
 // 以不区分大小写的方式查找指定字符串在目标字符串中最后一次出现的位置，找不到返回-1。
 func PosRI(haystack, needle string, startOffset...int) int {

g/text/gstr/gstr_z_unit_test.go

@@ -68,26 +68,6 @@ func Test_UcWords(t *testing.T) {
     })
 }
 
-func Test_SearchArray(t *testing.T) {
-    gtest.Case(t, func() {
-        array := []string{"a", "b", "c"}
-        gtest.Assert(gstr.SearchArray(array, "a"),  0)
-        gtest.Assert(gstr.SearchArray(array, "b"),  1)
-        gtest.Assert(gstr.SearchArray(array, "c"),  2)
-        gtest.Assert(gstr.SearchArray(array, "d"), -1)
-    })
-}
-
-func Test_InArray(t *testing.T) {
-    gtest.Case(t, func() {
-        array := []string{"a", "b", "c"}
-        gtest.Assert(gstr.InArray(array, "a"), true)
-        gtest.Assert(gstr.InArray(array, "b"), true)
-        gtest.Assert(gstr.InArray(array, "c"), true)
-        gtest.Assert(gstr.InArray(array, "d"), false)
-    })
-}
-
 func Test_IsLetterLower(t *testing.T) {
     gtest.Case(t, func() {
         gtest.Assert(gstr.IsLetterLower('a'), true)

g/util/gconv/gconv_map.go

@@ -99,10 +99,13 @@ func Map(i interface{}, noTagCheck...bool) map[string]interface{} {
                     rt   := rv.Type()
                     name := ""
                     for i := 0; i < rv.NumField(); i++ {
-                        // 检查json tag
+                        // 检查tag, 支持gconv, json标签, 优先使用gconv
                         if len(noTagCheck) == 0 || !noTagCheck[0] {
-                            if name = rt.Field(i).Tag.Get("json"); name == "" {
-                                name = rt.Field(i).Name
+                            tag := rt.Field(i).Tag
+                            if name = tag.Get("gconv"); name == "" {
+                                if name = tag.Get("json"); name == "" {
+                                    name = rt.Field(i).Name
+                                }
                             }
                         }
                         m[name] = rv.Field(i).Interface()

geg/database/kafka/gkafka_consumer.go

@@ -1,35 +0,0 @@
-package main
-
-import (
-    "fmt"
-    "github.com/gogf/gf/g/database/gkafka"
-)
-
-// 创建kafka消费客户端
-func newKafkaClientConsumer(topic, group string) *gkafka.Client {
-    kafkaConfig               := gkafka.NewConfig()
-    kafkaConfig.Servers        = "localhost:9092"
-    kafkaConfig.AutoMarkOffset = false
-    kafkaConfig.Topics         = topic
-    kafkaConfig.GroupId        = group
-    return gkafka.NewClient(kafkaConfig)
-}
-
-func main () {
-    group  := "test-group"
-    topic  := "test"
-    client := newKafkaClientConsumer(topic, group)
-    defer client.Close()
-
-    // 标记开始读取的offset位置
-    client.MarkOffset(topic, 0, 6)
-    for {
-        if msg, err := client.Receive(); err != nil {
-            fmt.Println(err)
-            break
-        } else {
-            fmt.Println(msg.Partition, msg.Offset, string(msg.Value))
-            msg.MarkOffset()
-        }
-    }
-}

geg/database/kafka/gkafka_producer.go

@@ -1,28 +0,0 @@
-package main
-
-import (
-    "github.com/gogf/gf/g/database/gkafka"
-    "fmt"
-    "github.com/gogf/gf/g/os/gtime"
-    "time"
-)
-
-// 创建kafka生产客户端
-func newKafkaClientProducer(topic string) *gkafka.Client {
-    kafkaConfig               := gkafka.NewConfig()
-    kafkaConfig.Servers        = "localhost:9092"
-    kafkaConfig.AutoMarkOffset = false
-    kafkaConfig.Topics         = topic
-    return gkafka.NewClient(kafkaConfig)
-}
-
-func main () {
-    client := newKafkaClientProducer("test")
-    defer client.Close()
-    for {
-        if err := client.SyncSend(&gkafka.Message{Value: []byte(gtime.Now().String())}); err != nil {
-            fmt.Println(err)
-        }
-        time.Sleep(time.Second)
-    }
-}

geg/database/kafka/gkafka_topics.go

@@ -1,16 +0,0 @@
-package main
-
-import (
-    "github.com/gogf/gf/g/database/gkafka"
-    "fmt"
-)
-
-func main () {
-    config        := gkafka.NewConfig()
-    config.Servers = "localhost:9092"
-
-    client := gkafka.NewClient(config)
-    defer client.Close()
-
-    fmt.Println(client.Topics())
-}

geg/net/ghttp/server/template/layout/main.go

@@ -0,0 +1,23 @@
+package main
+
+import (
+    "github.com/gogf/gf/g"
+    "github.com/gogf/gf/g/net/ghttp"
+)
+
+func main() {
+    s := g.Server()
+    s.BindHandler("/main1", func(r *ghttp.Request) {
+        r.Response.WriteTpl("layout.html", g.Map{
+            "mainTpl" : "main/main1.html",
+        })
+    })
+    s.BindHandler("/main2", func(r *ghttp.Request) {
+        r.Response.WriteTpl("layout.html", g.Map{
+            "mainTpl" : "main/main2.html",
+        })
+    })
+    g.View().SetPath("template")
+    s.SetPort(8199)
+    s.Run()
+}

geg/net/ghttp/server/template/layout/template/footer.html

@@ -0,0 +1 @@
+<h1>FOOTER</h1>

geg/net/ghttp/server/template/layout/template/header.html

@@ -0,0 +1 @@
+<h1>HEADER</h1>

geg/net/ghttp/server/template/layout/template/layout.html

@@ -0,0 +1,3 @@
+{{include "header.html" .}}
+{{include .mainTpl .}}
+{{include "footer.html" .}}

geg/net/ghttp/server/template/layout/template/main/main1.html

@@ -0,0 +1 @@
+<h1>MAIN1</h1>

geg/net/ghttp/server/template/layout/template/main/main2.html

@@ -0,0 +1 @@
+<h1>MAIN2</h1>

geg/os/gcron/gcron2.go

@@ -1,17 +1,18 @@
 package main
 
 import (
+    "fmt"
     "github.com/gogf/gf/g/os/gcron"
-    "github.com/gogf/gf/g/os/glog"
     "time"
 )
 
-func main() {
-    cron := gcron.New()
-    glog.Println("start")
-    cron.DelayAddOnce(1, "* * * * * *", func() {
-        glog.Println("run")
-    })
+func test() {
 
+}
+
+func main() {
+    _, err := gcron.Add("*/10 * * * * ?", test)
+    fmt.Println(err)
+    fmt.Println(gcron.Entries())
     time.Sleep(10*time.Second)
 }

geg/other/test.go

@@ -2,18 +2,13 @@ package main
 
 import (
     "fmt"
-    "github.com/gogf/gf/g/container/garray"
 )
 
 func main() {
-    value1 := []interface{}{0,1,2,3,4,5,6}
-    fmt.Println(value1[1:2])
-    return
-    array1 := garray.NewArrayFrom(value1)
-    a := array1.Range(0, 1)
-    fmt.Println(a)
-    fmt.Println(array1.Slice())
-    a = append(a, 100)
-    fmt.Println(a)
-    fmt.Println(array1.Slice())
+    s := "abc我是中国人é"
+    fmt.Println(len(s))
+
+    for i := 0; i < len(s); i++ {
+        fmt.Println(s[i])
+    }
 }

third/github.com/DataDog/zstd/.travis.yml

@@ -1,30 +0,0 @@
-language: go
-
-go:
-  - 1.9.x
-  - 1.10.x
-  - 1.11.x
-
-os:
-  - linux
-  - osx
-
-matrix:
-  include:
-    name: "Go 1.11.x CentOS 32bits"
-    language: go
-    go: 1.11.x
-    os: linux
-    services:
-      - docker
-    script:
-      # Please update Go version in travis_test_32 as needed
-      - "docker run -i -v \"${PWD}:/zstd\" toopher/centos-i386:centos6 /bin/bash -c \"linux32 --32bit i386 /zstd/travis_test_32.sh\""
-
-install:
-  - "wget https://github.com/DataDog/zstd/files/2246767/mr.zip"
-  - "unzip mr.zip"
-script:
-  - "go build"
-  - "PAYLOAD=`pwd`/mr go test -v"
-  - "PAYLOAD=`pwd`/mr go test -bench ."

third/github.com/DataDog/zstd/LICENSE

@@ -1,27 +0,0 @@
-Simplified BSD License
-
-Copyright (c) 2016, Datadog <info@datadoghq.com>
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-      this list of conditions and the following disclaimer.
-    * Redistributions in binary form must reproduce the above copyright notice,
-      this list of conditions and the following disclaimer in the documentation
-      and/or other materials provided with the distribution.
-    * Neither the name of the copyright holder nor the names of its contributors
-      may be used to endorse or promote products derived from this software
-      without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

third/github.com/DataDog/zstd/README.md

@@ -1,120 +0,0 @@
-# Zstd Go Wrapper
-
-[C Zstd Homepage](https://github.com/Cyan4973/zstd)
-
-The current headers and C files are from *v1.3.4* (Commit
-[2555975](https://github.com/facebook/zstd/releases/tag/v1.3.4)).
-
-## Usage
-
-There are two main APIs:
-
-* simple Compress/Decompress
-* streaming API (io.Reader/io.Writer)
-
-The compress/decompress APIs mirror that of lz4, while the streaming API was
-designed to be a drop-in replacement for zlib.
-
-### Simple `Compress/Decompress`
-
-
-```go
-// Compress compresses the byte array given in src and writes it to dst.
-// If you already have a buffer allocated, you can pass it to prevent allocation
-// If not, you can pass nil as dst.
-// If the buffer is too small, it will be reallocated, resized, and returned bu the function
-// If dst is nil, this will allocate the worst case size (CompressBound(src))
-Compress(dst, src []byte) ([]byte, error)
-```
-
-```go
-// CompressLevel is the same as Compress but you can pass another compression level
-CompressLevel(dst, src []byte, level int) ([]byte, error)
-```
-
-```go
-// Decompress will decompress your payload into dst.
-// If you already have a buffer allocated, you can pass it to prevent allocation
-// If not, you can pass nil as dst (allocates a 4*src size as default).
-// If the buffer is too small, it will retry 3 times by doubling the dst size
-// After max retries, it will switch to the slower stream API to be sure to be able
-// to decompress. Currently switches if compression ratio > 4*2**3=32.
-Decompress(dst, src []byte) ([]byte, error)
-```
-
-### Stream API
-
-```go
-// NewWriter creates a new object that can optionally be initialized with
-// a precomputed dictionary. If dict is nil, compress without a dictionary.
-// The dictionary array should not be changed during the use of this object.
-// You MUST CALL Close() to write the last bytes of a zstd stream and free C objects.
-NewWriter(w io.Writer) *Writer
-NewWriterLevel(w io.Writer, level int) *Writer
-NewWriterLevelDict(w io.Writer, level int, dict []byte) *Writer
-
-// Write compresses the input data and write it to the underlying writer
-(w *Writer) Write(p []byte) (int, error)
-
-// Close flushes the buffer and frees C zstd objects
-(w *Writer) Close() error
-```
-
-```go
-// NewReader returns a new io.ReadCloser that will decompress data from the
-// underlying reader.  If a dictionary is provided to NewReaderDict, it must
-// not be modified until Close is called.  It is the caller's responsibility
-// to call Close, which frees up C objects.
-NewReader(r io.Reader) io.ReadCloser
-NewReaderDict(r io.Reader, dict []byte) io.ReadCloser
-```
-
-### Benchmarks (benchmarked with v0.5.0)
-
-The author of Zstd also wrote lz4. Zstd is intended to occupy a speed/ratio
-level similar to what zlib currently provides.  In our tests, the can always
-be made to be better than zlib by chosing an appropriate level while still
-keeping compression and decompression time faster than zlib.
-
-You can run the benchmarks against your own payloads by using the Go benchmarks tool.
-Just export your payload filepath as the `PAYLOAD` environment variable and run the benchmarks:
-
-```go
-go test -bench .
-```
-
-Compression of a 7Mb pdf zstd (this wrapper) vs [czlib](https://github.com/DataDog/czlib):
-```
-BenchmarkCompression               5     221056624 ns/op      67.34 MB/s
-BenchmarkDecompression           100      18370416 ns/op     810.32 MB/s
-
-BenchmarkFzlibCompress             2     610156603 ns/op      24.40 MB/s
-BenchmarkFzlibDecompress          20      81195246 ns/op     183.33 MB/s
-```
-
-Ratio is also better by a margin of ~20%.
-Compression speed is always better than zlib on all the payloads we tested;
-However, [czlib](https://github.com/DataDog/czlib) has optimisations that make it
-faster at decompressiong small payloads:
-
-```
-Testing with size: 11... czlib: 8.97 MB/s, zstd: 3.26 MB/s
-Testing with size: 27... czlib: 23.3 MB/s, zstd: 8.22 MB/s
-Testing with size: 62... czlib: 31.6 MB/s, zstd: 19.49 MB/s
-Testing with size: 141... czlib: 74.54 MB/s, zstd: 42.55 MB/s
-Testing with size: 323... czlib: 155.14 MB/s, zstd: 99.39 MB/s
-Testing with size: 739... czlib: 235.9 MB/s, zstd: 216.45 MB/s
-Testing with size: 1689... czlib: 116.45 MB/s, zstd: 345.64 MB/s
-Testing with size: 3858... czlib: 176.39 MB/s, zstd: 617.56 MB/s
-Testing with size: 8811... czlib: 254.11 MB/s, zstd: 824.34 MB/s
-Testing with size: 20121... czlib: 197.43 MB/s, zstd: 1339.11 MB/s
-Testing with size: 45951... czlib: 201.62 MB/s, zstd: 1951.57 MB/s
-```
-
-zstd starts to shine with payloads > 1KB
-
-### Stability - Current state: STABLE
-
-The C library seems to be pretty stable and according to the author has been tested and fuzzed.
-
-For the Go wrapper, the test cover most usual cases and we have succesfully tested it on all staging and prod data.

third/github.com/DataDog/zstd/ZSTD_LICENSE

@@ -1,30 +0,0 @@
-BSD License
-
-For Zstandard software
-
-Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
-
-Redistribution and use in source and binary forms, with or without modification,
-are permitted provided that the following conditions are met:
-
- * Redistributions of source code must retain the above copyright notice, this
-   list of conditions and the following disclaimer.
-
- * Redistributions in binary form must reproduce the above copyright notice,
-   this list of conditions and the following disclaimer in the documentation
-   and/or other materials provided with the distribution.
-
- * Neither the name Facebook nor the names of its contributors may be used to
-   endorse or promote products derived from this software without specific
-   prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

third/github.com/DataDog/zstd/bitstream.h

@@ -1,471 +0,0 @@
-/* ******************************************************************
-   bitstream
-   Part of FSE library
-   header file (to include)
-   Copyright (C) 2013-2017, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
-****************************************************************** */
-#ifndef BITSTREAM_H_MODULE
-#define BITSTREAM_H_MODULE
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-/*
-*  This API consists of small unitary functions, which must be inlined for best performance.
-*  Since link-time-optimization is not available for all compilers,
-*  these functions are defined into a .h to be included.
-*/
-
-/*-****************************************
-*  Dependencies
-******************************************/
-#include "mem.h"            /* unaligned access routines */
-#include "error_private.h"  /* error codes and messages */
-
-
-/*-*************************************
-*  Debug
-***************************************/
-#if defined(BIT_DEBUG) && (BIT_DEBUG>=1)
-#  include <assert.h>
-#else
-#  ifndef assert
-#    define assert(condition) ((void)0)
-#  endif
-#endif
-
-
-/*=========================================
-*  Target specific
-=========================================*/
-#if defined(__BMI__) && defined(__GNUC__)
-#  include <immintrin.h>   /* support for bextr (experimental) */
-#endif
-
-#define STREAM_ACCUMULATOR_MIN_32  25
-#define STREAM_ACCUMULATOR_MIN_64  57
-#define STREAM_ACCUMULATOR_MIN    ((U32)(MEM_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64))
-
-
-/*-******************************************
-*  bitStream encoding API (write forward)
-********************************************/
-/* bitStream can mix input from multiple sources.
- * A critical property of these streams is that they encode and decode in **reverse** direction.
- * So the first bit sequence you add will be the last to be read, like a LIFO stack.
- */
-typedef struct
-{
-    size_t bitContainer;
-    unsigned bitPos;
-    char*  startPtr;
-    char*  ptr;
-    char*  endPtr;
-} BIT_CStream_t;
-
-MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* dstBuffer, size_t dstCapacity);
-MEM_STATIC void   BIT_addBits(BIT_CStream_t* bitC, size_t value, unsigned nbBits);
-MEM_STATIC void   BIT_flushBits(BIT_CStream_t* bitC);
-MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC);
-
-/* Start with initCStream, providing the size of buffer to write into.
-*  bitStream will never write outside of this buffer.
-*  `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code.
-*
-*  bits are first added to a local register.
-*  Local register is size_t, hence 64-bits on 64-bits systems, or 32-bits on 32-bits systems.
-*  Writing data into memory is an explicit operation, performed by the flushBits function.
-*  Hence keep track how many bits are potentially stored into local register to avoid register overflow.
-*  After a flushBits, a maximum of 7 bits might still be stored into local register.
-*
-*  Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers.
-*
-*  Last operation is to close the bitStream.
-*  The function returns the final size of CStream in bytes.
-*  If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable)
-*/
-
-
-/*-********************************************
-*  bitStream decoding API (read backward)
-**********************************************/
-typedef struct
-{
-    size_t   bitContainer;
-    unsigned bitsConsumed;
-    const char* ptr;
-    const char* start;
-    const char* limitPtr;
-} BIT_DStream_t;
-
-typedef enum { BIT_DStream_unfinished = 0,
-               BIT_DStream_endOfBuffer = 1,
-               BIT_DStream_completed = 2,
-               BIT_DStream_overflow = 3 } BIT_DStream_status;  /* result of BIT_reloadDStream() */
-               /* 1,2,4,8 would be better for bitmap combinations, but slows down performance a bit ... :( */
-
-MEM_STATIC size_t   BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize);
-MEM_STATIC size_t   BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits);
-MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD);
-MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* bitD);
-
-
-/* Start by invoking BIT_initDStream().
-*  A chunk of the bitStream is then stored into a local register.
-*  Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
-*  You can then retrieve bitFields stored into the local register, **in reverse order**.
-*  Local register is explicitly reloaded from memory by the BIT_reloadDStream() method.
-*  A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished.
-*  Otherwise, it can be less than that, so proceed accordingly.
-*  Checking if DStream has reached its end can be performed with BIT_endOfDStream().
-*/
-
-
-/*-****************************************
-*  unsafe API
-******************************************/
-MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, size_t value, unsigned nbBits);
-/* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */
-
-MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC);
-/* unsafe version; does not check buffer overflow */
-
-MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
-/* faster, but works only if nbBits >= 1 */
-
-
-
-/*-**************************************************************
-*  Internal functions
-****************************************************************/
-MEM_STATIC unsigned BIT_highbit32 (U32 val)
-{
-    assert(val != 0);
-    {
-#   if defined(_MSC_VER)   /* Visual */
-        unsigned long r=0;
-        _BitScanReverse ( &r, val );
-        return (unsigned) r;
-#   elif defined(__GNUC__) && (__GNUC__ >= 3)   /* Use GCC Intrinsic */
-        return 31 - __builtin_clz (val);
-#   else   /* Software version */
-        static const unsigned DeBruijnClz[32] = { 0,  9,  1, 10, 13, 21,  2, 29,
-                                                 11, 14, 16, 18, 22, 25,  3, 30,
-                                                  8, 12, 20, 28, 15, 17, 24,  7,
-                                                 19, 27, 23,  6, 26,  5,  4, 31 };
-        U32 v = val;
-        v |= v >> 1;
-        v |= v >> 2;
-        v |= v >> 4;
-        v |= v >> 8;
-        v |= v >> 16;
-        return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
-#   endif
-    }
-}
-
-/*=====    Local Constants   =====*/
-static const unsigned BIT_mask[] = {
-    0,          1,         3,         7,         0xF,       0x1F,
-    0x3F,       0x7F,      0xFF,      0x1FF,     0x3FF,     0x7FF,
-    0xFFF,      0x1FFF,    0x3FFF,    0x7FFF,    0xFFFF,    0x1FFFF,
-    0x3FFFF,    0x7FFFF,   0xFFFFF,   0x1FFFFF,  0x3FFFFF,  0x7FFFFF,
-    0xFFFFFF,   0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF,
-    0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */
-#define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0]))
-
-/*-**************************************************************
-*  bitStream encoding
-****************************************************************/
-/*! BIT_initCStream() :
- *  `dstCapacity` must be > sizeof(size_t)
- *  @return : 0 if success,
- *            otherwise an error code (can be tested using ERR_isError()) */
-MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC,
-                                  void* startPtr, size_t dstCapacity)
-{
-    bitC->bitContainer = 0;
-    bitC->bitPos = 0;
-    bitC->startPtr = (char*)startPtr;
-    bitC->ptr = bitC->startPtr;
-    bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer);
-    if (dstCapacity <= sizeof(bitC->bitContainer)) return ERROR(dstSize_tooSmall);
-    return 0;
-}
-
-/*! BIT_addBits() :
- *  can add up to 31 bits into `bitC`.
- *  Note : does not check for register overflow ! */
-MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC,
-                            size_t value, unsigned nbBits)
-{
-    MEM_STATIC_ASSERT(BIT_MASK_SIZE == 32);
-    assert(nbBits < BIT_MASK_SIZE);
-    assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    bitC->bitContainer |= (value & BIT_mask[nbBits]) << bitC->bitPos;
-    bitC->bitPos += nbBits;
-}
-
-/*! BIT_addBitsFast() :
- *  works only if `value` is _clean_, meaning all high bits above nbBits are 0 */
-MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
-                                size_t value, unsigned nbBits)
-{
-    assert((value>>nbBits) == 0);
-    assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    bitC->bitContainer |= value << bitC->bitPos;
-    bitC->bitPos += nbBits;
-}
-
-/*! BIT_flushBitsFast() :
- *  assumption : bitContainer has not overflowed
- *  unsafe version; does not check buffer overflow */
-MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
-{
-    size_t const nbBytes = bitC->bitPos >> 3;
-    assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    MEM_writeLEST(bitC->ptr, bitC->bitContainer);
-    bitC->ptr += nbBytes;
-    assert(bitC->ptr <= bitC->endPtr);
-    bitC->bitPos &= 7;
-    bitC->bitContainer >>= nbBytes*8;
-}
-
-/*! BIT_flushBits() :
- *  assumption : bitContainer has not overflowed
- *  safe version; check for buffer overflow, and prevents it.
- *  note : does not signal buffer overflow.
- *  overflow will be revealed later on using BIT_closeCStream() */
-MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
-{
-    size_t const nbBytes = bitC->bitPos >> 3;
-    assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    MEM_writeLEST(bitC->ptr, bitC->bitContainer);
-    bitC->ptr += nbBytes;
-    if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
-    bitC->bitPos &= 7;
-    bitC->bitContainer >>= nbBytes*8;
-}
-
-/*! BIT_closeCStream() :
- *  @return : size of CStream, in bytes,
- *            or 0 if it could not fit into dstBuffer */
-MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC)
-{
-    BIT_addBitsFast(bitC, 1, 1);   /* endMark */
-    BIT_flushBits(bitC);
-    if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */
-    return (bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0);
-}
-
-
-/*-********************************************************
-*  bitStream decoding
-**********************************************************/
-/*! BIT_initDStream() :
- *  Initialize a BIT_DStream_t.
- * `bitD` : a pointer to an already allocated BIT_DStream_t structure.
- * `srcSize` must be the *exact* size of the bitStream, in bytes.
- * @return : size of stream (== srcSize), or an errorCode if a problem is detected
- */
-MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize)
-{
-    if (srcSize < 1) { memset(bitD, 0, sizeof(*bitD)); return ERROR(srcSize_wrong); }
-
-    bitD->start = (const char*)srcBuffer;
-    bitD->limitPtr = bitD->start + sizeof(bitD->bitContainer);
-
-    if (srcSize >=  sizeof(bitD->bitContainer)) {  /* normal case */
-        bitD->ptr   = (const char*)srcBuffer + srcSize - sizeof(bitD->bitContainer);
-        bitD->bitContainer = MEM_readLEST(bitD->ptr);
-        { BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
-          bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0;  /* ensures bitsConsumed is always set */
-          if (lastByte == 0) return ERROR(GENERIC); /* endMark not present */ }
-    } else {
-        bitD->ptr   = bitD->start;
-        bitD->bitContainer = *(const BYTE*)(bitD->start);
-        switch(srcSize)
-        {
-        case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16);
-                /* fall-through */
-
-        case 6: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[5]) << (sizeof(bitD->bitContainer)*8 - 24);
-                /* fall-through */
-
-        case 5: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[4]) << (sizeof(bitD->bitContainer)*8 - 32);
-                /* fall-through */
-
-        case 4: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[3]) << 24;
-                /* fall-through */
-
-        case 3: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[2]) << 16;
-                /* fall-through */
-
-        case 2: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[1]) <<  8;
-                /* fall-through */
-
-        default: break;
-        }
-        {   BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
-            bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0;
-            if (lastByte == 0) return ERROR(corruption_detected);  /* endMark not present */
-        }
-        bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize)*8;
-    }
-
-    return srcSize;
-}
-
-MEM_STATIC size_t BIT_getUpperBits(size_t bitContainer, U32 const start)
-{
-    return bitContainer >> start;
-}
-
-MEM_STATIC size_t BIT_getMiddleBits(size_t bitContainer, U32 const start, U32 const nbBits)
-{
-#if defined(__BMI__) && defined(__GNUC__) && __GNUC__*1000+__GNUC_MINOR__ >= 4008  /* experimental */
-#  if defined(__x86_64__)
-    if (sizeof(bitContainer)==8)
-        return _bextr_u64(bitContainer, start, nbBits);
-    else
-#  endif
-        return _bextr_u32(bitContainer, start, nbBits);
-#else
-    assert(nbBits < BIT_MASK_SIZE);
-    return (bitContainer >> start) & BIT_mask[nbBits];
-#endif
-}
-
-MEM_STATIC size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits)
-{
-    assert(nbBits < BIT_MASK_SIZE);
-    return bitContainer & BIT_mask[nbBits];
-}
-
-/*! BIT_lookBits() :
- *  Provides next n bits from local register.
- *  local register is not modified.
- *  On 32-bits, maxNbBits==24.
- *  On 64-bits, maxNbBits==56.
- * @return : value extracted */
-MEM_STATIC size_t BIT_lookBits(const BIT_DStream_t* bitD, U32 nbBits)
-{
-#if defined(__BMI__) && defined(__GNUC__)   /* experimental; fails if bitD->bitsConsumed + nbBits > sizeof(bitD->bitContainer)*8 */
-    return BIT_getMiddleBits(bitD->bitContainer, (sizeof(bitD->bitContainer)*8) - bitD->bitsConsumed - nbBits, nbBits);
-#else
-    U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
-    return ((bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> 1) >> ((regMask-nbBits) & regMask);
-#endif
-}
-
-/*! BIT_lookBitsFast() :
- *  unsafe version; only works if nbBits >= 1 */
-MEM_STATIC size_t BIT_lookBitsFast(const BIT_DStream_t* bitD, U32 nbBits)
-{
-    U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
-    assert(nbBits >= 1);
-    return (bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> (((regMask+1)-nbBits) & regMask);
-}
-
-MEM_STATIC void BIT_skipBits(BIT_DStream_t* bitD, U32 nbBits)
-{
-    bitD->bitsConsumed += nbBits;
-}
-
-/*! BIT_readBits() :
- *  Read (consume) next n bits from local register and update.
- *  Pay attention to not read more than nbBits contained into local register.
- * @return : extracted value. */
-MEM_STATIC size_t BIT_readBits(BIT_DStream_t* bitD, U32 nbBits)
-{
-    size_t const value = BIT_lookBits(bitD, nbBits);
-    BIT_skipBits(bitD, nbBits);
-    return value;
-}
-
-/*! BIT_readBitsFast() :
- *  unsafe version; only works only if nbBits >= 1 */
-MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, U32 nbBits)
-{
-    size_t const value = BIT_lookBitsFast(bitD, nbBits);
-    assert(nbBits >= 1);
-    BIT_skipBits(bitD, nbBits);
-    return value;
-}
-
-/*! BIT_reloadDStream() :
- *  Refill `bitD` from buffer previously set in BIT_initDStream() .
- *  This function is safe, it guarantees it will not read beyond src buffer.
- * @return : status of `BIT_DStream_t` internal register.
- *           when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */
-MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD)
-{
-    if (bitD->bitsConsumed > (sizeof(bitD->bitContainer)*8))  /* overflow detected, like end of stream */
-        return BIT_DStream_overflow;
-
-    if (bitD->ptr >= bitD->limitPtr) {
-        bitD->ptr -= bitD->bitsConsumed >> 3;
-        bitD->bitsConsumed &= 7;
-        bitD->bitContainer = MEM_readLEST(bitD->ptr);
-        return BIT_DStream_unfinished;
-    }
-    if (bitD->ptr == bitD->start) {
-        if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer;
-        return BIT_DStream_completed;
-    }
-    /* start < ptr < limitPtr */
-    {   U32 nbBytes = bitD->bitsConsumed >> 3;
-        BIT_DStream_status result = BIT_DStream_unfinished;
-        if (bitD->ptr - nbBytes < bitD->start) {
-            nbBytes = (U32)(bitD->ptr - bitD->start);  /* ptr > start */
-            result = BIT_DStream_endOfBuffer;
-        }
-        bitD->ptr -= nbBytes;
-        bitD->bitsConsumed -= nbBytes*8;
-        bitD->bitContainer = MEM_readLEST(bitD->ptr);   /* reminder : srcSize > sizeof(bitD->bitContainer), otherwise bitD->ptr == bitD->start */
-        return result;
-    }
-}
-
-/*! BIT_endOfDStream() :
- * @return : 1 if DStream has _exactly_ reached its end (all bits consumed).
- */
-MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream)
-{
-    return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8));
-}
-
-#if defined (__cplusplus)
-}
-#endif
-
-#endif /* BITSTREAM_H_MODULE */

third/github.com/DataDog/zstd/compiler.h

@@ -1,111 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_COMPILER_H
-#define ZSTD_COMPILER_H
-
-/*-*******************************************************
-*  Compiler specifics
-*********************************************************/
-/* force inlining */
-#if defined (__GNUC__) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
-#  define INLINE_KEYWORD inline
-#else
-#  define INLINE_KEYWORD
-#endif
-
-#if defined(__GNUC__)
-#  define FORCE_INLINE_ATTR __attribute__((always_inline))
-#elif defined(_MSC_VER)
-#  define FORCE_INLINE_ATTR __forceinline
-#else
-#  define FORCE_INLINE_ATTR
-#endif
-
-/**
- * FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
- * parameters. They must be inlined for the compiler to elimininate the constant
- * branches.
- */
-#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR
-/**
- * HINT_INLINE is used to help the compiler generate better code. It is *not*
- * used for "templates", so it can be tweaked based on the compilers
- * performance.
- *
- * gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the
- * always_inline attribute.
- *
- * clang up to 5.0.0 (trunk) benefit tremendously from the always_inline
- * attribute.
- */
-#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
-#  define HINT_INLINE static INLINE_KEYWORD
-#else
-#  define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR
-#endif
-
-/* force no inlining */
-#ifdef _MSC_VER
-#  define FORCE_NOINLINE static __declspec(noinline)
-#else
-#  ifdef __GNUC__
-#    define FORCE_NOINLINE static __attribute__((__noinline__))
-#  else
-#    define FORCE_NOINLINE static
-#  endif
-#endif
-
-/* target attribute */
-#ifndef __has_attribute
-  #define __has_attribute(x) 0  /* Compatibility with non-clang compilers. */
-#endif
-#if defined(__GNUC__)
-#  define TARGET_ATTRIBUTE(target) __attribute__((__target__(target)))
-#else
-#  define TARGET_ATTRIBUTE(target)
-#endif
-
-/* Enable runtime BMI2 dispatch based on the CPU.
- * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
- */
-#ifndef DYNAMIC_BMI2
-  #if (defined(__clang__) && __has_attribute(__target__)) \
-      || (defined(__GNUC__) \
-          && (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8))) \
-      && (defined(__x86_64__) || defined(_M_X86)) \
-      && !defined(__BMI2__)
-  #  define DYNAMIC_BMI2 1
-  #else
-  #  define DYNAMIC_BMI2 0
-  #endif
-#endif
-
-/* prefetch */
-#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86))  /* _mm_prefetch() is not defined outside of x86/x64 */
-#  include <mmintrin.h>   /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
-#  define PREFETCH(ptr)   _mm_prefetch((const char*)ptr, _MM_HINT_T0)
-#elif defined(__GNUC__)
-#  define PREFETCH(ptr)   __builtin_prefetch(ptr, 0, 0)
-#else
-#  define PREFETCH(ptr)   /* disabled */
-#endif
-
-/* disable warnings */
-#ifdef _MSC_VER    /* Visual Studio */
-#  include <intrin.h>                    /* For Visual 2005 */
-#  pragma warning(disable : 4100)        /* disable: C4100: unreferenced formal parameter */
-#  pragma warning(disable : 4127)        /* disable: C4127: conditional expression is constant */
-#  pragma warning(disable : 4204)        /* disable: C4204: non-constant aggregate initializer */
-#  pragma warning(disable : 4214)        /* disable: C4214: non-int bitfields */
-#  pragma warning(disable : 4324)        /* disable: C4324: padded structure */
-#endif
-
-#endif /* ZSTD_COMPILER_H */

third/github.com/DataDog/zstd/cpu.h

@@ -1,216 +0,0 @@
-/*
- * Copyright (c) 2018-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_COMMON_CPU_H
-#define ZSTD_COMMON_CPU_H
-
-/**
- * Implementation taken from folly/CpuId.h
- * https://github.com/facebook/folly/blob/master/folly/CpuId.h
- */
-
-#include <string.h>
-
-#include "mem.h"
-
-#ifdef _MSC_VER
-#include <intrin.h>
-#endif
-
-typedef struct {
-    U32 f1c;
-    U32 f1d;
-    U32 f7b;
-    U32 f7c;
-} ZSTD_cpuid_t;
-
-MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) {
-    U32 f1c = 0;
-    U32 f1d = 0;
-    U32 f7b = 0;
-    U32 f7c = 0;
-#ifdef _MSC_VER
-    int reg[4];
-    __cpuid((int*)reg, 0);
-    {
-        int const n = reg[0];
-        if (n >= 1) {
-            __cpuid((int*)reg, 1);
-            f1c = (U32)reg[2];
-            f1d = (U32)reg[3];
-        }
-        if (n >= 7) {
-            __cpuidex((int*)reg, 7, 0);
-            f7b = (U32)reg[1];
-            f7c = (U32)reg[2];
-        }
-    }
-#elif defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__)
-    /* The following block like the normal cpuid branch below, but gcc
-     * reserves ebx for use of its pic register so we must specially
-     * handle the save and restore to avoid clobbering the register
-     */
-    U32 n;
-    __asm__(
-        "pushl %%ebx\n\t"
-        "cpuid\n\t"
-        "popl %%ebx\n\t"
-        : "=a"(n)
-        : "a"(0)
-        : "ecx", "edx");
-    if (n >= 1) {
-      U32 f1a;
-      __asm__(
-          "pushl %%ebx\n\t"
-          "cpuid\n\t"
-          "popl %%ebx\n\t"
-          : "=a"(f1a), "=c"(f1c), "=d"(f1d)
-          : "a"(1)
-          :);
-    }
-    if (n >= 7) {
-      __asm__(
-          "pushl %%ebx\n\t"
-          "cpuid\n\t"
-          "movl %%ebx, %%eax\n\r"
-          "popl %%ebx"
-          : "=a"(f7b), "=c"(f7c)
-          : "a"(7), "c"(0)
-          : "edx");
-    }
-#elif defined(__x86_64__) || defined(_M_X64) || defined(__i386__)
-    U32 n;
-    __asm__("cpuid" : "=a"(n) : "a"(0) : "ebx", "ecx", "edx");
-    if (n >= 1) {
-      U32 f1a;
-      __asm__("cpuid" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1) : "ebx");
-    }
-    if (n >= 7) {
-      U32 f7a;
-      __asm__("cpuid"
-              : "=a"(f7a), "=b"(f7b), "=c"(f7c)
-              : "a"(7), "c"(0)
-              : "edx");
-    }
-#endif
-    {
-        ZSTD_cpuid_t cpuid;
-        cpuid.f1c = f1c;
-        cpuid.f1d = f1d;
-        cpuid.f7b = f7b;
-        cpuid.f7c = f7c;
-        return cpuid;
-    }
-}
-
-#define X(name, r, bit)                                                        \
-  MEM_STATIC int ZSTD_cpuid_##name(ZSTD_cpuid_t const cpuid) {                 \
-    return ((cpuid.r) & (1U << bit)) != 0;                                     \
-  }
-
-/* cpuid(1): Processor Info and Feature Bits. */
-#define C(name, bit) X(name, f1c, bit)
-  C(sse3, 0)
-  C(pclmuldq, 1)
-  C(dtes64, 2)
-  C(monitor, 3)
-  C(dscpl, 4)
-  C(vmx, 5)
-  C(smx, 6)
-  C(eist, 7)
-  C(tm2, 8)
-  C(ssse3, 9)
-  C(cnxtid, 10)
-  C(fma, 12)
-  C(cx16, 13)
-  C(xtpr, 14)
-  C(pdcm, 15)
-  C(pcid, 17)
-  C(dca, 18)
-  C(sse41, 19)
-  C(sse42, 20)
-  C(x2apic, 21)
-  C(movbe, 22)
-  C(popcnt, 23)
-  C(tscdeadline, 24)
-  C(aes, 25)
-  C(xsave, 26)
-  C(osxsave, 27)
-  C(avx, 28)
-  C(f16c, 29)
-  C(rdrand, 30)
-#undef C
-#define D(name, bit) X(name, f1d, bit)
-  D(fpu, 0)
-  D(vme, 1)
-  D(de, 2)
-  D(pse, 3)
-  D(tsc, 4)
-  D(msr, 5)
-  D(pae, 6)
-  D(mce, 7)
-  D(cx8, 8)
-  D(apic, 9)
-  D(sep, 11)
-  D(mtrr, 12)
-  D(pge, 13)
-  D(mca, 14)
-  D(cmov, 15)
-  D(pat, 16)
-  D(pse36, 17)
-  D(psn, 18)
-  D(clfsh, 19)
-  D(ds, 21)
-  D(acpi, 22)
-  D(mmx, 23)
-  D(fxsr, 24)
-  D(sse, 25)
-  D(sse2, 26)
-  D(ss, 27)
-  D(htt, 28)
-  D(tm, 29)
-  D(pbe, 31)
-#undef D
-
-/* cpuid(7): Extended Features. */
-#define B(name, bit) X(name, f7b, bit)
-  B(bmi1, 3)
-  B(hle, 4)
-  B(avx2, 5)
-  B(smep, 7)
-  B(bmi2, 8)
-  B(erms, 9)
-  B(invpcid, 10)
-  B(rtm, 11)
-  B(mpx, 14)
-  B(avx512f, 16)
-  B(avx512dq, 17)
-  B(rdseed, 18)
-  B(adx, 19)
-  B(smap, 20)
-  B(avx512ifma, 21)
-  B(pcommit, 22)
-  B(clflushopt, 23)
-  B(clwb, 24)
-  B(avx512pf, 26)
-  B(avx512er, 27)
-  B(avx512cd, 28)
-  B(sha, 29)
-  B(avx512bw, 30)
-  B(avx512vl, 31)
-#undef B
-#define C(name, bit) X(name, f7c, bit)
-  C(prefetchwt1, 0)
-  C(avx512vbmi, 1)
-#undef C
-
-#undef X
-
-#endif /* ZSTD_COMMON_CPU_H */

third/github.com/DataDog/zstd/divsufsort.h

@@ -1,67 +0,0 @@
-/*
- * divsufsort.h for libdivsufsort-lite
- * Copyright (c) 2003-2008 Yuta Mori All Rights Reserved.
- *
- * Permission is hereby granted, free of charge, to any person
- * obtaining a copy of this software and associated documentation
- * files (the "Software"), to deal in the Software without
- * restriction, including without limitation the rights to use,
- * copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the
- * Software is furnished to do so, subject to the following
- * conditions:
- *
- * The above copyright notice and this permission notice shall be
- * included in all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
- * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
- * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
- * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
- * OTHER DEALINGS IN THE SOFTWARE.
- */
-
-#ifndef _DIVSUFSORT_H
-#define _DIVSUFSORT_H 1
-
-#ifdef __cplusplus
-extern "C" {
-#endif /* __cplusplus */
-
-
-/*- Prototypes -*/
-
-/**
- * Constructs the suffix array of a given string.
- * @param T [0..n-1] The input string.
- * @param SA [0..n-1] The output array of suffixes.
- * @param n The length of the given string.
- * @param openMP enables OpenMP optimization.
- * @return 0 if no error occurred, -1 or -2 otherwise.
- */
-int
-divsufsort(const unsigned char *T, int *SA, int n, int openMP);
-
-/**
- * Constructs the burrows-wheeler transformed string of a given string.
- * @param T [0..n-1] The input string.
- * @param U [0..n-1] The output string. (can be T)
- * @param A [0..n-1] The temporary array. (can be NULL)
- * @param n The length of the given string.
- * @param num_indexes The length of secondary indexes array. (can be NULL)
- * @param indexes The secondary indexes array. (can be NULL)
- * @param openMP enables OpenMP optimization.
- * @return The primary index if no error occurred, -1 or -2 otherwise.
- */
-int
-divbwt(const unsigned char *T, unsigned char *U, int *A, int n, unsigned char * num_indexes, int * indexes, int openMP);
-
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif /* __cplusplus */
-
-#endif /* _DIVSUFSORT_H */

third/github.com/DataDog/zstd/entropy_common.c

@@ -1,221 +0,0 @@
-/*
-   Common functions of New Generation Entropy library
-   Copyright (C) 2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
-*************************************************************************** */
-
-/* *************************************
-*  Dependencies
-***************************************/
-#include "mem.h"
-#include "error_private.h"       /* ERR_*, ERROR */
-#define FSE_STATIC_LINKING_ONLY  /* FSE_MIN_TABLELOG */
-#include "fse.h"
-#define HUF_STATIC_LINKING_ONLY  /* HUF_TABLELOG_ABSOLUTEMAX */
-#include "huf.h"
-
-
-/*===   Version   ===*/
-unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; }
-
-
-/*===   Error Management   ===*/
-unsigned FSE_isError(size_t code) { return ERR_isError(code); }
-const char* FSE_getErrorName(size_t code) { return ERR_getErrorName(code); }
-
-unsigned HUF_isError(size_t code) { return ERR_isError(code); }
-const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); }
-
-
-/*-**************************************************************
-*  FSE NCount encoding-decoding
-****************************************************************/
-size_t FSE_readNCount (short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
-                 const void* headerBuffer, size_t hbSize)
-{
-    const BYTE* const istart = (const BYTE*) headerBuffer;
-    const BYTE* const iend = istart + hbSize;
-    const BYTE* ip = istart;
-    int nbBits;
-    int remaining;
-    int threshold;
-    U32 bitStream;
-    int bitCount;
-    unsigned charnum = 0;
-    int previous0 = 0;
-
-    if (hbSize < 4) return ERROR(srcSize_wrong);
-    bitStream = MEM_readLE32(ip);
-    nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG;   /* extract tableLog */
-    if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX) return ERROR(tableLog_tooLarge);
-    bitStream >>= 4;
-    bitCount = 4;
-    *tableLogPtr = nbBits;
-    remaining = (1<<nbBits)+1;
-    threshold = 1<<nbBits;
-    nbBits++;
-
-    while ((remaining>1) & (charnum<=*maxSVPtr)) {
-        if (previous0) {
-            unsigned n0 = charnum;
-            while ((bitStream & 0xFFFF) == 0xFFFF) {
-                n0 += 24;
-                if (ip < iend-5) {
-                    ip += 2;
-                    bitStream = MEM_readLE32(ip) >> bitCount;
-                } else {
-                    bitStream >>= 16;
-                    bitCount   += 16;
-            }   }
-            while ((bitStream & 3) == 3) {
-                n0 += 3;
-                bitStream >>= 2;
-                bitCount += 2;
-            }
-            n0 += bitStream & 3;
-            bitCount += 2;
-            if (n0 > *maxSVPtr) return ERROR(maxSymbolValue_tooSmall);
-            while (charnum < n0) normalizedCounter[charnum++] = 0;
-            if ((ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
-                ip += bitCount>>3;
-                bitCount &= 7;
-                bitStream = MEM_readLE32(ip) >> bitCount;
-            } else {
-                bitStream >>= 2;
-        }   }
-        {   int const max = (2*threshold-1) - remaining;
-            int count;
-
-            if ((bitStream & (threshold-1)) < (U32)max) {
-                count = bitStream & (threshold-1);
-                bitCount += nbBits-1;
-            } else {
-                count = bitStream & (2*threshold-1);
-                if (count >= threshold) count -= max;
-                bitCount += nbBits;
-            }
-
-            count--;   /* extra accuracy */
-            remaining -= count < 0 ? -count : count;   /* -1 means +1 */
-            normalizedCounter[charnum++] = (short)count;
-            previous0 = !count;
-            while (remaining < threshold) {
-                nbBits--;
-                threshold >>= 1;
-            }
-
-            if ((ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
-                ip += bitCount>>3;
-                bitCount &= 7;
-            } else {
-                bitCount -= (int)(8 * (iend - 4 - ip));
-                ip = iend - 4;
-            }
-            bitStream = MEM_readLE32(ip) >> (bitCount & 31);
-    }   }   /* while ((remaining>1) & (charnum<=*maxSVPtr)) */
-    if (remaining != 1) return ERROR(corruption_detected);
-    if (bitCount > 32) return ERROR(corruption_detected);
-    *maxSVPtr = charnum-1;
-
-    ip += (bitCount+7)>>3;
-    return ip-istart;
-}
-
-
-/*! HUF_readStats() :
-    Read compact Huffman tree, saved by HUF_writeCTable().
-    `huffWeight` is destination buffer.
-    `rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32.
-    @return : size read from `src` , or an error Code .
-    Note : Needed by HUF_readCTable() and HUF_readDTableX?() .
-*/
-size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
-                     U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize)
-{
-    U32 weightTotal;
-    const BYTE* ip = (const BYTE*) src;
-    size_t iSize;
-    size_t oSize;
-
-    if (!srcSize) return ERROR(srcSize_wrong);
-    iSize = ip[0];
-    /* memset(huffWeight, 0, hwSize);   *//* is not necessary, even though some analyzer complain ... */
-
-    if (iSize >= 128) {  /* special header */
-        oSize = iSize - 127;
-        iSize = ((oSize+1)/2);
-        if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
-        if (oSize >= hwSize) return ERROR(corruption_detected);
-        ip += 1;
-        {   U32 n;
-            for (n=0; n<oSize; n+=2) {
-                huffWeight[n]   = ip[n/2] >> 4;
-                huffWeight[n+1] = ip[n/2] & 15;
-    }   }   }
-    else  {   /* header compressed with FSE (normal case) */
-        FSE_DTable fseWorkspace[FSE_DTABLE_SIZE_U32(6)];  /* 6 is max possible tableLog for HUF header (maybe even 5, to be tested) */
-        if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
-        oSize = FSE_decompress_wksp(huffWeight, hwSize-1, ip+1, iSize, fseWorkspace, 6);   /* max (hwSize-1) values decoded, as last one is implied */
-        if (FSE_isError(oSize)) return oSize;
-    }
-
-    /* collect weight stats */
-    memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
-    weightTotal = 0;
-    {   U32 n; for (n=0; n<oSize; n++) {
-            if (huffWeight[n] >= HUF_TABLELOG_MAX) return ERROR(corruption_detected);
-            rankStats[huffWeight[n]]++;
-            weightTotal += (1 << huffWeight[n]) >> 1;
-    }   }
-    if (weightTotal == 0) return ERROR(corruption_detected);
-
-    /* get last non-null symbol weight (implied, total must be 2^n) */
-    {   U32 const tableLog = BIT_highbit32(weightTotal) + 1;
-        if (tableLog > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
-        *tableLogPtr = tableLog;
-        /* determine last weight */
-        {   U32 const total = 1 << tableLog;
-            U32 const rest = total - weightTotal;
-            U32 const verif = 1 << BIT_highbit32(rest);
-            U32 const lastWeight = BIT_highbit32(rest) + 1;
-            if (verif != rest) return ERROR(corruption_detected);    /* last value must be a clean power of 2 */
-            huffWeight[oSize] = (BYTE)lastWeight;
-            rankStats[lastWeight]++;
-    }   }
-
-    /* check tree construction validity */
-    if ((rankStats[1] < 2) || (rankStats[1] & 1)) return ERROR(corruption_detected);   /* by construction : at least 2 elts of rank 1, must be even */
-
-    /* results */
-    *nbSymbolsPtr = (U32)(oSize+1);
-    return iSize+1;
-}

third/github.com/DataDog/zstd/error_private.c

@@ -1,48 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* The purpose of this file is to have a single list of error strings embedded in binary */
-
-#include "error_private.h"
-
-const char* ERR_getErrorString(ERR_enum code)
-{
-    static const char* const notErrorCode = "Unspecified error code";
-    switch( code )
-    {
-    case PREFIX(no_error): return "No error detected";
-    case PREFIX(GENERIC):  return "Error (generic)";
-    case PREFIX(prefix_unknown): return "Unknown frame descriptor";
-    case PREFIX(version_unsupported): return "Version not supported";
-    case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter";
-    case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding";
-    case PREFIX(corruption_detected): return "Corrupted block detected";
-    case PREFIX(checksum_wrong): return "Restored data doesn't match checksum";
-    case PREFIX(parameter_unsupported): return "Unsupported parameter";
-    case PREFIX(parameter_outOfBound): return "Parameter is out of bound";
-    case PREFIX(init_missing): return "Context should be init first";
-    case PREFIX(memory_allocation): return "Allocation error : not enough memory";
-    case PREFIX(workSpace_tooSmall): return "workSpace buffer is not large enough";
-    case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
-    case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
-    case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
-    case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
-    case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
-    case PREFIX(dictionary_wrong): return "Dictionary mismatch";
-    case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
-    case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
-    case PREFIX(srcSize_wrong): return "Src size is incorrect";
-        /* following error codes are not stable and may be removed or changed in a future version */
-    case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
-    case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
-    case PREFIX(maxCode):
-    default: return notErrorCode;
-    }
-}

third/github.com/DataDog/zstd/error_private.h

@@ -1,76 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* Note : this module is expected to remain private, do not expose it */
-
-#ifndef ERROR_H_MODULE
-#define ERROR_H_MODULE
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-
-/* ****************************************
-*  Dependencies
-******************************************/
-#include <stddef.h>        /* size_t */
-#include "zstd_errors.h"  /* enum list */
-
-
-/* ****************************************
-*  Compiler-specific
-******************************************/
-#if defined(__GNUC__)
-#  define ERR_STATIC static __attribute__((unused))
-#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
-#  define ERR_STATIC static inline
-#elif defined(_MSC_VER)
-#  define ERR_STATIC static __inline
-#else
-#  define ERR_STATIC static  /* this version may generate warnings for unused static functions; disable the relevant warning */
-#endif
-
-
-/*-****************************************
-*  Customization (error_public.h)
-******************************************/
-typedef ZSTD_ErrorCode ERR_enum;
-#define PREFIX(name) ZSTD_error_##name
-
-
-/*-****************************************
-*  Error codes handling
-******************************************/
-#undef ERROR   /* reported already defined on VS 2015 (Rich Geldreich) */
-#define ERROR(name) ZSTD_ERROR(name)
-#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
-
-ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
-
-ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); }
-
-
-/*-****************************************
-*  Error Strings
-******************************************/
-
-const char* ERR_getErrorString(ERR_enum code);   /* error_private.c */
-
-ERR_STATIC const char* ERR_getErrorName(size_t code)
-{
-    return ERR_getErrorString(ERR_getErrorCode(code));
-}
-
-#if defined (__cplusplus)
-}
-#endif
-
-#endif /* ERROR_H_MODULE */

third/github.com/DataDog/zstd/errors.go

@@ -1,35 +0,0 @@
-package zstd
-
-/*
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-*/
-import "C"
-
-// ErrorCode is an error returned by the zstd library.
-type ErrorCode int
-
-// Error returns the error string given by zstd
-func (e ErrorCode) Error() string {
-	return C.GoString(C.ZSTD_getErrorName(C.size_t(e)))
-}
-
-func cIsError(code int) bool {
-	return int(C.ZSTD_isError(C.size_t(code))) != 0
-}
-
-// getError returns an error for the return code, or nil if it's not an error
-func getError(code int) error {
-	if code < 0 && cIsError(code) {
-		return ErrorCode(code)
-	}
-	return nil
-}
-
-// IsDstSizeTooSmallError returns whether the error correspond to zstd standard sDstSizeTooSmall error
-func IsDstSizeTooSmallError(e error) bool {
-	if e != nil && e.Error() == "Destination buffer is too small" {
-		return true
-	}
-	return false
-}

third/github.com/DataDog/zstd/errors_test.go

@@ -1,29 +0,0 @@
-package zstd
-
-import (
-	"testing"
-)
-
-const (
-	// ErrorUpperBound is the upper bound to error number, currently only used in test
-	// If this needs to be updated, check in zstd_errors.h what the max is
-	ErrorUpperBound = 1000
-)
-
-// TestFindIsDstSizeTooSmallError tests that there is at least one error code that
-// corresponds to dst size too small
-func TestFindIsDstSizeTooSmallError(t *testing.T) {
-	found := 0
-	for i := -1; i > -ErrorUpperBound; i-- {
-		e := ErrorCode(i)
-		if IsDstSizeTooSmallError(e) {
-			found++
-		}
-	}
-
-	if found == 0 {
-		t.Fatal("Couldn't find an error code for DstSizeTooSmall error, please make sure we didn't change the error string")
-	} else if found > 1 {
-		t.Fatal("IsDstSizeTooSmallError found multiple error codes matching, this shouldn't be the case")
-	}
-}

third/github.com/DataDog/zstd/fse.h

@@ -1,704 +0,0 @@
-/* ******************************************************************
-   FSE : Finite State Entropy codec
-   Public Prototypes declaration
-   Copyright (C) 2013-2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
-****************************************************************** */
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-#ifndef FSE_H
-#define FSE_H
-
-
-/*-*****************************************
-*  Dependencies
-******************************************/
-#include <stddef.h>    /* size_t, ptrdiff_t */
-
-
-/*-*****************************************
-*  FSE_PUBLIC_API : control library symbols visibility
-******************************************/
-#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
-#  define FSE_PUBLIC_API __attribute__ ((visibility ("default")))
-#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1)   /* Visual expected */
-#  define FSE_PUBLIC_API __declspec(dllexport)
-#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
-#  define FSE_PUBLIC_API __declspec(dllimport) /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
-#else
-#  define FSE_PUBLIC_API
-#endif
-
-/*------   Version   ------*/
-#define FSE_VERSION_MAJOR    0
-#define FSE_VERSION_MINOR    9
-#define FSE_VERSION_RELEASE  0
-
-#define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE
-#define FSE_QUOTE(str) #str
-#define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str)
-#define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION)
-
-#define FSE_VERSION_NUMBER  (FSE_VERSION_MAJOR *100*100 + FSE_VERSION_MINOR *100 + FSE_VERSION_RELEASE)
-FSE_PUBLIC_API unsigned FSE_versionNumber(void);   /**< library version number; to be used when checking dll version */
-
-/*-****************************************
-*  FSE simple functions
-******************************************/
-/*! FSE_compress() :
-    Compress content of buffer 'src', of size 'srcSize', into destination buffer 'dst'.
-    'dst' buffer must be already allocated. Compression runs faster is dstCapacity >= FSE_compressBound(srcSize).
-    @return : size of compressed data (<= dstCapacity).
-    Special values : if return == 0, srcData is not compressible => Nothing is stored within dst !!!
-                     if return == 1, srcData is a single byte symbol * srcSize times. Use RLE compression instead.
-                     if FSE_isError(return), compression failed (more details using FSE_getErrorName())
-*/
-FSE_PUBLIC_API size_t FSE_compress(void* dst, size_t dstCapacity,
-                             const void* src, size_t srcSize);
-
-/*! FSE_decompress():
-    Decompress FSE data from buffer 'cSrc', of size 'cSrcSize',
-    into already allocated destination buffer 'dst', of size 'dstCapacity'.
-    @return : size of regenerated data (<= maxDstSize),
-              or an error code, which can be tested using FSE_isError() .
-
-    ** Important ** : FSE_decompress() does not decompress non-compressible nor RLE data !!!
-    Why ? : making this distinction requires a header.
-    Header management is intentionally delegated to the user layer, which can better manage special cases.
-*/
-FSE_PUBLIC_API size_t FSE_decompress(void* dst,  size_t dstCapacity,
-                               const void* cSrc, size_t cSrcSize);
-
-
-/*-*****************************************
-*  Tool functions
-******************************************/
-FSE_PUBLIC_API size_t FSE_compressBound(size_t size);       /* maximum compressed size */
-
-/* Error Management */
-FSE_PUBLIC_API unsigned    FSE_isError(size_t code);        /* tells if a return value is an error code */
-FSE_PUBLIC_API const char* FSE_getErrorName(size_t code);   /* provides error code string (useful for debugging) */
-
-
-/*-*****************************************
-*  FSE advanced functions
-******************************************/
-/*! FSE_compress2() :
-    Same as FSE_compress(), but allows the selection of 'maxSymbolValue' and 'tableLog'
-    Both parameters can be defined as '0' to mean : use default value
-    @return : size of compressed data
-    Special values : if return == 0, srcData is not compressible => Nothing is stored within cSrc !!!
-                     if return == 1, srcData is a single byte symbol * srcSize times. Use RLE compression.
-                     if FSE_isError(return), it's an error code.
-*/
-FSE_PUBLIC_API size_t FSE_compress2 (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);
-
-
-/*-*****************************************
-*  FSE detailed API
-******************************************/
-/*!
-FSE_compress() does the following:
-1. count symbol occurrence from source[] into table count[]
-2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog)
-3. save normalized counters to memory buffer using writeNCount()
-4. build encoding table 'CTable' from normalized counters
-5. encode the data stream using encoding table 'CTable'
-
-FSE_decompress() does the following:
-1. read normalized counters with readNCount()
-2. build decoding table 'DTable' from normalized counters
-3. decode the data stream using decoding table 'DTable'
-
-The following API allows targeting specific sub-functions for advanced tasks.
-For example, it's possible to compress several blocks using the same 'CTable',
-or to save and provide normalized distribution using external method.
-*/
-
-/* *** COMPRESSION *** */
-
-/*! FSE_count():
-    Provides the precise count of each byte within a table 'count'.
-    'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1).
-    *maxSymbolValuePtr will be updated if detected smaller than initial value.
-    @return : the count of the most frequent symbol (which is not identified).
-              if return == srcSize, there is only one symbol.
-              Can also return an error code, which can be tested with FSE_isError(). */
-FSE_PUBLIC_API size_t FSE_count(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize);
-
-/*! FSE_optimalTableLog():
-    dynamically downsize 'tableLog' when conditions are met.
-    It saves CPU time, by using smaller tables, while preserving or even improving compression ratio.
-    @return : recommended tableLog (necessarily <= 'maxTableLog') */
-FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
-
-/*! FSE_normalizeCount():
-    normalize counts so that sum(count[]) == Power_of_2 (2^tableLog)
-    'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1).
-    @return : tableLog,
-              or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_normalizeCount(short* normalizedCounter, unsigned tableLog, const unsigned* count, size_t srcSize, unsigned maxSymbolValue);
-
-/*! FSE_NCountWriteBound():
-    Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'.
-    Typically useful for allocation purpose. */
-FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog);
-
-/*! FSE_writeNCount():
-    Compactly save 'normalizedCounter' into 'buffer'.
-    @return : size of the compressed table,
-              or an errorCode, which can be tested using FSE_isError(). */
-FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
-
-
-/*! Constructor and Destructor of FSE_CTable.
-    Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
-typedef unsigned FSE_CTable;   /* don't allocate that. It's only meant to be more restrictive than void* */
-FSE_PUBLIC_API FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog);
-FSE_PUBLIC_API void        FSE_freeCTable (FSE_CTable* ct);
-
-/*! FSE_buildCTable():
-    Builds `ct`, which must be already allocated, using FSE_createCTable().
-    @return : 0, or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
-
-/*! FSE_compress_usingCTable():
-    Compress `src` using `ct` into `dst` which must be already allocated.
-    @return : size of compressed data (<= `dstCapacity`),
-              or 0 if compressed data could not fit into `dst`,
-              or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_compress_usingCTable (void* dst, size_t dstCapacity, const void* src, size_t srcSize, const FSE_CTable* ct);
-
-/*!
-Tutorial :
-----------
-The first step is to count all symbols. FSE_count() does this job very fast.
-Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells.
-'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0]
-maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value)
-FSE_count() will return the number of occurrence of the most frequent symbol.
-This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
-
-The next step is to normalize the frequencies.
-FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'.
-It also guarantees a minimum of 1 to any Symbol with frequency >= 1.
-You can use 'tableLog'==0 to mean "use default tableLog value".
-If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(),
-which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default").
-
-The result of FSE_normalizeCount() will be saved into a table,
-called 'normalizedCounter', which is a table of signed short.
-'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells.
-The return value is tableLog if everything proceeded as expected.
-It is 0 if there is a single symbol within distribution.
-If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()).
-
-'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount().
-'buffer' must be already allocated.
-For guaranteed success, buffer size must be at least FSE_headerBound().
-The result of the function is the number of bytes written into 'buffer'.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small).
-
-'normalizedCounter' can then be used to create the compression table 'CTable'.
-The space required by 'CTable' must be already allocated, using FSE_createCTable().
-You can then use FSE_buildCTable() to fill 'CTable'.
-If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()).
-
-'CTable' can then be used to compress 'src', with FSE_compress_usingCTable().
-Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize'
-The function returns the size of compressed data (without header), necessarily <= `dstCapacity`.
-If it returns '0', compressed data could not fit into 'dst'.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
-*/
-
-
-/* *** DECOMPRESSION *** */
-
-/*! FSE_readNCount():
-    Read compactly saved 'normalizedCounter' from 'rBuffer'.
-    @return : size read from 'rBuffer',
-              or an errorCode, which can be tested using FSE_isError().
-              maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */
-FSE_PUBLIC_API size_t FSE_readNCount (short* normalizedCounter, unsigned* maxSymbolValuePtr, unsigned* tableLogPtr, const void* rBuffer, size_t rBuffSize);
-
-/*! Constructor and Destructor of FSE_DTable.
-    Note that its size depends on 'tableLog' */
-typedef unsigned FSE_DTable;   /* don't allocate that. It's just a way to be more restrictive than void* */
-FSE_PUBLIC_API FSE_DTable* FSE_createDTable(unsigned tableLog);
-FSE_PUBLIC_API void        FSE_freeDTable(FSE_DTable* dt);
-
-/*! FSE_buildDTable():
-    Builds 'dt', which must be already allocated, using FSE_createDTable().
-    return : 0, or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_buildDTable (FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
-
-/*! FSE_decompress_usingDTable():
-    Decompress compressed source `cSrc` of size `cSrcSize` using `dt`
-    into `dst` which must be already allocated.
-    @return : size of regenerated data (necessarily <= `dstCapacity`),
-              or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_decompress_usingDTable(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, const FSE_DTable* dt);
-
-/*!
-Tutorial :
-----------
-(Note : these functions only decompress FSE-compressed blocks.
- If block is uncompressed, use memcpy() instead
- If block is a single repeated byte, use memset() instead )
-
-The first step is to obtain the normalized frequencies of symbols.
-This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount().
-'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short.
-In practice, that means it's necessary to know 'maxSymbolValue' beforehand,
-or size the table to handle worst case situations (typically 256).
-FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'.
-The result of FSE_readNCount() is the number of bytes read from 'rBuffer'.
-Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that.
-If there is an error, the function will return an error code, which can be tested using FSE_isError().
-
-The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'.
-This is performed by the function FSE_buildDTable().
-The space required by 'FSE_DTable' must be already allocated using FSE_createDTable().
-If there is an error, the function will return an error code, which can be tested using FSE_isError().
-
-`FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable().
-`cSrcSize` must be strictly correct, otherwise decompression will fail.
-FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`).
-If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small)
-*/
-
-#endif  /* FSE_H */
-
-#if defined(FSE_STATIC_LINKING_ONLY) && !defined(FSE_H_FSE_STATIC_LINKING_ONLY)
-#define FSE_H_FSE_STATIC_LINKING_ONLY
-
-/* *** Dependency *** */
-#include "bitstream.h"
-
-
-/* *****************************************
-*  Static allocation
-*******************************************/
-/* FSE buffer bounds */
-#define FSE_NCOUNTBOUND 512
-#define FSE_BLOCKBOUND(size) (size + (size>>7))
-#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size))   /* Macro version, useful for static allocation */
-
-/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
-#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue)   (1 + (1<<(maxTableLog-1)) + ((maxSymbolValue+1)*2))
-#define FSE_DTABLE_SIZE_U32(maxTableLog)                   (1 + (1<<maxTableLog))
-
-/* or use the size to malloc() space directly. Pay attention to alignment restrictions though */
-#define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue)   (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable))
-#define FSE_DTABLE_SIZE(maxTableLog)                   (FSE_DTABLE_SIZE_U32(maxTableLog) * sizeof(FSE_DTable))
-
-
-/* *****************************************
-*  FSE advanced API
-*******************************************/
-/* FSE_count_wksp() :
- * Same as FSE_count(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= `1024` unsigned
- */
-size_t FSE_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                 const void* source, size_t sourceSize, unsigned* workSpace);
-
-/** FSE_countFast() :
- *  same as FSE_count(), but blindly trusts that all byte values within src are <= *maxSymbolValuePtr
- */
-size_t FSE_countFast(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize);
-
-/* FSE_countFast_wksp() :
- * Same as FSE_countFast(), but using an externally provided scratch buffer.
- * `workSpace` must be a table of minimum `1024` unsigned
- */
-size_t FSE_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned* workSpace);
-
-/*! FSE_count_simple() :
- * Same as FSE_countFast(), but does not use any additional memory (not even on stack).
- * This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr` (presuming it's also the size of `count`).
-*/
-size_t FSE_count_simple(unsigned* count, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize);
-
-
-
-unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus);
-/**< same as FSE_optimalTableLog(), which used `minus==2` */
-
-/* FSE_compress_wksp() :
- * Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`).
- * FSE_WKSP_SIZE_U32() provides the minimum size required for `workSpace` as a table of FSE_CTable.
- */
-#define FSE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue)   ( FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) + ((maxTableLog > 12) ? (1 << (maxTableLog - 2)) : 1024) )
-size_t FSE_compress_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
-
-size_t FSE_buildCTable_raw (FSE_CTable* ct, unsigned nbBits);
-/**< build a fake FSE_CTable, designed for a flat distribution, where each symbol uses nbBits */
-
-size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue);
-/**< build a fake FSE_CTable, designed to compress always the same symbolValue */
-
-/* FSE_buildCTable_wksp() :
- * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
- * `wkspSize` must be >= `(1<<tableLog)`.
- */
-size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
-
-size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits);
-/**< build a fake FSE_DTable, designed to read a flat distribution where each symbol uses nbBits */
-
-size_t FSE_buildDTable_rle (FSE_DTable* dt, unsigned char symbolValue);
-/**< build a fake FSE_DTable, designed to always generate the same symbolValue */
-
-size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, FSE_DTable* workSpace, unsigned maxLog);
-/**< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DTABLE_SIZE_U32(maxLog)` */
-
-typedef enum {
-   FSE_repeat_none,  /**< Cannot use the previous table */
-   FSE_repeat_check, /**< Can use the previous table but it must be checked */
-   FSE_repeat_valid  /**< Can use the previous table and it is asumed to be valid */
- } FSE_repeat;
-
-/* *****************************************
-*  FSE symbol compression API
-*******************************************/
-/*!
-   This API consists of small unitary functions, which highly benefit from being inlined.
-   Hence their body are included in next section.
-*/
-typedef struct {
-    ptrdiff_t   value;
-    const void* stateTable;
-    const void* symbolTT;
-    unsigned    stateLog;
-} FSE_CState_t;
-
-static void FSE_initCState(FSE_CState_t* CStatePtr, const FSE_CTable* ct);
-
-static void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* CStatePtr, unsigned symbol);
-
-static void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* CStatePtr);
-
-/**<
-These functions are inner components of FSE_compress_usingCTable().
-They allow the creation of custom streams, mixing multiple tables and bit sources.
-
-A key property to keep in mind is that encoding and decoding are done **in reverse direction**.
-So the first symbol you will encode is the last you will decode, like a LIFO stack.
-
-You will need a few variables to track your CStream. They are :
-
-FSE_CTable    ct;         // Provided by FSE_buildCTable()
-BIT_CStream_t bitStream;  // bitStream tracking structure
-FSE_CState_t  state;      // State tracking structure (can have several)
-
-
-The first thing to do is to init bitStream and state.
-    size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize);
-    FSE_initCState(&state, ct);
-
-Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError();
-You can then encode your input data, byte after byte.
-FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time.
-Remember decoding will be done in reverse direction.
-    FSE_encodeByte(&bitStream, &state, symbol);
-
-At any time, you can also add any bit sequence.
-Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders
-    BIT_addBits(&bitStream, bitField, nbBits);
-
-The above methods don't commit data to memory, they just store it into local register, for speed.
-Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
-Writing data to memory is a manual operation, performed by the flushBits function.
-    BIT_flushBits(&bitStream);
-
-Your last FSE encoding operation shall be to flush your last state value(s).
-    FSE_flushState(&bitStream, &state);
-
-Finally, you must close the bitStream.
-The function returns the size of CStream in bytes.
-If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible)
-If there is an error, it returns an errorCode (which can be tested using FSE_isError()).
-    size_t size = BIT_closeCStream(&bitStream);
-*/
-
-
-/* *****************************************
-*  FSE symbol decompression API
-*******************************************/
-typedef struct {
-    size_t      state;
-    const void* table;   /* precise table may vary, depending on U16 */
-} FSE_DState_t;
-
-
-static void     FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt);
-
-static unsigned char FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
-
-static unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr);
-
-/**<
-Let's now decompose FSE_decompress_usingDTable() into its unitary components.
-You will decode FSE-encoded symbols from the bitStream,
-and also any other bitFields you put in, **in reverse order**.
-
-You will need a few variables to track your bitStream. They are :
-
-BIT_DStream_t DStream;    // Stream context
-FSE_DState_t  DState;     // State context. Multiple ones are possible
-FSE_DTable*   DTablePtr;  // Decoding table, provided by FSE_buildDTable()
-
-The first thing to do is to init the bitStream.
-    errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize);
-
-You should then retrieve your initial state(s)
-(in reverse flushing order if you have several ones) :
-    errorCode = FSE_initDState(&DState, &DStream, DTablePtr);
-
-You can then decode your data, symbol after symbol.
-For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'.
-Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out).
-    unsigned char symbol = FSE_decodeSymbol(&DState, &DStream);
-
-You can retrieve any bitfield you eventually stored into the bitStream (in reverse order)
-Note : maximum allowed nbBits is 25, for 32-bits compatibility
-    size_t bitField = BIT_readBits(&DStream, nbBits);
-
-All above operations only read from local register (which size depends on size_t).
-Refueling the register from memory is manually performed by the reload method.
-    endSignal = FSE_reloadDStream(&DStream);
-
-BIT_reloadDStream() result tells if there is still some more data to read from DStream.
-BIT_DStream_unfinished : there is still some data left into the DStream.
-BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled.
-BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed.
-BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted.
-
-When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop,
-to properly detect the exact end of stream.
-After each decoded symbol, check if DStream is fully consumed using this simple test :
-    BIT_reloadDStream(&DStream) >= BIT_DStream_completed
-
-When it's done, verify decompression is fully completed, by checking both DStream and the relevant states.
-Checking if DStream has reached its end is performed by :
-    BIT_endOfDStream(&DStream);
-Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible.
-    FSE_endOfDState(&DState);
-*/
-
-
-/* *****************************************
-*  FSE unsafe API
-*******************************************/
-static unsigned char FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
-/* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */
-
-
-/* *****************************************
-*  Implementation of inlined functions
-*******************************************/
-typedef struct {
-    int deltaFindState;
-    U32 deltaNbBits;
-} FSE_symbolCompressionTransform; /* total 8 bytes */
-
-MEM_STATIC void FSE_initCState(FSE_CState_t* statePtr, const FSE_CTable* ct)
-{
-    const void* ptr = ct;
-    const U16* u16ptr = (const U16*) ptr;
-    const U32 tableLog = MEM_read16(ptr);
-    statePtr->value = (ptrdiff_t)1<<tableLog;
-    statePtr->stateTable = u16ptr+2;
-    statePtr->symbolTT = ((const U32*)ct + 1 + (tableLog ? (1<<(tableLog-1)) : 1));
-    statePtr->stateLog = tableLog;
-}
-
-
-/*! FSE_initCState2() :
-*   Same as FSE_initCState(), but the first symbol to include (which will be the last to be read)
-*   uses the smallest state value possible, saving the cost of this symbol */
-MEM_STATIC void FSE_initCState2(FSE_CState_t* statePtr, const FSE_CTable* ct, U32 symbol)
-{
-    FSE_initCState(statePtr, ct);
-    {   const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
-        const U16* stateTable = (const U16*)(statePtr->stateTable);
-        U32 nbBitsOut  = (U32)((symbolTT.deltaNbBits + (1<<15)) >> 16);
-        statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits;
-        statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
-    }
-}
-
-MEM_STATIC void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* statePtr, U32 symbol)
-{
-    FSE_symbolCompressionTransform const symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
-    const U16* const stateTable = (const U16*)(statePtr->stateTable);
-    U32 const nbBitsOut  = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16);
-    BIT_addBits(bitC, statePtr->value, nbBitsOut);
-    statePtr->value = stateTable[ (statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
-}
-
-MEM_STATIC void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* statePtr)
-{
-    BIT_addBits(bitC, statePtr->value, statePtr->stateLog);
-    BIT_flushBits(bitC);
-}
-
-
-/* ======    Decompression    ====== */
-
-typedef struct {
-    U16 tableLog;
-    U16 fastMode;
-} FSE_DTableHeader;   /* sizeof U32 */
-
-typedef struct
-{
-    unsigned short newState;
-    unsigned char  symbol;
-    unsigned char  nbBits;
-} FSE_decode_t;   /* size == U32 */
-
-MEM_STATIC void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt)
-{
-    const void* ptr = dt;
-    const FSE_DTableHeader* const DTableH = (const FSE_DTableHeader*)ptr;
-    DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
-    BIT_reloadDStream(bitD);
-    DStatePtr->table = dt + 1;
-}
-
-MEM_STATIC BYTE FSE_peekSymbol(const FSE_DState_t* DStatePtr)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    return DInfo.symbol;
-}
-
-MEM_STATIC void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    U32 const nbBits = DInfo.nbBits;
-    size_t const lowBits = BIT_readBits(bitD, nbBits);
-    DStatePtr->state = DInfo.newState + lowBits;
-}
-
-MEM_STATIC BYTE FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    U32 const nbBits = DInfo.nbBits;
-    BYTE const symbol = DInfo.symbol;
-    size_t const lowBits = BIT_readBits(bitD, nbBits);
-
-    DStatePtr->state = DInfo.newState + lowBits;
-    return symbol;
-}
-
-/*! FSE_decodeSymbolFast() :
-    unsafe, only works if no symbol has a probability > 50% */
-MEM_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    U32 const nbBits = DInfo.nbBits;
-    BYTE const symbol = DInfo.symbol;
-    size_t const lowBits = BIT_readBitsFast(bitD, nbBits);
-
-    DStatePtr->state = DInfo.newState + lowBits;
-    return symbol;
-}
-
-MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr)
-{
-    return DStatePtr->state == 0;
-}
-
-
-
-#ifndef FSE_COMMONDEFS_ONLY
-
-/* **************************************************************
-*  Tuning parameters
-****************************************************************/
-/*!MEMORY_USAGE :
-*  Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
-*  Increasing memory usage improves compression ratio
-*  Reduced memory usage can improve speed, due to cache effect
-*  Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */
-#ifndef FSE_MAX_MEMORY_USAGE
-#  define FSE_MAX_MEMORY_USAGE 14
-#endif
-#ifndef FSE_DEFAULT_MEMORY_USAGE
-#  define FSE_DEFAULT_MEMORY_USAGE 13
-#endif
-
-/*!FSE_MAX_SYMBOL_VALUE :
-*  Maximum symbol value authorized.
-*  Required for proper stack allocation */
-#ifndef FSE_MAX_SYMBOL_VALUE
-#  define FSE_MAX_SYMBOL_VALUE 255
-#endif
-
-/* **************************************************************
-*  template functions type & suffix
-****************************************************************/
-#define FSE_FUNCTION_TYPE BYTE
-#define FSE_FUNCTION_EXTENSION
-#define FSE_DECODE_TYPE FSE_decode_t
-
-
-#endif   /* !FSE_COMMONDEFS_ONLY */
-
-
-/* ***************************************************************
-*  Constants
-*****************************************************************/
-#define FSE_MAX_TABLELOG  (FSE_MAX_MEMORY_USAGE-2)
-#define FSE_MAX_TABLESIZE (1U<<FSE_MAX_TABLELOG)
-#define FSE_MAXTABLESIZE_MASK (FSE_MAX_TABLESIZE-1)
-#define FSE_DEFAULT_TABLELOG (FSE_DEFAULT_MEMORY_USAGE-2)
-#define FSE_MIN_TABLELOG 5
-
-#define FSE_TABLELOG_ABSOLUTE_MAX 15
-#if FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX
-#  error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported"
-#endif
-
-#define FSE_TABLESTEP(tableSize) ((tableSize>>1) + (tableSize>>3) + 3)
-
-
-#endif /* FSE_STATIC_LINKING_ONLY */
-
-
-#if defined (__cplusplus)
-}
-#endif

third/github.com/DataDog/zstd/fse_compress.c

@@ -1,849 +0,0 @@
-/* ******************************************************************
-   FSE : Finite State Entropy encoder
-   Copyright (C) 2013-2015, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
-****************************************************************** */
-
-/* **************************************************************
-*  Includes
-****************************************************************/
-#include <stdlib.h>     /* malloc, free, qsort */
-#include <string.h>     /* memcpy, memset */
-#include <stdio.h>      /* printf (debug) */
-#include "bitstream.h"
-#include "compiler.h"
-#define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
-#include "error_private.h"
-
-
-/* **************************************************************
-*  Error Management
-****************************************************************/
-#define FSE_isError ERR_isError
-#define FSE_STATIC_ASSERT(c) { enum { FSE_static_assert = 1/(int)(!!(c)) }; }   /* use only *after* variable declarations */
-
-
-/* **************************************************************
-*  Templates
-****************************************************************/
-/*
-  designed to be included
-  for type-specific functions (template emulation in C)
-  Objective is to write these functions only once, for improved maintenance
-*/
-
-/* safety checks */
-#ifndef FSE_FUNCTION_EXTENSION
-#  error "FSE_FUNCTION_EXTENSION must be defined"
-#endif
-#ifndef FSE_FUNCTION_TYPE
-#  error "FSE_FUNCTION_TYPE must be defined"
-#endif
-
-/* Function names */
-#define FSE_CAT(X,Y) X##Y
-#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
-#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
-
-
-/* Function templates */
-
-/* FSE_buildCTable_wksp() :
- * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
- * wkspSize should be sized to handle worst case situation, which is `1<<max_tableLog * sizeof(FSE_FUNCTION_TYPE)`
- * workSpace must also be properly aligned with FSE_FUNCTION_TYPE requirements
- */
-size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
-{
-    U32 const tableSize = 1 << tableLog;
-    U32 const tableMask = tableSize - 1;
-    void* const ptr = ct;
-    U16* const tableU16 = ( (U16*) ptr) + 2;
-    void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableLog ? tableSize>>1 : 1) ;
-    FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
-    U32 const step = FSE_TABLESTEP(tableSize);
-    U32 cumul[FSE_MAX_SYMBOL_VALUE+2];
-
-    FSE_FUNCTION_TYPE* const tableSymbol = (FSE_FUNCTION_TYPE*)workSpace;
-    U32 highThreshold = tableSize-1;
-
-    /* CTable header */
-    if (((size_t)1 << tableLog) * sizeof(FSE_FUNCTION_TYPE) > wkspSize) return ERROR(tableLog_tooLarge);
-    tableU16[-2] = (U16) tableLog;
-    tableU16[-1] = (U16) maxSymbolValue;
-
-    /* For explanations on how to distribute symbol values over the table :
-    *  http://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */
-
-    /* symbol start positions */
-    {   U32 u;
-        cumul[0] = 0;
-        for (u=1; u<=maxSymbolValue+1; u++) {
-            if (normalizedCounter[u-1]==-1) {  /* Low proba symbol */
-                cumul[u] = cumul[u-1] + 1;
-                tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1);
-            } else {
-                cumul[u] = cumul[u-1] + normalizedCounter[u-1];
-        }   }
-        cumul[maxSymbolValue+1] = tableSize+1;
-    }
-
-    /* Spread symbols */
-    {   U32 position = 0;
-        U32 symbol;
-        for (symbol=0; symbol<=maxSymbolValue; symbol++) {
-            int nbOccurences;
-            for (nbOccurences=0; nbOccurences<normalizedCounter[symbol]; nbOccurences++) {
-                tableSymbol[position] = (FSE_FUNCTION_TYPE)symbol;
-                position = (position + step) & tableMask;
-                while (position > highThreshold) position = (position + step) & tableMask;   /* Low proba area */
-        }   }
-
-        if (position!=0) return ERROR(GENERIC);   /* Must have gone through all positions */
-    }
-
-    /* Build table */
-    {   U32 u; for (u=0; u<tableSize; u++) {
-        FSE_FUNCTION_TYPE s = tableSymbol[u];   /* note : static analyzer may not understand tableSymbol is properly initialized */
-        tableU16[cumul[s]++] = (U16) (tableSize+u);   /* TableU16 : sorted by symbol order; gives next state value */
-    }   }
-
-    /* Build Symbol Transformation Table */
-    {   unsigned total = 0;
-        unsigned s;
-        for (s=0; s<=maxSymbolValue; s++) {
-            switch (normalizedCounter[s])
-            {
-            case  0: break;
-
-            case -1:
-            case  1:
-                symbolTT[s].deltaNbBits = (tableLog << 16) - (1<<tableLog);
-                symbolTT[s].deltaFindState = total - 1;
-                total ++;
-                break;
-            default :
-                {
-                    U32 const maxBitsOut = tableLog - BIT_highbit32 (normalizedCounter[s]-1);
-                    U32 const minStatePlus = normalizedCounter[s] << maxBitsOut;
-                    symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
-                    symbolTT[s].deltaFindState = total - normalizedCounter[s];
-                    total +=  normalizedCounter[s];
-    }   }   }   }
-
-    return 0;
-}
-
-
-size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
-{
-    FSE_FUNCTION_TYPE tableSymbol[FSE_MAX_TABLESIZE];   /* memset() is not necessary, even if static analyzer complain about it */
-    return FSE_buildCTable_wksp(ct, normalizedCounter, maxSymbolValue, tableLog, tableSymbol, sizeof(tableSymbol));
-}
-
-
-
-#ifndef FSE_COMMONDEFS_ONLY
-
-/*-**************************************************************
-*  FSE NCount encoding-decoding
-****************************************************************/
-size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
-{
-    size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog) >> 3) + 3;
-    return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND;  /* maxSymbolValue==0 ? use default */
-}
-
-static size_t FSE_writeNCount_generic (void* header, size_t headerBufferSize,
-                                       const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
-                                       unsigned writeIsSafe)
-{
-    BYTE* const ostart = (BYTE*) header;
-    BYTE* out = ostart;
-    BYTE* const oend = ostart + headerBufferSize;
-    int nbBits;
-    const int tableSize = 1 << tableLog;
-    int remaining;
-    int threshold;
-    U32 bitStream;
-    int bitCount;
-    unsigned charnum = 0;
-    int previous0 = 0;
-
-    bitStream = 0;
-    bitCount  = 0;
-    /* Table Size */
-    bitStream += (tableLog-FSE_MIN_TABLELOG) << bitCount;
-    bitCount  += 4;
-
-    /* Init */
-    remaining = tableSize+1;   /* +1 for extra accuracy */
-    threshold = tableSize;
-    nbBits = tableLog+1;
-
-    while (remaining>1) {  /* stops at 1 */
-        if (previous0) {
-            unsigned start = charnum;
-            while (!normalizedCounter[charnum]) charnum++;
-            while (charnum >= start+24) {
-                start+=24;
-                bitStream += 0xFFFFU << bitCount;
-                if ((!writeIsSafe) && (out > oend-2)) return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-                out[0] = (BYTE) bitStream;
-                out[1] = (BYTE)(bitStream>>8);
-                out+=2;
-                bitStream>>=16;
-            }
-            while (charnum >= start+3) {
-                start+=3;
-                bitStream += 3 << bitCount;
-                bitCount += 2;
-            }
-            bitStream += (charnum-start) << bitCount;
-            bitCount += 2;
-            if (bitCount>16) {
-                if ((!writeIsSafe) && (out > oend - 2)) return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-                out[0] = (BYTE)bitStream;
-                out[1] = (BYTE)(bitStream>>8);
-                out += 2;
-                bitStream >>= 16;
-                bitCount -= 16;
-        }   }
-        {   int count = normalizedCounter[charnum++];
-            int const max = (2*threshold-1)-remaining;
-            remaining -= count < 0 ? -count : count;
-            count++;   /* +1 for extra accuracy */
-            if (count>=threshold) count += max;   /* [0..max[ [max..threshold[ (...) [threshold+max 2*threshold[ */
-            bitStream += count << bitCount;
-            bitCount  += nbBits;
-            bitCount  -= (count<max);
-            previous0  = (count==1);
-            if (remaining<1) return ERROR(GENERIC);
-            while (remaining<threshold) { nbBits--; threshold>>=1; }
-        }
-        if (bitCount>16) {
-            if ((!writeIsSafe) && (out > oend - 2)) return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-            out[0] = (BYTE)bitStream;
-            out[1] = (BYTE)(bitStream>>8);
-            out += 2;
-            bitStream >>= 16;
-            bitCount -= 16;
-    }   }
-
-    /* flush remaining bitStream */
-    if ((!writeIsSafe) && (out > oend - 2)) return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-    out[0] = (BYTE)bitStream;
-    out[1] = (BYTE)(bitStream>>8);
-    out+= (bitCount+7) /8;
-
-    if (charnum > maxSymbolValue + 1) return ERROR(GENERIC);
-
-    return (out-ostart);
-}
-
-
-size_t FSE_writeNCount (void* buffer, size_t bufferSize, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
-{
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);   /* Unsupported */
-    if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC);   /* Unsupported */
-
-    if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog))
-        return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0);
-
-    return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1);
-}
-
-
-
-/*-**************************************************************
-*  Counting histogram
-****************************************************************/
-/*! FSE_count_simple
-    This function counts byte values within `src`, and store the histogram into table `count`.
-    It doesn't use any additional memory.
-    But this function is unsafe : it doesn't check that all values within `src` can fit into `count`.
-    For this reason, prefer using a table `count` with 256 elements.
-    @return : count of most numerous element.
-*/
-size_t FSE_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
-                        const void* src, size_t srcSize)
-{
-    const BYTE* ip = (const BYTE*)src;
-    const BYTE* const end = ip + srcSize;
-    unsigned maxSymbolValue = *maxSymbolValuePtr;
-    unsigned max=0;
-
-    memset(count, 0, (maxSymbolValue+1)*sizeof(*count));
-    if (srcSize==0) { *maxSymbolValuePtr = 0; return 0; }
-
-    while (ip<end) {
-        assert(*ip <= maxSymbolValue);
-        count[*ip++]++;
-    }
-
-    while (!count[maxSymbolValue]) maxSymbolValue--;
-    *maxSymbolValuePtr = maxSymbolValue;
-
-    { U32 s; for (s=0; s<=maxSymbolValue; s++) if (count[s] > max) max = count[s]; }
-
-    return (size_t)max;
-}
-
-
-/* FSE_count_parallel_wksp() :
- * Same as FSE_count_parallel(), but using an externally provided scratch buffer.
- * `workSpace` size must be a minimum of `1024 * sizeof(unsigned)`.
- * @return : largest histogram frequency, or an error code (notably when histogram would be larger than *maxSymbolValuePtr). */
-static size_t FSE_count_parallel_wksp(
-                                unsigned* count, unsigned* maxSymbolValuePtr,
-                                const void* source, size_t sourceSize,
-                                unsigned checkMax, unsigned* const workSpace)
-{
-    const BYTE* ip = (const BYTE*)source;
-    const BYTE* const iend = ip+sourceSize;
-    unsigned maxSymbolValue = *maxSymbolValuePtr;
-    unsigned max=0;
-    U32* const Counting1 = workSpace;
-    U32* const Counting2 = Counting1 + 256;
-    U32* const Counting3 = Counting2 + 256;
-    U32* const Counting4 = Counting3 + 256;
-
-    memset(workSpace, 0, 4*256*sizeof(unsigned));
-
-    /* safety checks */
-    if (!sourceSize) {
-        memset(count, 0, maxSymbolValue + 1);
-        *maxSymbolValuePtr = 0;
-        return 0;
-    }
-    if (!maxSymbolValue) maxSymbolValue = 255;            /* 0 == default */
-
-    /* by stripes of 16 bytes */
-    {   U32 cached = MEM_read32(ip); ip += 4;
-        while (ip < iend-15) {
-            U32 c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-            c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-            c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-            c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-        }
-        ip-=4;
-    }
-
-    /* finish last symbols */
-    while (ip<iend) Counting1[*ip++]++;
-
-    if (checkMax) {   /* verify stats will fit into destination table */
-        U32 s; for (s=255; s>maxSymbolValue; s--) {
-            Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s];
-            if (Counting1[s]) return ERROR(maxSymbolValue_tooSmall);
-    }   }
-
-    {   U32 s;
-        if (maxSymbolValue > 255) maxSymbolValue = 255;
-        for (s=0; s<=maxSymbolValue; s++) {
-            count[s] = Counting1[s] + Counting2[s] + Counting3[s] + Counting4[s];
-            if (count[s] > max) max = count[s];
-    }   }
-
-    while (!count[maxSymbolValue]) maxSymbolValue--;
-    *maxSymbolValuePtr = maxSymbolValue;
-    return (size_t)max;
-}
-
-/* FSE_countFast_wksp() :
- * Same as FSE_countFast(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= `1024` unsigned */
-size_t FSE_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                          const void* source, size_t sourceSize,
-                          unsigned* workSpace)
-{
-    if (sourceSize < 1500) /* heuristic threshold */
-        return FSE_count_simple(count, maxSymbolValuePtr, source, sourceSize);
-    return FSE_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, 0, workSpace);
-}
-
-/* fast variant (unsafe : won't check if src contains values beyond count[] limit) */
-size_t FSE_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
-                     const void* source, size_t sourceSize)
-{
-    unsigned tmpCounters[1024];
-    return FSE_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, tmpCounters);
-}
-
-/* FSE_count_wksp() :
- * Same as FSE_count(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= `1024` unsigned */
-size_t FSE_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                 const void* source, size_t sourceSize, unsigned* workSpace)
-{
-    if (*maxSymbolValuePtr < 255)
-        return FSE_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, 1, workSpace);
-    *maxSymbolValuePtr = 255;
-    return FSE_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace);
-}
-
-size_t FSE_count(unsigned* count, unsigned* maxSymbolValuePtr,
-                 const void* src, size_t srcSize)
-{
-    unsigned tmpCounters[1024];
-    return FSE_count_wksp(count, maxSymbolValuePtr, src, srcSize, tmpCounters);
-}
-
-
-
-/*-**************************************************************
-*  FSE Compression Code
-****************************************************************/
-/*! FSE_sizeof_CTable() :
-    FSE_CTable is a variable size structure which contains :
-    `U16 tableLog;`
-    `U16 maxSymbolValue;`
-    `U16 nextStateNumber[1 << tableLog];`                         // This size is variable
-    `FSE_symbolCompressionTransform symbolTT[maxSymbolValue+1];`  // This size is variable
-Allocation is manual (C standard does not support variable-size structures).
-*/
-size_t FSE_sizeof_CTable (unsigned maxSymbolValue, unsigned tableLog)
-{
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
-    return FSE_CTABLE_SIZE_U32 (tableLog, maxSymbolValue) * sizeof(U32);
-}
-
-FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog)
-{
-    size_t size;
-    if (tableLog > FSE_TABLELOG_ABSOLUTE_MAX) tableLog = FSE_TABLELOG_ABSOLUTE_MAX;
-    size = FSE_CTABLE_SIZE_U32 (tableLog, maxSymbolValue) * sizeof(U32);
-    return (FSE_CTable*)malloc(size);
-}
-
-void FSE_freeCTable (FSE_CTable* ct) { free(ct); }
-
-/* provides the minimum logSize to safely represent a distribution */
-static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
-{
-    U32 minBitsSrc = BIT_highbit32((U32)(srcSize - 1)) + 1;
-    U32 minBitsSymbols = BIT_highbit32(maxSymbolValue) + 2;
-    U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
-    assert(srcSize > 1); /* Not supported, RLE should be used instead */
-    return minBits;
-}
-
-unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus)
-{
-    U32 maxBitsSrc = BIT_highbit32((U32)(srcSize - 1)) - minus;
-    U32 tableLog = maxTableLog;
-    U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
-    assert(srcSize > 1); /* Not supported, RLE should be used instead */
-    if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
-    if (maxBitsSrc < tableLog) tableLog = maxBitsSrc;   /* Accuracy can be reduced */
-    if (minBits > tableLog) tableLog = minBits;   /* Need a minimum to safely represent all symbol values */
-    if (tableLog < FSE_MIN_TABLELOG) tableLog = FSE_MIN_TABLELOG;
-    if (tableLog > FSE_MAX_TABLELOG) tableLog = FSE_MAX_TABLELOG;
-    return tableLog;
-}
-
-unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
-{
-    return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2);
-}
-
-
-/* Secondary normalization method.
-   To be used when primary method fails. */
-
-static size_t FSE_normalizeM2(short* norm, U32 tableLog, const unsigned* count, size_t total, U32 maxSymbolValue)
-{
-    short const NOT_YET_ASSIGNED = -2;
-    U32 s;
-    U32 distributed = 0;
-    U32 ToDistribute;
-
-    /* Init */
-    U32 const lowThreshold = (U32)(total >> tableLog);
-    U32 lowOne = (U32)((total * 3) >> (tableLog + 1));
-
-    for (s=0; s<=maxSymbolValue; s++) {
-        if (count[s] == 0) {
-            norm[s]=0;
-            continue;
-        }
-        if (count[s] <= lowThreshold) {
-            norm[s] = -1;
-            distributed++;
-            total -= count[s];
-            continue;
-        }
-        if (count[s] <= lowOne) {
-            norm[s] = 1;
-            distributed++;
-            total -= count[s];
-            continue;
-        }
-
-        norm[s]=NOT_YET_ASSIGNED;
-    }
-    ToDistribute = (1 << tableLog) - distributed;
-
-    if ((total / ToDistribute) > lowOne) {
-        /* risk of rounding to zero */
-        lowOne = (U32)((total * 3) / (ToDistribute * 2));
-        for (s=0; s<=maxSymbolValue; s++) {
-            if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) {
-                norm[s] = 1;
-                distributed++;
-                total -= count[s];
-                continue;
-        }   }
-        ToDistribute = (1 << tableLog) - distributed;
-    }
-
-    if (distributed == maxSymbolValue+1) {
-        /* all values are pretty poor;
-           probably incompressible data (should have already been detected);
-           find max, then give all remaining points to max */
-        U32 maxV = 0, maxC = 0;
-        for (s=0; s<=maxSymbolValue; s++)
-            if (count[s] > maxC) { maxV=s; maxC=count[s]; }
-        norm[maxV] += (short)ToDistribute;
-        return 0;
-    }
-
-    if (total == 0) {
-        /* all of the symbols were low enough for the lowOne or lowThreshold */
-        for (s=0; ToDistribute > 0; s = (s+1)%(maxSymbolValue+1))
-            if (norm[s] > 0) { ToDistribute--; norm[s]++; }
-        return 0;
-    }
-
-    {   U64 const vStepLog = 62 - tableLog;
-        U64 const mid = (1ULL << (vStepLog-1)) - 1;
-        U64 const rStep = ((((U64)1<<vStepLog) * ToDistribute) + mid) / total;   /* scale on remaining */
-        U64 tmpTotal = mid;
-        for (s=0; s<=maxSymbolValue; s++) {
-            if (norm[s]==NOT_YET_ASSIGNED) {
-                U64 const end = tmpTotal + (count[s] * rStep);
-                U32 const sStart = (U32)(tmpTotal >> vStepLog);
-                U32 const sEnd = (U32)(end >> vStepLog);
-                U32 const weight = sEnd - sStart;
-                if (weight < 1)
-                    return ERROR(GENERIC);
-                norm[s] = (short)weight;
-                tmpTotal = end;
-    }   }   }
-
-    return 0;
-}
-
-
-size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
-                           const unsigned* count, size_t total,
-                           unsigned maxSymbolValue)
-{
-    /* Sanity checks */
-    if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
-    if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC);   /* Unsupported size */
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);   /* Unsupported size */
-    if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC);   /* Too small tableLog, compression potentially impossible */
-
-    {   static U32 const rtbTable[] = {     0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
-        U64 const scale = 62 - tableLog;
-        U64 const step = ((U64)1<<62) / total;   /* <== here, one division ! */
-        U64 const vStep = 1ULL<<(scale-20);
-        int stillToDistribute = 1<<tableLog;
-        unsigned s;
-        unsigned largest=0;
-        short largestP=0;
-        U32 lowThreshold = (U32)(total >> tableLog);
-
-        for (s=0; s<=maxSymbolValue; s++) {
-            if (count[s] == total) return 0;   /* rle special case */
-            if (count[s] == 0) { normalizedCounter[s]=0; continue; }
-            if (count[s] <= lowThreshold) {
-                normalizedCounter[s] = -1;
-                stillToDistribute--;
-            } else {
-                short proba = (short)((count[s]*step) >> scale);
-                if (proba<8) {
-                    U64 restToBeat = vStep * rtbTable[proba];
-                    proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
-                }
-                if (proba > largestP) { largestP=proba; largest=s; }
-                normalizedCounter[s] = proba;
-                stillToDistribute -= proba;
-        }   }
-        if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) {
-            /* corner case, need another normalization method */
-            size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue);
-            if (FSE_isError(errorCode)) return errorCode;
-        }
-        else normalizedCounter[largest] += (short)stillToDistribute;
-    }
-
-#if 0
-    {   /* Print Table (debug) */
-        U32 s;
-        U32 nTotal = 0;
-        for (s=0; s<=maxSymbolValue; s++)
-            printf("%3i: %4i \n", s, normalizedCounter[s]);
-        for (s=0; s<=maxSymbolValue; s++)
-            nTotal += abs(normalizedCounter[s]);
-        if (nTotal != (1U<<tableLog))
-            printf("Warning !!! Total == %u != %u !!!", nTotal, 1U<<tableLog);
-        getchar();
-    }
-#endif
-
-    return tableLog;
-}
-
-
-/* fake FSE_CTable, for raw (uncompressed) input */
-size_t FSE_buildCTable_raw (FSE_CTable* ct, unsigned nbBits)
-{
-    const unsigned tableSize = 1 << nbBits;
-    const unsigned tableMask = tableSize - 1;
-    const unsigned maxSymbolValue = tableMask;
-    void* const ptr = ct;
-    U16* const tableU16 = ( (U16*) ptr) + 2;
-    void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableSize>>1);   /* assumption : tableLog >= 1 */
-    FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
-    unsigned s;
-
-    /* Sanity checks */
-    if (nbBits < 1) return ERROR(GENERIC);             /* min size */
-
-    /* header */
-    tableU16[-2] = (U16) nbBits;
-    tableU16[-1] = (U16) maxSymbolValue;
-
-    /* Build table */
-    for (s=0; s<tableSize; s++)
-        tableU16[s] = (U16)(tableSize + s);
-
-    /* Build Symbol Transformation Table */
-    {   const U32 deltaNbBits = (nbBits << 16) - (1 << nbBits);
-        for (s=0; s<=maxSymbolValue; s++) {
-            symbolTT[s].deltaNbBits = deltaNbBits;
-            symbolTT[s].deltaFindState = s-1;
-    }   }
-
-    return 0;
-}
-
-/* fake FSE_CTable, for rle input (always same symbol) */
-size_t FSE_buildCTable_rle (FSE_CTable* ct, BYTE symbolValue)
-{
-    void* ptr = ct;
-    U16* tableU16 = ( (U16*) ptr) + 2;
-    void* FSCTptr = (U32*)ptr + 2;
-    FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*) FSCTptr;
-
-    /* header */
-    tableU16[-2] = (U16) 0;
-    tableU16[-1] = (U16) symbolValue;
-
-    /* Build table */
-    tableU16[0] = 0;
-    tableU16[1] = 0;   /* just in case */
-
-    /* Build Symbol Transformation Table */
-    symbolTT[symbolValue].deltaNbBits = 0;
-    symbolTT[symbolValue].deltaFindState = 0;
-
-    return 0;
-}
-
-
-static size_t FSE_compress_usingCTable_generic (void* dst, size_t dstSize,
-                           const void* src, size_t srcSize,
-                           const FSE_CTable* ct, const unsigned fast)
-{
-    const BYTE* const istart = (const BYTE*) src;
-    const BYTE* const iend = istart + srcSize;
-    const BYTE* ip=iend;
-
-    BIT_CStream_t bitC;
-    FSE_CState_t CState1, CState2;
-
-    /* init */
-    if (srcSize <= 2) return 0;
-    { size_t const initError = BIT_initCStream(&bitC, dst, dstSize);
-      if (FSE_isError(initError)) return 0; /* not enough space available to write a bitstream */ }
-
-#define FSE_FLUSHBITS(s)  (fast ? BIT_flushBitsFast(s) : BIT_flushBits(s))
-
-    if (srcSize & 1) {
-        FSE_initCState2(&CState1, ct, *--ip);
-        FSE_initCState2(&CState2, ct, *--ip);
-        FSE_encodeSymbol(&bitC, &CState1, *--ip);
-        FSE_FLUSHBITS(&bitC);
-    } else {
-        FSE_initCState2(&CState2, ct, *--ip);
-        FSE_initCState2(&CState1, ct, *--ip);
-    }
-
-    /* join to mod 4 */
-    srcSize -= 2;
-    if ((sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) && (srcSize & 2)) {  /* test bit 2 */
-        FSE_encodeSymbol(&bitC, &CState2, *--ip);
-        FSE_encodeSymbol(&bitC, &CState1, *--ip);
-        FSE_FLUSHBITS(&bitC);
-    }
-
-    /* 2 or 4 encoding per loop */
-    while ( ip>istart ) {
-
-        FSE_encodeSymbol(&bitC, &CState2, *--ip);
-
-        if (sizeof(bitC.bitContainer)*8 < FSE_MAX_TABLELOG*2+7 )   /* this test must be static */
-            FSE_FLUSHBITS(&bitC);
-
-        FSE_encodeSymbol(&bitC, &CState1, *--ip);
-
-        if (sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) {  /* this test must be static */
-            FSE_encodeSymbol(&bitC, &CState2, *--ip);
-            FSE_encodeSymbol(&bitC, &CState1, *--ip);
-        }
-
-        FSE_FLUSHBITS(&bitC);
-    }
-
-    FSE_flushCState(&bitC, &CState2);
-    FSE_flushCState(&bitC, &CState1);
-    return BIT_closeCStream(&bitC);
-}
-
-size_t FSE_compress_usingCTable (void* dst, size_t dstSize,
-                           const void* src, size_t srcSize,
-                           const FSE_CTable* ct)
-{
-    unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize));
-
-    if (fast)
-        return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1);
-    else
-        return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0);
-}
-
-
-size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }
-
-#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
-#define CHECK_F(f)   { CHECK_V_F(_var_err__, f); }
-
-/* FSE_compress_wksp() :
- * Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`).
- * `wkspSize` size must be `(1<<tableLog)`.
- */
-size_t FSE_compress_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
-{
-    BYTE* const ostart = (BYTE*) dst;
-    BYTE* op = ostart;
-    BYTE* const oend = ostart + dstSize;
-
-    U32   count[FSE_MAX_SYMBOL_VALUE+1];
-    S16   norm[FSE_MAX_SYMBOL_VALUE+1];
-    FSE_CTable* CTable = (FSE_CTable*)workSpace;
-    size_t const CTableSize = FSE_CTABLE_SIZE_U32(tableLog, maxSymbolValue);
-    void* scratchBuffer = (void*)(CTable + CTableSize);
-    size_t const scratchBufferSize = wkspSize - (CTableSize * sizeof(FSE_CTable));
-
-    /* init conditions */
-    if (wkspSize < FSE_WKSP_SIZE_U32(tableLog, maxSymbolValue)) return ERROR(tableLog_tooLarge);
-    if (srcSize <= 1) return 0;  /* Not compressible */
-    if (!maxSymbolValue) maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
-    if (!tableLog) tableLog = FSE_DEFAULT_TABLELOG;
-
-    /* Scan input and build symbol stats */
-    {   CHECK_V_F(maxCount, FSE_count_wksp(count, &maxSymbolValue, src, srcSize, (unsigned*)scratchBuffer) );
-        if (maxCount == srcSize) return 1;   /* only a single symbol in src : rle */
-        if (maxCount == 1) return 0;         /* each symbol present maximum once => not compressible */
-        if (maxCount < (srcSize >> 7)) return 0;   /* Heuristic : not compressible enough */
-    }
-
-    tableLog = FSE_optimalTableLog(tableLog, srcSize, maxSymbolValue);
-    CHECK_F( FSE_normalizeCount(norm, tableLog, count, srcSize, maxSymbolValue) );
-
-    /* Write table description header */
-    {   CHECK_V_F(nc_err, FSE_writeNCount(op, oend-op, norm, maxSymbolValue, tableLog) );
-        op += nc_err;
-    }
-
-    /* Compress */
-    CHECK_F( FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, scratchBuffer, scratchBufferSize) );
-    {   CHECK_V_F(cSize, FSE_compress_usingCTable(op, oend - op, src, srcSize, CTable) );
-        if (cSize == 0) return 0;   /* not enough space for compressed data */
-        op += cSize;
-    }
-
-    /* check compressibility */
-    if ( (size_t)(op-ostart) >= srcSize-1 ) return 0;
-
-    return op-ostart;
-}
-
-typedef struct {
-    FSE_CTable CTable_max[FSE_CTABLE_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE)];
-    BYTE scratchBuffer[1 << FSE_MAX_TABLELOG];
-} fseWkspMax_t;
-
-size_t FSE_compress2 (void* dst, size_t dstCapacity, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog)
-{
-    fseWkspMax_t scratchBuffer;
-    FSE_STATIC_ASSERT(sizeof(scratchBuffer) >= FSE_WKSP_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE));   /* compilation failures here means scratchBuffer is not large enough */
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
-    return FSE_compress_wksp(dst, dstCapacity, src, srcSize, maxSymbolValue, tableLog, &scratchBuffer, sizeof(scratchBuffer));
-}
-
-size_t FSE_compress (void* dst, size_t dstCapacity, const void* src, size_t srcSize)
-{
-    return FSE_compress2(dst, dstCapacity, src, srcSize, FSE_MAX_SYMBOL_VALUE, FSE_DEFAULT_TABLELOG);
-}
-
-
-#endif   /* FSE_COMMONDEFS_ONLY */

third/github.com/DataDog/zstd/fse_decompress.c

@@ -1,309 +0,0 @@
-/* ******************************************************************
-   FSE : Finite State Entropy decoder
-   Copyright (C) 2013-2015, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
-****************************************************************** */
-
-
-/* **************************************************************
-*  Includes
-****************************************************************/
-#include <stdlib.h>     /* malloc, free, qsort */
-#include <string.h>     /* memcpy, memset */
-#include "bitstream.h"
-#include "compiler.h"
-#define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
-#include "error_private.h"
-
-
-/* **************************************************************
-*  Error Management
-****************************************************************/
-#define FSE_isError ERR_isError
-#define FSE_STATIC_ASSERT(c) { enum { FSE_static_assert = 1/(int)(!!(c)) }; }   /* use only *after* variable declarations */
-
-/* check and forward error code */
-#define CHECK_F(f) { size_t const e = f; if (FSE_isError(e)) return e; }
-
-
-/* **************************************************************
-*  Templates
-****************************************************************/
-/*
-  designed to be included
-  for type-specific functions (template emulation in C)
-  Objective is to write these functions only once, for improved maintenance
-*/
-
-/* safety checks */
-#ifndef FSE_FUNCTION_EXTENSION
-#  error "FSE_FUNCTION_EXTENSION must be defined"
-#endif
-#ifndef FSE_FUNCTION_TYPE
-#  error "FSE_FUNCTION_TYPE must be defined"
-#endif
-
-/* Function names */
-#define FSE_CAT(X,Y) X##Y
-#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
-#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
-
-
-/* Function templates */
-FSE_DTable* FSE_createDTable (unsigned tableLog)
-{
-    if (tableLog > FSE_TABLELOG_ABSOLUTE_MAX) tableLog = FSE_TABLELOG_ABSOLUTE_MAX;
-    return (FSE_DTable*)malloc( FSE_DTABLE_SIZE_U32(tableLog) * sizeof (U32) );
-}
-
-void FSE_freeDTable (FSE_DTable* dt)
-{
-    free(dt);
-}
-
-size_t FSE_buildDTable(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
-{
-    void* const tdPtr = dt+1;   /* because *dt is unsigned, 32-bits aligned on 32-bits */
-    FSE_DECODE_TYPE* const tableDecode = (FSE_DECODE_TYPE*) (tdPtr);
-    U16 symbolNext[FSE_MAX_SYMBOL_VALUE+1];
-
-    U32 const maxSV1 = maxSymbolValue + 1;
-    U32 const tableSize = 1 << tableLog;
-    U32 highThreshold = tableSize-1;
-
-    /* Sanity Checks */
-    if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE) return ERROR(maxSymbolValue_tooLarge);
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
-
-    /* Init, lay down lowprob symbols */
-    {   FSE_DTableHeader DTableH;
-        DTableH.tableLog = (U16)tableLog;
-        DTableH.fastMode = 1;
-        {   S16 const largeLimit= (S16)(1 << (tableLog-1));
-            U32 s;
-            for (s=0; s<maxSV1; s++) {
-                if (normalizedCounter[s]==-1) {
-                    tableDecode[highThreshold--].symbol = (FSE_FUNCTION_TYPE)s;
-                    symbolNext[s] = 1;
-                } else {
-                    if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
-                    symbolNext[s] = normalizedCounter[s];
-        }   }   }
-        memcpy(dt, &DTableH, sizeof(DTableH));
-    }
-
-    /* Spread symbols */
-    {   U32 const tableMask = tableSize-1;
-        U32 const step = FSE_TABLESTEP(tableSize);
-        U32 s, position = 0;
-        for (s=0; s<maxSV1; s++) {
-            int i;
-            for (i=0; i<normalizedCounter[s]; i++) {
-                tableDecode[position].symbol = (FSE_FUNCTION_TYPE)s;
-                position = (position + step) & tableMask;
-                while (position > highThreshold) position = (position + step) & tableMask;   /* lowprob area */
-        }   }
-        if (position!=0) return ERROR(GENERIC);   /* position must reach all cells once, otherwise normalizedCounter is incorrect */
-    }
-
-    /* Build Decoding table */
-    {   U32 u;
-        for (u=0; u<tableSize; u++) {
-            FSE_FUNCTION_TYPE const symbol = (FSE_FUNCTION_TYPE)(tableDecode[u].symbol);
-            U32 const nextState = symbolNext[symbol]++;
-            tableDecode[u].nbBits = (BYTE) (tableLog - BIT_highbit32(nextState) );
-            tableDecode[u].newState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
-    }   }
-
-    return 0;
-}
-
-
-#ifndef FSE_COMMONDEFS_ONLY
-
-/*-*******************************************************
-*  Decompression (Byte symbols)
-*********************************************************/
-size_t FSE_buildDTable_rle (FSE_DTable* dt, BYTE symbolValue)
-{
-    void* ptr = dt;
-    FSE_DTableHeader* const DTableH = (FSE_DTableHeader*)ptr;
-    void* dPtr = dt + 1;
-    FSE_decode_t* const cell = (FSE_decode_t*)dPtr;
-
-    DTableH->tableLog = 0;
-    DTableH->fastMode = 0;
-
-    cell->newState = 0;
-    cell->symbol = symbolValue;
-    cell->nbBits = 0;
-
-    return 0;
-}
-
-
-size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits)
-{
-    void* ptr = dt;
-    FSE_DTableHeader* const DTableH = (FSE_DTableHeader*)ptr;
-    void* dPtr = dt + 1;
-    FSE_decode_t* const dinfo = (FSE_decode_t*)dPtr;
-    const unsigned tableSize = 1 << nbBits;
-    const unsigned tableMask = tableSize - 1;
-    const unsigned maxSV1 = tableMask+1;
-    unsigned s;
-
-    /* Sanity checks */
-    if (nbBits < 1) return ERROR(GENERIC);         /* min size */
-
-    /* Build Decoding Table */
-    DTableH->tableLog = (U16)nbBits;
-    DTableH->fastMode = 1;
-    for (s=0; s<maxSV1; s++) {
-        dinfo[s].newState = 0;
-        dinfo[s].symbol = (BYTE)s;
-        dinfo[s].nbBits = (BYTE)nbBits;
-    }
-
-    return 0;
-}
-
-FORCE_INLINE_TEMPLATE size_t FSE_decompress_usingDTable_generic(
-          void* dst, size_t maxDstSize,
-    const void* cSrc, size_t cSrcSize,
-    const FSE_DTable* dt, const unsigned fast)
-{
-    BYTE* const ostart = (BYTE*) dst;
-    BYTE* op = ostart;
-    BYTE* const omax = op + maxDstSize;
-    BYTE* const olimit = omax-3;
-
-    BIT_DStream_t bitD;
-    FSE_DState_t state1;
-    FSE_DState_t state2;
-
-    /* Init */
-    CHECK_F(BIT_initDStream(&bitD, cSrc, cSrcSize));
-
-    FSE_initDState(&state1, &bitD, dt);
-    FSE_initDState(&state2, &bitD, dt);
-
-#define FSE_GETSYMBOL(statePtr) fast ? FSE_decodeSymbolFast(statePtr, &bitD) : FSE_decodeSymbol(statePtr, &bitD)
-
-    /* 4 symbols per loop */
-    for ( ; (BIT_reloadDStream(&bitD)==BIT_DStream_unfinished) & (op<olimit) ; op+=4) {
-        op[0] = FSE_GETSYMBOL(&state1);
-
-        if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8)    /* This test must be static */
-            BIT_reloadDStream(&bitD);
-
-        op[1] = FSE_GETSYMBOL(&state2);
-
-        if (FSE_MAX_TABLELOG*4+7 > sizeof(bitD.bitContainer)*8)    /* This test must be static */
-            { if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) { op+=2; break; } }
-
-        op[2] = FSE_GETSYMBOL(&state1);
-
-        if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8)    /* This test must be static */
-            BIT_reloadDStream(&bitD);
-
-        op[3] = FSE_GETSYMBOL(&state2);
-    }
-
-    /* tail */
-    /* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */
-    while (1) {
-        if (op>(omax-2)) return ERROR(dstSize_tooSmall);
-        *op++ = FSE_GETSYMBOL(&state1);
-        if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
-            *op++ = FSE_GETSYMBOL(&state2);
-            break;
-        }
-
-        if (op>(omax-2)) return ERROR(dstSize_tooSmall);
-        *op++ = FSE_GETSYMBOL(&state2);
-        if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
-            *op++ = FSE_GETSYMBOL(&state1);
-            break;
-    }   }
-
-    return op-ostart;
-}
-
-
-size_t FSE_decompress_usingDTable(void* dst, size_t originalSize,
-                            const void* cSrc, size_t cSrcSize,
-                            const FSE_DTable* dt)
-{
-    const void* ptr = dt;
-    const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr;
-    const U32 fastMode = DTableH->fastMode;
-
-    /* select fast mode (static) */
-    if (fastMode) return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 1);
-    return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 0);
-}
-
-
-size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, FSE_DTable* workSpace, unsigned maxLog)
-{
-    const BYTE* const istart = (const BYTE*)cSrc;
-    const BYTE* ip = istart;
-    short counting[FSE_MAX_SYMBOL_VALUE+1];
-    unsigned tableLog;
-    unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
-
-    /* normal FSE decoding mode */
-    size_t const NCountLength = FSE_readNCount (counting, &maxSymbolValue, &tableLog, istart, cSrcSize);
-    if (FSE_isError(NCountLength)) return NCountLength;
-    //if (NCountLength >= cSrcSize) return ERROR(srcSize_wrong);   /* too small input size; supposed to be already checked in NCountLength, only remaining case : NCountLength==cSrcSize */
-    if (tableLog > maxLog) return ERROR(tableLog_tooLarge);
-    ip += NCountLength;
-    cSrcSize -= NCountLength;
-
-    CHECK_F( FSE_buildDTable (workSpace, counting, maxSymbolValue, tableLog) );
-
-    return FSE_decompress_usingDTable (dst, dstCapacity, ip, cSrcSize, workSpace);   /* always return, even if it is an error code */
-}
-
-
-typedef FSE_DTable DTable_max_t[FSE_DTABLE_SIZE_U32(FSE_MAX_TABLELOG)];
-
-size_t FSE_decompress(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize)
-{
-    DTable_max_t dt;   /* Static analyzer seems unable to understand this table will be properly initialized later */
-    return FSE_decompress_wksp(dst, dstCapacity, cSrc, cSrcSize, dt, FSE_MAX_TABLELOG);
-}
-
-
-
-#endif   /* FSE_COMMONDEFS_ONLY */

third/github.com/DataDog/zstd/huf.h

@@ -1,327 +0,0 @@
-/* ******************************************************************
-   Huffman coder, part of New Generation Entropy library
-   header file
-   Copyright (C) 2013-2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
-****************************************************************** */
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-#ifndef HUF_H_298734234
-#define HUF_H_298734234
-
-/* *** Dependencies *** */
-#include <stddef.h>    /* size_t */
-
-
-/* *** library symbols visibility *** */
-/* Note : when linking with -fvisibility=hidden on gcc, or by default on Visual,
- *        HUF symbols remain "private" (internal symbols for library only).
- *        Set macro FSE_DLL_EXPORT to 1 if you want HUF symbols visible on DLL interface */
-#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
-#  define HUF_PUBLIC_API __attribute__ ((visibility ("default")))
-#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1)   /* Visual expected */
-#  define HUF_PUBLIC_API __declspec(dllexport)
-#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
-#  define HUF_PUBLIC_API __declspec(dllimport)  /* not required, just to generate faster code (saves a function pointer load from IAT and an indirect jump) */
-#else
-#  define HUF_PUBLIC_API
-#endif
-
-
-/* ========================== */
-/* ***  simple functions  *** */
-/* ========================== */
-
-/** HUF_compress() :
- *  Compress content from buffer 'src', of size 'srcSize', into buffer 'dst'.
- * 'dst' buffer must be already allocated.
- *  Compression runs faster if `dstCapacity` >= HUF_compressBound(srcSize).
- * `srcSize` must be <= `HUF_BLOCKSIZE_MAX` == 128 KB.
- * @return : size of compressed data (<= `dstCapacity`).
- *  Special values : if return == 0, srcData is not compressible => Nothing is stored within dst !!!
- *                   if HUF_isError(return), compression failed (more details using HUF_getErrorName())
- */
-HUF_PUBLIC_API size_t HUF_compress(void* dst, size_t dstCapacity,
-                             const void* src, size_t srcSize);
-
-/** HUF_decompress() :
- *  Decompress HUF data from buffer 'cSrc', of size 'cSrcSize',
- *  into already allocated buffer 'dst', of minimum size 'dstSize'.
- * `originalSize` : **must** be the ***exact*** size of original (uncompressed) data.
- *  Note : in contrast with FSE, HUF_decompress can regenerate
- *         RLE (cSrcSize==1) and uncompressed (cSrcSize==dstSize) data,
- *         because it knows size to regenerate (originalSize).
- * @return : size of regenerated data (== originalSize),
- *           or an error code, which can be tested using HUF_isError()
- */
-HUF_PUBLIC_API size_t HUF_decompress(void* dst,  size_t originalSize,
-                               const void* cSrc, size_t cSrcSize);
-
-
-/* ***   Tool functions *** */
-#define HUF_BLOCKSIZE_MAX (128 * 1024)                  /**< maximum input size for a single block compressed with HUF_compress */
-HUF_PUBLIC_API size_t HUF_compressBound(size_t size);   /**< maximum compressed size (worst case) */
-
-/* Error Management */
-HUF_PUBLIC_API unsigned    HUF_isError(size_t code);       /**< tells if a return value is an error code */
-HUF_PUBLIC_API const char* HUF_getErrorName(size_t code);  /**< provides error code string (useful for debugging) */
-
-
-/* ***   Advanced function   *** */
-
-/** HUF_compress2() :
- *  Same as HUF_compress(), but offers control over `maxSymbolValue` and `tableLog`.
- * `maxSymbolValue` must be <= HUF_SYMBOLVALUE_MAX .
- * `tableLog` must be `<= HUF_TABLELOG_MAX` . */
-HUF_PUBLIC_API size_t HUF_compress2 (void* dst, size_t dstCapacity,
-                               const void* src, size_t srcSize,
-                               unsigned maxSymbolValue, unsigned tableLog);
-
-/** HUF_compress4X_wksp() :
- *  Same as HUF_compress2(), but uses externally allocated `workSpace`.
- * `workspace` must have minimum alignment of 4, and be at least as large as HUF_WORKSPACE_SIZE */
-#define HUF_WORKSPACE_SIZE (6 << 10)
-#define HUF_WORKSPACE_SIZE_U32 (HUF_WORKSPACE_SIZE / sizeof(U32))
-HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
-                                     const void* src, size_t srcSize,
-                                     unsigned maxSymbolValue, unsigned tableLog,
-                                     void* workSpace, size_t wkspSize);
-
-#endif   /* HUF_H_298734234 */
-
-/* ******************************************************************
- *  WARNING !!
- *  The following section contains advanced and experimental definitions
- *  which shall never be used in the context of a dynamic library,
- *  because they are not guaranteed to remain stable in the future.
- *  Only consider them in association with static linking.
- * *****************************************************************/
-#if defined(HUF_STATIC_LINKING_ONLY) && !defined(HUF_H_HUF_STATIC_LINKING_ONLY)
-#define HUF_H_HUF_STATIC_LINKING_ONLY
-
-/* *** Dependencies *** */
-#include "mem.h"   /* U32 */
-
-
-/* *** Constants *** */
-#define HUF_TABLELOG_MAX      12      /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
-#define HUF_TABLELOG_DEFAULT  11      /* default tableLog value when none specified */
-#define HUF_SYMBOLVALUE_MAX  255
-
-#define HUF_TABLELOG_ABSOLUTEMAX  15  /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
-#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
-#  error "HUF_TABLELOG_MAX is too large !"
-#endif
-
-
-/* ****************************************
-*  Static allocation
-******************************************/
-/* HUF buffer bounds */
-#define HUF_CTABLEBOUND 129
-#define HUF_BLOCKBOUND(size) (size + (size>>8) + 8)   /* only true when incompressible is pre-filtered with fast heuristic */
-#define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size))   /* Macro version, useful for static allocation */
-
-/* static allocation of HUF's Compression Table */
-#define HUF_CTABLE_SIZE_U32(maxSymbolValue)   ((maxSymbolValue)+1)   /* Use tables of U32, for proper alignment */
-#define HUF_CTABLE_SIZE(maxSymbolValue)       (HUF_CTABLE_SIZE_U32(maxSymbolValue) * sizeof(U32))
-#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
-    U32 name##hb[HUF_CTABLE_SIZE_U32(maxSymbolValue)]; \
-    void* name##hv = &(name##hb); \
-    HUF_CElt* name = (HUF_CElt*)(name##hv)   /* no final ; */
-
-/* static allocation of HUF's DTable */
-typedef U32 HUF_DTable;
-#define HUF_DTABLE_SIZE(maxTableLog)   (1 + (1<<(maxTableLog)))
-#define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) \
-        HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = { ((U32)((maxTableLog)-1) * 0x01000001) }
-#define HUF_CREATE_STATIC_DTABLEX4(DTable, maxTableLog) \
-        HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = { ((U32)(maxTableLog) * 0x01000001) }
-
-
-/* ****************************************
-*  Advanced decompression functions
-******************************************/
-size_t HUF_decompress4X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< single-symbol decoder */
-size_t HUF_decompress4X4 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< double-symbols decoder */
-
-size_t HUF_decompress4X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< decodes RLE and uncompressed */
-size_t HUF_decompress4X_hufOnly(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /**< considers RLE and uncompressed as errors */
-size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /**< considers RLE and uncompressed as errors */
-size_t HUF_decompress4X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< single-symbol decoder */
-size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);   /**< single-symbol decoder */
-size_t HUF_decompress4X4_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< double-symbols decoder */
-size_t HUF_decompress4X4_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);   /**< double-symbols decoder */
-
-
-/* ****************************************
- *  HUF detailed API
- * ****************************************/
-
-/*! HUF_compress() does the following:
- *  1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h")
- *  2. (optional) refine tableLog using HUF_optimalTableLog()
- *  3. build Huffman table from count using HUF_buildCTable()
- *  4. save Huffman table to memory buffer using HUF_writeCTable()
- *  5. encode the data stream using HUF_compress4X_usingCTable()
- *
- *  The following API allows targeting specific sub-functions for advanced tasks.
- *  For example, it's possible to compress several blocks using the same 'CTable',
- *  or to save and regenerate 'CTable' using external methods.
- */
-unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
-typedef struct HUF_CElt_s HUF_CElt;   /* incomplete type */
-size_t HUF_buildCTable (HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits);   /* @return : maxNbBits; CTable and count can overlap. In which case, CTable will overwrite count content */
-size_t HUF_writeCTable (void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog);
-size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
-
-typedef enum {
-   HUF_repeat_none,  /**< Cannot use the previous table */
-   HUF_repeat_check, /**< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat */
-   HUF_repeat_valid  /**< Can use the previous table and it is asumed to be valid */
- } HUF_repeat;
-/** HUF_compress4X_repeat() :
- *  Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
- *  If it uses hufTable it does not modify hufTable or repeat.
- *  If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
- *  If preferRepeat then the old table will always be used if valid. */
-size_t HUF_compress4X_repeat(void* dst, size_t dstSize,
-                       const void* src, size_t srcSize,
-                       unsigned maxSymbolValue, unsigned tableLog,
-                       void* workSpace, size_t wkspSize,    /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
-                       HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);
-
-/** HUF_buildCTable_wksp() :
- *  Same as HUF_buildCTable(), but using externally allocated scratch buffer.
- * `workSpace` must be aligned on 4-bytes boundaries, and its size must be >= HUF_CTABLE_WORKSPACE_SIZE.
- */
-#define HUF_CTABLE_WORKSPACE_SIZE_U32 (2*HUF_SYMBOLVALUE_MAX +1 +1)
-#define HUF_CTABLE_WORKSPACE_SIZE (HUF_CTABLE_WORKSPACE_SIZE_U32 * sizeof(unsigned))
-size_t HUF_buildCTable_wksp (HUF_CElt* tree, const U32* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize);
-
-/*! HUF_readStats() :
- *  Read compact Huffman tree, saved by HUF_writeCTable().
- * `huffWeight` is destination buffer.
- * @return : size read from `src` , or an error Code .
- *  Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */
-size_t HUF_readStats(BYTE* huffWeight, size_t hwSize,
-                     U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize);
-
-/** HUF_readCTable() :
- *  Loading a CTable saved with HUF_writeCTable() */
-size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize);
-
-
-/*
- * HUF_decompress() does the following:
- * 1. select the decompression algorithm (X2, X4) based on pre-computed heuristics
- * 2. build Huffman table from save, using HUF_readDTableX?()
- * 3. decode 1 or 4 segments in parallel using HUF_decompress?X?_usingDTable()
- */
-
-/** HUF_selectDecoder() :
- *  Tells which decoder is likely to decode faster,
- *  based on a set of pre-computed metrics.
- * @return : 0==HUF_decompress4X2, 1==HUF_decompress4X4 .
- *  Assumption : 0 < dstSize <= 128 KB */
-U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize);
-
-/**
- *  The minimum workspace size for the `workSpace` used in
- *  HUF_readDTableX2_wksp() and HUF_readDTableX4_wksp().
- *
- *  The space used depends on HUF_TABLELOG_MAX, ranging from ~1500 bytes when
- *  HUF_TABLE_LOG_MAX=12 to ~1850 bytes when HUF_TABLE_LOG_MAX=15.
- *  Buffer overflow errors may potentially occur if code modifications result in
- *  a required workspace size greater than that specified in the following
- *  macro.
- */
-#define HUF_DECOMPRESS_WORKSPACE_SIZE (2 << 10)
-#define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32))
-
-size_t HUF_readDTableX2 (HUF_DTable* DTable, const void* src, size_t srcSize);
-size_t HUF_readDTableX2_wksp (HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize);
-size_t HUF_readDTableX4 (HUF_DTable* DTable, const void* src, size_t srcSize);
-size_t HUF_readDTableX4_wksp (HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize);
-
-size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
-size_t HUF_decompress4X2_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
-size_t HUF_decompress4X4_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
-
-
-/* ====================== */
-/* single stream variants */
-/* ====================== */
-
-size_t HUF_compress1X (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);
-size_t HUF_compress1X_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);  /**< `workSpace` must be a table of at least HUF_WORKSPACE_SIZE_U32 unsigned */
-size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
-/** HUF_compress1X_repeat() :
- *  Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
- *  If it uses hufTable it does not modify hufTable or repeat.
- *  If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
- *  If preferRepeat then the old table will always be used if valid. */
-size_t HUF_compress1X_repeat(void* dst, size_t dstSize,
-                       const void* src, size_t srcSize,
-                       unsigned maxSymbolValue, unsigned tableLog,
-                       void* workSpace, size_t wkspSize,   /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
-                       HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);
-
-size_t HUF_decompress1X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /* single-symbol decoder */
-size_t HUF_decompress1X4 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /* double-symbol decoder */
-
-size_t HUF_decompress1X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);
-size_t HUF_decompress1X_DCtx_wksp (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);
-size_t HUF_decompress1X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< single-symbol decoder */
-size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);   /**< single-symbol decoder */
-size_t HUF_decompress1X4_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);   /**< double-symbols decoder */
-size_t HUF_decompress1X4_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);   /**< double-symbols decoder */
-
-size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);   /**< automatic selection of sing or double symbol decoder, based on DTable */
-size_t HUF_decompress1X2_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
-size_t HUF_decompress1X4_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
-
-/* BMI2 variants.
- * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
- */
-size_t HUF_decompress1X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2);
-size_t HUF_decompress1X2_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2);
-size_t HUF_decompress4X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2);
-size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2);
-
-#endif /* HUF_STATIC_LINKING_ONLY */
-
-#if defined (__cplusplus)
-}
-#endif

third/github.com/DataDog/zstd/huf_compress.c

@@ -1,788 +0,0 @@
-/* ******************************************************************
-   Huffman encoder, part of New Generation Entropy library
-   Copyright (C) 2013-2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
-****************************************************************** */
-
-/* **************************************************************
-*  Compiler specifics
-****************************************************************/
-#ifdef _MSC_VER    /* Visual Studio */
-#  pragma warning(disable : 4127)        /* disable: C4127: conditional expression is constant */
-#endif
-
-
-/* **************************************************************
-*  Includes
-****************************************************************/
-#include <string.h>     /* memcpy, memset */
-#include <stdio.h>      /* printf (debug) */
-#include "bitstream.h"
-#include "compiler.h"
-#define FSE_STATIC_LINKING_ONLY   /* FSE_optimalTableLog_internal */
-#include "fse.h"        /* header compression */
-#define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
-#include "error_private.h"
-
-
-/* **************************************************************
-*  Error Management
-****************************************************************/
-#define HUF_isError ERR_isError
-#define HUF_STATIC_ASSERT(c) { enum { HUF_static_assert = 1/(int)(!!(c)) }; }   /* use only *after* variable declarations */
-#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
-#define CHECK_F(f)   { CHECK_V_F(_var_err__, f); }
-
-
-/* **************************************************************
-*  Utils
-****************************************************************/
-unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
-{
-    return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 1);
-}
-
-
-/* *******************************************************
-*  HUF : Huffman block compression
-*********************************************************/
-/* HUF_compressWeights() :
- * Same as FSE_compress(), but dedicated to huff0's weights compression.
- * The use case needs much less stack memory.
- * Note : all elements within weightTable are supposed to be <= HUF_TABLELOG_MAX.
- */
-#define MAX_FSE_TABLELOG_FOR_HUFF_HEADER 6
-size_t HUF_compressWeights (void* dst, size_t dstSize, const void* weightTable, size_t wtSize)
-{
-    BYTE* const ostart = (BYTE*) dst;
-    BYTE* op = ostart;
-    BYTE* const oend = ostart + dstSize;
-
-    U32 maxSymbolValue = HUF_TABLELOG_MAX;
-    U32 tableLog = MAX_FSE_TABLELOG_FOR_HUFF_HEADER;
-
-    FSE_CTable CTable[FSE_CTABLE_SIZE_U32(MAX_FSE_TABLELOG_FOR_HUFF_HEADER, HUF_TABLELOG_MAX)];
-    BYTE scratchBuffer[1<<MAX_FSE_TABLELOG_FOR_HUFF_HEADER];
-
-    U32 count[HUF_TABLELOG_MAX+1];
-    S16 norm[HUF_TABLELOG_MAX+1];
-
-    /* init conditions */
-    if (wtSize <= 1) return 0;  /* Not compressible */
-
-    /* Scan input and build symbol stats */
-    {   CHECK_V_F(maxCount, FSE_count_simple(count, &maxSymbolValue, weightTable, wtSize) );
-        if (maxCount == wtSize) return 1;   /* only a single symbol in src : rle */
-        if (maxCount == 1) return 0;         /* each symbol present maximum once => not compressible */
-    }
-
-    tableLog = FSE_optimalTableLog(tableLog, wtSize, maxSymbolValue);
-    CHECK_F( FSE_normalizeCount(norm, tableLog, count, wtSize, maxSymbolValue) );
-
-    /* Write table description header */
-    {   CHECK_V_F(hSize, FSE_writeNCount(op, oend-op, norm, maxSymbolValue, tableLog) );
-        op += hSize;
-    }
-
-    /* Compress */
-    CHECK_F( FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, scratchBuffer, sizeof(scratchBuffer)) );
-    {   CHECK_V_F(cSize, FSE_compress_usingCTable(op, oend - op, weightTable, wtSize, CTable) );
-        if (cSize == 0) return 0;   /* not enough space for compressed data */
-        op += cSize;
-    }
-
-    return op-ostart;
-}
-
-
-struct HUF_CElt_s {
-  U16  val;
-  BYTE nbBits;
-};   /* typedef'd to HUF_CElt within "huf.h" */
-
-/*! HUF_writeCTable() :
-    `CTable` : Huffman tree to save, using huf representation.
-    @return : size of saved CTable */
-size_t HUF_writeCTable (void* dst, size_t maxDstSize,
-                        const HUF_CElt* CTable, U32 maxSymbolValue, U32 huffLog)
-{
-    BYTE bitsToWeight[HUF_TABLELOG_MAX + 1];   /* precomputed conversion table */
-    BYTE huffWeight[HUF_SYMBOLVALUE_MAX];
-    BYTE* op = (BYTE*)dst;
-    U32 n;
-
-     /* check conditions */
-    if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
-
-    /* convert to weight */
-    bitsToWeight[0] = 0;
-    for (n=1; n<huffLog+1; n++)
-        bitsToWeight[n] = (BYTE)(huffLog + 1 - n);
-    for (n=0; n<maxSymbolValue; n++)
-        huffWeight[n] = bitsToWeight[CTable[n].nbBits];
-
-    /* attempt weights compression by FSE */
-    {   CHECK_V_F(hSize, HUF_compressWeights(op+1, maxDstSize-1, huffWeight, maxSymbolValue) );
-        if ((hSize>1) & (hSize < maxSymbolValue/2)) {   /* FSE compressed */
-            op[0] = (BYTE)hSize;
-            return hSize+1;
-    }   }
-
-    /* write raw values as 4-bits (max : 15) */
-    if (maxSymbolValue > (256-128)) return ERROR(GENERIC);   /* should not happen : likely means source cannot be compressed */
-    if (((maxSymbolValue+1)/2) + 1 > maxDstSize) return ERROR(dstSize_tooSmall);   /* not enough space within dst buffer */
-    op[0] = (BYTE)(128 /*special case*/ + (maxSymbolValue-1));
-    huffWeight[maxSymbolValue] = 0;   /* to be sure it doesn't cause msan issue in final combination */
-    for (n=0; n<maxSymbolValue; n+=2)
-        op[(n/2)+1] = (BYTE)((huffWeight[n] << 4) + huffWeight[n+1]);
-    return ((maxSymbolValue+1)/2) + 1;
-}
-
-
-size_t HUF_readCTable (HUF_CElt* CTable, U32* maxSymbolValuePtr, const void* src, size_t srcSize)
-{
-    BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1];   /* init not required, even though some static analyzer may complain */
-    U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1];   /* large enough for values from 0 to 16 */
-    U32 tableLog = 0;
-    U32 nbSymbols = 0;
-
-    /* get symbol weights */
-    CHECK_V_F(readSize, HUF_readStats(huffWeight, HUF_SYMBOLVALUE_MAX+1, rankVal, &nbSymbols, &tableLog, src, srcSize));
-
-    /* check result */
-    if (tableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
-    if (nbSymbols > *maxSymbolValuePtr+1) return ERROR(maxSymbolValue_tooSmall);
-
-    /* Prepare base value per rank */
-    {   U32 n, nextRankStart = 0;
-        for (n=1; n<=tableLog; n++) {
-            U32 current = nextRankStart;
-            nextRankStart += (rankVal[n] << (n-1));
-            rankVal[n] = current;
-    }   }
-
-    /* fill nbBits */
-    {   U32 n; for (n=0; n<nbSymbols; n++) {
-            const U32 w = huffWeight[n];
-            CTable[n].nbBits = (BYTE)(tableLog + 1 - w);
-    }   }
-
-    /* fill val */
-    {   U16 nbPerRank[HUF_TABLELOG_MAX+2]  = {0};  /* support w=0=>n=tableLog+1 */
-        U16 valPerRank[HUF_TABLELOG_MAX+2] = {0};
-        { U32 n; for (n=0; n<nbSymbols; n++) nbPerRank[CTable[n].nbBits]++; }
-        /* determine stating value per rank */
-        valPerRank[tableLog+1] = 0;   /* for w==0 */
-        {   U16 min = 0;
-            U32 n; for (n=tableLog; n>0; n--) {  /* start at n=tablelog <-> w=1 */
-                valPerRank[n] = min;     /* get starting value within each rank */
-                min += nbPerRank[n];
-                min >>= 1;
-        }   }
-        /* assign value within rank, symbol order */
-        { U32 n; for (n=0; n<nbSymbols; n++) CTable[n].val = valPerRank[CTable[n].nbBits]++; }
-    }
-
-    *maxSymbolValuePtr = nbSymbols - 1;
-    return readSize;
-}
-
-
-typedef struct nodeElt_s {
-    U32 count;
-    U16 parent;
-    BYTE byte;
-    BYTE nbBits;
-} nodeElt;
-
-static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
-{
-    const U32 largestBits = huffNode[lastNonNull].nbBits;
-    if (largestBits <= maxNbBits) return largestBits;   /* early exit : no elt > maxNbBits */
-
-    /* there are several too large elements (at least >= 2) */
-    {   int totalCost = 0;
-        const U32 baseCost = 1 << (largestBits - maxNbBits);
-        U32 n = lastNonNull;
-
-        while (huffNode[n].nbBits > maxNbBits) {
-            totalCost += baseCost - (1 << (largestBits - huffNode[n].nbBits));
-            huffNode[n].nbBits = (BYTE)maxNbBits;
-            n --;
-        }  /* n stops at huffNode[n].nbBits <= maxNbBits */
-        while (huffNode[n].nbBits == maxNbBits) n--;   /* n end at index of smallest symbol using < maxNbBits */
-
-        /* renorm totalCost */
-        totalCost >>= (largestBits - maxNbBits);  /* note : totalCost is necessarily a multiple of baseCost */
-
-        /* repay normalized cost */
-        {   U32 const noSymbol = 0xF0F0F0F0;
-            U32 rankLast[HUF_TABLELOG_MAX+2];
-            int pos;
-
-            /* Get pos of last (smallest) symbol per rank */
-            memset(rankLast, 0xF0, sizeof(rankLast));
-            {   U32 currentNbBits = maxNbBits;
-                for (pos=n ; pos >= 0; pos--) {
-                    if (huffNode[pos].nbBits >= currentNbBits) continue;
-                    currentNbBits = huffNode[pos].nbBits;   /* < maxNbBits */
-                    rankLast[maxNbBits-currentNbBits] = pos;
-            }   }
-
-            while (totalCost > 0) {
-                U32 nBitsToDecrease = BIT_highbit32(totalCost) + 1;
-                for ( ; nBitsToDecrease > 1; nBitsToDecrease--) {
-                    U32 highPos = rankLast[nBitsToDecrease];
-                    U32 lowPos = rankLast[nBitsToDecrease-1];
-                    if (highPos == noSymbol) continue;
-                    if (lowPos == noSymbol) break;
-                    {   U32 const highTotal = huffNode[highPos].count;
-                        U32 const lowTotal = 2 * huffNode[lowPos].count;
-                        if (highTotal <= lowTotal) break;
-                }   }
-                /* only triggered when no more rank 1 symbol left => find closest one (note : there is necessarily at least one !) */
-                /* HUF_MAX_TABLELOG test just to please gcc 5+; but it should not be necessary */
-                while ((nBitsToDecrease<=HUF_TABLELOG_MAX) && (rankLast[nBitsToDecrease] == noSymbol))
-                    nBitsToDecrease ++;
-                totalCost -= 1 << (nBitsToDecrease-1);
-                if (rankLast[nBitsToDecrease-1] == noSymbol)
-                    rankLast[nBitsToDecrease-1] = rankLast[nBitsToDecrease];   /* this rank is no longer empty */
-                huffNode[rankLast[nBitsToDecrease]].nbBits ++;
-                if (rankLast[nBitsToDecrease] == 0)    /* special case, reached largest symbol */
-                    rankLast[nBitsToDecrease] = noSymbol;
-                else {
-                    rankLast[nBitsToDecrease]--;
-                    if (huffNode[rankLast[nBitsToDecrease]].nbBits != maxNbBits-nBitsToDecrease)
-                        rankLast[nBitsToDecrease] = noSymbol;   /* this rank is now empty */
-            }   }   /* while (totalCost > 0) */
-
-            while (totalCost < 0) {  /* Sometimes, cost correction overshoot */
-                if (rankLast[1] == noSymbol) {  /* special case : no rank 1 symbol (using maxNbBits-1); let's create one from largest rank 0 (using maxNbBits) */
-                    while (huffNode[n].nbBits == maxNbBits) n--;
-                    huffNode[n+1].nbBits--;
-                    rankLast[1] = n+1;
-                    totalCost++;
-                    continue;
-                }
-                huffNode[ rankLast[1] + 1 ].nbBits--;
-                rankLast[1]++;
-                totalCost ++;
-    }   }   }   /* there are several too large elements (at least >= 2) */
-
-    return maxNbBits;
-}
-
-
-typedef struct {
-    U32 base;
-    U32 current;
-} rankPos;
-
-static void HUF_sort(nodeElt* huffNode, const U32* count, U32 maxSymbolValue)
-{
-    rankPos rank[32];
-    U32 n;
-
-    memset(rank, 0, sizeof(rank));
-    for (n=0; n<=maxSymbolValue; n++) {
-        U32 r = BIT_highbit32(count[n] + 1);
-        rank[r].base ++;
-    }
-    for (n=30; n>0; n--) rank[n-1].base += rank[n].base;
-    for (n=0; n<32; n++) rank[n].current = rank[n].base;
-    for (n=0; n<=maxSymbolValue; n++) {
-        U32 const c = count[n];
-        U32 const r = BIT_highbit32(c+1) + 1;
-        U32 pos = rank[r].current++;
-        while ((pos > rank[r].base) && (c > huffNode[pos-1].count)) {
-            huffNode[pos] = huffNode[pos-1];
-            pos--;
-        }
-        huffNode[pos].count = c;
-        huffNode[pos].byte  = (BYTE)n;
-    }
-}
-
-
-/** HUF_buildCTable_wksp() :
- *  Same as HUF_buildCTable(), but using externally allocated scratch buffer.
- *  `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as a table of HUF_CTABLE_WORKSPACE_SIZE_U32 unsigned.
- */
-#define STARTNODE (HUF_SYMBOLVALUE_MAX+1)
-typedef nodeElt huffNodeTable[HUF_CTABLE_WORKSPACE_SIZE_U32];
-size_t HUF_buildCTable_wksp (HUF_CElt* tree, const U32* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize)
-{
-    nodeElt* const huffNode0 = (nodeElt*)workSpace;
-    nodeElt* const huffNode = huffNode0+1;
-    U32 n, nonNullRank;
-    int lowS, lowN;
-    U16 nodeNb = STARTNODE;
-    U32 nodeRoot;
-
-    /* safety checks */
-    if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC);  /* must be aligned on 4-bytes boundaries */
-    if (wkspSize < sizeof(huffNodeTable)) return ERROR(workSpace_tooSmall);
-    if (maxNbBits == 0) maxNbBits = HUF_TABLELOG_DEFAULT;
-    if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
-    memset(huffNode0, 0, sizeof(huffNodeTable));
-
-    /* sort, decreasing order */
-    HUF_sort(huffNode, count, maxSymbolValue);
-
-    /* init for parents */
-    nonNullRank = maxSymbolValue;
-    while(huffNode[nonNullRank].count == 0) nonNullRank--;
-    lowS = nonNullRank; nodeRoot = nodeNb + lowS - 1; lowN = nodeNb;
-    huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS-1].count;
-    huffNode[lowS].parent = huffNode[lowS-1].parent = nodeNb;
-    nodeNb++; lowS-=2;
-    for (n=nodeNb; n<=nodeRoot; n++) huffNode[n].count = (U32)(1U<<30);
-    huffNode0[0].count = (U32)(1U<<31);  /* fake entry, strong barrier */
-
-    /* create parents */
-    while (nodeNb <= nodeRoot) {
-        U32 n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
-        U32 n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
-        huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count;
-        huffNode[n1].parent = huffNode[n2].parent = nodeNb;
-        nodeNb++;
-    }
-
-    /* distribute weights (unlimited tree height) */
-    huffNode[nodeRoot].nbBits = 0;
-    for (n=nodeRoot-1; n>=STARTNODE; n--)
-        huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;
-    for (n=0; n<=nonNullRank; n++)
-        huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;
-
-    /* enforce maxTableLog */
-    maxNbBits = HUF_setMaxHeight(huffNode, nonNullRank, maxNbBits);
-
-    /* fill result into tree (val, nbBits) */
-    {   U16 nbPerRank[HUF_TABLELOG_MAX+1] = {0};
-        U16 valPerRank[HUF_TABLELOG_MAX+1] = {0};
-        if (maxNbBits > HUF_TABLELOG_MAX) return ERROR(GENERIC);   /* check fit into table */
-        for (n=0; n<=nonNullRank; n++)
-            nbPerRank[huffNode[n].nbBits]++;
-        /* determine stating value per rank */
-        {   U16 min = 0;
-            for (n=maxNbBits; n>0; n--) {
-                valPerRank[n] = min;      /* get starting value within each rank */
-                min += nbPerRank[n];
-                min >>= 1;
-        }   }
-        for (n=0; n<=maxSymbolValue; n++)
-            tree[huffNode[n].byte].nbBits = huffNode[n].nbBits;   /* push nbBits per symbol, symbol order */
-        for (n=0; n<=maxSymbolValue; n++)
-            tree[n].val = valPerRank[tree[n].nbBits]++;   /* assign value within rank, symbol order */
-    }
-
-    return maxNbBits;
-}
-
-/** HUF_buildCTable() :
- * @return : maxNbBits
- *  Note : count is used before tree is written, so they can safely overlap
- */
-size_t HUF_buildCTable (HUF_CElt* tree, const U32* count, U32 maxSymbolValue, U32 maxNbBits)
-{
-    huffNodeTable nodeTable;
-    return HUF_buildCTable_wksp(tree, count, maxSymbolValue, maxNbBits, nodeTable, sizeof(nodeTable));
-}
-
-static size_t HUF_estimateCompressedSize(HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue)
-{
-    size_t nbBits = 0;
-    int s;
-    for (s = 0; s <= (int)maxSymbolValue; ++s) {
-        nbBits += CTable[s].nbBits * count[s];
-    }
-    return nbBits >> 3;
-}
-
-static int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) {
-  int bad = 0;
-  int s;
-  for (s = 0; s <= (int)maxSymbolValue; ++s) {
-    bad |= (count[s] != 0) & (CTable[s].nbBits == 0);
-  }
-  return !bad;
-}
-
-size_t HUF_compressBound(size_t size) { return HUF_COMPRESSBOUND(size); }
-
-FORCE_INLINE_TEMPLATE void
-HUF_encodeSymbol(BIT_CStream_t* bitCPtr, U32 symbol, const HUF_CElt* CTable)
-{
-    BIT_addBitsFast(bitCPtr, CTable[symbol].val, CTable[symbol].nbBits);
-}
-
-#define HUF_FLUSHBITS(s)  BIT_flushBits(s)
-
-#define HUF_FLUSHBITS_1(stream) \
-    if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*2+7) HUF_FLUSHBITS(stream)
-
-#define HUF_FLUSHBITS_2(stream) \
-    if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*4+7) HUF_FLUSHBITS(stream)
-
-FORCE_INLINE_TEMPLATE size_t
-HUF_compress1X_usingCTable_internal_body(void* dst, size_t dstSize,
-                                   const void* src, size_t srcSize,
-                                   const HUF_CElt* CTable)
-{
-    const BYTE* ip = (const BYTE*) src;
-    BYTE* const ostart = (BYTE*)dst;
-    BYTE* const oend = ostart + dstSize;
-    BYTE* op = ostart;
-    size_t n;
-    BIT_CStream_t bitC;
-
-    /* init */
-    if (dstSize < 8) return 0;   /* not enough space to compress */
-    { size_t const initErr = BIT_initCStream(&bitC, op, oend-op);
-      if (HUF_isError(initErr)) return 0; }
-
-    n = srcSize & ~3;  /* join to mod 4 */
-    switch (srcSize & 3)
-    {
-        case 3 : HUF_encodeSymbol(&bitC, ip[n+ 2], CTable);
-                 HUF_FLUSHBITS_2(&bitC);
-		 /* fall-through */
-        case 2 : HUF_encodeSymbol(&bitC, ip[n+ 1], CTable);
-                 HUF_FLUSHBITS_1(&bitC);
-		 /* fall-through */
-        case 1 : HUF_encodeSymbol(&bitC, ip[n+ 0], CTable);
-                 HUF_FLUSHBITS(&bitC);
-		 /* fall-through */
-        case 0 : /* fall-through */
-        default: break;
-    }
-
-    for (; n>0; n-=4) {  /* note : n&3==0 at this stage */
-        HUF_encodeSymbol(&bitC, ip[n- 1], CTable);
-        HUF_FLUSHBITS_1(&bitC);
-        HUF_encodeSymbol(&bitC, ip[n- 2], CTable);
-        HUF_FLUSHBITS_2(&bitC);
-        HUF_encodeSymbol(&bitC, ip[n- 3], CTable);
-        HUF_FLUSHBITS_1(&bitC);
-        HUF_encodeSymbol(&bitC, ip[n- 4], CTable);
-        HUF_FLUSHBITS(&bitC);
-    }
-
-    return BIT_closeCStream(&bitC);
-}
-
-#if DYNAMIC_BMI2
-
-static TARGET_ATTRIBUTE("bmi2") size_t
-HUF_compress1X_usingCTable_internal_bmi2(void* dst, size_t dstSize,
-                                   const void* src, size_t srcSize,
-                                   const HUF_CElt* CTable)
-{
-    return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
-}
-
-static size_t
-HUF_compress1X_usingCTable_internal_default(void* dst, size_t dstSize,
-                                      const void* src, size_t srcSize,
-                                      const HUF_CElt* CTable)
-{
-    return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
-}
-
-static size_t
-HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
-                              const void* src, size_t srcSize,
-                              const HUF_CElt* CTable, const int bmi2)
-{
-    if (bmi2) {
-        return HUF_compress1X_usingCTable_internal_bmi2(dst, dstSize, src, srcSize, CTable);
-    }
-    return HUF_compress1X_usingCTable_internal_default(dst, dstSize, src, srcSize, CTable);
-}
-
-#else
-
-static size_t
-HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
-                              const void* src, size_t srcSize,
-                              const HUF_CElt* CTable, const int bmi2)
-{
-    (void)bmi2;
-    return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
-}
-
-#endif
-
-size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
-{
-    return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
-}
-
-
-static size_t
-HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
-                              const void* src, size_t srcSize,
-                              const HUF_CElt* CTable, int bmi2)
-{
-    size_t const segmentSize = (srcSize+3)/4;   /* first 3 segments */
-    const BYTE* ip = (const BYTE*) src;
-    const BYTE* const iend = ip + srcSize;
-    BYTE* const ostart = (BYTE*) dst;
-    BYTE* const oend = ostart + dstSize;
-    BYTE* op = ostart;
-
-    if (dstSize < 6 + 1 + 1 + 1 + 8) return 0;   /* minimum space to compress successfully */
-    if (srcSize < 12) return 0;   /* no saving possible : too small input */
-    op += 6;   /* jumpTable */
-
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, segmentSize, CTable, bmi2) );
-        if (cSize==0) return 0;
-        assert(cSize <= 65535);
-        MEM_writeLE16(ostart, (U16)cSize);
-        op += cSize;
-    }
-
-    ip += segmentSize;
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, segmentSize, CTable, bmi2) );
-        if (cSize==0) return 0;
-        assert(cSize <= 65535);
-        MEM_writeLE16(ostart+2, (U16)cSize);
-        op += cSize;
-    }
-
-    ip += segmentSize;
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, segmentSize, CTable, bmi2) );
-        if (cSize==0) return 0;
-        assert(cSize <= 65535);
-        MEM_writeLE16(ostart+4, (U16)cSize);
-        op += cSize;
-    }
-
-    ip += segmentSize;
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, iend-ip, CTable, bmi2) );
-        if (cSize==0) return 0;
-        op += cSize;
-    }
-
-    return op-ostart;
-}
-
-size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
-{
-    return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
-}
-
-
-static size_t HUF_compressCTable_internal(
-                BYTE* const ostart, BYTE* op, BYTE* const oend,
-                const void* src, size_t srcSize,
-                unsigned singleStream, const HUF_CElt* CTable, const int bmi2)
-{
-    size_t const cSize = singleStream ?
-                         HUF_compress1X_usingCTable_internal(op, oend - op, src, srcSize, CTable, bmi2) :
-                         HUF_compress4X_usingCTable_internal(op, oend - op, src, srcSize, CTable, bmi2);
-    if (HUF_isError(cSize)) { return cSize; }
-    if (cSize==0) { return 0; }   /* uncompressible */
-    op += cSize;
-    /* check compressibility */
-    if ((size_t)(op-ostart) >= srcSize-1) { return 0; }
-    return op-ostart;
-}
-
-typedef struct {
-    U32 count[HUF_SYMBOLVALUE_MAX + 1];
-    HUF_CElt CTable[HUF_SYMBOLVALUE_MAX + 1];
-    huffNodeTable nodeTable;
-} HUF_compress_tables_t;
-
-/* HUF_compress_internal() :
- * `workSpace` must a table of at least HUF_WORKSPACE_SIZE_U32 unsigned */
-static size_t HUF_compress_internal (
-                void* dst, size_t dstSize,
-                const void* src, size_t srcSize,
-                unsigned maxSymbolValue, unsigned huffLog,
-                unsigned singleStream,
-                void* workSpace, size_t wkspSize,
-                HUF_CElt* oldHufTable, HUF_repeat* repeat, int preferRepeat,
-                const int bmi2)
-{
-    HUF_compress_tables_t* const table = (HUF_compress_tables_t*)workSpace;
-    BYTE* const ostart = (BYTE*)dst;
-    BYTE* const oend = ostart + dstSize;
-    BYTE* op = ostart;
-
-    /* checks & inits */
-    if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC);  /* must be aligned on 4-bytes boundaries */
-    if (wkspSize < sizeof(*table)) return ERROR(workSpace_tooSmall);
-    if (!srcSize) return 0;  /* Uncompressed */
-    if (!dstSize) return 0;  /* cannot fit anything within dst budget */
-    if (srcSize > HUF_BLOCKSIZE_MAX) return ERROR(srcSize_wrong);   /* current block size limit */
-    if (huffLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
-    if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
-    if (!maxSymbolValue) maxSymbolValue = HUF_SYMBOLVALUE_MAX;
-    if (!huffLog) huffLog = HUF_TABLELOG_DEFAULT;
-
-    /* Heuristic : If old table is valid, use it for small inputs */
-    if (preferRepeat && repeat && *repeat == HUF_repeat_valid) {
-        return HUF_compressCTable_internal(ostart, op, oend,
-                                           src, srcSize,
-                                           singleStream, oldHufTable, bmi2);
-    }
-
-    /* Scan input and build symbol stats */
-    {   CHECK_V_F(largest, FSE_count_wksp (table->count, &maxSymbolValue, (const BYTE*)src, srcSize, table->count) );
-        if (largest == srcSize) { *ostart = ((const BYTE*)src)[0]; return 1; }   /* single symbol, rle */
-        if (largest <= (srcSize >> 7)+1) return 0;   /* heuristic : probably not compressible enough */
-    }
-
-    /* Check validity of previous table */
-    if ( repeat
-      && *repeat == HUF_repeat_check
-      && !HUF_validateCTable(oldHufTable, table->count, maxSymbolValue)) {
-        *repeat = HUF_repeat_none;
-    }
-    /* Heuristic : use existing table for small inputs */
-    if (preferRepeat && repeat && *repeat != HUF_repeat_none) {
-        return HUF_compressCTable_internal(ostart, op, oend,
-                                           src, srcSize,
-                                           singleStream, oldHufTable, bmi2);
-    }
-
-    /* Build Huffman Tree */
-    huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
-    {   CHECK_V_F(maxBits, HUF_buildCTable_wksp(table->CTable, table->count,
-                                                maxSymbolValue, huffLog,
-                                                table->nodeTable, sizeof(table->nodeTable)) );
-        huffLog = (U32)maxBits;
-        /* Zero unused symbols in CTable, so we can check it for validity */
-        memset(table->CTable + (maxSymbolValue + 1), 0,
-               sizeof(table->CTable) - ((maxSymbolValue + 1) * sizeof(HUF_CElt)));
-    }
-
-    /* Write table description header */
-    {   CHECK_V_F(hSize, HUF_writeCTable (op, dstSize, table->CTable, maxSymbolValue, huffLog) );
-        /* Check if using previous huffman table is beneficial */
-        if (repeat && *repeat != HUF_repeat_none) {
-            size_t const oldSize = HUF_estimateCompressedSize(oldHufTable, table->count, maxSymbolValue);
-            size_t const newSize = HUF_estimateCompressedSize(table->CTable, table->count, maxSymbolValue);
-            if (oldSize <= hSize + newSize || hSize + 12 >= srcSize) {
-                return HUF_compressCTable_internal(ostart, op, oend,
-                                                   src, srcSize,
-                                                   singleStream, oldHufTable, bmi2);
-        }   }
-
-        /* Use the new huffman table */
-        if (hSize + 12ul >= srcSize) { return 0; }
-        op += hSize;
-        if (repeat) { *repeat = HUF_repeat_none; }
-        if (oldHufTable)
-            memcpy(oldHufTable, table->CTable, sizeof(table->CTable));  /* Save new table */
-    }
-    return HUF_compressCTable_internal(ostart, op, oend,
-                                       src, srcSize,
-                                       singleStream, table->CTable, bmi2);
-}
-
-
-size_t HUF_compress1X_wksp (void* dst, size_t dstSize,
-                      const void* src, size_t srcSize,
-                      unsigned maxSymbolValue, unsigned huffLog,
-                      void* workSpace, size_t wkspSize)
-{
-    return HUF_compress_internal(dst, dstSize, src, srcSize,
-                                 maxSymbolValue, huffLog, 1 /*single stream*/,
-                                 workSpace, wkspSize,
-                                 NULL, NULL, 0, 0 /*bmi2*/);
-}
-
-size_t HUF_compress1X_repeat (void* dst, size_t dstSize,
-                      const void* src, size_t srcSize,
-                      unsigned maxSymbolValue, unsigned huffLog,
-                      void* workSpace, size_t wkspSize,
-                      HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
-{
-    return HUF_compress_internal(dst, dstSize, src, srcSize,
-                                 maxSymbolValue, huffLog, 1 /*single stream*/,
-                                 workSpace, wkspSize, hufTable,
-                                 repeat, preferRepeat, bmi2);
-}
-
-size_t HUF_compress1X (void* dst, size_t dstSize,
-                 const void* src, size_t srcSize,
-                 unsigned maxSymbolValue, unsigned huffLog)
-{
-    unsigned workSpace[HUF_WORKSPACE_SIZE_U32];
-    return HUF_compress1X_wksp(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, workSpace, sizeof(workSpace));
-}
-
-/* HUF_compress4X_repeat():
- * compress input using 4 streams.
- * provide workspace to generate compression tables */
-size_t HUF_compress4X_wksp (void* dst, size_t dstSize,
-                      const void* src, size_t srcSize,
-                      unsigned maxSymbolValue, unsigned huffLog,
-                      void* workSpace, size_t wkspSize)
-{
-    return HUF_compress_internal(dst, dstSize, src, srcSize,
-                                 maxSymbolValue, huffLog, 0 /*4 streams*/,
-                                 workSpace, wkspSize,
-                                 NULL, NULL, 0, 0 /*bmi2*/);
-}
-
-/* HUF_compress4X_repeat():
- * compress input using 4 streams.
- * re-use an existing huffman compression table */
-size_t HUF_compress4X_repeat (void* dst, size_t dstSize,
-                      const void* src, size_t srcSize,
-                      unsigned maxSymbolValue, unsigned huffLog,
-                      void* workSpace, size_t wkspSize,
-                      HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
-{
-    return HUF_compress_internal(dst, dstSize, src, srcSize,
-                                 maxSymbolValue, huffLog, 0 /* 4 streams */,
-                                 workSpace, wkspSize,
-                                 hufTable, repeat, preferRepeat, bmi2);
-}
-
-size_t HUF_compress2 (void* dst, size_t dstSize,
-                const void* src, size_t srcSize,
-                unsigned maxSymbolValue, unsigned huffLog)
-{
-    unsigned workSpace[HUF_WORKSPACE_SIZE_U32];
-    return HUF_compress4X_wksp(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, workSpace, sizeof(workSpace));
-}
-
-size_t HUF_compress (void* dst, size_t maxDstSize, const void* src, size_t srcSize)
-{
-    return HUF_compress2(dst, maxDstSize, src, srcSize, 255, HUF_TABLELOG_DEFAULT);
-}

third/github.com/DataDog/zstd/mem.h

@@ -1,362 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef MEM_H_MODULE
-#define MEM_H_MODULE
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-/*-****************************************
-*  Dependencies
-******************************************/
-#include <stddef.h>     /* size_t, ptrdiff_t */
-#include <string.h>     /* memcpy */
-
-
-/*-****************************************
-*  Compiler specifics
-******************************************/
-#if defined(_MSC_VER)   /* Visual Studio */
-#   include <stdlib.h>  /* _byteswap_ulong */
-#   include <intrin.h>  /* _byteswap_* */
-#endif
-#if defined(__GNUC__)
-#  define MEM_STATIC static __inline __attribute__((unused))
-#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
-#  define MEM_STATIC static inline
-#elif defined(_MSC_VER)
-#  define MEM_STATIC static __inline
-#else
-#  define MEM_STATIC static  /* this version may generate warnings for unused static functions; disable the relevant warning */
-#endif
-
-/* code only tested on 32 and 64 bits systems */
-#define MEM_STATIC_ASSERT(c)   { enum { MEM_static_assert = 1/(int)(!!(c)) }; }
-MEM_STATIC void MEM_check(void) { MEM_STATIC_ASSERT((sizeof(size_t)==4) || (sizeof(size_t)==8)); }
-
-
-/*-**************************************************************
-*  Basic Types
-*****************************************************************/
-#if  !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
-# include <stdint.h>
-  typedef   uint8_t BYTE;
-  typedef  uint16_t U16;
-  typedef   int16_t S16;
-  typedef  uint32_t U32;
-  typedef   int32_t S32;
-  typedef  uint64_t U64;
-  typedef   int64_t S64;
-#else
-  typedef unsigned char      BYTE;
-  typedef unsigned short      U16;
-  typedef   signed short      S16;
-  typedef unsigned int        U32;
-  typedef   signed int        S32;
-  typedef unsigned long long  U64;
-  typedef   signed long long  S64;
-#endif
-
-
-/*-**************************************************************
-*  Memory I/O
-*****************************************************************/
-/* MEM_FORCE_MEMORY_ACCESS :
- * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
- * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
- * The below switch allow to select different access method for improved performance.
- * Method 0 (default) : use `memcpy()`. Safe and portable.
- * Method 1 : `__packed` statement. It depends on compiler extension (i.e., not portable).
- *            This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`.
- * Method 2 : direct access. This method is portable but violate C standard.
- *            It can generate buggy code on targets depending on alignment.
- *            In some circumstances, it's the only known way to get the most performance (i.e. GCC + ARMv6)
- * See http://fastcompression.blogspot.fr/2015/08/accessing-unaligned-memory.html for details.
- * Prefer these methods in priority order (0 > 1 > 2)
- */
-#ifndef MEM_FORCE_MEMORY_ACCESS   /* can be defined externally, on command line for example */
-#  if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
-#    define MEM_FORCE_MEMORY_ACCESS 2
-#  elif defined(__INTEL_COMPILER) || defined(__GNUC__)
-#    define MEM_FORCE_MEMORY_ACCESS 1
-#  endif
-#endif
-
-MEM_STATIC unsigned MEM_32bits(void) { return sizeof(size_t)==4; }
-MEM_STATIC unsigned MEM_64bits(void) { return sizeof(size_t)==8; }
-
-MEM_STATIC unsigned MEM_isLittleEndian(void)
-{
-    const union { U32 u; BYTE c[4]; } one = { 1 };   /* don't use static : performance detrimental  */
-    return one.c[0];
-}
-
-#if defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==2)
-
-/* violates C standard, by lying on structure alignment.
-Only use if no other choice to achieve best performance on target platform */
-MEM_STATIC U16 MEM_read16(const void* memPtr) { return *(const U16*) memPtr; }
-MEM_STATIC U32 MEM_read32(const void* memPtr) { return *(const U32*) memPtr; }
-MEM_STATIC U64 MEM_read64(const void* memPtr) { return *(const U64*) memPtr; }
-MEM_STATIC size_t MEM_readST(const void* memPtr) { return *(const size_t*) memPtr; }
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; }
-MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; }
-MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(U64*)memPtr = value; }
-
-#elif defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==1)
-
-/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
-/* currently only defined for gcc and icc */
-#if defined(_MSC_VER) || (defined(__INTEL_COMPILER) && defined(WIN32))
-    __pragma( pack(push, 1) )
-    typedef struct { U16 v; } unalign16;
-    typedef struct { U32 v; } unalign32;
-    typedef struct { U64 v; } unalign64;
-    typedef struct { size_t v; } unalignArch;
-    __pragma( pack(pop) )
-#else
-    typedef struct { U16 v; } __attribute__((packed)) unalign16;
-    typedef struct { U32 v; } __attribute__((packed)) unalign32;
-    typedef struct { U64 v; } __attribute__((packed)) unalign64;
-    typedef struct { size_t v; } __attribute__((packed)) unalignArch;
-#endif
-
-MEM_STATIC U16 MEM_read16(const void* ptr) { return ((const unalign16*)ptr)->v; }
-MEM_STATIC U32 MEM_read32(const void* ptr) { return ((const unalign32*)ptr)->v; }
-MEM_STATIC U64 MEM_read64(const void* ptr) { return ((const unalign64*)ptr)->v; }
-MEM_STATIC size_t MEM_readST(const void* ptr) { return ((const unalignArch*)ptr)->v; }
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value) { ((unalign16*)memPtr)->v = value; }
-MEM_STATIC void MEM_write32(void* memPtr, U32 value) { ((unalign32*)memPtr)->v = value; }
-MEM_STATIC void MEM_write64(void* memPtr, U64 value) { ((unalign64*)memPtr)->v = value; }
-
-#else
-
-/* default method, safe and standard.
-   can sometimes prove slower */
-
-MEM_STATIC U16 MEM_read16(const void* memPtr)
-{
-    U16 val; memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC U32 MEM_read32(const void* memPtr)
-{
-    U32 val; memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC U64 MEM_read64(const void* memPtr)
-{
-    U64 val; memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC size_t MEM_readST(const void* memPtr)
-{
-    size_t val; memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value)
-{
-    memcpy(memPtr, &value, sizeof(value));
-}
-
-MEM_STATIC void MEM_write32(void* memPtr, U32 value)
-{
-    memcpy(memPtr, &value, sizeof(value));
-}
-
-MEM_STATIC void MEM_write64(void* memPtr, U64 value)
-{
-    memcpy(memPtr, &value, sizeof(value));
-}
-
-#endif /* MEM_FORCE_MEMORY_ACCESS */
-
-MEM_STATIC U32 MEM_swap32(U32 in)
-{
-#if defined(_MSC_VER)     /* Visual Studio */
-    return _byteswap_ulong(in);
-#elif defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)
-    return __builtin_bswap32(in);
-#else
-    return  ((in << 24) & 0xff000000 ) |
-            ((in <<  8) & 0x00ff0000 ) |
-            ((in >>  8) & 0x0000ff00 ) |
-            ((in >> 24) & 0x000000ff );
-#endif
-}
-
-MEM_STATIC U64 MEM_swap64(U64 in)
-{
-#if defined(_MSC_VER)     /* Visual Studio */
-    return _byteswap_uint64(in);
-#elif defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)
-    return __builtin_bswap64(in);
-#else
-    return  ((in << 56) & 0xff00000000000000ULL) |
-            ((in << 40) & 0x00ff000000000000ULL) |
-            ((in << 24) & 0x0000ff0000000000ULL) |
-            ((in << 8)  & 0x000000ff00000000ULL) |
-            ((in >> 8)  & 0x00000000ff000000ULL) |
-            ((in >> 24) & 0x0000000000ff0000ULL) |
-            ((in >> 40) & 0x000000000000ff00ULL) |
-            ((in >> 56) & 0x00000000000000ffULL);
-#endif
-}
-
-MEM_STATIC size_t MEM_swapST(size_t in)
-{
-    if (MEM_32bits())
-        return (size_t)MEM_swap32((U32)in);
-    else
-        return (size_t)MEM_swap64((U64)in);
-}
-
-/*=== Little endian r/w ===*/
-
-MEM_STATIC U16 MEM_readLE16(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_read16(memPtr);
-    else {
-        const BYTE* p = (const BYTE*)memPtr;
-        return (U16)(p[0] + (p[1]<<8));
-    }
-}
-
-MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val)
-{
-    if (MEM_isLittleEndian()) {
-        MEM_write16(memPtr, val);
-    } else {
-        BYTE* p = (BYTE*)memPtr;
-        p[0] = (BYTE)val;
-        p[1] = (BYTE)(val>>8);
-    }
-}
-
-MEM_STATIC U32 MEM_readLE24(const void* memPtr)
-{
-    return MEM_readLE16(memPtr) + (((const BYTE*)memPtr)[2] << 16);
-}
-
-MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val)
-{
-    MEM_writeLE16(memPtr, (U16)val);
-    ((BYTE*)memPtr)[2] = (BYTE)(val>>16);
-}
-
-MEM_STATIC U32 MEM_readLE32(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_read32(memPtr);
-    else
-        return MEM_swap32(MEM_read32(memPtr));
-}
-
-MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32)
-{
-    if (MEM_isLittleEndian())
-        MEM_write32(memPtr, val32);
-    else
-        MEM_write32(memPtr, MEM_swap32(val32));
-}
-
-MEM_STATIC U64 MEM_readLE64(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_read64(memPtr);
-    else
-        return MEM_swap64(MEM_read64(memPtr));
-}
-
-MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64)
-{
-    if (MEM_isLittleEndian())
-        MEM_write64(memPtr, val64);
-    else
-        MEM_write64(memPtr, MEM_swap64(val64));
-}
-
-MEM_STATIC size_t MEM_readLEST(const void* memPtr)
-{
-    if (MEM_32bits())
-        return (size_t)MEM_readLE32(memPtr);
-    else
-        return (size_t)MEM_readLE64(memPtr);
-}
-
-MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val)
-{
-    if (MEM_32bits())
-        MEM_writeLE32(memPtr, (U32)val);
-    else
-        MEM_writeLE64(memPtr, (U64)val);
-}
-
-/*=== Big endian r/w ===*/
-
-MEM_STATIC U32 MEM_readBE32(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_swap32(MEM_read32(memPtr));
-    else
-        return MEM_read32(memPtr);
-}
-
-MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32)
-{
-    if (MEM_isLittleEndian())
-        MEM_write32(memPtr, MEM_swap32(val32));
-    else
-        MEM_write32(memPtr, val32);
-}
-
-MEM_STATIC U64 MEM_readBE64(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_swap64(MEM_read64(memPtr));
-    else
-        return MEM_read64(memPtr);
-}
-
-MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64)
-{
-    if (MEM_isLittleEndian())
-        MEM_write64(memPtr, MEM_swap64(val64));
-    else
-        MEM_write64(memPtr, val64);
-}
-
-MEM_STATIC size_t MEM_readBEST(const void* memPtr)
-{
-    if (MEM_32bits())
-        return (size_t)MEM_readBE32(memPtr);
-    else
-        return (size_t)MEM_readBE64(memPtr);
-}
-
-MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val)
-{
-    if (MEM_32bits())
-        MEM_writeBE32(memPtr, (U32)val);
-    else
-        MEM_writeBE64(memPtr, (U64)val);
-}
-
-
-#if defined (__cplusplus)
-}
-#endif
-
-#endif /* MEM_H_MODULE */

third/github.com/DataDog/zstd/pool.c

@@ -1,283 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/* ======   Dependencies   ======= */
-#include <stddef.h>  /* size_t */
-#include "pool.h"
-#include "zstd_internal.h"  /* ZSTD_malloc, ZSTD_free */
-
-/* ======   Compiler specifics   ====== */
-#if defined(_MSC_VER)
-#  pragma warning(disable : 4204)        /* disable: C4204: non-constant aggregate initializer */
-#endif
-
-
-#ifdef ZSTD_MULTITHREAD
-
-#include "threading.h"   /* pthread adaptation */
-
-/* A job is a function and an opaque argument */
-typedef struct POOL_job_s {
-    POOL_function function;
-    void *opaque;
-} POOL_job;
-
-struct POOL_ctx_s {
-    ZSTD_customMem customMem;
-    /* Keep track of the threads */
-    ZSTD_pthread_t *threads;
-    size_t numThreads;
-
-    /* The queue is a circular buffer */
-    POOL_job *queue;
-    size_t queueHead;
-    size_t queueTail;
-    size_t queueSize;
-
-    /* The number of threads working on jobs */
-    size_t numThreadsBusy;
-    /* Indicates if the queue is empty */
-    int queueEmpty;
-
-    /* The mutex protects the queue */
-    ZSTD_pthread_mutex_t queueMutex;
-    /* Condition variable for pushers to wait on when the queue is full */
-    ZSTD_pthread_cond_t queuePushCond;
-    /* Condition variables for poppers to wait on when the queue is empty */
-    ZSTD_pthread_cond_t queuePopCond;
-    /* Indicates if the queue is shutting down */
-    int shutdown;
-};
-
-/* POOL_thread() :
-   Work thread for the thread pool.
-   Waits for jobs and executes them.
-   @returns : NULL on failure else non-null.
-*/
-static void* POOL_thread(void* opaque) {
-    POOL_ctx* const ctx = (POOL_ctx*)opaque;
-    if (!ctx) { return NULL; }
-    for (;;) {
-        /* Lock the mutex and wait for a non-empty queue or until shutdown */
-        ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-
-        while (ctx->queueEmpty && !ctx->shutdown) {
-            ZSTD_pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
-        }
-        /* empty => shutting down: so stop */
-        if (ctx->queueEmpty) {
-            ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-            return opaque;
-        }
-        /* Pop a job off the queue */
-        {   POOL_job const job = ctx->queue[ctx->queueHead];
-            ctx->queueHead = (ctx->queueHead + 1) % ctx->queueSize;
-            ctx->numThreadsBusy++;
-            ctx->queueEmpty = ctx->queueHead == ctx->queueTail;
-            /* Unlock the mutex, signal a pusher, and run the job */
-            ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-            ZSTD_pthread_cond_signal(&ctx->queuePushCond);
-
-            job.function(job.opaque);
-
-            /* If the intended queue size was 0, signal after finishing job */
-            if (ctx->queueSize == 1) {
-                ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-                ctx->numThreadsBusy--;
-                ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-                ZSTD_pthread_cond_signal(&ctx->queuePushCond);
-        }   }
-    }  /* for (;;) */
-    /* Unreachable */
-}
-
-POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
-    return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
-}
-
-POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
-    POOL_ctx* ctx;
-    /* Check the parameters */
-    if (!numThreads) { return NULL; }
-    /* Allocate the context and zero initialize */
-    ctx = (POOL_ctx*)ZSTD_calloc(sizeof(POOL_ctx), customMem);
-    if (!ctx) { return NULL; }
-    /* Initialize the job queue.
-     * It needs one extra space since one space is wasted to differentiate empty
-     * and full queues.
-     */
-    ctx->queueSize = queueSize + 1;
-    ctx->queue = (POOL_job*)ZSTD_malloc(ctx->queueSize * sizeof(POOL_job), customMem);
-    ctx->queueHead = 0;
-    ctx->queueTail = 0;
-    ctx->numThreadsBusy = 0;
-    ctx->queueEmpty = 1;
-    (void)ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL);
-    (void)ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL);
-    (void)ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL);
-    ctx->shutdown = 0;
-    /* Allocate space for the thread handles */
-    ctx->threads = (ZSTD_pthread_t*)ZSTD_malloc(numThreads * sizeof(ZSTD_pthread_t), customMem);
-    ctx->numThreads = 0;
-    ctx->customMem = customMem;
-    /* Check for errors */
-    if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; }
-    /* Initialize the threads */
-    {   size_t i;
-        for (i = 0; i < numThreads; ++i) {
-            if (ZSTD_pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) {
-                ctx->numThreads = i;
-                POOL_free(ctx);
-                return NULL;
-        }   }
-        ctx->numThreads = numThreads;
-    }
-    return ctx;
-}
-
-/*! POOL_join() :
-    Shutdown the queue, wake any sleeping threads, and join all of the threads.
-*/
-static void POOL_join(POOL_ctx* ctx) {
-    /* Shut down the queue */
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    ctx->shutdown = 1;
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-    /* Wake up sleeping threads */
-    ZSTD_pthread_cond_broadcast(&ctx->queuePushCond);
-    ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
-    /* Join all of the threads */
-    {   size_t i;
-        for (i = 0; i < ctx->numThreads; ++i) {
-            ZSTD_pthread_join(ctx->threads[i], NULL);
-    }   }
-}
-
-void POOL_free(POOL_ctx *ctx) {
-    if (!ctx) { return; }
-    POOL_join(ctx);
-    ZSTD_pthread_mutex_destroy(&ctx->queueMutex);
-    ZSTD_pthread_cond_destroy(&ctx->queuePushCond);
-    ZSTD_pthread_cond_destroy(&ctx->queuePopCond);
-    ZSTD_free(ctx->queue, ctx->customMem);
-    ZSTD_free(ctx->threads, ctx->customMem);
-    ZSTD_free(ctx, ctx->customMem);
-}
-
-size_t POOL_sizeof(POOL_ctx *ctx) {
-    if (ctx==NULL) return 0;  /* supports sizeof NULL */
-    return sizeof(*ctx)
-        + ctx->queueSize * sizeof(POOL_job)
-        + ctx->numThreads * sizeof(ZSTD_pthread_t);
-}
-
-/**
- * Returns 1 if the queue is full and 0 otherwise.
- *
- * If the queueSize is 1 (the pool was created with an intended queueSize of 0),
- * then a queue is empty if there is a thread free and no job is waiting.
- */
-static int isQueueFull(POOL_ctx const* ctx) {
-    if (ctx->queueSize > 1) {
-        return ctx->queueHead == ((ctx->queueTail + 1) % ctx->queueSize);
-    } else {
-        return ctx->numThreadsBusy == ctx->numThreads ||
-               !ctx->queueEmpty;
-    }
-}
-
-
-static void POOL_add_internal(POOL_ctx* ctx, POOL_function function, void *opaque)
-{
-    POOL_job const job = {function, opaque};
-    assert(ctx != NULL);
-    if (ctx->shutdown) return;
-
-    ctx->queueEmpty = 0;
-    ctx->queue[ctx->queueTail] = job;
-    ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize;
-    ZSTD_pthread_cond_signal(&ctx->queuePopCond);
-}
-
-void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque)
-{
-    assert(ctx != NULL);
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    /* Wait until there is space in the queue for the new job */
-    while (isQueueFull(ctx) && (!ctx->shutdown)) {
-        ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
-    }
-    POOL_add_internal(ctx, function, opaque);
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-}
-
-
-int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque)
-{
-    assert(ctx != NULL);
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    if (isQueueFull(ctx)) {
-        ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-        return 0;
-    }
-    POOL_add_internal(ctx, function, opaque);
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-    return 1;
-}
-
-
-#else  /* ZSTD_MULTITHREAD  not defined */
-
-/* ========================== */
-/* No multi-threading support */
-/* ========================== */
-
-
-/* We don't need any data, but if it is empty, malloc() might return NULL. */
-struct POOL_ctx_s {
-    int dummy;
-};
-static POOL_ctx g_ctx;
-
-POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
-    return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
-}
-
-POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
-    (void)numThreads;
-    (void)queueSize;
-    (void)customMem;
-    return &g_ctx;
-}
-
-void POOL_free(POOL_ctx* ctx) {
-    assert(!ctx || ctx == &g_ctx);
-    (void)ctx;
-}
-
-void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque) {
-    (void)ctx;
-    function(opaque);
-}
-
-int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) {
-    (void)ctx;
-    function(opaque);
-    return 1;
-}
-
-size_t POOL_sizeof(POOL_ctx* ctx) {
-    if (ctx==NULL) return 0;  /* supports sizeof NULL */
-    assert(ctx == &g_ctx);
-    return sizeof(*ctx);
-}
-
-#endif  /* ZSTD_MULTITHREAD */

third/github.com/DataDog/zstd/pool.h

@@ -1,74 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef POOL_H
-#define POOL_H
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-
-#include <stddef.h>   /* size_t */
-#define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_customMem */
-#include "zstd.h"
-
-typedef struct POOL_ctx_s POOL_ctx;
-
-/*! POOL_create() :
- *  Create a thread pool with at most `numThreads` threads.
- * `numThreads` must be at least 1.
- *  The maximum number of queued jobs before blocking is `queueSize`.
- * @return : POOL_ctx pointer on success, else NULL.
-*/
-POOL_ctx* POOL_create(size_t numThreads, size_t queueSize);
-
-POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem);
-
-/*! POOL_free() :
-    Free a thread pool returned by POOL_create().
-*/
-void POOL_free(POOL_ctx* ctx);
-
-/*! POOL_sizeof() :
-    return memory usage of pool returned by POOL_create().
-*/
-size_t POOL_sizeof(POOL_ctx* ctx);
-
-/*! POOL_function :
-    The function type that can be added to a thread pool.
-*/
-typedef void (*POOL_function)(void*);
-/*! POOL_add_function :
-    The function type for a generic thread pool add function.
-*/
-typedef void (*POOL_add_function)(void*, POOL_function, void*);
-
-/*! POOL_add() :
-    Add the job `function(opaque)` to the thread pool. `ctx` must be valid.
-    Possibly blocks until there is room in the queue.
-    Note : The function may be executed asynchronously, so `opaque` must live until the function has been completed.
-*/
-void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque);
-
-
-/*! POOL_tryAdd() :
-    Add the job `function(opaque)` to the thread pool if a worker is available.
-    return immediately otherwise.
-   @return : 1 if successful, 0 if not.
-*/
-int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque);
-
-
-#if defined (__cplusplus)
-}
-#endif
-
-#endif

third/github.com/DataDog/zstd/threading.c

@@ -1,75 +0,0 @@
-/**
- * Copyright (c) 2016 Tino Reichardt
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- *
- * You can contact the author at:
- * - zstdmt source repository: https://github.com/mcmilk/zstdmt
- */
-
-/**
- * This file will hold wrapper for systems, which do not support pthreads
- */
-
-/* create fake symbol to avoid empty trnaslation unit warning */
-int g_ZSTD_threading_useles_symbol;
-
-#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
-
-/**
- * Windows minimalist Pthread Wrapper, based on :
- * http://www.cse.wustl.edu/~schmidt/win32-cv-1.html
- */
-
-
-/* ===  Dependencies  === */
-#include <process.h>
-#include <errno.h>
-#include "threading.h"
-
-
-/* ===  Implementation  === */
-
-static unsigned __stdcall worker(void *arg)
-{
-    ZSTD_pthread_t* const thread = (ZSTD_pthread_t*) arg;
-    thread->arg = thread->start_routine(thread->arg);
-    return 0;
-}
-
-int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
-            void* (*start_routine) (void*), void* arg)
-{
-    (void)unused;
-    thread->arg = arg;
-    thread->start_routine = start_routine;
-    thread->handle = (HANDLE) _beginthreadex(NULL, 0, worker, thread, 0, NULL);
-
-    if (!thread->handle)
-        return errno;
-    else
-        return 0;
-}
-
-int ZSTD_pthread_join(ZSTD_pthread_t thread, void **value_ptr)
-{
-    DWORD result;
-
-    if (!thread.handle) return 0;
-
-    result = WaitForSingleObject(thread.handle, INFINITE);
-    switch (result) {
-    case WAIT_OBJECT_0:
-        if (value_ptr) *value_ptr = thread.arg;
-        return 0;
-    case WAIT_ABANDONED:
-        return EINVAL;
-    default:
-        return GetLastError();
-    }
-}
-
-#endif   /* ZSTD_MULTITHREAD */

third/github.com/DataDog/zstd/threading.h

@@ -1,123 +0,0 @@
-/**
- * Copyright (c) 2016 Tino Reichardt
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- *
- * You can contact the author at:
- * - zstdmt source repository: https://github.com/mcmilk/zstdmt
- */
-
-#ifndef THREADING_H_938743
-#define THREADING_H_938743
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
-
-/**
- * Windows minimalist Pthread Wrapper, based on :
- * http://www.cse.wustl.edu/~schmidt/win32-cv-1.html
- */
-#ifdef WINVER
-#  undef WINVER
-#endif
-#define WINVER       0x0600
-
-#ifdef _WIN32_WINNT
-#  undef _WIN32_WINNT
-#endif
-#define _WIN32_WINNT 0x0600
-
-#ifndef WIN32_LEAN_AND_MEAN
-#  define WIN32_LEAN_AND_MEAN
-#endif
-
-#undef ERROR   /* reported already defined on VS 2015 (Rich Geldreich) */
-#include <windows.h>
-#undef ERROR
-#define ERROR(name) ZSTD_ERROR(name)
-
-
-/* mutex */
-#define ZSTD_pthread_mutex_t           CRITICAL_SECTION
-#define ZSTD_pthread_mutex_init(a, b)  ((void)(b), InitializeCriticalSection((a)), 0)
-#define ZSTD_pthread_mutex_destroy(a)  DeleteCriticalSection((a))
-#define ZSTD_pthread_mutex_lock(a)     EnterCriticalSection((a))
-#define ZSTD_pthread_mutex_unlock(a)   LeaveCriticalSection((a))
-
-/* condition variable */
-#define ZSTD_pthread_cond_t             CONDITION_VARIABLE
-#define ZSTD_pthread_cond_init(a, b)    ((void)(b), InitializeConditionVariable((a)), 0)
-#define ZSTD_pthread_cond_destroy(a)    ((void)(a))
-#define ZSTD_pthread_cond_wait(a, b)    SleepConditionVariableCS((a), (b), INFINITE)
-#define ZSTD_pthread_cond_signal(a)     WakeConditionVariable((a))
-#define ZSTD_pthread_cond_broadcast(a)  WakeAllConditionVariable((a))
-
-/* ZSTD_pthread_create() and ZSTD_pthread_join() */
-typedef struct {
-    HANDLE handle;
-    void* (*start_routine)(void*);
-    void* arg;
-} ZSTD_pthread_t;
-
-int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
-                   void* (*start_routine) (void*), void* arg);
-
-int ZSTD_pthread_join(ZSTD_pthread_t thread, void** value_ptr);
-
-/**
- * add here more wrappers as required
- */
-
-
-#elif defined(ZSTD_MULTITHREAD)   /* posix assumed ; need a better detection method */
-/* ===   POSIX Systems   === */
-#  include <pthread.h>
-
-#define ZSTD_pthread_mutex_t            pthread_mutex_t
-#define ZSTD_pthread_mutex_init(a, b)   pthread_mutex_init((a), (b))
-#define ZSTD_pthread_mutex_destroy(a)   pthread_mutex_destroy((a))
-#define ZSTD_pthread_mutex_lock(a)      pthread_mutex_lock((a))
-#define ZSTD_pthread_mutex_unlock(a)    pthread_mutex_unlock((a))
-
-#define ZSTD_pthread_cond_t             pthread_cond_t
-#define ZSTD_pthread_cond_init(a, b)    pthread_cond_init((a), (b))
-#define ZSTD_pthread_cond_destroy(a)    pthread_cond_destroy((a))
-#define ZSTD_pthread_cond_wait(a, b)    pthread_cond_wait((a), (b))
-#define ZSTD_pthread_cond_signal(a)     pthread_cond_signal((a))
-#define ZSTD_pthread_cond_broadcast(a)  pthread_cond_broadcast((a))
-
-#define ZSTD_pthread_t                  pthread_t
-#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
-#define ZSTD_pthread_join(a, b)         pthread_join((a),(b))
-
-#else  /* ZSTD_MULTITHREAD not defined */
-/* No multithreading support */
-
-typedef int ZSTD_pthread_mutex_t;
-#define ZSTD_pthread_mutex_init(a, b)   ((void)(a), (void)(b), 0)
-#define ZSTD_pthread_mutex_destroy(a)   ((void)(a))
-#define ZSTD_pthread_mutex_lock(a)      ((void)(a))
-#define ZSTD_pthread_mutex_unlock(a)    ((void)(a))
-
-typedef int ZSTD_pthread_cond_t;
-#define ZSTD_pthread_cond_init(a, b)    ((void)(a), (void)(b), 0)
-#define ZSTD_pthread_cond_destroy(a)    ((void)(a))
-#define ZSTD_pthread_cond_wait(a, b)    ((void)(a), (void)(b))
-#define ZSTD_pthread_cond_signal(a)     ((void)(a))
-#define ZSTD_pthread_cond_broadcast(a)  ((void)(a))
-
-/* do not use ZSTD_pthread_t */
-
-#endif /* ZSTD_MULTITHREAD */
-
-#if defined (__cplusplus)
-}
-#endif
-
-#endif /* THREADING_H_938743 */