Latest 1.1.0
Homepage https://github.com/Meniny/Tokenizer
License MIT
Platforms ios 8.0, osx 10.10, tvos 9.0, watchos 2.0
Frameworks Foundation
Authors




VersionAuthorBuild PassingSwift
PlatformsMIT
CocoapodsCarthageSPM

Introduction

What’s this?

Tokenizer is a tiny human language tokenizer framework in Swift..

Preview

Preview

Installation

CocoaPods

pod 'Tokenizer'

Contribution

You are welcome to fork and submit pull requests.

License

Tokenizer is open-sourced software, licensed under the MIT license.

Usage

import Tokenizer

let stackoverflow = "Stack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers."
let baidu = "全球最大的中文搜索引擎、最大的中文网站。1999年底,身在美国硅谷的李彦宏看到了中国互联网及中文搜索引擎服务的巨大发展潜力,抱着技术改变世界的梦想,他毅然辞掉硅谷的高薪工作,携搜索引擎专利技术,于 2000年1月1日在中关村创建了百度公司。"

dump(stackoverflow.tokenize())
dump(Tokenizer.tokenize(text: baidu))

Latest podspec

{
    "name": "Tokenizer",
    "version": "1.1.0",
    "summary": "A tiny human language tokenizer framework in Swift.",
    "description": "Tokenizer is a tiny human language tokenizer framework in Swift..",
    "homepage": "https://github.com/Meniny/Tokenizer",
    "license": "MIT",
    "authors": {
        "Elias Abel": "[email protected]"
    },
    "source": {
        "git": "https://github.com/Meniny/Tokenizer.git",
        "tag": "1.1.0"
    },
    "social_media_url": "https://meniny.cn/",
    "platforms": {
        "ios": "8.0",
        "osx": "10.10",
        "tvos": "9.0",
        "watchos": "2.0"
    },
    "source_files": "Tokenizer/**/*.{h,swift}",
    "public_header_files": "Tokenizer/**/*{.h}",
    "frameworks": "Foundation",
    "pushed_with_swift_version": "4.0"
}

Pin It on Pinterest

Share This