ToolsTokenize Method (String)

Splits the given text into individual atomic words, irrespective of punctuation and other marks.

Namespace: Accord.MachineLearning
Assemblies: Accord.MachineLearning (in Accord.MachineLearning.dll) Version: 3.8.0
Accord.Text (in Accord.Text.dll) Version: 3.8.0

Syntax

Copy

public static string[][] Tokenize(
	this string[] x
)

<ExtensionAttribute>
Public Shared Function Tokenize ( 
	x As String()
) As String()()

Request Example View Source

Parameters

x: Type: SystemString

Return Value

Type: String

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type . When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

Tools Class

Tokenize Overload

Accord.MachineLearning Namespace

ToolsAddLanguageSpecificTextSet("LSTCAEE6538_0?cpp=::|nu=.");Tokenize Method (AddLanguageSpecificTextSet("LSTCAEE6538_1?cpp=array&lt;");StringAddLanguageSpecificTextSet("LSTCAEE6538_2?cpp=&gt;|vb=()|nu=[]");)

Parameters

Return Value

Usage Note

Reference

ToolsTokenize Method (String)