In this article, I share with you a simple C++ class that wraps the string tokenization function strtok(). The QuickTokenizer class makes it easy to tokenize and enumerate the tokens in a thread-safe manner.
The class code
//////////////////////////////////////////////////////////////////////////
class QuickTokenizer
{
private:
char *buf;
char *token;
char *ctx;
void FreeBuffers()
{
if (this->token != NULL)
{
free(this->token);
this->token = NULL;
}
if (this->buf != NULL)
{
free(this->buf);
this->buf = NULL;
}
}
public:
QuickTokenizer() : buf(NULL), token(NULL)
{
}
const char *Tokenize(
const char *str,
const char *tok)
{
this->buf = _strdup(str);
this->token = _strdup(tok);
return strtok_s(buf, token, &ctx);
}
const char *NextToken()
{
return strtok_s(NULL, token, &ctx);
}
~QuickTokenizer()
{
FreeBuffers();
}
};
Do you want to master Batch Files programming? Look no further, the Batchography is the right book for you.
Available in print or e-book editions from Amazon.
Explanation and usage example
- Unlike strtok(), this class takes a copy of the string to be tokenized and thus the source string is a “const char*” instead of “char *”
- The class uses the safe version of strtok() which is strtok_s(). This allows you to do thread-safe tokenization
- Using the C++ destructor, you don’t have to worry about memory allocation / deallocation
Example
The following example, tokenizes a semi-colon separated string and iterates over each token:
void test()
{
char *in_str = "a;b;c;d;e;f;g;h";
QuickTokenizer qt;
for (const char *t = qt.Tokenize(in_str, ";"); t != NULL; t = qt.NextToken())
{
printf("token is: %s\n", t);
}
}
You might also like: