utf8.h File Reference

UTF-8 information and validation functions. More...

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Enumerations
enum	ast_utf8_validation_result { AST_UTF8_VALID, AST_UTF8_INVALID, AST_UTF8_UNKNOWN }

Functions
void	ast_utf8_copy_string (char *dst, const char *src, size_t size)
	Copy a string safely ensuring valid UTF-8. More...
int	ast_utf8_init (void)
	Register UTF-8 tests. More...
int	ast_utf8_is_valid (const char *str)
	Check if a zero-terminated string is valid UTF-8. More...
int	ast_utf8_is_validn (const char *str, size_t size)
	Check if the first size bytes of a string are valid UTF-8. More...
void	ast_utf8_validator_destroy (struct ast_utf8_validator *validator)
	Destroy a UTF-8 validator. More...
enum ast_utf8_validation_result	ast_utf8_validator_feed (struct ast_utf8_validator *validator, const char *data)
	Feed a zero-terminated string into the UTF-8 validator. More...
enum ast_utf8_validation_result	ast_utf8_validator_feedn (struct ast_utf8_validator *validator, const char *data, size_t size)
	Feed a string into the UTF-8 validator. More...
int	ast_utf8_validator_new (struct ast_utf8_validator **validator)
	Create a new UTF-8 validator. More...
void	ast_utf8_validator_reset (struct ast_utf8_validator *validator)
	Reset the state of a UTF-8 validator. More...
enum ast_utf8_validation_result	ast_utf8_validator_state (struct ast_utf8_validator *validator)
	Get the current UTF-8 validator state. More...

Detailed Description

UTF-8 information and validation functions.

Definition in file utf8.h.

Enumeration Type Documentation

ast_utf8_validation_result

enum ast_utf8_validation_result

Enumerator
AST_UTF8_VALID	The consumed sequence is valid UTF-8. The bytes consumed thus far by the validator represent a valid sequence of UTF-8 bytes. If additional bytes are fed into the validator, it can transition into either AST_UTF8_INVALID or AST_UTF8_UNKNOWN
AST_UTF8_INVALID	The consumed sequence is invalid UTF-8. The bytes consumed thus far by the validator represent an invalid sequence of UTF-8 bytes. Feeding additional bytes into the validator will not change its state.
AST_UTF8_UNKNOWN	The validator is in an intermediate state. The validator is in the process of validating a multibyte UTF-8 sequence and requires additional data to be fed into it to determine validity. If additional bytes are fed into the validator, it can transition into either AST_UTF8_VALID or AST_UTF8_INVALID. If you have no additional data to feed into the validator the UTF-8 sequence is invalid.

Definition at line 71 of file utf8.h.

                                {
   /*! \brief The consumed sequence is valid UTF-8
    *
    * The bytes consumed thus far by the validator represent a valid sequence of
    * UTF-8 bytes. If additional bytes are fed into the validator, it can
    * transition into either \a AST_UTF8_INVALID or \a AST_UTF8_UNKNOWN
    */
   AST_UTF8_VALID,

   /*! \brief The consumed sequence is invalid UTF-8
    *
    * The bytes consumed thus far by the validator represent an invalid sequence
    * of UTF-8 bytes. Feeding additional bytes into the validator will not
    * change its state.
    */
   AST_UTF8_INVALID,

   /*! \brief The validator is in an intermediate state
    *
    * The validator is in the process of validating a multibyte UTF-8 sequence
    * and requires additional data to be fed into it to determine validity. If
    * additional bytes are fed into the validator, it can transition into either
    * \a AST_UTF8_VALID or \a AST_UTF8_INVALID. If you have no additional data
    * to feed into the validator the UTF-8 sequence is invalid.
    */
   AST_UTF8_UNKNOWN,
};

Function Documentation

ast_utf8_copy_string()

void ast_utf8_copy_string	(	char *	dst,
		const char *	src,
		size_t	size
	)

Copy a string safely ensuring valid UTF-8.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

This is similar to ast_copy_string, but it will only copy valid UTF-8 sequences from the source string into the destination buffer. If an invalid UTF-8 sequence is encountered, or the available space in the destination buffer is exhausted in the middle of an otherwise valid UTF-8 sequence, the destination buffer will be truncated to ensure that it only contains valid UTF-8.

Parameters

dst	The destination buffer.
src	The source string
size	The size of the destination buffer

Returns

Nothing.

Definition at line 133 of file utf8.c.

References ast_assert, decode(), UTF8_ACCEPT, and UTF8_REJECT.

Referenced by test_copy_and_compare().

{
   uint32_t state = UTF8_ACCEPT;
   char *last_good = dst;

   ast_assert(size > 0);

   while (size && *src) {
      if (decode(&state, (uint8_t) *src) == UTF8_REJECT) {
         /* We _could_ replace with U+FFFD and try to recover, but for now
          * we treat this the same as if we had run out of space */
         break;
      }

      *dst++ = *src++;
      size--;

      if (size && state == UTF8_ACCEPT) {
         /* last_good is where we will ultimately write the 0 byte */
         last_good = dst;
      }
   }

   *last_good = '\0';
}

ast_utf8_init()

int ast_utf8_init

(

void

)

Register UTF-8 tests.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Does nothing unless TEST_FRAMEWORK is defined.

Returns

Always returns 0

Definition at line 362 of file utf8.c.

References ast_register_cleanup(), AST_TEST_REGISTER, and test_utf8_shutdown().

Referenced by asterisk_daemon().

{
   AST_TEST_REGISTER(test_utf8_is_valid);
   AST_TEST_REGISTER(test_utf8_copy_string);
   AST_TEST_REGISTER(test_utf8_validator);

   ast_register_cleanup(test_utf8_shutdown);

   return 0;
}

ast_utf8_is_valid()

int ast_utf8_is_valid

(

const char *

str

)

Check if a zero-terminated string is valid UTF-8.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Parameters

str	The zero-terminated string to check

Return values

0	if the string is not valid UTF-8
Non-zero	if the string is valid UTF-8

Definition at line 110 of file utf8.c.

References decode(), and UTF8_ACCEPT.

Referenced by AST_TEST_DEFINE().

{
   uint32_t state = UTF8_ACCEPT;

   while (*src) {
      decode(&state, (uint8_t) *src++);
   }

   return state == UTF8_ACCEPT;
}

ast_utf8_is_validn()

int ast_utf8_is_validn	(	const char *	str,
		size_t	size
	)

Check if the first size bytes of a string are valid UTF-8.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Similar to ast_utf8_is_valid() but checks the first size bytes or until a zero byte is reached, whichever comes first.

Parameters

str	The string to check
size	The number of bytes to evaluate

Return values

0	if the string is not valid UTF-8
Non-zero	if the string is valid UTF-8

Definition at line 121 of file utf8.c.

References decode(), and UTF8_ACCEPT.

Referenced by AST_TEST_DEFINE().

{
   uint32_t state = UTF8_ACCEPT;

   while (size && *src) {
      decode(&state, (uint8_t) *src++);
      size--;
   }

   return state == UTF8_ACCEPT;
}

ast_utf8_validator_destroy()

void ast_utf8_validator_destroy

(

struct ast_utf8_validator *

validator

)

Destroy a UTF-8 validator.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Parameters

validator

The validator instance to destroy

Definition at line 215 of file utf8.c.

References ast_free.

Referenced by AST_TEST_DEFINE().

{
   ast_free(validator);
}

ast_utf8_validator_feed()

enum ast_utf8_validation_result ast_utf8_validator_feed	(	struct ast_utf8_validator *	validator,
		const char *	data
	)

Feed a zero-terminated string into the UTF-8 validator.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Parameters

validator	The validator instance
data	The zero-terminated string to feed into the validator

Returns

The ast_utf8_validation_result indicating the current state of the validator.

Definition at line 189 of file utf8.c.

References ast_utf8_validator_state(), decode(), and ast_utf8_validator::state.

Referenced by AST_TEST_DEFINE().

{
   while (*data) {
      decode(&validator->state, (uint8_t) *data++);
   }

   return ast_utf8_validator_state(validator);
}

ast_utf8_validator_feedn()

enum ast_utf8_validation_result ast_utf8_validator_feedn	(	struct ast_utf8_validator *	validator,
		const char *	data,
		size_t	size
	)

Feed a string into the UTF-8 validator.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Similar to ast_utf8_validator_feed but will stop feeding in data if a zero byte is encountered or size bytes have been read.

Parameters

validator	The validator instance
data	The string to feed into the validator
size	The number of bytes to feed into the validator

Returns

The ast_utf8_validation_result indicating the current state of the validator.

Definition at line 199 of file utf8.c.

References ast_utf8_validator_state(), decode(), and ast_utf8_validator::state.

{
   while (size && *data) {
      decode(&validator->state, (uint8_t) *data++);
      size--;
   }

   return ast_utf8_validator_state(validator);
}

ast_utf8_validator_new()

int ast_utf8_validator_new

(

struct ast_utf8_validator **

validator

)

Create a new UTF-8 validator.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Parameters

[out]

validator

The validator instance

Return values

0	on success
-1	on failure

Definition at line 163 of file utf8.c.

References ast_malloc, ast_utf8_validator::state, tmp(), and UTF8_ACCEPT.

Referenced by AST_TEST_DEFINE().

{
   struct ast_utf8_validator *tmp = ast_malloc(sizeof(*tmp));

   if (!tmp) {
      return 1;
   }

   tmp->state = UTF8_ACCEPT;
   *validator = tmp;
   return 0;
}

ast_utf8_validator_reset()

void ast_utf8_validator_reset

(

struct ast_utf8_validator *

validator

)

Reset the state of a UTF-8 validator.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Resets the provided UTF-8 validator to its initial state so that it can be reused.

Parameters

validator

The validator instance to reset

Definition at line 210 of file utf8.c.

References ast_utf8_validator::state, and UTF8_ACCEPT.

{
   validator->state = UTF8_ACCEPT;
}

ast_utf8_validator_state()

enum ast_utf8_validation_result ast_utf8_validator_state

(

struct ast_utf8_validator *

validator

)

Get the current UTF-8 validator state.

Since

13.36.0, 16.13.0, 17.7.0, 18.0.0

Parameters

validator

The validator instance

Returns

The ast_utf8_validation_result indicating the current state of the validator.

Definition at line 176 of file utf8.c.

References AST_UTF8_INVALID, AST_UTF8_UNKNOWN, AST_UTF8_VALID, ast_utf8_validator::state, UTF8_ACCEPT, and UTF8_REJECT.

Referenced by ast_utf8_validator_feed(), and ast_utf8_validator_feedn().

{
   switch (validator->state) {
   case UTF8_ACCEPT:
      return AST_UTF8_VALID;
   case UTF8_REJECT:
      return AST_UTF8_INVALID;
   default:
      return AST_UTF8_UNKNOWN;
   }
}