In the early stages of programming, different regions and their own standards for representing characters used in their writing systems, i.e., they had their own encoding systems. Some like the most common ASCII fit their entire characters with a positive signed single byte (0-127) effectively, while others utilize an unsigned char byte (0-255) to represent their characters, among other encoding schemes (known as code pages, extended ASCII, or platform-specific encodings like Shift-JIS, EUC-KR, GB2312, and Big5). However, this resulted in conflict, as text in one encoding might be misinterpreted in another, described as "garbled text," "mojibake," or the "encoding nightmare." What exacerbated the issues was that the encoding was implemented not only in programming language compilers/editors but also in the OS and display servers/terminals.

The advent of Unicode

With Unicode support in place, the OS or display servers effectively understood different Unicode encodings (UTF-8, UTF-16, UTF-32). Code editors and compilers were also modified to appreciate the new ecosystem. Therefore, it made it possible to store Unicode characters in a C string, which is effectively encoded as a UTF-8 string.

#include <stdio.h>

int main() {

// char ch = '😀'; // uncommneting this will result in warning

char ch1[] = "😀";

char ch2[] = "ㅁ"; // sound /m/ in Hangeul

char ch3[] = "é";

printf("%s %s %s\n", ch1, ch2, ch3); // Outputs: 😀 ㅁ é

return 0;

}

What can we learn from the above code?

Size of UTF-8 characters vs size of C char: C char type occupies only a single byte, while UTF-8 ranges from 1-4 bytes, with most of its characters occupying more than one byte (emojis, etc). Thus, even though the assigned literals are made up of a single Unicode character, they are wrapped in C as strings, not as single characters (double quotes, not single quotes), since in C each is considered to be made up of more than one character. Failure to do so will result in a warning during compilation and undefined behaviours in printing, due to integer overflow.
Why use empty square brackets? We are not sure how many bytes each occupies; thus, we define them as empty to avoid memory waste if the Unicode character occupies fewer bytes, i.e., to ensure enough memory is allocated to each character. Using a character pointer would also suffice.
How the above works: The editor and the compiler, when set to UTF-8, ensure each character is encoded and parsed as Unicode effectively. The process will buffer the output to be displayed, and not a single character at a time. Therefore, the display interfaces will parse the buffer code point by code point, rendering it without issue.

Why do we need other character sizes, then?

There is a problem with char strings: although we treat them as UTF-8, they only allow access to a single byte at a time, whereas UTF-8 characters can span multiple bytes. Therefore, when accessing characters, e.g., in a for loop, for characters occupying more than a byte, we access the bytes rather than the code points. This affects all string functionalities supported in C; functions from string.h will result in unexpected results.

#include <stdio.h>

#include <string.h>

int main() {

char ch1[] = "😀"; // Unicode bytes string '\xf0\x9f\x98\x80'

char ch2[] = "ㅁ"; // sound /m/ in Hangeul: Unicode bytes string: '\xe3\x85\x81'

char ch3[] = "é"; // Unicode bytes string: '\xc3\xa9'

printf("%zu %zu %zu\n", strlen(ch1), strlen(ch2), strlen(ch3)); // 4 3 2

return 0;

}

It can be seen that, though one character, the strlen print different values depending on the number of bytes required to represent each character. Also, though single characters, we cannot use equality signs with them, since they are not represented as a single primitive type but as a collection. Hence, it would be effective to use a representation that treats each Unicode character/code point as a unit, enabling counting and other processes, such as traversing.

Wide Characters (wchar_t)

A wchar_t (wide character type) It is a C/C++ data type for storing characters beyond the basic ASCII set, designed for Unicode and international text, typically 16 (mainly in Windows) or 32 bits, allowing representation of many more characters than a standard char. The advantage is that each character is stored in more than one byte. For access, the entire character can be indexed and printed individually via string indexing. It won't result in gibberish, as in char type, as there is no character splitting. Moreover, it includes its own functions to aid processes such as counting, which will now be based on the size of a wide character rather than on a single byte at a time. Finally, one can define a character literal and assign to it.

#include <stdio.h>

#include <wchar.h>

#include <locale.h>

int main(void) {

setlocale(LC_ALL, ""); // set ti default system locale

wchar_t ch1 = L'😀'; // Not functional in Windows since the emoji 4 bytes

wchar_t ch2[] = L"ㅁ"; // Hangul letter mieum

wchar_t ch3[] = L"é";

wprintf(L"%lc %ls %ls\n", ch1, ch2, ch3); // 😀 ㅁ é

wprintf(L"%zu %zu\n", wcslen(ch2), wcslen(ch3)); // 1 1

return 0;

}

NOTES:

The above output is only possible in Linux/MacOS systems where wide characters are 32-bit types, allowing the representation of all Unicode characters. In Windows, there might be issues with characters that occupy 3 or 4 bytes (17-32 bits), such as emojis, because accessing an index might split a character into multiple units.
A wide character is 2 bytes in Windows; the first declaration will cause an integer overflow.
Locale is changed when using wchar_t functions like wprintf need to know which specific legacy encoding (like UTF-8) the system should use to convert those "wide" characters into the actual bytes the display server understands. The default locale ("C") limits it to ASCII, which is not what is required.
It can be inferred wchar_t It is not portable due to a lack of a standard size across systems. Thus, to manipulate 16-bit or 32-bit Unicode units, it is recommended to use the types defined in <uchar.h>, which have guaranteed sizes:
char16_t: Always 16-bit (UTF-16).
char32_t: Always 32-bit (UTF-32).

Why are these representations (wide character and its alternatives) a failure in Unicode encoding?

A) Memory Wastage

Not every character is of the same size. Assuming a Linux system, for every wchar_t Four bytes will be allocated; however, some characters occupy 1, 2, or 3 bytes, so other spaces are not yet used but are allocated.

B) Partical Compatibility with Unicode

We use these types to enhance accessing a character at a specific point. This is effective for mosts characters but does not meet a single criterion in Unicode. Unicode has three different concepts that mean a character:

Code Unit (Byte/Half-word): This represents the smallest storage unit (8 bits for UTF-8, 16 bits for UTF-16, and 32 bits for UTF-32).
Code Point: A single numerical identifier defined in the Unicode standard (for example, U+1F600 for 😀).
Grapheme (User-Perceived Character): The unit a person recognizes as a single character. It may consist of multiple code points combined (for instance, an "e" along with a "combining accent" counts as two code points but forms one grapheme: é).

These representations don't handle the last definition. For example, "é" and "é" seem the same to the eye, while the latter is composed of two characters (e + ◌́). A comparison between the two strings will return false even if they appear the same. The same problem occurs with other modified emojis, such as those with skin-tone modifiers. However, this problem is not limited to C, as other programming languages, such as Python/JavaScript/C++, face the same issues.

Conclusion

Wide characters solve defining a variable for a single code point or accessing code point per code point issues that would require much effort with char type. However, it has a trade-off as it wastes memory. Finally, it does not solve grapheme issues.