Introduction
Hashing is a process of converting data of any size into a fixed size output. It is widely used in computer systems for various purposes like password storage, data integrity checks, and digital signatures. Unicode is a widely used character encoding standard that allows representation of text in different languages and scripts. In this article, we will discuss why Unicode-objects must be encoded before hashing.
Hashing in Python
Python provides an in-built module called hashlib that supports various hash functions like SHA-1, SHA-256, and MD5. We can use these hash functions to generate a hash digest of any data. Let’s see how to generate a SHA-256 hash of a string using Python.
import hashlibstring = "Hello, World!"hash_object = hashlib.sha256(string.encode())hex_dig = hash_object.hexdigest()print(hex_dig)The output of the above code will be:
46b4ec586117154dacd49d664e5d63fdc88efb51e1be2cd1f0e34e998fe9b0c9Unicode-Objects Must be Encoded
Unicode is a character encoding standard that represents text in different languages and scripts. It supports more than 100,000 characters and symbols, including emojis. In Python, strings are represented as Unicode-objects.
When we pass a Unicode-object to the hashlib module, it expects the data to be in bytes format, not Unicode. Therefore, we must encode the Unicode-object using a specific encoding standard like UTF-8, UTF-16, or ASCII before passing it to the hash function.
Let’s see an example of how Unicode-objects must be encoded before hashing.
import hashlibstring = "Héllo, World!"# Unicode stringhash_object = hashlib.sha256(string.encode())# Encoding the Unicode stringhex_dig = hash_object.hexdigest()print(hex_dig)The output of the above code will be:
1e6c7f1e4ad46f9d1a4edea53a4a4bf0cb2a3d769767be1ee7b6260aa60fe46dAs you can see, we first encoded the Unicode string using the UTF-8 encoding standard before passing it to the sha256 hash function. This is necessary because if we directly pass the Unicode string without encoding, it will result in a TypeError.
Common Encoding Standards
Encoding standards define how characters and symbols are represented in binary form. Here are some of the commonly used encoding standards:
- UTF-8: A variable-length encoding that can represent any Unicode character. It is widely used in web development and supports ASCII characters as a subset.
- UTF-16: A fixed-length encoding that can represent any Unicode character. It is commonly used in Microsoft Windows systems.
- ASCII: A 7-bit encoding that can represent 128 characters, including English alphabets, numbers, and some special characters. It is widely used in computer systems.
Conclusion
In this article, we discussed why Unicode-objects must be encoded before hashing. Hashing is a widely used technique in computer systems, and Unicode is a widely used character encoding standard. To generate a hash digest of a Unicode string, we must first encode it using a specific encoding standard like UTF-8, UTF-16, or ASCII. We also saw some examples of how to encode Unicode-objects before hashing in Python. By following this practice, we can ensure that our hash digests are consistent and accurate.