What are hashes and how are they used

What are hashes and how are they used?
The hash sum (hash, hash code) is the result of processing some data by the hash function (hashing).

Hashes: type definition, checksum count, non-standard and iterated hashes

What are hashes

Hash transformation of an input array of arbitrary length into a (output) bit string of fixed length, performed by a certain algorithm. The function that implements the algorithm and performs the conversion is called the “hash function” or “convolution function”. The source data is called the input array, “key” or “message”. The result of the conversion (output) is called “hash”, “hash code”, “hash sum”, “message summary”.

This property of hash functions allows you to apply them in the following cases:

when building associative arrays;
when searching for duplicates in a series of data sets;
when building unique identifiers for data sets;
when calculating checksums from the data (signal) for the subsequent detection of errors in them (arising randomly or intentionally made) arising during storage and / or transmission of data;
when saving passwords in security systems in the form of a hash code (to recover a password using a hash code, a function is required which is inverse to the hash function used);
when generating an electronic signature (in practice, it is often not the message itself that is signed, but its “hash image”);
and etc.

One of the uses of hashes is to store passwords. The idea is as follows: when you come up with a password (for a website or an operating system), it is not the password itself that is saved, but its hash (the result of password processing by the hash function). This ensures that if the password storage system is compromised (the website is hacked and the attacker gains access to the password database), he will not be able to find out the user passwords, as they are stored as hashes. Those. even having cracked the password database, he will not be able to enter the site under the user credentials. When you need to check the user’s password, the hash is also calculated for the entered value and the system compares two hashes, not the passwords themselves.

For this reason, the pentester may be faced with the need to work with hashes. One of the typical tasks is hacking a password to get a password (they also say “a password in plain text” – since we already have a password in the form of a hash). In fact, hacking is the selection of such a string (password), which will give the same value with a breakable hash when hashing.

Hashcat is used to crack hashes . Regardless of the tool you choose, you need to know what type of hash we have.

How to determine hash type

There is a large number of hashes. Some of them are universal and are used by various applications, for example, MD5, SHA1, CRC8 and others. Some hashes are used only in certain applications (MySQL, vBulletin) and protocols.

In addition to popular hashes, developers can use various combinations of universal hashes (for example, calculate the hash using MD5, and then get the SHA1 hash for the resulting string), or iterated (with repetition) hashes (for example, the MD5 hash is calculated for the password, then the MD5 hash is calculated again, then the resulting string is again considered MD5 – and so on a thousand times).

For hacking, sometimes a hash is called a string or file formed in a certain way that is not used by the target application, but which was calculated from the source data so that it allows you to crack the password of the target file or protocol.

An example of such a line for WinZip:

$ zip2 $ * 0 * 3 * 0 * b5d2b7bf57ad5e86a55c400509c672bd * d218 * 0 ** ca3d736d03a34165cfa9 * $ / zip2 $

Example line to break a PDF password 1.7 Level 8 (Acrobat 10 – 11):

$ pdf $ 5 * 6 * 256 * 1 * -4 * 16 * 381692e488413f5502fa7314a78c25db * 48 * e5bf81a2a23c88f3dccb44bc7da68bb5606b653b733bcf9adaa5eb2c8ccf53abba66539044eb1957eda68469b1d0b9b5 * 48 * b222df06deb308bf919d13447e688775fdcab972faed2c866dc023a126cb4cd4bbffab3683ecde243cf8d88967184680

An example of a hash file for hacking VeraCrypt PBKDF2-HMAC-RIPEMD160 + AES: https://hashcat.net/misc/example_hashes/vc/hashcat_ripemd160_aes_13711.vc

Usually the pentester knows the source of the hash, and he knows its type. But there are exceptions. In this situation, you need to “guess” what hash in front of us.

This can be done by comparing the original hash with the samples. Or based on the number of characters and character set used.

You can also use tools that greatly speed up the process. Programs to determine the type of hash are hashID and HashTag.

hashID

This program is already installed by default in Kali Linux. It identifies the various types of hashes used to encrypt data, primarily passwords.

A hashID is a tool written in Python 3 that supports the identification of more than 220 unique hash types using regular expressions.
Using the program is very simple:

hashid hash_for_identification

A couple of important notes:

it is always better to specify a hash in single quotes (and not without quotes and not in double quotes)
there is an option -m , which when used displays information about the mode Hashcat

Hashcat hash mode is a hash type convention that must be specified with the -m , –hash-type option .
For example, I need to identify the $ S $ C33783772bRXEx1aCsvY.dqgaaSu76XmVlKrW9Qu8IQlvxHlmzLf hash:

hashid -m '$S$C33783772bRXEx1aCsvY.dqgaaSu76XmVlKrW9Qu8IQlvxHlmzLf'

As you can see from the screenshot, this is Drupal> v7.x in Hashcat. To break this hash, you must specify the mode 7900.
We identify the $ 1 $ VnG / 6ABB $ t6w9bQFxvI9tf0sFJf2TR hash:

hashid -m '$1$VnG/6ABB$t6w9bQFxvI9tf0sFJf2TR.'

We get several options at once:

MD5cryp is an algorithm that calls standard MD5 a thousand times to complicate the process.
For reference: MD5 was used to hash passwords. In UNIX, each user has his own password and only the user knows it. Hash protection is used to protect passwords. It was assumed that you can get a real password only by brute force. When UNIX appeared, the only way to hash was DES (Data Encryption Standard), but only US residents could use it, because DES source codes could not be exported from the country. FreeBSD solved this problem. US users could use the DES library, and other users have a method allowed for export. Therefore, in FreeBSD they started using MD5 by default. Some Linux systems also use MD5 for storing passwords.

Another $ 6 $ q8C1F6tv $ zTP / eEVixqyQBEfsSbTidUJfnaE2ojNIpTwTHava / UhFORv3V4ehyTOGdQEoFo1dEVG6UcXwhG.UHvyQyERz01 hash:

hashid -m '$6$q8C1F6tv$zTP/eEVixqyQBEfsSbTidUJfnaE2ojNIpTwTHava/UhFORv3V4ehyTOGdQEoFo1dEVG6UcXwhG.UHvyQyERz01'

The program says that it is SHA-512 Crypt – i.e. SHA512 (Unix).
Hashtag
HashTag is a python tool that parses and identifies various password hashes based on their type. HashTag supports the definition of more than 250 types of hashes and compares them with more than 110 hashcat modes. HashTag is able to identify a single hash, parse a single file and determine the hashes inside it or bypass the directory and all the subdirectories in search of potential hash files and identify all the hashes found.
Those. This is similar to the previous program.
By default, it is missing in Kali Linux, so you need to download it:

git clone https://github.com/SmeegeSec/HashTag.git
cd HashTag/
python2 HashTag.py -h

The hash for HashTag should also be placed in single quotes. The hash should be written after the -sh option . But immediately, without additional options are displayed modes.

We identify the same hashes:

python2 HashTag.py -sh '$S$C33783772bRXEx1aCsvY.dqgaaSu76XmVlKrW9Qu8IQlvxHlmzLf'

python2 HashTag.py -sh '$1$VnG/6ABB$t6w9bQFxvI9tf0sFJf2TR.'

As you can see, the results are similar.

Hash examples

You can find a large number of classic hashes, as well as hashes specially compiled for breaking a password and hash files here .

On that page you can:

try to identify your hash by sample
find the error in the compiled hash to crack the password by comparing it with the correct format
check hash identification programs

The hashID and HashTag programs do not always correctly identify the hash (at least hashID is noticed in explicit errors).
For example, I am interested in the hash c73d08de890479518ed60cf670d17faa26a4a71f995c1dcc978165399401a6c4: 53743528:

hashid -m 'c73d08de890479518ed60cf670d17faa26a4a71f995c1dcc978165399401a6c4:53743528'

This is clearly an erroneous result, since the salt after the colon was allegedly sent for the identification of the hash.

python2 HashTag.py -sh 'c73d08de890479518ed60cf670d17faa26a4a71f995c1dcc978165399401a6c4:5374'

We get a more correct result:

In fact, it is sha256 ($ pass. $ Salt).

How to calculate hash (checksum)

Linux has programs for calculating and checking popular hashes:

b2sum – calculates and checks the cryptographic hash function BLAKE2 (512-bit)
cksum – prints CRC checksum and number of bytes
md5sum – prints or checks MD5 checksum (128-bit)
sha1sum – prints or checks the SHA1 checksum (160-bit)
sha224sum – prints or checks the SHA224 checksum (224-bit)
sha256sum – prints or checks the SHA256 checksum (256-bit)
sha384sum – prints or checks the SHA384 checksum (384-bit)
sha512sum – prints or checks the SHA512 checksum (512-bit)

Information about SHA-2 (secure hashing algorithm, version 2) – a family of cryptographic algorithms (SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/256 and SHA-512/224.)

All these programs are installed by default in most Linux distributions, they allow you to calculate hashes for files or for strings.
The use of all these programs is similar – you need to specify the file name, or pass a string to the standard input.
If you use echo to calculate the hash of a string , it is extremely important to specify the -n option , which prevents the addition of a newline character – otherwise every hash for the string will be wrong!
An example of the SHA1 hash calculation for the test line:

echo -n 'test' | sha1sum
a94a8fe5ccb19ba61c4c0873d391e987982fbbd3  -

Another way to pass a string without adding the ending character is newline.

printf '%s' 'test' | md5sum

As you can see, after the hash should be a space and the file name (in the case of standard input – indicated by a dash), to show only the hash, you can use to add | awk ‘{print $ 1}’ or | cut -d “” -f1 :

echo -n 'test' | sha1sum | awk '{print $1}'

The same result can be obtained by the following construction:

echo -n 'test' | sha1sum | cut -d" " -f1

Tools for calculating various hashes

In addition to the Linux utilities listed above, there are other programs that can calculate checksums. Often they support several hashing algorithms at once, they may have additional input and output options (support various formats and encodings), some of them are prepared to perform file system auditing (identifying unauthorized changes to files).
A list of some popular hash calculation programs:

hashrat
hashdeep
Hasher
omnihash

Consecutive hashing using pipe (|)

For example, we need to calculate the sha256 hash for the string ‘HackWare’; and then for the resulting string (hash), calculate the md5 hash. The task seems very trivial:

echo -n 'Kalitut' | sha256sum | md5sum

But this is the wrong option. Since the result of the execution in any case is an incomprehensible string of random characters, it is difficult not only to detect the error, but even to understand what it is. And there are several mistakes here! And each of them leads to getting completely wrong data.
Even very experienced Linux command line users will not immediately understand what the problem is, but discovering the first problem will not immediately realize that there is one more.

It is very important to remember that in the string, along with the hash, the file name is always displayed, therefore, performing a fairly obvious command like the following:

echo -n 'Kalitut' | sha256sum | md5sum

we will not get the result that we expect. We assume to calculate the sha256 hash of the string ‘Kalitut’, and then for the resulting string (hash) to calculate the new hash md5. In fact, md5sum calculates the hash of the string to which is added ” – “. Those. It turns out a completely different result.
It has already been considered above how to delete “ – ” from the output ; it seems that now everything should be fine:

echo -n 'Kalitut' | sha256sum | awk '{print $1}' | md5sum

Let’s break this action down into separate commands:

echo -n 'Kalitut' | sha256sum | awk '{print $1}'

Get
d13417d8521cd36dc19ffee408905446209a410988898334f989907fd0a17b5e
Second stage of hashing:

echo -n 'd13417d8521cd36dc19ffee408905446209a410988898334f989907fd0a17b5e' | md5sum 
48d5b6128ce34d59484f9a4cca56189a  -

This is the correct answer.
Let’s try to perform

echo -n 'Kaliut' | sha256sum | awk '{print $1}' | md5sum

We’ll get: 4b932d30caeb4b1a2992169dc82f3df5 –
The problem is that when an intermediate hash is output, a newline character is added to it, and the second hash is considered for this full line, including an invisible character!

Using printf you can display the result without the newline ending character:

printf '%s' `echo -n 'Kalitut' | sha256sum | awk '{print $1}'` | md5sum

The result is again correct:
48d5b6128ce34d59484f9a4cca56189a –

Not everyone is friendly with printf and it’s problematic to use the above construction if you need to hash more than three times, so it’s better to use tr :

echo -n 'Kalitut' | sha256sum | awk '{print $1}' | tr -d '\n' | md5sum

Again the correct result:
48d5b6128ce34d59484f9a4cca56189a –
Or even make it even better – with the awk program we will use printf instead of print (this is the most convenient and short version):

echo -n 'Kalitut' | sha256sum | awk '{printf $1}' | md5sum

How to calculate iterated hashes

Iteration is the reapplication of an operation. With reference to cryptography, iterations are called multiple hashing of the data, which are obtained as a result of hashing. For example, for the original string in plain text, the SHA1 hash is calculated. The resulting value is again hashed – the SHA1 hash is calculated, and so on many times.

Iteration is a very effective method for dealing with rainbow tables and brute force, so iterated hashes are very popular in cryptography.

An example of code that calculates MD5 hash with 1000 iterations:

#!/bin/bash
 
text='Kalitut'
iterations=1000
 
function iterateMD5 {
    echo -n "$text" | md5sum
}
 
count=1
while [[ $count -le $iterations ]]; do
    text="$(iterateMD5 | awk '{printf $1}')"
    count=$((count + 1))
done
 
echo "$text"

Here:

Kalitut – string for hashing
1000 – the number of iterations
md5sum – used hash function