Hello and welcome back to this course. In the previous video, we talked about performing some traffic analysis for command and control. Our goal there was to look at the various protocols, types of traffic, packets and packet fields, and try to identify a good option for command and control. We want something that's rather common, has some space for storing command and control data, and where the data that we're going to transmit doesn't stick out. That third objective, not sticking out is what we're going to be talking about a little bit more in this video. In that video, we talked briefly about entropy, which is a measurement of the randomness in data or the amount of information a particular piece of data can encode. For what we're doing, we want the ability to have some relatively high entropy values. Because if you have data that's relatively same, or it all looks the same, things like English text, they have relatively low entropy. If you're used to having low entropy data like English texts in a field, and suddenly you have something that's got a bit more randomness to it, like computer code, then that computer code might stand out a little bit in that particular field. As part of our traffic analysis, we looked at the entropy of various fields using a function called fieldEntropy. Today in this video, we're going to be looking at what that fieldEntropy function actually does. Let's take a look at it now. Our fieldEntropy function is designed to calculate the entropy of a field. In the previous video, we briefly mentioned that we put some constraints on that field that we're going to calculate entropy for. The first is that we were looking for fields of certain types. When we're trying to transmit data over command and control infrastructure, we need some space to actually send more than a single bit or a small value in each packet. The less data we can fit into each packet, the more packets we have to send, and the more noticeable it is. One of our constraints is that we want fields that store certain types of data. Strings, bytes and bytearrays are three different Python data types that essentially come down to here's a set of bytes. They might be interpreted differently, but it says that, we might be able to store multiple bytes of information within a single field. This section of the code here is testing if we have a field of type string, bytes, or bytearray, and if so we want to convert that to a bytearray. Because a bytearray in Python essentially just a list of bytes as opposed to something like a string, which is interpreted as a string. With our list of individual bytes, we can treat each byte as a independent occurrence and calculate the entropy on it. Our other constraint that we talked about is we wanted a minimum length. If we can only store a single byte of information in each packet, we're going to need a lot of packets to achieve our goals. In this case, we've set our minimum length to five, but we could use something larger if we chose. If our length is greater than the minimum length, then we're going to calculate the entropy of that bytearray. To do so, we're going to use a couple of Python libraries called Pandas and SciPy. Pandas gives us the ability to represent data as a series and then count the number of occurrences of each item in that series. We'll take our bytearray of data, which might be a string, it might just be an array of bytes, etc. We're going to store it in a series data type. Then we can call value counts to say, how many bytes with value zero do we have? How many bytes with value one do we have, etc. In the end we should have a array which says for this particular value, we had this many occurrences. These counts aren't quite probabilities, but they're pretty close. When we're calculating entropy, we use observed probabilities, and we might compare those two expected probabilities. For example, if you're flipping a coin, the probability should be 50 percent that it is heads and 50 percent that it is tails for every coin flip. That's the ideal case. Some people can force a coin to flip a little bit more heads than tails, etc based off of few different factors. But in theory, half heads, half tails. That's the theory. In practice, if you flip a coin 100 times, the odds aren't great, that you'd get exactly 50 of heads and 50 of tails. You might have a 51, 49 split or something like that. Those are our observed probabilities in this particular case. When we're talking about the entropy of our packet fields, we don't have our expected probabilities. We could say that there's a one in 256 chance of each byte occurrence, and that's probably true. But what we do have is observed probabilities. We say, out of the 200 bytes in our byte array, five of them are the letter A. That observed probability is 5/200 or 140th and that's our observed probability of A versus our theoretical probability of 1 out of 256. Based off of all of these observed probabilities, we can calculate the entropy or the amount of randomness or amount of information that can be encoded in our byte array. Ideally, we should have observed probabilities pretty close to 1/256 for every single potential byte value. That would give us a nice high entropy and tell us that we can put any type of obfuscated data in our field. However, if we have something like English text or Spanish text or Latin text or pick a language text, we have a much lower entropy because there's a lot of structure to languages, so the amount of randomness in a string is much lower. If I say have the phrase H-E-L-L-O W-O-R-L in English, you know what the next letter is going to be. There's no randomness to it. Therefore, there's less information encoded in that string than if we have something that's completely random. To demonstrate this, I've got a few lines here that we're going to use our field entropy function to calculate the randomness of. We're going to look at the string "Hello world." Then we're going to look at an equal length string, 12 bytes, but they're going to be random. We'll import from the random library the rand bytes function to do that. Same calculation in both cases. One of them's English texts, one of them's completely random bytes. If we run our entropy function here, give it a moment, we see that the entropy of our random bytes is higher than the entropy of our English text. If we run this repeatedly, got lucky in this case, because what I wanted to demonstrate was, in most cases our entropies are going to be the same here for both of these because hello world doesn't change and this is the entropy of our 12 random bytes where they're all unique. However, down here we get a different entropy and the reason why is in this random string, there are repeated bytes somewhere. Here we go. We see a xfc and a xfc. Because there's a repetition there, our probabilities are slightly off of the 1 out of 256 that we're expecting and our entropy goes down a little bit. We can run this repeatedly as many times as we want and we see here that our entropies are the same in the first and the third, meaning that we got one occurrence of each byte that we see, which is about what we'd expect. This demonstrates just as Entropy.py function, which is just designed to determine if a particular field meets some criteria, the right types of value stored in it, and enough length. Then also to calculate the entropy of the data in that field to see what we might be able to store in it. Thank you.
