One of the major issues facing businesses working with user-generated online content in the modern era is speech moderation offensive and hate speech. The current approach to solving the problem is to manually keep a list of all possible words and phrases that could be considered offensive or hateful, and use it to filter out questionable content. There are a few problems with this approach: first and foremost, such an approach is not sustainable; by having to manually maintain a list of hateful and offensive words and phrases, the list very quickly becomes quite long, and keeping track of which variations of words and phrases have been added to the list becomes too manual to be worth the effort . With this project, the goal was to create an artificial intelligence model capable of detecting hate speech and offensive words and phrases in text data without the need for manually maintained lists of all possible qualified words and phrases of offensive hate speech. Say no to plagiarism. Get a tailor-made essay on “Why violent video games should not be banned”? Get the original essay For this project, the dataset used for training and testing the artificial intelligence model is from Github; This dataset was first generated by Davidson, Thomas and Warmsley, Dana and Macy, Michael and Weber, Ingmar for their paper titled “Automated Hate Speech Detection and the Offensive Language Problem”; The article was first published in “Proceedings of the 11th AAAI International Conference on Web and Social Media”. The original dataset consists of 24,783 unique tweets extracted from Twitter and is stored as a CSV file. This file contains six columns for each tweet: the full text of the tweet, the number of people who manually classified each tweet as offensive, hate speech or normal, the number of users who found the tweet offensive, the number of users who rate the tweet as hate speech, the number of users who rate the tweet as normal, and the majority class for each tweet based on user ratings. The dataset is also limited to only showing tweets that were rated by at least three users, whether they were offensive, hateful, or normal; this increases the probability that the classifications are legitimate and reduces noise in the model. Since the data initially comes from Twitter, the length of each text is limited; the average length of tweets is 85 characters, with a maximum length of 754 characters and a minimum length of 5 characters. Additionally, because the texts are tweets from Twitter, they contain special and extraneous characters, retweet information, URLs, and user mentions; None of this data is useful for the purposes of this project and was therefore processed during the data pre-processing stage. Another important note about the dataset is that all tweets are in English, or social media colloquial English. The original raw dataset is also unbalanced in terms of classes; Seventy-seven percent of the recordings were classified as offensive speech, six percent as hate speech, and the remaining seventeen percent were classified as neither hate speech nor offensive speech. When extracting the data into Python, most of the columns were removed as unnecessary for this project; the only two columns retained were the tweetsand the class to which they belonged. Once the dataset was loaded, it required a considerable amount of preprocessing to prepare the data to be used for training and testing an artificial intelligence model; Much of the data cleaning was done using the Regex library in Python. The first step was to remove all mentions included in tweets from the dataset; This was done by compiling a regular expression pattern that looked for any "at" symbol followed by characters and then spaces. The next step was to remove any information relating to retweets; This was done by using regex to remove all substrings containing "RT" and followed by characters and whitespace. Then, all tweet URLs were removed; Additionally, all foreign characters found in tweets were removed to reduce the complexity of the model's vocabulary. Additionally, all numbers were removed from the tweets in the dataset, as the numbers do not add anything of value to the purpose of this model. The next step was to remove all possible stop words from the tweets; In natural language processing, there is a concept known as stop words, which are words that generally do not add any significant meaning or value to the overall meaning of the text; In this project, the NLTK library was used to remove all possible stop words from tweets to ensure that the AI ​​model was not negatively affected by noise in the data. At this point, the tweets in the dataset are as clean as possible, so the final step in preparing the dataset for the model is to remove excess spaces in the tweets and remove all tweets from the dataset which eventually became empty strings. above cleaning processes have been completed. After all this data preprocessing was completed, the dataset was left with 24,768 cleaned tweets for model training and testing; these remaining tweets averaged 54 characters, with the longest being 343 characters and the shortest three characters. To enhance the value and generalizability of the artificial intelligence model being built, the dataset was supplemented with more hate speech and offensive words and phrases that had been identified by the Cludo Data Science team . These additional examples of hate speech and offensive words and phrases were compiled in twenty-five different languages ​​by the Cludo Data Science team over the course of 2018; Cludo is a website search company that provides clients with in-depth analytics and search solutions for their websites. In order to incorporate these new examples into the original dataset and keep them in line with the rest of the dataset, a function was developed to generate new fake tweets containing these examples of hate speech and offensive words and expressions. This was accomplished by creating a list of the most common words from the cleaned original tweet dataset and randomly sampling a random number of words based on the length distribution of tweets in the set of original data. Once these new imputed tweets were created, words and phrases were randomly sampled from the hateful and offensive speech identified by Cludo and randomly inserted into the new imputed tweets. To further augment the original dataset and avoid letting the AI ​​model learn from mistakes. relationships, the dataset was expanded incopying tweets and mixing up the word order in each tweet. One possible problem a model like this might encounter is incorrectly memorizing the specific order of words in order to classify offensive or hateful words or phrases, instead of actually learning what is and is not hate speech or offensive. To avoid this, the cleaned dataset was copied multiple times, and each of the tweets contained in the copies had the order of the words in the tweet shuffled randomly; this process deliberately added noise to the dataset and caused the model to be forced to actually learn from the data instead of memorizing it. Once this process was completed, all duplicate records in the new dataset were removed to reduce the risk of overfitting. At the end of this stage, the new dataset included 131,941 observations with approximately the same distribution between classes as the original dataset; With a significantly larger and augmented dataset, the AI ​​model is able to learn better and generalize to new examples in the future. Now that the dataset has been cleaned, preprocessed, and augmented, it's time to prepare the data for training and evaluation. the model of Artificial Intelligence. The first step was to divide the data set into our variables X and Y; In the case of this model, our X variable was the cleaned and preprocessed tweets and our Y variable was the class indicator column. After dividing the dataset into variables X and Y, the Tokenizer functionality of the Keras text preprocessing library was applied to the variable X; This Tokenizer used TFIDF to convert the tweet texts into arrays of numbers indicating each word in each tweet. The next step was to split the X and Y variables into training and testing data sets; This was accomplished using the train_test_split function from the sklearn model_selection library. Since this model is built to predict multiple classes, the dataset was split randomly while using stratified sampling to ensure that each of the three classes was more or less evenly distributed across the training datasets and testing; the chosen split between train and testing was seventy percent for training and thirty percent for testing. The final step in preparing the data for model training and testing was to supplement the now sequential data to normalize their lengths; To reduce the complexity of model training, the sequences were supplemented and/or limited to 500 units of length. Additionally, at this point, the Y variable has been changed from an integer value representing the class to a one-hot-coded variation, using Keras' to_categorical function. A few different variations of the neural networks were tested on a small subset of training data. determine the best combination of architecture and hyperparameters to solve this problem; Finally, a bidirectional LSTM was chosen as the core of the model. The input data form for the model was 92,359 observations with 500 features, while the test data form was 39,583 observations and 500 features. The first layer after the input layer was an embedding layer with a hidden layer size of 256 nodes; by making the first hidden layer an embedding layer, this allows artificial intelligence to learn more about the data by defining words contextually linked to an embedding representation encoding theirrelationship. After the integration layer is the bidirectional LSTM layer; this layer had 256 nodes and a 30% dropout rate to reduce the risk of overfitting and returned sequences; the bidirectional variant of the LSTM model was chosen because bidirectional LSTMs are better at learning context from natural language data compared to a standard LSTM. The bidirectional LSTM layer is fed into a Flatten layer, to allow subsequent layers to use the dense Keras layers. After the Flatten layer, a Dense layer was implemented using 256 hidden nodes and the ReLU activation function; ReLU was chosen due to its associated speed performance in larger networks, as well as its ability to learn nonlinear relationships. This Dense layer feeds into another Dropout layer to help reduce the risk of model overfitting; this dropout layer was set to a dropout rate of 30%. After this combination of Dense and Dropout layers, another set of Dense and Dropout layers was used, with the same parameters as the previous combination; Dense layer with ReLU activation and 256 hidden nodes, and a Dropout layer with a 30% dropout rate. The goal was to create an AI model that was robust and powerful enough to successfully learn from the dataset, without overfitting the model and keeping the model training time at an acceptable level. The final layer of the model was a Dense output layer with 3 nodes (one for each class) and Softmax activation; the Softmax activation function was chosen for the output layer because it maps the output of each of the classes such that adding the outputs will sum to 1, giving what is essentially the probability for each class. After building the model, the model was compiled with a categorical_crossentropy loss function (since the model is intended for a multiclass classification problem), an Adam optimization function and categorical_accuracy chosen as the parameter metric. Adam was chosen as the optimization function because it is relatively computationally efficient while remaining robust for this type of data, and categorical_accuracy was chosen as the training metric because we want to be sure that the model correctly classifies the inputs and not just grouping. all entries in a single class and calling it "precise". Other important parameters to note for this artificial intelligence were a batch size of 64 observations and 100 epochs; Generally, AI models learn better when presented with smaller batch sizes, but it is necessary to balance this with training time and computational efficiency, which is why the size of the batch was set to 64. Additionally, 100 epochs were chosen to ensure that the model had enough training time to learn everything possible about the data, but early stopping was also implemented to ensure that the model does not train longer than necessary. For the early stopping parameters, the model was oriented to monitor the validation loss metric with emphasis on its minimization, and with a “Patience” parameter of 10 epochs; this means that the model would automatically stop if the validation loss did not improve significantly over the 10 epochs of training. In total, this model had 72,549,379 trainable parameters. Model training was performed locally on an MSI brand laptop using the integrated NVIDIA GeForce GTX1060 GPU. Originally, model training started using.