# Aim ## All content published is intended as Linux Primer for IT professionals with a special focus for Security Professionals # History Linux was created by Linus Torvalds. The main component of Linux after which it is named is the kernel i.e., Linux Kernel. Initially, Torvalds wanted to call the kernel as *Freax (to resemble Unix-like),* but his friend suggested the name as Linux. Linux was given as open-source project with minimal distribution for hobbyists Linus holds the "Linux" trademark to monitor its use. He has the highest contribution in writing of Linux kernel. He is the main person which is responsible for committing changes onto the standard Linux kernel. As per Wikipedia, the Linux Foundation currently sponsors Torvalds so he can work full-time on improving Linux. There are mainly three family of distributions: * Debian family of Distributions - Ubuntu, Kali, and ParrotOS * RPM family of Distributions - Fedora, CentOS, and RedHat * ARCH family of Distributions - ARCH, Alpine and CRUX # Components of Linux OS Linux is made up of the following components: * BootLoader - It is the first thing that runs on startup * Linux Kernel - Explained on next page * Background Services - such as networking services * OS Shell (also known as Terminal) - way to interact with OS with the help of commands 1. sh - short form of Shell. Had major limitations like typing commands again if user makes a mistake while typing. 2. bash (Bourne Again Shell ) - default shell in most distros 3. ash and dash (Debian Almquist Shell) - lightweight clones of sh shell 4. zsh (Z shell) - Better than bash. Default on Kali Linux 5. csh (Cshell) & Tcsh - Allow for use of C++ syntax for scripting. 6. ksh (Korn shell) 7. rbash (Restricted bash) * Graphics Server (X) - The service responsible for GUI and Desktop Environment (XFCE, KDE, GNOME) which actually uses this service. i.e. Graphical server is the middleware between desktop environment to pass the user input captured by desktop environment to the operating systems. * Applications. * File System - Linux OS uses Filesystem Hierarchy Standard (FHS) and the hierarchy is as follows: 1. Volumes - Partitions on physical or virtual storage devices. (usually the best practice is to keep the OS and user data on separate volumes). Volumes are usually mounted (way to connect and interact on the same) which are then presented as directories known as mount points. 2. Directories - Unlike Windows OS, these are just files which show the indexing of file within. 3. Files - Actual building block. In Linux everything is just like a file. Note that hidden files are usually indicated as .\ or dot files Note: A disk’s file system has a number of control blocks called *inodes (index nodes).* These are basically metadata that describe the details of files and other objects, including where on the disk any associated data is stored. Entries in a Linux directory associate a name, with an inode number. As per [this ](https://www.site24x7.com/learn/linux/inode.html)site, > On a Linux system, files are stored in the form of blocks of a certain size. If a file is larger than this predetermined size, it’s broken down into chunks and stored in empty blocks wherever they’re available on the disk. With an increase in the number of files, this can easily get confusing. Inodes help the system organize data * [x] Overview of FHS Structure

Source: https://linuxconfig.org/wp-content/uploads/2013/03/Directory-Filesystem-Hierarchy-Standard.avif

Note: /dev which is a short-hand notation for devices as it usually contains drivers and /var which contains variable files is because the files inside /var are constantly changing # Linux Kernel Linux is the core of the Operating system i.e., it is the kernel. It is the closest thing that runs to the underling hardware such as CPU, Memory and I/O devices i.e. • It controls CPU • It controls Memory • It controls Devices ![](https://275986271-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPaRFhO7J6sRJrjn8Haee%2Fuploads%2Fiyd9tuSz6dBbCeSWJ0C0%2Fimage.png?alt=media\&token=e42a7015-593d-424a-a2f6-ab818a25e0d5) Note it is possible to change the Linux Kernel i.e., basically one can install or update the Linux kernel since it is open source, but the same cannot be done with Windows since NT kernel is closed source. # Linux Commands This page sheds light on commands which may not be known to newbies of Linux OS Just like apt (the package manager) is used to install services, "apt purge" can be used to remove the same. > **If a space is typed before the command, the bash history won’t log it.** > Recursive search on the bash history enables. On Pressing Ctrl+R, the prompt enters into\ > reverse search mode. Typing any character makes the search to suggest commands from the history. The user needs to press ENTER once the desired command is found. ### nl Just like cat command is used to display the contents of the file, if suppose line number is also required against the content, then nl is the command for it

{% hint style="info" %} cat /dev/null > filename is used to empty a given file {% endhint %} ### tac It is simply reverse of cat.

### bat One needs to install it using apt. In Debian based distros, it is executed as batcat. To put it simply, bat is like cat but on steroids. It has nl like feature which shows about line number and it also, formats text in nice and readable manner. It is especially useful for reading scripts and code based files.

### Wildcards ?, \*, \[], \[!] are some of the wildcards that can be used in linux. The ? wildcard represents one character, the \* represents any number of characters, the list of characters contained in the \[] represents one character, but the character must be a member of the list, and the list of characters contained in \[!] represents one character, but the character cannot be a member of the list.

### sort and uniq As per help section, sort has the following options:

Uniq is useful for finding unique entries in a set of data

One of the use cases can be used to find unique entries in your history. So for example, history command. First get rid of the history line item number using awk and then to place identical commands together, we can sort it and then we can get a unique count and lastly we can get that output in descending order using sort -nr

### who, w and last who is the command which is equivalent of "query user" or "qwinsta" in Windows, which basically tells the currently logged in users. Similarly, last is used to check the last logged on users, basically the login history. Lastly w tells the same information as who but with more details.

{% hint style="info" %} Note, in Debian/Ubuntu login logs (equivalent of WIndows Event Viewer -> Security Logs) are stored in information is stored in /var/log/auth.log, while for Redhat/CentOS, it is stored in /var/log/secure and for Debian/Ubuntu system logs (equivalent of WIndows Event Viewer -> System Logs) are stored at /var/log/syslog while for RHEL/CentOS machine, the output is stored in /var/log/messages {% endhint %} {% hint style="info" %} Note: A user in sudoers group can switch to profile of other user . Command is: sudo su - \ {% endhint %} {% hint style="info" %} To check which user is in sudoers group, one can simply do a \ cat /etc/group | grep sudo\ \ However there is a separate file called as /etc/sudoers for check information as shown below:\ \ ![](https://275986271-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPaRFhO7J6sRJrjn8Haee%2Fuploads%2FBWL8FXnErkorHZTz62wM%2Fimage.png?alt=media\&token=a12c2fcb-8055-494b-b9c2-444f8545d325) {% endhint %} {% hint style="info" %} In Fedora Distributions, the sudoers group is called wheel {% endhint %} ### cut The cut command helps to extract data from input with the help of a delimiter and determining whether to cut by byte position, field, or character.

-d " " is used for delimiting and -f1 or f2 is used for selecting column

### xargs Xargs is an utility that can simplify the processing of complex operations, repetitive tasks, batch processing, and large data manipulation. Xargs reads data from stdin, and then executes the command supplied to it as an argument. By default, xargs reads data separated by blanks because it treats spaces as delimiter, and then executes a command once for each argument. Note that blank lines are ignored.

### test, expr and let These commands can use the following operands:

*test* is used to check file types and compare values and returns true or false. Another way to use test is to use the construct `[...]`

{% hint style="info" %} Note that test would return 0 if true and 1 if false. this can be demonstrated using echo $? which tells the op of last command ![](https://275986271-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPaRFhO7J6sRJrjn8Haee%2Fuploads%2FLqyV4Tp4xkQDK1rABz9o%2Fimage.png?alt=media\&token=f01acd3a-abd5-4bb8-bc36-a002d00628fc) {% endhint %} *expr* command evaluates arithmetic or string expressions and sends the result to stdout. When one use the *expr* command to perform multiplication in an expression, one must use a backslash as an escape character before the multiplication operator, as the *\** character is recognized by the shell as a special character. This is as shown below. The construct for expr is `$((...))`

{% hint style="info" %} Note, to substitute a variable, use $variable-name {% endhint %} *let* command is equivalent to enclosing an arithmetic expression in double parentheses. The construct for *let* is `((...))`

### tree This is equivalent of the following command but displays files in form of branches of a tree: ```bash ls -R ```

### ln It is used for creating hard links and symlinks (soft links) in linux. Note: Symlinks are file-system objects used for creating symbolic links which are just pointers pointing to a file or directory, where as Hard links creates a direct association with the target file by acting as its mirror (because a hard link creates an alias, or an *associated* name with the same inode number)

{% hint style="info" %} Note, to revoke permissions in linux, one can simply use chmod with zero i.e., chmod 700 will only take permissions from group and others and will keep rwx with owner. {% endhint %} ### head and tail head lets user view only the top 10 lines of data in a file. one can also pass the argument n to mention how much lines should the head command display. tail is the exact opposite which displays the last 10 lines of a file.

### wc wc or word count simply tells the number of lines, word count, byte and characters count

### more and less These are cat-like commands with a scrollbar to avoid displaying the entire file at once. Both the command can be used to read the contents of a text file one page (one screen) at a time. This askubuntu [ans ](https://askubuntu.com/questions/1191862/what-is-the-difference-between-more-and-less-commands)summarizes the commands very easily:

### diff This is used to compare two files.

{% hint style="info" %} Note: zless and zdiff are basically commands used to the same operations as less and diff but for files in an compressed archive. {% endhint %} ### which and whereis which is a tool for locating the path to a command while whereis also does the same but also returns the location of manpage

### locate and find locate and find - both are used to search a file by its name. The key difference is that locate uses a previously built database while file searches in the filesystem. While locate is faster, find is more aggressive with options such as -name, -type and -exec

### file It is used to know more about the type of file to the user. please note that file commands reports on the basis of the contents inside the file and not based on file extension.

{% hint style="info" %} Just like the Sysinternals utility strings64.exe, in linux, there is a command called as strings which dumps printable character sequences in any file regardless of its type {% endhint %} ### apropos This command is used to find other commands based on the keyword that has been passed as an argument

{% hint style="info" %} sha1sum /bin/bash gives sha1 hash md5sum /bin/bash gives md5 hash {% endhint %} ### top, htop, atop, and iotop top is similar to task manager of windows i.e., to see the running processes. htop unlike top is interactive process viewer with functionality such as searching, tree view and capability to use a mouse with it

atop

iotop

{% hint style="info" %} ps lists a snapshot of the running processes. By default, it only shows processes owned by the user executing the command. The -a flag shows processes of all owners. The -u flag provides verbose o/p. The -x flag includes processes i.e., services started by the host on startup. {% endhint %} ### ss To list running services, one can use the ss command

{% hint style="info" %} **useradd, usermod** and **userdel** are used for creating, modifying and deleting a user. {% endhint %} ### lsof lsof command stands for **List Open Files.** Since Linux sees every object as a file, such as devices, directories, etc., unidentified open files prevent users from modifying them.

### sed It is stream editor. It brings the contents of a file into a stream, where modification operations such as text substitution or other text manipulation operations like insert, delete, search, and more. Please note that it doesn't change the actual file but in the stream. To change in the actual file, one can use re-director (>) to ultimately save the content in new file.

### Redirector {% hint style="info" %} In the world of Linux and Unix, there is something called as file descriptors. File descriptors are used to represent and manage input and output streams between processes and files, sockets, pipes, and other sources or destinations of data. A file descriptor is a unique non-negative integer that serves as a handle or identifier for an open file or communication channel. At least one file descriptor exists for every open file on the system. When a program asks to open a file — or another data resource, like a network socket — the kernel grants access, creates an entry in the global file table and provides the software with the location of that entry i.e., the global file table has entries pointing to inode number where a particular stream of a file is sent. {% endhint %}

Source: https://www.computerhope.com/jargon/f/file-descriptor.htm

Usually there are two out streams: one is main o/p and one is error. When only > is used, only o/p is redirected to a file or process but when both needs to be redirected, then 2>&1 has to be used

Source: https://catonmat.net/bash-one-liners-explained-part-three

### tee It is used to take the stdin and write to stdout or files. It is similar to Out-File of powershell example: ps -ef | tee file1

### screen Use screen to preserve the console output and running tasks in case of connection\ break (mostly useful while working on remote machines). List the running screen sessions: screen -list Attach to the screen: screen -r ### gpg and gpgtar gpg (GNU Privacy Guard) is used for creating encrypted keys and actually encrypting a file. As part of encryption for data at rest, gpgtar is used for encrypting files or directories using tar for compression of the said file.

{% hint style="info" %} Note, **encfs** is usually used for mounting or creating an encrypted virtual filesystem. **cryptsetup** is used for encryption of entire volume. {% endhint %} {% hint style="info" %} /dev/null - Whatever is written to this file vanishes, and reading from the file outputs nothing. So one can use this file is used to eliminate the standard output or error of a shell command, by redirecting it to this file. /dev/zero - Upon reading, this file provides an endless stream of zeros and anything written to the file vanishes. {% endhint %} ### dd, du and df dd is also known as disk/data duplicator. As per manpage, dd can be explained as follows:

```bash // To take backup of entire disk by copy pasting using dd dd if=/dev/sda of=/dev/sdb conv=noerror, sync ``` ```bash // Note, because default bs is 512, a 1.5 kb file will take 3 turns to copy as 1.5 kb=512bytes*3 and if we want to copy 1/3 of a file then we can simply provide count=1, so that only 1st default size of 512 bytes is copied, so command can be: dd if=/etc/shadow of=realdeal count=1 ``` du is used to check the space of an entire directory w\.r.t to what size a particular file or subdirectory has taken while df is used to check the overall disk space. du -sh <--summarize and human readable and du -sh \* <— for all files in current directory and without \* only gives details of the directory

> To view only the memory stats, one can use free command. It display the amount of free and used memory in the system. > > ![](https://275986271-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPaRFhO7J6sRJrjn8Haee%2Fuploads%2FUly1iksQhYFiNxDFFXLM%2Fimage.png?alt=media\&token=6aae4189-59b5-4ba1-b5a9-ae59a40259b0) ### tmux Also known as terminal multiplexer. This is particularly used to maintain terminal continuity across sessions. So, let's just say, one person is taking ssh and ran a command which will take, let's just 16 hours and in between, the person closed the terminal, so when he/she comes back, the command that was running needs to be re-run because when the session was closed, the command was also mid-way terminated. To solve, this problem, tmux was brought in. The logic is that once ssh access is taken, one needs to start a tmux session in that terminal and type all commands in that tmux, so even if ssh access is closed, ssh session when regained, can access the tmux to continue the work.

// For creating a new named session
tmux new -s <name-of-session> 

// For attaching to a named session
tmux attach -t <name-of-session>

// For copying from a tmux session
Ctrl + b [

// For increasing the scrollback buffer to 2 lac lines
Ctrl +b
: set -g history-limit 200000

// For deteaching a tmux session
Ctrl + b 
d 
OR
Ctrl + b
: detach

// For listing a tmux session, one can do the following:
tmux list-sessions

// For killing a tmux session, once you are inside the session
Ctrl + b
: kill-session 

// For moving up inside a tmux session,
Ctrl + b
[
Use up arrows to go up and down in copy mode and then press q to come back

// For splitting the screen horizontally in tmux, do the following:
After pressing, Ctrl + b, leave the keyboard and then press "

// For splitting the screen vertically, do the following:
After pressing, Ctrl + b, leave the keyboard and then press %

Note: After splitting, press Ctrl + b, and then use arrow keys to mopve around. 
Lastly, its better to use take ssh from cmd instead of putty for terminal splitting

### dig It is the newer alternative for older nslookup. It has following options:

Source: https://phoenixnap.com/kb/linux-dig-command-examples

### ssh #### Type of Shell ssh has a switch option called as -t which can be used to specify the shell one wants while ssh'ing One can also determine the type of shell using the command: ```bash script_shell="$(readlink /proc/$$/exe | sed "s/.*\///")"; echo $script_shell ```

#### Local Port Forwarding Everyone knows ssh, but suppose one is accessing a remote server via ssh and the remote server doesn't have GUI, only terminal then the following commands can use tunnelling using port forwarding to access an application running locally on the remote server from host machine.

```bash ssh @ -N -f -o ServerAliveInterval=60 -L :: ``` Here, -N is only for port forwarding and not executing remote commands, -f backgrounds ssh, -o is used for passing configuration values (serveraliveinterval is used to keep the connection alive after every 60 seconds), -L is used to forward a local port to the remote server's port Note: One can also use -q for quiet mode For example, to access the python server which is running on localhost of linux server, one can issue the following command: ssh \@\ -N -f -o ServerAliveInterval=60 -L 8000:localhost:8000

#### Dynamic Port forwarding Here, -D is used for the remote linux server to act as socks proxy and -C is used for compression to save bandwidth. ```bash ssh -D -f -C -N -q @ ``` The operation can be easily understood as shown below:

Source: https://ma.ttias.be/wp-content/uploads/2017/01/socks5_proxy_encryption.png

Source: https://ma.ttias.be/wp-content/uploads/2017/01/socks5_browser_settings.png

#### Remote Port forwarding Now, let's just assume that there is a remote server, where one can ssh into from local machine, but from that remote server, if one wants to access a webserver hosted on local machine, then reverse port forwarding can be quite helpful. ```bash ssh -N -f -R :: @ ``` Let's take the following example: On local machine, a webserver is hosted on port 8088

Now, for this webserver to be available to linux server, one can open terminals, one for reverse port forwarding (telling that remote server when access webserver locally on 8088, it will connect with webserver hosted on local machine) and other for interaction as shown below:

{% hint style="info" %} One more important option is use of -J which is to specify the jump host, so if a server A is not accessible directly, but via server B only, then ssh -J user\@serverB user\@serverA This command would work when password authentication is being used on both serverB and serverA For it to work via pem key, one need to create the following config file under .ssh and access it via the command: ssh remote {% endhint %} ```keepalived Host jump HostName IdentityFile .ssh/.pem Port 22 User ec2-user Host remote HostName IdentityFile .ssh/.pem Port 22 User ec2-user ProxyJump jump ``` ### socat Its purpose is to establish a relationship between two data sources, where each data source can be a file, a Unix socket, UDP, TCP, or standard input. It can be best described as written below from manpages:

One of the most used option is to use socat to connect to two TCP streams on separate streams. As per syntax, socat options \ \

Note that address can be of any order listen first or connect first also. Thus the following command can be used to connect two data streams as shown below: ```bash socat TCP-L:443,reuseaddr,fork TCP:localhost:8088 & ``` Note: **fork -** put every connection into a new process, **reuseaddr -** keeps the port open after a connection is made, **& - b**ackgrounds the application

Note, if one wants to create only listener or only connect, then give the 1st bidirectional address as hypen (-) like directly that tcp stream would be stdout Examples of usage: In Machine A, terminal 1 --> socat - TCP4-CONNECT:\:\ In either Machine A or Machine B terminal --> socat TCP-LISTEN:\,reuseaddr,pf=ip4,fork - The pf=ip4 option explicitly restricts the TCP-LISTEN address to use IPv4 (Internet Protocol version 4) for listening. It ensures that socat will only bind to an IPv4 address and only accept incoming connections that use IPv4, since by default, it often tries to listen on both IPv4 and IPv6 (Internet Protocol version 6) if the system supports it. The fork is crucial for allowing the socat listener to handle multiple incoming connections sequentially or in parallel. When a program like socat (or any server) is shut down, the socket it was using to listen on a port does not immediately become available. The operating system holds the socket in a TIME\_WAIT state. The `reuseaddr` option is a socket option that allows to bind immediately to the same port. ### tr As per help section, tr is simply used to convert small-case to upper-case or delete characters

Print the last 5 lines and replace the space character (' ') with underscore character ('\_')\ using tr utility.\ Command: cat apache\_access.log | tail -5 | tr ' ' '-' ### grep grep has the following switches which are commonly used: ![](https://275986271-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPaRFhO7J6sRJrjn8Haee%2Fuploads%2FfGcs0J3KGA1G7XnG2v40%2Fimage.png?alt=media\&token=cb16a73f-6b1a-4cad-8b6c-ced339009b84) grep's -v is used to grep everything in file except the passed argument:

### tldr Its an alternative to manpages

### at and cronjob It is like a non-recurring cron job.

Cronjob: To define the time you can provide concrete values for minute (m), hour (h), day of month (dom), month (mon), and day of week (dow) or use '\*' in these fields (for 'any'). Exact Pattern: minutes hour dom mon dow command

Note that all crontab can be added in single file itself using crontab -e ### timeout It is used to limit the execution of a command to time in seconds mentioned. example ping is in infinite loop without timeout and with timeout, it is limited as shown below:

{% hint style="info" %} lynx and browsh are terminal based browser, useful for submitting forms {% endhint %} # AWK It is more of a script than an command, which is heavily used for text processing and data manipulation and produce formatted reports. It is typically used as a data extraction and reporting tool. It is a standard feature of most Linux operating systems and is useful when handling text files that are formatted in a predictable way. awk parses and manipulates tabular data on a line-by-line basis, and it iterates through the entire file. By default, awk uses whitespace—for example, spaces and tabs—as a delimiter to separate fields The syntax is as follows: ```awk awk GNU_OR_POSIX_OPTIONS 'pattern_selection_criteria {action}' input-file ```

As shown above, -F is important to tell the delimiter awk has built-in variables such as: * **`$0`**. Used to specify the whole line. * **`$1`**. Specifies the first field or first column. (example - awk '{print $1}') * **`$2`**. Specifies the second field. * NR: Counts the number of input records (usually lines). Awk command performs the pattern/action statements once for each record in a file. * FS: Just like the command line argument -F, the field separator can also be passed via variable FS. * RS: Stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline. * OFS: Stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter. * ORS: Stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print. * FNR: It is the current record number in the current file. For the file, NR is going to be equal to FNR as FNR will reset to 1 for every file but NR will keep increasing. * NF: Variable whose value is the number of fields in the current record.

# Regex Regex is advanced search pattern which can be used to search non-specific and specific data, as well as enhance the quality of your programming code. Please note that if one can avoid regex instead of something simpler, one must avoid using it. Sometimes using regex where it is not required can break things. and [https://regex101.com](https://regex101.com/) are very helpful online tools to build regex queries. To identify any pattern, different pattern formation elements (regex structures) are used which are as follows: * **Character Classes**: List of characters that can appear in the pattern. Character classes are defined by square brackets around the list * [x] \[a-z] - all small letters * [x] \[A-Z] - all capital letters * [x] \[a-zA-Z] - both small and capital letters * [x] \[0-9] - all digits * [x] \[afgh] - elements only from this list of a,f,g and h * [x] \[^a-d] - ^ means negative or not when in square brackets, i.e. any character which is not small a, b, c and d * [x] \[\[:alnum:]] - All alphanumeric characters * **Meta Characters, Anchors and Escape characters**: They have special meaning within regex and usually start with \\ * [x] / - start or end of an expression * [x] ^ - Start of the line * [x] $ - End of the line * [x] \w - any word small or capital and \W - not any word * [x] \s - whitespace and \S - whitespace * [x] \d - digits and \D - not digits * [x] \b - backspace character or word boundary and \B - non word boundary * [x] a|b - match either a or b. | - means match any character before or after | * [x] \N or **.** - matches any character other than newline * [x] \\\ - matching \ itself * [x] \\\* - matching \* * [x] \\. - matching dot * [x] \n - match newline * [x] \t - match a tab character * [x] \r - match carriage return * [x] \0 - to match null character * **Occurrences:** They usually tell how much to match with the help of wildcards * [x] {1,3} - Define a range—the first digit is the minimum value, and the second is the maximum value * [x] {4} - Number of times the pattern should be matched * [x] \+ - Match one (1) or more of the specified preceding pattern, \* - Match zero (0) or more of the specified preceding pattern * [x] ? – Match (0) zero or one (1) of the specified preceeding pattern. * Quantifiers: Combining occurrences with the previous two regex structures can give something as quantifiers.

Note, capture group can be used with (), so if one wants to capture a pattern which has anything in between two words, let's just say, WORD1 aasdkjaslkj WORD2 asdka WORD3, then to capture from word1 to word2, regex can use the capture group as (.\*?) which means match anything except newline which can be any number of times including zero and capture it only once Examples: 1. To search gmail id of let's just raghav, but if there are many raghavs' (like , ) and there may be additional emails such as that of yahoo, outlook etc , then the regex query can be as follows: ```regex raghav\d*@\w{2,}\.\w{1,} ```

Here, raghav is matched and then it can have a digit or not i.e.digits can be 0 or more hence \*, followed by @ of the email domain and then domain can be be any word greater than 2 characters followed by a dot, which is lastly followed by a word of characters 1 or more to tell root domains 2. To search Aadhaar card number which is a set of 12 digits with spaces in between, the regex query can be: ```regex /\d{4}\s\d{4}\s\d{4}/g ``` 3. To search credit card separated by either spaces or dash, regex query can be: ```regex \d{4}\s?\-?\d{4}\s?\-?\d{4}\s?\-?\d{4} ``` 4. To extract IP address from IIS source file, one can use regex101.com to form a query, grep -Po to extract and awk to print only required information as shown below: ```bash grep -Po '\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s[GPHOT]' iis-sample-logs | awk '{print $1}' ```

Note: awk consider field separator as space by default # Log Scanning Log Scanning is beneficial for troubleshooting service errors and searching malicious activity Logwatch is an excellent tool to achieve the above objective. One can refer [this](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-logwatch-log-analyzer-and-reporter-on-a-vps) link to learn the same. auditd is the daemon that has to be installed to achieve the same. One can refer [this ](https://sematext.com/glossary/auditd/)link to learn more. aureport is used for generation of report and ausearch is used for searching within the logs collected by auditd. auditctl is used for configuring log collection rules Note: Please check the locale using command date '+%x' and then pass it to ausearch with start and end dates as: ausearch -ts \ -te \ # SE-Linux # AIM ## All content published is intended as an AWS Cloud Primer for IT professionals. # General Stuff about AWS 1. The on demand self-service delivery of compute power, DB, storage, app, and other IT resources through a cloud services platform via the Internet with pay-as-yo-go pricing is the essence of any cloud and AWS is no different. In short, AWS started as a DataCenter as a service. 2. When one uses AWS, they are kind of accepting the shared responsibility model, where **AWS** is responsible for **security of the cloud** and **customers** are responsible for **security in the cloud,** as shown below, and it changes with type of the service:

Source: https://aws.amazon.com/blogs/industries/applying-the-aws-shared-responsibility-model-to-your-gxp-solution/

3. AWS has something called as metadata resource tags, which can be used to segregate workloads across all AWS Services. 4. AWS Pricing Calculator can be used to get an estimated cost before moving workloads into the AWS cloud. 5. AWS Cost Explorer is useful for segregation wise cost by either linked account or linked accounts. 6. AWS Organizations is a simple way to get consolidated billing and apply common policies by having a master account and then creating department wise accounts under that. 7. Whenever events affecting customers of AWS occur, AWS Personal Health Dashboard can provide alerts and guidance related to remediation. 8. **AWS Trusted Advisor** is an automated tool to check AWS usage against best practices. All AWS customers get access to seven core checks. It gives recommendations related to Cost Optimization, Performance, **Security,** Fault Tolerance and Service Limits. For complete Trusted Advisor Checks, it is recommended to opt for AWS Business Support. 9. Where Cloud like AWS shines, is Serverless computing offering. It takes PaaS a step further, by introducing the developers with event driven based architecture, which means whatever code the devs are writing, will get an execution environment when an external event is triggered and they have to pay only for the compute required when the code is executed. # AWS Global Infrastructure * AWS Regions are the main data centers in a particular geography. * AWS Regions are further divided into availability zones by default that are there for fail-over and high availability. Example: ap-south-1 is Region Name and ap-south-1a is availability zone. * AWS Local Zones can be said as an extension of AWS Region. They are just strategically located AWS data center which is well connected with main AWS Region Data Center. * Outside of AWS Regions, specific points of presence elements exist. They are Edge Locations and Regional Edge Caches. Edge Locations are nodes used by Content Delivery Network service of AWS, named as Amazon CloudFront and their DNS service named as Amazon Route 53.

* To get information about AWS IP ranges, one can get navigate to and get a gist of the same:

{% hint style="info" %} Many times, Hackers take these IP Ranges and then checks across these ranges using online scanners, if there are any misconfigured or vulnerable AWS Service present on the same {% endhint %} # Interacting with AWS ## Accessing AWS Services * AWS Management Web Console * AWS Mobile Application * AWS CLI \[Available for Windows/Linux or use **CloudShell**] > Note: AWS CLI comes preinstalled on AMI Linux. Also,Bundled installations like the one as shown below are difficult to modify default behavior like changing User Agent, so its recommended to passit via Proxy (Shown later) > > One can install AWS CLI in Debian based Linux using the following commands: > > ```bash > curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" > unzip awscliv2.zip > sudo ./aws/install > aws --version > aws configure > aws sts get-caller-identity > ```

* AWS SDK \[Some of the supported languages are Python, C++, Java, .NET, Ruby, Node.js, Javascript (Browser), Kotlin, Swift etc.] # AWS Identity and Access Management ### Introduction AWS IAM is one of the foundational services in the cloud, responsible to control access to AWS resources in a particular AWS account across the Internet. During sign up process on AWS, Root user is created automatically and hence they have unrestricted access to all AWS services in that account. Therefore it is vital to have MFA on root account. In addition to this, following are the best practices: * Disable Root access key for cli usage * Rotate credentials * Don't share root user credentials Hence, for day to day activities, it is not at all recommended to use root account. So what's the solution? AWS provides IAM (Identity and Access Management) service to create users, groups, roles and policies to provide privileges to the AWS cloud account as required, following the principle of least privilege. As the name suggests, **IAM is responsible for Identity and Access which is cooler way of redefining Authentication and Authorization. Under Authentication, Creation and management of identities happens and under authorization, permissions to identities happens.** * [x] When the word "Identities" is written, it represents any of the following: * IAM User - Can be a User or an Application * IAM Group - can be a collection of IAM Users * IAM Role - Used to assign temporary credentials that has privileges to perform and access certain AWS resources; so IAM Users, application or external identities can assume the role. * [x] When the word "permissions" is used, it usually refers to permissions inside a policy, which is used to do access control (a way for authorization mechanism) for the above identities. IAM user with Administrative privileges is the 2nd most powerful account that can be used safely instead of root account for day to day privileged operations in the AWS cloud. *Note by default, AWS IAM Identities don't have access to billing console. It needs to be explicitly permitted.* AWS IAM can provide secure access, provide granular control over that access and also ensure that access is temporary. This is all facilitated to AWS IAM Identities, which can be a IAM user or IAM Group or even IAM role. ### Account ID and Account Alias Account ID (which is 12 digit numerical id) is created on its own when one sign-up for AWS. Now, one can simply create account alias which will be bound to account id. Its similar to why DNS is being used. Humans cannot remember numbers, so they need simple names and that is where this comes up.

Once created, the new URL for login for IAM Users can be as follows:

### ARN All resources have unique Amazon Resource Name (ARN) It is of the following format ```json arn:partitition:service:region:account-id:resourcetype/resource/qualifier ``` Note that partition in arn can also be "aws-us-gov" or "aws-cn", if not, it is usually "aws" for most case Example: ```json arn:aws:s3::::images-of-football The reason region and account id are not mentioned in this is because s3 is a global service, so no region is required and also when region and account is not give, it is assumed that default account and default region is selected resource type is also set to default to bucket resource is images-of-football as this is a bucket arn ``` ``` arn:aws:ec2:us-east-1:112233445566:instance/i-054dsfg34gdsfg38 Here, one can can see that for ec2, resource type is given as instance and resource is instance id (i-054dsfg34gdsfg38) If it ever requires that all instances would have ARN of arn:aws:ec2:us-east-1:112233445566:instance/* ``` ### AWS STS One of the foundational service for IAM is AWS STS. AWS Security Token Service (AWS STS) is a web service that enables to request temporary, limited-privilege credentials for trusted users in form of tokens that can control access to AWS resources. The duration of the validity of the token or lifetime of the token can range from 15 mins and go upto 36 hours. The working of AWS STS can be explained as follows:

Source: https://cloudpatterns.org/wp-content/uploads/2023/03/aws-sts-chart-2.png

{% hint style="info" %} **Each IAM entity (user, group, or role) has a defined aws:userid variable.** {% endhint %} Note that a **Session ARN is a unique identifier for a session when a user assumes an IAM role. This session is created when an IAM user or another role assumes a role using the AssumeRole API call**. The Session ARN typically looks like this: arn:aws:sts::ACCOUNT\_ID:assumed-role/ROLE\_NAME/SESSION\_NAME ### AWS Requests {% hint style="info" %} To access a resource, the principal sends a request to AWS with the help of following components: * Name of the principal that can be IAM User or IAM Role * Actions or operations to perform * Resource over which the operation has to be performed * Resource specific data * Environment in which the request has to be performed (Specific conditions such as IP if implemented) {% endhint %} A sample of Request and Response made using AWS CLI can look like this:

> **For those who are interested to know how to intercept AWS CLI Requests in Windows/Linux, here are the steps:** > > 1. Download Burp Certificate and head over to Linux machine or ubuntu wsl and issue the command: openssl x509 -inform der -in path/to/Burp-certificate.der -out /path/to/Burp-certificate.pem > 2. copy the contents of .pem file generated in above step. > 3. Append the above certificate in "C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem" at the end. Note for linux, the path is "/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem" > 4. Download the AWS-Sig4 Burpsuite extension from Portswigger. > 5. Import the keys into the extension. > 6. In Windows CMD prompt, use the following commands for Windows/Linux (for particular session): > > * HTTP Proxy > > `set HTTP_PROXY=http://127.0.0.1:8080` OR`export HTTP_PROXY=http://127.0.0.1:8080` > > * HTTPS Proxy > > `set HTTP_PROXY=http://127.0.0.1:8080` OR`export HTTP_PROXY=http://127.0.0.1:8080` > 7. Finally use awscli commands and it would be intercepted. > 8. One can also use this to change User Agent if required # AWS Organizations As the name suggests, it is used to organize different AWS Accounts (which can be used by different departments in a company) to have consolidated billing and gives the flexibility for different departments to have different policies:

Source: https://td-mainsite-cdn.tutorialsdojo.com/wp-content/uploads/2018/12/AWS-Training-AWS-Organizations-2.jpg

By default, when one goes to AWS Organizations menu, the context can be as shown below:

Now, one can create an OU and then create an account under it:

# Users Every IAM user receives Amazon Resource Name (ARN). It is similar to object id in Active Directory and is of the format: * arn:aws:iam::\:root * arn:aws:iam::\:user/iam-username More details about ARN are already explained [here](/aws/aws-identity-and-access-management#arn). ### Console Access When logging into AWS, using root account, one can simply navigate to , which will redirect to domain [signin.aws.amazon.com](https://signin.aws.amazon.com)

However, when logging using IAM account, one can simply use the 2nd option as mentioned in the above screenshot and simply provide the options as mentioned in the below screenshot:

Now one can either provide account ID (which is 12 digit numerical id) created when one sign-up for AWS or simply provide an account alias (as its difficult to remember the 12 digit number) in place of account id to access further and lastly provide the IAM user and password to login, provided the IAM user has console access.

Now, one can understand the steps can be referred to see how to create an IAM user and provide the user console access in IAM console. If its the first IAM user, then one has to use root account privileges. To be specific, the following account that is created is the administrator account which can be used to manage day to day operations in AWS.

Instead of creating user with administrator access individually, a group has been created with administrator privileges as shown [here](/aws/aws-identity-and-access-management/groups-and-roles).

{% hint style="info" %} Every IAM user access resources based on assigned permissions. {% endhint %}

{% hint style="info" %} Note, one can also access AWS console via https\://\.signin.aws.amazon.com for AWS IAM Users to login {% endhint %} ### Forgot Password

As said above, IAM users would have to go to their IAM administrators for password reset. This is particularly useful to prevent SSPR (Self Service Password Reset) abuse in phishing attacks. ### Programmatic Access Next, one can also create access keys to let the user access AWS programmatically or using cli. It is like **user-id (Access Key ID)** and **password (Secret Access Key)** for programmatic access. Moreover, the best practices for them are as follows: * Rotate access keys every 90 days * Store access keys in secure manner **Note**: Refer [here ](https://guides.radifine.com/how-to-securely-access-and-transfer-file-to-an-ec2-instance-in-private-subnet-of-vpc-in-aws#heading-step-24-for-access-via-awscli-more-work-is-necessary)on how to use access keys and MFA to create[ short term credentials](https://notes.radifine.com/aws/aws-iam/federation#aws-sts) for best practices. This is particularly useful to avert the risk of long term damage in case the user's system is compromised. **Note**, if the access key is for an application, then one can create a different user with only cli access that has access to particular resource in AWS, store it in AWS secrets manager and then give EC2 IAM instance profile to fetch only that particular AWS key from AWS secrets manager as security best practice * Delete unused keys * Monitor usage of access keys It can be created as shown below:

{% hint style="info" %} Note: If the primary use case of access keys is going to be to use it to perform operations across AWS service to do repetitive tasks or save time to do something over cli for better productivity, then **CloudShell** can also be used as shown below. It is simply better because there is no requirement to ever store access keys locally in user's system in this case. {% endhint %}

Note that Access Key ID starts with AKIA for long term credentials and ASIA for short term credentials ### Password Policy It is basically used to configure the following: * Character requirement like small alphabets, capital alphabets etc. * Minimum Password Length. * Validity of the password in terms of its expiration. * Number of times it is allowed to do attempt entering password for users. * Reusability of Passwords Password Policy can be managed as shown below:

### Setting up MFA This is used to setup extra layer of security. Login to AWS console and go to Security Credentials

For demo, lets configure using Software based Authenticator app. OneAuth, Microsoft Authenticator, Google Authenticator, Orange Authenticator etc are supported. We can also configure using browser extension from authenticator.cc

# Policies and Permissions ### IAM Policy Permission policy is an authorization mechanism. Hence, until the policy is attached to an AWS Identity, that identity cannot do anything. AWS Identities are authorized to perform or forbidden to perform certain actions based on these policies. This is possible a single policy can include multiple permissions or statements. Also, multiple statements in policies are evaluated in OR logic IAM Policy is a JSON-formatted document in which permissions are listed. It consists of 4 different categories such as Effect, Action, Resource and Condition where Effect can be to allow or not to allow, action can be create or delete, resource can be AWS service and condition can answer whether to perform or not based on if it is true or not. These policies itself are like template. These needs to be attached to an AWS identity like user, group or IAM Roles for them to shape up to become permissions.

{% hint style="info" %} Note, for ease of creation of an IAM policy, one can always try using the following: {% endhint %} IAM policy is written in Access Policy Language (a fancy name indeed for a JSON Document)

Note that there is a Version: 2012-10-17 statement in the above policy. It states when AWS created this version of policy writing. there was 2008 also, which was older version. Also, statement ID (SID) is optional The Action follows Noun-Verb format (example: s3(noun):Get-Object(ACL) The Condition follows Key:Value pair (example:"Condition":{"StringsEquals":{"aws:username":"Senapati"}}) and value is case sensitive by default, otherwise condition can be set to ignore case for value of Senapati. Also Policy condition types can be global condition keys or service-specific condition keys, examples of which is as follows:

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html

Please note that for these policies to take effect, these need to be attached to an IAM Identity like User, Group or Role, thereby becoming permissions for the attached identity. ### Policy Creation Steps to create a policy are as follows:

{% hint style="info" %} Personal Interpretation, the AWS IAM policies are evaluated on every action, especially for Console Access, meaning: Let's just a user ABC is logged onto the console and has the permission in a policy attached to him/her that allows to upload a file inside S3 bucket. Now, if the admin or any user with change permissions over IAM removes the policy that allows ABC to upload files, then he/she would be unable to do so, the next time they want to perform the file upload permission (PutObject) inside the s3 bucket. {% endhint %} Note that there is a ### Summaries in IAM The policy summary have hyperlinks to Service Summary and it has hyperlinks to action summary. These summaries are an absolute gold when it comes to analysing and understanding a particular IAM policy and can be navigated as shown below:

Source: https://docs.aws.amazon.com/images/IAM/latest/UserGuide/images/policy_summaries-pol-sum.png

### NotResource, NotAction NotResource is exact opposite of Resource in an IAM policy, With NottResource, one can setup policy like Everything except the one mentioned in NotResource

NotAction matches everything except the specified action. When effect is allow, it allows all actions except the ones mentioned in NotAction. When effect is deny, it denies all actions except the ones mentioned in NotAction.

### Types of Identity-based Policies * **Managed Policies** * [x] These managed policies are independent resources with their own unique ARNs * [x] They can be attached to multiple AWS Identities * [x] It can be further categorized as **AWS managed** (cannot be edited or updated by anyone including root except AWS and used for common use cases\[also referred as **Job Functions - which are like full job role that can be assigned to a person**]) or **Customer managed** (as the name suggests, they are managed by the customer of that particular AWS account) * [x] It has version control, i.e., it is possible to have multiple versions of a particular policy. Max number of policy versions are 5. * [x] The newest version of the policy is default, so if one wants to choose another policy version to be applied, simply change that version as default. * **Inline Policies** * [x] They are embedded in a single identity. * [x] They are inline hence cannot be attached to other identity. * [x] They are also managed by the customer themselves, hence customizable at any given time ### Policy Evaluation (I) {% hint style="info" %} **An AWS identity with multiple policies also follows OR logic when evaluating the final access** {% endhint %} IAM policy works with the following precedence: Explicit Deny -> Explicit Allow -> finally Implicit Deny. Example: A new user is created and no access was given during creation time, so access to all services in AWS are implicitly denied, post that, if the user is allowed to access certain things, then that would be the case of explicitly allowing or granting them permissions to do certain task, and then if the same user is explicitly denied to access anything then takes the highest priority for denying access to certain AWS services or resources as applicable. This is particularly useful when one has to temporary ban the usage of AWS account by assigning Explicit Deny on the account that prevents the usage of the account. The evaluation logic flowchart whenever any action would be attempted to be performed is as follows:

Source: https://docs.aws.amazon.com/images/sns/latest/dg/images/AccessPolicyLanguage_Evaluation_Flow.gif

The evaluation logic can be summarized as follows: * By default all requests are denied. * An explicit allow in any permissions policy/ies override the default. * An organization's SCP, IAM permissions boundary or a session policy overrides the allow. * In short, they must all allow the request for anything to be permitted * An Explicit deny in any policy overrides any allows The decision making algorithm flowchart for policy evaluation considering all relevant policies can be as follows:

Source: https://docs.aws.amazon.com/images/IAM/latest/UserGuide/images/PolicyEvaluationHorizontal111621.png

### Policy Simulator Policy simulator allows a way to test and simulate policy changes without applying them. Testing can be done based on policies attached to AWS IAM Identities or based on services, actions and resources. It can be useful to test overlapping policies to understand that if an action is allowed or denied and which statement works and which is preceded Head over to

### Permission Boundaries It indicates maximum permissions that an IAM identity can have and can only be attached to an IAM user or role. Note that there can be one permission boundary per IAM identity. This is useful to prevent accidental application of IAM policies to an AWS IAM identity in case there are multiple people in organization that has the privileges to change or add permissions to a Identity. It works in conjunction with Permission policy as shown below:

Source: https://securityboulevard.com/2023/10/aws-permission-boundary-what-is-it-and-how-to-use-it/

One can setup a boundary as shown below:

So effectively whoever assumes the role of s3kacommander, will only have the common access which would be that of s3bucketaccess policy simply because it has less permissions and they are the only actions rather than the overly permissive policy of tests3fullaccess and therefore senapati will only be able to access based on the condition set even when senapati assumes the role of s3kacommander ### Other Policy Types The entire discussion that happened above is for Identity-based policies. Other tyes of policies are as follows: * **Resource-based Policies**: Available for certain resources such as [S3 ](/aws/aws-storage-services/s3#resource-based-policy)or SQS * **Organization Service Control Policies:** * [x] To understand this, it is recommended that one reads about [AWS Organizations](/aws/aws-identity-and-access-management/aws-organizations) first and understand OU or Organization Unit * [x] Sometimes despite privileges, a user is unable to execute commands. This is despite the fact one can have privileges. The reason for that is SCP i.e., Service Control Policies which is always set in Management Account * [x] It sets the maximum permissions for an OU and limits permission for entities in member accounts, including each AWS root Account * **Access Control Lists**: Relevant to storage and buckets and ACLs in firewall (especially for public access). It defines which principals in another account can assume a role. * **Session Policies:** * [x] When one is assuming a role, session policy can be applied to limit what all role can do. It is similar to permission boundary, but there is a difference based on applicability. Session policy is applied only when one is assuming a role. So if the organization wants to limit what a user assuming a role can and can't do, then it can be defined in session policy.

When an IAM user assumes a role to access a s3, if s3 has a resource policy, and if it mentions Resource ARN, then internally the resource policy is converted to IAM policy to provide access accordingly. If it is session ARN, then it is not converted. * [x] As per above diagram Scenario 1: A resource-based policy can specify the ARN of the user or role as a principal. In that case, the permissions from the resource-based policy are added to the role or user's identity-based policy **before the session is created**. The session policy limits the total permissions granted by the resource-based policy and the identity-based policy. The resulting session's permissions are the intersection of the session policies and the resource-based policies plus the intersection of the session policies and identity-based policies. ```json // Identity-Based Policy for DataAccessRole: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] } // Resource-Based Policy on example-bucket: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:role/DataAccessRole" }, "Action": "s3:*", "Resource": "arn:aws:s3:::example-bucket/specific-object" } ] } // Session Policy: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::example-bucket/specific-object" } ] } // Effective Permissions The effective permissions will be limited to: Actions: s3:GetObject, s3:PutObject on arn:aws:s3:::example-bucket/specific-object ``` * [x] As per above diagram Scenario 2: A resource-based policy can specify the ARN of the session as a principal. In that case, the permissions from the resource-based policy are **added after the session is created**. The resource-based policy permissions are not limited by the session policy. The resulting session has all the permissions of the resource-based policy *plus* the intersection of the identity-based policy and the session policy.

// Identity-Based Policy (attached to DataAccessRole):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

// Resource-Based Policy on example-bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:sts::123456789012:assumed-role/DataAccessRole/*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::example-bucket/*"
    }
  ]
}

// Session Policy (applied during the session):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::example-bucket",
        "arn:aws:s3:::example-bucket/*"
      ]
    }
  ]
}

// Effective Permissions
The effective permissions will be limited to:
Actions: s3:GetObject, s3:PutObject, s3:ListBucket for all objects in arn:aws:s3:::example-bucket

* [x] As per above diagram Scenario 3: A permissions boundary can set the maximum permissions for a user or role that is used to create a session. In that case, the resulting session's permissions are the intersection of the session policy, the permissions boundary, and the identity-based policy. However, a permissions boundary does not limit permissions granted by a resource-based policy that specifies the ARN of the resulting session. ```json // Identity-Based Policy for DataAccessRole: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] } // Permissions Boundary: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "arn:aws:s3:::example-bucket/*" } ] } // Session Policy: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::example-bucket/specific-object" } ] } //Effective Permissions: The effective permissions will be limited to: Actions: s3:GetObject, s3:PutObject on arn:aws:s3:::example-bucket/specific-object ``` {% hint style="info" %} Note that **Resource groups** and **Tag editor** are another methods to provide access via IAM {% endhint %} ### Policy Evaluation (II) The evaluation logic can be summarized as follows: * By default all requests are denied. * An explicit allow in any permissions policy/ies override the default. * An organization's SCP, IAM permissions boundary or a session policy overrides the allow. * In short, they must all allow the request for anything to be permitted * An Explicit deny in any policy overrides any allows The decision making algorithm flowchart for policy evaluation considering all relevant policies can be as follows:

### # Groups and Roles ## IAM Groups An IAM Group is a collection of IAM Users. This is used to prevent applying common policies (bunch of permissions) to users individually and instead apply them to the group, of which the users will be part of. This works, because the policies are flow down to individual users, as inheritance of the group membership. Hence, any policy changes done to an AWS Group will automatically be reflected to AWS IAM Users within the particular group. {% hint style="info" %} Note that IAM Group doesn't have unique identifier i.e. ARN {% endhint %} {% hint style="info" %} There is no concept of group within a group in AWS i.e. Groups nesting is not present {% endhint %} ### Group Creation Creating Group with Administrator privileges:

## IAM Roles IAM Roles are temporary permissions given to either people or apps to perform a certain task. Consider IAM Role as a hat, whoever assumes it, gets the permissions attached to the role. An example of It being particularly useful is to consultants or contractors who join the company for a specific project and don't need permanent access to the client's company. They are a also popular choice for Applications i.e., whatever application requires, give it via an IAM role. IAM Role can be assigned to an AWS service or AWS IAM user account (same account or even different AWS account) or a federated identity (both Web based or SAML 2.0). Benefits: * No long term credential * No requiring embedding of permanent credentials to applications * No IAM user creation for consultants/contractors or other AWS account users * Allows for federation AWS AssumeRole enables entities like IAM users or AWS services to temporarily acquire the permissions and access rights of another AWS entity by assuming a specific role. This delegation is defined through a trust policy, which specifies the trusted entities that can assume the role. Here’s how it works: 1. An entity requests to assume a role by calling the AssumeRole API operation or assumes the role via the AWS Management Console. 2. AWS validates the entity’s permissions against the trust policy attached to the role. The trust policy specifies who can assume the role, typically within the same or different AWS accounts. 3. Once authorized, AWS generates temporary security credentials for the entity, granting access based on the permissions defined in the role’s policies. 4. The entity can now perform actions and access resources using the temporary credentials, limited to the permissions of the assumed role. To assume an IAM role, one needs a trust relationship that will be configured in trust policy. Just like MSFT Active Directory, there is a trusting account (whose resources are accessed) and there is trusted account (that contains the users accessing the resources of the trusting account). So, to summarize, for IAM Roles to be effective, there are two types of Policy that are required: * Permission Policy - Define what actions and resources for an IAM Role * Trust Policy - Define who is allowed to assume an IAM Role. Note in trust policy, another role is also allowed, i.e. role within a role or role chaining.

Source: https://miro.medium.com/v2/resize:fit:640/format:webp/1*e_faUe39DVwrnAlg2nSOfg.png

A Trust policy looks as below: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:user/AnotherUser" }, "Action": "sts:AssumeRole" "Resource": "arn:aws:iam::123456789012:role/Role-A" } ] } ``` ### Service Role AWS services assume the service role to perform action on user's behalf. Two types of service roles, with the difference between the two is that one is user defined (service role) and other can be AWS predefined (example EC2 Autoscaling, CloudFormation etc). Note it is usually not possible to delete AWS managed Service-linked policy even by the root account

Source: https://anjanashankar.com/2021/05/20/role-based-access-aws/

{% hint style="info" %} **Note that instead of creating long term credentials, as shown in the above image, the EC2 instance profile assumes an role that has access policy to S3 buckets. So, by the virtue of presence of applications on the EC2 instances, they are able to generate temporary credentials to access S3. Here, the EC2 has that trust relationship in form of service role, this means that if tomorrow instead of XYZ application, ABC application gets deployed on that EC2 instance, they will still be able to access the S3 bucket as they are associated with the EC2 rather than long-term credentials which can be hardcoded in the application. This also means, if more than one application is hosted on EC2, every app would be able to access the s3 bucket.** **Basically, the applications access the s3 bucket using IMDS (Instance Metadata Service) endpoint on a link-local address of 169.254.169.254 to get short term credentials at runtime and access s3 using that.** {% endhint %}

### Role Creation Note, for role creation, first ensure that AWS Policy is created as shown on below link: {% embed url="" %}

The sample role can then be attached to an IAM role in EC2 as shown below during EC2 instance launch configuration:

{% hint style="info" %} Note that you can also attach the role in existing EC2 {% endhint %} ### PassRole There is something called IAM:PassRole, which allows users to delegate permissions in form of Roles to AWS Services so that they can access other resources or AWS Service. Note that usually the IAM Admin create these roles and allow users to delegate the role to AWS services. Visually, it can be demonstrated as follows: ```scss // Generated by ChatGPT [ User/Service ] --( PassRole )--> [ IAM Role (EC2ReadS3Role) ] | | | v | [ EC2 Instance ] | | |------------------------------------------> [ S3 Bucket ] ``` ### Delegation - For Cross Account Access Just like in MSFT's Active Directory, there is concept of delegation in AWS, i.e., permissions can be given to resources a user controls. As said earlier, the resource holding account is trusting account and users of the account who will access are trusted accounts and therefore trust policy is gonna be required. So, in short, cross account Role is used for delegation Steps are not given as it requires opening and signing up another AWS account, so due to economical constraints, request you to follow the guide [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). ### Effective Use case IAM services can interact with each other using IAM Role example: AWS Serverless LAMP stack

Source: https://d2908q01vomqb2.cloudfront.net/1b6453892473a467d07372d45eb05abc2031647a/2020/06/01/Screenshot-2020-06-01-at-17.59.43.png

Source: https://docs.aws.amazon.com/images/IAM/latest/UserGuide/images/roles-usingrole-ec2roleinstance.png

### Revoking a session Sometimes due to unforseen circumstances, an IAM Role assumed session has to be revoked. AWS does that by attaching explicit deny to that session, so that when any action is evaluated, it would be denied

Follow the steps to revoke the steps from here: # Federation The idea of Federation in AWS IAM is that it allows organizations to bring existing authentication in form of on-prem identities (MSFT's Active Directory) or online identities (Google Workspace Identity or MSFT Admin center users or MSFT Entra-ID etc) here in AWS and assign permissions to them, i.e. use existing corporate credentials for authentication and authorization into AWS Console, CLI or direct API calls. ### Types AWS supports different types of Federation, as given below: * Cross Account Access * SAML 2.0 * Web Identity * AWS Directory Service

Source: https://d1.awsstatic.com/security-center/SecurityBlog/federated_auth_with_adfs_5.69a994ecc1645a52e53648efaae211cdfcfaa55a.png

### Cross Account Access This can be best understood from the following diagram:

Source: https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2016/11/16/Diagram_KZ_111616_d.png

Please note that temporary security credentials can be valid from 15 minutes to upto 12 hours. ### SAML 2.0 - For existing Corporate Users Security Assertion Markup Language or SAML is an open standard supported by many identity providers (IdPs) such as auth0, Microsoft Active Directory, Azure, Google Workspace Identity etc. It requires first setting up metadata, which lays the ground rule for how the communication will happen between the IdP and SP (AWS in this case). Once the authentication is done between IdP and user, then SAML assertion document will be sent by IdP to SP to prove that the user is who he/she claims to be and then based on authorization set in AWS based on IAM role, will get access to AWS services. The following will explain the process better:

Source: https://docs.aws.amazon.com/images/IAM/latest/UserGuide/images/saml-based-sso-to-console.diagram.png

Note that just like there is sts:AssumeRole, for SAML, there is sts:AssumeRolewithSAML and for Web Users, there is sts:AssumeRolewithWebIdentity ### Web Identity - For general Web users Popular method for Application logins. It supports Amazon, FB, Google or any **OIDC** Connect compatible Identity providers and integrates with Amazon Cognito as identity broker. Best benefit of these is that End-Users don't need AWS account, and they can still login and access AWS resources (like an image within S3 bucket, let's just say via the application) . The IAM Role that has access to the particular resource can be said to be assumed by the user proving his/her identity using above mentioned providers.

Source: https://miro.medium.com/v2/resize:fit:720/format:webp/0*adBXuxmDWcJWCgth.png

### AWS Directory Service There are basically 3 options, Simple AD, AWS Managed Microsoft Active Directory and Active Directory Connector * Simple AD - Standalone Samba4 powered Active Directory compatible server. Supports Kerberos and does have user accounts, group membership, group policies but no support for MFA, no trust relationships or PowerShell etc. It works well with AWS apps and services such as Amazon Connect, Amazon QuickSight, Amazon WorkDocs, Amazon WorkMail, Amazon Workspaces etc. * AWS Managed Microsoft Active Directory, as the name suggests, is simply Microsoft AD in the cloud. It is usually a pair of domain controllers which supports almost all features, supports RADIUS, MFA and secure TLS communication. It requires a VPC with two subnets in different availability zones for redundancy, but does not support NAT. This can be used to keep AD-aware workloads on AWS as well as provide SSO to Office365 and other cloud apps. It is also provides connectivity to On-Prem.

Source: https://docs.aws.amazon.com/images/directoryservice/latest/admin-guide/images/ms_ad_use_cases2.png

* Active Directory Connector - A simple gateway to redirect all requests to on-premises Microsoft AD over a VPN connection or AWS direct Connect. It also requires Kerberos pre-auth enabled. It can also be used to join EC2 instances directly to cloud.

Source: https://aws.amazon.com/blogs/security/how-to-connect-your-on-premises-active-directory-to-aws-using-ad-connector/

# Access Control (via available tools) ## IAM Access Control Tools that can help in monitoring Access Control are as follows: * **Credential Report** - It can be used to identify unused credentials and maintain compliance evidences by providing a CSV about password and access keys usage of a particular IAM user with its ARN, creation time, password or access key last used, last changed, next rotation date and MFA active, last used service and also X.509 certificates

* **Access Analyzer** - A feature in IAM that allows to monitor the usage of policies so that unwanted policies can either be modified or removed. It has service name, policy granting permissions, last accessed, access by members. This is particularly useful to see if a person who has changed jobs or moved from the organization does not have that same level of access, which it discovers by telling IAM policy not getting used. Apart from all these, it can be used to monitor external access also

In short, it identifies resources shared with external principals for few resources such as: * [x] S3 * [x] AWS KMS keys * [x] Lambda functions * [x] AWS IAM Roles * [x] Amazon SQS queues * [x] AWS Secrets Manager secrets Feel free to navigate to the following link to have a more hands-on approach: * **AWS CloudTrail** - One of the foundational AWS logging service that can be used to monitor as collect forensic evidence. More details on[ AWS CloudTrail page](/aws/other-aws-services/logging-and-monitoring/aws-cloudtrail). * **AWS Trusted Advisor** - Useful for newly created AWS account to know the must do's. It helps with Cost optimization, performance, fault tolerance, security etc. **It has lot of many checks, but are behind a paywall. First thing post setup** should be to check suggestions from IAM

* **AWS Config** - This paid service from Amazon, assesses, audits and evaluates configurations of AWS resources, but where it really shines is its nature for enforcement. It is kind of File Integrity Monitoring but for AWS configurations. Hence the feature list is as follows: 1. Continuous Monitoring of configurations when it differs from baseline 2. Continuous Assessment of configurations when it differs from baseline whether the change is permitted or not. 3. Change Management - view all the changes 4. Operational Troubleshooting - if any flow breaks, then it can help in that 5. Helps in Compliance

W\.r.t IAM, AWS config can help in: * [x] Monitoring of Root Account MFA enabled or disabled * [x] Changes to IAM password policy * [x] IAM policy blacklisted check helps in monitoring if a blacklisted user has suddenly got any policy. * [x] Monitoring if a group has suddenly got additions or subtractions in its members (useful for checking if any user is part of Admin access) * [x] If an IAM User has any policies applied to them * [x] If an IAM group has any user or is it is blank Setup example for AWS Config:

{% hint style="info" %} Note, one can create its own Rules using Lambda and Guard {% endhint %}

# AWS Cognito Amazon Cognito is a technology that lets app developers implement user sign-up, sign-in and role based access control in web and mobile applications to support serverless authentication and authorization, and when using federated identity, can act as identity broker. Note RBAC is implemented when users are divided into groups and each group is then provided different permissions Practical Example, Streaming Platforms can use AWS cognito to let users sign-up and sign-in through Google or Microsoft Accounts or Apple ID and different users would have different viewing permissions to the content the streaming platform will host based on if a user is on ad-supported free tier model or paid premium subscriber. ### Identity Sources in Cognito: Simply speaking, they represents the source of authentication or user pool. * **User Pool Directory** - It leverages Cognito itself as identity source and managing user directory by using cognito user pool that stores identity information (username and password), user profile information (photo, URLs etc), compatible with AWS Lambda as well as supports categorization of users into groups * **Identity Federation** - It enables users to sign-in with an existing identity source (enterprise, social) (Example: Sign in with MSFT AD, Sign in with Google etc) * **Identity Pools** - It enables to grant users access to other AWS services. Note this can be used in conjunction with user pool or even separately. Use cases - Direct service integration (client apps use aws direct creds of the service to integrate and communicate with AWS services) and Permission pass-through (it can be AWS Lamda function or other functions that uses the permissions of the user and not the function itself) ### Authentication flow: History and typical use case of AWS Cognito * Previously, the web application tier would be divided into client and server where upon authentication, server would provide cookies to client and subsequent requests would be interacted with that session cookie. Usually the application logic was mainly on server side and client used to only display rendered output. * This got changed to API based architecture, where the application logic or main heavy lifting shifted to client side and only required data would be fetched from server. So app clients would communicate with API tier on server i.e., instead of web requests, API requests (which can be said as standalone functions) would be made and based on data received, app at client side can display output. Now, instead of session cookie, the better solution that was implemented would be to use either use OIDC standard via JWT tokens or Cognito User Pool. Either OIDC that allows third party identity providers such as Google, Facebook, Microsoft etc or Cognito User pools allows to authenticate and provide user profile back in those JWTs (this is what actually replaces session cookies and information is provided back and forth into JWT components which are trusted due to the it being digitally signed). * Note JWTs have 3 components: first one is HEADER shows token type and algo, the second one being PAYLOAD has the main part where data about the user and its relevant profile is present and last component is the SIGNATURE part. There are basically three types of JWTs at play, which are ID Token (Contains user identity info), Access Token (Contains groups and scopes) and Refresh Token (Used to get new ID and access tokens), which are obtained upon successful authentication and validity of these tokens can be changed and is mostly configured inside Cognito user Pool . * It is these access tokens that is used to access resources in AWS. The validity of a token can be verified whether it is for specified user pool (iss), whether the client\_id is valid, whether toke is not expired, whether token is not expired(exp), whether is time is in past (iat), verifies the token scope, verifies the signature with the user pool keys. * So, how this works is that once authentication happens, let's just say using Cognito user pool, all three JWT tokens are received, which will be sent in future API requests. * To make this more meaningful, IAM roles can be created that has access to specific AWS services like S3, so whoever assumes the role can access the required resources and these IAM roles are then mapped to Identity pool. So the authentication flows works like that first do authentication and obtain JWT tokens. Using the access tokens, which on user basis, can obtain short term credentials for particular AWS IAM role, which can then be used to obtain resources from other AWS services like S3

Source: http://blog.jacobmarks.com/2016/12/amazon-cognito-user-pool-admin.html

### Creating User Pool

### Creating Identity Pool

# AWS IAM Identity Center ### Features AWS IAM Identity Center (previously known as AWS Single Sign On) offers a secure solution for the central management of workforce identities, allowing to establish or link user identities seamlessly across AWS accounts and applications. That means, if an organization have multiple AWS Accounts, across AWS Organizations, a single sign on to all AWS accounts can be enabled. AWS recommends Identity center as the method for implementing workforce authentication and authorization within AWS, catering to organisations of diverse sizes and types. With IAM Identity Center, AWS customers gain the ability to efficiently create and oversee user identities within the AWS ecosystem. It can be used to integrate existing identity sources, including but not limited to Microsoft Active Directory, Okta, Ping Identity, JumpCloud, Google Workspace, and Microsoft Entra ID (Azure AD).

### Use-Cases **Enable multi-account access to AWS accounts:** IAM Identity Center simplifies multi-account access management by allowing users to employ their directory credentials for seamless single sign-on access to various AWS accounts. Through their personalised web user portal, users can conveniently view and access their assigned roles across multiple AWS accounts from one centralised location. This streamlined authentication experience extends to AWS CLI, SDKs, and the AWS Console Mobile Application, ensuring a uniform and secure authentication process.

Source: https://cdn.hashnode.com/res/hashnode/image/upload/v1684949549669/df98e960-5aed-49fc-9e1c-f8e4f804eaaa.png?auto=compress,format&format=webp

**Enable single sign-on access to AWS Apps:** IAM Identity Center offers integrated access to AWS applications such as Amazon SageMaker Studio, AWS Systems Manager Change Manager, and AWS IoT SiteWise, enabling zero-configuration authentication and authorization. These applications seamlessly interact with IAM Identity Center to provide a unified view of users and groups, enhancing resource sharing and collaboration within the application environment. This integration ensures an efficient and consistent user experience for accessing AWS applications. {% hint style="info" %} **Access EC2 instances running Windows using IAM Identity Center** {% endhint %} **Other Use cases** include Logon to M365 applications, Salesforce and SAML based applications, etc. It can also work with multiple Identity Providers such as Active Directory, Okta, IAM stores etc. {% hint style="info" %} A web-based portal where users can sign in to access their assigned AWS accounts and applications is generally used as shown above. Organizations can customize the portal with their branding for a cohesive user experience. {% endhint %} By default, access to the AWS portal can be obtained via a URL following this format: ‘**d-xxxxxxxxxx**.awsapps.com/start.’ But most organizations customize it as per company branding.

### Components When setting up IAM Identity Center, AWS account will typically create an **SSO instance**. It serves as the centralized management point for user identities, access controls, and authentication across multiple AWS accounts and applications. Here are the main components: **AWS Accounts** * AWS consumers can link multiple AWS accounts to your AWS Identity Center instance, allowing centralized management of access across those accounts. #### **Identity Sources** Individual identities or collection of such identities that can be created and managed within AWS Identity Center or imported from an external identity provider (IdP). * **Internal Users**: Users can be created directly in AWS Identity Center. Custom attributes can be created to store additional information about users, for finetuned access control and management * **External Identity Providers**: Integration with external IdPs like Microsoft Active Directory, Okta, or SAML 2.0 providers for user authentication and management. #### **Permission Sets** * A permission set is a collection of IAM policies that define what actions users can take and which resources they can access within an AWS account. * It can include AWS managed policies, customer-managed policies, or custom policies. * Permission sets are assigned to users or groups for specific AWS accounts, determining what resources and actions they can access. * When creating a permission set, AWS consumers can define how long a user’s session remains active before requiring re-authentication, adding an extra layer of security. * Sample Policy example: ```json // Created by ChatGPT // Role that has read only permissions for Account A: 123456789012 and Account B: 210987654321 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:RunInstances", "ec2:StartInstances", "s3:PutObject", "s3:GetObject", "lambda:InvokeFunction" ], "Resource": [ "arn:aws:ec2:us-east-1:123456789012:*", "arn:aws:s3:::example-bucket-account-a/*", "arn:aws:lambda:us-east-1:123456789012:function:*" ] }, { "Effect": "Deny", "Action": "iam:*", "Resource": "*" } ] } ``` {% hint style="info" %} **AWS SSO Admin** refers to the AWS service that allows you to manage AWS Single Sign-On (SSO) features programmatically through the AWS CLI (Command Line Interface) or SDKs. {% endhint %} ### Device Code Authorization There is something called as the device code flow (The *device code* grant type) which is generally used on embedded devices. Example: While setting up Google Chromecast, it shows the pin and then one has to enter the same in Android's Home App, so that it reflects there. It works by starting the flow on the device (think apple TV, etc) and then providing the user a URL and 'user code' to access on a more powerful endpoint. The user will navigate to this URL in a full featured browser and provides auth and the user code. Meanwhile, the device, without a browser, is polling a specific endpoint with that same 'user code'. When the user completes authentication, the endpoint will return an access token or other authentication information to the embedded device. Similar is the case with AWS and Azure as Cloud providers when it wants user to authenticate with AWS CLI. Following are the steps: 1. The user 'registers their device' (creates an app) using the /client/register endpoint i.e., the client application (AWS CLI) registers an OIDC client by calling [sso-oidc:RegisterClient](https://docs.aws.amazon.com/singlesignon/latest/OIDCAPIReference/API_RegisterClient.html). It does not need authentication to do it. It will return an OIDC Client ID and Secret 2. The user 'authorizes their device' (starts Device Authorization Flow) using the /device\_authorization endpoint. This endpoint takes the client ID, client secret, and start\_url (provided by AWS). The result will be a DeviceCode, UserCode, and VerificationUrl. The client application calls [sso-oidc:StartDeviceAuthorization](https://docs.aws.amazon.com/singlesignon/latest/OIDCAPIReference/API_StartDeviceAuthorization.html). This generates an URL that looks like *

.amazonaws.com/?user\_code=ABCD-DCBA* 3. The user is asked to visit the verification URL and enter the code. Because this is typically all completed on the same machine, the code can be prefilled. Once the user visits the verification URL, they'll be asked to allow access to the device (this is an OIDC consent screen). Upon clicking allow they'll be redirected via the SP-initiated SAML Flow for authentication (if configured). Upon successfully completing authentication they'll be told they can close the window. 4. In the back end AWS is recording when the user code associated with the specific AWS SSO start url (and thereby account) has authenticated successfully. Meanwhile the CLI code has been polling the /token endpoint at 1 second intervals (or there abouts) with the deviceCode, UserCode, ClientId, and ClientSecret. Once AWS records the success, this endpoint will return an Access Token (OIDC compliant). Once the end user has accepted the prompt, the client applications calls [sso-oidc:CreateToken](https://docs.aws.amazon.com/singlesignon/latest/OIDCAPIReference/API_CreateToken.html) to retrieve an **AWS SSO access token**. The access token provided by AWS can be used in conjunction with many AWS endpoints including sts.assumeRole(), sso.listAccounts, and sso.listRoles. using sts.assumeRole, a CLI can request a short lived access key and secret for a provided role. With this access token, the client application can use the AWS SSO API, and in particular: * List all the AWS accounts to which the end user has access ([sso:ListAccounts](https://docs.aws.amazon.com/singlesignon/latest/PortalAPIReference/API_ListAccounts.html)) * List all the roles available to the end user in every AWS account ([sso:ListAccountRoles](https://docs.aws.amazon.com/singlesignon/latest/PortalAPIReference/API_ListAccountRoles.html)) * Assume any of these roles to retrieve temporary STS credentials ([sso:GetRoleCredentials](https://docs.aws.amazon.com/singlesignon/latest/PortalAPIReference/API_GetRoleCredentials.html)) #### Demonstration ```bash // Copy the below command to configure sso aws configure sso SSO session name (Recommended): session1 SSO start URL [None]: https://radifine.awsapps.com/start SSO region [None]: ap-south-1 SSO registration scopes [sso:account:access]: Attempting to automatically open the SSO authorization page in your default browser. If the browser does not open or you wish to use a different device to authorize this request, open the following URL: https://device.sso.ap-south-1.amazonaws.com/ Then enter the code: ABCD-DCBA ```

Once authenticated, it looks something like this: ```bash The only AWS account available to you is: 123456789012 Using the account ID 123456789012 The only role available to you is: Reader Using the role name "Reader" CLI default client Region [None]: ap-south-1 CLI default output format [None]: CLI profile name [Reader-123456789012]: radisession To use this profile, specify the profile name using --profile, as shown: aws s3 ls --profile radisession ``` # Networking and Content Delivery in AWS ## How do AWS provide Networking in its cloud infrastructure ? AWS implements Software Defined Networking (SDNs) to provide its users with various networking services. ### Key Options in AWS * Amazon VPC - Virtual Private Cloud, the foundational service of AWS. * AWS Route 53 - DNS Service of AWS, similar offerings such as given by GoDaddy, NameCheap etc. * AWS Direct Connect - Dedicated connection to AWS with existing on-premises network using a high speed (and low latency) optical fiber provided with AWS Direct Connect Router. * Elastic Load Balancing * AWS CloudFront * API Gateway # AWS VPC Virtual Private Cloud or VPC is an isolated virtual network within the AWS Infrastructure. It is similar to Internet side and Intranet side of an organization's own on-premises data center. Hence, just like any traditional network, VPC supports the following: * IPv4 and IPv6 addresses * NAT * Can be connected to new or existing VPCs * Can be connected to on-prem network with the help of VPN Also note that, VPCs can be designed to be equally scalable to accommodate more resources. Now, let's see some technicality of the VPCs. The fundamental blocks of VPCs can consist of the following: * **Subnets** of a particular CIDR block. * **Implied Router** implementing the route table. Route Tables controls the flow of traffic in, out and within a VPC. * **Internet Gateway** (IGW) providing path to and from the Internet for instances with public IP. * **NAT Gateway** (NGW) \[Public NGW for providing egress only path for instances with no public IP, to reach the internet i.e., internet devices cannot connect back to the source instance and Private NGW for communication with instances in other VPCs] * **Egress-only Internet Gateway** which supports only IPv6 internet traffic. * **Virtual Private Gateway** (VGW) for providing termination point in a VPC for a VPN Connection or AWS Direct Connect. *Note: One can also use NAT instances available as part of Community AMIs as shown below instead of NAT Gateways, simply because they are cheaper:*

## VPC Creation Process Now, let's use all the above concepts to see practical implementation: *Step 1*: Create a VPC with CIDR Block:

*Step 2*: Create Subnets from the CIDR block of the VPC:

*Step 3:* Associating the Subnet with VPC Route Table

*Step 4*: Creating Internet Gateway and making changes in route table for Internet Traffic.

Now, to put an instance such as a web server in the pub-net range, it is required to follow the steps of setting up an EC2 given [here](/aws/aws-compute-services/ec2).

## NAT Gateway in VPC Consider the scenario that there is a VPC, which has a subnet consisting of two EC2 instances with no Public IP Attached to them and the use case is to only provide outgoing traffic. The architecture might look like this:

**Well the above network architecture will fail miserably, because of following reasons:** * Internet Gateway is still required to sort of provide the path for the NAT Gateway Public IP to connect with Internet. This can be seen in solution point 2 as mentioned in AWS Docs (). Hence, the correct Network architecture can be as shown below:

*Step 1*: To construct the above diagram, one can first create two instances with no Public IPv4 addresses in private subnet:

*Step 2*: Create NAT GW in public subnet

*Step 3*: Edit the route table of a private subnet

*Step 4*: Similarly, add the route of Internet Gateway to a public subnet

{% hint style="info" %} What is **VPC Endpoints**? **These are basically endpoints that allows to connect AWS services that are completely private and not exposed to the Internet for direct communication or Access**. How VPC Endpoint can be created? {% endhint %} To connect with instances with only private IP Addresses, one requires a way to connect with them. This can be done with the help of **EC2 serial console** if the instance supports it or one can go with VPC Endpoint of type **EC2 instance connect;** the latter is shown below: ### Connecting to EC2s with Private IP

Now, let's connect with EC2 instances

*Step 5:* Post connecting to one of the instances in the private subnet, one can simply check outgoing traffic as shown below:

## VPC Peering For communication between instances in two separate VPCs, one can use **AWS VPC Peering connection.** *For the below peering connection to work, the route table of the two VPCs also needs to be edited to have a route in their respective route tables, such that when traffic to vpc-a is required to go from vpc-b, then vpc-b route table must have a route of source as ip range of vpc-a and target as peering connection.*

## Transit VPC Note that for Site (On-Prem) to VPC connection, apart from AWS Direct Connect, one can use a Transit VPC. *Transit VPC is like a common hub or gateway to AWS Cloud VPCs. Various Site to Site VPNs can terminate at Transit VPCs and this Transit VPC would have access to other VPCs, where necessary workloads are there. This effectively reduces the no. of connections as shown below:* *Before Transit VPC*:

*After Transit VPC:*

One thing that must be noted that for the transit VPC to be implemented, following are the steps: * *Create (Instantiate) a Virtual router with elastic N/W Interface & elastic IP and place it in transit VPC.* * *Create a Customer Gateway (just an object like representation of the physical or virtual router device) representing Office A Router, Office B Router and Virtual Router in Transit VPC.* * *Create a Virtual Private Gateway in VPC A, VPC B & VPC C in order to further create a Site to Site VPN between that of virtual private gateways and virtual routers.* * *Similarly, have a site to site connection between physical router of Office A & Office B with that virtual router for dynamic propagation of routes via BGP.* ## CloudHub Also note down that if two physical sites need to have a site to site VPN, then using concept of CloudHub in AWS, they can be joined. Implementation is similar to above transit VPC process, i.e. creating customer gateways object of routers of the two offices. Creating Virtual Private Gateway, which would be having VPN connection with both the customer gateways and lastly route propagation. # AWS Route 53 Simply speaking, Domain Name System is a translation of a Domain Name to IP Address. Route 53 is Amazon's DNS Service. Some of its features are as follows: * Its a global service rather than regional service. It uses edge locations. * It can be used for global resource routing i.e., it can be sued to redirect the nearest server to a user based on region or country or the server with fastest response time. * Its highly available with almost zero downtime i.e., it provides failover capability to route the user to different server if the server nearest to them is down w/o the user knowing it. Let's see how to get started with Route 53 to look into the following sub-services that are available with Route 53:

Route 53 after registering the domain can have various options as per the above features:

## Hosted Zone Let's suppose, we have registered a domain on Route 53 or any other domain registration website such as Bigrock or Godaddy or Namecheap. Now, one can purchase and register the domain at one place and can still definitely use AWS Hosted Zone to maintain the DNS records. This is done with the help of pointing the nameservers of the AWS at domain registration website, which can be done as shown below:

Now, there are two ways to create DNS record. One is one record at a time and one is zone wise i.e. by importing the zone file (*filenameofrecords.zone*). Let's look at the latter first.

To create a single record, the steps are as follows:

Now there is a concept of Alias records. These are not CNAME records but rather an AWS concept. They provide alternate Route 53 way to AWS services such as S3 or Elastic Load Balancers, CloudFront etc. One can simply set the alias record as shown below:

# Elastic Load Balancing Load Balancing is a networking concept that is used to distribute loads or in simpler terms, let's just say traffic to different servers of the same application. This is particularly useful to improve fault tolerance and performance of applications running in AWS. It integrates with EC2, EKS and Lambda easily. One point to be noted here that there are two types of scaling. * [x] One is vertical scaling, meaning to increase the compute or memory or storage capacity of the instance to meet the demand, but downside is that EC2 needs to be turned off before it can be scaled. * [x] Another type of scaling is Horizontal scaling that can leverage Elastic Load Balancers. Here instead of increasing capacity of existing EC2, another EC2 is spun up (can be done using [auto scaling groups ](/aws/aws-compute-services/ec2#auto-scaling-groups)of EC2) and then using elastic load balancing, the traffic is routed between existing and new EC2. ELB also supports Availability Zone Balancing; meaning if different servers hosted on EC2/EKS/Lamda for the same application or use case are distributed across different availability zones, then ELB can be used to serve the user with nearest one, for enhancing the user-experience. ### Working of Load Balancer: * Load balancers have a listener that receives traffic and forwards it to internal web or app server depending on the application architecture. * Usually the recipient of the load balancer traffic is configured in form of target group. * Also important to note that load balancer configuration has something called as target type, which can be either instance or IP. When instance is there, the traffic is forwarded to its Internal IP Address on primary interface. And when IP is selected, private IP has to be specified which can be associated with an interface, other than primary, considering the instances has more than one IP other interfaces. * The members of the target group are also subjected to a health check to know if traffic can be passed to it or not. This health check is usually done in form of a GET request and successful response ensures that the instance member has passed it or not. There are four types of Load Balancers:

## Application Load Balancers (ALB) * For passing traffic based on request attributes. * It uses Round Robin algorithm to select the target each time. * Supports WAF to be used in conjunction with ELB. * Please note that it terminates the client connection by responding to them post looking at HTTP headers and can use any port from 1-65535 for HTTP and HTTPS. * It supports Path and host-based routing (particularly useful for microservices based architecture)

## Network Load Balancers (NLB) * For passing traffic based on layer 4 protocol and port number. * Supports any TCP Connection. * It uses Flow Hash algorithm for even selection of all targets. * It passes the traffic as it is and does not terminate the HTTP or HTTPS client connections. Hence, the target remains the same tillthe TCP connection is timed out. * It does not support host or path based routing

## GateWay Load Balancers (GWLB) * For passing the traffic through security appliances.

## Classic Load Balancers (CLB) * For load balancing based on EC2 instances individually. * This is usually used for Network of old EC2 Classic Instances (applicable before 2014) and hence not recommended for VPC deployments

Note: *Sticky sessions make sure that when a user connection received to ELB gets forwarded to a server, all the subsequent connection remains to that server for a specified period of time to prevent bouncing of connection to other load balanced server* Note: *Idle Time outs ensures a client connection that is no longer being used, gets closed after a specified number of seconds.* ### Configuring Application Load Balancers. Consider an application with following architecture:

webs-1, webs-2 and webs-3 are three web servers in three different availability zone and are part of subnets websnet-1, websnet-2 and websnet-3 respectively which is having private IP range of 172.31.1.x/24, 172.31.2.x/24 and 172.31.3.x/24 correspondingly. To load balance, the first thing needed is a target group, which can be configured as follows:

Now, let's see how one can find the creation page for a load balancer on AWS portal:

*Note, if the load balancer would have been internal, then instead of internet facing, we would have selected internal in the scheme area as shown above.*

*Note that for HTTPS, one can set the listener to 443 and since it uses TLS, a certificate is necessary to be given to load balancer listener as shown below:*

Rest from below, common wizard options are shown:

***Note that one can create an alias A record to load balancer for routing traffic via domain name.***

Now, if one wishes to do rule based routing, one can setup the rules accordingly. Example: There is one domain configured to a load balancer. The requirement is that when the path after the domain name is different in the URL, a different target group (of maybe the same target servers but the service is listening on different port number) should be reached out, so that the resource can be loaded accordingly. To do this, one can set the rules accordingly.

Other rules can be on the basis of:

### Configuring Network Load Balancers The process is almost the same as Application Load Balancers. The only difference is that instead of HTTP/HTTPS, TCP has to be selected.

and rest of the process is exactly the same. *Please note for having HTTPS connection over Network Load Balancer, each target must have the certificate installed individually because one can't install the certificate on load balancer itself as the connection is pass through and terminates directly on the target itself.* ## Session Management Sticky sessions ensures that a particular client connection always gets forwarded to the same target that it originally gets load balanced to. To do this, the load balancer uses cookie which has a unique encrypted value that changes with each request. When different clients and load balancer are communicating with each other, load balancer identifies a particular client with the help of that changing value cookie and on basis of that, the load balancer forwards every subsequent request to a particular target of the target group. Stickiness duration can range from 1 second to 7 days. It can be configured as shown below:

Idle Timeouts maintains the duration till how long a TCP connection for a client is open as it is this TCP connection in which HTTP request and response traverses. idle simply means no data is passing between client and server and default timeout is 60 seconds. If simply due to business or non performance of the a particular target of the target group result into timeout, then client receives a gateway timeout error with 504 code.

Just like idle timeouts are for connection waiting time for clients , keep alive time is for waiting time of load balancer to receive response from the a particular target of the target group before terminating the connection.

# AWS CloudFront ## Introduction to CDN Suppose there is a new website in a local area that is gaining traction. Soon, the organization that is behind the website is experiencing a surge in their traffic from other local areas as well as other countries, but realizes that if they don't serve the content of the website with faster response time, this popularity will eventually fizz out and they won't be able to cash in at this opportunity. Why this is important? Because the websites that loads faster will provide a better user experience as no user in today's time wants to wait longer than expected to consume the content served on their website. So, what's the solution to the above problem? Add more servers that hosts web front end content (Static Content such as HTML, CSS and Dynamic Content such as Javascript). Yeah that solves this problem on a small area wise division where those servers would be put. But what if website is receiving traffic from a brand new area. The organization just can't keep adding servers as this would be costly as well as non scalable. Here enters the service to rescue such situation and that solution is none other than CDN. Yes, Content Delivery Network can be used to provide a scalable solution to resolve the serving of web content faster by having a bunch of servers that are geographically distributed across different locations. This will server the content from the nearest CDN node server to the user. One can consider the CDN server as remote cache of the website. They are basically reducing the distance between the origin of server hosting the content and the user of the website, automatically reducing the load times. Other than the faster load times for better user experience, using caching and optimizations, CDNs are able to save bandwidth, increase content availability and redundancy to hardware failures of web servers by providing alternate servers and may also be able to protect against DDOS attacks and provide fresh TLS/SSL certificates to host. They can also compress files and minification. Apart from that, CDNs use load balancing, failover and anycast routing to make sure that even if other CDN servers or even if the data center goes down, the experience would be seamless and problems would be unknown to the end user and website is always up. CDNs can also provide or enable WAFs to protect the websites against vulnerabilities. Hence CDNs are important for any individual or organization with an internet property. CloudFront is AWS's answer to the service of CDN. ## Tech behind CloudFront As explained in AWS Global infrastructure page, there are many edge locations and regional edge locations that allow caching of the original website. These actually help because the regional edge may cache your website first and then other edge locations can take a copy from regional edge caches instead of taking it from the source directly. This is known as tiered caching. CloudFront uses optimizations such as TCP fast open (TFO - alternate to continue TCP connection w/o 3 way handshake for repeated connections), and keep alive connections have definitely improved performance. One Important thing which defines CDN is Caching Behavior, which are set of rules bound to an origin that define how the CDN handles and process incoming requests. This behavior is controlled by multiple configuration settings such as * Path Patterns (it is like caching will depend on matching pattern rule example: /images/\*.jpg -> /images/\* -> \*.gif, so a /image/sample.gif will match with 2nd pattern first so that caching behavior will be used to cache the object from the origin server), * Multiple Origins (Website components or the entire website can be kept at multiple origins which may depend on the region, i.e, content may change depending on country like in Netflix), * Query Strings (example: search parameter in Get URL of the website can get cached to serve the content faster to newer audience if there is a surge in a certain search query after the first person searches where the content is directly served from origin server and then it gets cached for subsequent searches), * Securing Objects through HTTPS (depending on whether http is hit or https, different cache can be served) * TTL (Different TTLs for different cached elements) The above configuration are complimentary to other elements such as origin behavior, cache duration, cookie or query string forwarding, request headers, compression, encryption etc. CloudFront's cache retention is made to keep objects longer in cache and to minimize cache churn using techniques like tiered caching (explained earlier) and de-duplication (Instead of caching based on URL for every new request which is different from previous URL hits, it is always better to divide the entire content from origin server into different small elements, also known as chunks; these chunks of elements is kept with unique reference such that these references ultimately add up to serve any content. This ultimately helps to remove redundant chunks and only if new content is asked, then only new element chunk is made). ## CloudFront Features * CloudFront also supports Pay-as-you-go model. Hence one can use CloudFront when let's just say they have a anticipated demand like during some sale to increase app availability. * CloudFront Data transfer is free , if the origin server is an AWS endpoint like S3 or EC2 etc as long as the traffic is within AWS. * CloudFront also supports multiple origins that will enable redundancy. * CloudFront also supports micro services etc to support modern day application tier. * CloudFront also supports Access Control to kind of provide geographical restrictions to your website. * CloudFront supports WebSocket, as well as the HTTP protocol. * CloudFront integrates easily with AWS Shield and WAF to protect against a large variety of attacks including DDOS * CloudFront integrates easily with AWS Certificate Manager to automatically handle renewal of TLS certificates and provide features such as OCSP stapling (OCSP is a real-time check of the status of a certificate validity, that browser normally performs when a user visits on https website, by checking with the CA that issued certificate has not been revoked and OCSP stapling is a process to keep a digitally-signed and time-stamped version of the OCSP response directly on the webserver for faster TLS /SSL handshakes), session tickets (Session Tickets help speed up the time spent restarting or resuming an SSL session by encrypting SSL session information and storing it in a *ticket* that the client can use to resume a secure connection instead of repeating the SSL handshake process), perfect forward secrecy (Forward secrecy achieved by generating new session keys for each message ensuring that past communications cannot be decrypted even if the secret long-term key is compromised.), and field level encryption (Field-level encryption configurations helps protect specific data that end users inserts in `POST` requests. CloudFront encrypts the data at the edge location, using a public key that is provided to it, before forwarding the request to the origin. The specific application component that has the appropriate private key can decrypt the data, but the data remains protected as it’s passed through other parts of the system). * One can restrict access to content through a number of capabilities. example, with signed URLs and signed cookies, one can support token authentication to restrict access to only authenticated viewers. * Through Geo-restriction, one can prevent users in specific geographic locations from accessing certain content. With OAI, or origin access identity, one can restrict access to an S3 bucket to only be accessible from CloudFront. * In CloudFront, Lamda\@Edge can help to serve requests faster than previously by using Edge Locations. * When certain portion of website (such as css) is updated, CloudFront supports invalidation to ultimately re-cache the new file. If one wants to update files frequently, it is recommended to primarily use file versioning for faster processing. At max, CloudFront supports 3000 invalidations (it can be any combination - 3000 requests for 1 file each, one request for 3000 files or 30 requests for 100 files each). First 1000 invalidations in a month are free, rest are chargeable. These 1000 invalidation is overall limit. If an organization has more than 1 distribution, so 1000 free will be applicable as changes over sum of both and not individually. * CloudFront also supports Error Handling for better user experience Few important pointers before moving ahead: The flow of request is is as follows:

{% hint style="info" %} When the response is served from Cache, it is called as Cache hit and when CDN has to go to origin request to fetch response, then it is called as cache miss. {% endhint %} ## Setting up AWS CloudFront > Prerequisite : An application hosted on EC2 or Elastic Beanstalk with or w/o Application Load Balancer is required. There are mainly 3 config parts/areas, which are Origin Settings, Cache Behavior and distribution settings. * Let's begin with the first one: Origin Settings:

* Cache Behavior Settings

> For static content, managed cache policy of "Caching Enabled" can be easily used

> Please refer [this](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/field-level-encryption.html#field-level-encryption-setting-up-step4) for more details about field level encryption configuration * Other Distribution Settings

### Additional Configuration Options in CloudFront

{% hint style="info" %} The above snapshot makes one thing very clear that for a single distribution, there can be multiple behaviors based on different URLs or headers or cookies etc. Hence, it can be said that **CloudFront Distribution is a collection of an origin and all associated caching rules while handling its traffic.** {% endhint %} ### Error Handling

### Invalidations

{% hint style="info" %} Invalidations as shown above is basically a way developer or AWS Admin telling CloudFront that its needs to update its cache at the edge location by taking the content from Origin. It does it when the viewer requests the updated file and due to this invalidation configuration, CloudFront considers it as cache miss and gets the file from origin. {% endhint %} ### Geo Restrictions

### WAF

### Origin Access Two ways of Origin Access: Origin Access Control and Origin Access Identity. OAC uses signed URL's/Cookies to restrict access to certain users. OAI uses canonical user of CloudFront to have permission to read a bucket. OAC is more stringent than OAI. * OAC requires use of Amazon library during development process of application code present in the origin server.

{% hint style="info" %} Create 2048 bit new priv key pair online using: {% endhint %}

Save the above and users will be served with signed URLs of the format:

* OAI can be best explained with the following diagram:

Select appropriate settings and necessary details and output can be obtained. ### Field Level Encryption The use case is explained earlier, i.e. protection of post body parameters. The way to set this is as follows:

Post that it is possible to set Field Level Distribution in Behaviour page ### Lambda\@Edge Lambda\@Edge allows devs to deploy Node or Python code at the same edge locations where the content is cached. It's a fully managed solution, so there's no servers to manage, which helps on security, compliance, and also your time and resources. Points to Remember: * Only Node and Python are supported languages for running on Lambda\@Edge. One can't just deploy Lambda\@Edge by itself. It has to be associated with a CloudFront distribution in order to deploy to the edge locations. * Use Case of Lambda: CloudFront can customize the content delivery perhaps in such a manner that one can send search engine crawlers to a static version of the content while delivering real users to the dynamic version. Another use case is to do content security by adding or altering headers. * Working of Lamda\@Edge:

# Amazon API Gateway ### Features API GW can be said to be a bridge between different AWS services or between Users of a Application and their corresponding backend. It can act as the "front door" for applications to access data, business logic, or functionality from the backend services.

Source: https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html

**Amazon API Gateway is a fully managed service to create, publish, maintain, monitor, and secure APIs at any scale with the help of API Keys and its tracking via usage plan**. Note that usage plan can be used to put limitations/restrictions on a particular API with the help of Quotas or throttling etc. API Gateway supports RESTful APIs, WebSocket APIs and even HTTP APIs They support containerized and serverless workloads, as well as web applications. One such use case is its integration with Lambda:

Source: https://dev.to/aws-builders/two-ways-to-directly-integrate-aws-lambda-function-with-amazon-api-gateway-3can

API Gateway supports thousands of concurrent API calls, including traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. The url is of the format: https\://<**random-api-key/id**>.execute-api.<**region**>-amazonaws.com/<**stage-such-as-dev**>/<**resource-name-such-as-default**> Note that Stage can also have stage variables, which can be used to store configuration without modifying the code (example, URI for uat stage and prod stage can be different and this is stored in stage variables, so dynamically the value can be taken at runtime and used further by the API) An HTTP or REST API GW can have versions with each version having a method (GET/POST...), Authorization and API Key

API Gateway can also be private and **can be only accessed via VPC endpoints**.

Source: https://javapatel.wordpress.com/wp-content/uploads/2023/10/rest-private-api-coverimage.png?w=816

### Authentication and Authorization * API GW also supports Standard AWS IAM roles and policies, Endpoint policies for interface VPC endpoints * Just like S3, it can also have Resource Policy, which can be used to configure for IAM users, Source IP Address ranges or CIDR blocks and even specified VPCs or VPC endpoints (any AWS account). {% hint style="info" %} Please note that it best practice to specify VPC ID for private API GW in condition. {% endhint %} ```json // Generated by ChatGPT { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "execute-api:Invoke", "Resource": "arn:aws:execute-api:us-east-1:123456789012:abcd1234ef/*", "Condition": { "IpAddress": { "aws:SourceIp": "203.0.113.0/24" } } }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::098765432109:role/MyRole" }, "Action": "execute-api:Invoke", "Resource": "arn:aws:execute-api:us-east-1:123456789012:abcd1234ef/*" } ] } ``` The authorization workflow with IAM and resource policy can be highlighted as shown below:

Source: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-authorization-flow.html#apigateway-resource-policies-iam-policies-interaction

* It also supports Amazon Cognito User Pools * One different authorization it supports is Lamda Authorizers ### Lambda Authorizers It is a lambda function to control the access to a APIs behind a API Gateway. It can be used to build custom authentication or authorization apart from the usual IAM based or API key based. It uses custom authorization scheme: * Based on bearer token authentication strategy * Uses request parameters to determine the caller's identity There are two types: * TOKEN authorizer (token-based) * REQUEST authorizer (request parameter based) Based on authorizer values, access can be granted or denied as shown below:

Source: https://docs.aws.amazon.com/images/apigateway/latest/developerguide/images/custom-auth-workflow.png

Note that policy is usually cached for 30 seconds ### Steps to setup an API Gateway

> Head back to Lambda to complete setting up trigger and then use API Gateway URL

# AWS Storage Services It can be divided into three main categories * File Storage (EBS, EFS, FSx) * Object Storage (S3) * Databases (RDS, DynamoDB, Elasticache, Redshift, DocumentDB etc) For large-scale data transfer, AWS Snowball and AWS Snowmobile can be used. * AWS Snowball - It is a physical device delivered by AWS that can be connected to a network to physically migrate upto petabytes of data into it, which is then carried by AWS to a AWS data center and then the data is uploaded to S3 * AWS Snowmobile - It is a ruggedized shipping container (on a truck) that supports exabyte scale transfer where AWS sets up a connection to customer's existing network. AWS then loads the data to snowmobile and then the container is transferred to an AWS data center to upload it to S3.

Source: https://datavalley-ai-prod.s3.amazonaws.com/files/20230320125139/image-141.png

# Amazon EBS Basic Description about the same can be found from [EC2](/aws/aws-compute-services/ec2) page. One can create Volumes as shown below:

EBS Snapshots can be taken for periodic backup and DR purposes. Hence, to take any snapshot particularly related to EBS, its necessary to either stop the instance (for root volumes) or unmount the instance (for extra drives) to prevent any further write operation, post which the snapshot can be taken. The EBS Snapshots can then later be made available to other new volumes, which can then be attached to another EC2 instances all together.

# Amazon EFS It is a fully managed NFS file system for linux workloads It supports data across multiple AZs and provides two different storage classes i.e., standard and infrequent access Unlike EBS, it can be attached to multiple instances even though they are in different availability zones and even with on-prem network

Source: https://miro.medium.com/v2/resize:fit:720/format:webp/1*Gig18TiVgWdxXeedc778UA.png

# Amazon FSx Just like EFS is for Linux, FSx is a fully native Windows file system It supports the following: * Windows NTFS (note, it is a file system from MSFT for Windows, just like linux has file hierarchy system - FHS, where everything is a file) * SMB Support * MSFT AD Integration * SSD for low latency # S3 ## Features It is one of the core or foundational storage services from Amazon that lets AWS account users store files as objects in buckets where Buckets are storage containers. It stores data across multiple availability zone and enable URL Access for the bucket as well as objects inside it It offers the ability to configure rules for data lifecycle such as expiration time kinda like of OneDrive links that can expire It can also be used to host static content, so a website can entirely be hosted inside s3. Note that the nomenclature for such is of the format of https\://\.s3-website.\.amazonaws.com/\.\ Cross Region Replication of buckets and objects inside the bucket is available for backup and availability across multiple regions, but that requires versioning of source and destination buckets to be enabled as well as an IAM role might to perform the replication between the two buckets is required. {% hint style="info" %} Note even though the bucket can be present in an availability zone, its name has to be globally unique {% endhint %} {% hint style="info" %} Object locking in S3 is supported where the data cannot be deleted, rewritten, or tampered with. It follows WORM philosophy of Write Once and Read Many, particularly useful to establish the inherent trust in data by an unchangeable permanent copy of data for auditing or record-keeping to meet compliance or regulatory requirements. They are a special choice to prove integrity of data when an organization is hit by ransomware attacks. The objects cannot be tampered, either for a set period (retention) or indefinitely until the lock is removed. {% endhint %} ## Storage Classes S3 provides the following storage classes for different use cases: * Non-Archival Storage Classes * [x] S3 Standard: Default storage class, used for frequently accessed data * [x] S3 Intelligent-Tiering: Applied on a object and moves those objects to the correct storage class based on usage i.e., moves objects in s3 buckets automatically based on whether it is frequent access or infrequent access. If an object is not accessed for 30 days, IA will move the object into Infrequent access tier, If not accessed for 90 days, Glacier Archive and if not accessed for 180 days, it would be moved to Glacier Deep Archive. * [x] S3 Standard-Infrequent Access: For infrequently accessed data with standard resilience i.e., it combines the lower cost of Infrequent access along with resiliency of multiple Availability Zones * [x] S3 One Zone-Infrequent Access: Same as above just with one availability Zone * Archival Storage Classes * [x] S3 Glacier Instant Retrieval: Retrieval time is higher i.e, in milliseconds. Doesn't require frequent access, but when access is required, it is required instantly * [x] S3 Glacier Flexible Retrieval: Flexible in such a time that it can have configurable retrieval times from minutes to hours. It is recommended for backup and Disaster Recovery * [x] S3 Glacier Deep Archive: Deep archive that has retrieval time within 12 hours. It is recommended to hold some data for many years maybe for compliance requirement but cost is more important than retrieval times ### S3 Lifecycle Policies The objects in a bucket can be moved to another storage classes based on the criteria set by the AWS account user. The criteria can be time-based. It also enables to delete objects based on age. The policies can also factor in versions of a specific object in the bucket, so the policy can set to delete a object that is not current version after a certain period of time ### S3 Transfer Acceleration It is a feature that can be enabled per bucket that allows for faster upload of data using the AWS Edge locations as a part of AWS CloudFront {% hint style="info" %} Note that S3 supports bucket as well as objects level logging to know about the details of when they were accessed, who accessed them etc using AWS CloudTrail {% endhint %} ## ACL in S3 Note that Access Control in S3 can be applied through various ways., which can be: IAM Policies -> S3 bucket and IAM User has to be in the same aws account Bucket Policies -> Can be used to provide cross account access Access Control List -> Can be set on bucket and objects in bucket * READ * When applied on bucket, READ ACL allows user to list all objects in the bucket * When applied on Object, READ ACL allows user to retrieve the content of objects and object meta-data * READ\_ACP * When applied, READ\_ACP allows user to read the Access Control Policy * WRITE * When applied, WRITE ACL allows user to write an object (Bucket must have write permission) * WRITE\_ACP * When applied, WRITE\_ACP ACL allows user user to write ACL of object and bucket. * FULL\_CONTROL * It allows all the above permissions ACL is deployed using Grantee as explained below:

ACL Example: ```json // Here Type needs to have ID, Group needs to have URI { "Grants": [ { "Grantee": { "Type": "CanonicalUser", "ID": "AWS_ACCOUNT_ID" }, "Permission": "FULL_CONTROL" }, { "Grantee": { "Type": "Group", "URI": "http://acs.amazonaws.com/groups/global/AllUsers" }, "Permission": "READ" }, { "Grantee": { "Type": "Group", "URI": "http://acs.amazonaws.com/groups/global/AuthenticatedUsers" }, "Permission": "WRITE" }, { "Grantee": { "Type": "AmazonCustomerByEmail", "EmailAddress": "exampleuser@example.com" }, "Permission": "READ" } ], "Owner": { "ID": "AWS_ACCOUNT_ID" } } ``` > Kindly read through the [AWS KMS service](/aws/other-aws-services/aws-key-management-services-kms) before reading down. ## Steps for creating S3 and exercising access control using bucket policy:

#### Resource Based policy for S3 Buckets

{% hint style="info" %} Note that Each IAM entity (user, group, or role) has a defined aws:userid variable. This variable is required for use within the bucket policy to specify the role or user as an exception in a conditional element. An assumed-role’s aws:userId value is defined as UNIQUE-ROLE-ID:ROLE-SESSION-NAME (for example, AROAEXAMPLEID:userdefinedsessionname) {% endhint %}

# AWS Databases Services Amazon offers many DB Services. Some of the common and noteworthy ones are as follows: * Amazon RDS (PaaS) - Example: Amazon Aurora Platform * Amazon DynamoDB (SaaS - DB as a service) * Amazon Redshift (SaaS - DB as a service) * Amazon Elasticache (SaaS - DB as a service) * AWS Database Migration Service (DMS) - It is possible to move existing on-prem databases into AWS Cloud's RDS. This supports one time and continual migration of data. The pay for this migration is not much as it is only for the compute leveraged in migration process. Additional Database Services can be as follows: * **Document DB**: It is MongoDB compatible database, which is ideal to experiment in AWS for testing with MongoDB workloads. It is also a PaaS offering similar to RDS. * **Neptune**: It is managed graph database service, which is ideal for mapping relationships between data points. Very useful for social media companies * **MemoryDB:** It is redis compatible service ideal for a fast in-memory primary database * **Timestream:** It is serverless time series database service ideal for storing and retrieving time series data points. Very useful for IoT sensors # Amazon RDS ### Features It is a relational database that supports MySQL, PostgreSQL, MariaDB, Oracle Database, SQL Server as well as Amazon Aurora Its a fully managed service offering from Amazon that falls under PaaS category, so it handles provisioning, patching, backup and recovery. It provides both general purpose SSD and provisioned IOPS SSD drive. Because it is a PaaS offering, it is basically a DB application/service on an EC2 instance in a particular VPC. **Deployment:** The deployment is supported in multiple availability zones. Users can easily launch a relational database instance in RDS. AWS manages the provisioning of hardware, database setup, patching, and backups, reducing administrative overhead. Amazon's own RDS: **Amazon Aurora** is a MySQL and PostgreSQL compatible relational database built for the cloud that combines performance of traditional enterprise databases with simplicity and cost-effectiveness of open source databases. **Database Engines:** RDS supports multiple database engines, allowing users to choose the one that best suits their application requirements. **Scaling:** RDS provides options for vertical and horizontal scaling to accommodate growing workloads. Users can resize instances or create read replicas for improved performance and availability. **Automated Backups:** RDS automated database backups, making it simple to recover data in case of failures or user errors. Users can also set up automated snapshots for data retention. **Security:** RDS offers robust security features, including network isolation, encryption at rest and in transit, and integration with AWS Identity and Access Management (IAM) for fine-grained access control. **Monitoring and Metrics:** AWS CloudWatch integration allows users to monitor database performance and set up alarms for proactive issue resolution. **High Availability:** RDS supports Multi-AZ deployments for failover and read replicas for improved availability and fault tolerance. AWS RDS simplifies database management, ensuring that organisations can focus on their applications while AWS takes care of the database infrastructure, backups, and maintenance. It’s a valuable service for those looking to run relational databases in the cloud with ease and reliability.

Source: https://alexanderhose.com/content/images/size/w1000/2023/01/aws-rds-architecture-diagram.png

### Steps for Implementation Navigate to AWS RDS

{% hint style="info" %} Refer either of the below links for setting up IAM authentication to RDS over SSM: \ \ OR\ {% endhint %}

# Amazon DymanoDB ### Features In last page, Relational DBs were discussed. The service offering from Amazon for Non-Relational DBs which supports NoSQL for querying part, is Amazon DynamoDB. Unlike Amazon RDS, which is a PaaS, since this is a SaaS offering, not only, it is fully managed, it is also not required to manage the database layer, so need to think of schema It provides both key-value and document database. It provides consistent, single-digit millisecond response times at any scale. It supports ACID (Atomicity, Consistency, Isolation and Durability) transaction It supports Identity and Access controls on Table It supports automated scaling without excessive maintenance and has extremely low latency It offers in-memory cache with DynamoDB Accelerator (DAX) It supports server-less applications and it autoscales to support growing demand ### Steps for creating a DynamoDB Table When creating a table, not that every table is associated with a partition key

Note, DynamoDB supports PartiQL which is used to query NoSQL DB like DynamoDB using SQL queries

# Amazon Elasticache ### Features It is the fully managed in-memory data store offering from AWS. It is supported by by both in memory engines - Memcached and Redis It provides low latency in response times It enables scaling and replicas to meet sudden surge in demand from the application It can be used for database later caching i.e., between the application and actual database, Elasticache can act as buffer to hold the data When a web application is doing session storage, Elasticache with Redis engine can also used to get quicker responses on session data ## # Amazon RedShift ## Features It is a scalable data warehouse service from AWS that can be used for user analytics It supports warehousing upto petabytes of data It uses high performance disks and columnar storage It has in built feature to fully encrypt contents and also provides isolation within a VPC It has something Redshift Spectrum to support querying exabytes of data in s3 # Amazon DocumentDB It is a NoSQL Database # AWS Compute Services ## What is AWS Compute? A service that enables to leverage virtual machines on cloud for workloads. This could be for hosting website, running a database, or calculating statistics from a data set. ### Key Compute Options in AWS * Amazon EC2 - VMs on AWS - **IaaS (Infrastructure as a Service)** * AWS Elastic Beanstalk - For scaling WebApps and services - **PaaS (Platform as a Service)** * AWS Lambda - Server-less Architecture * AWS Container Services such as AWS AppRunner, AWS EKS and AWS ECS # EC2 ## Features It is **E**lastic **C**ompute **C**loud. But What is Elasticity? - *It is the ability for the infrastructure supporting a use case to grow and contract based on how much it is used at a point in time*. The name has elastic simply because of ability to resize capacities of vCPUs or RAM or SSD. The use cases are limitless. * One can launch EC2 on basis of the factors such as Instance Types \[Predefined bundle of vCPU (processor), RAM (memory), SSD (storage) Network Bandwidth etc with varying options of General Purpose OR Compute or Memory or Storage optimized etc.), AMI, Purchase Options and Root Device type. * What is AMI? - Its Amazon Machine Image. Its kind of a like a template of a OS with different flavors of add-ons. AMI when instantiated, usually requires of Root Volume Template, Launch Permissions, Block Device Mapping etc. * [x] Root Volumes are usually like hard disk that hold the OS. There are two types of storage (Root Volumes) that can be attached to EC2: 1. The first one is Instance Store (Ephemeral Storage) - local storage on the bare metal server where instance is stored and it is like RAM (shut down and data gone) & it is physically connected to host on which VM is running, mostly used for temporary computations, storing caches, storing non-critical logs etc 2. The other is [Elastic Block Storage](#user-content-fn-1)[^1] (Persistent Storage), **where Hard disk is separately connected from the host (Network attached storage)** on which VM is running but it will be persistent. Note that EBS can be attached to only instance at a time and it supports Amazon EBS encryption for meeting regulatory requirements. It supports petabytes of data and supports multiple volume types based on need such as General purpose SSD (general workloads), provisioned IOPS SSD (High performance low latency), throughput optimized HDD (designed for frequently accessed data) , cold HDD (less frequent workloads). It supports data across multiple Availability Zones * [x] Launch Permissions decides which AWS Accounts can launch instances of AMI. * [x] Block Device Mapping helps in attaching another storage to EC2 * One Can Create Custom AMI images or use community AMIs or commercial AMIs (all available from AWS Marketplace) for providing OS to the EC2 * **In AWS EC2, there are different instance types, such as t3.medium or m5.large (t starting or m starting is general compute - the difference is that t is burstable and m is non burstable, i.e., for predictive workload, m type can be cheaper as the flexibility of t type may cost more), c5d.24xlarge (c starting is compute optimized), p3.16xlarge (p starting is accelerated compute optimized), i3.16xlarge (i starting is storage optimized), r4.large (r starting is memory optimized)** * There are different types of Instances - On-Demand, Reserved, Spot, Savings Plan, Spot and Dedicated. 1. On-Demand as name suggests can be spun up and closed at any time. There are no contracts and mostly used for experimental POC purpose. 2. Reserved - Cheaper when compared to On-Demand Instances, because of commitment to use a Virtual Machine over a defined period (1 year or 3 year). There are multiple offering class, as mentioned below Note that the pricing for the above are based on the following configuration options: * Platform - OS such as Linux or Windows * Tenancy - Underlying hardware can be dedicated to AWS Account or Default (Shared among multiple instances, even though they cannot communicate with each other) * Offering Class - * [x] Standard (for steady workloads, but lock-in so, can't change), * [x] Convertible (some attributes can change, but good for steady workloads), * [x] Scheduled (change possible as per time like weekend, you can increase); * Instance Type - These are as defined above in bold. In Convertible, instance types may be changeable. * Term - It can be 1 year or 3 year * Payment Option - Payment terms can be All Upfront, Partial Upfront or No Upfront * Availability Zone - If this is selected during configuration of RI, it can be termed as Reserved RI, if all availability zones within an entire region is selected for discounts to be applied otherwise discount can be applied in a single availability zone, to reserved capacity. *Note that if suppose an AWS user is signing up for a Reserved Instance for a 3 year plan and then the user does not require that Reserved Instance, let's say after 2 year, then the user can have the option to sell Reserved Instances on Amazon Reserved Instance Marketplace, where there are also opportunities to buy instances for terms shorter than one year.* 3. Savings Plan is rather commitment to pay per hour that can be utilized for any of the compute services that matches the criteria given at the time of purchasing the plan, rather than reserved instances, where you pay for commitment to use a instance for a standard 1 or 3 year plan. 4. Spot is like of stock market. If the price offered by you to AWS is higher than the spot price, your instance is up. If the spot price goes higher than what you were paying per hour, your instance is terminated. The price difference can be verified from the AWS pricing calculator:

These are meant for those applications which can be easily operated by starting and stopping them i.e. something which don't have a real time usage and are used to carry out independent work. 5. Dedicated host is like full physical server for the customer/user in AWS data center. This is the most expensive option. Note the difference between dedicated instance and dedicated host may sound a bit complex, but is not. Dedicated instances means the various instances created by a single customer (AWS User Account) onto the same underlying hardware; that hardware can be the same or different depending upon how many dedicated hardware we have opted for in AWS. Dedicated Host, on the other hand ensures that hardware will always be the same for a particular instances that the user will launch. This is usually helpful for running softwares which binds with hardware device id. ## Setting up an EC2 instance * To Launch an instance on-demand with default settings, one can follow the steps as shown below:

Now, there are three ways to go ahead (One is Auto Assign Public IP, One is Assign Elastic IP and last is not to assign an public IPv4). * [x] For letting AWS assign and attach an interface with changing IP addresses during server restart (stop instance and then again start instances)

* [x] Another way is to create N/W interface and assign elastic IP (which won't change even after when the server restarts, post stopping am instance as shown below:

* [x] No Public IP to be assigned. For providing Internet, one can use NAT Gateway

From below, steps to further configure an AWS instance merges from the above three NIC variants shown above

**Here, IAM Instance Profile can be used to provide role based access to EC2 instance**. **The EC2 instance profile assumes an role that has access policy to S3 buckets. So, by the virtue of presence of applications on the EC2 instances, they are able to generate temporary credentials to access S3. Here, the EC2 has that trust relationship in form of service role, this means that if tomorrow instead of XYZ application, ABC application gets deployed on that EC2 instance, they will still be able to access the S3 bucket as they are associated with the EC2 rather than long-term credentials which can be hardcoded in the application. This also means, if more than one application is hosted on EC2, every app would be able to access the s3 bucket.** **Basically, the applications access the s3 bucket using IMDS (Instance Metadata Service) endpoint on a link-local address of 169.254.169.254 to get short term credentials at runtime and access s3 using that.** {% hint style="info" %} The IMDS service can be used to fetch the following metadata. It is the way by which AWS makes configuration data available to EC2 * Public IP Address * UserData * Security Groups * Instance details * IAM Access Credentials for the EC2 instance The url is **** AWS IMDSv1 (Instance Metadata Service Version 1) and IMDSv2 (Instance Metadata Service Version 2) are services provided by Amazon Web Services (AWS) that allow EC2 instances to retrieve metadata about themselves and interact with the instance’s local configuration. While both services serve similar purposes, they have significant differences in terms of security and functionality. **IMDSv1** was the original metadata service provided by AWS. It allowed EC2 instances to access metadata by making HTTP requests to a predefined local IP address (). While it was useful for retrieving instance information and user data, it had some security concerns, as it didn’t offer strong protection against potential attacks. **IMDSv2**, introduced to address these security concerns, enhances the metadata service significantly. IMDSv2 provides a more secure and robust way to access instance metadata. It uses a session-based approach, requiring an instance to authenticate and use a short-lived token to access metadata. This mitigates risks related to potential exploitation of the metadata service by unauthorized entities, making it more resilient against various attacks. **For the instance’s hostname**: curl **For the instance’s AMI ID**: curl **For the instance type**: curl **For getting the public IPv4 address assigned to the instance**: curl **For getting security groups associated with the instance**: curl **For getting the EC2 Instance Profile credentials of the IAM role (if a role is attached to the instance)**: curl {% endhint %}

*Note:* *a) Placement Group allows to specify how instances are placed on underlying hardware. There are three types, namely cluster, partition and spread.* 1. *Cluster helps to keep the two instances very near to each other within an availability zone with having a far better instance to instance communication for low latency and high throughput.* 2. *Spread offers hardware isolation, i.e., each instance in run on separate hardware.* 3. *Partition is a hybrid of above two, i.e., suppose there are 8 instances then there would be group of 4-4 instances within a partition. So, instance to instance communication within the same partition is very fast; at the same time, they are isolated with the other set of partition on hardware level. These are also called as nodes* *b) The On Demand Capacity Reservation is different from Reserved instances. This was introduced during Covid crisis. Because of huge shift to cloud, many new users were launching EC2 instances, and old users were getting error of type "No capacity on host to run the instances". Hence, AWS came up with the idea to pre-book a slot for on-demand instances at no extra cost, but providing users a way to get guaranteed capacity to run shorter term workloads which can be turned on and off at any given time.*

*Note: Whenever selecting AMI (Amazon Machine Image) from Community or AWS Marketplace, one can find that Amazon supports two types of images. One is HVM (Hardware Virtual Machine) and other is PV (ParaVirtual). To learn the difference, kindly check out the below data sourced from the AWS Docs:*

## Reserved Instances Let's now see about reserved instances launch:

* Post this, launch the instance of the exact same configuration just like an on demand instance and it would eventually be shown into the reserved instances section, solidifying that reserved instance discounts are being applied on the turned on instance ## Spot Instances

## Dedicated Hosts

## AMI *Practical usage for AMI: Suppose a user created an EC2 instance, configured everything that is required in the system such as settings and softwares. Now, the user wishes to create copies of this EC2 instance with the previous OS and instance settings as golden copy, then it can be done by creating image or snapshot of the machine, which can be replicated with new instances.*

Rest of the steps to launch the instance is similar Note that an EC2 can have different stages such as AMI -> Pending -> Running -> Stopping -> Stopped -> Shutting Down -> Terminated. When an instance is in Hibernation mode (Stopped Stage), the contents of its RAM are quickly flushed into the EBS, so that next time the instance is made up, the RAM can be repopulated with the same data, making the transition feel seamless. ## How to move EC2 instances from one subnet or Availability Zone to other? Steps are as follows: Step 1: Stop the instance. Step 2: Note down the Security Groups attached to the instance Step 3: If an elastic IP is attached to it, make note of it as well as the network interface. Step 4: Create an AMI of the same as shown above. Step 5: Terminate the instance. This will detach the the network interface from the instance. Step 6: Under the network interface, disassociate address and select the elastic IP.![](https://3681896347-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjfQTFfcSjS8MYnjfKw2c%2Fuploads%2FfWFEyohYu1x5QyFm7K2o%2Fimage.png?alt=media\&token=7be868d0-9bcf-4c1e-8997-c6d10b578f77)![](https://3681896347-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjfQTFfcSjS8MYnjfKw2c%2Fuploads%2FOaCAb3TZDhcQ0jvA98vf%2Fimage.png?alt=media\&token=d2a0eece-6781-4a5e-b9cf-adc1afef677b) Step 7: Create a new network interface **in the required subnet or availability zone with new private IPv4 address as per the subnetwork** and **associate the elastic IP with this interface.** Step 8: Create an instance from that AMI. While doing that, in the wizard, attach the new network interface which has association of previous elastic IP and relevant security group.

## Auto Scaling Groups Suppose there is a video streaming application with fluctuating workloads, data gathered from Cloud Watch and is as shown below:

This typical week shows demand of the application by its consumers with release of final trailer on Thursday and movie on Friday and general viewing habits peaking in the Weekend. If the video streaming platform wants to optimize costs of the application, it needs to scale the application underlying EC2 instances with demand rather than keeping the high cost instances up all the time. This can be done with the help of Auto Scaling Groups. Auto Scaling allows to add or remove EC2 instances on demand based on conditions specified to maintain performance and optimize costs. This results in automated scalability. There are two types of scalability: * Scaling Out: Increase in EC2 instances by launching them. * Scaling In: Decrease in EC2 instances by terminating them. The steps to achieve Auto Scaling happens in three stages: 1. Auto Scaling Launch Configuration 2. Auto Scaling Group 3. Auto Scaling Policy *Note that the best way of Auto Scaling is Dynamic one. Let us take an example: There are two instances, supported by* [*Elastic Load Balancing*](/aws/networking-and-content-delivery-in-aws/elastic-load-balancing)*. As ELB sees a surge in demand in number of requests hitting it, a CloudWatch alarm is triggered. This results in auto scaling event to be executed, which can be to scale out to meet the rising demand.* Note that it also supports MAC OS instances [^1]: [https://notes.radifine.com/aws/](https://notes.radifine.com/aws/aws-storage-services/aws-ebs) # Elastic BeanStalk WS Elastic Beanstalk, automates the process of deploying and scaling compute workloads on EC2. The main difference is instead of AWS user to deal with those servers directly, AWS will manage all that for the user. It allows developers to concentrate on dev part and helps them with the deployment and administration part. Technically, everything is still running all of the compute on EC2, but the process of managing servers, handling things such as provisioning, load balancing, scaling, and monitoring are all handled automatically through the work of Elastic Beanstalk service. *This is particularly helpful for developers because the services helps in administration of the server on which the app is going to be hosted.* Hence, as normal thumb rule goes, with ease, comes limitation and restriction. This is because unlike EC2, where user is free to do whatever he/she wishes to, here, in Elastic Beanstalk shows limited compatibility with technologies such as Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker. The following are the steps to deploy a sample application on AWS Beanstalk, to understand the overall **PaaS** concept better:

# AWS Lambda ### Features Lambda is the so called as "server-less architecture" of AWS. It is different from the previous two as it is event driven; so the code uploaded in Lambda, is not given an execution environment to run code, until the defined event is triggered, and in this process, the user does not get to either provision or manage servers, and hence user only pay for the time used to consume compute resources. Due to this, there is a limitation, that whatever code is uploaded in Lambda, should complete execution within 5 minutes Here, lot of application or back‑end service is supported and as there is no scope for administration, it integrates with many AWS services. One of the most important features is that it is highly scalable. Pricing changes with memory. An AWS user can consume memory from 128 to 3008 MB allocated for functions that are running on AWS Lambda.

Source: https://vticloud.io/wp-content/uploads/2021/03/deep-dive-on-aws-lambda-vti-cloud.jpg

As explained in IAM, Roles are used for one AWS service to communicate with other service

Source: https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2018/01/21/AR_Diagram_010418.png

AWS Lambda supports versioning (max versions are 5) and each version is immutable, i.e., once published, they cannot be changed and if changes are required, publish a new version, so in all, it can look like this: * LambdaFunctionName * LambdaFunctionName:1 <- Immutable * LamdaFunctionName:2 <- Immutable ### Execution Environment Lifecycle Execution Environments are secure and isolated runtime environments The lifecycle can be defined as shown below:

Source: https://trackit.io/aws-lambda-lifecycle/ and https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html and https://tmmr.uk/post/lambda/lambda-execution-environment/

Extension INIT - In the Extension Init phase the system starts all extensions to let them apply their logic to the Lambda Runtime INIT - The runtime is the combination of an operating system, programming language and a combination of software libraries required to invoke the Lambda function {% hint style="info" %} Note that when AWS Lambda was not used and is initialized for the 1st time at any given time, i.e., it was not active, then for its first run, it will take a little time to respond. This is called **Cold Start** When it has become active and is responding to 2nd or 3rd trigger or request, it is called as **Warm Start**. Here initialization doesn't happen and request goes straight to invoke stage {% endhint %} ### Types of Invocations AWS Lambda supports parallel invocations:

Source: https://blog.awsfundamentals.com/lambda-limitations

The characteristics of Lambda Execution environment are as follows: * Read Only File System * Memory based on Lambda configuration during setup * 512 MB of disk space in /tmp directory * No inbound connections * Only TCP/IP Outbound connections are allowed * Max timeout of 15 minutes {% hint style="info" %} If API Gateway is used for passing the trigger to lambda function and it has a timeout of 30 secs, so and if Lambda function code has a sleep of 35 seconds then it won't work, because there is timeout at API Gateway due to which API Gateway will not be able to relay the response back from Lambda to API Gateway all the way back to user. What is the solution then? {% endhint %} The solution to above problem can be diagrammatically summarized as: Use Asynchronous events or Stream (Poll-based)

Source: https://d2908q01vomqb2.cloudfront.net/fc074d501302eb2b93e2554793fcaf50b3bf7291/2019/06/27/Screen-Shot-2019-06-27-at-2.23.51-PM-1024x510.png

Source: https://media.licdn.com/dms/image/v2/D5612AQEkftRQJaBD3g/article-cover_image-shrink_600_2000/article-cover_image-shrink_600_2000/0/1721199571323?e=2147483647&v=beta&t=vyOf-dB-uRgAxj5sQe83_sWT71MnrKrnoiZqcE-KBuw

#### Improving Response Time by keeping Lambda Functions Warm

### Layers in Lambda Functions

Source: https://aws.amazon.com/blogs/mobile/how-to-use-lambda-layers-with-the-amplify-cli/

Layers can also have upto 5 versions and as they have versioning, they are also immutable. Lambda Functions can use multiple layers which can be compatible with a runtime. They are extracted to /opt directory as shown below: * Python - /opt/python/lib/python\/site-packages * Node.js - /opt/nodejs/node\_modules * Binaries - /bin * Java - /opt/java/lib * ```

``` Each layer has specific ARN. Layers can be stacked and ordering of layer matters Max size of a layer is 250 MB ### Lambda Environment Variables

Source: https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html

Note that the id (which tell username) for an lambda environment generally tarts with **sbx\_user\** ### Lambda Alias Routing Lambda supports not only creating alias of a function but also supports routing to it. This is generally used to test new features by traffic shifting to send a small percentage of traffic to a second function alias or version for a rolling deployment before production rollout. This is commonly called a [canary release](https://wa.aws.amazon.com/wat.concept.canary-deployment.en.html) Example: 90% of invocations route to the *stable* version while 10% route to alias *new-feature* pointing to version *3*. If the 10% is successful, deployment can continue until all traffic is migrated to version 3, and the *stable* alias is then pointed to version 3.

Source: https://d2908q01vomqb2.cloudfront.net/1b6453892473a467d07372d45eb05abc2031647a/2020/06/15/5-9-AWS-Lambda-alias-routing.png

### Event and Context The `event` parameter contains data that is passed to the Lambda function at the time of invocation. This data is specific to the event source that triggers the Lambda function.

Source: https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html#apigateway-example-event

When Lambda runs the function, it passes a `context` object to the [handler](https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html). This object provides methods and properties that provide information about the invocation, function, and execution environment. It's main role is to provide information about the current execution environment. Unlike `event`, the methods and properties of the `context` object remain the same regardless of the lambda was invoked or triggered Sample Code: ```python // Generated by ChatGPT def lambda_handler(event, context): # Accessing event data print("Received event:", event) # Accessing context data print("Request ID:", context.aws_request_id) print("Function Name:", context.function_name) # Your processing logic here return { 'statusCode': 200, 'body': 'Hello from Lambda!' } ``` ### Runtime It is mostly a script (**bootstrap executable file**) that executes everything in order for getting the proper work from everyone and giving the output to user, via Runtime API and Lambda service

So, runtime runs the function's setup code. It reads the handler name from an environment variable, reads the invocation event from lambda runtime API, passes the event data to function handler and posts the response from handler back to Lambda. It can distributed as public or private layers Working and Execution Flow can be explained as follows:

Source: https://attackdefense.com/listing?labtype=aws-cloud-security-bootcamp&subtype=aws-cloud-security-bootcamp-recordings

Runtime API Path (): * Invocation Request - GET "http\://${AWS\_LAMBDA\_RUNTIME\_API}/2018-06-01/runtime/invocation/next" * Invocation Response - POST "http\://${AWS\_LAMBDA\_RUNTIME\_API}/2018-06-01/runtime/invocation/AwsRequestId/response" * Invocation Error - POST "http\://${AWS\_LAMBDA\_RUNTIME\_API}/2018-06-01/runtime/invocation/AwsRequestId/error" * Invocation Error Path - POST "http\://${AWS\_LAMBDA\_RUNTIME\_API}/2018-06-01/runtime/init/error" ### Steps for using AWS Lambda Example Scenario: Generate Symmetric keys using Lambda

Now, to create a lambda function using dotnet cli, ensure that .NET SDK is installed from [here](https://dotnet.microsoft.com/en-us/download/dotnet/6.0). Create a folder to host the source code ```powershell cd dotnet nuget add source https://api.nuget.org/v3/index.json -n nuget.org dotnet new --install Amazon.Lambda.Templates dotnet new lambda.EmptyFunction --name SymKeyGen cd SymKeyGen cd src notepad SymKeyGen.csproj // make the following changes ``` ``` net6.0 //Change here enable enable true Lambda true false //Change here ``` ```powershell dotnet add package Amazon.Lambda.Core dotnet add package Amazon.Lambda.APIGatewayEvents dotnet add package AWSSDK.S3 dotnet add package Newtonsoft.Json notepad Function.cs ```

//Generated by ChatGPT

using System;
using System.IO;
using System.Security.Cryptography;
using System.Text;
using Amazon.Lambda.Core;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.S3;
using Amazon.S3.Model;
using Newtonsoft.Json;
using System.Text.RegularExpressions;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

public class Function
{
    private readonly string bucketName = "radifineuniquebucketname"; // Update with your S3 bucket name
    private readonly IAmazonS3 s3Client;

    public Function()
    {
        s3Client = new AmazonS3Client();
    }

    // Entry point for the Lambda function
    public APIGatewayProxyResponse FunctionHandler(APIGatewayProxyRequest request, ILambdaContext context)
    {
        return GenerateKey(request, context);
    }

    public APIGatewayProxyResponse GenerateKey(APIGatewayProxyRequest request, ILambdaContext context)
    {
        // Parse input parameters from the request
        if (request.QueryStringParameters == null || 
            !request.QueryStringParameters.TryGetValue("input", out var hexInput))
        {
            return new APIGatewayProxyResponse
            {
                StatusCode = 400,
                Body = JsonConvert.SerializeObject(new { Error = "Input parameter is required." }),
                Headers = new System.Collections.Generic.Dictionary<string, string>
                {
                    { "Content-Type", "application/json" }
                }
            };
        }

        // Decode hex input to original string
        string inputParam = HexToString(hexInput);

        // Validate the input
        if (!IsValidInput(inputParam))
        {
            return new APIGatewayProxyResponse
            {
                StatusCode = 400,
                Body = JsonConvert.SerializeObject(new { Error = "Invalid input parameter." }),
                Headers = new System.Collections.Generic.Dictionary<string, string>
                {
                    { "Content-Type", "application/json" }
                }
            };
        }

        // Generate a symmetric key based on the input parameter
        byte[] symmetricKey = GenerateKeyFromInput(inputParam);
        var base64Key = Convert.ToBase64String(symmetricKey);

        // Save the key to S3
        string fileName = $"{inputParam}.txt"; // Name the file using the input string
        var putRequest = new PutObjectRequest
        {
            BucketName = bucketName,
            Key = fileName,
            ContentBody = base64Key,
            ContentType = "text/plain"
        };

        // Upload the key to S3
        s3Client.PutObjectAsync(putRequest).Wait();

        // Return the key and success message
        var response = new
        {
            Key = base64Key,
            Message = "Key generated and saved successfully."
        };

        return new APIGatewayProxyResponse
        {
            StatusCode = 200,
            Body = JsonConvert.SerializeObject(response),
            Headers = new System.Collections.Generic.Dictionary<string, string>
            {
                { "Content-Type", "application/json" }
            }
        };
    }

    private bool IsValidInput(string input)
    {
        // Check for null or empty
        if (string.IsNullOrWhiteSpace(input))
            return false;

        // Check length (e.g., maximum length of 255 characters)
        if (input.Length > 255)
            return false;

        // Validate against a regex pattern (adjust as necessary)
        string pattern = @"^[a-zA-Z0-9\-\.]+$"; // Example pattern: allows alphanumeric, hyphens, and dots
        return Regex.IsMatch(input, pattern);
    }

    private byte[] GenerateKeyFromInput(string input)
    {
        using (var sha256 = SHA256.Create())
        {
            // Hash the input to create a fixed-length key
            return sha256.ComputeHash(Encoding.UTF8.GetBytes(input));
        }
    }

    private string HexToString(string hex)
    {
        StringBuilder result = new StringBuilder(hex.Length / 2);
        for (int i = 0; i < hex.Length; i += 2)
        {
            result.Append((char)Convert.ToByte(hex.Substring(i, 2), 16));
        }
        return result.ToString();
    }
}

```powershell dotnet build dotnet publish -c Release -o ./publish Compress-Archive -Path * -DestinationPath ../SymKeyGenSrc.zip ```

Edit the permissions for access to S3

> For those interested, can head over API Gateway to have complete scenario based understanding.

Now, it should work # Container Services Containers are packages of software that contain all of the necessary elements to run in any environment. Simply speaking, they are just virtualized environments which has all dependencies (configuration, runtime environment and additional packages) to run a particular code (application). This makes them run on top of any operating system hence are OS platform agnostic. AWS App Runner, is one of the many such container service offered by AWS with a focus on web applications or APIs. ### ECR There is another service called ECR (Elastic Container Registry) which is a service for storing, managing and deploying container container images such as Docker images. It is like a library where one keep all the Docker images so they can be easily accessed and used later. #### Working of ECR: **Image Repository:** Users can create repositories in ECR to store Docker images. Each repository can hold multiple image versions, making it a centralised hub for container images. **Image Push and Pull:** Developers can push (upload) their Docker images to ECR, while deployment environments can pull (download) the images for running containerized applications. **Integration with ECS and EKS:** ECR seamlessly integrates with ECS and EKS, simplifying container deployment and orchestration. ### ECS and EKS There are two container orchestration services offered by AWS, namely, Amazon Elastic Container Service ECS and Amazon Elastic Kubernetes Service EKS. Container orchestration services are tools that help manage the deployment, scaling, and operation of containerized applications. The main difference as EKS is Kubernetes based specifically and ECS supports Docker & other Open Container Initiative (OCI) standards based containers. For ECS or EKS, user has to select two options for Compute Engine; they are EC2 and AWS Fargate. If EC2 is selected as compute engine, the onus to manage scaling and defining the compute resources such as no. of resources such as vCPU, RAM etc. and also the no. of such instances. AWS Fargate on the other hand is a serverless compute engine for use with container orchestration services. i.e. the user don't have to manage anything around scaling no. of instances to run containers. That part is taken up by AWS in the form of Fargate. # ECR and ECS A **registry** is a storage location for Docker container images in AWS. It is a logical namespace that can contain multiple **repositories.** A **repository** is a collection of related Docker container images within a registry. It stores different versions (tags) of container images. A repository is where one can push, pull, and manage the lifecycle of Docker images. ### Steps for using ECR

* The **URI** is what is used to reference the repository images while creating containers on ECS clusters. Format: \.dkr.ecr.\.amazonaws.com/ \ * The **Tag immutability** column lists its status, if tag immutability is enabled it will prevent image pushes with pre-existing tags from overwriting the images. * The **Encryption type** column lists the encryption properties of the repository, it shows the default encryption types such as AES-256, or has KMS enabled encryption. * The **Pull through cache** column lists its status, if Pull through cache status is Active it will cache repositories in an external public repository into the private repository

* The **Permissions** section allows to apply a registry policy to grant permissions to an AWS principal at the private registry level. These allow us to scope access to the Replication and pull through cache configuration features of private registry. * The **Pull through cache configuration** lets set pull through cache rules for the repository. The Replication configuration lets set rules to manage cross region and cross account repository replication. On the private repositories listing, The **Pull through cache** column lists its status, if Pull through cache status is Active it will cache repositories in an external public repository into the private repository * The **Scanning configuration** has two options: * **Basic scanning** is a free service that allows manual scans and scans on push of images in the registry. * **Enhanced scanning** provides automated continuous scanning that identifies vulnerabilities in both operating systems and enhanced scanning

{% hint style="info" %} The Public tab lists the repositories which are publicly accessible from this account. A unique default alias is added to the repository name at creation, it is used for identification of this repository from the multitude of public repositories on AWS. {% endhint %} To Push Docker Images to ECR * **Tag Docker Image**: ```bash docker tag your-image-name:latest .dkr.ecr..amazonaws.com/your-repository:your-tag ``` * **Login to ECR**: Authenticate Docker to your ECR registry: ```bash aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com ``` * **Push the Image**: Push the tagged image to ECR: ```bash docker push .dkr.ecr..amazonaws.com/your-repository:your-tag ``` * **Pull the Image**: `docker pull .dkr.ecr..amazonaws.com/your-repository:your-tag` Example:

{% hint style="info" %} Integrate ECR with CI/CD tools (like AWS CodePipeline or Jenkins) to automate the process of building, tagging, and pushing images to ECR whenever changes are made to your codebase. {% endhint %} ```bash //After pulling an image, if one wants to run a docker image docker images <-- this will give iamge ID docker run -it -p : <-- It will run docker exec -it bash ``` * Implement a tagging strategy that includes version numbers and identifiers for easy tracking and updates. When one create new versions of the application, use a consistent tagging strategy (like semantic versioning) to clearly indicate changes (e.g., `v1.0.0`, `v1.1.0`). * After pushing new images, update deployment configurations in ECS or EKS to use the new tags. This is done on the basis of Task Definition as one can create it such that specifies container image with a particular tag. One can use environment variables or parameters to dynamically set the image tag based on requirements. **ECS**, is a logical group of EC2 instances on which you can run an application without having to scale your own cluster management infrastructure because ECS manages that for you. If you manage to compromise service running in **ECS**, the **metadata endpoints change**. <-Will update later, so ignore the below-> Deploy Images Using Tags in ECS or EKS //Give snapshot images below for the steps * When creating or updating an ECS Task Definition or an EKS Deployment, specify the image URI with the tag: * Example: `.dkr.ecr..amazonaws.com/nameof-repository:tag-given`. * For ECS, you can do this in the Task Definition: * Go to **ECS** in the console. * Select **Task Definitions** and either create a new one or edit an existing definition. * Under **Container Definitions**, specify the image URL with the tag. # EKS Some part of this page is summarized using ChatGPT ## PreContext ### What is Kubernetes? Kubernetes is an open-source platform that acts as an orchestration tool, meaning it automates the processes involved in deploying, managing, and scaling containers. This includes starting and stopping containers, monitoring their health, and handling failures. Kubernetes can manage hundreds or thousands of containers spread across multiple servers (nodes), handling load balancing, service discovery, and failover automatically.

Source: https://www.nccgroup.com/us/research-blog/detection-engineering-for-kubernetes-clusters/

### USP Of Kubernetes Kubernetes provides the capability to scale up horizontally as opposed to scaling vertically, i.e., spreading the workload by adding more nodes instead of adding more resources to existing nodes Kubernetes provides the capability to scale up horizontally as opposed to scaling vertically, i.e., spreading the workload by adding more nodes instead of adding more resources to existing nodes ### How different it is from from Docker? While Docker handles the creation and management of individual containers, Kubernetes takes it a step further by orchestrating those containers across multiple hosts. Many organizations use Docker to create their containers and then deploy them in a Kubernetes cluster i.e., Kubernetes can use Docker as its container runtime, meaning that Kubernetes can manage and orchestrate containers created with Docker. However, Kubernetes is also compatible with other container runtimes. ### Concepts of Kubernetes: Kubernetes uses services to define a logical set of pods and a policy for accessing them. This abstraction allows for stable communication between different parts of an application, even as the underlying pods change. * **Containers:** At its core, Kubernetes manages containers. Containers are lightweight, portable units that package an application along with its dependencies, libraries, and configuration files. Containers ensure that an application runs consistently across different environments. Container share the operating system kernel but run in isolated user spaces, which makes them efficient and quick to start. * **Pods:** The smallest deployable units in Kubernetes are called pods. **A pod can contain one or more containers that share the same network namespace and storage as they can be closely related and need to work together**. This facilitates that containers within the pod can communicate with each other using `localhost`, while sharing storage volumes. Example: A Pod containing a Web Server Container and a Log Server Container.

Source: https://kubernetes.io/images/docs/kubernetes-cluster-architecture.svg

* **Cluster Architecture:** The Kubernetes cluster is made up of nodes. Nodes are the environment in which pods operate. A node can be both baremetal machine or Virtual Machine. The cluster consists of a master node (which manages the cluster) and worker nodes (where the containers run). This architecture is what makes Kubernetes help to realize the essence of distributed computing. * **Master Node:** This node manages the Kubernetes cluster and is responsible for maintaining the desired state of the applications. It handles scheduling, scaling, and monitoring. It can automatically scale applications up or down based on demand. If more users are accessing the application, it can increase the number of pods to handle the load and then scale down when traffic decreases. It has following components: #### 1. etcd * **Definition:** etcd is a distributed key-value store used by Kubernetes to store all its configuration data and state information. * **Data Storage:** etcd holds the entire cluster state, including information about nodes, pods, services, and configurations. * **Consistency:** etcd ensures data consistency and reliability by data replication. This is crucial for fault tolerance. #### 2. API Server * **Definition:** The API server is the central management component of Kubernetes that exposes the Kubernetes API. * **Communication Hub:** It acts as the main interface for all interactions with the cluster, allowing users, applications, and other components to communicate. * **RESTful Interface:** The API server provides a RESTful interface for clients to send requests to create, update, or delete resources in the cluster. * **Authentication and Authorization:** It handles requests and applies security policies, ensuring that only authorized users and components can access or modify resources. Kubernetes uses roles to determine if a user or pod is authorized to make a specific connection/call. Roles are scoped to either the entire cluster (ClusterRole) or a particular namespace (Role). These roles contain lists of resources (the object) the role can grant access to, and a list of verbs that the role can perform on the said resource, are declared and then attached to RoleBindings. RoleBindings pretty much link the role (permissions) with the users and systems. More information on this functionality can be found [here](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) in the Kubernetes official documentation.

// Sample taken from Kubernetes documentation
      
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        namespace: default
        name: pod-reader
      rules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get", "watch", "list"]

#### 3. Scheduler * **Definition:** The scheduler is responsible for assigning pods to worker nodes based on resource availability and requirements. * **Pod Placement:** When a new pod is created, the scheduler determines the best node to run that pod, considering factors like resource requests, node capacity, and affinity/anti-affinity rules. * **Load Balancing:** It helps balance workloads across nodes to ensure efficient resource utilization and prevent overloading any single node. * **Custom Scheduling Policies:** Users can define custom scheduling policies, such as placing pods closer to certain resources or spreading them across nodes for high availability. #### 4. Controllers * **Definition:** Kubernetes user can define the desired application state (how many replicas, what images to use, etc.) in configuration files, and Kubernetes works to ensure the current state matches the desired state with the help of controllers. Controllers are control loops that monitor the state of the cluster and make adjustments to ensure the desired state matches the actual state. * **Deployment Controller:** Manages deployments by ensuring that the desired number of pod replicas are running and up-to-date. * **ReplicaSet Controller:** Ensures that a specified number of replicas for a pod are running at any given time. * **Node Controller:** Monitors the health of nodes and manages node lifecycle events, such as marking nodes as unavailable when they fail. It has the ability to automatically replace or restart failed containers, ensuring that the desired state of the application is maintained * **Job and CronJob Controllers:** Manage the execution of jobs and scheduled tasks. * **Worker Nodes:** These nodes run the actual application workloads, hosting the pods that contain the containers. #### 1. Kubelet * **Definition:** It is an agent that runs on each worker node in a Kubernetes cluster. Its primary job is to manage the pods and containers on that node. * Kubelet receives pod specifications (instructions) from the Kubernetes API server and ensures that the specified containers are running in those pods. * It continuously monitors the health of the pods and their containers, reporting back to the Kubernetes control plane. If a container crashes or fails, Kubelet will attempt to restart it. * Kubelet manages resources on the node, including CPU and memory, to ensure that pods have what they need to run efficiently. #### 2. Kube-proxy * **Definition:** It is a network proxy that runs on each worker node in a Kubernetes cluster. Its role is to facilitate communication between services and pods. * Kube-proxy maintains network rules on the nodes, allowing for load balancing and routing of requests to the appropriate pod based on the service configuration. * Kube-proxy supports various protocols, including TCP and UDP, allowing for versatile network communication #### 3. Pods * Already discussed about the pod in earlier points. * **Service:** Kubernetes services provide a stable endpoint for accessing a set of pods. Services allow for load balancing and automatic DNS resolution, making applications accessible within the cluster or externally. Some of the Kubernetes service are:

Source: https://www.linkedin.com/pulse/day-33-working-namespaces-services-kubernetes-kartik-bhatt-hloyc

* **Namespace:** Within a physical Kubernetes cluster, namespace is a logical, virtual cluster which provide a way to segregate and isolate resources and objects within a cluster, creating distinct scopes for applications and services

{% hint style="info" %} kubectl is the command-line tool used to interact with the Kubernetes cluster. It acts as the primary way to communicate with the Kubernetes API server. kubectl allows engineers to apply configuration files (YAML or JSON) that define how their applications should run in the cluster. {% endhint %} {% hint style="info" %} **ConfigMaps** in Kubernetes are a way to manage configuration data for applications in a decoupled manner. They allow engineers to separate configuration from the application code, making it easier to manage and update configurations without rebuilding the container images. Here’s a breakdown of what ConfigMaps are and how they work: A ConfigMap is a Kubernetes API object that allows to store non-confidential data in key-value pairs. This can include configuration settings, command-line arguments, environment variables, or any other configuration details needed by the applications. {% endhint %} ## AWS EKS Features: Amazon Elastic Kubernetes Service (AWS EKS) is a managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications using Kubernetes. AWS EKS abstracts the complexities of Kubernetes cluster management and offers a reliable and scalable platform for running containerized workloads. Key components include: **EKS Cluster:** The EKS cluster is the central management entity that hosts multiple worker nodes and manages their orchestration. It’s responsible for maintaining the desired state of Kubernetes applications. **Worker Nodes:** Worker nodes are EC2 instances within your EKS cluster that run containerized applications. These nodes are managed by the EKS control plane.

Source: https://www.devopsschool.com/blog/wp-content/uploads/2021/03/Amazon-Elastic-Kubernetes-Service-EKS-Explained-Diagram-5.png

Source: https://rafay.co/the-kubernetes-current/getting-started-with-amazon-eks/

## Bonus Learning: Rancher Rancher is a powerful and popular open-source container management platform that simplifies the deployment, orchestration, and management of containerized applications in Kubernetes clusters across multiple environments. It provides a comprehensive suite of tools and features that streamline the containerization journey for organisations, making it easier to harness the full potential of container technology. It provides a user-friendly interface for managing containers, clusters, and applications, making it easier to deploy and maintain container-based applications. {% hint style="info" %} **In short, Rancher builds on top of Kubernetes to manage multiple Kubernetes cluster that can span across different Cloud Providers** {% endhint %}

Source: https://miro.medium.com/v2/resize:fit:720/format:webp/1*TnFotcV01sGOnJ0_4Q8Eww.png

Some key aspects and benefits of using Rancher: **Centralised Management:** Rancher offers a unified, web-based interface that allows users to manage and monitor multiple Kubernetes clusters across different environments from a single, centralised dashboard. This simplifies the management of complex container infrastructures, whether they are on-premises, in the cloud, or at the edge. **Kubernetes Made Accessible:** Rancher abstracts much of the complexity of Kubernetes, making it accessible to a wider range of users, from DevOps teams to developers. It provides an intuitive user experience for cluster provisioning, scaling, and lifecycle management, reducing the learning curve associated with Kubernetes. **Multi-Cluster Management:** Organisations often operate multiple Kubernetes clusters for various purposes, such as development, testing, and production. Rancher excels at managing multiple clusters, making it easy to ensure consistency, security, and compliance across these clusters. It simplifies the process of creating, upgrading, and patching clusters. **Extensibility:** Rancher’s open architecture allows users to easily integrate additional tools and services. It supports a wide range of third-party plugins and extensions, making it adaptable to various infrastructure and application requirements. This extensibility enhances Rancher’s capabilities, such as networking, storage, and security. **Security and Compliance:** Rancher offers robust security features, including role-based access control (RBAC), identity and access management (IAM), and security scanning for container images. These capabilities help organisations enforce security policies and ensure compliance with industry standards and regulations.

Source: https://www.webagesolutions.com/blog/rancher-introduction

Additional Reading Material: # AppRunner # Other AWS Services # CloudFormation It can be create AWS Environments at scale in a much quicker way, as it is a way to code a infrastructure. It is a template to setup/configure resources which is given in YAML or JSON It can be used to create stack (AWS Resources such as VPC or S3 or EC2 or any other service supported) as IAM user or Assume Role # AWS Key Management Services (KMS) When data is moved into the cloud, it is best to secure it at both the stages - when data is in transit and when the data is at rest and one of the services from AWS that can be foundational into dealing with keys for encryption is KMS or Key Management Service, mainly used to provide, store and manage encryption keys for data at rest. Note that KMS also allows auditing of who used the KMS keys with the help of [CloudTrail](/aws/other-aws-services/logging-and-monitoring/aws-cloudtrail). It is also possible to exercise access control to KMS keys by creating usage policies. KMS allows centralized key management that has compatibility to a variety of other AWS services. AWS KMS facilitates the creation of both Symmetric key and Asymmetric keys (like for EC2 access over ssh) KMS keys can be of two types: * Customer Master Keys (CMKs) - It can be customer managed (uploaded by the customer or generated by KMS for the customer) or AWS-managed (generated in KMS by AWS whenever required). It can act as one of the input in generating unique data keys as well as used to encrypt and decrypt the generated data keys, i.e. CMKs can act as Key encryption keys and this process is known as Envelope Encryption * Data Encryption Keys - It is also known as data keys, which is used to encrypt and decrypt the data. Note that master key never leaves the KMS and DEK generated each time is unique, for better security. Using the keys in KMS, AWS encrypts and decrypts the data in S3 or s3 glacier or EBS or EFS or FSx or RedShift or RDS or Amazon EMR and even CloudTrail. This process is called Server-Side Encryption. Different flavours of SSE are as follows: * Server Side Encryption using AWS Managed Keys (SSE-KMS) - Data Encryption key (DEK) is used, which itself is encrypted using Customer Master Key. * [x] Example: To encrypt [EBS](/aws/aws-storage-services/amazon-ebs) volumes, KMS is used which holds the customer master keys (responsible for generating data keys) and data keys (for encryption of data on EBS volumes). * Server Side Encryption using Customer Provided Encryption key (SSE-C) - The customer generated key is uploaded into KMS is used for encryption data directly, unlike in previous case, where DEK (data encryption key) is generated using Customer Master key {% hint style="info" %} Note that in most data at rest cases, let's just say s3; so when data is uploaded into s3, the DEK is part of metadata for the encrypted s3 object in the bucket, therefore when required access to the encrypted object is requested, AWS s3 should be able able to get that quickly {% endhint %} There is something called Client-Side Encryption, where the client only supplies 256-bit key and key encrypting key and encrypt the data and encrypted blob is directly uploaded as it is (encrypted) in s3. Please note that it is essential for KMS Administrative tasks such as * Control access to IAM users and role that can administer and may or may not use the key * Rotate the keys * Disable keys and Re-enable keys when required * Audit the usage of keys AWS Console for KMS keys shows the following options:

## Steps for triggering creation of AWS managed (KMS key/Customer Master key):

## Steps for creating KMS generated Customer managed (KMS key/Customer Master key):

## Steps for importing customer uploaded Customer managed (KMS key/Customer Master key): <-will be added later-> ## Steps for using customer provided data encryption key: <-will be added later-> # AWS Secrets Manager ### Features AWS Secrets Manager is where secrets (from POV of the customer or AWS account holder) can be stored. is a managed service provided by Amazon Web Services (AWS) that simplifies the secure storage and management of sensitive information . Secrets can include passwords, API keys, tokens, and other sensitive data that applications, services, or IT resources need to access. i.e., It can be used to store database credentials or API keys or tokens or any other app secret etc. It operates by allowing users to create and store secrets, in a centralized and highly secure manner. The primary goal of AWS Secrets Manager is to safeguard access to services and applications by eliminating the need to hard-code sensitive information in plaintext, thereby enhancing security and compliance Typical use case is webapp hosted on AWS securely gets access to RDS database. It also supports DocumentDB or RedShift. Key features of AWS Secrets Manager include: **Secret Creation:** Users can create secrets using the AWS Management Console, SDKs, or the AWS Command Line Interface (CLI). These secrets can include database credentials, API keys, or any other sensitive data. **Centralized Management**: It provides a centralized place to manage all secrets, making it easier to organize, access, and control them across AWS environment. **Encryption:** AWS Secrets Manager encrypts these secrets both at rest and in transit using industry-standard encryption protocols, ensuring the confidentiality and integrity of the stored data. The secrets managed by Secrets Manager can be encrypted using keys managed by KMS. **Access Control:** Users can define fine-grained access control policies to specify who can access and manage secrets, helping to maintain the principle of least privilege. **Audit and Monitoring**: Integration with AWS CloudTrail provides a record of calls to AWS Secrets Manager for compliance and auditing purposes. **Cross-Region Replication**: It allows to replicate secrets across multiple AWS regions for disaster recovery purposes. **Rotation:** One of the standout features of AWS Secrets Manager is its ability to automate the rotation of credentials. The secrets can be auto-rotated during configuration. It can periodically generate new credentials for a secret and update the applications that use them, thus enhancing security by regularly changing access credentials. **Integration:** AWS Secrets Manager can seamlessly integrate with AWS services and other applications, allowing for secure retrieval of secrets during application runtime. This eliminates the need for hard coding credentials within applications. ### Steps for usage

Once the rotation function is given, review and create the secret to later integrate it with the application # AWS Certificate Manager (ACM) ACM is useful only, if one wants to use SSL/TLS Certificate with other AWS services such as Elastic Load Balancing, Amazon CloudFront, Amazon Cognito, AWS Elastic Beanstalk, AWS App Runner, Amazon API Gateway, AWS Nitro Enclaves, AWS CloudFormation, AWS Amplify, Amazon OpenSearch Service, and AWS Network Firewall. To get ACM certificate, steps are as follows:

# AWS Messaging Services AWS SNS and AWS SQS are the two messaging services provided by Amazon Web Services. Both have use cases to be used to be integrated for applications for messaging Example: Using Amazon SNS, messages are being subscribed to a topic which based on business logic can be queued to appropriate Lambda function as trigger

Source: https://miro.medium.com/v2/resize:fit:720/format:webp/1*Zh0muEji5v4nKw60pUeMAA.png

Another example can also be illustrated as follows:

Source: https://media.amazonwebservices.com/blog/sns_sqs_image_proc_2.png

Order Fulfilment at eCommerce site:

Source: https://www.slideshare.net/JungHakLee4/awskrug-aws-sns-sqs-and-ses

# AWS SNS (Simple Notification Service) Amazon Simple Notification Service (AWS SNS) is a fully managed publish/subscribe messaging service. What is pub/sub messaging service? As per ChatGPT the following can be summarized:

{% hint style="info" %} Apache **KAFKA** is also an example of such messaging service {% endhint %} It enables to create decoupled applications i.e., Publishers do not need to know the details of the subscribers, and subscribers do not need to know who publishes messages. It enables the distribution of messages and notifications across different channels such as email, SMS, HTTP/HTTPS, push notifications etc. AWS SNS uses a publish-subscribe model, where publishers send messages to topics, and subscribers receive those messages based on their interest in specific topics i.e., **subscribers will receive notifications for what they have subscribed for.** A Topic ARN is a unique identifier for an SNS topic within AWS. It’s used to address and target specific topics when publishing messages. Topic ARNs are essential for routing messages to the appropriate subscribers and play a key role in the access control policies that define who can publish and subscribe to specific topics. # AWS Simple Queue Service (SQS) Where AWS SNS lags is that it just passes messages, but cannot store the message for very long. This is where AWS Simple Queue Service comes into picture, where message can reside for upto 14 days. It is also a fully managed service that is fully fault tolerant, meaning if something go down (such as power outages at the consumer end of the queue), because messages in queue can wait, it can be processed later. SQS offers two types of queues: * **Standard Queues**: These provide at-least-once delivery, meaning a message might be delivered multiple times **but is processed in no particular order**. * **FIFO (First-In-First-Out) Queues**: These ensure that messages are processed exactly once and in the order they are sent. * **Dead-Letter Queues:** If a message cannot be processed after a specified number of attempts, it can be sent to a dead-letter queue for further investigation and debugging. # AWS Systems Manager Previously called as Simple Systems Manager ### Features AWS Systems Manager is for management of AWS Resources. It groups our resources into Resource Groups. It analyse our workloads to gain insights to operational data and it can also let AWS users perform actions on resource groups Systems Manager Automation enables you to run a sequence of workflows (e.g., AWS API calls, provisioning instances, creating buckets, and firing off Lambda functions) through the convenience of a single SSM Automation document. AWS Automation can helps to automate mundane or repetitive tasks for maintenance or deployment of cloud native or hybrid infrastructure. Via a series of steps/actions defined in either JSON or YAML, SSM Automation offers useful plugins to carry out commands/API calls Chief use case is to achieve operational efficiency for the workload running in the cloud. Examples: * Automation - Creation of golden AMI Image. * Patch Management - For deploying specific patches to worklaods in the cloud. * Run Command - Can be used to execute commands onto all EC2 systems. * Parameter Store - Can be used to store secrets and other configuration data in hierarchical storage. * OpsCenter - Can be used to have centralized view to responds to alerts. * Session Manager - Can be used to connect to private EC2 machines replacing bastion hosts. It has many sub services under Systems Manager:

# Application Management Two very vital services under application management subservice umbrella of AWS Systems Manager are AppConfig and Parameter Store. AppConfig as the name says, is used to to create, manage and deploy app configs and Parameter store secure hierarchial storage for secrets and data ### Steps for AppConfig When an application is hosted on EC2, with the help of privileges assigned to IAM role, it can fetch the configuration of AppConfig, which internally can source configuration values from various sources such as S3, or Parameter store etc and can have multiple app config profiles with the help of Environments like Dev or Prod.

### Steps for Parameter Store As explained earlier, it can be used to hold key value pair to hold data in plaintext as well as secure format. This can include Database credentials also, which can internally be used in app config, which in turn can be used by the application to access the Database Note every parameter can have version for tracking history

# Node Management As noted previously, **Systems Session Manager** is a service that is used to provide cli based access to EC2 workloads without the need to expose it on the internet or without the requirement of bastion host or jump host. It is part of Node Management sub service of AWS Systems Manager AWS SSM Session Manager is an excellent feature to connect to and manage all hybrid infrastructure remotely without having to use SSH for Linux or RDP for Windows instances. It provides the added bonus of security, as IT Teams don't need SSH/RDP ports open and access control to who can establish shell sessions using IAM policies. It also allows for logging and auditing of all activity during a shell session via Session Manager. For this to work, an SSM agent either needs to be preinstalled or needs to be installed on the EC2 instance. It is compatible with both Windows as well as Linux The steps to setup the same are laid out in detail [here](https://guides.radifine.com/how-to-securely-access-and-transfer-file-to-an-ec2-instance-in-private-subnet-of-vpc-in-aws). SSM agent allows administrators to remotely execute commands, automate tasks, and manage configuration settings on those instances. It provides a secure and efficient way to perform various operational tasks, such as software patching, inventory management, and software installation with the highest privileges on the EC2 instances Note that to connect Virtual Machines in other cloud providers i.e., Non-AWS Virtual machines or on-prem machines, **Hybrid Activations** can also be used. However, it requires Key activation for SSM Agent when installed on them. It can as summarized below:

Note all of the instances that have AWS SSM agent installed can be collectively managed into **Fleet Manager** Another important feature to understand the types of automation that can be done is to check SSM Documents (present under **Shared Resources -> Documents** ). A sample SSM Document format can look like as shown below: # Logging and Monitoring # AWS CloudTrail AWS CloudTrail is an AWS Service that records all activities performed in an AWS account (via cli or Mgmt Console or SDKs). This logging and auditing service from AWS can also be integrated in SIEM Solution for helping not only in Visibility, Incident Response, Debugging as well as fulfil the compliance requirements. Event Logging in AWS:

Source: https://quizlet.com/616135849/aws-cloudtrail-flash-cards/

As shown above, there are mainly two types of events, management events: (control plane operations) and data events (data plane operations) and both types of events will log AWS Account, User ID/Role, IP Address, Time and resource details etc. Events from 90 days can be viewed from Event History. Some of the things that should be considered while setting up CloudTrail * Not all services are supported * Only 5 trails per region are allowed The logs collected in CloudTrail can be used with S3, Lamda, Athena and even CloudWatch for analysing, storage etc. ### CloudTrail Creation: To create an audit trail in this service, steps can be as follows:

Quick Trail Console:

More Detailed Console:

Finally create the trail:

Event History would look something like this:

# AWS CloudWatch # AWS Macie # AWS Inspector # AWS GuardDuty # Welcome to Notes of Radifine Labs!! These are personalized notes of Yobroda to help the community in general with concepts ## Table of Contents ### Linux Fundamentals You can visit it here: ### AWS Fundamentals You can visit it here: