What is a ddos ​​attack. DDoS attack: how to do it? Programs for DDoS attacks

If you read our guide and implement all the described technologies - secure your computer from hacker threats! Don't neglect it!

In the field of information security, ddos ​​attacks occupy one of the leading places in the ranking of electronic threats. But most users have very limited knowledge in this subject. Now we will try to cover this topic in the most detailed and accessible way, so that you can imagine what this type of electronic threat means, how it is carried out, and, accordingly, how to deal with it effectively. So get acquainted - DDOS attack.

Terminology

To speak the same language, we must introduce terms and their definitions.

Dos attack is a denial of service attack. Hence the English abbreviation dos - Denial of Service. One of the subtypes is a distributed attack, which is carried out simultaneously from several, and as a rule, from a large number of hosts. We will devote the main part of the discussion to these options, because a ddos ​​attack carries more destructive consequences, and a significant difference is only in the number of hosts used for the attack.

To make it easier for you to understand. Such actions are aimed at temporarily stopping the operation of any service. This may be a separate site on the network, a large Internet or cellular provider, as well as a separate service (accepting plastic cards). In order for the attack to succeed and bring destructive actions, it must be carried out from a large number of points (this point will be discussed in more detail later). Hence the "distributed attack". But the essence remains the same - to interrupt the operation of a certain system.

To complete the picture, you need to understand who and for what purpose conducts such actions.

Denial-of-service attacks, like other computer crimes, are punishable by law. Therefore, the material is presented for informational purposes only. They are carried out by IT-specialists, people who are well versed in the topics of "computers" and "computer networks", or, as it has become fashionable to say, hackers. Basically, this event is aimed at making a profit, because, as a rule, ddos ​​attacks are ordered by unscrupulous competitors. Here it would be appropriate to give a small example.

Suppose there are two large Internet providers in the service market of a small city. And one of them wants to oust a competitor. They order a distributed dos attack on a competitor's server from hackers. And the second provider, due to the overload of its network, is no longer able to provide Internet access to its users. As a result - the loss of customers and reputation. Hackers get their reward, an unscrupulous provider - new customers.

But there are cases when they "ddo" just for fun, or to hone their skills.

Distributed DDoS attack

Let's agree right away - we will analyze computer attacks. Therefore, if we are talking about several devices from which the attack is carried out, these will be computers with illegal software.

Here, too, it is appropriate to make a small digression.. In fact, in order to stop the operation of any service or service, you need to exceed the maximum allowable load for it. The simplest example is accessing a website. One way or another, it is designed for a certain peak attendance. If at a certain point in time ten times more people visit the site, then, accordingly, the server will not be able to process such a volume of information, and will stop working. And connections at this moment will be made from a large number of computers. These will be the same nodes that were discussed above.

Let's see what it looks like in the diagram below:

As you can see, the hacker took control of a large number of user computers and installed his spyware on them. It is thanks to him that he can now perform the necessary actions. In our case - to carry out a ddos ​​attack.

Thus, if you do not follow the safety rules when working at a computer, you can be exposed to a virus infection. And perhaps your computer will be used as a host for malicious activities.

You will need: we described some aspects of security in the article.

But how they will be used depends on which option is chosen by the attacker

Classification of ddos ​​attacks

The following types of attacks can be attempted by attackers:

  1. Bandwidth congestion. In order for computers connected to the network to communicate normally, the communication channel through which they are connected must work normally and provide sufficient parameters for specific tasks (for example, bandwidth). This type of attack is aimed precisely at overloading network communication channels. This is achieved by constantly sending incoherent or system information (ping command)
  2. Resource limit. We have already considered this type above, in the example with access to a website. As we noted - the server was able to handle a limited number of simultaneous connections. An attacker needs to direct a large number of simultaneous connections to the server. As a result, the server will not cope with the load, and will stop working.
  3. Attack on DNS servers. In this case, the DDOS attack is also designed to stop access to the website. Another option is to redirect the user from the correct site to the fake one. This may be done in order to steal personal data. This is achieved by attacking the DNS servers, and substituting ip-addresses for fake ones. Let's explore this with an example. A certain bank uses its website to pay via the Internet. The user needs to go to it and enter the details of his plastic card. An attacker, in order to steal this information, creates a site of the same type and attacks the DNS servers (name servers). The purpose of this event is to redirect the user to the attacker's website when he tries to access the bank's website. If this succeeds, the user, unaware of the threat, will enter his personal data on the attacker's website, and he will gain access to them
  4. Software bugs. The most difficult is this type of attack. Malefactors reveal defects in the software, and use them for the purpose of destruction of system. To order such a ddos ​​attack, you will need to spend a lot of money

How to conduct a DDOS attack with your own hands

As an example, we decided to show you how you can carry out a DDOS attack using special software.

To get started, download the program here. After that, launch it. You should see the start window:

You need to make the minimum settings:

  1. In the column "URL" we write the address of the site that we want to attack
  2. Then press the button "Lock on" - We will see the target resource
  3. We put the TCP method
  4. Choose the number of threads (Threads)
  5. Set the upload speed using the slider
  6. When all the settings are completed, press the "IMMA CHARGIN MAH LAZER" button

All - the attack began. Once again, all actions are presented for informational purposes.

How to protect yourself from DDOS attacks

You probably already understood that this type of threat is very dangerous. Therefore, it is very important to know the methods and principles of combating and preventing distributed attacks.

  1. Setting up filtering systems is a task for system administrators and hosting providers
  2. Acquisition of protection systems against DDOS attacks (software and hardware systems)
  3. Use of Firewall and access control lists (ACLs) - this measure is aimed at filtering suspicious traffic
  4. Increasing available resources and installing redundancy systems
  5. Response technical and legal measures. Up to bringing the perpetrator to justice

Video for the article:

Conclusion

Now you probably understand the danger of DDOS attacks. The issues of ensuring the security of your resources must be approached very responsibly, sparing no time, effort and money. Better yet, have a separate specialist, or an entire information security department.

Regular readers very often asked the question of how you can edit the text if the file is in PDF format. The answer can be found in the material -

To protect your data, you can use a whole range of measures. One of these options is

If you need to edit your video online, we have prepared an overview of the popular ones for you.

Why look for information on other sites, if everything is collected from us?

Fighting DDoS attacks is not only difficult, but also exciting work. It is not surprising that every system administrator first of all tries to organize defense on his own - especially since this is still possible.

We decided to help you in this difficult task and publish some short, trivial and non-universal tips for protecting your site from attacks. These recipes will not help you deal with any attack, but they will save you from most dangers.

The Right Ingredients

The harsh truth is that many sites can be taken down by anyone using the Slowloris attack, which kills Apache tightly, or by arranging a so-called SYN flood using a farm of virtual servers raised in a minute in the Amazon EC2 cloud. All of our further tips for protecting yourself from DDoS on your own are based on the following important conditions.

1. Get rid of Windows Server

Practice suggests that a site that runs on Windows (2003 or 2008 - it doesn't matter) is doomed in the event of DDoS. The reason for the failure lies in the Windows network stack: when there are a lot of connections, the server will certainly begin to respond poorly. We don't know why Windows Server performs so badly in such situations, but we've come across this more than once or twice. For this reason, this article will focus on the means of protection against DDoS attacks in the case when the server is running on Linux. If you are a happy owner of a relatively modern kernel (starting from 2.6), then the primary tools will be the iptables and ipset utilities (for quickly adding IP addresses), with which you can quickly ban bots. Another key to success is a well-prepared network stack, which we will also talk about next.

2. Part ways with Apache

The second important condition is the rejection of Apache. If you have Apache, then at least put a caching proxy in front of it - nginx or lighttpd. Apache "is extremely difficult to give files away, and, even worse, it is fundamentally (that is, incorrigibly) vulnerable to the most dangerous Slowloris attack, which allows you to flood the server almost from a mobile phone. To combat various types of Slowloris, Apache users first came up with a patch Anti-slowloris.diff, then mod_noloris, then mod_antiloris, mod_limitipconn, mod_reqtimeout... But if you want to sleep well at night, it's easier to take an HTTP server that is immune to Slowloris at the level of code architecture.So all our further recipes are based on the assumption that nginx is used on the frontend.

Fighting off DDoS

What to do if DDoS comes? The traditional self-defense technique is to read the HTTP server log file, write a grep pattern (that catches bot requests), and ban anyone who falls under it. This technique will work... if you're lucky. There are two types of botnets, both dangerous, but in different ways. One completely comes to the site instantly, the other - gradually. The first one kills everything at once, but the whole thing appears in the logs, and if you grepaet them and ban all IP addresses, then you are the winner. The second botnet lays down the site gently and carefully, but you will probably have to ban it for a day. It is important for any administrator to understand: if you plan to fight with grep, then you need to be prepared to devote a couple of days to fighting the attack. The following are tips on where you can put straws in advance so that it is not so painful to fall.

3. Use the testcookie module

Perhaps the most important, effective and operational recipe of this article. If DDoS comes to your site, then the testcookie-nginx module, developed by @kyprizel, can become the most effective way to fight back. The idea is simple. Most often, bots that implement HTTP flooding are rather dumb and do not have HTTP cookie and redirect mechanisms. Sometimes more advanced ones come across - they can use cookies and process redirects, but almost never a DoS bot carries a full-fledged JavaScript engine (although this is becoming more and more common). Testcookie-nginx acts as a quick filter between bots and the backend during an L7 DDoS attack, allowing you to filter out junk requests. What is included in these checks? Does the client know how to perform HTTP Redirect, does it support JavaScript, is it the browser it claims to be (because JavaScript is different everywhere and if the client says that it is, say, Firefox, then we can check this). The verification is implemented using cookies using different methods:

  • "Set-Cookie" + redirect with 301 HTTP Location;
  • "Set-Cookie" + redirect using HTML meta refresh;
  • arbitrary template, and you can use JavaScript.

To avoid automatic parsing, the validation cookie can be encrypted with AES-128 and later decrypted on the JavaScript client side. The new version of the module has the ability to set cookies via Flash, which also allows you to effectively weed out bots (which Flash, as a rule, does not support), but, however, blocks access for many legitimate users (in fact, all mobile devices). It is noteworthy that it is extremely easy to start using testcookie-nginx. The developer, in particular, gives several clear examples of use (for different cases of attack) with config samples for nginx.

In addition to the advantages, testcookies also have disadvantages:

  • cuts all bots, including Googlebot. If you plan to leave the testcookie on a permanent basis, make sure that you do not disappear from the search results;
  • creates problems for users with browsers Links, w3m and the like;
  • does not save from bots equipped with a full-fledged browser engine with JavaScript.

In a word, testcookie_module is not universal. But from a number of things, such as, for example, primitive toolkits in Java and C #, it helps. Thus, you cut off part of the threat.

4. Code 444

DDoSers often target the most resource-intensive part of a site. A typical example is a search that performs complex database queries. Naturally, attackers can take advantage of this by loading several tens of thousands of requests to the search engine at once. What we can do? Temporarily disable search. While customers won't be able to search for the information they need with built-in tools, the entire main site will remain up and running until you find the root of all problems. Nginx supports the non-standard 444 code, which allows you to simply close the connection and return nothing in response:

Location /search ( return 444; )

Thus, it is possible, for example, to quickly implement filtering by URL. If you are sure that requests to location /search are only coming from bots (for example, your confidence is based on the fact that your site does not have a /search section at all), you can install the ipset package on the server and ban bots with a simple shell script:

Ipset -N ban iphash tail -f access.log | while read LINE; do echo "$LINE" | \ cut -d""" -f3 | cut -d" " -f2 | grep -q 444 && ipset -A ban "$(L%% *)"; done

If the format of the log files is non-standard (not combined) or you need to ban on other grounds than the status of the response, you may need to replace cut with a regular expression.

5. Geo-banim

The non-standard response code 444 can also be useful for quickly banning clients based on geo-reference. You can severely restrict individual countries that you feel uncomfortable from. For example, it is unlikely that an online camera store from Rostov-on-Don has many users in Egypt. This is not a very good way (to put it bluntly - disgusting), because the GeoIP data is inaccurate, and Rostovites sometimes fly to Egypt on vacation. But if you have nothing to lose, then follow the instructions:

  1. Connect the GeoIP module to nginx (wiki.nginx.org/HttpGeoipModule).
  2. Display georeference information in the access log.
  3. Next, after modifying the above shell script, grep nginx's accesslog and add geographically kicked clients to the ban.

If, for example, the bots were mostly from China, then this might help.

6. Neural network (PoC)

Finally, you can repeat the experience of the user @SaveTheRbtz, who took the PyBrain neural network, stuffed the log into it and analyzed the requests (habrahabr.ru/post/136237). The method is working, although not universal :). But if you really know the insides of your site - and you, as a system administrator, should - then chances are that in the most tragic situations, such a tool based on neural networks, training and information gathered in advance will help you. In this case, it is very useful to have access.log before the start of DDoS, since it describes almost 100% of legitimate clients, and therefore, an excellent dataset for training a neural network. Moreover, bots are not always visible in the log.

Problem Diagnosis

The site is not working - why? Is it DDoSed or is it an engine bug not noticed by the programmer? Doesn't matter. Don't look for an answer to this question. If you think that your site may be attacked, contact companies that provide protection against attacks - a number of anti-DDoS services offer free first day after connection - and do not waste any more time looking for symptoms. Focus on the problem. If the site is running slowly or not opening at all, then something is wrong with its performance, and - regardless of whether there is a DDoS attack or not - you, as a professional, are obliged to understand what is causing this. We have repeatedly witnessed how a company experiencing difficulties with the operation of its site due to a DDoS attack, instead of looking for weaknesses in the site engine, tried to send applications to the Ministry of Internal Affairs in order to find and punish the attackers. Don't make such mistakes. The search for cybercriminals is a difficult and lengthy process, complicated by the very structure and principles of the Internet, and the problem with the operation of the site must be resolved promptly. Get the technicians to find out what is causing the site performance drop, and the lawyers can write the complaint.

7. Use a profiler and debugger

For the most common website building platform - PHP + MySQL - the bottleneck can be found using the following tools:

  • the Xdebug profiler will show which calls the application spends the most time on;
  • the built-in APD debugger and debug output to the error log will help you find out exactly which code makes these calls;
  • in most cases, the dog is buried in the complexity and heaviness of database queries. The explain SQL directive built into the database engine will help here.

If the site is lying on its back and you do not lose anything, disconnect from the network, look at the logs, try to play them. If it doesn't, then go through the pages, look at the base.

The example is for PHP, but the idea is valid for any platform. A developer writing software products in any programming language must be able to quickly use both the debugger and the profiler. Practice ahead of time!

8. Analyze mistakes

Analyze the amount of traffic, server response time, number of errors. See logs for this. In nginx, the server response time is recorded in the log by two variables: request_time and upstream_response_time. The first is the total execution time of the request, including network delays between the user and the server; the second tells how long the backend (Apache, php_fpm, uwsgi...) has been executing the request. The upstream_response_time value is extremely important for sites with a lot of dynamic content and active communication between the frontend and the database, and should not be neglected. You can use the following config as the log format:

Log_format xakep_log "$remote_addr - $remote_user [$time_local] " ""$request" $status $body_bytes_sent " ""$http_referer" "$http_user_agent" $request_time \ $upstream_response_time";

This is a combined format with added timing fields.

9. Track requests per second

Also look at the number of requests per second. In the case of nginx, you can roughly estimate this value with the following shell command (the ACCESS_LOG variable contains the path to the nginx request log in combined format):

Echo $(($(fgrep -c "$(env LC_ALL=C date [email protected]$(($(date \ +%s)-60)) +%d/%b/%Y:%H:%M)" "$ACCESS_LOG")/60))

Compared to the normal level for this time of day, the number of requests per second can both fall and grow. They grow if a large botnet arrives, and fall if the incoming botnet crashes the site, making it completely inaccessible to legitimate users, and at the same time, the botnet does not request statics, but legitimate users do. The drop in the number of requests is observed precisely due to statics. But, one way or another, we are talking about serious changes in indicators. When this happens suddenly - while you are trying to solve the problem on your own and if you do not see it right away in the log, it is better to quickly check the engine and contact specialists in parallel.

10. Don't Forget About tcpdump

Many people forget that tcpdump is an awesome diagnostic tool. I will give a couple of examples. In December 2011, a bug was discovered in the Linux kernel when it opened a TCP connection when the SYN and RST TCP segment flags were set. The first bug report was sent by a system administrator from Russia, whose resource was attacked by this method - the attackers found out about the vulnerability earlier than the whole world. Obviously, such a diagnosis helped him. Another example: nginx has one not very nice property - it writes to the log only after the request has been completely processed. There are situations when the site is down, nothing works and there is nothing in the logs. This is because all the requests that are currently loading the server have not yet completed. Tcpdump will help here too.

It is so good that I advised people not to use binary protocols until they are sure that everything is in order, because text protocols are easy to debug with tcpdump, but binary protocols are not. However, the sniffer is good as a diagnostic tool - as a means of maintaining production"and he's scary. It can easily lose multiple packages at once and mess up your user history. It is convenient to watch its output, and it will be useful for manual diagnostics and a ban, but try not to base anything critical on it. Another favorite tool for "warming requests" - ngrep - generally by default tries to request around two gigabytes of non-swapable memory and only then begins to reduce its requirements.

11. Attack or not?

How to distinguish a DDoS attack, for example, from the effect of an advertising campaign? This question may seem ridiculous, but this topic is no less complicated. There are some pretty funny cases. For some good guys, when they tensed up and thoroughly screwed up caching, the site fell ill for a couple of days. It turned out that for several months this site had been imperceptibly datamined by some Germans, and before optimizing the caching of the site page for these Germans, with all the pictures, it took quite a long time to load. When the page began to be issued from the cache instantly, the bot, which did not have any timeouts, also began to collect them instantly. It was hard. The case is especially difficult for the reason that if you yourself changed the setting (turned on caching) and the site stopped working after that, then who, in your opinion and the boss's opinion, is to blame? Exactly. If you see a sharp increase in the number of requests, then look, for example, in Google Analytics, who came to which pages.

Web server tuning

What are the other key points? Of course, you can set the "default" nginx and hope that everything will be fine for you. However, good things don't always happen. Therefore, the administrator of any server must devote a lot of time to fine-tuning and tuning nginx.

12. Limit resources (buffer sizes) in nginx

What should be remembered first of all? Each resource has a limit. First of all, it concerns RAM. Therefore, the sizes of headers and all used buffers should be limited to adequate values ​​per client and per server as a whole. They must be registered in the nginx config.

  • client_header_buffer_size_ _ Sets the buffer size for reading the client request header. If the request string or request header field does not fit entirely into this buffer, then larger buffers are allocated, specified by the large_client_header_buffers directive.
  • large_client_header_buffers Specifies the maximum number and size of buffers to read the large client request header.
  • client_body_buffer_size Sets the buffer size for reading the client request body. If the request body is larger than the specified buffer, then the entire request body or only part of it is written to a temporary file.
  • client_max_body_size Sets the maximum size allowed for a client request body, specified in the Content-Length field of the request header. If the size is larger than the specified one, then the client returns error 413 (Request Entity Too Large).

13. Set up timeouts in nginx

Time is also a resource. Therefore, the next important step should be to set all the timeouts, which again, it is very important to carefully register in the nginx settings.

  • reset_timedout_connection on; Helps deal with sockets stuck in the FIN-WAIT phase.
  • client_header_timeout Specifies a timeout when reading a client request header.
  • client_body_timeout Specifies a timeout when reading the body of a client request.
  • keepalive_timeout Sets the timeout during which the keep-alive connection with the client will not be closed by the server. Many are afraid to set large values ​​here, but we are not sure that this fear is justified. You can optionally set a timeout value in the Keep-Alive HTTP header, but Internet Explorer is notorious for ignoring this value.
  • send_timeout Specifies a timeout when sending a response to the client. If the client does not receive anything after this time, the connection will be closed.

Immediately the question is: what parameters of buffers and timeouts are correct? There is no universal recipe here, each situation has its own. But there is a proven approach. It is necessary to set the minimum values ​​at which the site remains operational (in peacetime), that is, pages are returned and requests are processed. This is determined only by testing - both from desktops and from mobile devices. The algorithm for finding the values ​​of each parameter (buffer size or timeout):

  1. We set the mathematical minimum value of the parameter.
  2. We start running site tests.
  3. If all the functionality of the site works without problems, the parameter is defined. If not, increase the value of the parameter and go to step 2.
  4. If the value of the parameter exceeded even the default value, this is a reason for discussion in the development team.

In some cases, the revision of these parameters should lead to refactoring / redesign of the site. For example, if the site does not work without three-minute AJAX long polling requests, then you do not need to increase the timeout, but replace long polling with something else - a botnet of 20 thousand machines, hanging on requests for three minutes, will easily kill an average cheap server.

14. Limit connections in nginx (limit_conn and limit_req)

Nginx also has the ability to limit connections, requests, and so on. If you're not sure how a certain part of your site will behave, then ideally you should test it, figure out how many requests it can handle, and write that into your nginx configuration. It's one thing when the site is down and you are able to come and pick it up. And it's a completely different matter - when it lay down to such an extent that the server went into swap. In this case, it is often easier to reboot than to wait for his triumphant return.

Let's assume that the site has sections with the telling names /download and /search. In doing so, we:

  • we don't want bots (or people with overzealous recursive download managers) to fill up our TCP connection table with their downloads;
  • we do not want bots (or stray search engine crawlers) to exhaust the computing resources of the DBMS with many search queries.

For these purposes, the following configuration will fit:

Http ( limit_conn_zone $binary_remote_addr zone=download_c:10m; limit_req_zone $binary_remote_addr zone=search_r:10m \ rate=1r/s; server ( location /download/ ( limit_conn download_c 1; # Other configuration location ) location /search/ ( limit_req zone= search_r burst=5; # Other configuration location ) ) )

It usually makes sense to set limits limit_conn and limit_req for locations where scripts that are expensive to execute are located (search is indicated in the example, and this is no accident). Constraints must be chosen based on the results of load and regression testing, as well as common sense.

Notice the 10m parameter in the example. It means that a dictionary with a buffer of 10 megabytes and not a megabyte more will be allocated for the calculation of this limit. In this configuration, this will allow 320,000 TCP sessions to be monitored. To optimize memory usage, the $binary_remote_addr variable is used as a key in the dictionary, which contains the user's IP address in binary form and takes up less memory than the regular $remote_addr string variable. It should be noted that the second parameter to the limit_req_zone directive can be not only IP, but also any other nginx variable available in this context - for example, in the case when you do not want to provide a more forgiving mode for the proxy, you can use $binary_remote_addr$http_user_agent or $binary_remote_addr$http_cookie_myc00kiez - but such constructions should be used with caution, because, unlike the 32-bit $binary_remote_addr, these variables can be much longer and the "10m" you declared may suddenly end.

Trends in DDoS

  1. The power of network and transport layer attacks is constantly growing. The potential of an average SYN flood attack has already reached 10 million packets per second.
  2. Attacks on the DNS have been in particular demand lately. UDP flooding with valid DNS requests with spoofed source IP addresses is one of the easiest attacks to implement and difficult to counter. Many large Russian companies (including hosting companies) have recently experienced problems as a result of attacks on their DNS servers. The further, the more such attacks will be, and their power will grow.
  3. Judging by external signs, most botnets are not controlled centrally, but through a peer-to-peer network. This gives attackers the opportunity to synchronize the actions of the botnet in time - if earlier control commands were distributed over a botnet of 5,000 machines in tens of minutes, now seconds count, and your site may suddenly experience an instant hundredfold increase in the number of requests.
  4. The share of bots equipped with a full-fledged browser engine with JavaScript is still small, but it is constantly growing. Such an attack is more difficult to repulse with built-in improvised means, so DIYers should watch this trend with caution.

preparing the OS

In addition to fine-tuning nginx, you need to take care of the settings for the network stack of the system. At the very least, immediately include net.ipv4.tcp_syncookies in sysctl to protect yourself from a small SYN flood attack at once.

15. Tune the core

Pay attention to the more advanced settings of the network part (kernel), again in terms of timeouts and memory. There are more important ones and less important ones. First of all, you need to pay attention to:

  • net.ipv4.tcp_fin_timeout The time the socket will spend in TCP phase FIN-WAIT-2 (waiting for a FIN/ACK segment).
  • net.ipv4.tcp_(,r,w)mem TCP socket receive buffer size. Three values: minimum, default value and maximum.
  • net.core.(r,w)mem_max Same for non-TCP buffers.

With a link of 100 Mbps, the default values ​​are still somehow suitable; but if you have at least gigabits per second available, then it's better to use something like:

sysctl -w net.core.rmem_max=8388608 sysctl -w net.core.wmem_max=8388608 sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608" sysctl -w net.ipv4.tcp_wmem="4096 65536 8388608" sysctl - w net.ipv4.tcp_fin_timeout=10

16. Revision /proc/sys/net/**

It is ideal to study all parameters /proc/sys/net/**. We need to see how they differ from the default ones, and understand how adequately they are exposed. A Linux developer (or system administrator) who understands the operation of the Internet service under his control and wants to optimize it should read with interest the documentation of all parameters of the kernel network stack. Perhaps he will find variables specific to his site there that will help not only protect the site from intruders, but also speed up its work.

Do not be afraid!

Successful DDoS attacks day by day extinguish e-commerce, shake the media, knock out the largest payment systems with one blow. Millions of Internet users are losing access to critical information. The threat is urgent, so you need to meet it fully armed. Do your homework, don't be afraid and keep your head cool. You are not the first nor the last to face a DDoS attack on your site, and it is up to you, guided by your knowledge and common sense, to minimize the consequences of the attack.

On a computing system in order to bring it to failure, that is, the creation of such conditions under which legal (lawful) users of the system cannot access the resources (servers) provided by the system, or this access is difficult. The failure of the “enemy” system can also be a step towards mastering the system (if in an emergency the software gives out any critical information - for example, version, part of the program code, etc.). But more often it is a measure of economic pressure: downtime of a revenue-generating service, bills from the provider, and measures to avoid an attack significantly hit the “target” in the pocket.

If an attack is carried out simultaneously from a large number of computers, one speaks of DDoS attack(from English. Distributed Denial of Service, distributed denial of service attack). In some cases, an unintended action leads to an actual DDoS attack, for example, placing a link on a popular Internet resource to a site hosted on a not very productive server (slash dot effect). A large influx of users leads to exceeding the allowable load on the server and, consequently, a denial of service for some of them.

Types of DoS attacks

There are various reasons why a DoS condition may occur:

  • Error in program code, resulting in access to an unused fragment of the address space, execution of an invalid instruction, or other unhandled exception when the server program crashes - the server program. A classic example is zero-based referencing. null) address.
  • Insufficient validation of user data, leading to an infinite or long cycle or increased long-term consumption of processor resources (up to the exhaustion of processor resources) or the allocation of a large amount of RAM (up to the exhaustion of available memory).
  • flood(English) flood- "flood", "overflow") - an attack associated with a large number of usually meaningless or incorrectly formatted requests to a computer system or network equipment, which has as its goal or led to a system failure due to the exhaustion of system resources - the processor, memory or communication channels.
  • Attack of the second kind- an attack that seeks to cause a false alarm of the protection system and thus lead to the unavailability of the resource.

If an attack (usually a flood) is carried out simultaneously from a large number of IP addresses - from several computers dispersed in the network - then in this case it is called distributed denial of service attack ( DDoS).

Exploitation of bugs

Exploit refers to a program, a piece of program code, or a sequence of program commands that exploit vulnerabilities in software and are used to attack a cyber system. Of the exploits that lead to a DoS attack, but are unsuitable, for example, to seize control of an "enemy" system, the most famous are WinNuke and Ping of death (Ping of death).

flood

For flooding as a violation of netiquette, see flooding.

flood they call a huge stream of meaningless requests from different computers in order to take the "enemy" system (processor, RAM or communication channel) with work and thereby temporarily disable it. The concept of “DDoS attack” is almost equivalent to the concept of “flood”, and in everyday life both of them are often interchangeable (“flood the server” = “DDoS’it the server”).

To create a flood, both ordinary network utilities like ping (this is known, for example, the Internet community " Upyachka"), and special programs can be used. The possibility of DDoS is often "sewn up" in botnets. If a cross-site scripting vulnerability or the ability to include images from other resources is found on a site with high traffic, this site can also be used for a DDoS attack.

Communication channel and TCP subsystem flood

Any computer that communicates with the outside world via the TCP / IP protocol is subject to these types of floods:

  • SYN flood - with this type of flood attack, a large number of SYN packets are sent to the attacked node via the TCP protocol (requests to open a connection). At the same time, after a short time, the number of sockets available for opening (software network sockets, ports) is exhausted on the attacked computer, and the server stops responding.
  • UDP flood - this type of flood does not attack the target computer, but its communication channel. Providers reasonably assume that UDP packets should be delivered first, while TCP can wait. A large number of UDP packets of different sizes clog the communication channel, and the server running over the TCP protocol stops responding.
  • ICMP flood - the same thing, but with the help of ICMP packets.

Application layer flood

Many services are designed in such a way that a small request can cause a large consumption of computing power on the server. In this case, it is not the communication channel or the TCP subsystem that is attacked, but the service (service) itself - a flood of such "sick" requests. For example, web servers are vulnerable to HTTP flooding - either a simple GET / or a complex database query like GET /index.php?search= can be used to disable a web server<случайная строка> .

DoS attack detection

There is an opinion that special tools for detecting DoS attacks are not required, since the fact of a DoS attack cannot be overlooked. In many cases this is true. However, successful DoS attacks were observed quite often, which were noticed by the victims only after 2-3 days. It happened that the negative consequences of an attack ( flood-attacks) resulted in excessive costs for paying for excess Internet traffic, which was found out only when receiving an invoice from an Internet provider. In addition, many intrusion detection methods are ineffective near the target of attack, but are effective on network backbones. In this case, it is advisable to install detection systems exactly there, and not wait until the user who has been attacked notices it himself and seeks help. In addition, in order to effectively counteract DoS attacks, it is necessary to know the type, nature, and other characteristics of DoS attacks, and detection systems make it possible to quickly obtain this information.

DoS attack detection methods can be divided into several large groups:

  • signature - based on a qualitative analysis of traffic.
  • statistical - based on a quantitative analysis of traffic.
  • hybrid (combined) - combining the advantages of both of the above methods.

DoS protection

Measures to counter DoS attacks can be divided into passive and active, as well as preventive and reactive.

Below is a brief list of the main methods.

  • Prevention. Prevention of the reasons that prompt certain individuals to organize and undertake DoS attacks. (Very often, cyberattacks in general are the result of personal grievances, political, religious and other disagreements, provocative behavior of the victim, etc.)
  • Filtering and blackholing. Blocking traffic from attacking machines. The effectiveness of these methods decreases as you get closer to the object of attack and increases as you get closer to the attacking machine.
  • Reverse DDOS- redirecting the traffic used for the attack to the attacker.
  • Elimination of vulnerabilities. Doesn't work against flood-attacks for which "vulnerability" is the finiteness of certain system resources.
  • Increasing resources. Naturally, it does not provide absolute protection, but it is a good background for applying other types of protection against DoS attacks.
  • Dispersal. Building distributed and duplicating systems that will not stop serving users, even if some of their elements become unavailable due to a DoS attack.
  • Evasion. Moving the immediate target of the attack (domain name or IP address) away from other resources that are often also affected along with the immediate target of the attack.
  • Active response. Impact on the sources, the organizer or the control center of the attack, both by man-made and organizational and legal means.
  • Using equipment to repel DoS attacks. For example DefensePro® (Radware), Perimeter (MFI Soft), Arbor Peakflow® and other manufacturers.
  • Acquisition of a service to protect against DoS attacks. Actual in case of exceeding the bandwidth of the network channel by the flood.

see also

Notes

Literature

  • Chris Kaspersky Computer viruses inside and out. - Peter. - St. Petersburg. : Peter, 2006. - S. 527. - ISBN 5-469-00982-3
  • Stephen Northcutt, Mark Cooper, Matt Fearnow, Karen Frederik. Analysis of typical security breaches in networks = Intrusion Signatures and Analysis. - New Riders Publishing (English) St. Petersburg: Williams Publishing House (Russian), 2001. - P. 464. - ISBN 5-8459-0225-8 (Russian), 0-7357-1063-5 ( English)
  • Morris, R.T.= A Weakness in the 4.2BSD Unix TCP/IP Software. - Computing Science Technical Report No.117. - AT&T Bell Laboratories, Feb 1985.
  • Bellovin, S.M.= Security Problems in the TCP/IP protocol Suite. - Computer Communication Review, Vol. 19, No.2. - AT&T Bell Laboratories, April 1989.
  • = daemon9 / route / infinity "IP-spooling Demystified: Trust Realationship Exploitation". - Phrack Magazine, Vol.7, Issue 48. - Guild Production, July 1996.
  • = daemon9 / route / infinity "Project Neptune". - Phrack Magazine, Vol.7, Issue 48. - Guild Production, July 1996.

Links

  • DoS attack in the Open Directory Project Link Directory (

DDOS attack. Explanation and example.

Hi all. This is the Computer76 blog, and now another article about the basics of hacker art. Today we will talk about what a DDOS attack is in simple words and examples. Before throwing in technical terms, there will be an introduction that everyone can understand.

Why is a DDOS attack used?

WiFi hacking is used to collect the password of a wireless network. Attacks in the form of “ ” will allow you to listen to Internet traffic. Vulnerability analysis with the subsequent loading of a specific one makes it possible to capture the target computer. What does a DDOS attack do? Ultimately, its goal is to take ownership of the resource from its rightful owner. I do not mean that the site or blog will not belong to you. This is in the sense that in the event of a successful attack on your site, you lose control of it. At least for a while.

However, in the modern interpretation of DDOS, an attack is most often used to disrupt the normal operation of any service. Hacker groups, the names of which are constantly heard, attack major government or government sites in order to draw attention to certain problems. But almost always behind such attacks there is a purely mercantile interest: the work of competitors or simple pranks with completely indecently unprotected sites. The main concept of DDOS is that a huge number of users, or rather requests from bot computers, access the site at the same time, which makes the load on the server unbearable. We often hear the expression “the site is not available”, but few people think about what is actually hidden behind this wording. Well, now you know.

DDOS attack - options

Option 1.

players crowded at the entrance

Imagine that you are playing an online multiplayer game. Thousands of players are playing with you. And you are familiar with most of them. You negotiate the details and do the following at hour X. You all go to the site at the same time and create a character with the same set of characteristics. Group in one place, blocking access to objects in the game by the number of simultaneously created characters for other conscientious users who do not suspect anything about your collusion.

Option 2.


Imagine that someone decides to disrupt the bus service in the city along a certain route in order to prevent conscientious passengers from using public transport services. Thousands of your friends at the same time go to stops at the beginning of the specified route and aimlessly ride in all cars from end to end until the money runs out. The trip is paid, but no one gets off at any stop other than the final destinations. And other passengers, standing at intermediate stops, look sadly at the receding minibuses, unable to push their way into the crowded buses. Everything is in a burnout: both taxi owners and potential passengers.

In reality, these options cannot be physically implemented. However, in the virtual world, your friends can be replaced by computers of unscrupulous users who do not bother to somehow protect their computer or laptop. And the vast majority are. There are many programs for carrying out a DDOS attack. Needless to say, such actions are illegal. And a ridiculously prepared DDOS attack, no matter how successful it is, is detected and punished.

How is a DDOS attack carried out?

By clicking on a website link, your browser sends a request to the server to display the page you are looking for. This request is expressed as a data packet. And not even one, but a whole package of packages! In any case, the amount of data transmitted per channel is always limited to a certain width. And the amount of data returned by the server is disproportionately larger than what is contained in your request. It takes the server effort and resources. The stronger the server, the more expensive it is for the owner and the more expensive the services it provides. Modern servers can easily cope with a sharply increased influx of visitors. But for any of the servers, there is still a critical value of users who want to get acquainted with the content of the site. The clearer is the situation with the server that provides website hosting services. Just a little, and the victim site is disconnected from service, so as not to overload the processors that serve thousands of other sites located on the same hosting. The work of the site stops until the moment when the DDOS attack itself stops. Well, imagine that you start reloading any of the site's pages a thousand times per second (DOS). And thousands of your friends are doing the same thing on their computers (distributed DOS or DDOS)... Large servers have learned to recognize that a DDOS attack has begun and counteract it. However, hackers are also improving their approaches. So, within the framework of this article, I can’t explain what a DDOS attack is in more detail.

What is a DDOS attack you can find out and try right now.

ATTENTION. If you decide to try, all unsaved data will be lost, you will need a button to return the computer to a working state. RESET. But you will be able to find out exactly what the server that was attacked “feels”. A detailed example is a paragraph below, and now - simple commands to reboot the system.

  • For Linux, in the terminal, type the command:
:(){ :|:& };:

The system will refuse to work.

  • For Windows, I suggest creating a bat file in Notepad with the code:
:1 Start goto 1

Name the type DDOS.bat

To explain the meaning of both commands, I think, is not worth it. Everything is visible to the naked eye. Both commands force the system to execute the script and immediately repeat it, sending it to the beginning of the script. Given the speed of execution, the system falls into a stupor after a couple of seconds. Game, as they say, over.

DDOS attack using software.

For a more visual example, use the Low Orbit Ion Cannon program (Ion cannon from low orbit). Or LOIC. The most downloaded distribution is located at (we work on Windows):

https://sourceforge.net/projects/loic/

ATTENTION ! Your antivirus should react to the file as malicious. It's okay: you already know what you're downloading. In the signature database, it is marked as a flood generator - translated into Russian, this is the ultimate goal of endless calls to a specific network address. I PERSONALLY did not notice any viruses or trojans. But you have the right to hesitate and postpone the download.

Since careless users bombard the resource with messages about a malicious file, Source Forge will take you to the following page with a direct link to the file:

As a result, I managed to download the utility only through .

The program window looks like this:

Point 1 Select target will allow an attacker to focus on a specific target (an IP address or site url is entered), point 3 Attack options will allow you to select the attacked port, protocol ( method) of the three TCP, UDP and HTTP. In the TCP/UDP message field, you can enter a message for the victim. After done, the attack starts at the press of a button IMMA CHARGIN MAH LAZER(this is a phrase on the verge of a foul from the once popular comicmeme; By the way, there are a lot of American swear words in the program). All.

WARNING

This option is for testing only for localhost. That's why:

  • against other people's sites it is illegal, and for this they are already really sitting in the West (which means they will soon be planted here too)
  • the address from which the flood is coming will be calculated quickly, they will complain to the provider, and he will give you a warning and remind you of the first point
  • in networks with a low throughput channel (that is, in all home networks), the little thing will not work. With the TOR network, everything is the same.
  • if you set it up properly, you will quickly clog YOUR communication channel than harm someone. So this is exactly the option when the pear beats the boxer, and not vice versa. And the option with a proxy will follow the same principle: no one will like the flood on your part.

Read: 9 326

Increasingly, in the official messages of hosting providers, here and there, mentions of reflected DDoS attacks flicker. Increasingly, users, having discovered the unavailability of their site, immediately assume DDoS. Indeed, at the beginning of March, the Runet experienced a whole wave of such attacks. At the same time, experts assure that the fun is just beginning. It is simply impossible to ignore a phenomenon so relevant, formidable and intriguing. So today let's talk about the myths and facts about DDoS. From the point of view of the hosting provider, of course.

memorial day

On November 20, 2013, for the first time in the 8-year history of our company, the entire technical site was unavailable for several hours due to an unprecedented DDoS attack. Tens of thousands of our customers throughout Russia and the CIS have suffered, not to mention ourselves and our Internet provider. The last thing the provider managed to fix before the white light faded for everyone was that its input channels were tightly clogged with incoming traffic. To visualize this, imagine your bathtub with an ordinary sink, into which Niagara Falls rushed.

Even upstream providers have felt the echoes of this tsunami. The graphs below clearly illustrate what happened that day with Internet traffic in St. Petersburg and in Russia. Pay attention to the steep peaks at 15:00 and 18:00, just when we recorded the attacks. On these sudden plus 500-700 GB.

It took several hours to localize the attack. The server to which it was sent was calculated. Then the purpose of Internet terrorists was calculated. Do you know who all this enemy artillery hit? One very ordinary, modest client site each.

Myth number one: “The object of attack is always the hosting provider. These are the machinations of his competitors. Not mine." In fact, the most likely target for Internet terrorists is a regular client site. That is, the site of one of your hosting neighbors. And maybe yours too.

Not all DDoS...

After the events on our technical site on November 20, 2013 and their partial repetition on January 9, 2014, some users began to assume DDoS in any particular failure of their own site: “This is DDoS!” and “Are you having DDoS again?”

It is important to remember that if we suffer such a DDoS that even customers feel it, we immediately report it ourselves.

We want to reassure those who are in a hurry to panic: if something is wrong with your site, then the probability that this is DDoS is less than 1%. Simply due to the fact that a lot of things can happen to the site and this “lot of things” happens much more often. We will talk about methods of self-quick diagnostics of what exactly is happening with your site in one of the following posts.

In the meantime - for the sake of accuracy of word usage - let's clarify the terms.

About terms

DoS attack (from English Denial of Service) - this is an attack designed to cause a server to be denied service due to its overload.

DoS attacks are not related to equipment damage or information theft; their goal - make the server stop responding. The fundamental difference between DoS is that the attack occurs from one machine to another. There are exactly two participants.

But in reality, we practically do not observe DoS attacks. Why? Because the objects of attacks are most often industrial objects (for example, powerful productive servers of hosting companies). And in order to cause any noticeable harm to the operation of such a machine, much more power is needed than its own. This is first. And secondly, the initiator of a DoS attack is quite easy to calculate.

DDoS - essentially the same as DoS, only the attack is distributed nature. Not five, not ten, not twenty, but hundreds and thousands of computers access one server simultaneously from different places. Such an army of machines is called botnet. It is almost impossible to calculate the customer and the organizer.

accomplices

What kind of computers are included in the botnet?

You will be surprised, but often these are the most ordinary home machines. Who knows?.. - quite possibly your home computer drawn to the side of evil.

It takes a little for this. An attacker finds a vulnerability in a popular operating system or application and uses it to infect your computer with a Trojan that, on a certain day and hour, instructs your computer to start performing certain actions. For example, send requests to a specific IP. Without your knowledge and participation, of course.

Myth number two: « DDoS is being done somewhere far away from me, in a special underground bunker where bearded hackers with red eyes sit. In fact, without knowing it, you, your friends and neighbors - anyone can be an unwitting accomplice.

This is really happening. Even if you don't think about it. Even if you are terribly far from IT (especially if you are far from IT!).

Entertaining hacking or DDoS mechanics

The DDoS phenomenon is heterogeneous. This concept combines many options for actions that lead to one result (denial of service). Consider the options for the troubles that DDoSers can bring us.

Overconsumption of server computing resources

This is done by sending packets to a specific IP, the processing of which requires a large amount of resources. For example, to load a page, you need to execute a large number of SQL queries. All attackers will request this particular page, which will cause a server overload and a denial of service for normal, legitimate site visitors.
This is an attack of the level of a schoolboy who has devoted a couple of evenings to reading the Hacker magazine. She is not a problem. The same requested URL is calculated instantly, after which access to it is blocked at the web server level. And this is just one of the solutions.

Overload of communication channels to the server (to the exit)

The difficulty level of this attack is about the same as the previous one. The attacker calculates the heaviest page on the site, and the botnet under his control begins to request it en masse.


Imagine that the part of Winnie the Pooh invisible to us is infinitely large
In this case, it is also very easy to understand what exactly is clogging the outgoing channel, and prohibit access to this page. Requests of the same type are easy to see with the help of special utilities that allow you to look at the network interface and analyze traffic. Then a rule is written for the Firewall, which blocks such requests. All this is done regularly, automatically and so lightning fast that most users are unaware of any attack.

Myth number three: "A However, they rarely visit my hosting often, and I always notice them.” In fact, 99.9% of attacks you don't see or feel. But the daily struggle with them - this is the everyday, routine work of a hosting company. This is our reality, in which the attack is cheap, the competition is off the charts, and not everyone demonstrates legibility in the methods of fighting for a place in the sun.

Overload of communication channels to the server (at the entrance)

This is already a puzzle for those who have been reading the Hacker magazine for more than one day.


Photo from the website of the radio "Echo of Moscow". They did not find anything more illustrative to represent DDoS with channel overload at the entrance.
To fill the channel with incoming traffic to capacity, you need to have a botnet, the power of which allows you to generate the required amount of traffic. But maybe there is a way to give away little traffic and get a lot?

There is, and not one. There are many options for enhancing the attack, but one of the most popular right now is attack through public DNS servers. Experts call this method of amplification DNS amplification(in case someone prefers expert terms). And if it's simpler, then imagine an avalanche: a small effort is enough to break it, and inhuman resources to stop it.

You and I know that public DNS server on request, informs anyone who wants data about any domain name. For example, we ask such a server: tell me about the sprinthost.ru domain. And he, without hesitation, throws out everything he knows to us.

Querying a DNS server is a very simple operation. It costs almost nothing to contact him, the request will be microscopic. For example, like this:

It remains only to choose a domain name, information about which will be an impressive data package. So the original 35 bytes with a slight movement of the hand turn into almost 3700. There is an increase of more than 10 times.

But how to make sure that the response is directed to the correct IP? How to forge the source IP of the request so that the DNS server issues its answers in the direction of the victim, who did not request any data?

The fact is that DNS servers work on UDP communication protocol, which doesn't need confirmation of the origin of the request at all. Forging an outgoing IP in this case is not a big deal for a doser. That's why this type of attack is so popular right now.

Most importantly, a very small botnet is enough to implement such an attack. And several disparate public DNS, which will not see anything strange in the fact that different users from time to time request data from the address of one host. And only then all this traffic will merge into one stream and nail one “pipe” tightly.

What the doser cannot know is the capacity of the attacker's channels. And if he does not calculate the power of his attack correctly and does not immediately block the channel to the server by 100%, the attack can be quickly and easily repelled. Using utilities like TCPdump it is easy to find out that incoming traffic comes from DNS, and at the Firewall level to prohibit it from being accepted. This option - refusing to accept traffic from DNS - is associated with a certain inconvenience for everyone, however, both the servers and the sites on them will continue to work successfully.

This is just one of many options to enhance the attack. There are many other types of attacks, we can talk about them another time. In the meantime, I would like to summarize that all of the above is true for an attack whose power does not exceed the width of the channel to the server.

If the attack is strong

If the attack power exceeds the capacity of the channel to the server, the following happens. The Internet channel is instantly clogged to the server, then to the hosting site, to its Internet provider, to the upstream provider, and so on and up in ascending order (in the future - to the most absurd limits), as far as the attack power is enough.

And then it becomes a global problem for everyone. And in short, this is what we had to deal with on November 20, 2013. And when large-scale upheavals occur, it's time to turn on special magic!


This is what special magic looks like. With the help of this magic, it is possible to calculate the server to which the traffic is directed and block its IP at the ISP level. So that it stops accepting any calls to this IP through its communication channels with the outside world (uplinks). To lovers of terms: experts call this procedure "black out", from English blackhole.

At the same time, the attacked server with 500-1500 accounts remains without its IP. A new subnet of IP addresses is allocated for it, over which client accounts are randomly distributed evenly. Further, experts are waiting for a repetition of the attack. It almost always repeats itself.

And when it repeats, there are no longer 500-1000 accounts on the attacked IP, but some dozen or two.

The circle of suspects is narrowing. These 10-20 accounts are again distributed to different IP addresses. Once again, the engineers lie in wait for another attack. Again and again they spread the accounts remaining under suspicion to different IPs and so, by gradual approximation, they calculate the object of attack. All other accounts at this point return to normal operation on the same IP.

As you know, this is not an instant procedure, it takes time to implement.

Myth number four:“When a massive attack happens, my hoster doesn’t have a plan of action. He just waits with his eyes closed for the end of the bombardment and answers my letters with the same type of replies.This is not the case: in the event of an attack, the hosting provider acts according to a plan in order to localize it and eliminate the consequences as soon as possible. And the same type of letters allow you to convey the essence of what is happening and at the same time save the resources necessary for the fastest possible processing of an emergency situation..

Is there light at the end of the tunnel?

Now we see that DDoS activity is constantly increasing. To order an attack has become very accessible and ugly inexpensive. To avoid accusations of propaganda, there will be no prooflinks. But take our word for it, it is.

Myth number five: “A DDoS attack is a very expensive event, and only business bigwigs can afford to order one. As a last resort, these are the machinations of the secret services!” In fact, such events have become extremely accessible.

Therefore, it is not necessary to expect that malicious activity will disappear by itself. Rather, it will only intensify. It remains only to forge and sharpen weapons. What we are doing, improving the network infrastructure.

The legal side of the issue

This is a very unpopular aspect of discussing DDoS attacks, as we rarely hear about cases of capture and punishment of the instigators. However, remember: A DDoS attack is a criminal offense. In most countries of the world, including Russia.

Myth number six: « Now I know enough about DDoS, I'll order a holiday for a competitor - And I won't get anything for it!" It is not excluded that it will. And if it does, it won't seem like much.

  • Tie history with DDoS payment system Assist
  • Exciting denouement

In general, we do not advise anyone to engage in the vicious practice of DDoS, so as not to incur the wrath of justice and not bend their own karma. And we, due to the specifics of our activity and a lively research interest, continue to study the problem, stand guard and improve defensive structures.

PS:we do not have enough kind words to express all our gratitude, so we just say"Thank you!" to our patient customers who have warmly supported us on a difficult day on November 20, 2013. You have said many words of encouragement in support of our

mob_info