Search for files on Linux, Freebsd by type, content, size and date

The Linux offers a number of tools that can be used on the command line. I have found the following useful

Find files with certain text

grep -r "Text to find" PATH
e.g. grep -r "bad text" /home/user/www/

If you just want the file names

grep -r "Text to find" PATH | cut -d: -f1
e.g.  grep -r "bad text" /home/user/www/ | cut -d: -f1

If you want to find certain files

find . -type f -name "101.php"
for a case insensitive search
find . -type f -iname "101.php"

Wildcards work too

find . -type f -name "*.php"
for a case insensitive search
find . -type f -iname "*.php"

Suppose you want to find all the files modified between a certain dates. This works

touch -t yyyymmddhhmm tempfile1
touch -t yyyymmddhhmm tempfile2
find /www/ -type f -newer tempfile1 -not -newer tempfile2

e.g.
touch -t 201510010000 /tmp/startfile
touch -t 201510010000 /tmp/endfile
find /www/ -type f -newer /tmp/startfile -not -newer /tmp/endfile

Find specific files, e.g. php files, between certain dates

touch -t yyyymmddhhmm tempfile1
touch -t yyyymmddhhmm tempfile2
find /www/ -type f -name "*.php" -newer tempfile1 -not -newer tempfile2

e.g.
touch -t 201510010000 /tmp/startfile
touch -t 201510010000 /tmp/endfile
find /www/ -type f -name "*.php"  -newer /tmp/startfile -not -newer /tmp/endfile

 References

• Mommy, I found it! — 15 Practical Linux Find Command Examples
• Daddy, I found it!, 15 Awesome Linux Find Command Examples (Part2)

Twitter Username/handle limited to 15 characters

Recently I discovered that Twitter username's have a 15 character limit. It just one of those things you did not know or note. It makes sense given that Twitter wants you to say what you have to in 140 characters. A reply to @twitterusername(a name that just made its 15 character limit) takes up 16 characters of the 140, the Twitter limit for posts.

I searched to confirm, and a writer at "The Curious Engineer" confirmed by my new finding.

Goes to the truth that you learn everyday.

Reference

• The 15 Character Limit Of Twitter Username

Centos7 changes from Centos 5 & 6

I have not yet started playing around with CENTOS 7. However,  I was scanning an article 
on Digital Ocean and saw this command "systemctl restart httpd" and it peaked my interest.

It seems in CENTOS 7 you there is a little change in the commands to restart services and how services are configured to start automatically or not on reboots.

So if you have a service called mybuzz instead of

• "service mybuzz restart" it is now "systemctl restart mybuzz"
• "service mybuzz restart|start|stop|status|reload|condestar|status" it is now "systemctl restart|start|stop|status|reload|condestar|status mybuzz" respectively

For chkconfig the changes are not that simple. The cheatsheet below will be useful

• "chkconfig mybuzz on" it is now "systemctl enable frobozz"
• "chkconfig mybuzz off" it is now "systemctl disable frobozz"
• "chkconfig mybuzz " it is now "systemctl is-enabled frobozz" (check if service is configured to start or not)

References

• SysVinit to Systemd Cheatsheet

Got fatal error 1236 from master when reading data from binary log

Today I encountered a MySQL replication error but thanks to Percona's Muhammad Irfan excellent blog the solution was quick and easy. While the error occurred with a MariaDB setup the MySQL instructions applied.

The error

Slave_IO_Running: No

Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position; the first event 'binlog.000202' at 307396531, the last event read from 'binlog.000202' at 4, the last byte read from 'binlog.000202' at 4.', Internal MariaDB error code: 1236 

It appear there was some issue on the server hosting the MariaDB and the server administrator opted to reboot. When the server restarted, replication stopped working.

Solution

On the Master
Firstly, on master, confirm that it is the end of the binary log. Browse to the directory

cd /var/lib/mysql (location will vary depending on your installation)

mysqlbinlog --base64-output=decode-rows --verbose --verbose --start-position=307396531 binlog.000202

I also peaked on the next log binlog.000203

mysqlbinlog --base64-output=decode-rows --verbose --verbose --start-position=0 binlog.000203

On the Slave

Stop the slave

All seems well, so I proceeded change the master log and master log position

CHANGE MASTER TO MASTER_LOG_FILE='binlog.000203', MASTER_LOG_POS=4;

Start the slave 

Once done, check the Slave Status and all should be well.

show slave status\G;

Please note that your binlog is e.g. mysql-bin.001 advance to the next log file mysql-bin-002

Why did this happen? According to Muhammad

I foresee master server crashed or rebooted and hence binary log events not synchronized on disk. This usually happens when sync_binlog != 1 on the master. You can investigate it as inspecting binary log contents as below:

user yogesh77 on the Percona Forum expanded this

"After sudden reboot mysql rolled back last transactions in binary logs however slave already incremented its binary position so after master is up slave is not able to get correct binary position. To resolve this issue you need to point slave to new binary file created after the server reboot and mysql restart. The same issue happened to be just 2 days back. After setting new binary position I skipped few entries on slave which were updated on slave and rolled back in binary position."

How to prevent
The suggestion is this little command.

On the master

SET GLOBAL sync_binlog=1;

Add to my.cnf or server.cnf for MariaDB to make permanent.

sync_binlog=1;

Also, try shutting down the server gracefully.

However there is a issue with sync_binlog. It appears enabling on certain file systems results in a significant performance hit. That discussion will have to be for another blog. Some discussions seem to indicate that it is getting better

n.b. If you have MYSQL replication set up, monitor it and have alerts emailed to you. Replication is a nice feature but must be monitored.

References

• MySQL Replication: ‘Got fatal error 1236′ causes and cures
• Last_IO_Error: Got fatal error 1236
• MySQL 5.6 Replication Performance
• MySQL sync_binlog performance and reliability
• Sync_binlog Options

PHP Tuning - realpath_cache_size integer and realpath_cache_ttl integer

Two interesting directives available since PHP 5.1 are

1. real_cache_size
2. real_cahe_ttl

The given definitions are as follows.

realpath_cache_size integer - Determines the size of the realpath cache to be used by PHP. This value should be increased on systems where PHP opens many files, to reflect the quantity of the file operations performed.
The size represents the total number of bytes in the path strings stored, plus the size of the data associated with the cache entry. This means that in order to store longer paths in the cache, the cache size must be larger. This value does not directly control the number of distinct paths that can be cached.
The size required for the cache entry data is system dependent.

realpath_cache_ttl integer - Duration of time (in seconds) for which to cache realpath information for a given file or directory. For systems with rarely changing files, consider increasing the value.

One user on a Reddit Thread  seems to suggest that on enabling he got 50-70% increase in performance. That is significant. What I am not sure is how it works with APC.


References

• PHP Manual  - realpath-cache-size
• Adding this to my Drupal 7 site's php.ini dramatically improved performance
• Improve the Performance and Scalability of Your Drupal Site

Using Varnish to block access to specific folders

If you ever need to block folder or folders using Varnish Cache, here are the simple steps.

Edit  sub vcl_recv and add the following

sub vcl_recv {
  # Ban outside access to #/user, /admin etc
  # works if you : if (req.url ~ "^/user" || req.url ~ "^/admin") {
  if ( (req.url ~ "^/user" || req.url ~ "^/admin" ) && !client.ip ~ yourallowedip) {
      # Have Varnish throw the error directly.
       error 405 "Sorry";
    }
#Other code
#....
}

Create...

acl yourallowedip {
    "1.1.1.1";
}

Restart varnish and you will be good to go.

service restart varnish

It is always good to test your configuration before restarting Varnish. The command to do is below. It there is an error it will let you know otherwise you will get a long display.

varnishd -C -f /etc/varnish/default.vcl

References

• Lockdown Wordpress
• Varnish Command Line tools
• Practical Guide to Varnish