File System Extension on Live Linux VMware Guest using vmdisk size extended

Many thanks to RAM for this Article.

---

This article explains, Filesystem extension on live Linux VMware Linux Guest where vmdisk size is extended and by not new disk added.

We had a scenario as follows:

1.       File system extension requirement on a live mounted file system without reboot.

2.       It’s a Linux guest on VMware required a FS extension from 600 GB to 900 GB. The FS was a single 600 GB disk  /dev/sdb

3.       While assigning storage, the team did increase the underlying disk to 900 GB than adding a new disk.

4.       Even after extension,  /dev/sdb was not picking up the additional 300 GB space. [ rescan or partprobe did not help here ]

Note: The case also applies for situations where you have the underlying partition has been changed ( using fdisk ).

Following are the steps taken to make the kernel recognize the new partition structure and to extend the filesystem

First we verified the disk sizes and allocations

# pvs
# vgs
# lvdisplay -m /dev/vg_name/lv_name  [ to get the underlying block devices ]
 

Now we had the partition table re-read for the underlying block device.

# blockdev --rereadpt /dev/sdb
OR# sfdisk –R /dev/sdb

Do note that if you are doing this on a physical machine where we have multipath involved, we would need to re-read the partition tables for all the underlying disks involved.

Now that we have the partition table re-read, we would need to make PV resized to the new disk. Else it would still show the old size.

# pvresize /dev/sdb

Check pvs / vgs output to see whether the new size is detected:

# pvs
# vgs

Once you have the new size detected, you can use the standard procedure to extend the filesystems

# lvextend -L +300G /dev/vg/lv
# resize2fs /dev/vg/lv

Check whether the new file systems are showing the correct sizes:

# df -h

Following are the screenshots of the entire activity which I performed in a test VM. A test VG and LV were created for this activity.

Verify current disk size of the mounted volume :

Check and verify on the available disk space on the underlying disk(s)

Increased the size of the vmware disk than adding a new disk in the virtual machine settings in vCenter.

Now,

Make the new sizes/partition visible on the system without reboot or taking the volume offline:

Extend the LV:

Resize FS:

Allow SSH and Web Connections in IP Tables in Linux

To Allow web and ssh connections in IP Tables

SSH and web both require out going messages on established tcp connections.

iptables -A OUTPUT -o eth0 -m state –state ESTABLISHED,RELATED -j ACCEPT

Then you need to allow incomming connections on port 80 and 22 and possibly 443

iptables -A INPUT -p tcp -i eth0 –dport 22 –sport 1024:65535 -m state –state NEW -j ACCEPTiptables -A INPUT -p tcp -i eth0 –dport 80 –sport 1024:65535 -m state –state NEW -j ACCEPTiptables -A INPUT -p tcp -i eth0 –dport 443 –sport 1024:65535 -m state –state NEW -j ACCEPT

To allow a DNS server to operate use the following rules (assuming your blocking inbound and outbound in iptables)

DNS communicated in to destination port 53 but can come from any port in the upper range. So these rules require a large section of ports to allow access as long as they want to talk to 53.

iptables -A OUTPUT -p udp –dport 53 –sport 1024:65535 -j ACCEPTiptables -A INPUT -p udp –dport 53 –sport 1024:65535 -j ACCEPT

migratepv VS replacepv

what is the difference between migratepv and replacepv?

replacepv command simply moves all the logical partitions on one physical volume to another physical volume.  The command is designed to make it easy to replace a disk in a mirrored configuration.

migratepv command also very similar.

The biggest difference is that migratepv allows you to copy the LPs on a logical volume basis, not just on a physical volume basis. For example, if you have a disk that has two logical volumes on it and you want to reorganize and put each logical volume on a different disk, migratepv can do it.

migratepv -l lv01 hdisk1 hdisk2
migratepv -l lv02 hdisk1 hdisk3

In this case, the logical partitions from logical volume lv01 are moved from hdisk1 to hdisk2.
The logical partitions from logical volume lv02 are moved to hdisk3.

What is umask?

umask will be used for setting the default file creation permissions.When a file is created, its permissions are set by default, depending on the umask setting configured.

This value is usually set for all users in /etc/profile and can be obtained by typing command umask:

testuser$ umask
0022

The default umask value is usually 022. It is an octal number which indicates what rights will be removed by default to all newly created files by a user.For example, 022 indicates that write permissions will not be given to group and other.

For example, with umask value of 000, files get mode 666 and directories get mode 777. As a result, with a default umask value of 022, newly created files get a default mode 644 (666 - 022 = 644) and directories get a default mode 755 (777 - 022 = 755).

To change the umask value:

For example, if you want by default all the,

• New directories to get permissions rwxr-x----  (Mode: 750)
• New files to get permissions rw-r-----  (Mode 640)

You need to use a umask value which removes all rights to other, and write permissions to the group and the value is 027. The command to use to set this:

# umask 027

Changing default gateway in SuSE Linux

Adding default Gateway in Linux SUSE

To change the default route permanently in SuSe Linux, make an entry in /etc/sysconfig/network/routes file.

For example, to make 192.168.10.1 as default route, add the following line into /etc/sysconfig/network/routes file. 

default 192.168.2.1 - -

Using route command:

To route all the traffic via 192.168.1.1 gateway connected via eth1 network interface:

# route add default gw 192.168.1.1 eth1

To view routes configured:

# netstat -rn 

Setting Limits for Users in Linux

We can set useful limits for users which is useful to control the resource utilization in Linux. This can be configured in /etc/security/limits.conf. To activate these limits you need to add the following line to the bottom of /etc/pam.d/login file in your Linux server.

session required /lib/security/pam_limits.so.

Entries in limits.conf file have the following structure:

[username or @groupname]          [type resource]        [limit]

Group names must be preceded by the @ to distinguish them from usernames.
The type must be either soft or hard. Soft-limits can be exceeded and are usually warning marks whereas hard-limits cannot be exceeded. A resource type can be one of the following keywords:

core	Limits the size of a core file (KB).
data	Maximum data size (KB).
fsize	Maximum file size (KB).
memlock	Maximum locked in memory address space (KB).
nofile	Maximum number of open files.
rss	Maximum resident set size (KB).
stack	Maximum stack size (KB).
cpu	Maximum CPU time in minutes.
nproc	Maximum number of processes.
as	Address space limit.
maxlogins	Maximum number of logins allowed for this user.

Below is example of limits.conf file. In this example, oracle user set to memlock limit of 12582912 and all the users in the server set with nproc and nofile limits.

# Added for Oracle Database Server

* soft nproc 2047

* hard nproc 16384

* soft nofile 1024

* hard nofile 65536

oracle soft memlock 12582912

oracle hard memlock 12582912

#*               soft    core            0

#*               hard    rss             10000

#@student        hard    nproc           20

#@faculty        soft    nproc           20

#@faculty        hard    nproc           50

#ftp             hard    nproc           0

#@student        -       maxlogins       4

# End of file

Special Shell Variables - Useful for Scripting

Below are the special shell variables. These are important to know for everyone, especially who is willing to learn shell scripting. Hope the list helps.

Name	Description
$1 - $9	These variables are referring the parameters passed to command or script. $1 refers to the first argument and $2 refers second and so on.
$0	The name of the command or script currently being executed.
$#	The number of arguments passed to the command/script or invocation of the shell.
$?	The exit status of the last command executed is given as a decimal string.  When a command completes successfully, it returns the exit status of 0 (zero), otherwise it returns a non-zero exit status.
$$	The process number of the currently executing command or script. - Useful for including in filenames, to make them unique.
$!	The process ID of the last command runs in the background.
$-	The current options supplied to the command or script.
$*	A string containing all the arguments passed to the command/script or shell, starting at $1. When quoted, "$*" is a single word, comprising all the arguments to the shell, joined together with spaces. For example ‘a b' c becomes "a b c".
$@	Same as above, except when quoted. When quoted, "$@" is identical to the arguments received by the shell, the resulting list of words completely match what was given to the shell. For example '1 2' 3 becomes "1 2" "3"

Apart from these, there are some standard variables which are set through .profile or .bashrc. Try "env" command to view all of them.

Network cards changed to '__tmpXXXX' instead of ethX after reboot in Linux

We have faced a strange issue recently and would like to share that to LazySystemAdmin readers.

Problem / Issue:

After upgrading the kernel, we rebooted the server. After reboot, some of the Ethernet network cards (NIC) are renamed to '__tmpxxxx' instead of ethX. Ethernet  interface keeps changing into  '__tmpxxxx' even after two more reboot of the server.

"ifconfig -a" output is like below: Invalid network interface names after reboot

root:testsrv1#  ifconfig -a | grep HW
__tmp1428126851 Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:53
__tmp1516900339 Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:52
__tmp1854964292 Link encap:Ethernet  HWaddr 78:E7:D1:FB:B1:2F
__tmp1950613216 Link encap:Ethernet  HWaddr 78:E7:D1:FB:B1:2E
bond0     Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:50
eth0      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:50
eth1      Link encap:Ethernet  HWaddr 68:B5:99:B4:9F:E8
eth4      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:50
eth5      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:51

While checking the backup files which were taken before reboot. Network cards looks like below:

root:testsrv1# cd /root/backup-testsrv1-24-05-12/
root:testsrv1# grep HW  network-interfaces.24-05-12
bond0     Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:50
eth0      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:50
eth1      Link encap:Ethernet  HWaddr 68:B5:99:B4:9F:E8
eth2      Link encap:Ethernet  HWaddr 78:E7:D1:FB:B1:2E
eth3      Link encap:Ethernet  HWaddr 78:E7:D1:FB:B1:2F
eth4      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:50
eth5      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:51
eth6      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:52
eth7      Link encap:Ethernet  HWaddr 1C:C1:DE:72:4D:53

Why this happens? what is the root cause?

It is because of the behavior of udev. udev does not load modules sequentially but loads the modules in parallel. You will get non-deterministic Ethernet device ordering if you have Multiple Network Drivers in the machine. It's inevitable.

So, it is required to use "HWADDR= " in the ifcfg files to accomplish that mapping.

How to fix this issue?

To prevent this from occurring, the "HWADDR=" parameter should be used in /etc/sysconfig/network-scripts/ifcfg-ethX. You should mention the HWADDR in the ifcfg-eth* files or remove any "#" in start of that line and restart the network service.

# vi /etc/sysconfig/network-scripts/ifcfg-ethX
# service network restart

Interview Questions and Answers - Linux Administrator - 3

What is the major difference between ext2 and ext3 file systems?

The main difference between ext2 and ext3 is, ext3 allows journaling. (Journaling is a type of log file, which tracks all the file system changes. so that you can recover in case of filesystem crash)

Explain Linux Booting Process?

• When the computer is switched on, it automatically invokes BIOS [a ROM chip embedded in the motherboard].

• The BIOS will start the processor and perform a POST [power on self test] to check whether the connected device are ready to use and are working properly.

• Once the POST is completes BIOS will check for the booting device. The boot sector is always the first sector of the hard disk and BIOS will load the MBR into the memory. MBR holds the boot loader of the OS.

• From here the boot loader takes the control of the booting process.

• GRUB is the boot loader for Linux. 

• Depending on the boot option selected the kernel is loaded first.

• After kernel is loaded the kernel will take the control of the booting process

• Initrd will be loaded which contains drivers to detect hardware (Initialization of RAM Disk)

• Then it will initialize all the hardware including I/O processors etc.

• Kernel then mounts the root partition as read-only

• INIT is loaded as the first process.

• INIT will mount the root partition and other partitions as read/write and checks for file system errors.

• Sets the System Clock, hostname etc..

• Based on the Runlevel, it will load the services and runs the startup scripts which are located in /etc/rcX.d/ (Network, cups, nfs, SSH etc.)

• Finally it runs the rc.local script.

• Now the login prompt will appear.

How do you extend a LV? For example, How do you expand /var file system with additional 2GB space?

• Check which logical volume (LV) holds the  /var file system using df –h
• Now, find out this particular LV belongs to which VG using lvdisplay <lv_name>
• Check the free space available in that Volume Group (VG) using vgdisplay <vgname>. Look for "Free PE / Space" line in the ouput.
• If the free space available in VG, now you can expand the LV using lvextend –L +2G <lv_name>. Now, Logical Volume has been expanded. Now we have to expand the file system using resize2fs /var. All these can be done in online without unmounting the filesystems.

How do you find out hardware errors inside Linux?

• dmesg
• /var/log/messages
• dmidecode –t system 
• IML (Integrated Management Logs) - An iLO console feature
• hpacucli - To check RAID array status
• use grep or less commands on

• /var/log/messages and /var/log/warn
• /var/log/debug
• /var/log/kern.log
• /var/log/mcelog

How do find out what are the files inside an RPM before installing it?

rpm -qlp package.rpm  (Example : $ rpm -qlp rpm -qlp gnupg-1.4.5-1.i386.rpm)

kblockd Process - High Utilization in Linux - Quick View

I  have came across the situation where multiple kblockd process are utilizing the CPU heavily and causing server load high in Linux servers. I was wondered what is kblockd and why it is taking high CPU utilization. But there is not much information available in internet about kblockd. After lot of  research, the below is what I learned.


What is kblockd?

In a general, the kblockd kernel threads are responsible for performing low-level disk operations.


Why kblockd processes are heavily utilizing the CPU which causing server load?

A high utilization of these could indicate that the server IO queue is backed up and the server is not managing to perform its disk writes quick enough. Most of the times, At that point, the SD drivers fails the IO and fails it to EXT3, which then aborts the journal for safety reasons. So when it is middle of transaction, it forgets about it and retrying to rollback. Also the kblockd message is a symptom of  "server running low on memory and starting to fail normal kernel memory allocations".

Most of the times this is a sign of pathological behavior by the kernel or merely a symptom of an overloaded server, depends on the workload of the server and its hardware. Please note that there is always a potential for hangs when something can't allocate memory.


What can be done to resolve this issue?

For now, I don't find any immediate resolution to fix this issue. However keeping your kernel version and block device driver modules up to date might help fixing this issue. Also Upgrading the Linux server to the latest available service pack level is recommended. These might fix any known bugs in earlier versions.

Disable Ping on Linux Server - Quick HOWTO

How do you disable ping to Linux server? Here is the quick steps:

 To disable ping:

echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all

To enable ping:

echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all

That's all..! 

Change Username Max Length in AIX - Quick HOWTO

In AIX 5.3,

To change the default length of username:

# chdev -l sys0 -a max_logname=9
sys0 changed

To check the current length of username:

# lsattr -El sys0 -a max_logname
max_logname 9 Maximum login name length at boot time True
# getconf LOGIN_NAME_MAX 21

Hope this quick tip helps someone..!

Find WWN and WWPN in Solaris

How do you find out WWN and WWPN of HBA in Solaris server?

Here is the quick way to get wwnn and wwpn in Solaris:

testuser@testsunsrv # /usr/sbin/prtconf -vp | grep -i wwn
            port-wwn:  210000e0.4m1e143f
            node-wwn:  200000e0.4m1e143f
            port-wwn:  210100e0.4m3e143f
            node-wwn:  200000e0.4m3e143f
            port-wwn:  21000003.fc2f1c36
            node-wwn:  20000003.fc2f1c36
            port-wwn:  210000e0.4m103794
            node-wwn:  200000e0.4m103794
            port-wwn:  210100e0.4m303794
            node-wwn:  200000e0.4m303794
            port-wwn:  210000e0.4m0a575b
            node-wwn:  200000e0.4m0a575b
            port-wwn:  210100e0.4m2a575b
            node-wwn:  200000e0.4m2a575b
            port-wwn:  210000e0.4m0a55ab
            node-wwn:  200000e0.4m0a55ab
            port-wwn:  210100e0.4m2a55ab
            node-wwn:  200000e0.4m2a55ab

testuser@testsunsrv #  prtpicl -v -c scsi-fcp | grep -i wwn
  :node-wwn      20  00  00  e0  4m  1e  14  3f
  :port-wwn      21  00  00  e0  4m  1e  14  3f
  :node-wwn      20  00  00  e0  4m  3e  14  3f
  :port-wwn      21  01  00  e0  4m  3e  14  3f
  :node-wwn      20  00  00  03  fc  2f  1c  36
  :port-wwn      21  00  00  03  fc  2f  1c  36
  :node-wwn      20  00  00  e0  4m  10  37  94
  :port-wwn      21  00  00  e0  4m  10  37  94
  :node-wwn      20  00  00  e0  4m  30  37  94
  :port-wwn      21  01  00  e0  4m  30  37  94
  :node-wwn      20  00  00  e0  4m  0a  57  5b
  :port-wwn      21  00  00  e0  4m  0a  57  5b
  :node-wwn      20  00  00  e0  4m  2a  57  5b
  :port-wwn      21  01  00  e0  4m  2a  57  5b
  :node-wwn      20  00  00  e0  4m  0a  55  ab
  :port-wwn      21  00  00  e0  4m  0a  55  ab
  :node-wwn      20  00  00  e0  4m  2a  55  ab
  :port-wwn      21  01  00  e0  4m  2a  55  ab
testuser@testsunsrv #

Just grep for "port-wwn" in the above command to see only WWPN. In most of the cases, WWPN is the required information for storage zoning.

testuser@testsunsrv #  prtpicl -v -c scsi-fcp | grep -i port-wwn

 Hope this helps..!

hpacucli - Check RAID Information from Linux Shell

Have you ever tried to check how the hardware RAID Array configured on server from your Linux Shell? Have you ever wanted to change or modify your Hardware RAID configurations without rebooting the server and without leaving your Linux shell?

hpacucli utility is there to help you, If your server is HP Hardware. hpacucli  (HP Array Configuration Utility CLI) is a command line based disk configuration program for Smart Array Controllers and RAID Array Controllers. You can download and  install hpacucli tool from HP website.

Quick Abbreviations:

chassisname = ch
controller = ctrl
logicaldrive = ld
physicaldrive = pd
drivewritecache = dwc

As root, just type "hpacucli" and you will be into hpacucli command line interface. Let me give you a quick example of what you can do with this hpacucli.

To Get the quick details about the RAID controller and its Health:

=> ctrl all show status

Smart Array P400 in Slot 9
   Controller Status: OK
   Cache Status: OK
   Battery Status: OK

=>

To get a quick idea of How the disks are grouped and which raid level used:

=> ctrl all show

Smart Array P400 in Slot 9           (sn: PXXXXXXXXXXXXX)

=> ctrl all show config

Smart Array P400 in Slot 9           (sn: P6YYYYYYYYYYYY)

   array A (SAS, Unused Space: 0 MB)

      logicaldrive 1 (68.3 GB, RAID 1, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)

To Get complete details about how the raid configured in the server:

=> ctrl all show config detail

Smart Array P400 in Slot 9
   Bus Interface: PCI
   Slot: 9
   Serial Number: PXXXXXXXXX
   Cache Serial Number: PAXXXXXXXXT
   RAID 6 (ADG) Status: Enabled
   Controller Status: OK
   Chassis Slot:
   Hardware Revision: Rev D
   Firmware Version: 7.08
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Accelerator Ratio: 25% Read / 75% Write
   Drive Write Cache: Disabled
   Total Cache Size: 512 MB
   Battery Pack Count: 1
   Battery Status: OK
   SATA NCQ Supported: True

   Array: A
      Interface Type: SAS
      Unused Space: 0 MB
      Status: OK

      Logical Drive: 1
         Size: 68.3 GB
         Fault Tolerance: RAID 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 17562
         Stripe Size: 128 KB
         Status: OK
         Array Accelerator: Enabled
         Unique Identifier: 600508XXXXXXXXXXXXXXXX0002
         Disk Name: /dev/cciss/c0d0
         Mount Points: /boot 103 MB, swap 8.0 GB
         Logical Drive Label: A08923XXX61630G9SVI3RJCC0A
         Mirror Group 0:
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
         Mirror Group 1:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 72 GB
         Rotational Speed: 10000
         Firmware Revision: HPDA
         Serial Number:         PXXXXXXA
         Model: HP      DG072A4951
         PHY Count: 1
         PHY Transfer Rate: Unknown
      physicaldrive 1I:1:2
         Port: 1I
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 72 GB
         Rotational Speed: 10000
         Firmware Revision: HPDA
         Serial Number:         PXXXXXXA
         Model: HP      DG072A4951
         PHY Count: 1
         PHY Transfer Rate: Unknown

=>

 Be sure to verify your version of hpacucli and refer the ReadMe always, before you trying to modify the configuration of RAID or Smart Array controllers.